Running jobs in parallel with the same model_dir causes issues

Observed error

Running several jobs in parallel with the same model_dir may cause issues.

Explanation

When a model_dir is used in multiple parallel runs, it could cause a race condition (two operations being performed on the same object at the same time) as each run attempts to write their file-backed tensors to the same directory.

Workaround

• Use different model directories, or

• Add a unique artifact sub-directory by generating a uuid.