Running jobs in parallel with the same model_dir causes issues#
Running several jobs in parallel with the same
model_dir may cause issues.
model_dir is used in multiple parallel runs, it could cause a race condition (two operations being performed on the same object at the same time) as each run attempts to write their file-backed tensors to the same directory.
Use different model directories, or
Add a unique artifact sub-directory by generating a