Troubleshooting

• Cannot load Cerebras checkpoints in GPUs
  • Work around
• Custom PT training script spawns multiple compile jobs
  • Observed Error
  • Explanation
  • Work around
• Loss compilation issues with Autogen
  • Custom loss functions with AutoGen
  • Improving loss function performance
• Error parsing metadata
  • Observed Error
  • Explanation
  • Work around
• Error Receiving Activation
  • cerebras.appliance.errors.ApplianceUnknownError: Ran into error while receiving activation tensor <custom-call …>
• Failed mount directory during execution
  • Observed Error
  • Work around
• Failing to automatically load checkpoints
  • Explanation
  • Work around
• Failure to trace due to functionalization error
  • Observed Error
  • Explanation
  • Work around
• Input Starvation
• Out of memory errors and system resources
  • Determining if your job is queued
  • Determining if job failed because of an OOM error
  • Determining if job failed because of system could not fit requested memory
  • Troubleshooting OOM errors
• Model is too large to fit in memory
  • Observed Error
  • Causes and Possible Solutions
• ModuleNotFoundError
  • ModuleNotFoundError: No module named <’_bz2’, ‘_sqlite3’>
  • ModuleNotFoundError: No module named <…>
• Numerical issues
  • Observed Error
  • Explanation
  • Work around
• Throughput spike after saving checkpoints
• Training fails when logged-in as root
  • Observed Error
  • Explanation
• Vocabulary Size Troubleshooting
  • Large vocabulary size
  • Small vocabulary size

previous

Optimizing performance with automatic microbatching

next

Cannot load Cerebras checkpoints in GPUs