Troubleshooting

• Cannot load Cerebras checkpoints in GPUs
  • Work around
• Custom PT training script spawns multiple compile jobs
  • Observed Error
  • Explanation
  • Work around
• Enable kernel generalizability with Autogen
  • Example use cases
• Error parsing metadata
  • Observed Error
  • Explanation
  • Work around
• Error Receiving Activation
  • cerebras_appliance.errors.ApplianceUnknownError: Ran into error while receiving activation tensor <custom-call …>
• Failed mount directory during execution
  • Observed Error
  • Work around
• Failing to automatically load checkpoints
  • Explanation
  • Work around
• Failure to trace due to functionalization error
  • Observed Error
  • Explanation
  • Work around
• Input Starvation
• Out of memory errors and system resources
  • Determining if your job is queued
  • Determining if job failed because of an OOM error
  • Determining if job failed because of system could not fit requested memory
  • Troubleshooting OOM errors
• Model is too large to fit on the device
  • Observed Error
• ModuleNotFoundError
  • ModuleNotFoundError: No module named <’_bz2’, ‘_sqlite3’>
  • ModuleNotFoundError: No module named <…>
• Throughput spike after saving checkpoints
• Training fails when logged-in as root
  • Observed Error
  • Explanation
• Vocabulary Size Troubleshooting
  • Large vocabulary size
  • Small vocabulary size

previous

Train with gradient accumulation

next

Cannot load Cerebras checkpoints in GPUs