modelzoo.transformers.data_processing.scripts.hdf5_preprocessing.utils.add_lm_args

modelzoo.transformers.data_processing.scripts.hdf5_preprocessing.utils.add_lm_args(parser)

The language-modeling format is common enough (FIM is very similar) that we can re-use the arguments for it

previous

modelzoo.transformers.data_processing.scripts.hdf5_preprocessing.utils.add_common_args

next

modelzoo.transformers.data_processing.scripts.hdf5_preprocessing.utils.check_fim_special_tokens