modelzoo.transformers.data_processing.scripts.hdf5_preprocessing.utils.fim#

modelzoo.transformers.data_processing.scripts.hdf5_preprocessing.utils.fim(sample_array, sample_idx, tokenizer, fim_rate, spm_rate, suffix_tok_id, prefix_tok_id, middle_tok_id, fim_pad_tok_id, eos_tok_id, opt_bos_tok_id)[source]#

Takes in an array of input_ids, mask, and labels, and performs the FIM operation to re-arrange into PSM and SPM format with some probability

Parameters
  • sample_array (np.array) – Stack of input_ids, mask, and labels after tokenization. Labels are off-by-one of input_ids

  • training (as in standard auto-regressive) –

  • i (int) – Index of sample from dataset, used for logging.

  • tokenizer (Tokenizer) – Tokenizer object

  • fim_rate (float) – Determines what percentage of contexts are FIM’ed

  • spm_rate (float) – Determines what percentage of FIM’ed contexts are in SPM format. 1 - spm_rate determines PSM

  • suffix_tok_id (int) – Id for special token denoting suffix section in a FIM’ed context

  • prefix_tok_id (int) – Id for special token denoting prefix section in a FIM’ed context

  • middle_tok_id (int) – Id for special token denoting middle section in a FIM’ed context

  • fim_pad_tok_id (int) – Id for padding

  • eos_tok_id (int) – Id for the end-of-seqence

  • opt_bos_tok_id (list) – Optionally a list containing the bos token id, otherwise will be empty list. Empty list will be a no-op in the concatenation. Bos-token will only exist if model’s tokenizer adds bos-token by default.

Returns

Stack of input_ids, mask, and labels after FIM transformation. Mask and labels have been adjusted to still filter padding tokens and represent the following token, respectively.

Return type

fim_outputs (np.array)