cerebras.modelzoo.data_preparation.nlp.hdf5_preprocessing.utils.create_features_summarization#
- cerebras.modelzoo.data_preparation.nlp.hdf5_preprocessing.utils.create_features_summarization(prompt_ids, completion_ids, max_sequence_length, eos_id=0, sep_id=None, pad_id=0, min_len=10, inverted_mask=False, input_ids_dtype='int32', input_mask_dtype='int32', labels_dtype='int32')[source]#
Given a list of prompt_ids and completion_ids, generate input sequence and labels.
- Parameters
prompt_ids (sequence) – List containing token ids for the prompt to create features,labels and input mask from.
completion_ids (sequence) – List containing token ids for the completion create features,labels and input mask from.
max_sequence_length (int) – Maximum sequence length for data writes.
eos_id (int) – Id for end of sequence token. Defaults to 0.
sep_id (int) – Id for separator token. Defaults to None.
pad_id (int) – Id for pad token. Defaults to 0.
min_len (int) – Minimum length of token_ids to be considered a valid sequence.
inverted_mask (bool) – Invert mask if specified for runtime execution. Defaults to False.
input_ids_dtype (str) – Dtype as string for input ids. Defaults to int32.
input_mask_dtype (str) – Dtype as string for input mask. Defaults to int32.
labels_dtype (str) – Dtype as string for labels. Defaults to int32.