cerebras.modelzoo.data_preparation.nlp.hdf5_preprocessing.utils.create_features_auto_lm_vsl#

cerebras.modelzoo.data_preparation.nlp.hdf5_preprocessing.utils.create_features_auto_lm_vsl(bin, max_sequence_length, num_pad, pad_id=0, inverted_mask=False, input_ids_dtype='int32', input_mask_dtype='int32', labels_dtype='int32', attention_span_dtype='int32', position_ids_dtype='int32')[source]#

Given a list of VSL sequences, generate input features and labels.

Parameters

  • bin (list(sequence)) – list of VSL sequences.

  • max_sequence_length (int) – Maximum sequence length for data writes.

  • num_pad (int) – number of padding tokens in the sequence.

  • pad_id (int) – Id for pad token. Defaults to 0.

  • inverted_mask (bool) – Invert mask if specified for runtime execution. Defaults to False.

  • input_ids_dtype (str) – Dtype as string for input ids. Defaults to int32.

  • input_mask_dtype (str) – Dtype as string for input mask. Defaults to int32.

  • labels_dtype (str) – Dtype as string for labels. Defaults to int32.

  • attention_span_dtype (str) – Dtype as string for keys attention span in VSL. Defaults to int32.

  • position_ids_dtype (str) – Dtype as string for position ids and attention span in VSL. Defaults to int32.

Returns

Tuple containing features and labels

previous

cerebras.modelzoo.data_preparation.nlp.hdf5_preprocessing.utils.create_features_auto_lm

next

cerebras.modelzoo.data_preparation.nlp.hdf5_preprocessing.utils.create_features_llava_phase1