cerebras.modelzoo.data.nlp.bert.BertSumCSVDataProcessor.BertSumCSVDataProcessor#

class cerebras.modelzoo.data.nlp.bert.BertSumCSVDataProcessor.BertSumCSVDataProcessor(*args, **kwargs)[source]#

Bases: torch.utils.data.IterableDataset

Reads csv file containing the input_token_ids, and label_ids. Creates attention_mask and segment_ids on the fly :param <dict> params: dict containing input parameters for creating dataset.

Methods

create_dataloader

Classmethod to create the dataloader object.

load_buffer

Generator to read the data in chunks of size of data_buffer.
create_dataloader()[source]#

Classmethod to create the dataloader object.

load_buffer()[source]#

Generator to read the data in chunks of size of data_buffer.

Returns

Yields the data stored in the data_buffer.

previous

cerebras.modelzoo.data.nlp.bert.BertSumCSVDataProcessor.create_bertsum_feature

next

cerebras.modelzoo.data.nlp.bert.BertTokenClassifierDataProcessor