modelzoo.transformers.pytorch.bert.fine_tuning.extractive_summarization.input.BertSumCSVDataProcessor.BertSumCSVDataProcessor#

class modelzoo.transformers.pytorch.bert.fine_tuning.extractive_summarization.input.BertSumCSVDataProcessor.BertSumCSVDataProcessor[source]#

Bases: torch.utils.data.IterableDataset

Reads csv file containing the input_token_ids, and label_ids. Creates attention_mask and segment_ids on the fly :param <dict> params: dict containing input parameters for creating dataset.

Methods

create_dataloader

Classmethod to create the dataloader object.

load_buffer

Generator to read the data in chunks of size of data_buffer.
__call__(*args: Any, **kwargs: Any) → Any#

Call self as a function.

__init__(params)[source]#
static __new__(cls, *args: Any, **kwargs: Any) → Any#
create_dataloader()[source]#

Classmethod to create the dataloader object.

load_buffer()[source]#

Generator to read the data in chunks of size of data_buffer.

Returns

Yields the data stored in the data_buffer.

previous

modelzoo.transformers.pytorch.bert.fine_tuning.extractive_summarization.input.BertSumCSVDataProcessor.create_bertsum_feature

next

modelzoo.transformers.pytorch.bert.fine_tuning.extractive_summarization.input.utils