cerebras.modelzoo.data_preparation.nlp.bert.bertsum_data_processor.Tokenizer

class cerebras.modelzoo.data_preparation.nlp.bert.bertsum_data_processor.Tokenizer(params)

Bases: object

Tokenizes files from the input path into output path. Stanford CoreNLP is used for tokenization. :param params: dict params: Tokenizer configuration parameters.

Methods

process

previous

cerebras.modelzoo.data_preparation.nlp.bert.bertsum_data_processor.RougeBasedLabelsFormatter

next

cerebras.modelzoo.data_preparation.nlp.bert.create_csv