modelzoo.transformers.data_processing.bert.bertsum_data_processor.Tokenizer#

class modelzoo.transformers.data_processing.bert.bertsum_data_processor.Tokenizer[source]#

Bases: object

Tokenizes files from the input path into output path. Stanford CoreNLP is used for tokenization. :param params: dict params: Tokenizer configuration parameters.

Methods

process

__init__(params)[source]#

Tokenizes files from the input path into output path. Stanford CoreNLP is used for tokenization. :param params: dict params: Tokenizer configuration parameters.

previous

modelzoo.transformers.data_processing.bert.bertsum_data_processor.RougeBasedLabelsFormatter

next

modelzoo.transformers.data_processing.bert.dynamic_processor