modelzoo.transformers.pytorch.bert.input.utils.parse_text#

modelzoo.transformers.pytorch.bert.input.utils.parse_text(text, do_lower)[source]#

Postprocessing of the CSV file. This code should parse commas that are part of the token strings. :param: str text: String with an input text.

Returns

List of parsed tokens.

previous

modelzoo.transformers.pytorch.bert.input.utils.get_whole_word_span

next

modelzoo.transformers.pytorch.bert.input.utils.shard_and_shuffle_data