modelzoo.transformers.data_processing.tokenizers.BPETokenizer.get_pairs#

modelzoo.transformers.data_processing.tokenizers.BPETokenizer.get_pairs(word)[source]#

Return set of symbol pairs in a word.

Word is represented as tuple of symbols (symbols being variable-length strings).

previous

modelzoo.transformers.data_processing.tokenizers.BPETokenizer.bytes_to_unicode

next

modelzoo.transformers.data_processing.tokenizers.BPETokenizer.BPETokenizer