modelzoo.transformers.pytorch.gpt2.input.InferenceDataProcessor

Functions

get_token_ids

Get encoded token ids from a string using the specified tokenizer.

Classes

InferenceDataProcessor

previous

modelzoo.transformers.pytorch.gpt2.input.HuggingFaceIterableDataProcessorEli5.HuggingFaceIterableDataProcessorEli5

next

modelzoo.transformers.pytorch.gpt2.input.InferenceDataProcessor.get_token_ids