cerebras.modelzoo.data_preparation.nlp.t5.utils.shuffle#

cerebras.modelzoo.data_preparation.nlp.t5.utils.shuffle(dataset, shuffle_buffer, rng)[source]#

Perform a buffered shuffle on an iterator.

Parameters

  • dataset (iterable) – The iterable to shuffle.

  • shuffle_buffer (int) – The number of elements to buffer for the shuffle.

  • rng (random.Random) – The random generator used as a source of randomness for shuffling.

Yields

Elements of the original iterable in a different order.

previous

cerebras.modelzoo.data_preparation.nlp.t5.utils.select_random_chunk

next

cerebras.modelzoo.data_preparation.nlp.t5.utils.split_sequences