modelzoo.transformers.data_processing.slimpajama.dedup.to_hash#

Functions

generate_hashes

get_documents

get_features

output_results

to_minhash

previous

modelzoo.transformers.data_processing.slimpajama.dedup.generate_duplicates_dict.generate_duplicates

next

modelzoo.transformers.data_processing.slimpajama.dedup.to_hash.generate_hashes