cerebras.modelzoo.data_preparation.nlp.write_csv_ner#

File: write_csv_ner.py

Use to create pre-processed CSV files for the Data Processor from the NER raw dataset CSV files.

Based on https://github.com/NVIDIA/DeepLearningExamples/blob/master/TensorFlow/LanguageModeling/BERT/run_ner.py with minor modifications

Example Usage:

python write_csv_ner.py –data_dir /cb/ml/language/datasets/blurb/data_generation/data/BC5CDR-chem/ –vocab_file /cb/ml/language/datasets/pubmed_abstracts_baseline_fulltext_vocab/Pubmed_fulltext_vocab.txt –output_dir /cb/ml/language/datasets/ner-pt/bc5cdr-chem-csv –do_lower_case

Functions

`convert_examples_to_features_and_write`
`convert_single_example`
`update_parser`	Add required command-line arguments.
`write_csv_files`

Classes

InputFeatures

Create new instance of InputFeatures(tokens, labels)

cerebras.modelzoo.data_preparation.nlp.write_csv_data.convert_to_bert_format_files

cerebras.modelzoo.data_preparation.nlp.write_csv_ner.convert_examples_to_features_and_write