examples : add tokenization tests and refactor codes (#186)

examples : add tokenization tests and refactor codes (#186)

* examples : [refactor] remove unnecessary lines and segments

* examples : [feature] add tokenization test for gpt-neox

* examples : [feature] handle multibyte character set

* examples : [refactor] find the longest token for word

* examples : [refactor] move test_tokenizer to common.cpp as the function affects other models

* add 'test_tokenizer' function after loading the model

* examples : [feature] add test cases for checking tokenization

* examples : [feature] tokenize with huggingface tokenizers for currently supported models

* examples : add tokenization test cases for each model

* revert conversion from string to utf-8 encoded byte strings

* [refactor] make util functions for testing tokenizers available

* [bug fix] test replit using functions and variables (e.g. tokenizer struct, tokenization method) defined in its main.cpp

* [refactor] modify function name test_tokenizer -> test_gpt_tokenizer

* [refactor] put parenthesis on single line for-loops and if-statements

* [refactor] withdraw <filesystem> and use <iostream> and <dirent.h>

* [refactor] remove 'find_test_file' function and directly set test file path from 'test_gpt_tokenizer' function

* call a function for testing tokenizer with filename specified

* revert test tokenizer in replit (replit uses seperate methods for tokenzation and decoding)

* compare vector of id to check if two tokenizations are identical.

* write token ids instead of strings.

* [refactor] use --token_test rather than --test for token-test argument

* add english test cases

* update test cases with more english prompts

* examples : tokenizer testing fixes

---------

Co-authored-by: Georgi Gerganov <redacted>