sharpetronics/gpt4all
1
0
Fork 0
mirror of https://github.com/nomic-ai/gpt4all.git synced 2025-11-18 00:13:03 -05:00
Code Issues Projects Releases Wiki Activity
840 Commits 42 Branches 71 Tags
Commit Graph

2 Commits

Author SHA1 Message Date
Aaron Miller
d59c77ac55 buf_ref.into() can be const now 2023-05-30 12:05:57 -04:00
Aaron Miller
bbcee1ced5 New tokenizer implementation for MPT and GPT-J 
Improves output quality by making these tokenizers more closely
match the behavior of the huggingface `tokenizers` based BPE
tokenizers these models were trained with.

Featuring:
 * Fixed unicode handling (via ICU)
 * Fixed BPE token merge handling
 * Complete added vocabulary handling
 2023-05-30 12:05:57 -04:00