#
bpe
Here are 6 public repositories matching this topic...
Fast and versatile tokenizer for language models with BPE, Unigram and WordPiece tokenization. Compatible with SentencePiece, Tokenizers, Tiktoken and more.
-
Updated
Oct 1, 2024 - Rust
This crate is a rust porting of Andrej Karpathy implementation of Byte Pair Encoding (BPE) algorithm available here https://github.com/karpathy/minbpe
-
Updated
Feb 19, 2024 - Rust
Improve this page
Add a description, image, and links to the bpe topic page so that developers can more easily learn about it.
Add this topic to your repo
To associate your repository with the bpe topic, visit your repo's landing page and select "manage topics."