awesome-tokenizers
github.com/nlpoptimize/awesome-tokenizers ↗A curated list of tokenizer libraries for blazing-fast NLP processing.
12
GitHub Stars
29
Curated Resources
3
Categories
23 hours ago
Last Refreshed
🔹 WordPiece Tokenizer Implementations🔹 BPE (Byte Pair Encoding) Implementations🔹 SentencePiece Implementations
Use this list with your AI agent
Add the Context Awesome MCP server to Claude, Cursor, or any MCP client, then ask:
"Show me 🔹 wordpiece tokenizer implementations resources from awesome-tokenizers"
Installation instructions →What's inside
🔹 WordPiece Tokenizer Implementations
🔹 BPE (Byte Pair Encoding) Implementations
🔹 SentencePiece Implementations
Showing a sample of 29 resources. View the full list on GitHub →