Huggingface Tokenizers

Fast BPE/WordPiece tokenization and custom vocab training.

By Orchestra Research · Official Docs · Date unknown

Fast, production-ready tokenizers with Rust performance and Python ease-of-use.

Platforms: linux, macos, windows.

hermes skills install official/mlops/huggingface-tokenizers

• #skills
• #optional
• #mlops
• #tokenization
• #huggingface
• #bpe
• #wordpiece