Lib.rs

› Keywords #bpe #llm #openai #hugging-face #nlp #tokenize #chat-template

#tiktoken


  1. llm-tokenizer

    LLM tokenizer library with caching and chat template support

    v1.4.2 232K #chat-template #tiktoken #hugging-face #llm

  2. tiktoken

    A high-performance pure-Rust implementation of OpenAI's tiktoken BPE tokenizer

    v3.5.1 5.2K #bpe #openai #tokenizer

  3. riptoken

    Fast BPE tokenizer for LLMs — a faster, drop-in compatible reimplementation of tiktoken

    v0.3.0 49K #tiktoken #bpe #llm #nlp

  4. dynamo-tokenizers

    Standalone tokenizer implementations (HuggingFace, tiktoken, FastTokenizers) for LLM inference serving

    v1.5.2 61K #inference #tiktoken #llm #hugging-face #tokenizer

  5. miktik

    A unified, multi-backend tokenizer library for LLMs

    v0.2.0 420 #tiktoken #llm #hugging-face #rust #tokenizer

  6. skimtoken

    Fast token count estimation library

    v0.2.2 180 #token-count #estimation #python #parameters #memory #startup-time #tiktoken #statistical-model

  7. toktrie_tiktoken

    tiktoken (OpenAI BPE) library support for toktrie and llguidance

    v1.7.6 #tiktoken #structured-output #bpe #openai #llguidance #toktrie #constrained-decoding #context-free-grammar #json-schema #llama

  8. mkcontext

    that provides functionality for creating context

    v0.7.3 800 #globs #context #run-command #limit #exclude #tiktoken #glob-pattern #chatgpt #clipboard #command-line-tool

  9. tiktoken-wasm

    WASM bindings for the tiktoken BPE tokenizer

    v3.5.1 #tiktoken #wasm-bindings #bpe #model-info #model-name #string-encoding #model-provider #wasm-memory #pricing

  10. tt-tokenize

    Shared token estimation (tiktoken cl100k + heuristic fallback) used by preview, dispatch, and routing

    v0.1.0 #tiktoken #estimation #preview #openai #routing #heuristics #anthropic #gemini #bpe #medium

  11. rustbpe

    A BPE (Byte Pair Encoding) tokenizer written in Rust with Python bindings

    v0.1.0 #bpe #python-bindings #tiktoken #training #gpt-4 #tokenizer

  12. wordchipper-cli-util

    Wordchipper CLI

    v0.9.2 #tokenize #wordchipper #lexer #cli #python-bindings #bpe #tiktoken #gpt-2 #openai

  13. loctok

    Count LOC (lines of code) & TOK (LLM tokens), fast

    v0.2.0 #token #llm #loc #tiktoken #lines-of-code

  14. toklab-core

    Pure-Rust core for toklab: bulk tokenizer + counter for OpenAI BPE encodings

    v0.1.1 #bpe #tiktoken #openai #llm

  15. tokenmonster

    Greedy tiktoken-like tokenizer with embedded vocabulary (cl100k-base approximator)

    v0.1.0 #tokenize #tiktoken #nlp #tokenizer

    16. Try searching with DuckDuckGo.


  16. chat-splitter

    Never exceed OpenAI's chat models' maximum number of tokens when using the async_openai Rust crate

    v0.1.1 #chat #openai #chat-completion #max #split #exceed #tiktoken

  17. tokin

    Experimental fast tokenizer

    v0.1.0 #word-piece #tokenize #nlp #tiktoken #bpe #tokenizer