Lib.rs

› Keywords #nlp #bpe #llm #hugging-face #japanese #sql-parser #sql #word-piece

#tokenizer


  1. tokenizers

    today's most used tokenizers, with a focus on performances and versatility

    v0.23.1 3.0M #bpe #nlp #hugging-face #tokenize #tokenizer

  2. svgtypes

    SVG types parser

    v0.16.1 2.7M #svg-parser #svg #tokenizer

  3. xmlparser

    Pull-based, zero-allocation XML parser

    v0.13.6 8.2M #tokenize #xml #tokenizer

  4. charabia

    detect the language, tokenize the text and normalize the tokens

    v0.9.9 73K #tokenize #normalize #tokenizer

  5. sqlite3-parser

    SQL parser (as understood by SQLite)

    v0.17.0 165K #tokenize #sql-parser #sql #tokenizer #parser

  6. htmlparser

    Pull-based, zero-allocation HTML parser

    v0.2.1 179K #tokenize #tokenizer

  7. tiktoken

    A high-performance pure-Rust implementation of OpenAI's tiktoken BPE tokenizer

    v3.5.1 5.2K #bpe #openai #tokenizer

  8. html5gum

    A WHATWG-compliant HTML5 tokenizer and tag soup parser

    v0.8.4 49K #tokenize #whatwg #html5 #html-parser #tokenizer

  9. azul-simplecss

    A very simple CSS 2.1 tokenizer with CSS nesting support

    v0.2.0 54K #css-parser #css #nested #tokenizer

  10. splintr

    Fast Rust tokenizer (BPE + SentencePiece + WordPiece) with Python bindings

    v0.10.0 5.0K #bpe #llm #sentence-piece #tokenizer #word-piece

  11. scnr2

    Scanner/Lexer with regex patterns and multiple modes

    v0.5.2 14K #lexer #lexer-tokenizer #tokenizer

  12. vaporetto

    pointwise prediction based tokenizer

    v0.6.5 17K #japanese #tokenize #analyzer #morphological #tokenizer

  13. kitoken

    Fast tokenizer for language models, supporting BPE, Unigram and WordPiece tokenization

    v0.11.0 2.5K #bpe #unigram #nlp #tokenizer #word-piece

  14. lindera-tantivy

    Lindera Tokenizer for Tantivy

    v4.0.0 5.5K #tokenize #lindera #tantivy #tokenizer

  15. tok3niz3r-cli

    Command-line tool to train, run, and inspect byte-level BPE tokenizers. Installs the tok3 binary.

    v0.2.0 #bpe #nlp #tokenizer

  16. fastokens

    Fast Tokenizer

    v0.2.1 143K #bpe #python #llm #source #built #tokenizer

  17. vibrato-rkyv

    Vibrato: viterbi-based accelerated tokenizer with rkyv support for fast dictionary loading

    v0.7.10 130 #japanese #tokenize #morphological #analyzer #tokenizer

  18. dynamo-tokenizers

    Standalone tokenizer implementations (HuggingFace, tiktoken, FastTokenizers) for LLM inference serving

    v1.5.2 61K #inference #tiktoken #llm #hugging-face #tokenizer

  19. lexsim

    Dictionary-free, multilingual lexical similarity engine with hybrid word segmentation (tokenize + Jaccard + BM25 + TextRank + content hash)

    v0.6.0 210 #tokenize #bm25 #jaccard #nlp #similarity #tokenizer

  20. alyze

    High-performance text analysis for full-text search

    v0.1.5 2.5K #tokenize #nlp #analysis #unicode-segmentation #unicode #tokenizer

  21. tokie

    Blazingly fast tokenizer - 50x faster tokenization, 10x smaller model files, 100% accurate drop-in replacement for HuggingFace

    v0.0.10 #bpe #nlp #transformer #tokenize #tokenizer #word-piece

  22. octofhir-fhirpath-parser

    Parser and tokenizer for FHIRPath expressions

    v0.4.20 1.0K #tokenize #fhir #tokenizer #fhirpath

  23. bareun_rs

    an unofficial Rust library for Bareun, a Korean morphological analyzer

    v1.8.0 #korean #nlp #analyzer #api-bindings #tokenizer

  24. language-tokenizer

    Text tokenizer for linguistic purposes, such as text matching. Supports more than 40 languages, including English, French, Russian, Japanese, Thai etc.

    v0.3.0 12K #tokenize #text-tokenizer #language #tokenizer

  25. chaptr

    Filename tokenizer for manga, manhwa, manhua, and light novels

    v1.5.0 #manga #tokenize #filename #lightnovel #tokenizer

  26. utokenizer

    CLI tool for building a local model-tokenizer registry and counting input tokens across model families

    v0.1.1 #llm #cli #tokenizer

  27. tok3niz3r-trainer

    Trainer for byte-level BPE tokenizers: learns a vocabulary and merge table from a text corpus. The training half of tok3niz3r.

    v0.2.0 #bpe #training #nlp #tokenizer

  28. tantivy-stemmers

    A collection of Tantivy stemmer tokenizers

    v0.4.0 3.3K #stemming #tantivy #tokenizer

  29. xmltok

    Pull-based, zero-allocation XML tokenizer with compact, lifetime-free tokens. A fork of xmlparser.

    v0.14.0 #tokenize #xml-parser #xml #tokenizer

  30. bpe

    Fast byte-pair encoding implementation

    v0.2.1 12K #algorithm #encoding #tokenizer

  31. normy

    Ultra-fast, zero-copy text normalization for Rust NLP pipelines & tokenizers

    v0.1.4 #llm #nlp #zero-copy #normalization #tokenizer

  32. scnr2_generate

    Scanner/Lexer with regex patterns and multiple modes

    v0.5.2 13K #lexer #lexer-tokenizer #tokenizer

  33. trustformers-tokenizers

    Tokenizers for TrustformeRS

    v0.1.3 #bpe #nlp #tokenize #tokenizer

  34. vibrato

    viterbi-based accelerated tokenizer

    v0.5.2 3.6K #japanese #tokenize #tokenizer #morphological

  35. tokmat

    Standalone high-performance Canadian address parsing engine core

    v0.3.3 #tokenize #regex #address #address-parser #canada #tokenizer

  36. tokstream-cli

    CLI token stream simulator using Hugging Face tokenizers

    v0.1.2 #tokenize #streaming #cli #tokenizer

  37. libsimple

    Rust bindings to simple, a SQLite3 fts5 tokenizer which supports Chinese and PinYin

    v0.9.0 130 #sqlite-extension #fts5 #tokenizer

  38. oxibonsai-tokenizer

    Pure Rust BPE tokenizer for OxiBonsai (MeCrab-compatible)

    v0.2.2 #bpe #llm #nlp #wasm #tokenizer

  39. bbpe

    Binary byte pair encoding (BPE) trainer and CLI compatible with Hugging Face tokenizers

    v0.6.3 #bpe #malware #hugging-face #binary #tokenizer

  40. segtok

    Sentence segmentation and word tokenization tools

    v0.1.5 103K #tokenize #split #tokenizer #word

  41. token-calibrator

    Calibrated, self-learning token estimator for live context-size display

    v1.0.1 #llm #estimation #regression #tokenize #tokenizer

  42. unitoken

    Fast BPE tokenizer/trainer with a Rust core and Python bindings

    v0.1.4 #bpe #nlp #tokenize #tokenizer

  43. unobtanium-segmenter

    A text segmentation toolbox for search applications inspired by charabia and tantivy

    v0.5.2 #tokenize #text-segmentation #language #tokenizer

  44. tekken-rs

    Mistral Tekken tokenizer with audio support

    v0.1.1 50K #tokenize #mistral #nlp #artificial-intelligence #audio #tokenizer

  45. neurorvq-rs

    NeuroRVQ biosignal tokenizer — inference in Rust (Burn and/or RLX)

    v0.2.0 #tokenize #eeg #ecg #bci #emg #tokenizer

  46. libsql-sqlite3-parser

    SQL parser (as understood by SQLite) (libsql fork)

    v0.13.0 237K #tokenize #sql #sql-parser #parser #tokenizer

  47. miktik

    A unified, multi-backend tokenizer library for LLMs

    v0.2.0 420 #tiktoken #llm #hugging-face #rust #tokenizer

  48. mecab-ko

    한국어 형태소 분석기 - MeCab-Ko의 순수 Rust 구현

    v0.7.2 120 #tokenize #nlp #korean #mecab #morphology #tokenizer

  49. wordchipper

    HPC Rust LLM Tokenizer Library

    v0.9.2 140 #artificial-intelligence #bpe #tokenizer

  50. bpe-openai

    Prebuilt fast byte-pair encoders for OpenAI

    v0.3.0 11K #bpe #algorithm #encoding #tokenizer

  51. tok3niz3r-runtime

    Encoder and decoder for byte-level BPE tokenizers, with rkyv (zero-copy) and HuggingFace tokenizer.json loading. The runtime half of tok3niz3r.

    v0.2.0 #bpe #hugging-face #nlp #tokenizer

  52. rust_tokenizers

    High performance tokenizers for Rust

    v8.1.1 5.8K #nlp #machine-learning #tokenize #tokenizer

  53. tok3niz3r-core

    Core types and on-disk schema for the tok3niz3r byte-level BPE tokenizer: token ids, vocabulary, merge table, and the GPT-2 byte-level mapping

    v0.2.0 #bpe #nlp #tokenizer

  54. spider-markup5ever

    Common code for xml5ever and html5ever (Send-able fork via spider-tendril)

    v0.39.0 7.3K #html5ever #send #html #concurrency #tokenizer

  55. svgrtypes

    SVG types parser

    v0.45.1 #svg-parser #tokenize #svg #tokenizer

  56. nlpo3

    Thai natural language processing library, with Python and Node bindings

    v1.4.0 1.0K #nlp #tokenize #thai #word-segmentation #tokenizer

  57. tokid

    Token-native IDs for LLM-facing systems

    v0.1.0-alpha.4 #llm #identifier #tokenizer

  58. html5tokenizer

    An HTML5 tokenizer with code span support

    v0.5.2 #html #html5 #whatwg #html-parser #tokenizer

  59. tokelang-core

    Compression engine for Tokelang Lite: English-prompt compression middleware over standard tokenizers, with a content-recall validator and safe passthrough fallback

    v0.9.6 #llm-prompt #llm #compression #nlp #nlp-prompt #tokenizer

  60. wordchipper-cli

    A command-line interface for the Wordchipper tokenizer library

    v0.9.2 #llm #model #command-line-interface #cat #suite #multi-tool #tokenizer #gpt-2 #vocabulary #user-facing

  61. english-pos-tagger

    part-of-speech tagging and lemmatization of English text

    v0.1.0 #part-of-speech #nlp #pos-tagging #part-of-speech-tagging #lemmatizer #tokenizer

  62. limbo_sqlite3_parser

    SQL parser (as understood by SQLite)

    v0.0.22 3.4K #tokenize #sql-parser #sql #parser #tokenizer

  63. ailed-soulsteal

    Fast game data extractor — drains PGN into tokenized training data for ML models. 60K+ games/sec.

    v26.3.0 #chess #tokenize #pgn #game-ai #machine-learning #tokenizer

  64. fastokens-b10

    Fast Tokenizer

    v0.2.3 #bpe #python #fastokens #llm #source #tokenizer

  65. lnhash

    Tokenización determinista — texto a Togens con detección de dominio (lenguaje natural/código/datos) e idioma

    v0.1.2 #hashing #language-detection #domain-detection #tokenizer

  66. tokenx-rs

    Fast token count estimation for LLMs at 96% accuracy without a full tokenizer

    v0.1.0 270 #llm #claude #tokenize #tokenizer

  67. smoltok-core

    Byte-Pair Encoding tokenizer implementation in Rust

    v0.1.1 #bpe #encoding #text-processing #tokenizer

  68. pos-tagger

    Brill-style English part-of-speech tagger (lexicon lookup + 8 contextual rewrite rules)

    v0.1.0 #nlp #pos-tagging #brill #penn-treebank #tokenizer

  69. english-tokenizer

    Regex-driven tokenizer that splits a sentence and type-tags each token (word, number, emoji, url, email, hashtag, mention, time, currency, and more)

    v0.1.0 #tokenize #nlp #unicode-text #unicode #tokenizer

  70. rust-forth-tokenizer

    A Forth tokenizer written in Rust

    v0.2.1 480 #tokenize #forth #tokenizer

  71. latex-math

    Command-line interface (CLI) tool and library for parsing and representing mathematical expressions

    v0.1.2 #math #equations #tokenize #latex #tokenizer

  72. bleuscore

    A fast bleu score calculator

    v0.2.0 #tokenize #nlp #deep-learning #bleu #tokenizer

  73. mecrab

    A high-performance, thread-safe morphological analyzer compatible with MeCab, written in pure Rust

    v0.1.0 #japanese #morphology #nlp #mecab #tokenizer

  74. punkt

    sentence tokenizer

    v1.0.5 #tokenize #sentence #tokenizer

  75. pretokie

    Fast, zero-allocation pretokenizers for BPE tokenizers

    v0.0.4 130 #bpe #nlp #pretokenizer #transformer #tokenizer

  76. panko

    A small, zero-copy text tokenizer that crumbles strings into Words, Symbols, and Newlines

    v0.2.0 #tokenize #text-tokenizer #unicode #split #nlp #unicode-text #tokenizer

  77. tokstream-core

    Core tokenizer streaming engine for tokstream

    v0.1.2 #tokenize #simulation #streaming #tokenizer

  78. fuzzy-pickles

    A low-level parser of Rust source code with high-level visitor implementations

    v0.1.1 #tokenize #rust #tokenizer

  79. fibpetokenizer

    A blazing fast Byte Pair Encoding (BPE) tokenizer library with Python bindings

    v0.1.0 #bpe #nlp #machine-learning #bpe-tokenizer #tokenizer

  80. lindera-tokenizer

    A morphological analysis library

    v0.32.3 7.9K #morphological-analysis #tokenize #tokenizer #analysis #morphological

  81. indent_tokenizer

    Generate tokens based on indentation

    v0.4.0 #tokenize #indentation #tokenizer

  82. turso_sqlite3_parser

    SQL parser (as understood by SQLite)

    v0.2.0-pre.7 1.5K #tokenize #sql-parser #sql #parser #tokenizer

  83. wordchipper-training

    BPE LLM Tokenizer Training for wordchipper

    v0.9.2 #bpe #training #llm #wordchipper #suite #tokenizer

  84. bundle_repo

    Pack a local or remote Git Repository to XML for LLM Consumption

    v0.6.0 440 #git #llm #tokenizer

  85. fast-bpe-rs

    Fast Byte Pair Encoding (BPE) tokenizer with Python bindings powered by PyO3

    v0.6.1 #bpe #pyo3 #nlp #python #tokenizer

  86. vaporetto_rules

    Rule-base filters for Vaporetto

    v0.6.5 850 #japanese #tokenize #morphological #analyzer #tokenizer

  87. lang_pt

    A parser tool to generate recursive descent top down parser

    v0.1.2 #tokenize #top-down-parser #recursive-descent #tokenizer #parser

  88. string-view

    Work with views into string slices. Safely extend, reduce without losing parent string size. Modify str slices in-place to avoid allocations.

    v0.4.1 #compiler #text #no-alloc #string #tokenizer

  89. char-token-est

    Tokenless byte/char-based token-count estimator for LLM prompts. Per-model-family calibration for Claude, GPT, Gemini, Llama. Zero deps.

    v0.1.0 #llm #tokenizer #estimate

  90. bpetok

    CLI for tokenizing text input using Byte Pair Encoding (BPE)

    v0.1.2 #bpe #cli #tokenizer

  91. avila-tokenizers

    The most complete tokenizer library in Rust - BPE, WordPiece, Unigram, with native support for GPT, BERT, Llama, Claude

    v0.1.0 #llm #bert #nlp #tokenizer

  92. kevy-text

    Dictionary-free full-text search core: CJK bigram tokenizer, inverted segments, BM25 scoring

    v3.17.0 #full-text-search #tokenize #cjk #bm25 #inverted-index #tokenizer

  93. rllm-tokenizer

    HF tokenizer loading, encoding, decoding, streaming detokenization, and chat templates for rLLM

    v0.1.1 #chat-template #streaming #hf #encoding-decoding #tokenizer

  94. rustbpe

    A BPE (Byte Pair Encoding) tokenizer written in Rust with Python bindings

    v0.1.0 #bpe #python-bindings #tiktoken #training #gpt-4 #tokenizer

  95. mipl

    Minimal Imperative Parsing Library

    v0.2.1 #tokenize #tokenizer #parser

  96. tantivy-czech-stemmer

    Czech stemmer as Tantivy tokenizer

    v0.2.1 #tantivy #stemming #czech #tokenizer

  97. giron

    ECMAScript parser which outputs ESTree JSON

    v0.1.2 #javascript-parser #javascript #tokenizer

  98. sql-script-parser

    iterates over SQL statements in SQL script

    v0.1.2 #sql #mysql #sql-parser #tokenizer

  99. tokenizers-enfer

    today's most used tokenizers, with a focus on performances and versatility

    v0.21.1 #bpe #hugging-face #nlp #tokenize #tokenizer #word-piece

  100. bpe-tokenizer

    A BPE Tokenizer library

    v0.1.4 150 #bpe #byte #encoding #pair #tokenize #byte-pair #tokenizer

  101. alith-models

    Load and Download LLM Models, Metadata, and Tokenizers

    v0.4.3 #gguf #llm-model #metadata #hugging-face #local-llm #tokenizer #embedding

  102. tokenizer

    Thai text tokenizer

    v0.1.2 #tokenize #localization #thai #text-tokenizer #tokeniser

  103. punkt_n

    Punkt sentence tokenizer

    v1.0.5 #tokenize #sentence #punkt #tokenizer

    104. Try searching with DuckDuckGo.


  104. sentencepiece-model

    SentencePiece model parser generated from the SentencePiece protobuf definition

    v0.1.4 16K #sentence-piece #tokenize #nlp #machine-learning #tokenizer

  105. rust_transformers

    High performance tokenizers for Rust

    v0.2.0 #bpe #transformer-models #bert #py #tokenizer #integration-tests #rust-nightly #benchmark

  106. tele_tokenizer

    A CSS tokenizer

    v0.2.0 #tokenize #css #telecss #tokenizer

  107. tokengeex

    efficient tokenizer for code based on UnigramLM and TokenMonster

    v1.1.0 #llm #nlp #codegeex #tokenizer

  108. uscan

    A universal source code scanner

    v0.1.3 #tokenize #compiler #tokenizer

  109. nipah_tokenizer

    A powerful yet simple text tokenizer for your everyday needs!

    v0.1.0 #tokenize #text-tokenizer #nlp #tokenizer

  110. smoltoken

    A fast library for Byte Pair Encoding (BPE) tokenization

    v0.2.0 360 #bpe #tokenizer #ai

  111. rs-jptxt2tokens

    wrapper to convert the jp txt to tokens

    v0.1.0 #japanese #nlp #wrapper #tokenizer

  112. aleph-alpha-tokenizer

    A fast implementation of a wordpiece-inspired tokenizer

    v0.3.1 #tokenize #aleph-alpha #nlp #tokenizer

  113. json-parser

    JSON parser

    v1.0.2 #tokenize #json #tokenizer

  114. indentation_flattener

    From indented input, generate plain output with indentation PUSH and POP codes

    v0.1.0 #indentation #tokenize #tokenizer #parser

  115. basic_lexer

    Basic lexical analyzer for parsing and compiling

    v0.2.1 #tokenize #lexical-analysis #white-space #tokenizer

  116. tinytoken

    tokenizing text into words, numbers, symbols, and more, with customizable parsing options

    v0.1.4 130 #tokenize #text-input #numbers #tokenizer

  117. mecab-ko-core

    한국어 형태소 분석 핵심 엔진 - Lattice, Viterbi, 토크나이저

    v0.7.2 130 #tokenize #viterbi #nlp #korean #morphology #tokenizer

  118. tokenmonster

    Greedy tiktoken-like tokenizer with embedded vocabulary (cl100k-base approximator)

    v0.1.0 #tokenize #tiktoken #nlp #tokenizer

  119. regex-tokenizer

    A regex tokenizer

    v0.1.1 #tokenize #regex #tokenizer

  120. scanny

    A advanced text scanning library for Rust

    v0.1.0 #lexical-token #tokenizer #parser

  121. tokeneer

    tokenizer crate

    v0.1.0 #bpe #nlp #tokenizer

  122. llm_models

    Load and download LLM models, metadata, and tokenizers

    v0.0.3 #gguf #llm #model #metadata #artificial-intelligence #tokenizer #candle

  123. gpt_tokenizer

    Rust BPE Encoder Decoder (Tokenizer) for GPT-2 / GPT-3

    v0.1.0 #chatgpt #gpt-3 #bpe #openai #tokenizer

  124. tocken

    Clustering algorithms

    v0.1.0 #machine-learning #vector-search #tokenize #text-tokenizer #text #tokenizer

  125. rustpotion

    Blazingly fast word embeddings with Tokenlearn

    v0.3.0 #tokenize #embedding #rag #model2vec #tokenizer

  126. scnr2_macro

    Scanner/Lexer with regex patterns and multiple modes

    v0.5.2 4.0K #lexer-tokenizer #lexer #tokenizer

  127. tuker

    A small tokenizer/parser library with an emphasis on usability

    v0.1.0 #lexer #lexer-tokenizer #tokenizer

  128. token

    string-tokenizer (and sentence splitter) Note: If you find that you would like to use the name for something more appropriate, please just send me a mail at jaln at itu dot dk

    v1.0.0-rc1 #string-tokenizer #splitter #sentence #string #tokenizer

  129. bytepiece_rs

    The Bytepiece Tokenizer Implemented in Rust

    v0.2.2 110 #tokenize #tokenizer #deeplearning

  130. blingfire-sys

    Bindings to the BlingFire C++ library

    v1.0.1 1.2K #tokenize #nlp #machine-learning #tokenizer