pg_tokenizer

pg_tokenizer is a standalone multilingual text analyzer and vocabulary tokenizer extension for PostgreSQL, designed to power full-text BM25 ranking with vchord_bm25 and hybrid retrieval with vchord.

Website | Documentation | vchord | vchord_bm25 | Blog | PGEXT | Releases | Issues | Upstream

---

Vchord Suite

pg_tokenizer is part of the Vchord extension family maintained by PGSTY:

• vchord (Docs / Meta): Scalable, high-performance, and disk-friendly vector similarity search.
• vchord_bm25 (Docs / Meta): Native BM25 full-text ranking index (Block-WeakAnd).
• pg_tokenizer (Docs / Meta): Standalone multilingual text analyzer and vocabulary tokenizer.

---

Features

• Modular Text Analysis Pipeline: Chain character filters (to_lowercase, unicode_normalization), pre-tokenizers (unicode_segmentation, regex, Chinese jieba), and token filters (stopwords, Snowball stemmer, synonym, pg_dict, ngram) via declarative TOML configurations.
• Built-in & Hugging Face Vocabularies: Ship with built-in pre-trained models (bert_base_uncased, wiki_tocken, gemma2b, llmlingua2) or import any Hugging Face tokenizer.json directly into PostgreSQL.
• Corpus-Trained Custom Models: Automatically build domain-specific vocabularies from your own table columns and maintain token embeddings via database triggers (create_custom_model_tokenizer_and_trigger).
• Multilingual Support: Native support for English, European languages (30 Snowball stemmers), Chinese (jieba), Japanese/Korean (lindera), and multilingual LLM vocabularies.

---

Version 1.1.0

Version 1.1.0 adds WAL-replicated definition invalidation for standbys and promotion, schema-bound PostgreSQL dictionaries, correct writer parallel markings, and configurable analysis budgets. Runtime vocabulary tables and their sequences live in tokenizer_data, so database owners no longer need CREATE on tokenizer_catalog.

It also fixes Lindera server-file permission checks, UTF-8 splitting of long custom tokens, repeated-lexeme tsvector conversion, trigger target coercion, and Jieba's swapped full/precise modes. Regenerate persisted output and rebuild dependent indexes if either Jieba mode was used in 1.0.0 or earlier. Analysis budgets reduce expansion risk but do not impose a hard memory or cancellation-time limit on third-party libraries.

PREPARE TRANSACTION now discards local derived objects, including configurations loaded after direct catalog DML. A later rollback of the prepared transaction cannot leave a stale configuration cached when another transaction reuses its version number. Read-only transactions can still be prepared; definition-changing APIs continue to reject PREPARE.

Release acceptance for 1.1.0 covers fresh installations on PostgreSQL 14–18. Upgrade scripts remain available, but upgrade compatibility, default-privilege migration and snapshots spanning an upgrade are outside this release's acceptance scope. See the English release notes, Chinese release notes, and GitHub release.

Version 1.0.0 was the first PGSTY release. It builds with pgrx 0.19.3 for PostgreSQL 14–18 and moves to Lindera 6.2, Jieba 0.11, and Hugging Face tokenizers 0.23. It adds native PostgreSQL tsvector output, filtered custom vocabularies with schema-qualified source tables, safe concurrent vocabulary writes, database-owner privileges, transaction-aware definition caches, preload models that survive cache invalidation and broken entries, and definition tables included in pg_dump. It fixes ngram for non-ASCII and short tokens, calls pg_dict dictionaries the way ts_lexize does, reserves the built-in stopword lists, and takes its Snowball stemmers from pgsty/rust-stemmers 1.2.1 with unchanged output. Existing tokenize() calls still return INT[].

See custom models, native full-text search, upgrade instructions, and release/PGXN preparation. The source package includes a Makefile for PGXN clients; PGXN publication is a separate release step.

Quick Start

For full setup instructions (Debian/ZIP packages and building from source), see the Installation Guide or docs/01-installation.md.

1. Install via Native Linux Packages (pig)

On EL 8/9/10, Debian 12/13, and Ubuntu 22.04/24.04/26.04 (PostgreSQL 14–18), install the native package built for your distribution and architecture from the Pigsty repository (see the package matrix on pgext.cloud):

curl -fsSL https://repo.pigsty.io/pig | bash
pig repo add pgsql -u
pig ext add pg_tokenizer

Native RPM/DEB builds do not embed Lindera dictionaries; supply a dictionary built with Lindera 6. GitHub release packages enable lindera-ipadic and include IPADIC. Other embedded dictionaries require their corresponding build features. Lindera dictionary and user-dictionary files on the server can be loaded only by superusers and roles with the privileges of pg_read_server_files; see database-owner permissions.

Then add pg_tokenizer.so to shared_preload_libraries and restart PostgreSQL. The example assumes no other preloaded libraries; preserve any existing entries when adding it:

psql -U postgres -c "ALTER SYSTEM SET shared_preload_libraries = 'pg_tokenizer.so';"
sudo systemctl restart postgresql

2. Run with Docker

The image below is maintained upstream and does not establish availability of PGSTY's 1.1.0 release. Use the matching 1.1.0 package or source tag when deploying this release.

You can also start a ready-to-use container with the tensorchord/vchord-suite image, which bundles PostgreSQL, vchord, vchord_bm25, and pg_tokenizer:

docker run \
  --name vchord-suite \
  -e POSTGRES_PASSWORD=postgres \
  -p 5432:5432 \
  -d tensorchord/vchord-suite:pg18-latest
  # Or from GHCR: ghcr.io/tensorchord/vchord-suite:pg18-latest

Connect to the database using psql:

psql -h localhost -p 5432 -U postgres

3. Basic SQL Usage

Enable the extension and add tokenizer_catalog to search_path:

CREATE EXTENSION IF NOT EXISTS pg_tokenizer;
SET search_path TO "$user", public, tokenizer_catalog;

Create a tokenizer backed by the built-in llmlingua2 model and tokenize text into an integer array of token IDs:

SELECT create_tokenizer('tokenizer1', $$
model = "llmlingua2"
$$);

SELECT tokenize(
    'PostgreSQL is a powerful, open-source object-relational database system. It has over 15 years of active development.',
    'tokenizer1'
);

More SQL examples (custom corpus models, Chinese jieba, Japanese lindera) can be found in docs/03-examples.md.

When upgrading from Lindera 0.42 to 6.2, regenerate stored Lindera token arrays and rebuild their search indexes: the embedded IPADIC dictionary now assigns different token IDs. Existing model configurations remain accepted; see the Lindera upgrade notes.

---

Documentation

Full online documentation is available at vchord.pgsty.com/docs/tokenizer, alongside the local docs in docs/:

• Installation Guide (docs/01-installation.md)
• Tokenizer Overview & Usage (docs/04-usage.md)
• Text Analyzer Pipeline (docs/05-text-analyzer.md)
• Models (Built-in, Custom, Hugging Face, Lindera) (docs/06-model.md)
• Multilingual Segmentation & Examples (docs/03-examples.md)
• SQL & Configuration Reference (docs/00-reference.md)
• Development · Limitations

---

License

This software is licensed under the Apache License 2.0 (Apache-2.0).

• Copyright (c) 2025 TensorChord Inc.
• Copyright (c) 2026 PGSTY PTE. LTD.
• Maintained by Ruohang Feng @ PGSTY