Skip to content

Python: fix(redis): escape RediSearch query syntax in memory search text - #10039

Open
Fish (wxhking) wants to merge 2 commits into
microsoft:mainfrom
wxhking:fix/redis-search-text-escaping
Open

Fish (wxhking) wants to merge 2 commits into
microsoft:mainfrom
wxhking:fix/redis-search-text-escaping

Conversation

@wxhking

Copy link
Copy Markdown

Fixes #10002

Summary

RedisContextProvider._redis_search passed the user's message straight into TextQuery / AggregateHybridQuery as the search text. RediSearch reads several punctuation marks as query syntax there — | unions terms, ? is a fuzzy marker, backticks quote exact phrases — so ordinary messages like ps aux | grep python, Why ? or a markdown table reached the engine as malformed grammar (e.g. @content:(ps | aux | | | grep | python)) and raised IntegrationInvalidRequestException: Redis text search failed: ... Syntax error, failing the whole agent run before the model was ever called.

The search text is plain prose, not a query: escape the RediSearch syntax characters with a TokenEscaper configured for them before building either query.

  • spaces stay unescaped, so terms keep tokenizing and BM25 scoring works exactly as before
  • tag filter values were already escaped via _vector_store's escaper; only the free-text path was exposed
  • reproduced at the query-string level: TextQuery(text='ps aux | grep python').query_string() yields '@content:(ps | aux | | | grep | python)' before the fix; after it the terms are escaped and the broken | | shape is gone

Test plan

  • new parametrized regression tests in python/packages/redis/tests/test_context_provider_edges.py: all five failing messages from the issue are asserted to produce an escaped query string (no unescaped | ? / | | shapes, escaped chars present), for both the text and hybrid-vector paths — 7 new cases
  • pytest tests -m 'not integration' in python/packages/redis — 337 passed, 58 deselected (integration), no new failures
  • ruff check / format clean on both changed files

Copilot AI balanced review requested due to automatic review settings November 3, 2026 13:11
@agent-framework-automation agent-framework-automation Bot added the python Usage: [Issues, PRs], Target: Python label Nov 3, 2026
@github-actions github-actions Bot changed the title fix(redis): escape RediSearch query syntax in memory search text Python: fix(redis): escape RediSearch query syntax in memory search text Nov 3, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Escaping the text before embedding makes hybrid search use different embedding input from stored content, potentially changing retrieval rankings; preserve the original embedding input and add a regression assertion.

1 open issue
What changed in this PR

Addresses #10002 by escaping RediSearch syntax in user messages used for Redis memory retrieval, aiming to prevent punctuation from causing syntax errors that abort agent runs.

Changes:

  • Adds a search-text escaper shared by text and hybrid queries, leaving spaces unescaped.
  • Adds regression tests for punctuation-heavy messages and escaped text passed to hybrid queries.
File Description
python/​packages/​redis/​tests/​test_context_provider_edges.py Adds text-query escaping cases and a hybrid-query regression test.
python/​packages/​redis/​agent_framework_redis/​_context_provider.py Escapes search text before constructing Redis queries.

Review effort: Balanced

@wxhking

Copy link
Copy Markdown
Author

@microsoft-github-policy-service agree

RedisContextProvider passed the user's message straight into TextQuery /
AggregateHybridQuery as the search text. RediSearch reads several
punctuation marks as query syntax there — '|' unions terms, '?' is a fuzzy
marker, backticks quote exact phrases — so ordinary messages like
'ps aux | grep python' or 'Why ?' reached the engine as malformed grammar
(e.g. '@content:(ps | aux | | | grep | python)') and raised a syntax error
that failed the whole agent run; the model was never called.

Escape the search text with a TokenEscaper configured for the RediSearch
syntax characters before building either query. Spaces stay unescaped, so
the terms keep tokenizing and BM25 scoring works as before. Tag filter
values were already escaped through _vector_store's escaper; only the
free-text path was exposed.

Fixes microsoft#10002
@wxhking
Fish (wxhking) force-pushed the fix/redis-search-text-escaping branch from 4dfe2ea to 37bb445 Compare November 8, 2026 02:17
@wxhking
Fish (wxhking) deployed to github-app-auth November 8, 2026 02:17 — with GitHub Actions Active
redisvl's TextQuery escapes every term itself: _tokenize_and_escape_query
runs the default TokenEscaper per token, and that escaper also escapes the
backslash a hand-escaped term already carries. The pre-escaped '|' from the
TokenEscaper built here therefore landed as a literal backslash-pipe that
was re-escaped again — the produced query still carried the bare '|', so
'ps aux | grep python' stayed broken, and tokens like 'v1.2' picked up a
backslash that changed BM25 matching.

Instead of escaping, strip the syntax characters redisvl's escaper does not
cover ('|', '?', backtick) and rejoin the remaining tokens as plain terms;
TextQuery then tokenizes and escapes each of them again as designed.

Suggested-by: Aman Agnihotri <163747435+its-amann@users.noreply.github.com>
@wxhking
Fish (wxhking) deployed to github-app-auth November 9, 2026 05:34 — with GitHub Actions Active

This branch was successfully deployed

1 active deployment
github-app-auth — d12730eb Deployed Nov 9, 2026 by wxhking via add_label #27370
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

python Usage: [Issues, PRs], Target: Python

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Python: [Bug]: RedisContextProvider fails the whole agent run when a message contains ?, | or a backtick

3 participants