Skip to content

gh-161524: Speed up copying sparse dictionaries - #161636

Open
localzet wants to merge 1 commit into
python:mainfrom
localzet:contrib/gh-161524-sparse-dict-copy
Open

localzet wants to merge 1 commit into
python:mainfrom
localzet:contrib/gh-161524-sparse-dict-copy

Conversation

@localzet

@localzet localzet commented Nov 8, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Copy live entries directly into a compact keys table when cloning a sparse combined dictionary, rather than inserting each surviving key/value pair through the generic merge path.

  • Preserve insertion order, key/value identities, and cached hashes without invoking user-defined __hash__ methods again.
  • Use the compact-copy path for dict.copy() and copying into empty dictionaries (including frozendict construction).
  • Keep the existing path for very sparse tables (fewer than approximately 1/64 live entries), where scanning tombstones is less beneficial.
  • Preserve dictionary watcher notifications when an empty destination clones a source.

Tests

The patch was tested in the contributor fork with:

  • Debug builds: test_dict, test_capi.test_dict, and test_capi.test_watchers (249 tests).
  • Debug free-threaded build: the same test modules (249 tests).
  • Reference-leak checks: -R 3:3 on these modules.
  • AddressSanitizer build: the same test modules (detect_leaks=0, matching CPython's ASan CI configuration).
  • Additional randomized sparse-dictionary stress tests, including hash collisions and concurrent copies, in a separate research branch.

Verification: https://github.com/localzet/cpython/actions/runs/42510556632

Performance

Linux x86-64, GCC PGO + LTO; median of repeated source.copy() measurements against the unmodified baseline (3c92a06e). Times are ns per operation.

Keys (live / original) Key type Baseline Patch Speedup
30,000 / 100,000 str 891,644 235,320 3.79x
30,000 / 100,000 int 592,779 206,562 2.87x
3,000 / 100,000 str 118,462 46,237 2.56x
3,000 / 100,000 int 111,283 76,323 1.46x
1,500 / 100,000 int 87,039 89,021 0.98x

These are results from one CI benchmark run, not a statistically controlled multi-machine study. Extremely sparse integer-key cases are approximately unchanged; some cases show a small measured slowdown.

Benchmark: https://github.com/localzet/cpython/actions/runs/42497448717

AI assistance was used for source exploration, patch development, and test/benchmark automation. The patch and its behavior were examined and tested against the configurations listed above.

@bedevere-app bedevere-app Bot added awaiting review type-feature A feature request or enhancement labels Nov 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

awaiting review type-feature A feature request or enhancement

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant