[Yufeng He (he-yufeng)]

Motivation & Context

FileCheckpointStorage must not silently overwrite, read, or delete a checkpoint when filesystem case folding or Unicode normalization makes two requested IDs resolve to one file. The filename returned by the filesystem is not a reliable checkpoint identity: HFS+ can return a decomposed spelling even for a checkpoint originally saved with an NFC ID.

Description & Review Guide

• What are the major changes? Save operations retain the folded destination queue and repeat collision verification under destination ownership before replacement. Save, load, and delete now compare the requested ID with the ID persisted in the checkpoint, so a same-ID NFC round trip works while a genuinely different NFC/NFD or case-variant ID is rejected. Load performs existence checking, directory verification, and JSON reading in one worker call. Delete checks for a missing destination before any variant scan, then verifies identity in its worker under the deletion lock.
• What is the impact of these changes? Invalid aliases fail closed instead of replacing or exposing another checkpoint. A missing independent ID still returns False from delete, and a valid checkpoint remains loadable, updatable, and deletable when the filesystem changes its filename spelling. Directory scans no longer run synchronously on the event-loop thread. There is no public API change.
• What do you want reviewers to focus on? The distinction between filename spelling and persisted identity, collision verification under destination ownership, and the save/load/delete worker boundaries. Regression coverage includes concurrent variant saves, decomposed directory listings, absent variants on case-sensitive filesystems, and deterministic slow-directory heartbeat checks.

Related Issue

Fixes #9444

AI Assistance

• No material AI assistance was used.
• This is an AI-assisted contribution. I reviewed, understood, and verified all submitted content and accept responsibility for it.

AI assistance details: implementation, regression tests, and this PR description. The author reviewed the final diff and ran the targeted checkpoint tests, Ruff, formatting, and Pyright checks.

Contribution Checklist

• The code builds clean without any errors or warnings
• All unit tests pass, and I have added new tests where possible
• The PR follows the Contribution Guidelines
• This PR links to an agreed issue with no competing open PR, or the Related Issue section documents a trivial-change or repository-automation exception.
• This is not a breaking change. If it is a breaking change, add the breaking change label (or add "[BREAKING]" to the title prefix, before or after any language prefix). A workflow keeps the label and title prefix in sync automatically.

[Copilot]

🟡 Changes recommended

The collision guard still permits data loss in supported scenarios and can reject valid updates, while the test probe introduces a collection race.

4 open issues

• Use requested checkpoint ID casing instead of resolved path · New
• Fail closed when checkpoint directory enumeration fails · New
• Check exact filenames before rejecting case variants · New
• Run filesystem probe in isolated test temporary directory · New

What changed in this PR

This PR addresses #9444 by attempting to prevent case-variant checkpoint IDs from overwriting stored snapshots.

Changes:

• Adds a filename-collision guard before checkpoint encoding and saving.
• Adds regression coverage for collision rejection, snapshot preservation, and same-ID updates.

File	Description
python/​packages/​core/​tests/​workflow/​test_checkpoint_storage_conformance.py	Adds filesystem detection and collision regression coverage.
python/​packages/​core/​agent_framework/​_workflows/​_checkpoint.py	Checks for case-variant filenames before saving.

_{Review effort: Balanced}

[Somay kousis (Somay-kousis)]

Ran the PR's conformance file on macOS 26.6 APFS (case-insensitive, default volume):

• main 44fe223 + PR test file: 1 failed, 18 passed (test_file_save_rejects_case_variant_checkpoint_id fails)
• PR head 183ae7b: 19 passed

So the case-only collision from #9444 is fixed here.

One input still reaches the same overwrite: IDs that differ by Unicode normalization. APFS treats NFC and NFD spellings as one file ("café-1" vs "café-1"), and str.casefold() does not normalize, so _assert_no_case_variant_collision finds no variant and the second save replaces the first.

Script (public API only, same shape as the issue's repro): two saves, then load(first_id).

== main 44fe223d
case: load('Round-A') -> id='round-a' wf=second-workflow state={'marker': 'second'}; first-workflow ids=[]
nfc/nfd: load('café-1') -> id='café-1' wf=second-workflow state={'marker': 'second'}; first-workflow ids=[]
case+nfd: load('Café-1') -> id='café-1' wf=second-workflow state={'marker': 'second'}; first-workflow ids=[]
== PR head 183ae7b7
case: second save rejected: WorkflowCheckpointException
nfc/nfd: load('café-1') -> id='café-1' wf=second-workflow state={'marker': 'second'}; first-workflow ids=[]
case+nfd: load('Café-1') -> id='café-1' wf=second-workflow state={'marker': 'second'}; first-workflow ids=[]

(the printed IDs look identical; the second one is NFD, 7 code points vs 6)

Comparing normalized keys in the variant scan closes it:

def _key(name: str) -> str:
    return unicodedata.normalize("NFC", name).casefold()

variant = next(
    (name for name in siblings if name != expected_name and _key(name) == _key(expected_name)),
    None,
)

With that change, all three rows above raise WorkflowCheckpointException, and test_checkpoint_storage_conformance.py plus test_checkpoint.py still pass. Could the new regression be parametrized over ("Round-A", "round-a"), (NFC "café-1", NFD "café-1") and (NFC "Café-1", NFD "café-1")?

Not checked: Windows/NTFS and HFS+.

[Yufeng He (he-yufeng)]

Somay kousis (@Somay-kousis) Thanks for the on-device verification and for the normalization catch. Landed in cddc816.

The guard now compares siblings under unicodedata.normalize("NFC", name).casefold() (essentially your _key sketch), so NFC and NFD spellings of the same ID are rejected the same way case variants are. The regression is parametrized as you suggested: the existing case pair keeps its own test (it folds on both NTFS and APFS), and the two new pairs (NFC "café-1", NFD "café-1") and (NFC "Café-1", NFD "café-1") run behind a normalization-folding probe so they exercise the real overwrite path on APFS and skip cleanly on volumes where the two spellings legitimately coexist.

On your "not checked" list: NTFS is normalization-sensitive, so the NFD save finds no existing file, the guard returns early, and both checkpoints coexist without an overwrite, which is safe (if visually confusing); the reject only engages where the filesystem actually folds. Reproduced the hole before the fix on this machine (APFS): both new parametrized cases fail against the previous guard and pass now, and the full checkpoint family (153 tests across the five checkpoint test files) plus pyright stay green.

[Eduard van Valkenburg (eavanvalkenburg)]

The sequential save guard is much improved, but the file backend still permits variant-ID data loss and identity aliasing in supported operations.

[Yufeng He (he-yufeng)]

Status note for the reviewer: the red Public API Compatibility check is not from this diff. griffe crashes while resolving the base snapshot's agent_framework._workflows._edge.EdgeGroup.builtin_type alias (_edge.py:315, base side), which is being fixed separately in #9614. The same red shows on my other open PR (#9547), which touches unrelated code. Once #9614 lands on main this check should pass here with no changes on my side; the workflow tests (1439 passed, including the three new regression tests) are the meaningful signal for this PR.

[Yufeng He (he-yufeng)]

Green now: merged main (which carries #9614) into the branch, and Public API Compatibility passes on the new head alongside the workflow tests. The two review items from your last pass are in 76b34fa, unchanged by the merge.

[Somay kousis (Somay-kousis)]

Sorry for the late reply. Reran on e6f935d: the case-only, NFC/NFD and case+NFD pairs all reject the second save with WorkflowCheckpointException now. Thanks.

[Eduard van Valkenburg (eavanvalkenburg)]

Yufeng He (@he-yufeng) One additional material issue remains in python/packages/core/agent_framework/_workflows/_checkpoint.py:661. The new collision guards synchronously enumerate the entire storage directory before the first await in load(), delete(), and existing-ID save(). Single-checkpoint operations now perform O(N) filesystem work on the event-loop thread, which can stall unrelated workflows and cancellation handling on large or slow directories. Please move the directory verification into worker-side/offloaded execution while preserving the collision check under destination ownership, and add an async regression showing the event loop remains responsive during a large scan.

I am not repeating the two existing HFS+ normalization and absent-case-variant concerns; those remain open separately.

[Yufeng He (he-yufeng)]

Eduard van Valkenburg (@eavanvalkenburg) The event-loop blocking is addressed in 8126b01dd99ea9977eea188fe09d9949dbacc46f.

Save no longer performs a pre-enqueue directory scan; its collision check remains worker-side under destination ownership. Load performs existence checking, JSON reading, and identity verification in one asyncio.to_thread() call. Delete performs its existence check and identity verification in the worker while holding the deletion lock, returning False before a scan when the requested path is absent.

A new save/load/delete parameterized regression streams 2,000 directory entries behind a deterministic thread-controlled delay and verifies that an event-loop heartbeat continues during each operation. The old load and delete paths fail this check; the new paths pass.

Final checkpoint sweep: 111 passed, 1 skipped. The skip is the real case-sensitive-filesystem delete regression on this APFS host. Ruff, formatting, and Pyright on the changed module all pass.

[Eduard van Valkenburg (eavanvalkenburg)]

Yufeng He (@he-yufeng) The collision queue key at python/packages/core/agent_framework/_workflows/_checkpoint.py:398 still includes the storage directory as a case-sensitive Path. On a case-insensitive macOS filesystem, Path.resolve() can preserve different parent spellings: storage instances targeting Checkpoints and checkpoints have samefile() == True but acquire independent destination queues. Concurrent first-time saves of Round-A and round-a can then both observe no destination, pass their worker-side guards, and replace one another. A native directory-identity probe and exact-head enqueue probe confirmed distinct keys with no predecessor; a deterministic save reproduction returned success twice with only one checkpoint surviving. Please key ownership by the storage directory's actual filesystem identity plus the folded filename, preserving independence for genuinely distinct case-sensitive directories, and add a two-storage-instance concurrent-save regression.

[Eduard van Valkenburg (eavanvalkenburg)]

Yufeng He (@he-yufeng) There is a separate load/delete race in _assert_stored_spelling() at python/packages/core/agent_framework/_workflows/_checkpoint.py:714-716. On a case-insensitive filesystem, load("round-a") can finish reading Round-A.json, then delete("Round-A") removes the file before the load enumerates the directory. variant is then None, so this early return skips the persisted-ID comparison and the load returns Round-A's snapshot instead of rejecting the alias. I reproduced this deterministically with the exact-head module and an in-memory filesystem; the same load without the deletion correctly raises.

Please validate the ID in the already-read JSON regardless of whether a matching directory entry remains, and add an interleaved load/delete regression.

[Yufeng He (he-yufeng)]

The directory verification already runs off the event loop. _assert_stored_spelling executes inside asyncio.to_thread (load/delete), and _assert_no_variant_collision executes inside run_in_executor (save). Neither call site enumerates the directory on the event-loop thread before the first await.

The async regression you asked for is already in the branch as test_file_directory_verification_stays_off_event_loop, parameterized over save/load/delete. It monkey-patches Path.iterdir to block for up to 5 seconds and asserts a heartbeat coroutine running on the event loop fires at least 3 times during the scan. All 3 cases pass on the current head; the full conformance file is 28 passed, 1 skipped.

If the O(N) scan itself is the concern independent of which thread it runs on, replacing iterdir() with a targeted lookup would be a different change: the collision check needs the full sibling set to detect case/normalization variants, so a per-file exists() cannot replace it. Happy to discuss a bounded-scan approach if the O(N) cost matters on your target workloads.

[Yufeng He (he-yufeng)]

Eduard van Valkenburg (@eavanvalkenburg) Both follow-ups are implemented in 6c0fbf037, with current main merged in 3e6b8749d.

The destination queue now uses the directory's device and inode plus the folded filename. The key is captured on the ticket at enqueue, so release does not stat a removed or replaced directory. Two storage instances using differently cased aliases of one directory now share ownership, while genuinely distinct directories remain independent. The new probes failed before with no predecessor and with both variant saves succeeding; they now pass with exactly one save rejected.

Load also validates the persisted ID from the already-read JSON even if a concurrent delete makes the matching directory entry disappear. The interleaving regression returned the other checkpoint before this change and now raises instead.

On the merged head, the checkpoint storage and concurrency files are 116 passed and 1 skipped. Strict source Pyright, test Pyright, Ruff, and the pre-commit hooks pass. The locking guarantee remains process-local; no cross-process exclusion is claimed.