Repository navigation
FEAT: Change saved registry entries through the API - #4072
Open
varunj-msft wants to merge 199 commits into
Open
varunj-msft wants to merge 199 commits into
varunj-msft wants to merge 199 commits into
Conversation
…API input External construction now rejects a parameter whose type includes Path unless the calling service passes the name in allowed_path_params after making the value safe. The converter service allows its upload parameters once they are stored, and the target service allows the server-set upload folder and the GitHubCopilotTarget working directory. The scorer-only check is removed, and the "Valid parameters" list leaves out path parameters callers cannot set.
The target catalog now applies the path rule to required parameters too, so a target that needs a disallowed path is left out instead of listed without the field, matching the scorer catalog. allowed_path_params is typed frozenset[str] so a bare string cannot turn the allow-list into a substring match.
The target catalog still lists a target whose required path parameter is its server-set upload folder when the server has one, and leaves it out otherwise.
DELETE /api/scenario-presets/{name} and /api/scenario-suites/{name} accept an
optional ?expected_version= query. When supplied, the delete is refused with 409
if the stored document changed after the caller read it, so confirming a delete
in the GUI cannot discard an edit the caller never saw. Without it the delete
stays unconditional, preserving existing callers.
Locally the version is compared inside the existing per-document lock; on Azure
Blob the delete carries the read ETag with MatchConditions.IfNotModified. A
versioned delete of a document that is already gone still reports 404.
The preset and suite libraries send the version they loaded and, on 409, close
the confirmation, reload, and explain that the item changed.
Follow-up to review feedback on microsoft#3057.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
# Conflicts: # tests/unit/backend/test_converter_service.py
…benchmark's recipes folder
Reverts 15e7a9e, which took the restore retry out of this stack after it was merged in by mistake. Deleting a saved entry needs the restore-failure list and retry it adds.
A saved entry is also refused for an empty key or headers it would take from this server, so the explanation other users see now says so. The retry tests' target sets the empty headers a saved OpenAI target needs.
The guard wraps ScenarioPresetStorage.delete_preset, which takes expected_version since microsoft#3656.
GET and DELETE /api/saved-entries/{kind}/{name} address a saved target, converter, scorer,
or technique by its file. Who may delete it comes from the file: whoever saved it or an
administrator, or anyone with sign-in off. The delete is refused while another saved entry
names it, a registered entry holds it, or work could be using it, and checks the version the
caller read. These routes answer while the runtime isn't ready, so an entry that failed to
restore can be deleted before a retry. Saving and deleting take turns, and a save checks again
that what it names still comes back after a restart.
Saved targets, converters, scorers, and techniques have a Delete button. Its dialog says who saved the entry and whether you may delete it, sends the version it showed, and offers Look again when the file changed. Entries that couldn't be restored can be deleted from the banner that lists them.
PUT /api/saved-entries/{kind}/{name} takes the recipe GET returns, changed, with the version
it read. The entry is rebuilt through the API that creates one, so every check a new entry
gets still applies, and its recipe is overwritten only if it is still that version. Its name,
class or attack type, and who saved it stay the same. The running entry is replaced, and the
old one is put back if the edit fails. Entries that hold the old one keep it until the next
apply or restart, and the response lists them. GET now returns the recipe to callers who may
change the entry.
The banner sat outside the theme provider, so it and the Delete dialog it now opens were drawn without the theme's colors. It now renders inside the theme, before the app's content.
…t/13178-Edit-Saved-Entries
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
Unresolved issues expose server paths during deletion failures, block recovery after editing unrestored entries, and lose keyboard focus after target or scorer deletion.
5 open issues
What changed in this PR
This PR extends PyRIT’s saved registry entries with version-checked API editing, building on the persistence and deletion work in #4071. Registry-tab editing remains out of scope.
Changes:
- Adds saved-entry edit request and response models, ownership metadata, and management-route integration.
- Carries forward persistence safeguards, restore recovery, saved-entry deletion controls, and version-checked preset and suite deletes.
- Expands regression coverage, isolates test storage, and documents saved-entry behavior.
| File | Description |
|---|---|
| tests/unit/setup/test_targets_initializer.py | Tests saved-target collision protection. |
| tests/unit/setup/test_scorer_initializer.py | Tests saved-scorer tagging protection. |
| tests/unit/setup/test_converter_initializer.py | Tests saved-converter collision protection. |
| tests/unit/setup/test_configuration_loader.py | Tests recipe storage configuration. |
| tests/unit/registry/test_scenario_suite_storage.py | Tests stale-version delete rejection. |
| tests/unit/registry/test_scenario_suite_registry.py | Tests version-checked suite deletion. |
| tests/unit/registry/test_scenario_preset_registry.py | Tests version-checked preset deletion. |
| tests/unit/registry/test_instance_registry.py | Tests saved-entry protection and origin metadata. |
| tests/unit/registry/test_converter_registry.py | Tests saved-name collision recording. |
| tests/unit/registry/test_attack_technique_registry.py | Tests saved-technique collision handling. |
| tests/unit/backend/test_technique_edit_service.py | Updates calls to keyword-only creation API. |
| tests/unit/backend/test_scenario_suite_service.py | Tests delete-version forwarding. |
| tests/unit/backend/test_scenario_suite_routes.py | Tests conflicts and sanitized storage errors. |
| tests/unit/backend/test_scenario_suite_execution_service.py | Tests saved-entry launch conflict messages. |
| tests/unit/backend/test_scenario_preset_service.py | Tests delete-version forwarding. |
| tests/unit/backend/test_scenario_preset_routes.py | Tests conflicts and sanitized storage errors. |
| tests/unit/backend/test_saved_entries.py | Tests temporary-reference save restrictions. |
| tests/unit/backend/test_mappers.py | Tests origin and ownership response mapping. |
| tests/unit/backend/test_initializer_service.py | Tests sanitized filesystem failures. |
| tests/unit/backend/test_auth_middleware.py | Tests authenticated object-ID lookup. |
| tests/unit/backend/test_analytics_benchmark.py | Tests benchmark recipe-storage isolation. |
| tests/unit/backend/conftest.py | Isolates saved-entry storage in backend tests. |
| tests/end_to_end/conftest.py | Gives launched backends temporary recipe storage. |
| test_support/analytics_benchmark.py | Configures isolated benchmark recipe storage. |
| pyrit/setup/initializers/tools.py | Rejects saved-entry collisions during tool setup. |
| pyrit/setup/initializers/targets.py | Rejects saved-target group collisions. |
| pyrit/setup/initializers/scorers.py | Prevents startup tags targeting saved replacements. |
| pyrit/setup/configuration_loader.py | Adds and validates recipe storage configuration. |
| pyrit/score/true_false/local_refusal_classifier_scorer.py | Declares model-backed save restrictions. |
| pyrit/score/true_false/audio_true_false_scorer.py | Declares audio scorer save restrictions. |
| pyrit/score/scorer.py | Defines scorer persistence-policy metadata. |
| pyrit/score/float_scale/azure_content_filter_scorer.py | Declares server credential fallback. |
| pyrit/score/float_scale/audio_float_scale_scorer.py | Declares audio scorer save restrictions. |
| pyrit/registry/scenario_suite_storage.py | Supports version-checked suite deletion. |
| pyrit/registry/scenario_preset_storage.py | Supports version-checked preset deletion. |
| pyrit/registry/components/scorer_registry.py | Reserves the scorer types route name. |
| pyrit/registry/components/scenario_suite_registry.py | Forwards suite deletion versions. |
| pyrit/registry/components/scenario_preset_registry.py | Forwards preset deletion versions. |
| pyrit/prompt_target/websocket_copilot_target.py | Declares account-dependent save restriction. |
| pyrit/prompt_target/prompt_shield_target.py | Declares credential and destination policies. |
| pyrit/prompt_target/openai/openai_target.py | Declares credential, header, and destination policies. |
| pyrit/prompt_target/litellm_chat_target.py | Declares API-key fallback policy. |
| pyrit/prompt_target/hugging_face/hugging_face_chat_target.py | Declares token and model-storage restrictions. |
| pyrit/prompt_target/http_target/http_target.py | Restricts saving raw HTTP requests. |
| pyrit/prompt_target/hack_a_prompt_target.py | Declares session and cookie fallback policies. |
| pyrit/prompt_target/github_copilot_target.py | Declares GitHub token fallback policy. |
| pyrit/prompt_target/common/prompt_target.py | Defines target persistence-policy metadata. |
| pyrit/prompt_target/azure_ml_chat_target.py | Declares credential and destination policies. |
| pyrit/prompt_target/azure_blob_storage_target.py | Declares token and container-address policies. |
| pyrit/prompt_target/a2a_target.py | Declares endpoint and agent-card destinations. |
| pyrit/models/catalog/target.py | Adds target origin and ownership fields. |
| pyrit/models/catalog/scorer.py | Adds scorer origin and ownership fields. |
| pyrit/models/catalog/__init__.py | Exports recipe and restore models lazily. |
| pyrit/converter/converter.py | Defines credential fallback metadata. |
| pyrit/converter/azure_speech_text_to_audio_converter.py | Declares Speech credential fallback. |
| pyrit/converter/azure_speech_audio_to_text_converter.py | Declares Speech credential fallback. |
| pyrit/backend/services/scenario_suite_service.py | Forwards suite deletion versions. |
| pyrit/backend/services/scenario_suite_execution_service.py | Exposes safe saved-entry conflict messages. |
| pyrit/backend/services/scenario_run_service.py | Reports launch-script saved-entry collisions. |
| pyrit/backend/services/scenario_preset_service.py | Forwards preset deletion versions. |
| pyrit/backend/routes/techniques.py | Adds persistence errors and caller ownership context. |
| pyrit/backend/routes/targets.py | Adds persistence errors and caller ownership context. |
| pyrit/backend/routes/scorers.py | Adds persistence errors and caller ownership context. |
| pyrit/backend/routes/scenarios.py | Documents saved-entry launch conflicts. |
| pyrit/backend/routes/scenario_suites.py | Adds conditional deletes and sanitized storage logging. |
| pyrit/backend/routes/scenario_presets.py | Adds conditional deletes and sanitized storage logging. |
| pyrit/backend/routes/restore_failures.py | Exposes restore failures and Retry. |
| pyrit/backend/routes/initializers.py | Sanitizes filesystem failure responses and logs. |
| pyrit/backend/routes/common.py | Shares ownership and save-refusal response helpers. |
| pyrit/backend/models/techniques.py | Adds persistence validation and ownership fields. |
| pyrit/backend/models/targets.py | Adds persistence validation and saveability metadata. |
| pyrit/backend/models/scorers.py | Adds scorer persistence validation. |
| pyrit/backend/models/scenario_suites.py | Documents delete-version usage. |
| pyrit/backend/models/scenario_presets.py | Documents delete-version usage. |
| pyrit/backend/models/saved_entries.py | Defines saved-entry inspection and editing contracts. |
| pyrit/backend/models/converters.py | Adds persistence validation and ownership fields. |
| pyrit/backend/middleware/runtime.py | Allows recovery and saved-entry management routes. |
| pyrit/backend/middleware/auth.py | Exposes the authenticated user’s object ID. |
| pyrit/backend/mappers/technique_mappers.py | Maps technique origin and ownership. |
| pyrit/backend/mappers/target_mappers.py | Maps target origin and ownership. |
| pyrit/backend/mappers/converter_mappers.py | Maps converter origin and ownership. |
| pyrit/backend/main.py | Registers recovery and saved-entry routers. |
| frontend/src/utils/registryOrigin.ts | Provides origin labels and ownership captions. |
| frontend/src/utils/registryName.ts | Generates valid, available registry names. |
| frontend/src/utils/registryName.test.ts | Tests name validation and suggestions. |
| frontend/src/test-utils/targetFixtures.ts | Supports origin and ownership fixtures. |
| frontend/src/services/errors.ts | Recognizes typed save refusals. |
| frontend/src/services/errors.test.ts | Tests save-refusal classification. |
| frontend/src/services/api.ts | Adds recovery, saved-entry, and conditional-delete calls. |
| frontend/src/services/api.test.ts | Tests delete-version request parameters. |
| frontend/src/hooks/useRuntime.styles.ts | Styles restore failure lists and actions. |
| frontend/src/components/Techniques/TechniqueEditor.test.tsx | Updates converter save-button expectations. |
| frontend/src/components/ScenarioSuites/ScenarioSuiteLibrary.tsx | Handles stale deletes and restores focus. |
| frontend/src/components/ScenarioSuites/ScenarioSuiteLibrary.test.tsx | Tests delete conflicts and focus recovery. |
| frontend/src/components/ScenarioPresets/ScenarioPresetLibrary.tsx | Handles stale deletes and restores focus. |
| frontend/src/components/ScenarioPresets/ScenarioPresetLibrary.test.tsx | Tests conditional deletes and focus recovery. |
| frontend/src/components/Registry/TechniqueRegistry.tsx | Adds origin, ownership filtering, and deletion. |
| frontend/src/components/Registry/ScorerRegistry.tsx | Adds origin, ownership filtering, and deletion. |
| frontend/src/components/Registry/ScorerRegistry.test.tsx | Tests ownership filtering and saved-row actions. |
| frontend/src/components/Registry/ScenarioPresetRegistry.test.tsx | Expects version-checked preset deletion. |
| frontend/src/components/Registry/SavedBy.tsx | Renders saved-entry ownership captions. |
| frontend/src/components/Registry/SavedBy.styles.ts | Styles ownership captions. |
| frontend/src/components/Registry/OriginBadge.tsx | Renders registry origin badges. |
| frontend/src/components/Registry/DeleteSavedEntryButton.styles.ts | Styles saved-entry deletion dialog content. |
| frontend/src/components/Registry/ConverterRegistry.tsx | Adds ownership filtering and saved-entry deletion. |
| frontend/src/components/Registry/ConverterRegistry.test.tsx | Tests origin, ownership, and deletion behavior. |
| frontend/src/components/Converters/SaveConverterDialog.tsx | Clarifies that converter copies remain temporary. |
| frontend/src/components/Converters/SaveConverterDialog.test.tsx | Tests temporary-copy messaging. |
| frontend/src/components/Config/TargetTable.tsx | Adds origin, ownership filtering, and saved deletion. |
| frontend/src/components/Config/TargetTable.styles.ts | Styles ownership captions and filter controls. |
| frontend/src/components/Config/TargetConfig.tsx | Refreshes targets after saved-entry deletion. |
| frontend/src/App.tsx | Moves the runtime banner inside the theme provider. |
| frontend/playwright.config.ts | Isolates seeded backend recipe storage. |
| frontend/e2e/fixtures/scenario_dataset_backend.py | Forwards delete versions through storage guards. |
| frontend/e2e/fixtures/manual_send_backend.py | Uses temporary saved-entry storage. |
| frontend/e2e/config.spec.ts | Tests target saving and ownership filtering. |
| frontend/e2e/_targets.ts | Adds origin and ownership mock fields. |
| doc/gui/0_gui.md | Documents saved entries, ownership, and deletion. |
| doc/getting_started/pyrit_conf.md | Documents recipe storage and restore recovery. |
| .pyrit_conf_example | Adds recipe storage configuration guidance. |
Review effort: Balanced
| status_code=status.HTTP_409_CONFLICT, | ||
| detail=f"Scenario preset '{name}' changed since it was read; re-read it and retry", | ||
| ) from None | ||
| except (AzureError, TimeoutError) as exc: |
Contributor
Author
| status_code=status.HTTP_409_CONFLICT, | ||
| detail=f"Scenario suite '{name}' changed since it was read; re-read it and retry", | ||
| ) from None | ||
| except (AzureError, TimeoutError) as exc: |
Contributor
Author
…hanges A 409 for a taken name or tag kept Save disabled until the name changed, so correcting the tag didn't help. The refusal now applies only to the name and tags it was given for.
A request that finished during the wait could register an entry that holds the one being deleted, and the delete then removed it and closed its resources anyway. Holders are now checked again after the wait, right before the entry is removed.
# Conflicts: # frontend/src/components/Registry/TechniqueRegistry.test.tsx
…rmed edits Editing an entry that failed to restore registered it even though the response said it wasn't running, so Retry then failed with a name clash. Its recipe is now only saved, and Retry or the next start restores it. What holds the old entry is listed after the rebuild, so an entry a request registered meanwhile keeps the old object open and is in still_using_old. A technique recipe with a different name or persist, a recipe with a different owner, and params that aren't an object are refused with a 422 instead of a 500 or being ignored.
A launch script that caught the refusal of a technique whose name is a saved technique's tag, or whose tag is a saved technique's name or another spelling of its tag, could leave the saved technique answering that selector. Those clashes are now recorded like a name clash, so the scan stops.
The deleted entry's row, and with the last scorer of a kind its whole table, unmounted with the focus in it, so focus fell back to the page body. Focus now goes to New Target, or to the Scorer Registry heading, as it goes to New Converter and New technique on those pages.
varunj-msft
requested review from
hannahwestra25 and
Richard Lundeen (richlundeen)
November 8, 2026 23:56
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This is one of the changes for saving Registry entries across restarts. Until the earlier PRs merge, the diff here also shows their changes.
Fixing a saved entry, including one that failed to restore, meant deleting it and saving it again under the same name.
GET /api/saved-entries/{kind}/{name}now returns the storedrecipeto a caller who may change the entry.PUT /api/saved-entries/{kind}/{name}takes it back, changed, with the version it read asexpected_version.persist. The name, the class or attack type, and who saved the entry stay the same, and the entry stays saved (422 otherwise).still_using_old. The old entry's resources are released only when nothing holds it and no work is in progress.runningis false); Retry restores it.What stays the same: creating, deleting and restoring work as before, and settings changed for one run in the Scanner are still never saved.
Not covered: