Repository navigation
FEAT: Save converter copies - #4069
Open
varunj-msft wants to merge 174 commits into
Open
varunj-msft wants to merge 174 commits into
varunj-msft wants to merge 174 commits into
Conversation
…API input External construction now rejects a parameter whose type includes Path unless the calling service passes the name in allowed_path_params after making the value safe. The converter service allows its upload parameters once they are stored, and the target service allows the server-set upload folder and the GitHubCopilotTarget working directory. The scorer-only check is removed, and the "Valid parameters" list leaves out path parameters callers cannot set.
The target catalog now applies the path rule to required parameters too, so a target that needs a disallowed path is left out instead of listed without the field, matching the scorer catalog. allowed_path_params is typed frozenset[str] so a bare string cannot turn the allow-list into a substring match.
The target catalog still lists a target whose required path parameter is its server-set upload folder when the server has one, and leaves it out otherwise.
# Conflicts: # tests/unit/backend/test_converter_service.py
…benchmark's recipes folder
# Conflicts: # tests/unit/registry/test_registry_recipe_rules.py
# Conflicts: # pyrit/backend/mappers/target_mappers.py # pyrit/backend/models/scorers.py # pyrit/backend/routes/converters.py # pyrit/backend/routes/targets.py # pyrit/backend/services/converter_service.py # pyrit/backend/services/scorer_service.py # pyrit/backend/services/target_service.py # pyrit/registry/instance_registry.py # tests/unit/backend/test_api_routes.py # tests/unit/backend/test_converter_service.py # tests/unit/backend/test_mappers.py # tests/unit/backend/test_runtime_lifecycle.py # tests/unit/backend/test_scorer_service.py # tests/unit/backend/test_target_service.py # tests/unit/registry/test_instance_registry.py
main now declares that TextTarget takes no credentials, with an empty supported_auth_modes. The saved-entry check read that as supporting no mode at all, so it refused saved TextTargets and left the saveability check for the target types with no mode to report. A class that declares no auth modes is now checked as the default api_key mode, which it has no key for.
…d to OpenAIChatTarget
# Conflicts: # pyrit/registry/registry_recipe_rules.py
A reference left to a configured default can be a list, such as a composite scorer's scorers. Each entry in the list is now named in the saved recipe, in order, and the list is refused if any of them is Temporary or not registered, instead of the whole list being refused as one unregistered entry.
# Conflicts: # frontend/src/components/Registry/CreateConverterDialog.tsx
# Conflicts: # pyrit/converter/azure_speech_audio_to_text_converter.py # pyrit/converter/azure_speech_text_to_audio_converter.py # pyrit/prompt_target/common/prompt_target.py
A copy can't change its source's connection settings even when it isn't saved, so that refusal now comes first instead of a not-saveable one that says the copy can still be created without saving it. Tests cover a credential change, a saved copy edited to send requests elsewhere, and the credential check for a class that lets copies change one.
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
Synchronous discovery and component construction in restore() can stall health and configuration requests during live apply and should be offloaded while preserving initialization that requires the runtime loop.
1 open issue
What changed in this PR
Adds persistent converter copies to PyRIT’s registry, building on the stacked persistence and connection-setting safeguards. Saved copies retain only their overrides and are rebuilt from their source after restart.
Changes:
- Adds saved-entry recipes, restore handling, origin metadata, and safeguards against replacing saved entries.
- Makes converter-copy saving the default in the dialog, with a Temporary fallback and name validation.
- Extends registry displays, documentation, and regression coverage, while isolating test recipe storage.
| File | Description |
|---|---|
| tests/unit/setup/test_targets_initializer.py | Tests saved-target collision protection. |
| tests/unit/setup/test_scorer_initializer.py | Tests saved-scorer tagging safeguards. |
| tests/unit/setup/test_converter_initializer.py | Tests saved-converter replacement refusal. |
| tests/unit/setup/test_configuration_loader.py | Tests recipe storage configuration. |
| tests/unit/registry/test_resolution.py | Tests copy-locked parameter metadata. |
| tests/unit/registry/test_instance_registry.py | Tests saved-entry protection and origins. |
| tests/unit/registry/test_converter_registry.py | Tests collisions and locked copy settings. |
| tests/unit/registry/test_attack_technique_registry.py | Tests saved-technique collision handling. |
| tests/unit/models/test_parameter.py | Updates parameter serialization expectations. |
| tests/unit/backend/test_temporary_components.py | Tests target-copy connection safeguards. |
| tests/unit/backend/test_technique_edit_service.py | Tests locked converter settings in edits. |
| tests/unit/backend/test_scorer_routes.py | Tests scorer persistence and API errors. |
| tests/unit/backend/test_scenario_suite_routes.py | Tests sanitized storage failures. |
| tests/unit/backend/test_scenario_suite_execution_service.py | Tests saved-entry launch conflict messages. |
| tests/unit/backend/test_scenario_run_service.py | Tests launch-script replacement safeguards. |
| tests/unit/backend/test_scenario_preset_routes.py | Tests sanitized storage responses and logs. |
| tests/unit/backend/test_saved_entries.py | Tests temporary-reference and credential refusals. |
| tests/unit/backend/test_mappers.py | Tests target origin and saver metadata. |
| tests/unit/backend/test_initializer_service.py | Tests local storage error sanitization. |
| tests/unit/backend/test_auth_middleware.py | Tests current-user ID extraction. |
| tests/unit/backend/test_analytics_benchmark.py | Tests isolated benchmark recipe storage. |
| tests/unit/backend/conftest.py | Isolates saved entries in backend tests. |
| tests/end_to_end/conftest.py | Uses temporary backend recipe storage. |
| test_support/analytics_benchmark.py | Configures benchmark recipe isolation. |
| pyrit/setup/initializers/tools.py | Rejects saved-entry substitutions. |
| pyrit/setup/initializers/targets.py | Protects saved names during target grouping. |
| pyrit/setup/initializers/scorers.py | Prevents tagging saved substitutes. |
| pyrit/setup/configuration_loader.py | Adds recipe storage configuration. |
| pyrit/score/true_false/local_refusal_classifier_scorer.py | Declares persistence restrictions. |
| pyrit/score/true_false/audio_true_false_scorer.py | Declares transcription credential restrictions. |
| pyrit/score/scorer.py | Adds scorer persistence policy metadata. |
| pyrit/score/float_scale/azure_content_filter_scorer.py | Declares credential fallback behavior. |
| pyrit/score/float_scale/audio_float_scale_scorer.py | Declares transcription persistence restrictions. |
| pyrit/registry/resolution.py | Derives copy-locked parameter metadata. |
| pyrit/registry/registry.py | Rejects locked external copy overrides. |
| pyrit/registry/components/scorer_registry.py | Reserves the scorer types route name. |
| pyrit/registry/components/attack_technique_registry.py | Preserves metadata and detects saved collisions. |
| pyrit/prompt_target/websocket_copilot_target.py | Declares target persistence restrictions. |
| pyrit/prompt_target/prompt_shield_target.py | Declares credential and destination policies. |
| pyrit/prompt_target/openai/openai_target.py | Declares persistence policies and locked settings. |
| pyrit/prompt_target/litellm_chat_target.py | Declares API-key fallback behavior. |
| pyrit/prompt_target/hugging_face/hugging_face_chat_target.py | Declares token and model persistence restrictions. |
| pyrit/prompt_target/http_target/http_target.py | Prevents saving credential-bearing HTTP requests. |
| pyrit/prompt_target/hack_a_prompt_target.py | Declares session and cookie fallback behavior. |
| pyrit/prompt_target/github_copilot_target.py | Declares token fallback behavior. |
| pyrit/prompt_target/common/prompt_target.py | Adds shared persistence and copy policies. |
| pyrit/prompt_target/azure_ml_chat_target.py | Declares credential and destination policies. |
| pyrit/prompt_target/azure_blob_storage_target.py | Declares SAS and destination policies. |
| pyrit/prompt_target/a2a_target.py | Declares destination validation metadata. |
| pyrit/models/parameter.py | Adds the copy-locked parameter flag. |
| pyrit/models/catalog/target.py | Adds target origin and saver fields. |
| pyrit/models/catalog/scorer.py | Adds scorer origin metadata. |
| pyrit/models/catalog/registry_recipe.py | Defines saved recipes and restore reports. |
| pyrit/models/catalog/__init__.py | Exports recipe and restore models. |
| pyrit/converter/converter.py | Adds shared copy and credential policies. |
| pyrit/converter/azure_speech_text_to_audio_converter.py | Locks speech connection settings. |
| pyrit/converter/azure_speech_audio_to_text_converter.py | Locks speech connection settings. |
| pyrit/backend/services/scorer_service.py | Saves scorer recipes and reports origins. |
| pyrit/backend/services/scenario_suite_execution_service.py | Exposes safe saved-entry conflict messages. |
| pyrit/backend/services/scenario_run_service.py | Detects launch-script saved-entry conflicts. |
| pyrit/backend/services/runtime_lifecycle.py | Restores recipes and reports readiness failures. |
| pyrit/backend/routes/targets.py | Passes saver context and maps persistence errors. |
| pyrit/backend/routes/scorers.py | Supports scorer persistence responses. |
| pyrit/backend/routes/scenarios.py | Documents saved-entry launch conflicts. |
| pyrit/backend/routes/scenario_suites.py | Sanitizes storage failure logging. |
| pyrit/backend/routes/scenario_presets.py | Sanitizes storage failure logging. |
| pyrit/backend/routes/initializers.py | Handles and sanitizes storage failures. |
| pyrit/backend/routes/converters.py | Maps save errors and blocks saved removal. |
| pyrit/backend/routes/common.py | Adds saver and save-refusal helpers. |
| pyrit/backend/README.md | Documents persistence APIs and converter copies. |
| pyrit/backend/models/targets.py | Adds target persistence request validation. |
| pyrit/backend/models/scorers.py | Adds scorer persistence request validation. |
| pyrit/backend/models/converters.py | Adds converter persistence and origin fields. |
| pyrit/backend/middleware/auth.py | Exposes the current user’s object ID. |
| pyrit/backend/mappers/target_mappers.py | Maps target origin and saver metadata. |
| pyrit/backend/mappers/converter_mappers.py | Maps converter origin metadata. |
| frontend/src/utils/registryOrigin.ts | Defines shared origin labels. |
| frontend/src/utils/registryName.ts | Generates valid, available registry names. |
| frontend/src/utils/registryName.test.ts | Tests name validation and suggestions. |
| frontend/src/types/index.ts | Adds persistence and origin API types. |
| frontend/src/test-utils/targetFixtures.ts | Supports origin and saver test data. |
| frontend/src/services/errors.ts | Recognizes typed save refusals. |
| frontend/src/services/errors.test.ts | Tests save-refusal detection. |
| frontend/src/components/Techniques/TechniqueEditor.test.tsx | Updates converter save expectations. |
| frontend/src/components/Registry/ScorerRegistry.tsx | Displays scorer origin badges. |
| frontend/src/components/Registry/ScorerRegistry.test.tsx | Tests scorer origin labels. |
| frontend/src/components/Registry/OriginBadge.tsx | Adds a shared registry origin badge. |
| frontend/src/components/Registry/ConverterRegistry.tsx | Shows origins and disables saved removal. |
| frontend/src/components/Registry/ConverterRegistry.test.tsx | Tests origin labels and removal restrictions. |
| frontend/src/components/Converters/SaveConverterDialog.tsx | Saves copies with Temporary fallback. |
| frontend/src/components/Converters/SaveConverterDialog.test.tsx | Tests copy saving, refusals, and names. |
| frontend/src/components/Config/TargetTable.tsx | Shows origins, savers, and ownership filtering. |
| frontend/src/components/Config/TargetTable.test.tsx | Tests origin and saver displays. |
| frontend/src/components/Config/TargetTable.styles.ts | Styles saver labels and filter controls. |
| frontend/src/components/Chat/ConverterPanel.save.test.tsx | Expects persistent converter-copy requests. |
| frontend/playwright.config.ts | Isolates seeded backend recipe storage. |
| frontend/e2e/fixtures/manual_send_backend.py | Isolates offline fixture recipe storage. |
| frontend/e2e/config.spec.ts | Tests target saving and saver filtering. |
| frontend/e2e/_targets.ts | Adds origin and saver mock fields. |
| doc/gui/0_gui.md | Explains saved entries and converter copies. |
| doc/getting_started/pyrit_conf.md | Documents recipe storage and restore failures. |
| doc/code/registry/0_registry.md | Documents locked copy connection settings. |
| .pyrit_conf_example | Adds commented recipe storage configuration. |
Review effort: Balanced
| # the target API builds them, because a constructor such as HuggingFaceChatTarget's starts a task | ||
| # on it. A slow constructor holds the loop while it runs. | ||
| saved = await asyncio.to_thread(restorer.read) | ||
| report = restorer.restore(saved=saved) |
Contributor
Author
There was a problem hiding this comment.
Leaving this as it is. The restore code comes from #3909 and builds entries on the loop on purpose: a constructor such as HuggingFaceChatTarget's starts a task on the running loop, and the target API builds them the same way. Reading the files already runs in a worker thread. A saved copy doesn't change that, since the converter it copies was built the same way at startup.
A copy whose changes cleared a reference, such as a converter's target, took the configured default while it was saved and wrote the cleared value, so a Temporary default wasn't refused and the copy depended on defaults set again at every start. The changes now name that entry as a new saved entry's do, and a Temporary or unregistered default is refused.
The cached service keeps the registry it was made with, so after another test reset the registry the request couldn't find the converter this test registered. The test now clears the cache and the registry before and after.
varunj-msft
requested review from
hannahwestra25 and
Richard Lundeen (richlundeen)
November 8, 2026 23:56
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This is one of the changes for saving Registry entries across restarts. It needs #3984, which keeps a copy's endpoint and credentials the same as its source's. Until the earlier PRs merge, the diff here also shows their changes.
Save to registry on a converter's changed settings, in chat or in a technique edit, made a Temporary copy that was gone after a restart. Copies can now be saved, and a saved copy is rebuilt from the converter it copies at every start.
sourceand"persist": trueis saved as a recipe that names the converter it copies and holds only the changed settings. At every start it's rebuilt from that converter as it is then, after it, so it keeps where that converter sends requests and how it signs in.What stays the same: a copy made with Create as Temporary is Temporary, as before, and a copy still can't change its source's connection settings (#3984). Target copies are unchanged.
Screenshots: Save to registry for a copy of a saved converter, the saved copy in the Converters table, and a copy of a Temporary converter offered as Temporary with the reason.
Not covered:
sourceon targets) and gives copies per-caller credentials.