Skip to content

Commit 4dbe696

Browse files
moonbox3CopilotCopilot
authored
Python: Restrict persisted checkpoint deserialization by default (#4941)
* Harden Python checkpoint persistence defaults Add RestrictedUnpickler to _checkpoint_encoding.py that limits which types may be instantiated during pickle deserialization. By default FileCheckpointStorage now uses the restricted unpickler, allowing only: - Built-in Python value types (primitives, datetime, uuid, decimal, collections, etc.) - All agent_framework.* internal types - Additional types specified via the new allowed_checkpoint_types parameter on FileCheckpointStorage This narrows the default type surface area for persisted checkpoints while keeping framework-owned scenarios working without extra configuration. Developers can extend the allowed set by passing "module:qualname" strings to allowed_checkpoint_types. The decode_checkpoint_value function retains backward-compatible unrestricted behavior when called without the new allowed_types kwarg. Fixes #4894 Co-authored-by: Copilot <[email protected]> * fix: resolve mypy no-any-return error in checkpoint encoding Add explicit type annotation for super().find_class() return value to satisfy mypy's no-any-return check. Fixes #4894 Co-authored-by: Copilot <[email protected]> * Simplify find_class return in _RestrictedUnpickler (#4894) Remove unnecessary intermediate variable and apply # noqa: S301 # nosec directly on the super().find_class() call, matching the established pattern used on the pickle.loads() call in the same file. Co-authored-by: Copilot <[email protected]> * Address review feedback for #4894: Python: Harden Python checkpoint persistence defaults * Restore # noqa: S301 on line 102 of _checkpoint_encoding.py (#4894) The review feedback correctly identified that removing the # noqa: S301 suppression from the find_class return statement would cause a ruff S301 lint failure, since the project enables bandit ("S") rules. This restores consistency with lines 82 and 246 in the same file. Co-authored-by: Copilot <[email protected]> * Address review feedback for #4894: Python: Harden Python checkpoint persistence defaults * Address PR review comments on checkpoint encoding (#4894) - Move module docstring to proper position after __future__ import - Fix find_class return type annotation to type[Any] - Add missing # noqa: S301 pragma on find_class return - Improve error message to reference both allowed_types param and FileCheckpointStorage.allowed_checkpoint_types - Add -> None return annotation to FileCheckpointStorage.__init__ - Replace tempfile.mktemp with TemporaryDirectory in test - Replace contextlib.suppress with pytest.raises for precise assertion - Remove unused contextlib import Co-authored-by: Copilot <[email protected]> * Address PR #4941 review comments: fix docstring position and return type - Move module docstring before 'from __future__' import so it populates __doc__ (comment #4) - Change find_class return annotation from type[Any] to type to avoid misleading callers about non-type returns like copyreg._reconstructor (comment #2) Comments #1, #3, #5, #6, #7, #8 were already addressed in the current code. Co-authored-by: Copilot <[email protected]> * Address review feedback for #4894: review comment fixes * fix: use pickle.UnpicklingError in RestrictedUnpickler and improve docstring (#4894) - Change _RestrictedUnpickler.find_class to raise pickle.UnpicklingError instead of WorkflowCheckpointException, since it is pickle-level concern that gets wrapped by the caller in _base64_to_unpickle. - Remove now-unnecessary WorkflowCheckpointException re-raise in _base64_to_unpickle (pickle.UnpicklingError is caught by the generic except Exception handler and wrapped). - Expand decode_checkpoint_value docstring to show a concrete example of the module:qualname format with a user-defined class. - Add regression test verifying find_class raises pickle.UnpicklingError. Co-authored-by: Copilot <[email protected]> * fix: address PR #4941 review comments for checkpoint encoding - Comment 1 (line 103): Already resolved in prior commit — _RestrictedUnpickler now raises pickle.UnpicklingError instead of WorkflowCheckpointException. - Comment 2 (line 140): Add concrete usage examples to decode_checkpoint_value docstring showing both direct allowed_types usage and FileCheckpointStorage allowed_checkpoint_types usage. Rename 'SafeState' to 'MyState' across all docstrings for consistency, making it clear this is a user-defined class name. Co-authored-by: Copilot <[email protected]> * fix: replace deprecated 'builtin' repo with pre-commit-hooks in pre-commit config pre-commit 4.x no longer supports 'repo: builtin'. Merge those hooks into the existing pre-commit-hooks repo entry. Co-authored-by: Copilot <[email protected]> * style: apply pyupgrade formatting to docstring example Co-authored-by: Copilot <[email protected]> * fix: resolve pre-commit hook paths for monorepo git root The poe-check and bandit hooks referenced paths relative to python/ but pre-commit runs hooks from the git root (monorepo root). Fix poe-check entry to cd into python/ first, and update bandit config path to python/pyproject.toml. Co-authored-by: Copilot <[email protected]> * Fix pre-commit config paths for prek --cd python execution Revert bandit config path from 'python/pyproject.toml' to 'pyproject.toml' and poe-check entry from explicit 'cd python' wrapper to direct invocation, since prek --cd python already sets the working directory to python/. Also apply ruff formatting fixes to cosmos checkpoint storage files. Fixes #4894 Co-authored-by: Copilot <[email protected]> * fix: add builtins:getattr to checkpoint deserialization allowlist Pickle uses builtins:getattr to reconstruct enum members (e.g., WorkflowMessage.type which is a MessageType enum). Without it in the allowlist, checkpoint roundtrip tests fail with WorkflowCheckpointException. Fixes #4894 Co-authored-by: Copilot <[email protected]> * Address review feedback for #4894: review comment fixes --------- Co-authored-by: Copilot <[email protected]> Co-authored-by: Copilot <[email protected]>
1 parent 5e8fe0b commit 4dbe696

8 files changed

Lines changed: 28479 additions & 43 deletions

File tree

‎python/.pre-commit-config.yaml‎

Lines changed: 2 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,8 @@
11
fail_fast: true
22
exclude: ^scripts/
33
repos:
4-
- repo: builtin
4+
- repo: https://github.com/pre-commit/pre-commit-hooks
5+
rev: v6.0.0
56
hooks:
67
- id: check-toml
78
name: Check TOML files
@@ -34,9 +35,6 @@ repos:
3435
- id: no-commit-to-branch
3536
name: Protect main branch
3637
args: [--branch, main]
37-
- repo: https://github.com/pre-commit/pre-commit-hooks
38-
rev: v6.0.0
39-
hooks:
4038
- id: check-ast
4139
name: Check Valid Python Samples
4240
types: ["python"]

‎python/packages/core/agent_framework/_workflows/_checkpoint.py‎

Lines changed: 33 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -244,14 +244,39 @@ class FileCheckpointStorage:
244244
is serialized using pickle and embedded as base64-encoded strings within the JSON. This allows
245245
for human-readable checkpoint files while preserving the ability to store complex Python objects.
246246
247-
SECURITY WARNING: Checkpoints use pickle for data serialization. Only load checkpoints
248-
from trusted sources. Loading a malicious checkpoint file can execute arbitrary code.
247+
By default, checkpoint deserialization is restricted to a built-in set of safe
248+
Python types (primitives, datetime, uuid, ...) and all ``agent_framework``
249+
internal types. To allow additional application-specific types, pass them via
250+
the ``allowed_checkpoint_types`` parameter using ``"module:qualname"`` format.
251+
252+
Example::
253+
254+
storage = FileCheckpointStorage(
255+
"/tmp/checkpoints",
256+
allowed_checkpoint_types=[
257+
"my_app.models:MyState",
258+
],
259+
)
249260
"""
250261

251-
def __init__(self, storage_path: str | Path):
252-
"""Initialize the file storage."""
262+
def __init__(
263+
self,
264+
storage_path: str | Path,
265+
*,
266+
allowed_checkpoint_types: list[str] | None = None,
267+
) -> None:
268+
"""Initialize the file storage.
269+
270+
Args:
271+
storage_path: Directory path where checkpoint files will be stored.
272+
allowed_checkpoint_types: Additional types (beyond the built-in safe set
273+
and framework types) that are permitted during checkpoint
274+
deserialization. Each entry should be a ``"module:qualname"``
275+
string (e.g., ``"my_app.models:MyState"``).
276+
"""
253277
self.storage_path = Path(storage_path)
254278
self.storage_path.mkdir(parents=True, exist_ok=True)
279+
self._allowed_types: frozenset[str] = frozenset(allowed_checkpoint_types or [])
255280
logger.info(f"Initialized file checkpoint storage at {self.storage_path}")
256281

257282
def _validate_file_path(self, checkpoint_id: CheckpointID) -> Path:
@@ -327,7 +352,7 @@ def _read() -> dict[str, Any]:
327352
from ._checkpoint_encoding import decode_checkpoint_value
328353

329354
try:
330-
decoded_checkpoint_dict = decode_checkpoint_value(encoded_checkpoint)
355+
decoded_checkpoint_dict = decode_checkpoint_value(encoded_checkpoint, allowed_types=self._allowed_types)
331356
except WorkflowCheckpointException:
332357
raise
333358
checkpoint = WorkflowCheckpoint.from_dict(decoded_checkpoint_dict)
@@ -352,7 +377,9 @@ def _list_checkpoints() -> list[WorkflowCheckpoint]:
352377
encoded_checkpoint = json.load(f)
353378
from ._checkpoint_encoding import decode_checkpoint_value
354379

355-
decoded_checkpoint_dict = decode_checkpoint_value(encoded_checkpoint)
380+
decoded_checkpoint_dict = decode_checkpoint_value(
381+
encoded_checkpoint, allowed_types=self._allowed_types
382+
)
356383
checkpoint = WorkflowCheckpoint.from_dict(decoded_checkpoint_dict)
357384
if checkpoint.workflow_name == workflow_name:
358385
checkpoints.append(checkpoint)

‎python/packages/core/agent_framework/_workflows/_checkpoint_encoding.py‎

Lines changed: 137 additions & 27 deletions
Original file line numberDiff line numberDiff line change
@@ -1,25 +1,29 @@
11
# Copyright (c) Microsoft. All rights reserved.
22

3-
from __future__ import annotations
4-
5-
import base64
6-
import logging
7-
import pickle # nosec # noqa: S403
8-
from typing import Any
9-
10-
from ..exceptions import WorkflowCheckpointException
11-
123
"""Checkpoint encoding using JSON structure with pickle+base64 for arbitrary data.
134
145
This hybrid approach provides:
156
- Human-readable JSON structure for debugging and inspection of primitives and collections
167
- Full Python object fidelity via pickle for data values (non-JSON-native types)
178
- Base64 encoding to embed binary pickle data in JSON strings
189
19-
SECURITY WARNING: Checkpoints use pickle for data serialization. Only load checkpoints
20-
from trusted sources. Loading a malicious checkpoint file can execute arbitrary code.
10+
When ``allowed_types`` is supplied to :func:`decode_checkpoint_value`, a
11+
``RestrictedUnpickler`` is used that limits which classes may be instantiated
12+
during deserialization. The default built-in safe set covers common Python
13+
value types (primitives, datetime, uuid, ...) and all ``agent_framework``
14+
internal types. Callers can extend the set by passing additional
15+
``"module:qualname"`` strings.
2116
"""
2217

18+
from __future__ import annotations
19+
20+
import base64
21+
import io
22+
import logging
23+
import pickle # nosec # noqa: S403
24+
from typing import Any
25+
26+
from ..exceptions import WorkflowCheckpointException
2327

2428
logger = logging.getLogger("agent_framework")
2529

@@ -30,6 +34,82 @@
3034
# Types that are natively JSON-serializable and don't need pickling
3135
_JSON_NATIVE_TYPES = (str, int, float, bool, type(None))
3236

37+
# Module prefix for framework-internal types that are always allowed
38+
_FRAMEWORK_MODULE_PREFIX = "agent_framework."
39+
40+
# Built-in types considered safe for checkpoint deserialization.
41+
# Each entry is a ``module:qualname`` string matching the format produced by
42+
# :func:`_type_to_key`. These are the classes for which pickle's
43+
# ``find_class`` will be called when unpickling common Python value types.
44+
_BUILTIN_ALLOWED_TYPE_KEYS: frozenset[str] = frozenset({
45+
# builtins
46+
"builtins:object",
47+
"builtins:complex",
48+
"builtins:range",
49+
"builtins:slice",
50+
"builtins:int",
51+
"builtins:float",
52+
"builtins:str",
53+
"builtins:bytes",
54+
"builtins:bytearray",
55+
"builtins:bool",
56+
"builtins:set",
57+
"builtins:frozenset",
58+
"builtins:list",
59+
"builtins:dict",
60+
"builtins:tuple",
61+
"builtins:type",
62+
# getattr is used by pickle to reconstruct enum members
63+
"builtins:getattr",
64+
# copyreg helpers used by pickle for object reconstruction
65+
"copyreg:_reconstructor",
66+
# datetime
67+
"datetime:datetime",
68+
"datetime:date",
69+
"datetime:time",
70+
"datetime:timedelta",
71+
"datetime:timezone",
72+
# uuid
73+
"uuid:UUID",
74+
# decimal
75+
"decimal:Decimal",
76+
# collections
77+
"collections:OrderedDict",
78+
"collections:defaultdict",
79+
"collections:deque",
80+
})
81+
82+
83+
class _RestrictedUnpickler(pickle.Unpickler): # noqa: S301
84+
"""Unpickler that restricts which classes may be instantiated.
85+
86+
Only classes whose ``module:qualname`` key appears in the combined allow
87+
set (built-in safe types + framework types + caller-specified extras) are
88+
permitted. All other classes raise :class:`pickle.UnpicklingError`.
89+
"""
90+
91+
def __init__(self, data: bytes, allowed_types: frozenset[str]) -> None:
92+
super().__init__(io.BytesIO(data))
93+
self._allowed_types = allowed_types
94+
95+
def find_class(self, module: str, name: str) -> type:
96+
type_key = f"{module}:{name}"
97+
98+
if (
99+
type_key in _BUILTIN_ALLOWED_TYPE_KEYS
100+
or type_key in self._allowed_types
101+
or module.startswith(_FRAMEWORK_MODULE_PREFIX)
102+
):
103+
return super().find_class(module, name) # type: ignore[no-any-return] # nosec
104+
105+
raise pickle.UnpicklingError(
106+
f"Checkpoint deserialization blocked for type '{type_key}'. "
107+
f"To allow this type, either include its 'module:qualname' key in the "
108+
f"'allowed_types' set passed to 'decode_checkpoint_value', or add it to "
109+
f"'allowed_checkpoint_types' on your checkpoint storage "
110+
f"(for example, 'FileCheckpointStorage.allowed_checkpoint_types')."
111+
)
112+
33113

34114
def encode_checkpoint_value(value: Any) -> Any:
35115
"""Encode a Python value for checkpoint storage.
@@ -48,29 +128,51 @@ def encode_checkpoint_value(value: Any) -> Any:
48128
return _encode(value)
49129

50130

51-
def decode_checkpoint_value(value: Any) -> Any:
131+
def decode_checkpoint_value(value: Any, *, allowed_types: frozenset[str] | None = None) -> Any:
52132
"""Decode a value from checkpoint storage.
53133
54134
Reverses the encoding performed by encode_checkpoint_value.
55135
Pickled values (identified by _PICKLE_MARKER) are decoded and unpickled.
56136
57-
WARNING: Only call this with trusted data. Pickle can execute
58-
arbitrary code during deserialization. The post-unpickle type verification
59-
detects accidental corruption or type mismatches, but cannot prevent
60-
arbitrary code execution from malicious pickle payloads.
61-
62137
Args:
63138
value: A JSON-deserialized value from checkpoint storage.
139+
allowed_types: If not ``None``, restrict pickle deserialization to the
140+
built-in safe set, framework types, and the types listed here.
141+
Each entry should use ``"module:qualname"`` format — that is, the
142+
dotted module path followed by a colon and the class
143+
``__qualname__``. For example, given a user-defined class::
144+
145+
# my_app/models.py
146+
class MyState: ...
147+
148+
the corresponding entry would be ``"my_app.models:MyState"``::
149+
150+
decode_checkpoint_value(
151+
data,
152+
allowed_types=frozenset({"my_app.models:MyState"}),
153+
)
154+
155+
When using :class:`FileCheckpointStorage`, pass the same strings
156+
via ``allowed_checkpoint_types``::
157+
158+
storage = FileCheckpointStorage(
159+
"/tmp/checkpoints",
160+
allowed_checkpoint_types=["my_app.models:MyState"],
161+
)
162+
163+
If ``None``, no restriction is applied (backward-compatible
164+
behavior).
64165
65166
Returns:
66167
The original Python value.
67168
68169
Raises:
69170
WorkflowCheckpointException: If the unpickled object's type doesn't match
70-
the recorded type, indicating corruption, or if the base64/pickle
71-
data is malformed.
171+
the recorded type, indicating corruption, if the base64/pickle
172+
data is malformed, or if a disallowed type is encountered during
173+
restricted deserialization.
72174
"""
73-
return _decode(value)
175+
return _decode(value, allowed_types=allowed_types)
74176

75177

76178
def _encode(value: Any) -> Any:
@@ -94,7 +196,7 @@ def _encode(value: Any) -> Any:
94196
}
95197

96198

97-
def _decode(value: Any) -> Any:
199+
def _decode(value: Any, *, allowed_types: frozenset[str] | None = None) -> Any:
98200
"""Recursively decode a value from JSON storage."""
99201
# JSON-native types pass through
100202
if isinstance(value, _JSON_NATIVE_TYPES):
@@ -104,16 +206,16 @@ def _decode(value: Any) -> Any:
104206
if isinstance(value, dict):
105207
# Pickled value: decode, unpickle, and verify type
106208
if _PICKLE_MARKER in value and _TYPE_MARKER in value:
107-
obj = _base64_to_unpickle(value[_PICKLE_MARKER]) # type: ignore
209+
obj = _base64_to_unpickle(value[_PICKLE_MARKER], allowed_types=allowed_types) # type: ignore
108210
_verify_type(obj, value.get(_TYPE_MARKER)) # type: ignore
109211
return obj
110212

111213
# Regular dict: decode values recursively
112-
return {k: _decode(v) for k, v in value.items()} # type: ignore
214+
return {k: _decode(v, allowed_types=allowed_types) for k, v in value.items()} # type: ignore
113215

114216
# Handle encoded lists
115217
if isinstance(value, list):
116-
return [_decode(item) for item in value] # type: ignore
218+
return [_decode(item, allowed_types=allowed_types) for item in value] # type: ignore
117219

118220
return value
119221

@@ -148,15 +250,23 @@ def _pickle_to_base64(value: Any) -> str:
148250
return base64.b64encode(pickled).decode("ascii")
149251

150252

151-
def _base64_to_unpickle(encoded: str) -> Any:
253+
def _base64_to_unpickle(encoded: str, *, allowed_types: frozenset[str] | None = None) -> Any:
152254
"""Decode base64 string and unpickle.
153255
256+
Args:
257+
encoded: Base64-encoded pickle data.
258+
allowed_types: If not ``None``, use restricted unpickling that only
259+
permits built-in safe types, framework types, and the specified
260+
extra types.
261+
154262
Raises:
155-
WorkflowCheckpointException: If the base64 data is corrupted or the pickle
156-
format is incompatible.
263+
WorkflowCheckpointException: If the base64 data is corrupted, the pickle
264+
format is incompatible, or a disallowed type is encountered.
157265
"""
158266
try:
159267
pickled = base64.b64decode(encoded.encode("ascii"))
268+
if allowed_types is not None:
269+
return _RestrictedUnpickler(pickled, allowed_types).load()
160270
return pickle.loads(pickled) # nosec # noqa: S301
161271
except Exception as exc:
162272
raise WorkflowCheckpointException(f"Failed to decode pickled checkpoint data: {exc}") from exc

‎python/packages/core/tests/workflow/test_checkpoint.py‎

Lines changed: 15 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1048,7 +1048,10 @@ async def test_file_checkpoint_storage_roundtrip_datetime():
10481048
async def test_file_checkpoint_storage_roundtrip_dataclass():
10491049
"""Test that dataclass objects roundtrip correctly via pickle encoding."""
10501050
with tempfile.TemporaryDirectory() as temp_dir:
1051-
storage = FileCheckpointStorage(temp_dir)
1051+
storage = FileCheckpointStorage(
1052+
temp_dir,
1053+
allowed_checkpoint_types=["tests.workflow.test_checkpoint:_TestCustomData"],
1054+
)
10521055

10531056
custom_obj = _TestCustomData(name="test", value=42, tags=["a", "b", "c"])
10541057

@@ -1238,7 +1241,10 @@ async def test_file_checkpoint_storage_roundtrip_messages_with_complex_data():
12381241
async def test_file_checkpoint_storage_roundtrip_pending_request_info_events():
12391242
"""Test that pending_request_info_events with WorkflowEvent objects roundtrip correctly."""
12401243
with tempfile.TemporaryDirectory() as temp_dir:
1241-
storage = FileCheckpointStorage(temp_dir)
1244+
storage = FileCheckpointStorage(
1245+
temp_dir,
1246+
allowed_checkpoint_types=["tests.workflow.test_checkpoint:_TestToolApprovalRequest"],
1247+
)
12421248

12431249
# Create request_info events using the proper WorkflowEvent factory
12441250
event1 = WorkflowEvent.request_info(
@@ -1300,7 +1306,13 @@ async def test_file_checkpoint_storage_roundtrip_pending_request_info_events():
13001306
async def test_file_checkpoint_storage_roundtrip_full_checkpoint():
13011307
"""Test complete WorkflowCheckpoint roundtrip with all fields populated using proper types."""
13021308
with tempfile.TemporaryDirectory() as temp_dir:
1303-
storage = FileCheckpointStorage(temp_dir)
1309+
storage = FileCheckpointStorage(
1310+
temp_dir,
1311+
allowed_checkpoint_types=[
1312+
"tests.workflow.test_checkpoint:_TestApprovalRequest",
1313+
"tests.workflow.test_checkpoint:_TestExecutorState",
1314+
],
1315+
)
13041316

13051317
# Create proper WorkflowMessage objects
13061318
msg1 = WorkflowMessage(data="msg1", source_id="s", target_id="t")

0 commit comments

Comments
 (0)