Pickle: Python Object Serialization

Overview

• Pickle Module: Implements binary protocols for serializing and de-serializing Python object structures.
• Pickling: Converting a Python object hierarchy to a byte stream.
• Unpickling: Converting a byte stream back to a Python object hierarchy.
• Security Warning: Pickle is not secure against incorrect or malicious data. Only unpickle data from trusted sources.

Comparison with Other Modules

Pickle vs. Marshal

• Pickle is preferred for serialization as it supports more types and maintains object sharing.
• Marshal is more primitive and not suitable for user-defined classes.
• Portability: Marshal is not portable across Python versions.

Pickle vs. JSON

• Format: Pickle is binary; JSON is text-based and human-readable.
• Scope: Pickle is Python-specific; JSON is language-independent.
• Security: JSON is safer for untrusted data.

Data Stream Format

• Pickle's format is Python-specific, which limits interoperability with non-Python programs.
• Supports 6 protocols, with higher versions requiring newer Python versions.

Module Interface

• Functions: dumps() for serializing, loads() for de-serializing.
• Pickler & Unpickler: Offer more control over the pickling process.

What Can Be Pickled

• Supports basic data types (integers, strings, etc.), collections, functions, classes, and instances of classes.
• Limitations: Highly recursive structures may lead to errors.

Pickling Class Instances

• Customization: Classes can define special methods like __getstate__ and __setstate__ to control pickling behavior.

Advanced Features

Persistent External Objects

• Uses persistent IDs to reference objects outside the pickle stream.
• Custom methods persistent_id and persistent_load are used to manage these references.

Dispatch Tables

• Allows customization of pickling for specific classes without affecting global behavior.

Handling Stateful Objects

• Example provided to show state management for pickled objects.

Custom Reduction

• Reducer Override: Subclassing Pickler to implement custom reduction logic for object serialization.

Out-of-band Buffers

• Available from protocol 5, allows for efficient large data transfers without copying.

Security

• Unpickling can invoke arbitrary code, hence restrict globals to ensure safety.
• Example of a restricted unpickler provided.

Performance

• Protocols 2+ offer efficient binary encodings and optimizations in C.

Examples

• Basic usage of dump() and load() for serializing a dictionary of diverse objects.

Additional Resources

• Modules like copyreg, pickletools, and shelve provide additional functionalities and tools associated with pickling.

Footnotes

• Provides additional insights and technical details on pickling process.