(Python) Why and how to serialize/deserialize objects.

Serialization is the process of converting in-memory data into a format, typically text or binary, that can be easily transmitted and stored for later reconstruction or use in a different location.

Photo by Ferenc Almasi on Unsplash

Explanations

Serialization is about grouping data together

When you have in-memory data, often the content isn’t stored in a contiguous memory space; you have part of the data in one place and references or pointers indicating where the other elements are located.

To store and later reconstruct the original data, you need to gather all the content in one place, replacing the references with actual values. That’s what serialization is about.

Deserialization is the process of recreating the in-memory structure.

As you can guess, deserialization involves performing the opposite action: taking the stored or transmitted data and rebuilding it into an in-memory structure that matches the structure used by the application.

Common use cases

Serialization/deserialization are often used for the following purposes:

• Transmitting data over a network or between applications or processes.
• Sending instructions to an external device. (I once read that the process was named after the serial port, but I’ve never been able to confirm it)
• Storing data for later use or as a cache.
• Performing remote procedure invocation.

Deserialization can also be used on its own to read configuration files.

What are the methods to do this in Python?

Simple (but unsafe) methods

Python has its own internal object serialization module called marshal, and there is also another module, pickle that is often mentioned.

Both are very fast. However, marshal has many limitations, as it is intended solely for creating pseudo-compiled code for Python modules (it’s the format of the .pyc files). Additionally, there is no guarantee that data serialized with marshal in one Python version will be readable in another version.

Pickle is more flexible and can be used with user-defined classes.

Both marshal and pickle work by storing (and reading) instructions on how to reconstruct the data. However, both are considered unsafe because arbitrary code can be executed.

If both the reading and writing processes are in Python, and there is no risk of the serialized data being tampered with, you can use pickle. In any other scenario, it’s best to avoid it.

More common methods

More common methods involve using language-agnostic formats such as:

• A binary format can be used when we need the data to be as small as possible. However, it requires a protocol to define how the content is structured (in what order). Google’s Protocol Buffers (protobuf) is a great tool for binary serialization; from a description file, we can generate code for serialization and deserialization in different languages — for example, serializing in Python and reading in Java.
• XML was widely used, but it is quite verbose and therefore not very efficient. The advantage of XML is that it is easy to validate using XML Schema or DTD, which can be referenced within the XML file itself.
• JSON is probably the most widely used format today. Its lightweight nature has made it more favorable than XML.
• YAML shares many similarities with JSON and includes some additional features. It is often used for configuration files, allowing users to create the file directly in the serialized format, which the application or process then deserializes.
• etc.

Using abstraction in Python for serializing and deserializing instances

The code examples are taken from an actual side project I am currently working on.

The source code handles creating/reading mapping objects (dictionaries) that are then converted to and from JSON.

Making a class serializable

Methods in serialization-compatible classes that enable exporting an instance as a mapping and constructing an instance from a mapping.

    def serialize(self, context: "Context") -> Mapping[str, Any]:
        # context is an object that give access to runtime/environment related elements
        # like providing methods to encrypt/decrypt
        return {
            "_meta": {  # _meta part include information about the class
                "module": self.__class__.__module__,
                "class": self.__class__.__name__,
            }
            # other information can be added later
        }

    @classmethod
    def deserialize(cls, context: "Context", data: Mapping[str, Any]) -> Any:
        # context is an object that give access to runtime related elements
        # the data object can contain information to pass to the constructor
        return cls()

Recreating instances

These are the two functions used to reconstruct an instance from a mapping.

from typing import Mapping, Any, Type

def deserialize_class(data: Mapping[str, Any]) -> Type:
    # helper method to find the class
    meta = data["_meta"]
    module_name = meta["module"]
    class_name = meta["class"]
    
    module = __import__(module_name, fromlist=[""])
    serialized_class = getattr(module, class_name)

    return serialized_class

def deserialize_instance(context: "Context", data: Mapping[str, Any]) -> Any:
    # helper method to re-create the instance
    _class = deserialize_class(data)

    if hasattr(_class, "deserialize"):
        # call the class method deserialize to get the instance
        return _class.deserialize(context, data)

    raise RuntimeError(f"{_class.__name__} does not have deserialize() class method")

Addendum:

I currently use mappings because they make the content easier to read. However, I might switch to using lists later to identify values by their position instead of their associated keys, which would result in a more compact format.

Sometimes, it is better to implement a factory.

When the object you want to recreate is an instance of a class you didn’t created (and therefore lacks a deserialize class method), you should consider using a factory class. The same applies when readingconfiguration files.

I developed a simple factory system for my current employer, which can be found in a public repository.

The concept is quite simple: for each class you may want to instantiate, you create a corresponding factory class.

Then you instantiate this factory (either statically or dynamically) and provide it with the parameters needed to construct the object.

After that, you simply call its build method.

from abc import ABC, abstractmethod
from typing import Generic, TypeVar, Mapping, Any

T = TypeVar("T")

class ABCFactory(ABC, Generic[T]):

    def set_parameters(self, params: Mapping[str, Any]):
        # validate and store the parameters
    
    @abstractmethod
    def build(self, context: "Context") -> T:
        # use the parameters to instantiate the object
        # use the context to access runtime related data

The disadvantage of using a factory is that you must create an additional class each time. However, the advantages are that:

• You have a lazy way of instantiating objects, as you only call the build method when needed.
• You can easily perform parameter validation.
• Using a context object in the build method, you can provide access to other elements to help you create the intended instance.

That’s all folks!

I wanted to discuss serialization and deserialization because they are essential parts of my side project, which involves distributing tasks to workers and also scheduling execution for later.