Pyson

Pyson converts between dicts/lists and python3 objects.

The basic version determines the types for (de)serialization from the __init__ function in the involved classes. Polymorphism is supported, so derived classes can (de)serialized from a superclass.

You can control the (de)serialization mapping using jackson-styled annotations.

Installation

For pyson you need python 3.8 or 3.9.

pip install https://tracinsy.ewi.tudelft.nl/pubtrac/Utilities/export/192/pyson/dist/pyson-1.0.0.tar.gz

or from your setup.py

    install_requires=[ "pyson@pip install https://tracinsy.ewi.tudelft.nl/pubtrac/Utilities/export/192/pyson/dist/pyson-1.0.0.tar.gz"],

NOTICE you may want to use the latest version. Check pyson/dist for the latest.

WARNING: there exists another other python library named "pyson", which is an entirely different project.

Basic Mechanism

The basic mechanism for pyson is to look at the __init__ function of the class under serialization. The arguments in the __init__ function are matched to the json fields. The arguments in the __init__ must be fully typed, and these types are used to determine how to interpret the json content.

You can use typing.Any to indicate you expect any PRIMITIVE type (int, str, dict, etc). For security reasons, pyson always requires explicit class for polymorphism and never deserializes just any class, only primitives can be handled this way.

Deserialization

For Deserialization of a json object, it goes like this

• if the target class has @JsonSubtypes,
  • check which the subclasses are and what their ID is.
  • Determine the actual class contained in the json
  • "unwrap" the json, so that we have the remaining json to deserialize the actual class
• If it's not a JsonSubtypes, then the actual class is the requested target class
• now that the actual class is known,
  • the parameter names and param-classes from the __init__ function of the actual class are taken
  • for each of the parameters, recursively deserialize the json value for that parameter, using the param-class as targetclass.
  • call the constructor of the target class, using the parsed json for each parameter. Missing parameters is allowed if the constructor has default values for that parameter.

Serialization

Serialization of an object is much more straightforward.

• Create a json dict, with keys the arguments of the __init__ function of the object and the value the serialized value returned by the getter (also considering @JsonGetter)
• If the object is an instance of a class with @JsonSubtypes, add/wrap the json with class info according to the @JsonTypeInfo

Primitive types

Primitive types are types that are built into Python and do not have the __init__ function as we expect, for instance float, str. Therefore these need to be intercepted and handled through special constructors and serializers.

The following types are recognised and handled separately:

NoneType, int, float, bool, str, complex, range, bytes, bytearray, datetime, URI, UUID, Decimal, dict, list, Exceptions (see below)

The dict and list are treated even more special as we also support the List[TYPE] and Dict[KEYTYPE, VALUETYPE] types.

WARNING: Using Decimal causes rounding errors. This is because pyson takes python's json representation format which contains floats for numbers. And floats can not represent all decimals accurately.

You can use Union[X, NoneType] where NoneType=type(None) or Optional[X]if the object can also be None. So for exmple you can use reservationBid:Union[Bid,type(None)] = None. Do not use the mypy.NoneType which is an entirely different class. We do not support Union[X,Y] where both X and Y are not NoneType.

Exceptions

Exceptions are handled as follows: Serialization of an exception gives {"message":"...", "cause": EX, "stackTrace":[]} where EX is either None or another serialized exception. This format tries to maximize compatibility with the jackson way of serializing exceptions. The "stackTrace" field is always empty because python does not include the stacktrace with exceptions; it's there only for compatibility of the format with java. Note that the type of the exception is not included (as in jackson). Deserialization takes a dict with at least a "message" field. The constructor of the provided class is called with this message, so make sure that your exception constructor takes the message as the constructor argument. If the dict contains a cause, the cause is also parsed recursively, as general Exception, and attribute __cause__ of the exception is filled with the result.

Annotations

The following annotations are available

@JsonGetter

Added to a function definition in a class. Takes "value" as argument, containing the name of a json field (which is also used as argument in the constructor).

Indicates that that function is to be used to get the value of the given json field.

@JsonSubTypes

Added to a class definition. Takes a list of strings as argument. Each string is the full class path to another class.

Indicates that that other class is a sub-class of the annotated class. The other class can be used for deserialization as well.

If this is used, the @JsonTypeInfo must also be provided.

@JsonValue

indicates that the value of annotated accessor (either field or "getter" method [a method with non-void return type, no args]) is to be used as the single value to serialize for the instance, instead of the usual method of collecting properties of value.

@JsonTypeInfo

Added to a class definition. Contains "Id" and "As" values.

Indicates that the class name/id should be included in the json code. This is especially useful in combination with @JsonSubTypes.

The "Id" value can have the value NONE, NAME, CLASS. We recommend to use NAME,

Id Value	meaning
NONE	Do not include class id at all. Do not use this with @JsonTypeInfo. Only included because it was availale in Jackson
NAME	Use the name of the class to refer to the class. All classes referred must have different name (not two the same names with different classpath).
CLASS	Use the full.class.path to refer to the class.

NAME should be used, the others are not properly implemented. We recommend this method anyway, because it is shorter and gives more readable json, and gives you flexibility to move around the actual classes if needed without breaking compabibility with existing json files

The "As" annotation indicates *how* to include the class name is included in/extracted from the json.

As value	meaning
PROPERTY	The json dict is extended with a type parameter containing the class Id
WRAPPER_OBJECT	A dict is created with the class Id as key, and as value the actual class contents
WRAPPER+ARRAY	An array is used for storing the contents of the class

At this moment WRAPPER_OBJECT should be used; the others are not properly implemented.

@JsonDeserialize

This annotation allows a custom hand-coded deserializer to be used. The argument is a full.class.path of a class implementing Deserializer. For example, @JsonDeserialize("geniusweb.issuevalue.ValueDeserializer.ValueDeserializer")

The Deserializer requires one implemented method: deserialize(self, data:object, clas: object). It takes the json object and the expected class, and then returns the deserialized object.

Inheritance of annotations

The usual inheritance mechanism of python applies also to the annotatinos.

Examples

See pyson/test/ObjectMapperTest.py​ for many examples.

A simple example, deserializng a dict with objects

from pyson.ObjectMapper import ObjectMapper
from pyson.JsonTypeInfo import JsonTypeInfo
from pyson.JsonTypeInfo import Id,As
from typing import Dict

@JsonTypeInfo(use=Id.NAME, include=As.WRAPPER_OBJECT)
class Simple:
    def __init__(self, a:int):
        self._a=a
    def geta(self)->int:
        return self._a
    def __eq__(self, other):
        return isinstance(other, self.__class__) and \
            self._a==other._a
    def __str__(self):
        return self._name+","+str(self._a)

pyson=ObjectMapper()
objson = { 'a':{"Simple":{'a':1}},'c':{"Simple":{'a':3}}}
obj=pyson.parse(objson, Dict[str,Simple])
obj['a'].geta()

A complex example showing many things at once

from pyson.ObjectMapper import ObjectMapper
from pyson.JsonSubTypes import JsonSubTypes
from pyson.JsonTypeInfo import JsonTypeInfo
from pyson.JsonTypeInfo import Id,As
from typing import Dict,List,Set
import json

class Props:
    '''
    compound class with properties, used for testing 
    '''
    def __init__(self, age:int, name:str):
        if age<0:
            raise ValueError("age must be >0, got "+str(age))
        self._age=age
        self._name=name;
    def __str__(self):
        return self._name+","+str(self._age)
    def getage(self):
        return self._age
    def getname(self):
        return self._name
    def __eq__(self, other):
        return isinstance(other, self.__class__) and \
            self._name==other._name and self._age==other._age


@JsonSubTypes(["__main__.Bear"])
@JsonTypeInfo(use=Id.NAME, include=As.WRAPPER_OBJECT)
class Animal:
    pass
    
    
class Bear(Animal):
    def __init__(self, props:Props):
        self._props=props
            
    def __str__(self):
        return "Bear["+str(self._props)+"]"
        
    def getprops(self):
        return self._props
    def __eq__(self, other):
        return isinstance(other, self.__class__) and \
            self._props==other._props

pyson=ObjectMapper()


obj=Bear(Props(1,'bruno'))
res=pyson.toJson(obj)
print("result:"+str(res))
bson={'Bear': {'props': {'age': 1, 'name': 'bruno'}}}
res=pyson.parse(bson, Animal)
print("Deserialized an Animal! -->"+str(res))

NOTICE: our code allows you to use objects as keys, as python does allow this. However json requires strings as keys.

And an example using @JsonGetter

from pyson.ObjectMapper import ObjectMapper
from pyson.JsonGetter import JsonGetter
class Getter:
    def __init__(self, a:int):
        self._a=a
    @JsonGetter("a")
    def getValue(self):
        return self._a

getter=Getter(17)
pyson=ObjectMapper()
pyson.toJson(getter)

Add/Extend annotations at runtime

You can also add/extend existing annotations at runtime. You just need to update the class and function attributes. For now, check the source code for details.

Download sources

svn co https://tracinsy.ewi.tudelft.nl/pub/svn/Utilities/pyson

Why not @datatype

The @datatype annotation from python offers serialization through the asdict method. However it does not support polymorphism, it does not include the information needed to properly support polymorphism, and it does not offer the reverse method (e.g. from-dict).

From from-dict functionality, you need to use a third party library, like ​dataclasses_fromdict, [​​https://github.com/konradhalas/dacite dacite] or ​mashumaro.

But even then polymorphic classes can not be deserialized, because the serialized form just does not contain the information required to do this.