Skip to content

Draft: Add JSON serialization to data objects

Alessandro Maissen requested to merge json_serialization into master

This PR adds custom encoders/decoders to serialize datasets (and all their contained objects: data blocks, data objects, transformations, normalization, domains, etc) to JSON files. I also added a couple of convenience methods to serialize and deserialize into a zip archive containing the JSON file (because using indent=2 to make it pretty, makes files pretty large).

I unit tested the serialization and deserialization of all the underlying classes (data objects, transforms, normalizations and domain stuff).For the "higher level" classes (datablock and dataset), I only tested with the existing example notebooks. In particular, the toy problem beam overlap. It seems to work, but I'm not entirely clear on how the loading process works, on my first attempts I inadvertently generated 5000 extra samples every time I loaded (which I guess it's expected behavior, I was just confused by the naming of the functions) I am attaching two files, one is my test jupyter notebook that loads a dataset directly from a zip file, and the second one is the PDF output of this notebook.

I kept the changes to the existing objects to a minimum, the only thing that I had to rename was dtype to array_dtype in some data type objects.

Edited by Alessandro Maissen

Merge request reports

Loading