Apache Arrow for variable-length data¶
Please check out the recipe on using Apache Arrow for variable-length data in its own repository:
https://gitlab.jsc.fz-juelich.de/sdlaml/arrow-example
The recipe can help you to write arbitrary data in an efficient storage format. The most obvious use case is reducing access to individual files by instead storing all data in one single, efficient-to-read file.