Understanding Apache Parquet: How columnar storage powers modern analytics

Parquet’s efficiency stems from its three-tier hierarchical structure. At the top level, row groups provide the unit of parallelisation – different threads or compute nodes can process separate row groups independently. Within each row group, column chunks store all values for a single column contiguously, allowing your query engine to read only the appropriate columns. Finally, column chunks are split into pages, which represent the atomic unit where compression and encoding occur.

This isn’t just clever organisation – it’s a deliberate balance of competing demands. Large blocks optimise sequential reads, while smaller pages enable precise access patterns. The structure allows query engines to make intelligent decisions about what data to read, when to read it, and how to process it in parallel.

Parquet’s sophisticated encoding schemes adapt to your data’s characteristics. Dictionary encoding transforms repetitive strings into compact integers, which work especially well for categorical data like trading venues or instrument types. Run-length encoding collapses sequences of identical values, while bit-packing ensures integers only consume the bits they actually need.

The Parquet V2 improvements, particularly DELTA_LENGTH_BYTE_ARRAY encoding, demonstrate continued innovation. By grouping string lengths separately from string data and applying delta encoding, it achieves better compression for the short strings common in financial datasets.

Post-encoding, compression algorithms like Snappy (fast but modest compression) or Zstandard (balanced performance) further reduce storage footprint. The choice matters: in I/O-bound systems, aggressive compression with ZSTD can actually improve query performance despite higher CPU usage.

Perhaps Parquet’s most sophisticated feature is its ability to efficiently handle complex, nested data structures commonly found in serialisation formats like JSON or Avro. Parquet achieves this with the “record shredding and assembly” algorithm pioneered by Google’s Dremel. The nested schemas are “shredded” into flat columns while preserving the ability to reconstruct the original nested structure.

This capability is especially valuable when working with semi-structured data from APIs, event streams or document stores. This allows us to maintain a single storage format across diverse data types.

Parquet’s wide adoption in data lake architectures stems from this careful engineering: hierarchical organisation for parallelism, sophisticated encoding for compression, rich metadata for optimisation and elegant handling of complex schemas. Understanding these capabilities provides a base of comparison as we evaluate alternatives throughout the series.

In our next post, we’ll explore Apache Arrow, not as an alternative to Parquet, but as a complement for in-memory processing. Future posts will explore emerging formats like Lance, Nimble, Vortex and BtrBlocks, each attempting to build on Parquet’s foundation in different dimensions.

Stay tuned for Part two: Apache Arrow: Accelerating in-memory analytics through zero-copy data sharing