This is part four in our six-part series exploring alternatives to Apache Parquet, based on research by Abdelrhman Elrawy. Having covered Parquet’s storage design, Arrow’s in-memory format and Lance’s fast random access, we now turn to BtrBlocks, a research format targeting a different bottleneck entirely: decompression.
In part three of our six-part series, we explore how Lance is rethinking columnar storage to enable faster random access, efficient data processing and support for modern AI and machine learning workloads.
Lance, our prior subject, was built to fetch a handful of rows quickly. BtrBlocks, developed at the Technical University of Munich and presented at SIGMOD 2023, solves the opposite problem: scanning whole columns as fast as the network can deliver them. Its starting observation is simple. Cloud instances with 100 Gbit/s networking are now routine, and once the network is that fast, the CPU becomes the limiting factor.
Parquet typically compresses data twice: first with lightweight encodings such as dictionary and run-length encoding, then with a general-purpose codec such as Snappy or Zstd. That second layer made sense when disks and networks were slow and spending CPU to move fewer bytes was a good trade.
The BtrBlocks authors show why that trade no longer makes sense. Every Parquet variant they tested decompresses at more than 50 GB/s in memory, which looks comfortably faster than a 100 Gbit/s (12.5 GB/s) network link. But that figure is measured in uncompressed output. For a network scan, what matters is how fast the CPU consumes compressed input. If it can’t keep pace with compressed bytes arriving off the wire, the network sits idle while you pay for compute. Measured that way, Parquet with Zstd reached only 24.8 Gbit/s on a 100 Gbit/s instance.
BtrBlocks drops general-purpose codecs entirely. It uses a pool of fast, lightweight encodings (including run-length, dictionary, frequency, SIMD bit-packing, Fast Static Symbol Table (FSST) for strings, and Roaring bitmaps for nulls) and applies them in cascades. The output of one encoding becomes the input to the next. A string column might be dictionary-encoded, its integer codes run-length encoded, and those run lengths bit-packed.
Choosing a cascade by hand would be impractical, so BtrBlocks chooses automatically for each 64,000-value block. It samples ten runs of 64 values (1% of the block), rules out unsuitable schemes using simple statistics, trial-compresses the sample with each remaining scheme, and keeps the best. It then repeats the process on the output, up to three levels deep by default. Selection takes about 1.2% of total compression time.
BtrBlocks introduces pseudodecimal encoding, a novel compression scheme for floating-point numbers. Their binary form compresses poorly: 0.99 has no exact binary representation, producing long, repetitive bit patterns. Pseudodecimal encoding splits each double into two integers, significant digits and a power-of-ten exponent, so 3.25 becomes (325, 2). Those integers then flow into the cascade like any other integer column. Values that don’t fit (like NaN, infinity, or very high-precision numbers) are stored separately as exceptions, so the encoding stays lossless. The authors note that its strength is low-precision values like prices; it performs poorly on high-precision data such as vector embeddings.
On the Public BI Benchmark (real-world data drawn from Tableau workbooks), BtrBlocks gives up some compression: 5.28x versus 6.05x for Parquet with Zstd, though it beats Parquet with Snappy at 4.79x. In return, it decompresses 3.8x faster in memory than Parquet with Zstd. In an end-to-end scan from Amazon S3 it reached 86 Gbit/s, close to the 91 Gbit/s the test client achieved on uncompressed data, and cost roughly 1.8x less than either compressed Parquet variant.
The authors also tested whether SIMD instructions alone explain the gap. A scalar-only build ran 17% slower but was still 2.3x faster than the fastest Parquet configuration, evidence that the gains come from the design, not low-level tuning.
Open-Source Software at G-Research
We’re committed to supporting the open-source community through our contributions to widely used software, ongoing project development and partnerships with open-source foundations.
BtrBlocks is deliberately narrow. It produces compressed blocks and nothing else. Metadata, statistics, and indexes are left to a separate layer, on the reasoning that a query planner should prune data before making high-latency requests to object storage.
The authors concluded that retrofitting these ideas into Parquet would break compatibility, instead choosing to open-source the project. The Vortex project took up the challenge, adopting a BtrBlocks-inspired compressor as its default.
Next in our series: We introduce you to Vortex, turning BtrBlocks’ compression research into a full, extensible file format.
Adam Reeve, Open-Source Software Developer
Understanding Apache Parquet: How columnar storage powers modern analytics
Hear from one of our Open-Source Software Developers on the engineering behind Apache Parquet and how its columnar architecture enables efficient, high-performance analytics at scale.
Apache Arrow: Accelerating in-memory analytics through zero-copy data sharing
Explore how Apache Arrow accelerates in-memory analytics through zero-copy data sharing, columnar memory and hardware-conscious design, and why it works so effectively alongside Apache Parquet.
Lance: Rebuilding columnar storage for random access and AI workloads
In part three of our six-part series exploring alternatives to Apache Parquet, discover how Lance is rethinking columnar storage for modern AI and machine learning workloads, enabling faster random access and more efficient data processing.
