Skip to main content

Vortex: An extensible columnar format for scans and random access

16 November 2026
  • News
  • Open-source software

This is part five in our six-part series exploring alternatives to Apache Parquet. Lance showed how to fetch individual rows quickly. BtrBlocks showed how to scan whole columns at network speed. Vortex attempts both in a single, extensible format.

In our previous articles, we learned that Lance and BtrBlocks approached Parquet’s limits from different angles. Lance was built for fast point lookups without giving up scan performance, while BtrBlocks focused more narrowly on decompressing scans at network speed. Vortex, started at SpiralDB and contributed to the Linux Foundation’s LF AI & Data in 2025, shares Lance’s goal of serving both access patterns. What sets it apart is how much of the format it leaves open. Encodings, layouts and compression strategies are all pluggable, so a writer can tune a file to its workload without stepping outside the specification.

Blue cloud outline floats centered on a plain white background presenting a minimal graphic icon style with thick continuous line Explore BtrBlocks

In part four of our six-part series, we explore how BtrBlocks uses lightweight compression techniques to accelerate decompression, improve performance and reduce costs in high-bandwidth cloud environments.

Read part four

Logical types, physical choices

Vortex separates what data is from how it is stored. A column’s logical type, such as a 64-bit integer or a UTF-8 string, says nothing about its encoding or arrangement on disk. That decision is made per chunk of data, so one column can be dictionary-encoded in one chunk and run-length encoded in the next, while readers still see a single string column.

The file itself is deliberately thin. It starts and ends with a magic number, and a small postscript at the end points to the schema, statistics and a footer that describes the full physical layout. A reader fetches the tail of the file, learns how everything else is arranged, then requests only what it needs. Because the file describes its own structure, new layouts don’t require a new specification.

Layouts instead of row groups

Parquet requires row groups: every column is cut at the same row boundaries. Vortex makes that a writer’s choice. The default layout splits a table into columns, stores pruning statistics (zone maps) for every 8K rows and chunks each column independently. A writer optimising for scans can use large contiguous chunks; one optimising for lookups can use small ones that keep read amplification low. Independent column chunking also helps wide tables, where a query on one narrow column shouldn’t pay for the I/O layout of its neighbours.

A BtrBlocks compressor, extended

Vortex’s default compressor follows the BtrBlocks approach from our last post: no general-purpose codec, just lightweight encodings applied in cascades, chosen by trial-compressing a sample. Vortex samples roughly 1% of each chunk (at least 1,024 values) and cascades up to three levels deep. It adds newer encodings: FastLanes for integer bit-packing, ALP for floating-point values and FSST for strings.

Two differences matter. First, because the encodings stay lightweight, Vortex can evaluate some filters and projections directly on compressed data, which an opaque Zstd layer prevents. Second, the compressor is pluggable: an optional “compact” strategy adds Zstd and PCodec when file size matters more than read speed.

Arrow in, Arrow out

Vortex is built to sit next to Apache Arrow, the subject of part two in our series. Arrow arrays convert to Vortex arrays without copying, and fully decompressed Vortex arrays convert back the same way. That makes integration with Arrow-based engines straightforward; the project lists Polars, DataFusion, DuckDB and Apache Spark among its integrations.

The claims and what's settled

The project claims 100x faster random access reads, 10–20x faster scans and 5x faster writes than Parquet, at similar compression ratios. These are the project’s own benchmarks. Its published TPC-H results (scale factor 10) show Vortex files 38% smaller than Parquet with Zstd and decompressing 10–25x faster, and Microsoft has reported 30% runtime reductions for Spark workloads using Vortex in Iceberg.

On stability, the file format carries a backward-compatibility guarantee: since version 0.36.0, newer Vortex releases keep read support for files written by older ones. One frequently cited feature is not yet real: embedding WebAssembly decoders in files, so older readers could decode unfamiliar encodings, is a proposal, not an implemented capability.

Whether Vortex can displace Parquet may depend less on benchmarks than on adoption across engines, which is still early. Its power may not be in any single capability, but rather their potential combinations. BtrBlocks-style cascading compression, zero-copy conversion to and from Arrow, and writer-chosen layouts all work together within a format designed to absorb new encodings as research produces them.

Next in our series: the conclusion, where we take a step back. We’ll compare all five formats, talk about what they offer to teams choosing a storage layer today and check in on what’s changed in this fast-moving space since we wrote the first article.

Adam Reeve, Open-Source Software Developer

Explore the full series

Part one

Understanding Apache Parquet: How columnar storage powers modern analytics

Hear from one of our Open-Source Software Developers on the engineering behind Apache Parquet and how its columnar architecture enables efficient, high-performance analytics at scale.

Read now
Part two

Apache Arrow: Accelerating in-memory analytics through zero-copy data sharing

Explore how Apache Arrow accelerates in-memory analytics through zero-copy data sharing, columnar memory and hardware-conscious design, and why it works so effectively alongside Apache Parquet.

Read now
Part three

Lance: Rebuilding columnar storage for random access and AI workloads

In part three of our six-part series exploring alternatives to Apache Parquet, discover how Lance is rethinking columnar storage for modern AI and machine learning workloads, enabling faster random access and more efficient data processing.

Read now
Part four

BtrBlocks: Columnar compression for the high-bandwidth cloud

In part four of our six-part series, we explore how BtrBlocks uses lightweight compression techniques to accelerate decompression, improve performance and reduce costs in high-bandwidth cloud environments.

Read now

Latest events

  • Quantitative engineering
  • Quantitative research

Women in Quant Finance 2026

22 Nov 2026 - 23 Nov 2026 1 Soho Place, London, W1D 3BG
  • Quantitative engineering
  • Quantitative research

Melbourne quant challenge 2026

30 Nov 2026 University of Melbourne, Alan Gilbert Building
  • Quantitative engineering
  • Quantitative research

Canberra quant challenge 2026

02 Dec 2026 University of Canberra

Stay up to date with G-Research

Join our talent network

Be the first to hear about new roles and opportunites to join our team.