#How we started: fast JSON parsing

Pydantic started as a library to validate and parse JSON data efficiently. When we built Logfire, we initially stored all attributes as JSON blobs in a single column and relied on Jiter (one of the fastest JSON parsers out there in any language) to rip through the JSON data as quickly as possible. This is the layout detailed above (attributes: Utf8View (JSON)).

This approach was simple but made a fundamental mistake: we were optimizing to do work faster, but we couldn't really avoid doing the work. Compression is also suboptimal since a single varying attribute makes Parquet level dictionary compression ineffective and even zstd struggles with highly variable data, all the JSON syntax overhead, etc.

The biggest problem really ends up being extra IO: since most of our data is stored on object storage, extra IO from downloading large JSON blobs, and especially the latency of starting these downloads, ends up dominating query runtimes much more so than any CPU work.

#Shredding columns to save IO and compute

To make queries fast, we need to avoid unnecessary work. For normal columns DataFusion / Parquet avoid unnecessary work by using columnar storage (allowing reading only the columns you need) and zone maps / min-max statistics (allowing skipping entire row groups or pages that don't match your filter predicates).

So when you write a query like:

SELECT count(*) FROM records WHERE duration < 0;

The system can efficiently read just the duration column and skip row groups where the max duration is < 0 based on the stored statistics. In this case this is a valid query, duration is a f64 so in theory it could have values <0. In practice though duration is always >= 0 so this query will be very fast: based on statistics we know no rows match and we can return 0 immediately.

So how can we enable this for semi-structured data like attributes stored as JSON?

Our initial solution was static shredding: we manually picked a small set of commonly queried attributes (like http.status_code, http.method, etc.) and extracted these into their own strongly typed columns during ingestion.

So when we receive:

Assuming our hardcoded shredding configuration extracts http.response.status_code, url.full, and db.query.text, we would store:

Now we can run queries like:

SELECT count(*) FROM records WHERE _lf_attributes_http.response.status_code = 500;

We can use statistics to prune files / row groups / pages that don't match, and then scan just the _lf_attributes_http.response.status_code column which is strongly typed (int64) and compresses well.

Of course you as a user don't have to use the _lf_attributes_http.response.status_code column directly, you can write:

SELECT count(*) FROM records WHERE attributes->'http.response.status_code' = 500;

And we rewrite this under the hood to use the shredded column.

I thought it worth mentioning the _lf_attributes prefix: one of the challenges with the current static shredding approach is that these columns are user-visible, e.g. if you run select * from records limit 10 you'll see these columns. Since these are an internal implementation detail, we'd rather not leak them to you. We want to keep the logical layout of attributes as a single JSON column from your perspective, while optimizing the physical layout under the hood.

#The Bad: Limited Flexibility, High Cost for Unshredded Attributes and Overheads

There are several issues with this scheme:

• Limited Flexibility: Only a small, hardcoded set of attributes are shredded. If users want to query other attributes (e.g., user_id, session_id, region), they pay the full cost of JSON storage and parsing. Queries on these attributes can get slow on projects with 100s of GBs of data. We can't easily add new shredded attributes without code changes and deployments. This is particularly painful with indexing; we only have indexes on the hardcoded set of shredded attributes.
• High Cost for Unshredded Attributes: Queries that filter on unshredded attributes (e.g., attributes->>'user_id' = 'user123') require scanning the entire JSON column, leading to high I/O, poor compression, and expensive JSON parsing. This results in slow queries and high resource usage. Since data is stored in chunks, even a single large unshredded attribute can bloat the entire column, making all queries expensive even if they are not touching that attribute.
• User-visible columns: The shredded columns are visible to users, leading to confusion and clutter in query results. Users may not understand the purpose of these internal columns, and they can complicate query writing and result interpretation.
• Overheads: As you can see, there are several columns that may be all nulls (e.g. if the application doesn't do DB queries, the _lf_attributes_db.query.text column is always null). Although Parquet / Arrow / DataFusion are quite efficient at handling nulls, there is still some overhead in storage and processing for these unused columns.

Every query touching attributes that isn't filtering on the small hardcoded set of shredded attributes pays these costs.

For example, consider the query:

SELECT * FROM events WHERE attributes->>'my_llm_response' like '%important%';

This query needs to scan the entire attributes JSON column since my_llm_response is not shredded. Even though my_llm_response is a large attribute it may be a rare attribute that only appears in a small fraction of rows. Think about it this way: you probably have many more spans that don't involve LLM calls than ones that do, even in an agentic application.

┌─────────────────────────────────────────┐
│  Cost 1: Download Entire _lf_attributes │
│  (could be GBs of data)                 │
└─────────────────────────────────────────┘
                    ↓
┌──────────────────────────────────────────┐
│  Cost 2: Poor Compression                │
│  Variable structure → zstd 10% efficiency│
└──────────────────────────────────────────┘
                    ↓
┌─────────────────────────────────────────┐
│  Cost 3: JSON Parsing                   │
│  Jiter does lazy parsing, worst case    │
│  must parse entire row.                 │
└─────────────────────────────────────────┘
                    ↓
┌─────────────────────────────────────────┐
│  Cost 4: In-Memory Operations           │
│  Create copies of extracted values      │
│  Do comparisons in memory               │
└─────────────────────────────────────────┘

The Result: this query has to parse and scan potentially GBs of JSON data, much of which is irrelevant to the query, just to get the few rows that contain my_llm_response.

Or consider the query:

SELECT * FROM events WHERE attributes->>'user_id' = 'user123';

This query now has the opposite problem: even if 1/3 of rows contain a user_id attribute, we have to scan the entire JSON column to find them, which includes those large my_llm_response attributes that bloat the column size.