Skip to main content

Your Data Doesn't Want to Move: Parquet Support Comes to Lucenia

· 7 min read
Wes Richardet
Wes Richardet
Lucenia Team

Every search deployment I've worked on starts the same way: with a copy. Your data lives in Parquet on object storage, the format the analytics world standardized on a decade ago, and before anyone can search it you build a pipeline. Extract, transform, bulk-index. Now you have two copies of the truth, a lag measured in hours, and a pipeline someone has to babysit forever.

Lucenia 0.12 ships a different answer. You mount the Parquet file. It becomes a searchable index in one API call, and the file never moves.

PUT /_plugins/parquet/vehicle-telemetry
{
  "path": "s3a://lakehouse/telemetry/2026-08.parquet",
  "id_column": "reading_id"
}

That's the whole integration. Lucenia validates the file, infers field mappings from the Parquet schema, and returns a queryable index: row count, typed fields, and a list of anything it skipped. Behind it is a custom Lucene codec that reads the Parquet file in place at query time. One row becomes one document, one column becomes one field, one file becomes one segment. The file stays the source of truth, and the index is a thin, read-only view over it. Local files work (absolute paths need an entry in parquet.allowed_paths), and so do Hadoop URIs: S3, HDFS, plus GCS and Azure via their connectors. Local mounts pin replicas to zero; mount from a Hadoop URI and replicas follow your cluster settings.

We call this index-free search, and it changes the math for enterprise AI retrieval.

Yes, I know what I wrote last year​

Seventeen months ago I published Apache Parquet: Good for Analytics, Not So Good for Search on this blog. I stand by every word of it. Parquet has no inverted indexes, no HNSW graphs, no spatial trees for geo and numeric data. Reassembling rows from column chunks is slow, and row-group statistics only help when your queries match assumptions somebody baked in at write time.

But that post left a question hanging: if the format is wrong for search and the data is never going to move, who blinks first? Enterprises were not going to abandon Parquet. Search engines were not going to stop needing indexes. Somebody had to go to the data. The codec-level mount is our answer: keep Lucene's query machinery, point it at the file where it sits, and be honest about which operations that can support. The limitations in that old post didn't disappear. They became this feature's design constraints.

The flywheel only spins if connecting is cheap​

AI model intelligence is compounding exponentially, yet 56% of CEOs report neither increased revenue nor lower costs from AI in the past 12 months, according to PwC's 29th Global CEO Survey of 4,454 chief executives across 95 countries. The diagnosis, as Atlan co-founder Prukalpa Sankar frames it, is that performance is intelligence plus context, and enterprises have no way to onboard agents onto their context the way they onboard employees.

Lucenia creates a flywheel by connecting the systems where data already lives. This lets AI bootstrap context through searching long term memory, and compound quality from there. Searching a new data source is the first step in that flywheel, and it has to be frictionless. The more sources you can connect, the more context your agents can access, the better their answers become, and the more sources you can justify connecting.

Notice the load-bearing assumption: connecting a source has to be nearly free. If every new data source costs a pipeline, a schema negotiation, and a second storage bill, the flywheel never reaches escape velocity. You'll connect the five sources that justify the engineering and ignore the five hundred that don't.

The context flywheel: connect a source, an agent searches it, answers get better, more sources become justified. Whether the loop turns depends on connect cost — a pipeline per source connects 5 of 500, while a one-API-call mount connects all 500

Mounting inverts that economics. The Parquet files your teams have been writing to for years, from telemetry and transactions to feature tables and one-off exports, become searchable the moment an agent needs them. Nothing has to be built or budgeted before the first query. Retrieval stops being a gate and becomes a default.

Your alpha is your data. Stop copying it into silos.​

When intelligence is a commodity, your data is the moat. That makes every proprietary copy of it a liability. Each copy is a sync problem and a governance surface. Each one buys a little more lock-in. And the search tier has been the worst offender for twenty years: re-ingest everything into a private format first, get results second.

Index-free search puts the search engine on the same side of the table as the lakehouse. Lucenia reads the open format where it sits. Your alpha stays in your format, on your object storage, under your access controls.

Agents are probabilistic writers. Databases were built for trusted, validated writers, and agents are not that. Which is why read-only mounts are a governance model, not a missing feature. An agent querying a mounted Parquet file can retrieve anything and corrupt nothing. When data earns a full index, promotion is one explicit, auditable step:

POST /_reindex
{
  "source": { "index": "satellite-telemetry" },
  "dest":   { "index": "satellite-telemetry-indexed" }
}

The trade-offs, stated plainly​

I like to think in trade-offs, and this feature is a deliberate one. Three ways to make lakehouse data searchable:

  • Traditional pipeline: full query surface and full control. Also a second copy, ingestion lag, and a pipeline to maintain before anyone runs query one.
  • Mount (PUT /_plugins/parquet/...): searchable in one API call, zero copies, zero ETL. Read-only, single shard per file. You get range queries on numerics via doc values, term and match queries, and numeric aggregations. Snappy or uncompressed files only, and per-column term dictionaries build in heap on first access, so size your nodes for big files.
  • Promote (_reindex): the full Lucenia query and aggregation surface, including keyword aggregations and sorting. You pay ingestion cost once, and only for data that proved it deserved it.

Three paths from a Parquet file on object storage to a searchable index: the traditional extract-transform-bulk-index pipeline that produces a second copy, the one-API-call mount where a Lucene codec reads the file in place, and promotion via reindex for files that earn it

The pattern that falls out is two-tier: mount everything, promote what earns it. Let actual retrieval traffic tell you which files deserve full indexing. That is usage-driven curation instead of guess-driven ETL.

Picture the agent workflow this enables. A support agent investigating a spike in device failures mounts last month's telemetry export, runs a handful of range queries and aggregations, finds the affected batch, and moves on. Total infrastructure provisioned for that investigation: zero. If the same file keeps drawing queries week after week, that is your signal to promote it and unlock the full surface. The data told you it mattered before you paid to index it.

Practical takeaway: if you're building agentic retrieval over an existing lakehouse, stop budgeting for an ingestion pipeline as step one. Mount a file this afternoon, point your agent at it, and let the queries tell you what's worth promoting.

The AI bottleneck moved from intelligence to access, from how smart the model is to whether it can reach the context trapped in your systems. Your data has been sitting in Parquet, ready, for a decade.

It was never the problem. The copies were.