A columnar file format landed in our tree today: an extension (.dfc), a fixed header, a schema section, one region per column, a reader, a writer, and a segmented writer that appends without rewriting. Nobody should write a file format on a whim, so here is the argument for this one, plus an honest account of which parts are code and which are still drawings.
Why not just write Parquet
We already read and write Parquet, and we're going to keep doing it. It's the interchange format. When data moves between our engine and Spark, DuckDB, pandas or a warehouse, Parquet is the answer.
Parquet is columnar, organized into row groups, with statistics per column chunk so a reader can skip the chunks it does not need. It's also designed to be written once. Changing one row means rewriting, because the offsets, encodings and statistics all describe the bytes as they were written.
That's the right trade for an analytical file and the wrong one for what I need here. The engine we're building holds a working table, data that is still inside the application that produced it. Rows arrive over time, some of them are wrong, and a few get deleted. I want a file that scans like an analytical file and corrects like a table.
What's in the file today
A file opens with a fixed header carrying the magic DFC1, the counts, and the offset of every section below it. That header gets written last, once the writer knows where everything actually landed. Then comes the schema, then one region per column, then a footer with a magic of its own.
Columns go down by kind, and the reason is what happens on the read side. A column of fixed-width values is one contiguous block, so a reader maps it and gets to any row by arithmetic instead of a search. Text gets an array of offsets followed by the bytes. Anything the format does not recognize falls back to JSON, because I would rather have a slow path than a type that cannot be stored at all. Nulls ride along as a bitmap next to the values, and a column with no nulls does not pay for one at all.
The string dictionary is the part I'm least sure about. The type exists and the header holds an offset for it, with the rule already written down: build one when at most 70 percent of the values are distinct and it saves at least a kilobyte. The writer doesn't build one yet.
The footer is honest in the same way. It has its own magic, the file size, and a slot for a checksum that is currently written as zero. There is no integrity checking in this format. The space is reserved for it, and that's all I can claim.
Appending without rewriting
An append doesn't touch the file you already have. It writes a new segment beside it, named after the base file (analysis.dfc, then analysis.dfcs1, analysis.dfcs2), each one a complete DFC file with the same schema. A read walks the main file and then the segments in order, and compaction merges them back into one.
That buys cheap appends, and it hands me the problem the rest of this post is about. A row now lives inside a segment, and nothing has taught the file how to forget one.
Deletes and updates, as designed
Before you can delete a row you need a stable way to point at it. With segments, the natural address is a 64-bit id split in half: the top half says which segment, the bottom half says which row inside it. It doesn't change while the row sits where it was written.
A delete is then an entry in a map keyed by that id, kept in a sidecar beside the .dfc, and a hash lookup on the read path. The row stays on disk and the reader steps over it. An update is the same trick with one more step: write the new values, mark the old row, and let compaction sort out the pair.
Two things follow, and both are worth knowing before anyone builds on this. The row count stops being one number, because rows written and rows still live are different figures, and code that confuses them reports the wrong thing. And an edit held in memory is not durable, so a change has to reach a log before it becomes visible, and be replayed when the file is opened. All of that is designed and none of it is built, which is why nothing here is crash-safe.
Where DFC and Parquet actually differ
| Question | Apache Parquet | DFC today |
|---|---|---|
| Layout | Row groups of column chunks | A region per column, a file per segment |
| Adding rows | Another file | A new segment beside the first |
| Deleting one row | Rewrite the file | A tombstone entry, designed not built |
| Updating one row | Rewrite the file | Tombstone plus append, designed not built |
| Skipping data on read | Column chunk statistics | Nothing yet, the footer is a placeholder |
| Who can read it | Many engines and languages | Our reader, nothing else |
That last row is the price, and it isn't a small one. A format only we can read traps your data, which is why the Parquet connector isn't going anywhere and why export stays a first-class operation. DFC is the working copy. Parquet is the copy you hand to someone else.
What's missing
Compression is next. The schema already carries a compression field per column, and every column today is uncompressed. After that, analytics that read columns directly instead of building rows first, and the pruning index the footer holds a place for.
None of this is available anywhere. It's a format in a repository, with tests and a working name we may not keep.