The root namespace of this project became Datafication.Core today. It's a small change to a small codebase: connectors for JSON and text files, filters (attribute, regex, type, head, tail, shape, flatten), a record type made of named fields, and tests. It targets .NET 6, and the core project has no third-party package references at all.
A rename is a good moment to write down what the thing is for.
The five hops
If you've ever watched data leave a C# application so somebody could answer a question about it, you know this sequence. I've seen it on every .NET project I've worked on that had data worth asking about.
The application is written in C#. It knows the types, the constraints, and what a valid order looks like. Then somebody asks which categories are growing, and the data leaves. It gets exported as text, reloaded somewhere the types are guessed again, run through a copy of rules that already exist a few directories away, pasted into a document, and, if it matters, written a third time back inside the application.
Nobody picks this. It's what happens when the work in the middle has no home on the platform. I argued the general version of that when we started the company.
LINQ covers half of it
The usual answer in .NET is LINQ, and for half the work it is the right answer. Where, Select, OrderBy, GroupBy, Join and the aggregate operators cover what application code needs against a collection it already has a type for.
Analytical work asks for the other half. Here is the split as I see it.
| The operation | LINQ to Objects | An analytical table |
|---|---|---|
| Keep matching rows | Where(x => x.Region == "NE"), given a row type |
The same test, named by column |
| Take some columns | Select(x => new { x.A, x.B }), a type per shape |
Columns named at run time |
| Total per group | GroupBy, then an aggregate in a Select |
One call, keys and totals |
| Moving average of seven rows | By hand | A window over an ordered column |
| Months across the top | By hand | A pivot |
| Fill the gaps in a column | By hand | A policy, per column |
| Join two files of different shapes | Join, once both have types |
Named keys and a join mode |
| Describe what just loaded | No equivalent | Names, types, null counts |
LINQ can express every row in that table, given a type and enough lines. What I care about is what you get without writing them. Here is a seven-day moving average, done the obvious way:
var moving = readings
.Select((r, i) => new
{
r.Date,
Average = readings
.Skip(Math.Max(0, i - 6))
.Take(Math.Min(i + 1, 7))
.Average(x => x.Value)
})
.ToList();
It's correct, it re-reads the window for every row, and it's the third time this year somebody has written it. Multiply that by pivots, percentiles and fills, and leaving the platform starts to look sensible.
The other options are real, and I want to be fair to them. Microsoft.Data.Analysis is Microsoft's DataFrame for .NET and has not reached a 1.0 version. Deedle is an F#-first data frame library. DuckDB reaches .NET through an ADO.NET provider, so the query language is SQL. System.Data.DataTable predates LINQ and stores rows. pandas and polars need a Python runtime, which puts them on the far side of the picture above.
None of that is a complaint. My point is about shape: no widely used .NET type holds a table by column, carries its schema, and offers the analytical operations as methods.
What we are committing to
Four decisions, written down now so you can judge them later.
Columns, not records. Today this codebase holds a list of records, each holding a list of named fields. That's the natural shape for a connector and the wrong shape for analysis, which reads one column across every row. A mean over a million rows should touch one contiguous list, not walk a million objects pulling one field out of each.
Every operation returns a new table. The filter pipeline already works this way: a filter takes records and returns records, and the input is untouched. Keep that rule everywhere and a pipeline reads top to bottom, with no method quietly changing what another step is holding.
A column carries more than a type. A field already has a name and a label. A column needs a description, a display format, whether it may be null, whether it is unique. That's the metadata that lets a table describe itself.
No dependencies in the core. The core project references nothing. The JSON connector references Newtonsoft.Json, the text-file connector references nothing, and whatever a format needs is that format's problem. The engine everything else sits on should drop into your application without an argument about transitive versions.
What we are not building
There won't be a LINQ provider, and IQueryable is not going on the table type. An IQueryable surface promises the whole of LINQ, so every operator that cannot be translated turns into a run-time surprise instead of a compile error. Pivot, window frames and fill policies are not LINQ operators to begin with.
The bigger reason is that the columns are not known at compile time. A CSV with forty columns shows up as forty names in a header row, and an expression tree over T needs T first. So the operations will be methods named after what they do, taking column names as arguments: a filter that takes a column, a value and a comparison, a grouping that takes a key column and an aggregate. Computed columns will need something more expressive, and I don't know its shape yet.
The cost we are accepting
A table that discovers its columns at run time holds values whose types it didn't know when it was compiled. That means boxing, an allocation and a pointer chase per value. We're choosing it on purpose. A general table that's awkward to use has failed at its job whether or not it allocates.
It's also why I expect to build a second thing. When data stops fitting in memory, the answer isn't a faster boxed list, it's typed storage on disk with the same operations over it.
Where this goes next
A CSV connector, because CSV is how data actually arrives. Then grouping and calculated fields in the pipeline, and serialization so a shaped result can be written back out. After that, the change that matters: the record list becomes a set of columns, and the operations move onto the table.
There's nothing to install yet. When there is, I'll write about what it does and what it does not.