SDK 1.0.10: faster Velocity queries, and one API removed

If you keep a large .dfc file around and run filters, sorts and group-bys over it all day, this release is for you. Version 1.0.10 is on NuGet, and nearly everything in it lives in one package, Datafication.Storage.Velocity, in the gap between Where and Execute.

That gap is where the work happens because a Velocity query is deferred. Select, Where, Sort, GroupByAggregate and the rest each add an operation to a plan, and nothing touches the file until Execute() runs it.

using var sales = new VelocityDataBlock("sales.dfc");

DataBlock top = sales
    .Where("Region", "West")
    .Where("Amount", 1000.0, ComparisonOperator.GreaterThan)
    .Sort(SortDirection.Descending, "Amount")
    .Head(50)
    .Execute();

By the time that last line runs, the whole query is known. That is what lets us pick a better route through the file instead of doing the obvious thing row by row.

What changed

Filters stopped asking one row at a time

The old filter path walked the rows and asked each one whether it qualified. Now it asks each column once and gets back one bit per row, and the conditions are combined as bit patterns before a single row is read out. On a numeric column that means several values are compared per instruction rather than one.

Not every filter gets that treatment, and it is worth knowing which. String equality does, because it compares dictionary IDs. Contains, StartsWith and EndsWith cannot be answered from an ID, so a query carrying one of them falls back to the older path. Mix the two and you get the best of it anyway: the numeric conditions run through the bitmask, and the pattern is tested only on the rows that survive.

Big files now use more than one core

The unit of parallel work used to be the segment, and a file written in one pass is a single segment. Forty million rows, one thread. Now a large segment splits into slices that scan independently, so the cores you have actually get used on the file you are waiting on.

One honest limit. This helps the numeric conditions. A string equality that follows an earlier condition still works segment by segment, not slice by slice.

Sorting when you only want the top of the list

Sort().Head(50) does not need a sorted file. It needs the best fifty rows, and a small bounded heap collects those in one pass. Double columns get a second saving on top of it: every DFC row group records its own minimum and maximum in the file index, so groups are visited most promising first, and once the heap is full a group that cannot beat it is never read.

Grouping got the same kind of attention. It now looks at the key column before it starts and picks how to do the work. Sorted keys need no hash table at all, which covers most time-series and log data, and a dictionary-encoded string column with few distinct values needs no hashing either, because a small integer is already an array index.

What I removed

VelocityResult and AsResult() are gone, along with VelocityDataRow, RowIndexStorage, ComputedColumnStorage and the Count(), Any() and FirstOrDefault() extension methods. If a chain of yours ends in AsResult(), end it in Execute() instead and you are done.

They were a second way to run a query, which meant a second copy of every optimization above. Everything worth doing had to be done twice. The commit that took them out added two lines and deleted 2,845.

One other behavior change to note: Variance() and StandardDeviation() now return NaN when there are fewer than two values.

Getting it

dotnet add package Datafication.Storage.Velocity

The full list of changes is in the changelog, and the file format underneath all of this is the subject of Designing DFC. If you have a query that still feels slower than it should, I would like to see its shape.