Datafication SDK

The problem it solves

Python has pandas and polars. .NET teams had LINQ, which is wonderful until you need a moving average, a pivot, a join across two files, or a pipeline with more than three steps. Analytical work ended up in a sidecar language, a spreadsheet, or a database nobody wanted to own, and the application that produced the data never got to use it.

The Datafication SDK is the data layer .NET never shipped: a columnar engine with the operations analysts expect, written to feel like .NET and to compile.

What the SDK is

At the center is the DataBlock, a column-oriented, in-memory analytical table. Every operation returns a new DataBlock, so a pipeline reads top to bottom and never mutates its input. Schemas carry labels, descriptions, formats, and constraints alongside the data.

using Datafication.Core.Data;
using Datafication.Extensions.Connectors.CsvConnector;

var sales = await DataBlock.Connector.LoadCsvAsync("sales.csv");

var result = sales
    .Where("region", "North America", ComparisonOperator.Equals)
    .Compute("profit_margin", "profit / revenue")
    .Where("profit_margin", 0.25, ComparisonOperator.GreaterThan)
    .GroupByAggregate("product_category", "revenue", AggregationType.Sum, "total_revenue")
    .Sort(SortDirection.Descending, "total_revenue");

Filtering with Where, WhereIn, and Filter; projection with Select; Sort; GroupBy and GroupByAggregate; Merge with inner, left, right, and full outer joins; aggregations from Sum to Percentile; computed columns through an expression language with arithmetic, comparison, logic, CASE WHEN, and about forty built-in functions; Pivot, Melt, and Transpose; and window functions.

When the data outgrows memory

VelocityDataBlock keeps the same query API and moves the data to disk in the DFC columnar format: typed, dictionary-encoded columns, SIMD-accelerated filters, LZ4 and Deflate compression, a write-ahead log, and compaction. The same Where, Compute, GroupByAggregate, and Sort calls build a query plan that runs when you call Execute().

Measured on a 1,000,000-row, 15-column dataset with batched cursor iteration, .NET 8 on an iMac, using the benchmark project in the samples. Your numbers will differ with hardware, column types, and query shape.

Compared with

Where DataBlock sits next to the ways .NET developers already work with tables.

Connectors

Each connector is its own package, so an application takes only the formats it reads.

The rest of the family

Licensing

The SDK ships under the Datafication SDK License Agreement, which is included in every package. It is free to use while fewer than five developers in your organization build with it and the organization's revenue is under $500,000 a year, and open-source projects are exempt from the developer limit. Above that, a commercial license applies, which adds priority email support and advance notice of breaking changes. support@datafication.co answers questions about which one applies to you.

Get the SDK