The Datafication SDK is on NuGet

The Datafication SDK is public. Eight packages went up on nuget.org today, all at version 1.0.1, all built for .NET 8.

I've written the words "it's on NuGet" about other people's projects plenty of times. Writing them about this one feels different. All of it started as code we needed for our own work and kept to ourselves, and as of this morning it's a dotnet add package away for anybody.

Here's what you get. The SDK does data work inside your own process: no daemon, no sidecar, no service to sign up for. DataBlock is a column-oriented in-memory engine. VelocityDataBlock is a disk-backed columnar store over a file format we call DFC. Five connectors fill a DataBlock from a CSV file, a Parquet file, a database, an S3 bucket, or a web page. One package serves registered DataBlocks over HTTP. Install the parts you need and leave the rest.

What shipped

Datafication.Core has no third-party dependencies at all, and that was on purpose. The package everything else depends on shouldn't drag a dependency graph into your application. Each connector adds the one library that parses its format, and nothing more.

dotnet add package Datafication.Core
dotnet add package Datafication.CsvConnector

What the code looks like

A load, four operations, a sort:

using Datafication.Core.Data;
using Datafication.Extensions.Connectors.CsvConnector;

var sales = await DataBlock.Connector.LoadCsvAsync("sales.csv");

var result = sales
    .Where("region", "North America", ComparisonOperator.Equals)
    .Compute("profit_margin", "profit / revenue")
    .Where("profit_margin", 0.25, ComparisonOperator.GreaterThan)
    .GroupByAggregate("product_category", "revenue", AggregationType.Sum, "total_revenue")
    .Sort(SortDirection.Descending, "total_revenue");

Two things there are worth pointing out. Compute takes an expression string, and Core compiles it: arithmetic, comparison, boolean logic, CASE WHEN, and about forty built-in functions for math, dates and strings. And notice the load goes into a variable before the chain starts. await binds to the whole expression, so chaining straight off await DataBlock.Connector.LoadCsvAsync(...) doesn't compile. That one still catches me.

Rows come back through a cursor:

using var cursor = result.GetRowCursor("product_category", "total_revenue");
while (cursor.MoveNext())
{
    Console.WriteLine($"{cursor["product_category"]}: {cursor["total_revenue"]}");
}

Why DataBlock is the center of it

A DataBlock holds one list of values per column instead of a list of row objects. That's the layout analytical work wants, because a filter, a sum or a group touches whole columns.

Every operation returns a new block, so the block you called it on is unchanged. You can branch a pipeline without making defensive copies.

Columns carry more than a name and a type. DataColumn has a Label, a Description, a Format, and IsNullable, IsUnique and IsIndexed flags, so the metadata that usually ends up in a side dictionary lives on the column instead. Info() prints column, label, type, non-null count and null count.

I made the case for building this in C# in a DataFrame-shaped hole in .NET. Today you can install it.

Velocity is a separate package on purpose

When the data won't sit in memory, VelocityDataBlock writes DFC, a columnar file format. If you want the reasoning behind the format, it's in Designing DFC.

It reads like DataBlock, with one difference that matters: nothing runs until Execute().

using Datafication.Storage.Velocity;

using var store = await VelocityDataBlock.SaveAsync("sales.dfc", sales);

var top = store
    .Where("region", "North America", ComparisonOperator.Equals)
    .GroupByAggregate("product_category", "revenue", AggregationType.Sum)
    .Sort(SortDirection.Descending, "sum_revenue")
    .Execute();

Each call adds to a query plan. Execute() runs it and hands back an ordinary in-memory DataBlock. The sum_ prefix in that sort comes from GroupByAggregate, which names output columns sum_, avg_, count_, min_, max_, std_ or var_ plus the source column name.

Velocity is optional. If the data fits in memory, Datafication.Core is the whole story.

The license, briefly

I'd rather you know the terms now than run into them later. The SDK ships under the Datafication SDK License Agreement, a proprietary license with a free tier. Free use is permitted while both conditions hold: fewer than five developers at your organization build software with the SDK, and the organization's gross annual revenue does not exceed $500,000 USD. Open-source projects are exempt from the developer count provided annual funding stays under $500,000 and the project does not compete with or replicate the SDK's core functionality. Everything else needs a commercial license, and support@datafication.co is where to ask.

The license is a contract, not a key. No shipped package gates a feature on a license file.

What you need, and where to go next

.NET 8.0 or later, on Windows, macOS, or Linux. Every package targets net8.0.

Each package has its own README and a samples/ folder in the overview repository, covering filtering, grouping, merges, window functions and expressions. The API reference is at datafication.co/help/api/, and the packages start at Datafication.Core.

Pull it down, run something real through it, and tell me where it gets in your way. On day one that's the feedback I most want.