A REST layer that speaks DataBlock

A DataBlock lives in one process. Sooner or later something outside that process wants the data: a dashboard, a notebook on somebody's laptop, a service written in another language.

You know how this usually goes. Revenue by region gets an endpoint. Then somebody needs a date range, so the endpoint grows a parameter, then a category parameter, and six months later there's a file of near-identical actions that all end in a group-by. The other way out is to serve raw rows and let every caller re-implement the analytics on their own side.

Datafication.Server.Core, one of the eight packages published last week, takes a third position. You register a DataBlock under an id, and the package puts a fixed set of routes over it. One of those routes accepts an analytical operation as JSON and runs it in process, before anything is serialized.

Two calls, then a name

Here's the whole of the setup.

using Datafication.Core.Data;
using Datafication.Extensions.Connectors.CsvConnector;
using Datafication.Server.Core.Extensions;
using Datafication.Server.Core.Registry;

var builder = WebApplication.CreateBuilder(args);


builder.Services.AddDataBlockRegistry();
builder.Services.AddDataficationServer(options =>
{
    options.RoutePrefix = "api/data";
    options.MaxRowsPerRequest = 5000;
});

// Your own authentication scheme goes here.

var app = builder.Build();

var registry = app.Services.GetRequiredService<IDataBlockRegistry>();
registry.RegisterDataBlock("orders", await DataBlock.Connector.LoadCsvAsync("orders.csv"));

app.UseAuthentication();
app.UseAuthorization();
app.MapControllers();
app.Run();

AddDataBlockRegistry puts a registry in the container. AddDataficationServer adds the controller, a connector registry, a sink registry, and two authorization policies. RegisterDataBlock gives a block a name, and that name is all a caller ever needs to know.

Anything that produces a DataBlock can be registered: a CSV load, a database query, a Parquet file, or a block your own code built row by row.

The routes, and the prefix

You don't write the prefix on any route. It comes from the RoutePrefix option, so the server above serves /api/data/datablocks, and changing that one line moves all twenty-three endpoints together.

The rest are an Info() summary, a metadata update, a sink listing, a registry search, and registry export, import, cache clearing, maintenance and analytics.

One request, one whole operation

The query route is the one I care most about. It takes a JSON body, and one request can filter, compute a column, group, sort and page. Here's a real one.

curl -X POST http://localhost:5199/api/data/datablocks/orders/query \
  -H "Content-Type: application/json" \
  -d '{
    "where": [ { "column": "Quantity", "operator": "gt", "value": 2 } ],
    "compute": [ { "resultColumn": "Revenue", "expression": "Quantity * UnitPrice" } ],
    "groupBy": { "column": "Region", "aggregations": { "Revenue": "sum", "OrderId": "count" } },
    "sort": { "column": "sum_Revenue", "direction": "desc" },
    "take": 5
  }'

And here's what comes back.

{
  "data": [
    { "Region": "North", "sum_Revenue": 882.85, "count_OrderId": 5 },
    { "Region": "West", "sum_Revenue": 159.92, "count_OrderId": 1 },
    { "Region": "East", "sum_Revenue": 119.94, "count_OrderId": 1 },
    { "Region": "South", "sum_Revenue": 118.21, "count_OrderId": 2 }
  ],
  "totalRows": 4,
  "returnedRows": 4,
  "columns": ["Region", "sum_Revenue", "count_OrderId"],
  "metadata": {
    "executionTimeMs": 18,
    "operationsApplied": [
      "Where: 1 filter(s)",
      "Compute: 1 column(s)",
      "GroupBy: Region",
      "Sort: sum_Revenue desc"
    ],
    "executedAt": "2026-01-14T09:41:22.364819Z"
  },
  "messages": null
}

Two details there are worth keeping in mind. totalRows is the count before paging, so a caller can page without a second request just to get a count. operationsApplied is the server's own account of what it did, and it's the quickest way to find out that a key in your body got ignored.

The output column names are the ones GroupByAggregate produces in process: sum_, avg_, count_, min_, max_, std_ and var_ in front of the source column name. That is why the sort refers to sum_Revenue, a column that did not exist when the request was written. Compute uses the same expression language you'd use in process, parsed and compiled server-side.

Operations run in a fixed order

The body is a set of operations, not a sequence. However you order the keys, the controller applies them in one order: merge, the three filters (where, whereIn, whereNot), dropNulls, fillNulls, dropDuplicates, compute, window, then groupBy or aggregate, sort, melt, transpose, select, and finally sample or skip and take.

That's why the example above can compute Revenue and then group on it in the same request: computed columns are added before grouping. It also means you cannot filter on a grouped result in one call, because filters run first. Worth knowing before you go hunting for a bug. The block is cloned before any of this runs, so a query never changes what the registry holds.

Filters take short or long operator names: eq, ne, gt, gte, lt, lte and their spelled-out forms, plus contains, startswith and endswith. An unrecognized operator falls back to equality rather than failing, which is worth knowing when a filter comes back with the wrong rows.

For .NET callers, a builder

If the caller is also .NET, you don't have to hand-write that JSON. DataBlockQueryBuilder, in the same package, builds the request object for you.

var request = new DataBlockQueryBuilder()
    .WhereGreaterThan("Quantity", 2)
    .Compute("Revenue", "Quantity * UnitPrice")
    .GroupBy("Region", new Dictionary<string, string>
    {
        { "Revenue", "sum" },
        { "OrderId", "count" }
    })
    .OrderByDescending("sum_Revenue")
    .Take(5)
    .Build();

It also carries MovingAverage, CumulativeSum, RowNumber, Lag, Lead, InnerJoin, LeftJoin and Transpose.

What the package does not do

It doesn't authenticate anyone, and I'd rather be plain about that than have you find it out in production. AddDataficationServer registers two authorization policies, DataBlockAccess and DataBlockAdmin, both requiring an authenticated user and a scope claim, and it never calls AddAuthentication. Whatever your host uses to establish identity is what protects these routes.

Output formats depend on the host too. format names a sink from the sink registry, and when no sink matches, you get the structured JSON above rather than an error.

One field is accepted and not applied. pivot deserializes and this endpoint never reads it, so a body carrying it comes back unpivoted, and operationsApplied shows it missing.

How this differs from a minimal API over a table

A minimal API in front of Entity Framework Core serves rows: the caller computes over them on its own side, or you write an endpoint per aggregate. That's a fine design, and a different one. Here the unit of a request is an analytical operation, run against a block that's already in memory and already typed, and the response carries the result plus a record of the steps taken. The trade is fewer endpoints and no per-question code, in exchange for a body the caller has to learn.

If you put this in front of a block, the thing I want to hear about is the key you reached for in that body and didn't find. That's the part of this I can still change.