Seven connectors, one shape

A data library is mostly its edges. The work you sat down to do happens on a table in memory, but the first hour goes on getting the table: a semicolon delimiter, a workbook with its header on row three, a bucket holding four hundred files.

Datafication.JsonConnector and Datafication.ExcelConnector went to nuget.org today at 1.0.9. That completes the set: seven connector packages, one for each way data tends to arrive. Seven packages, but one shape, and the shape is the part worth your time.

The shape

Every connector is a class you build from a configuration object, and every one implements IDataConnector, which is two methods:

Task<DataBlock> GetDataAsync();
Task<IStorageDataBlock> GetStorageDataAsync(IStorageDataBlock target, int batchSize = 10000);

The first reads the source into memory and hands back a DataBlock. The second streams it in batches into disk-backed storage, which in practice means a VelocityDataBlock writing a .dfc file. Same connector, same configuration, two ways of landing. Which one you pick is about size, not about where the data came from.

The seven

Four of them hang a shorthand off the static DataBlock.Connector property, for when the source is a path or a URL and the defaults are right:

var sales  = await DataBlock.Connector.LoadCsvAsync("sales.csv");
var events = await DataBlock.Connector.LoadJsonAsync("https://api.example.com/events");
var budget = await DataBlock.Connector.LoadExcelAsync("budget.xlsx");
var trips  = await DataBlock.Connector.LoadParquetAsync(new Uri("file:///data/trips.parquet"));

Parquet takes a Uri. The other three take a string and build the Uri themselves, from either a local path or an http address.

Defaults run out quickly, and then the same method takes a configuration instead. The CSV shorthand assumes a comma and a header row, so a tab-separated file without one gets written out:

var config = new CsvConnectorConfiguration
{
    Source = new Uri(Path.GetFullPath("sales.tsv")),
    Separator = '\t',
    HeaderRow = false
};

var sales = await DataBlock.Connector.LoadCsvAsync(config);

Separator is a char?, where null means comma, and HeaderRow is a bool. Excel's equivalent is where the awkward workbooks get handled: SheetName, SheetIndex, HasHeader, HeaderRow, SkipRows, UseColumns and NRows. If you've ever been handed a workbook with a title block above the header, you know why that list runs as long as it does.

ADO.NET brings no driver

Datafication.AdoConnector depends on Datafication.Core and nothing else. That's on purpose. You bring the provider and register it:

DbProviderFactories.RegisterFactory("Npgsql", NpgsqlFactory.Instance);

var recent = await DataBlock.Connector.LoadPostgresAsync(
    connectionString,
    "SELECT * FROM orders WHERE placed_at >= @since",
    new Dictionary<string, object> { ["@since"] = DateTime.UtcNow.AddDays(-7) });

There are four shorthands, LoadSqlServerAsync, LoadPostgresAsync, LoadSqliteAsync and LoadMySqlAsync, each with a parameterized overload, plus LoadAdoAsync(AdoConnectorConfiguration) for stored procedures and timeouts. The connector checks the provider name with DbProviderFactories.GetFactory first, so a missing driver is a clear error before anything opens a connection. The full configuration is ProviderName, ConnectionString, CommandText, CommandTimeout, CommandType and Parameters, which means any provider with a registered DbProviderFactory works, not only the four with shorthands.

The web connector is five connectors

HtmlTableConnector, CssSelectorConnector, LinkExtractorConnector, ImageExtractorConnector and PageMetadataConnector, all built from WebConnectorFactory:

var rates = await WebConnectorFactory
    .CreateCssSelectorConnector(new Uri("https://example.com/rates"), "table.rates td")
    .GetDataAsync();

Each configuration inherits UseBrowser, which defaults to false. False fetches the HTML and parses it with AngleSharp. True drives a headless browser through PuppeteerSharp, which is what you need when the page builds its table in JavaScript.

S3 is a connector that calls the others

var config = new S3ConnectorConfiguration
{
    Region = "us-east-1",
    BucketName = "noaa-ghcn-pds",
    ObjectKey = "csv/by_year/",
    AllowMultipleSegments = true
};

var connector = new S3DataConnector(config);
try
{
    using var velocity = new VelocityDataBlock("ghcn.dfc");
    await connector.GetStorageDataAsync(velocity, batchSize: 50000);
    await velocity.FlushAsync();
}
finally
{
    connector.Dispose();
}

ObjectKey is either one object or a prefix. A prefix needs AllowMultipleSegments = true and the storage method: GetDataAsync throws NotSupportedException on a prefix rather than pull an unknown number of files into memory. I'd rather hand you an exception than watch a machine run out of memory. The connector downloads each object, looks at its extension (.csv, .json, .parquet, .xlsx, .xls) and hands it to the matching connector, which is why this one package depends on four of the others. ServiceUrl and ForcePathStyle point it at MinIO and other S3-compatible stores.

On the way back out

Sinks return values, not files. CSV and JSON give you a string, Excel and Parquet give you bytes, and you decide where they go:

var text  = await block.CsvStringSinkAsync();
var bytes = await block.ParquetSinkAsync();
await File.WriteAllBytesAsync("trips.parquet", bytes);

ParquetSinkWithSkippedColumnsAsync returns the bytes together with the list of columns it could not write. Nested DataBlock columns are skipped, which matches the fact that nested Parquet data is not read on the way in either. The web package adds a screenshot of a table as a PNG and a PdfSink. ADO.NET and S3 have no sink and only read.

Why two of them were late

These connectors were written across nearly three years, in the order the need turned up: JSON and CSV in 2023, ADO.NET and web later that year, Parquet in 2024, Excel in 2025, S3 last. All seven were in the repository when the first packages went to NuGet on 9 January. The Excel and JSON builds were not switched on in the publish workflow, so eight packages shipped instead of ten. Today's 1.0.9 turns them on and puts both on nuget.org.

For what the shape looks like once the data is in, the CSV to Parquet walkthrough runs a file through a connector, a few transformations, and a sink, end to end. Point one at a source of yours, and if it comes back in a shape you didn't expect, that's the report I want.