"Data-driven" has been attached to so many things that it has stopped carrying much meaning. It sits in my own description of this company, so I owe you a definition.
A data-driven world is not one where everybody has a dashboard. It is one where the software that runs a business is also the software that learns from it, and where the distance between recording a fact and acting on it is short enough that acting is the normal thing to do.
By that definition, most of the world is not data-driven yet. It is data-collecting.
Count the hops
Picture an ordinary line-of-business application. It records orders, shipments, support tickets, sensor readings, whatever the business does. Every one of those records is written by code that understands it completely: the types, the constraints, the rules about what counts as a valid order.
Then somebody asks a question. Which product categories are growing in the Northeast? Which support tickets predict churn? How does yesterday's sensor drift compare with last quarter's?
Here is what usually happens next, counted in hops:
- The data is exported out of the application's database, usually as CSV, and loses its types on the way.
- It is loaded into a tool in another language, where the types get guessed again, sometimes wrong.
- The business rules the application already encodes are written a second time in that tool, slightly differently.
- The answer is pasted into a document, a slide, or a message, with nothing left connecting it to the data that produced it.
- If the answer matters, somebody rebuilds the analysis inside the application, a third time, in the original language.
Five hops, three implementations of the same rules, and a week between the question and the code that acts on the answer. Every hop is a place where the meaning leaks out. Every hop is also a reason not to ask the next question, and that is the expensive part, because nobody ever writes down the questions they decided against.
Why the hops are there
Nobody wanted them. They are there because the application's own platform never had the tools for the work in the middle.
A very large share of the applications that run businesses are written in .NET, and the analytical toolbox there has been thin. LINQ handles a filter and a projection. Past that, a grouped percentile, a moving average, a pivot, or a join between two files of different shapes meant writing it yourself or leaving for a platform where somebody already had. Leaving was the rational call. The tools were better over there.
So this is a tooling gap, and tooling gaps close. The interesting question is what the tools look like once this one does.
What closing it looks like
Three layers, and I think all three have to live on the same platform as the application.
A data layer that feels like the platform. A column-oriented table that loads from the formats data already lives in, does what analysts expect, reads as a pipeline, and compiles. Typed, so the guessing at hop two never happens. Embeddable, so the rules at hop three are written once, in the application, and reused by the analysis. And with an answer for the day the data no longer fits in memory that does not involve rewriting anything.
A loop for looking at the data. Understanding data is iterative: load, look, reshape, look again. The fastest loop wins, and today the fastest loop for a .NET developer is in another language. You should be able to open a notebook, reach for your application's own types and rules, and look at the data without an export step in front of it.
A way to check what came out the other end. All of this ends up as a screen somebody looks at. Checking that screen from .NET has meant borrowing a test runner from another ecosystem, with a process boundary sitting between the test and the page. It should not have to.
We have spent the better part of two years on the first layer. It is not public yet, and this is not an announcement. I am writing the aim down now so that when the packages do show up, you have something to judge them against.
The test I apply
Take those five hops and ask, one at a time, whether a tool removes the hop or only makes it faster.
A faster export is still an export. A better CSV parser still guesses types. A nicer dashboard still separates the answer from the code that produced it. The hops disappear only when the analysis runs on the same platform, with the same types and the same rules, as the application that produced the data.
That is what I mean by a data-driven world. Not more data, and not more dashboards. Fewer hops between a fact and a decision. I am building for the platform where I think the gap is widest and the payoff is biggest. If that is your platform too, I would like to hear which hop you are stuck on. The address is support@datafication.co.