Advancements in AI are going to change how work is done in every industry. Incumbents across the modern data stack are already working to add ChatGPT-like co-pilots to their offerings. These co-pilots will make it easier to write and maintain ETL jobs, structure databases, and build dashboards. All of these improvements will surely be productivity bumps, but they won’t be paradigm shifts.

What the existing solutions are missing is that AI has the power to completely change the data ecosystem. Rather than hard and inflexible data models powered by layers of ETL, AI will empower users to directly use data across organizations in the form they understand, regardless of the domain. Imagine a marketing analyst effortlessly querying real-time sales data from Salesforce, combined with customer feedback from Zendesk and campaign performance metrics from Google Analytics — all without writing a single line of integration code or waiting for an ETL process.

The Modern Data Stack (MDS) refers to a collection of tools used by organizations to manage, store, transform, and analyze their data. A few examples of layers in the stack are:

Many of the above products also provide more pieces of the stack or have a full MDS offering (like Palantir Foundry).

Not pictured above are the source systems or systems of record. These are the systems that originate the data, like applications and services. Some examples are application databases like Postgres, ElasticSearch, or Oracle, tracking services like Amplitude or Google Analytics, CRMs like Salesforce or HubSpot, and APIs like Stripe or Shopify.

Additionally, products like Customer Data Platforms (Segment, for example) have cropped up to add vertical specific views on top of source systems and warehouses. “Reverse ETL” capabilities of these products also allow you to write back to these systems to maintain that single source of truth.

The MDS exists for scale and convenience. Analytical teams want to answer questions that can only be answered by querying data produced by different silo’d data sources. High-scale event data often requires very fast analytical databases to query efficiently. For these reasons, the canonical solution has been to move data from where it’s created (applications, services, APIs) to a single place where it can all be queried efficiently together.

Unfortunately, completing a data transformation and moving enterprise data to a single data warehouse didn’t solve all data problems.

The MDS solved a lot of problems by centralizing data and governance in organizations, but it wasn’t able to entirely remove complexity. Complexity shifted left to the ETL (ingestion and transformation) of data instead. Now organizations need to build and maintain data pipelines that bring data into a data lake, then transform and clean it inside of a data warehouse. Services like Fivetran exist to manage these pipelines for organizations, but using these managed services means orgs are dependent on the included connectors.

If organizations are using a Business Intelligence layer that requires a strict semantic model (data model matching the business objects) like Looker or ThoughtSpot, then they also must manage transformations to get data to a BI-ready state. This is ultimately more work for data engineering teams to configure and manage, and that’s if you can get the various stakeholders internally to agree on what such a semantic model is supposed to look like. Solutions like dbt try to make defining this strict semantic model easier, but these solutions don’t make BI any more flexible.

The impact of this is that when a source system has a field or object that stakeholders need that isn’t included in the managed ingest connector, data lake, or BI layer, it can take weeks-to-months of coordination across data engineering, analytical, and business teams to get that new data into a dashboard. Wasn’t putting all the data in one place supposed to solve this back-and-forth?

The MDS exists to get all the data into one place, in a form that BI applications can read and people can understand. It enables point-and-click, self-serve analysis, as well as centralized data governance. So what’s next?

With the rise of Large Language Models (LLMs) and semantic search systems, suddenly we have the power to make data legible and composable across different systems without moving it around and transforming it. This opens up a massive opportunity for ad-hoc analytics. We are introducing the Liquid Data Layer to do just that.

The Liquid Data Layer (LDL) is the first and only AI-native data layer. Unlike semantic models or hard ontologies of the past, the LDL is a flexible data model. It combines knowledge of existing data source systems, business nuance, and context about user intent to create a virtual data model, purpose-built to answer a user’s question or to help a user discover data.
‍
The LDL plugs into every part of the Modern Data Stack, from source systems to ETL to BI to governance to build an AI-native, flexible model of an organization’s data ecosystem.
‍