Beyond metadata: building an open context layer for AI with OpenMetadata

Duration: 13:17 1936 transcript words EN summary

Video summary

The speaker introduces OpenMetadata as an open-source context layer designed to empower AI agents by providing them with a comprehensive understanding of organizational data. Drawing from a real-world scenario where leadership sought to build an internal AI data assistant, the presentation highlights that AI coding tools are not magic but rely heavily on high-quality context layers like code repositories. In contrast, early attempts at building a data assistant failed because they lacked shared context across five disparate systems, leading to hallucinations such as incorrect revenue calculations and misunderstandings of local business rules like fiscal quarters. These failures demonstrated that while AI models can generate SQL queries, they struggle without the semantic meaning and historical context necessary to interpret company-specific nuances. To address these challenges, OpenMetadata connects to various data sources through over 90 connectors, ingesting metadata, schemas, and business logic into a single unified graph. This architecture allows data teams to collaborate on defining data quality, governance, and ownership while simultaneously making this information accessible to AI agents via APIs and Model Context Protocol (MCP). The platform solves three critical problems for AI: it provides context by cataloging what data exists, adds semantics through glossaries and ontologies that help AI distinguish between similar column names like gross and net revenue, and offers memory capabilities to store organizational decisions. For instance, the system can automatically identify test data that should be excluded and understand complex relationships between financial metrics based on defined business rules rather than guessing from text alone. A key feature of this solution is its ability to create a persistent, versioned memory within a new interface called Context Center, which stores documents and decisions accessible to the entire organization. When an AI agent makes a mistake or learns a specific rule, such as correcting the definition of Q1 for a specific region, that correction is saved permanently rather than being lost after a session ends. This ensures that every interaction refines the organizational knowledge base, allowing AI agents to write back and improve over time without requiring a complete rebuild of the context layer when switching platforms. The open foundation built on standards like RDF and JSON Schema ensures flexibility and longevity, preventing vendors from locking organizations into proprietary ecosystems. In conclusion, OpenMetadata transforms how AI interacts with enterprise data by bridging the gap between human governance and machine execution. By centralizing metadata, business rules, and historical decisions into a searchable graph, it enables AI agents to generate accurate, context-aware responses that respect local constraints and global standards. The speaker emphasizes that this approach not only prevents costly errors like accidental database deletions but also fosters a culture where AI corrections become permanent assets for the company. Ultimately, the tool empowers organizations to leverage advanced AI capabilities safely and effectively by ensuring that machines have access to the same rich context that human experts rely on daily.

Read the full video transcript

Thanks uh thanks. My name is Mayur. I'm a software engineer at Collibra. I'll be talking about how you can build an open context layer using open metadata. For sake of this talk, let's let me wear my data engineering cap. And let's say this is the stack that we use within our organization. We have Postgres as our application database. We load it to Redshift via a Fivetran pipeline. There is DBT involved for transformation. And finally, we have Superset for our BI reporting need. And one day the leadership comes to us and ask, "Now we are generating more than 50% of code using AI. Why can't I ask question about our our data to AI and get a straight answer?" So they assign us the project to build an internal AI data assistant. And while analyzing this, we find out that AI for coding does not work by magic. It works because it was built on top of the best context layer available. That is a repository. The repository contains the code. It is centralized. It can be searchable via AI. So it it can explore the code on its own. It can find out what is available, what is not. And that it also contains a lot of tests. So whatever AI is generating, it can validate by running those tests and it can see if it is it is passing or not. Repository also has GitHub history. Uh so it can understand why a line change was made. And finally, it also has a good documentation and there is always human in the loop for providing feedback to the AI. Now let's take a look at how this would work for our data assistant. We have five different systems and five different UIs and there is no shared context in between them. And these five systems are not even providing a complete picture about the organizational knowledge. For example, a physical year rule is defined in some random Slack thread from 2024. Now, we try to build our version one for data assistant using We just dumping all the table schemas in into the agent and we ask it a simple question about generate me a report of for Q1 uh for the physical year and use the fact sale table. It very confidently generated a SQL query and it is well-formatted as well, but let's see what's the problem with that. Now, there are three main problems. I asked it to generate revenue, but it gave me seven random sums. Now, it is my job to figure out what is revenue from that. It is That is not providing answer. It is just dumping out information and asking me to figure it out. Another problem is that it considers that Q1 is from January to March, which is not the case for us in India. And this table also contains some test data, which inflates my revenue and it is not able to ignore that. So, the model knows how to generate SQL, but it does not know context about my company. And that is not the only problem uh for this data assistant. Let's say I want to also integrate a different all all of the systems into my AI agent because let's say I want to also debug broken pipelines. I want to debug broken reports. So, for that, I need the full context of our data. And to build it, I'll have to deal with authentication system for all of the system. I will have to build tools for AI to access the data if those are not already available. And finally, I'll also have to take care about data access to not not end up into situations like these where Claude just deletes production database. And that's where Open Metadata comes into the picture. Open Metadata is a open source context layer for AI. It has more than 450 plus contributors. It has 50 million plus Docker downloads and it is deployed across more than 4,000 large organization across the world. And let's see how it fits into our system. So, now Open Metadata is the one which is connecting to all of these five different systems. It pulls the metadata and information around metadata. And then data and business teams collaborate on the Open Metadata platform and provide meaning to the data. Now, initially it was designed for data and business team to collaborate and for fulfilling their data discovery, data quality, and data governance need. But now, since this has the full context in one graph, it has also been accessible to AI agent for becoming its context layer via MCB and APIs. So, let's see what Open Metadata is trying to solve for AI agents. It is tackling three problems. First is the context to make AI aware about what data exist in our organization. Second is to give semantics, like to give meaning to the data that exist. And finally, memory to keep record of all the decisions that we made across our organization. And for example, this is the example of the stack that I discussed earlier. I have five different systems. OpenMetadata is able to capture all of this in a single graph. And it also captures the flow of data, so I know that okay, Postgres is went into Redshift via Fivetran pipeline, and then that table is consumed for my BI report. And it not only captures information from these five tools. There are more than 90 plus connectors uh from where it can extract metadata, and including Apache Pig or We have a lot of dashboard connectors, a lot of pipeline connectors, and there are many more categories of connectors that we support. And this is the example of metadata is tries to pull, right? So, for example, this is a dim customer table. It automatically pulled up all the columns available, their data types. If they had uh description, we pull that as well. And we can also attach information like tags and glossaries to it. We can define the ownership. We can define the certification of if whether my table is of gold standard or silver standard or etc. And this information is not only available to humans via this UI, but it all it is also available to the AI agent via MCP, right? So, it can pull in the same information in just one context call. And let's see how it solves for semantics. Let's say I have these three columns in my fact sale table, and all of these are representing a monetary value, right? So, let's say if when I ask what is revenue, it will have to guess between all of these. But, that's where OpenMetadata tries to solve it, right? You We see in the previous slide that we can attach glossaries to the column and from that information it is able to determine that okay, gross sale is for gross revenue, net sale is for net revenue and total sale it does not even contribute to revenue because it has taxes and shipping details. And it not only interprets it from just a text view, it it can also gather that information from a relationship graph because Open Metadata now also supports defining ontologies on glossary terms, right? So, for example, in this case, net revenue can gross revenue is calculated from net revenue and this information is available to AI uh as well as a relationship graph. So, it is not determining this information based on text. It is able to reason it on top of the the graph. Uh and the final piece is the memory. So, we talked about how Q1 how it made mistake in our query where it assumed that Q1 is January to March, but it actually it should be from April to June, right? And the the test data should be ignored. So, let's see how it can we can do that. So, this is my Cloud desktop setup where I'm asking it the same query that generate me SQL query for fact sale table and for Q1, give me the report. And you can see that it is making call to the Open Metadata MCP and it is able to search metadata. And finally, it is able to generate the SQL, which is much cleaner than what we saw before, right? Now, it is not giving me seven random numbers. It is giving me proper net revenue and the currency it is used for. But still the problem of Q1 exists. So, what I do next is I'm telling Claude to remember that okay, Q1 is from April to June and you should exclude these columns because they are inflating my results. So, now this this step will not only remember the this information in my local session, but it will also create the same memory in OpenMetadata. So, OpenMetadata now has this new UI called context center where it can where it can store and create memory which can be accessed by anyone in the organization. And finally, uh after getting this information, it is able to generate the accurate query. It is able to figure out the Q1. It is able to figure out the test columns needs to be ignored. And let's go back and the context center not only stores the memory, it also can hold documents. It can hold hold the articles like like Notion to give the AI more context about the organizational knowledge. Okay. So, now with the same model, we are able to generate the accurate query. So, it was not the model which was not capable of determining this information. It was the missing context that OpenMetadata is solving. And this context layer is open. It is built on top of open foundation like MCP, RDF, JSON schema. So, let's say whenever you are deciding to switch EI platforms, you don't have to build the context layer from ground up. And finally, it is also compounding because EI agents cannot only read from the data, but it they can also write back. And every correction, every decision becomes permanent organizational knowledge. And that would be it. We saw how Open Metadata is trying to solve for context, semantics, and memory. Thank you. Do we have any questions? >> Thank you. Questions? >> Uh thanks for the nice demo on the cloud side. Uh question uh there needs to be a maker checker for context updates, right? As in at some point in time uh global context of the org cannot be updated via uh like individual chats, right? How do you maintain that? Uh is there a maker checker available? Uh how do you work with uh like uh context that gets uh polluted because of uh like local context, like for a singular analysis versus a global context. >> Okay. Yeah. So, the context center that I just showed you, it also captures the version history. So, basically, you can let's say if you if you figure out that okay, there is something going on wrong with the global context, you can go to that that version history, and you can make decision okay, there was something which was changed by someone which is not right, and you can always revert back to whatever is the correct uh context. Similar to how you would do it, let's say reset get reset command in in in the get, right? Yeah. Any more questions? >> Thank you. >> [clears throat] >> Thank