Video summary
The video introduces InMobi's DataBridge, a practical implementation of the data mesh philosophy designed to solve critical challenges in cross-domain data sharing within an adtech environment. The core problem addressed is the inefficiency and governance risks associated with traditional centralized data lakes or ad-hoc copying methods; centralization often dilutes ownership and creates cost accountability issues, while manual copying leads to fragmented local copies that are difficult to manage and prone to security breaches involving sensitive PII data. To overcome this deadlock where valuable exploration happens before materialization can be approved, the team developed a system prioritizing global discoverability without moving underlying data until a specific business use case justifies it. This approach ensures that engineering resources are not wasted on exploring potentially useless datasets by keeping them in their original silos initially and only replicating high-value data after rigorous approval processes involving legal and compliance teams.
The technical architecture relies heavily on Apache Iceberg tables to create a unified metadata layer where the actual data remains stationary within its respective business unit's storage, such as Google Cloud Storage or Databricks Unity Catalogs. Instead of physically migrating petabytes of raw data, DataBridge registers source catalogs into a central Polaris catalog and uses pointers to reference specific snapshots of that data. When queries are executed via engines like Trino or Spark, the system validates access permissions through an OPA agent before issuing short-lived tokens to query the original GCS locations directly. This federated querying model allows users to perform complex joins and aggregations across different domains seamlessly; only when a consumer team identifies a valuable intersection of data do they initiate a materialization request, triggering an automated process that creates a local copy while maintaining strict governance controls over who can access what.
To ensure the system remains useful despite being distributed, DataBridge employs hourly sync jobs to update metadata pointers and handles schema evolution by tracking version changes in Iceberg tables. The platform supports various materialization patterns including simple projections of specific columns, aggregations across partition keys, and join operations between different datasets. A significant learning from this implementation is the acceptance of eventual consistency rather than real-time synchronization for all data movements to avoid excessive latency; while current updates happen on a scheduled basis with plans for future real-time capabilities, the system effectively manages pointer drift by refreshing copies only when necessary based on snapshot differences at the partition level. Additionally, the team standardized column naming conventions across disparate business units using glossaries and conformed views to ensure that explorers see a uniform data structure regardless of where the source tables originated.
Ultimately, DataBridge demonstrates how organizations can balance autonomy with collaboration by enabling discovery without duplication until value is proven. The solution has successfully onboarded nearly 100 datasets from various sources, facilitating over 30 materializations totaling approximately three petabytes of processed data while maintaining robust audit trails and immutable definitions. By shifting the paradigm to "explore first, copy later," InMobi's team reduced unnecessary engineering effort and prevented the proliferation of unmanaged local copies that often lead to compliance risks. The key takeaway is that metadata management should be centralized for visibility and governance, but actual data movement must remain consumer-driven and justified by concrete business needs, creating a scalable framework where different teams can leverage each other's intelligence without compromising security or operational efficiency.
Read the full video transcript
Hey. Hi. Um, so this is uh a talk that
that we want to uh start with is in like
they're like uh in uh currently in my
company there are like different
business units and all of them have a
different cataloges. So this talk is
mostly uh
uh explaining how do we govern data
sharing across different uh lakehouses?
How do we make sure that that the data
that that has been shared how do you
have governance around it? So this is a
a kind kind of a talk of uh kind of
building a practical data mesh uh for
cross domain data sharing and we'll want
to introduce a kind of a principle where
you want to have a discoverability
across all the data sets uh but then
have certain use cases to just to
metalize locally. I'm uh Shom Gupta I'm
currently a staff engineer at InMobi uh
team. So, Inmovi is mostly an adtech uh
space company and this is from the
central platform team. Uh so, I'll just
go through the problem and then give you
the brief overview of what what we try
to build and how we want to uh make this
learning available for you. Uh this is
mostly an uh a talk where you want to go
through the architecture of how that we
build it and what are the constraints
and lessons that that we u had and what
what did we do about it.
So this is the main business problem uh
where you'll have multiple business
units. They are residing in their own
silos. Uh they have their own catalog.
Uh they have their own data formats. Uh
and each each of them are handled
independently across them. U autonomy is
kind kind of a very good where every
data sets resides into single islands
but then data sharing never happens. You
are not uh you don't know how to access
data across the domains.
uh but the value comes in when the data
where you can get the data of other view
do some interactions around it and then
create value out of it. So at least from
the ad tech space u so one of the
examples is where you can get an
audience overlap. So let's say if I get
a user doing some activity in in a
certain app if I can get that
information into my ad serving platform
where you can target that user again is
something that uh will be really useful
for us. So that is where uh your cross
domain identifying the intersection
becomes very critical for us. So how do
we get that uh so one major way uh what
people or company would have done is
like centralize everything. You have
data across the BUS um you'll have like
data moving in um push to a central data
lake. So that is like a data copy that
that you'll be doing from one BU. You
copy that data completely into a central
warehouse. This will uh dilute your
owner ownership and then there will be a
lot of issues on how do you want to
govern those things up? Who pays for it?
Uh so if you are moving the data across
to a central place um who pays and how
how do you get a cost accountability to
that particular system is kind of an an
issue here. Other problem or or like an
other solution which teams would have
tried is uh doing an ad hoc copy. What
what what it's do is like uh you get uh
the data of certain BUS and then request
ad hocly whether you want to do that.
What you do is like there like multiple
teams uh what you do is request certain
copies get some approvals get a local
copy into your system and then again you
want another data you request another
copy. So what happens end up creating is
that lot of local copies are created and
then uh what ends up happening is like
it becomes a struggle to actually manage
and identify who copies the data how
that data is managed. Uh what ends up
happening a lot of PII data uh is
present in these uh databases and and it
gets moved across laptops and people can
freely um check all these things. So the
copy diverges uh the data source. No,
nobody knows how how that thing happens.
So that is a big problem here with ad
hoc copies
and uh the biggest uh so this is kind of
a deadlock or like the heart of our talk
where uh you can't govern what you
haven't explored. Uh so let's say if the
data is useful, you want to know whether
that data is useful or not. What you try
to do is explore that data. But once you
try to explore that data, that data of
another view is not available. So what
you do is like um try to request a copy
uh from another BU to your BU. So what
you do is like uh get the copy uh do all
the governance reviews and approvals and
then build pipelines to actually copy
the data from another BU to your BU and
then what what ended up happening is
that the moment you uh like lot of
engineering effort is wasted in actually
copying the data and then finally after
weeks of effort you're trying to explore
the data which may or may not be useful.
So that is where the loop comes in. Uh
your old model asks you to pay for the
cost of exploration before making a
business decision and everything
follows. So that that's the main problem
that we want to solve. Uh that is where
we have built this kind of a system
where um onto a philosophy uh where uh
you want to have a global discovery. uh
you have your data across all the views
but make that available to be explored
across all the systems and once you've
identified that this is a particular
business use case uh and you are sure
that this makes sense then only you
materialize it u
so so until unless a use case or or a or
a majority of uh business is valid
there's no need to actually uh do a
mutilization
um this is kind of a a funnel you You
can think of it as like uh every data
sets that is available only a handful of
them will be useful for exploration if
you have that kind of a data and and uh
once you explore that data across the
Bus and then there'll be like a small
set of data which justifies uh enough
value to actually go ahead and request a
copy and your governance and your all
the approvals of how do you want to get
those accesses should reside on the last
layer where you actually want to
materialize and copy that data into the
view itself. So that's the core idea
that we want to uh pursue this uh system
around. U you want to have kind of a
data sets to be explored and uh be
available for exploration. uh do that
overlap analysis um with your data and
what whatever the data is present in a
specific view do that overlap analysis
and then only if there is a certain
business use case and you find a certain
data set which is useful for you then
only do a metalization so um so before
going into the architecture I'll just uh
start briefly on uh what is actually a
table how does it do uh this is mostly
an uh a standard
how the cataloges works uh into this
kind of a system. Uh there will be like
a tables um and then corresponding to
that table will be a pointer to a
metadata where the metadata saves all
the uh schema partition specs and all
the related details of where the actual
data resides and the data resides in
different location. So data can reside
into a completely different location
itself or it can be into a same GCS path
itself. So that's how an actually a
table in an iceberg or a standard
catalog works. U
so when you're doing a write um it just
creates a new snapshot. So let's say
there's a catalog pointer to a table it
may respond to a particular snapshot
there will be like a four or five uh
park files in into a bucket. So let's
say if you add a new data or a new
partition path, it creates a new meta uh
snapshot for that particular system and
then what happens like now your catalog
is pointing to an older data. Uh what
you just do is like point to the latest
snapshot. So that's how uh the table and
we want to use this kind of a schema to
build our uh system. Now coming to the
main architecture of uh what what we
have in our system is um this where your
data never moves. Um what what we try to
build is kind of a data mesh platform
where uh your data can reside into the
BU's uh so data does not move. So all of
your BU's data uh in their own uh
certain cataloges it can be either unity
or it can be a Polaris itself. What we
do is like we register all these
cataloges into our central Polaris. uh
so this is like a unified metadata
catalog. What we do is essentially
create tables using the references of
metadata present into the uh source
catalog. So essentially it is uh putting
a metadata layer on top of your u actual
catalog. So this is where your uh all
the sources u the data gets uh pulled
into a central unified catalog. So this
is um and then we build a kind of a
control plane which is kind of a a back
end to actually manage all these
onboarding utilization governance all
the audits corresponding to this
particular system that that we have
built and then uh you provide kind of a
a querying engine. So so you use train
overx to actually go ahead and query uh
our central catalog and which internally
is useful for exploration as well. So
that is where you go ahead and explore
the data and then we have uh kind of a a
UI for uh all all this. Uh so we used
open metadata which provides kind of a
way to actually use uh or visually see
all those tables that have been
onboarded into our central catalog and
then um all the metalization and all uh
the that we have built kind of a custom
um filters where you can go ahead and
see that data itself.
So going into the detail on um so this
is like what the architecture of our
system where data resides into your in
your own specific views. Data never
moves and then we have kind of a
metadata layer which actually points
that to a central or central system owns
the metadata and then you can go ahead
and search all the data. So the query
spans across the
um views and then you can go ahead and
make a decision whether you want to go
ahead and mutilize it.
So yeah uh now I'll just go through the
details of each and every steps how the
onboarding works. How do we uh make sure
that the data is updated? So first way
first step is like how do we onboard the
metadata? So it is just a registration
flow where you first register the actual
catalog. the source cataloges are
registered then you can just onboard the
table. So the onboarding is like uh get
getting the source table and then
pushing that metadata or like creating a
new table which is pointing the metadata
of a source view. So it essentially is
like a pointer to a metadata in the
source table and creating a new uh table
on the uh central polaris side. So once
that table is created so so we'll have
like a similar table uh structure
created here but it points to the
metadata of the catalog or the source
catalog and then once it's uh onboarded
or like enabled into the central polaris
we want that to be discoverable across
the systems. Uh so that is where the
metadata moves but uh but the data
remains the same. Uh so we have like a
three onboarding steps. uh so so um lot
of our views are using datab bricks is
unity catalog and some of them are using
an existing iceberg as well so we kind
of support onboarding all of them into
our catalog so for datab bricks uh unity
we use um kind of a uniform enabled so
what what it does is like creates a
metadata of uh the tables
correspondingly in a format which
iceberg can understand so iceberg
adapters are simple for plain delta
tables uh we use some something called
as an X table which creates the metadata
uh similar to a format which iceberg can
understand. So once the onboarding is
done with uh all of this tables have
kind of a metadata compatible to an
iceberg format and then that has been
onboarded to the central catalog.
So that is where your uh once the
registration happens uh your tables are
actually are present into your central
catalog and then um the next step is
like once the onboarding has been done
how do we make sure that your central
catalog or central uh actually stays
updated. So since uh what can happen is
like producers uh what we have built is
like producers can keep on uh producing
new data. Um what we have built is kind
of a sync service. So what it does is
like tries to see what is the latest
snapshot that is present onto the source
side and try to sync that into our
central polaris. So currently we have
kind of an hourly job. So what it does
is like tries to see what's the latest
snapshot or the latest metadata on the
source table try to get that snapshot or
metadata into our uh central polaris
itself. So what it does is like
essentially it is moving the pointers uh
from the previous snapshot to the latest
snapshot. So it's as you can see this is
just a a movement of pointers from um
the previous snapshot to an higher
snapshot
and then you can have all all this uh
can be explored or or via trino on
databick where you're and then
the same things can be done on the open
meta as well where where you can see the
latest snapshot or latest schema of the
tables that have been onboarded.
So currently this is an hourly sync
where we try to see if there has been
latest changes and we want to sync that
uh into a central vol. Now since the
date now the uh what we have done till
now is like uh moving the data or the
metadata into a central catalog and
making that up to date. Now how do we
want to query it? So when you do a query
uh what what we use is kind of a SQL
query and that query can be uh fired via
a stino and a spark. So we have kind of
a two offerings where um most of the
queries go via trino and then let's say
they are like uh some of the data
scientists and uh we are which are more
familiar to datab bricks for that we use
spark. So then what it does is like it
queries uh the central polaris to get
the metadata and uh central polaris
gives you kind of an uh scoped access.
So we use something called a vended
token. So it creates a a short-lived
token for the tables or or the GCS
location which you're trying to access
and then it uh runs the queries or like
once it gets the metadata of where that
actual location is and a correspondingly
a token then it fires those queries
across the the BU's data sets um and
then merges those uh cross joins or
across domain searches uh merges the
results and responds back to the UI. So
this is how the u uh the system that we
have built. Uh it's like only the
results are actually moving across the
network not the underlying data.
So yeah and this is how the actual query
runs uh where when you run a query uh it
tries to see uh whether you are the
correspondingly access to actually query
that system. So that is where your
engines where where we have set up some
some rules on how who and what can
access these kind of a data and we have
and then what what we do is like when
that queries is done on the polaris side
then there are like certain arbbacks
that we have set up so that like uh it
creates uh let's say checks the
permissions uh so so that person
querying the data actually has the
access to explore that data sets itself
and then once Polaris confirms it create
it lends a a token uh for you to access
that path itself.
So once that token is there you uh then
query engine goes directly to the GCS
bucket and tries to access that data
directly.
So yeah once the uh so we have created
kind of a unified uh analytics platform
where uh there are like certain systems
um that is built on top of uh polaris
catalog itself where you you have some
sort of a krino for our ad hoc queries
database for let's say uh mostly for ML
platforms and ML engineers to actually
go ahead and see that data superet is
mostly for some some reporting and vi
dashboards that you want to uh show up
Open metadata is where we want to
discover the it's mostly a discovery
portal where you want to discover all
the data set schemas and their
relationships lineages and and all the
stuff and all of them converge or let's
say converge to a central uh Polaris
catalog.
Uh now coming to the uh main part where
you want to mutilize it. Uh so
milization is where um the everything
comes together where you want to have a
a copy of that data into your uh DU's
network. So that is where you want to
have all the query running and explore
all the data sets. Once you have made
sure that this particular use case is
actually um useful for me and I can
productionize it, I can derive some
business unit out of it. Then you go
ahead and requestization. So this is
initiated by a a consumer where teams
asks for a local copy then it goes
through the approval flows where you
have an infosc and a legal approval
where whether you are uh you could go
ahead and actually uh make a copy of
that. So all the governance layer sits
here uh you have kind of a consent gates
whether since you are copying the data
from consumer to kind of a producer. So
we'll try to get approvals uh from the
consumer and producer side. um once the
once all those approvals are there then
only we go ahead and u complete the
mizilization request. So then we uh had
kind of an automation kicks in where we
try to create a a copy and we try to
keep that copy up to date as in when the
source data gets updated.
So we support kind of a three use cases
here uh currently. Um so first is mostly
a projection where you can do select uh
some filters uh some some select columns
and all so that uh the exact columns
gets transferred to the producer from
producer to the consumer. Other one is
aggregation where we support some sort
of a group by and some summation
aggregations across the partition key.
So let's say you want to have some
across a certain partition key of of our
table. So uh that that is another
pattern that that we support here.
Another pattern is actually an align
join. So what what you can do is like
you can select two particular tables and
then have kind of a joint query on top
of that and utilize that joint result
itself.
So this is the kind of a patterns that
we try to support here. Uh and then
coming to the mutilization. Um so what
we do initially is like just the get the
latest dump uh from the table and then
do a a complete copy into the B used
network. So the first metalization is
actually a complete refresh uh where we
check each and every partitions and uh
we just dump that information into the
BU's catalog and then the next
metalization when it runs it now needs
to know what changed. Um so what we want
to do is like kind of an incrementally
refresh that. Um so what we kind kind of
a use is kind of a no uh we save the
last state that we metized. So when we
did the first mutilization what we do is
like we try to uh watermark uh that this
is the snapshot of source um table that
that I tried to uh mutilize and then
when the next mutilization starts in
what it tries to check is like hey the
last mutilization run the source BU was
at snapshot 42 and then what's the
current snapshot so let's say the
snapshot is 548 now what it has to do is
now identify the difference between um
this particular snapshot and the latest
snapshot. So what we try to do is like
uh kind of a identify uh the partitions
that that have changed. Uh so iceberg
provides a capability of kind of a
logical uh it exposes kind of a logical
partition table where you can identify
that these are the partitions that have
changed across the partitions. Uh so in
that part uh partition table it shows
the last time snap and the last uh
snapshot version that updated it. So
using this mechanisms we are able to
identify that uh from this snapshot to
this uh through the latest snapshot what
all partitions have changed. Uh for us
um so so it mostly identifies at a
partition level what changes have been
done. So if there has been a certain a
single row that that has been changed uh
we'll go ahead end up uh updating the
complete partition. So this is kind of a
trade-off that we have taken um to
identify the changes uh in a faster way.
Uh so yeah that's how we identify the
change in the partition and um the next
thing is on the schema evolution uh
where if there has been certain changes
into your source uh catalog schema how
do we make sure that the data copy that
gets uh produced into the uh consumer
how do they get updated so we uh we use
we again use iceberg's native uh schema
version support. So iceberg actually
saves a lot of metadata of the schemas
that that that it has evolved to. So
it's uh so every new update that that
that you do it creates a new schema
version and all the schema versions are
available. So what we do is like we try
to save the the last schema version that
it was mutilized to and then when the
new malization kicks in it tries to
identify what's the schema version of
the latest uh metadata and then tries to
identify the div. So let's say there
have like new columns added if there
have been new columns dropped. Uh we can
then apply certain configs what to do if
the there has been certain changes into
the uh schema itself. So let's say there
are like new columns added uh if those
were not used in uh metalization then it
doesn't make sense. If the columns had
been dropped u then you want to see
whether those columns are actually used
into your metalized table. blend then
you want to have some sort of a config
whether you want to drop it, stop it or
like move forward. So that is what our
uh current system does. Um we have
currently exposed it as a configuration
file where you can uh try to stop the
tag uh whether you want to um stop if
there are has been changes in the schema
pause or like notify or there's some
config of auto evolve as well.
I'll just move on to the uh major
learnings that that we had uh with this
kind of a solution. Uh so the first
learning is like u we are moving the
metadata across like there will be some
changes uh there like pointer drift uh
uniform lag that could happen. So since
we are we avoided moving the data we
have created a kind of a consistency
problem. So what we have made sure is
like we took a conscious call to uh make
it an eventual consistence system. So if
there are any delays um so the next copy
or the next mutilization will actually
kick in and it will get that latest data
updated.
Another is like on the query uh because
what happens like these queries run
across the BUS um um since we have
avoided uh moving it to a central place.
Now your queries are running across the
BU itself. Um so that is where you want
to profile how your query layer is
performing. How can we do some sort of a
caching around it? Whether that can be
optimized. Um and we do a kind of a
token pending where we try to give you a
a token uh where a shortlived token. So
we try to do some optimization caching
those tokens als so so that like uh so
so that the queries can be run faster.
Another big problem of like let's say
having a a shared uh system is like um
lot of BU's data that gets onboarded
here may have a certain different uh
variable names um they can have the same
kind of for example let's say user
identifier same identifier can have
different values or let's say some views
can use user ID some can use user ID we
kind so when we are onboarding that data
into a central platform we want to have
kind of a uniformed view or or like a
uniformed uh column naming conventions
across the systems. So what we build is
kind of a kind of a glossery and a
conformed view. So when we onboard a
table, we try to see if there's actually
a glossery which maps this particular
column to an actual canonical uh column
name and then when it's onboarded it
creates a view out out of it. So that uh
and then the searches are on the view
not on the actual table. So that like
all the people actually searching the uh
or exploring the data have a uniform
column or uniform can canonical
structure around it.
So that's what uh our database platform
gives us today. It provides a a
federated discovery across every domain
governilization
uh with approval flow um all delta and
iceberg under a unified catalog. Um and
we have kind of a good amount of
auditability immutable definitions uh
and metalization actually driven by a
consumer uh platform.
These are mostly the numbers. Uh we have
like currently almost like 100 data sets
have been um available to discover
across our platform. uh 30 plus
mutilization around like we have
mutilized almost like 300 terabytes of
almost like three pabytes of data almost
like 12% of mutilizations that we have
done. The key takeaway of this is mostly
do have kind of an uh platform where you
want to explore first. Um have consumer
decide where when do you want to have a
a a copy of the data not the reverse uh
metadata is actually an harder part uh
have kind of an immutability and audit
across the the teams. So yeah um thank
you for everyone. Um so this particular
team uh the deep was kind of our main
lead here. Uh I am Shivam. I was mostly
looking into the malization stuff. Uh we
have Pushpendra and uh Kulib who were
looking into the sink and uh federation.
Uh if you can join
Thank you.
Thank you. Thank you. room
questions.
So you mentioned when you query it
centrally right so there are certain
kind of projections and joins and all
that you support. So can you explain a
bit like what layer does that execute?
Is it the TRO or the spark which is
doing that or you have some custom query
layer on top? Yeah, it's it's mostly TRO
and Spark which are actually running the
queries. So what happens like TRO
queries uh so first checks the catalog
gets the credentials or like a the
location of those files. So let's say
they are the locations in in a certain
GCS it gets that location from the uh
central catalog and then directly
queries the uh GCS or path to get get
the information. So mostly it is uh
Trino and Spark doing doing the actual
query
>> but in that case all the operations and
the dialect that is the operations that
the dialect of Trino and Spark will
support should be supported right. So
>> yeah yeah so we try to make sure that
like inter operability across the
database are handled into the system
itself.
>> Okay.
>> And the FGAC is managed what you have a
ranger setup or how do you where do you
define the user access permissions and
policies? Yeah. So that is like a a big
problem for us. U so initially what we
did is like kind of a have some sort of
an arbback uh defined into the trino and
a separate uh system for a database. So
what what we had is like two set of
arbbacks but then um um what what we
ended up doing is kind of a use an OPAI
agent uh which will be kind of a a
central uh thing where you define all
the accesses uh correspondingly to a
user and correspondingly let's say a
certain user has certain access to a
certain view with a certain data sets.
So all that accesses and arbback is
currently now defined into a single OPA
agent and all Trino and uh Espark
actually uses or queries that to get
that information.
you know you can still write a custom
plug-in to manage this but how do you do
this in spark
>> so for spark what we have done is like
we have created a dedicated clusters for
that so let's say um what we can have is
like uh different arbbacks created for
different clusters so we'll have a
certain group of users tagged to a
certain cluster itself
>> basically role specific clusters So
>> you may have covered this in the
beginning which I missed but the
question is this kind of architecture
what use cases are
this is most suitable for
>> so
>> scenarios yeah so I'll give you one one
of example where um we have certain uh
let's say you want to target a
particular user that user is trying to
um
uh let's say buy some uh things on
online. So what you would have is like
we have certain apps uh there's
different view uh which can uh which
sees that this particular system or this
particular user is trying to view a
certain uh event. Let's say he's trying
to uh he's seeing um the uh the he's
seeing the socks or or let's say he's
seeing certain characters or he's
browsing the e-commerce site. So what we
can get is if we get those event stream
flowing in into an ad serving platform
then we can target that user uh knowing
that this particular user is actually
seeing or doing some activity in in a
certain other view I can use that
intelligence and target into my system
itself. So this is one of the use cases
that we have in adtech. There can be
multiple use cases where you want to
have some sort of an intelligence coming
in from a another view and use that
intelligence to actually derive business
output output for you.
Does that make sense?
>> So is it like creating different data
representation based on local needs?
>> Yeah. Yeah.
Okay. Thank you.
Uh I think we have maybe it's time for
one more question here.
I'll just continue to the previous
question. Uh would it not add to the
latency because we are doing late
materialization and schema evolution is
also no bit delayed. So for the add tech
uh would it not impact the latency both
for inference?
>> Yeah. So uh currently our solution is
mostly uh an offline process where you
have a scheduled job running at a
certain frequency. So this is the
current state. So we are building on uh
the next way where you can have a
real-time malization itself. But that
that is actually in work in progress. Uh
so that's an issue with the current
system. It is an eventual consistenc
uh in this kind of a system. So yeah,
next logical progression is like can we
make that real time but yeah that that
is a work in progress for now.
>> Okay.