Video summary
The meeting focused on the challenges and opportunities surrounding automated reporting within the Linux Foundation's Decentralized Trust initiative, specifically addressing the limitations of current local models versus frontier models. Participants discussed how while local models are sufficient for high-level summaries, they often fail when tasked with synthesizing data from multiple input streams or generating complex reports suitable for slides. A significant concern raised was the reliance on personal free credits to prototype these solutions, which is viewed as unsustainable and undesirable for a formal pipeline. The consensus was that any viable path forward must not require individuals to donate their own resources but rather leverage infrastructure provided by the organization to ensure scalability and fairness for all members.
To address these limitations, a proposal was presented to build a robust reporting pipeline using a local model enhanced with Retrieval Augmented Generation (RAG). This system would utilize a Linux Foundation-hosted knowledge base that ingests information from various sources such as GitHub repositories, Confluence notes, meeting summaries, and project websites. By combining this rich context with small local LLMs, the initiative aims to create a customizable chatbot capable of generating detailed reports and facilitating cross-project searches. This approach allows members to query across the entire ecosystem without burning personal credits, effectively turning the collective knowledge of the foundation into an accessible tool for tracking progress in projects like Trust Over IP.
The discussion also highlighted the necessity of creating graduated levels of detail to cater to different user needs, ranging from high-level overviews for general constituents to deep technical insights for specific teams. The proposed solution involves developing skills and templates that enable a local model to capture and present information at varying granularities. However, the speakers acknowledged that building this infrastructure is a substantial undertaking requiring significant time, trial and error, and collaboration rather than quick fixes achievable by a small group in a hackathon setting. Consequently, the strategy shifted from individual experimentation to forming a broader coalition, inviting other groups working on similar pipelines to join forces and standardize their efforts under the Decentralized Trust umbrella.
The immediate next steps involve coordinating with key figures like David Boswell and Ben to formalize this collaborative approach and share progress among all interested parties. The goal is to transform isolated prototyping efforts into a unified, reusable infrastructure that benefits the entire Linux Foundation community. By pooling resources and expertise, the group hopes to overcome the technical hurdles of integrating diverse data sources and deploying reliable local AI models. The meeting concluded with an agreement to move forward collectively, ensuring that the development of these reporting tools is sustainable, accessible, and aligned with the strategic interests of all constituents within the foundation.
Read the full video transcript
Hello everyone.
>> Hello Connie.
>> That was an interesting
governance sorry graph meeting.
>> It is because I think there's a whole
bunch of work happening that is
happening at quite a rapid clip.
And I don't think one hour is good
enough to actually cover everything
meaningfully.
>> Yeah. Yeah, they are certainly very full
meetings.
Today is not a very full meeting.
So, I'm just going to wait like one more
minute and I I have some updates to
share and a proposal to go over, but
that shouldn't take uh uh definitely not
take the full hour.
Um
yes, but we'll just see if anyone else
comes in the next minute or two.
Is it very late for you where you are
right now?
>> It's 9:30 at night
at this word.
>> Okay.
So,
>> I'm usually
done with my work around 8:00, so yes,
it's late for me.
>> Okay. Okay.
Thank you for uh
coming to this meeting cuz I think yeah,
I I I I think this reporting thing
is going to be actually quite
interesting.
The uh I'm I'm working on similar kinds
of things about uh
how to work with more local models um
to get uh
to extract information and uh
boy, is there a big difference between
the local models running and the
frontier models you can access on the
web.
>> Yeah, but I think there's also
it depends on how much granularity or uh
sort of if you want to create reports
that are reasonably high level, but they
don't go in too deep, local models
should work in
for most cases.
>> Yeah.
>> But if you're trying to get to a point
where you would extract data from
multiple input streams,
synthesize them, draw inferences, and
then present it back as a form of a of a
report that can also be made into a
slide,
I don't think local models will help you
at that point because they're just
completely useless. They break down.
>> Yeah.
Yeah. Yeah, I I agree with that. So, um
so I don't think anyone I don't know if
anyone is coming but I mean if they do
all all all reshare.
>> Let me tell you this. If you if you're
using frontier models to do this, you're
using it out of your own account, right?
So, you're use burning up your credits
to do that, aren't you?
>> Yeah.
Yeah, but I don't
>> That is not that is not desirable.
>> No, but I I don't I'm using the free
credits. I have a account and I don't
use I use my work credit accounts for my
work work and I use the free chat GPT
account with the occasional credits
uh to do to do anything else such as
something like this because I mean
I I just I don't use the free tier. I
I usually have one thing to do like uh
yeah, close.
>> I I understand. All All I'm saying is
that
if
using
LLMs and uh and services is the path
that
uh Trust Over IP is going to walk down,
then the prototyping shouldn't require
somebody using their own personal
credits.
>> Yes. Yeah. Yeah, I agree. I agree with
that.
>> You are being generous by donating your
credits to this effort, but I don't
think that's the ideal way to go
forward.
>> Yeah. Yeah, if uh certainly for any kind
of um
uh
any kind of actual pipeline, that that
would be the case. So, here I'm going to
show
So, the proposal is because we're not
the only people who are thinking about
this
in the Linux Foundation uh in
Decentralized Trust. So, I have a
meeting with I'm need to establish a
meeting with Ben when he's back from
vacation. When we're both back from
vacation. So, I just got back. I think
he's been recently back. Follow-up
email, but it takes a little while to
start things again.
To the trust over to Linux Foundation
decentralized trust. So, David Boswell
and Ben, I shared this proposal
of
because then we're going to look
throughout the Linux Foundation
decentralized trust to see what kind of
pipelines and stuff we can build.
Not using our personal or our free
tokens or whatever else, but to
to come up with something
that we can that we can run ourselves
and potentially with a rag. Because we
could
the the local models
might not have the context window
sufficient to ingest the huge amount of
information, but instead the
we could do a a local model that is then
You know what a rag is, right?
>> I do.
>> Yeah. So, we could do a
a local model with a rag built by
Linux Foundation.
And and it might even be helpful um
because all the projects from the Linux
Foundation could be included in the rag
and then you could query cross project
within Linux Foundation decentralized
trust to be like, you know, write a
section of the report and highlight
where you think there is similarities or
opportunities and stuff like that.
Because we don't have enough
uh I don't have enough time to read
everything that all the other
decentralized trust
communities are producing and stuff like
that. So, anyway, um that would be the
knowledge base and then off of that we
would write the skills and templates in
order to query the local model.
Uh yeah, so you would have, uh, local
knowledge model, the reporting skills,
perhaps, uh, Linux Foundation
decentralized trust wide,
and the small local LLM, and then, uh,
uh, capture the information. So,
and then the, uh, Linux Foundation
pointed out this project,
which is, uh, a
I think this is, uh,
uh, in order to generate a rag.
It's a little older, but, uh, open
source way to generate the
vectors that go into the rag that we can
run on our own stuff.
And then the ingestion would be things
like, uh,
um,
the GitHub repos, the notes on
Confluence, the, uh, all of these
meetings generate AI summaries, which I
don't
have access to unless you're like
registered for the meeting, but
presumably, Linux Foundation would have
the ability to draw all that information
in.
The websites, etc., etc. So, that's the
proposal that I have sent on to David
Boswell and Ben.
>> This This is nice.
I think there is
So, in my mind, there is one element
that we also have to factor in,
which is that
the granularity of information that
something, uh, someone like you
who or Drummond will consume, vis-à-vis,
let's say, somebody who's generally
interested in the topic, but doesn't
want all the nitty-gritty. So,
uh, some of the details when we think
about how to present information. So,
one of I I realize that
this particular approach is
directed at ensuring that reporting back
to Linux Foundation is taken care of in
as much automated way as possible.
But, the reporting back to
Yes. And but the reporting back to Linux
Foundation
also implies that we are all reporting
back to the constituents of the Linux
Foundation, individuals and smaller
groups, and all the members.
And they don't consume all the
information in the same level of
granularity or detail. And so when
you are talking to them about rags and
reports and templates, I would request
you to keep this in mind that
it has to have the ability to produce a
graduated level
of
detail across the board.
>> Right. Okay. Yeah. Yeah. Yeah. So So
this would be the nuts and bolts and the
and the and the the
the overview would be we are going to
customize a chatbot that can
generate reporting that we need and be
also used by members of Linux Foundation
Decentralized Trust
to
do searches, etc., and learn more about
different projects through a chat
interface or reporting report generation
interface.
>> Yeah.
GitHub reporting is actually reasonably
easy. So
I'll post a link to something that
I was using locally for a long time and
then yesterday I boosted up with a bit
of AI and
sort of
pushed it to a GitHub repository.
>> Okay.
>> And
And it effectively allows me to on a
week-to-week basis keep track of
what is happening in DIF and TOIP.
>> Oh, okay.
And what
are you running So I'm I'm testing with
my work, I'm testing
some
small local models. And boy, what a
different I mean, I'm I'm just starting
right now
uh on on the um
Okay, sorry. I'm looking opening the
link. I'm just starting right now
uh about the um
Oh, that's a nice monitoring page, by
the way. Just starting right now on on
like the context window uh the context
and the and the difference between the
frontier model and what I can shove into
it and how complex it can and deliver.
And uh I think I'm using uh Quen
uh with 7 billion
uh
I don't know which Quen it is, but it's
not Yeah, anyway, it's a small local
model and my gosh, I have to take my
whole thing and put it into individual
chunks and say, "Give me information
about this one thing." and back and
forth.
So, I was wondering what your
experiences are if you have any with uh
any kind of local models.
>> So,
so this one, the one that you see uh as
the on screen as the monitoring page,
that doesn't use AI. It's just
essentially a whole bunch of Python
scripts that parse and extract the data.
>> Right.
>> Because it's GitHub, so it's easy to
create a lot of patterns out of it. But
that's not going to help when you're
trying to do Confluence, and I don't
even know if Confluence exposes a
service endpoint allowing LLMs to
consume, but anyways.
>> Yeah.
That would that would uh the Confluence
consumption would be we'd have to then
work with uh
uh Linux Foundation and and maybe there
we'd have to run some kind of
I don't know, some kind of script on
that server that just goes through and
extracts everything exports it into like
I don't know some kind of JSON
>> Yeah, and you basically need an existing
sort of pipeline that allow
>> Yeah.
>> It's going to be fun, and I I doubt it's
going to be I don't think the MVP is
going to happen.
If you are the only person working on
it.
>> Oh, yeah, yeah, yeah, yeah. I I cannot I
cannot make it happen. But
>> You need to You need to call for elves.
>> Oh, yeah, yeah,
for sure. And that's why that's why I'm
connecting
I would I would cancel this if it
weren't for the fact that David and Ben
have talked about and other groups are
already working on this and perhaps now
this is the time for us to join forces
and see how things are going.
>> So, if there are other groups join
working on this, is it not possible for
like almost like a big tent or an
umbrella meeting to happen as to people
who are working on it and
>> Yeah.
>> you take stock of what progress they
have made and then turn their pipeline
into the trust over IP thing and
determine if reusability is possible and
stuff.
>> Yeah.
>> Everything that cuts down your your work
should be encouraged, I guess.
>> Yeah, I'm going to
So, David mentioned that others are
looking into this and then when I have a
meeting with him, he wanted me to put
this proposal together to share amongst
the other groups
to see to see what other people are
doing and where their interests are or
what what anyway. So, so the next step
is absolutely meeting with David cuz I'm
not doing this on my own. It's not It's
not I don't I don't even have access or
anything like that.
>> Yeah. Yeah.
>> But there's something
>> But this is good. All right, let's do it
this way. Let Let me not hold you to
this meeting and given that I think we
have covered everything that needs to be
done. You can get have a lot of the time
back.
>> Yeah, you too. And and I'll and I'll
keep this
I Yeah, I don't know if I keep this
meeting or cancel it until cuz we can't
tailor any kinds of
the the project has changed from
prompting
which you can do like I could do with my
own free account which I needed to do in
order to generate the TAC reports. So
changed from that structure into let's
build an LLM pipeline with a rag which
is not what I can do. But Linux
Foundation is interested in it.
Do you think
>> is
infrastructure work that
is I mean it's going to take a lot of
time. It's going to take quite a few
individuals to do it on a full-time
basis.
And it is not
it's a lot of trial and error. It's It
doesn't have just because there is a
frontier model, this does not emerge
overnight by three people working
constantly 4 hours I mean for 4 days
straight at the at a hackathon kind of
thing. That doesn't happen.
>> Yeah.
Okay, I'll report back with David
Boswell that's me yeah tells me or Ben.
>> All righty.
>> Awesome. Thank you so much.
>> Thank you so much. Bye. Bye.