Video summary
The video presents a demonstration of an AI Editorial Assistant designed specifically for the Fedora community, aimed at helping users evaluate draft articles against established publication standards. The tool is capable of operating through multiple interfaces, including a graphical user interface (GUI), a command-line interface (CLI), and Jupyter notebooks, offering flexibility for different user preferences. Users can select between two specific publication types, such as the Fedora Magazine or the community blog, and choose from various large language models, with GMA 4 E4D recommended for current use. The system is built to be reproducible without requiring complex builds, allowing users to simply pull container images directly from Quay to run the analysis locally on their machines.
The core functionality of the assistant involves analyzing submitted text to ensure compliance with specific guidelines regarding tone, structure, and content quality. During the demo, the tool successfully identifies when a draft meets standards but also highlights missing features, such as featured images or proper header tags, which are crucial for publication readiness. If an article is deemed too brief or lacking in certain views, the AI provides detailed feedback that can be exported as a text file, allowing writers to make necessary revisions before resubmission. This process acts as a preliminary filter, ensuring that only high-quality drafts proceed to the final human review stage by the editorial team.
To support these evaluations, the project includes a comprehensive repository structure containing raw data, quality reports in JSON format, and scripts for fetching articles via the WordPress API. The demonstration also explores performance differences between CPU and GPU environments, noting that GPU provisioning significantly speeds up the analysis of faithfulness and accuracy. Furthermore, the tool allows for benchmarking and visualization of results, enabling users to compare good versus bad examples and understand specific issues within a draft. Documentation within the repository guides users through prerequisites, deployment on platforms like Local OpenShift, and step-by-step instructions for integrating these tools into their workflow without needing to build everything from scratch.
In conclusion, this AI Editorial Assistant serves as a robust foundation for improving content quality within the Fedora ecosystem while feeding into larger RPM packaging projects related to Retrieval-Augmented Generation (RAG). The presenter emphasizes that while the tool provides automated checks and recommendations, it is intended to support rather than replace human judgment, ensuring that the final published material meets the high standards of the community. By offering clear feedback on structural elements like images and headers, the system empowers contributors to refine their work efficiently. The video ends with acknowledgments to mentors and the broader Fedora community for making this internship opportunity possible, highlighting the collaborative nature of developing such open-source tools.
Read the full video transcript
So we will do a demo uh seeing the tool
in action like on real drafts. Um so at
first we will just try to launch the we
would paste in a draft expect to get
like a grounded review. So the first
demo is going to be about a good
article. So uh the tool in action should
load a draft that meets like uh sorry we
will try to load a draft that actually
meets standards. We have examples of
that and then the model
probably will have to confirm it's
compliant and then it sites
um so we have multiple paths right you
can actually do it using the streaml app
or even doing it like using the terminal
path uh which I showed earlier in in in
an earlier video like we I'm also going
to show it here uh we can actually try
to switch publications like it can
either be like the magazine or like the
com blog and then it's actually are
reproducible. So you don't need to like
to build anything. You can just pull
like the the OA images from Kore
directly and run. Now there is an honest
move uh that we would like to make here
is that be the final person to sort of
confirm it but at least the tool will
flag some missing features that you need
in order for your draft article to meet
the guideline and then you'll submit
something that's best for publication.
Uh so the next phase will be uh like
because this project was the foundation
on which we built or like we are sort of
feeding the larger RPM packaging uh rag
project which is the main project for my
internship. Um and of course for that I
would like to thank my mentors uh the
feder community uh at large internship
program for this outstanding
opportunity. So now let's get into the
real work.
when you download or like clone a copy
of I mean of the repository and then you
will see this app.py pipe which is the
file that we have here
and when you launch it it will open um
the GUI like the graphical user
interface you have here um you have the
publication type it can either be like
are you trying to test like a draft
article against uh the Fed community
blog like guidelines or is it against
the Feder magazine model that actually
exists right in in this case for now we
only have these two models and the
recommended one is like GMA 4 E4D
and then you can also choose the review
mode. Do you want to review like
entirely like the full editor review or
do you just want to review stuff before
this one um actually ends?
when we uh come back to um I said
I mean three possible ways of using this
which is the recommended one for
everybody uh like CLI like the command
line interface and then run like run
that is already there I give you the
model right wait so now it will be
waiting for a prompt here that it will
use and then it analyzed and then it
gave
the pillars of the tone, what you have
to avoid and so on and so forth until
the end. So you could do it using this
one if you prefer and if you have the
knowledge of it rather than using the
GUI, right?
You could take an article, paste it like
possibility. Uh you could also launch a
notebook, a Jupyter notebook uh that
shows you everything on local host
locally on your machine. And then here
if you run this you will see uh here set
up an imports try to run the quality
report analysis to know how how many of
those articles that you f either in like
the account blog or like magazine uh
worth or like keeping uh for your lag
enhancement and then it will give you
that output.
Then after that okay there is drinking
strategy experiment because like I said
before we for like memory outage and
then after that you can now do the right
query testing doing this one benchmark
results visualization. So here you can
even like sort of visualize the results
reference articles you would find like
some good articles and then like two bad
articles and then you can now try to
test them out and see what it what what
it like. Let's go back to the other one
uh and see hopefully if it's over. Okay,
it's still running. So
when you go back to the repository here
you tells uh in this
docs folder about first of all this guy
that MD give like what are the things
that you have to set up like the pre
requisites and so on and then what is
the contribution
like you have
like a quick uh a quick start and what
are documentation files that actually
help you understand this project as a
whole and sort
reproduce it. Tool was happy. We gave it
this um
uh we gave it this uh to be a good one
by my mentors and then of course the
tool actually told us that okay this is
ready to it wasn't able
to detect the feature image. Why?
because we copied the text content
rather than taking the source file or
like uh maybe because if you do a the
image tag like the HTML image like you
know tag and then it will okay there was
an image there was like a featured image
and stuff like that. So the reason why
it was not able to find the headers is
passed. Okay, talking about some some uh
tags and all like let's see now what's
the overall uh recommendation. So it
says it's ready to publish or need some
revision. So the article is too brief
and like some views and then you can go
ahead and try to download the review as
a txt file and see the recommendations
and try to adjust your draft article
like that before submitting submitting
it again. So, what I'm what I'm going to
do is uh I will simply copy this one.
Go back here. Copy this and go back to
the tool. Just get rid of this.
Paste it and run this one again. Finish.
It's going to be uh running the second
one here. Reviewing with JMA. So, in the
meantime, I can go ahead like I said
download to actually improve. So, which
is um a good way of having a feedback on
like a draft article that you actually
submitted. Um
review this article.
Okay, maybe I mistakenly stopped it. So,
we just have to Okay, it's still running
here. You can stop it or you can just
let it be until it's over. So here uh
select reviewing
and it would take some time. Um yeah
okay we spoke about this exploration uh
so we are going to go ahead and try to
shut it down. Yeah shut down this
Jupyter server. Yes.
All right it's down. So you see that we
can use like the Jupyter server like the
notebook or we can use it on like the
CLI environment to do it or like the
most preferred way the most recommended
way for anybody for anyone in the
community uh is the like the gooey like
the app finishes so running.
Um
there's a repo.
So maybe um here is what we actually got
when we um okay fetching the article and
then um like thinking of deploying it
using local open shift or CRC
internal to like Fedora and Red Hat. Um
so we were sort of preparing for that.
And then like I said we also have the uh
GPU environment that was provisioned
which is very much
better uh as opposed to running like the
tool or like doing everything in a CPU
environment work uh and checks it
against like faithfulness and accuracy
and other like uh criteria. So for the
repository structure maybe we will just
get back to this before and you have the
data folder in which you have like a
child folder or job directory of raw
data
and then the guidelines
right
like for magazine articles too uh and
that is also like the reports directory
uh in which you find like the quality
report JSON uh because when you fetch
all of those articles
some of them. So you need to run a
quality report to get to know those and
then you have uh the review that's saved
model review outputs.
good versus bad examples. And then you
have also the script directory in which
you have the script helps you fetch
articles using the WordPress API and
then you can uh the the stage corpus uh
what helps like sort of clean the data
set like the quality report generator.
Then the model benchmarking
review a good example and the model
directory in which we do have the
Ramadana configuration like
directory in which we have the details
about that and then the notebook the
quick store um that helps you sort of
use the tool core images no pipeline no
build or like set up from and then the
guide that is a complete step-by-step
like instructions uh guide for you to
the local open shift like deployment
guide and we also talk about like
community shift at some point I just
check this out oh okay I'll be right um
and then like the step by step also like
bastion login and then anible like we
were just sort of exploring platforms
like these tools for for the system that
we're building
um so yeah you'll see all of these and
then the requirement before instead of
like installing them one by one. Yeah,
it's actually timed out after 300
seconds. That's because I'm running it
on my laptop. It's CPU. It's on GPU. And
then because I run the first one that
was successful and this one just output
this one a bad article and then we know
the like the issues in that B article
and then we expect to return like to
flag the same problems or the specific
problems and then name or give
recommendations on um like how to fix it
to meet all of the guidelines. Thank you
so much for listening, watching