AI-Powered Content Intelligence: Multi-Modal Analysis for Drupal Media Management

All right. Um, hi. Hello all. Um, I'm Satish. I'm working as a staff engineer at uh, a company called Zuks. Um, Zuks is acquired by Amazon. Uh, at Zuks, we work um, on the technology to enable driverless cars. Um, if you're interested, um, the cars are we we have launched the cars in San Francisco. you can install the app called Zuks and then explore uh the driverless cars. Um and uh on the website I'm a multimedia expert. Uh I work on video, image and audio processing and also I'm a EI engine. Uh I closely work with the multimedia teams uh to um because for the cars I work on the sensors like cameras and lighters. uh I write the software to capture all these data and process them and making sure that the machine learning teams are uh happy with that their data to make the decisions. Uh that's how I'm I'm closely working with the multime media teams and um I when I'm going through the some of the work that Drupal has been doing uh it's it's tremendous um I thought the content intelligence uh artificial intelligence powered a powered content intelligence um would be really um keep the Drupal at the next level. Um that's uh that's what I I believe in and um uh I'm happy that the community is is also in the thinking in the same direction and it's been a great work is going on in the community and today's uh agenda um what I'm going through is I'm going to talk about what is content what I mean by the content and uh what is content intelligence and how it is going to be very useful in creating the websites. uh as fast as possible. Um and then uh what uh Drupal's uh like how does the content content intelligence going to be useful for the Drupal Drupal foundation like Drupal CMS uh platforms and uh coming to the content when I say content it's uh video image and audio files that has been uh widely used by the people uh day by day it it has been like uh explosively increasing the amount of multimedia data that has been used in creating the websites. Um, and it is it is a huge challenge for the uh for any platforms to manage such a huge humongous content. Um, and then making the content more accessible to the people who are building the websites. Uh, that has been a key uh key thing here. Um well in in case of challenges right uh what's happening is let's say we have a company logo that's an image um let's say 10 people are building the websites with the same logo right so it is it is not recommended to keep 10 different copies in the system it will wastage the memory right so uh that is one of the challenges as as like the multimedia content is growing day by And discovery of the content is one of the very important. Let's say somebody is building the website and they want to grab some of the images or related videos, related audio to to add to their websites to build their websites. So we have to design the system the platform so that people easily access the content related content what they are exactly looking for. uh that's that's one of the challenge as the content uh size is increasing number of content is increasing and and also meeting the accessibility SEO um search engine optimization gaps. So we have to me make sure that the content is meeting uh the SEO um standards which means the uh the alternate text let's say the image there is some image and video files um we we should we should be able to pass the content and um add what exactly happening the context context of the content as as part of the metadata to each file so that it is easily accessible for the website builders. Um and yeah and and the rankings rankings are very important. Let's say somebody is building website and they want to search for some video and we have like 50 videos. Um how do we keep them in a series right? What videos we are going to keep as the first and second in the category right? So that's when that's where the rankings will come. So uh there is something called the the relevance relevance uh index that is a models will go through the content and then add this relevance uh score uh with the with each uh media data so that we can rank these files and then show in a in a list. Uh then then it should be more relevant to the uh person who is visiting the website. Then they can they can fast they can fastly build the the uh complete work. And to do all of these things only one solution is uh we should have smarter and automated context workflows uh sorry content workflows which means that let's say somebody is is uploading some files. Um there should be a system which goes through those multimedia content uh like image or video and audio and it has to be automatically uh understand that content pass through the content and understand and add the the um the concept of that content as a metadata and we have to tag that metadata. If this flow can be automated and and stored in the system uh that can be uh that will be like more reusable for everybody to to access the content. Um that's about the content and the content intelligence. This is this is a a huge shift between passive media storage and uh the active content understanding. This this is two different very important things. Um most of the systems today of course they adapting to AI but still so much of work is there to adapt to the AI. Lot of systems where you're storing the multimedia content in a passive format. Let's say there is one two different images are there but same content but if the file name is different we'll store it as two different files. But in case of active content understanding, let's say we we watched a YouTube video, a 2-minute video. We understand who is the speaker and what the speaker is talking about and what is the topic they're talking about. All those things. So at the end of the video, we gain some knowledge. Let's say there is a system a content library uh has all these kind of capabilities and and uh we we upload the files and this content library is going through all the media content and gain all these capabilities and understanding about the multimedia content and then adding this as a metadata tagging with each file. That would be the uh that that we would call it as the content intelligence and that is going to be um uh a phenomenal shift for the people uh who is building the websites in a fast manner and um at the same time uh let's say there is an image uh there are some AI tools like OpenCV all such kind of tools it can detect the objects it can detect the human pose it can uh uh give the context text about like uh if a person is running so it will give you uh it it will understand and add the track to the image. So you can search like okay I want to see who are people who are jogging. So you can see all the kind of images. So this is all possible with AI tools by adding the by by AI tools analyzing this content and then adding the metadata to the content and uh the discoverable is uh discoverability is is one of the very important thing uh there is a natural language uh processing um let's say there is a content library which goes through the video and understands what exactly happening and then there is NLP processing which goes through this text and extracts the keywords and then understand the content and the topic attached to the uh video file so that we can easily um search related videos and then uh get the work done quickly and accessibility first the all text let's say for the people who are disabled who cannot see the video they can play the video and then they can understand what exactly happening in the video uh these all text will explain uh what is happening in the video let's say there is an image it will speak the text uh can be played and then it will explain what is happening in the image. So such kind of things uh can be done and accessibility first is designing a system keeping the accessibility first in in in mind so that uh the things can be uh developed quicker um for the website developers and that's about content intelligence and any questions uh please stop me if you want to discuss anything deeper and multimodel AI sensors there are many tools in the artificial intelligence. There are many modalities like computer vision, computer vision libraries, OpenCV uh libraries are there. Uh they are featurerich libraries. Uh they can detect the objects, faces, logos and uh text scenes all those things and they can also give us the description about that image. Uh what's happening in that image and also and so there is something called smart cropping and focal point. um it can it can identify what is the important portion of the image. Let's say you're opening a website in a in a big display. You can see complete image. Let's say you're opening the website in a mobile phone. You don't have that much space but still you want to see the image content. Then the smart cropping will be very useful where it will detect the important portion of the image and then crop and show only that portion of the image in the smaller displays. So such kind of things uh are very such kind of tools are very important in building the websites and of course natural processing it analyzes the transcripts extracts keywords and understands the topic what people are talking about in that particular um content multimedia content and the semantic relationship this is this is very important let's say somebody is building a website to host master degree uh tutorial and the person is checking for the documents and let's say there are more documents regarding postgraduation. So how do you relate these two technically these two are master degree and postgraduation. So you have to understand the context and then grab all these files and show in the search results. So that's where these semantic uh relationships will come into picture and there are many a models that that are capable of doing all these things and u it is important to understand uh and identify such kind of models and use for the uh uh the platform build and um audio AI recognizes voices it understands the speakers and it understands the the emotion under the uh uh emotion of the speaker and let's say somebody is trying to post some violent video uh that has the violence. So it is very important for us to understand the emotion of the uh audio uh and then if if that is creating some uh something undesirable we should be able to block or we can bring in human uh into the into the loop and then ask hey this video is good for the uploading or not. So such kind of decisions uh where these kind of tools uh will be very useful and crossmodel fusion uh now we have this open CV NLP and audio. So we can combine the output of all these models and then uh generate a meaningful context metadata and then attach with the metadata. This is the next level of uh uh creating the metadata and and we can track this kind of metadata with the files. So this is called cross model fusion and why Drupal foundation of course Drupal Drupal foundation is a great one a great content management system that has been widely used in the world and uh this is the platform where I feel like it has to take the advantage of artificial intelligence uh and then uh uh integrate all the um uh services EI related services into the platform. from so that website builders can can get the benefit out of it. Um the the main important reason is the the media architecture in the Drupal it has been u designed in a modular way like the media entity media types media libraries such kind of low-level basic blocks are well designed uh for example media entity means what is the file um uh we can add some fields to the file uh there are many many aspects of the file which we are calling it as a media entity and then media type is is it a image or video audio such kind of details and and supporting libraries underlying libraries. So these are all the things have been designed very well where uh we can integrate the AI services um as a as a microservices uh because of this API first approach uh has been very uh very welcoming to integrate the AI services easily into the uh Drupal foundation systems and entity relationships. uh as I mentioned we need to create uh we should be able to create the the relationship between different media files so that we can uh quickly access the related files in the search results and uh human in loop um of course we cannot do everything uh relying on AI we have to make sure that human in the loop and making sure that the the AI is doing what it has to do uh so that uh it is we safely enabling MBA into the systems and enterprise scalability. It has huge scope for the enterprise scalability. Uh there are many many companies are adapting. Uh so uh the the basic system is well designed so that more more models can be integrated and can get the benefit out of it. And this is about again image image tagging. uh we can detect the if the if the what is kind of image it is a car or it is a what what exactly happening in the image and I was mentioning about smart cropping this is the example of the smart cropping let's say there is a there is a big image where uh the person is sitting in the corner and rest of rest of this is just trees right see if we have smaller display then the smart cropping can understand the image where the important portions are there in the image and it can crop and it can show the the uh very relevant parts on the display and the focal points is it can the openc libraries can understand the important portions of the image where we can enhance enhance it and then the the display on the websites will be even more better. So those are called focal points and video intelligence there is a automatic transcription where uh the models understands what speaker is talking and then transcripts will be generated and uh then the topic understanding NLP will understand the text uh and then uh it will give you the topic what what kind of topics the people are discussing and then content summarization uh thereby we can sum we can have the summary of the content and then we can make it uh more searchable content. That's that's where the video intelligence and the uh natural language processing libraries are being used and audio like speaker recognition and then again topic detection sentiment analysis if they uh what is the sentiment uh has been uh let's say there is a podcast uh what is the topic people have discussed and what is the sentiment uh what is the summary of the content so those kind of things are very very useful for the discoverability purpose. And of course this has very uh there are many technical challenges to integrate the AI. Um it has to be done in a in a uh constructive way. Um if not uh GPU resources are that's very huge uh it can es escalate the the cost of the uh G video inference or multimedia inference processing. So these these are some of the things we have to take into consideration. There are there are couple of challenges here and also privacy compliance. Let's say there are some documents the a content library is processing. Uh it should not capture the personal details like email photos those kind of things those are very very important to individual. So it has to capture only the relevant content and and then neglect the rest of the personal details. So those things are very important. Uh we have to keep the user consent management uh in place for it. and um editorial trust uh we have to always keep the person in loop human in loop uh initially when we are rolling out the EI make sure that we are getting the feedback from the human let's say there are 100 files at least for 10 files you get the feedback from human making sure that the is doing the right thing and then we should also get the opinion from the from the user like how they are so that feedback is also very important to make sure that we are uh taking the feedback and then making necessary changes like for example model fine-tuning uh we can get the models open source models and then we can also fine-tune for the healthcare applications for different different kind of applications and the performance uh integrating AI can can easily in sometimes can easily improve the latency let's say the some somebody is uploading a file the content library is going through the file and then takes some time So this is also we have to keep in mind uh making sure that uh it is um it is taking as less as possible. Uh the best we have to do and then making sure that the website builder uh the person who is building the website they have the right um right environment to construct the website. Um and yeah challenges are solvable but then uh take it take it in a proper way and and then uh plan planning is very very important in this case to solve all these problems and some of the best practices edge computing let's say EI model is running if we can split the task and and assign to different devices so that can be we can get the result faster so that the the it takes it will be less latency So that is edge computing. So make sure that edge computing in the in place and privacy preserving a as I mentioned it is very important to have the privacy preserving a and human in the loop and also performance um there are content to delivery networks. Let's say one person is building a website in new years. So there should be a server close to the that person so that they can upload the video and gets the uh result from the video and then they can they can make the progress fast. So that is that is on the CDN and implementation strategy is like small start small. We we cannot enable ya for everything at one go. Start with images and then see how the uh how the work is going on and then roll out for video and audio because video and audio are very complex uh in nature uh in terms of processing. Um and then a gradual roll out uh as I mentioned and user training um the capabilities and limitations educate educating the teams like how how exactly it works and continuous monitoring how how it has been doing we have to continuously take the feedback and see how this is working out and refine the models as I mentioned fine-tuning the models the models can be uh fine-tuned the models already built also can be fine- tuned for our own applications um which is called transfer for learning uh that is that is also very important in case of uh rolling out these models into the system. Those are some of the uh best practices and uh yeah finally the key takeaways is uh operational transformation moving from manual to the uh automated uh content processing. uh that's where the content intelligence is is more useful uh in in case of building the uh websites and uh compliance compliance is uh compliance and accessibility is more important that makes it even more faster. Um if the if the files are attached with the proper metadata and tagged with right right information then we can quickly access the uh multimedia content for building the websites and scalable media ecosystems. um uh these are like future ready multimedia ecosystems. Uh the moment we we enrolled all these uh AI models once we proved that it is working for for some kind of system it is more scalable uh because most of the multimedia content is is is similar so it can be easily scalable. Yeah, that's um that's all. Um any questions? So, so you basically, if I may summarize, you came up with a strategy on how to implement media uh moderation for Drupal sites, but uh are you planning to do any of this or yeah, I have been exploring into Drupal uh websites uh like building the websites, but this is something I found out that uh like content intelligence is more important uh like one of the uh very important aspect that keeps the Drupal foundation to the next level. Uh and and people can easily build their websites in less time. Uh this is one of the strategy that I was I I could think of. >> Okay. Very good. >> Thanks.