From perception to action: World Foundation Models (WFMs) for autonomous robots and vehicles

Okay? Hello everyone. Uh I'm Abhishek. I work at KPIT as part of the autonomous driving team. And today I'm going to be talking about a change a recent change that has been happening in the autonomous driving space and autonomy in general. Uh so start. Yep. So, the topic of today I'm going to be talking about is autonomy 2.0. So, up until now uh autonomous driving stacks uh have had a a conventional sort of piece-by-piece architecture, whereas now that has been replaced that is being replaced by uh end-to-end AI models, right? So, to start off with I just have a simple prompt for everyone. I think every one of us knows prompts. Um that I want you to imagine a futuristic autonomously driving car, right? You think about it head. And can we define the objectives for such a car? What is an autonomous driving car is really supposed to do? Right? So, I can put this into a like very um simple context. One, go from point A to point B safely. And two, follow regulations. Right? So, it has to be as simple as that. Right? So, keeping this in mind, I want to first establish how the conventional or like the today's autonomous driving stack looks like. Right? So, you must have all seen the typical autonomous driving car where it it has all the different kinds of sensors mounted on top of it, different video sensors, radar, lidar, all of that. Right? So, all of that information is then passed on to first the perception stack, which establishes what objects are there in the scene, and then the localization, which establishes the 3D world around the vehicle. And based on that, uh certain planning uh rules are devised. Uh so, all of these acronyms are some or the other uh autonomous driving features. All of you must have heard about the AEB in the new Mahindra's, which is the auto emergency braking. ACC is the adaptive cruise control, lane keep assist, lane change assist, uh blind spot detection, traffic jam uh assist, and intelligent speed assist. All of these are like different features. Right? So, this follows the uh normal software development convention that a problem, a big problem, should be broken down into smaller and smaller pieces, and the pieces should be solved one by one, so that the when you put them all back together, the entire system makes sense and works reliably without failures. Right? So, but these, when applied to autonomous driving, uh pose a big problem. Right? And why does it do that? Because autonomous driving is a long-tail problem. Right? Um all of these autonomous driving stacks have been able to do almost 80% of the work fairly well. But they fail when they are not supposed to fail. Right? They fail when you are in an accident, they fail when there is uh a vehicle turned over and they're not able to brake. They fail in these very random and chaotic situations, which don't happen very uh regularly. Right? And situations that you cannot write rules for all the time. Right? So, the current approach of trying to break down into uh smaller and smaller problem statements, when applied to long tail, it leads to a game of whack-a-mole. You solve one problem, another one pops up. You solve that, third one pops up. Right? And this keeps on going. Right? And because of this problem, we've not been able to achieve the full autonomy. So, up until now, like 10 years ago, people assumed that yeah, we'll solve autonomous driving like whatever by 2025 or like things like that. Timelines like that. But that has not been the case because of this approach, because this long tail problem is too difficult to solve with without any generalization built into the stack. Right? So, imagine you're on an Indian road, for example. Uh all of these features are there in your like most advanced AD car. Right? So, you're driving. There's always someone at your blind spot, some two-wheeler guy trying to be in that one space, right? To avoid him, you slightly move towards the left. Um so, you uh move out of your lane. So, the lane keeper start start start spinning. The blind spot is already pinging. Now, there is like a lot of vehicles so that it the vehicle assumes that there's a traffic jam. So, that traffic jam assist also pings you. The intelligent speed assist pings up because technically, the speed limit is 30. We are doing 50. Who knows what is the right thing to do? So, intelligent speed assist also starts pinging you. Right? And in all of this, suddenly some guy comes in from the wrong side and your vehicle automatically brakes because it doesn't know what to do do with that. Right? So, the problem is there are just too many features and too many things, and I did not ask for this. I asked for a simple thing. Just get me from point A to point B and follow regulations. Why are you doing these 10 different things for me? Right? So, just to quote a uh just to quote a character from my favorite video game, we are deep in acronym hell right now. Right? There's just too many things happening and it's not coming together as it should. Right? And the next problem is the lack of generalization. Right? The entire stack needs customization and custom code to be written every time it has to be moved to a new geography. Right? That is one of the biggest reasons why autonomous driving as a whole has not moved beyond the global north. Right? Only now we are getting autonomous driving solutions in India. Even those are fairly basic, well-tested, and well-defined problems which are which they are solving. Not the general autonomous driving problem. Right? And that is because like there are different types of traffic governing objects in different geographies, different obstacles, different driving patterns. There are auto rickshaws in India, tuk-tuks in Thailand. All So, your object detection model or the perception stack is always going to give varying outputs in varying geographies. Right? So, the cycle time is high, cost is high. So, how do we solve that? Like, what is the replacement for this? So, a lot of the OEMs particularly some companies what they have done is just replace the entire stack with an AI model. Right? Of course, I'm putting it very simply in a simplistic fashion. But, the idea still remains the same that just you replace the entire stack with an AI model and then once your AI once your entire stack is just an AI model, you train with more and more data. Right? So, on paper this looks quite straightforward. You know, this is actually being used by a lot of organizations. They are the Tesla FSD that your cousin from Bay Area keeps bragging about. Right? That is end-to-end end The So, in recently at CES, uh Nvidea launched their open source stack for end to end AI. Um even in uh robotics, this is the same uh idea that is being used uh that the end to end AI with end to end AI, uh the robots will be able to learn on their own, right? So, this definitely is getting used, right? And so, what are the opportunities here? As I mentioned earlier, your entire stack is just an AI model now, right? So, you can train it at scale. You can generalize over a large range of scenarios. You can validate it virtually without the without even ever putting it onto any hardware or without the vehicle even driving a mile, right? It significantly reduces development costs, uh reduces uh porting to a porting to a virtual geography, uh the It has It provides you robustness against edge cases and long tail scenarios. Like um and it brings us closer to the original objective. I do not want to have 10 different things. I just want to get there where I want to get to and get there safely while following regulations, right? So, this uh approach is theoretically at least the closer approach to the uh starting objectives, right? But then, the main there is one problem with this. Um anybody want to guess what the problem would be here in this case? The problem is simple. You need data. You need a lot of data, right? You require geo-specific and this is all visual data to uh start with. Visual data which is geo-specific, thousand tens of thousands of hours in run time. Uh it has to fit something called as the operational design domain. So, this defines in which conditions you your vehicle should work, like the time of day, weather, geography, all these things. Right? Um so so this includes variations of different highway exits, all of this. Um right? So, and along with that it also needs to capture things which are out of distribution. Things that you cannot record, such as accidents, such as somebody running a stop sign. Right? All of these things cannot be orchestrated and recorded in physical world. And then finally it has to be high fidelity, at least 720p, as high as 4K in some of the cases that we've seen. Right? So, how do we generate this kind of data? Right? The generative AI models that we've seen today are really good at generating cat videos for some reason. But when you work for or work on generating data for autonomous driving part, there are significant challenges uh in um the diffusion back So, basically the models which have diffusion backbones uh right? Uh generative AI models, they have significant problems with that. So, what uh we are working on is essentially developing a world foundation model, particularly for autonomous driving domain, right? Um so, you can text uh take a text prompt, video, camera, point cloud. So, all of these are like the typical inputs that are available in autonomous driving. Um your video recordings are there, your lidar point cloud data is available, map data is available. And along with that we add a lot of context in the text prompt as well with respect to what environment and what time of day, weather, geography, all of these things. Uh scene details, traffic occupants, all of these details we add in the text prompt and we train the model to generate data specific to these kinds of conditions, right? And to generate data which can do scene generation, do uh uh planning output generation, control output generation, and map output generation as well, right? So, So, what So, the typical architecture of a world from this model So, this is something I've taken from Nvidia. Um but we also have our own in-house um model that we're developing. The idea is pretty simple. You try to put some or the other constraints onto the diffusion backbone. The diffusion backbone, as it is conditioned on text, is not good enough to generate realistic videos with the high amount of control that is required for autonomous driving. Right? So, you take additional domain-specific control inputs like 3D bounding boxes. Uh 3D bounding boxes are the boxes for cars, but when you're driving, you don't only perceive the vehicle as 2D, you also have depth perception. Similar thing has to be introduced when you are doing uh annotation for autonomous driving. So, that is why 3D uh map layouts. So, you need lane lines, road edges, uh traffic signals, or pedestrian crossings, all of that. And then additionally, if required, depth map and segmentation map of again the same imagery. Right? So, all of these control inputs uh can be provided in addition to your text prompt and then uh use uh that part to generate very specific kind of scenarios uh which can orchestrate a particular type of motion or scenario which you need to test your AD stack on or train your AD stack with. Right? So, these provide much higher control uh compared to the conventional diffusion models. They are They ensure physics-based realism because we are providing the 3D bounding box and map layout. So, they will put the lanes exactly where the lanes are supposed to be, vehicles where the 3D bounding boxes are, and they can be trained for geo-specific outputs, right? So, I will show a typical output. So, okay. Uh the GIF is running a little slowly, but essentially, if you see here, there is correlation between both the inputs and the outputs. There are 3D bounding boxes where of where the vehicles are, and for traffic signals as well, there are different These are the traffic signals. These are the lane lines, and these orange ones are the pedestrian crossings, right? So, this is typically how a scene is represented and the output is generated. Uh also follows that same layout, right? So, similarly, autonomous driving and like in general needs that doesn't just work on one view. It works on six views of data. So, the example that I that I showed in the previous screen, that is to be generated for six different views, right? So, this is something that we've done. We've trained a world function model for India data, and we've generated this So, this is like an example video which where we have generated like six view output, right? You can see the highway barriers look like the ones are in India, buses fairly similar, the overpass, everything. So, all of this geo-specific effect is coming in because we've trained this model with India data, right? Again, so the thing I was talking about earlier, you can change or create custom variations with respect to weather, time of day, even road conditions for that example. All of these are generated particularly based on like Indian scenarios, right? You will see potholes in Indian roads or you will see So, rainy time or fog essentially. So, all of that parameters are considered and all of these are based on the ODD that has been defined for the particular vehicle. Now, the advantage of having this is the OEM now doesn't need to go and record monsoon data in monsoon. Right? They can record the scenario in daytime clear daytime and then create these variations purely using a world condition model. Right? Or if they know that for this particular scenario their model is failing for some reason, they will want to validate it for all different times of days, the the weather conditions, and road conditions. So, again, orchestrating the exact scenario where your video fails is not possible again. Right? You cannot do that make that exact same thing happen again. Right? And to do that in different weather time of day again is becomes a challenge. So, you can do all of this using generated data. Right? Additionally, as I mentioned, there are certain scenarios which you just cannot create by just driving around. You cannot create or you cannot drive through a fire situation on the road or you know, any other scenarios. So, these things need to be generated using synthetic data generation methods. Okay? So, one more example I will show is So, this is the typical implementation that we do using simulators. So, the first row you will see is the bounding boxes for the vehicles. And the second row you will see is the road layout. Okay? So, you see here the first vehicle passes the ego vehicle and second vehicle is the one that the ego vehicle crashes into. If I want to generate this exact scenario, I can do that in a simulator first and then extract these particular intermediate representations of that scenario and pass that on to the world foundation model along with a text which defines just the geography. I can say this is like you know, North American intersection which is not signalized and there is this incident is happening. Once I provide that information along with this grounding, the model is able to generate very realistic looking outputs which show the first vehicle passing exactly the moment I wanted to, second vehicle coming in exactly at the moment I wanted it to and the crash happening exactly in the way I wanted it to. Right? So, this is the capability that world foundation models give us. Um so, if you have a world foundation model, if you are able to generate data, how do you measure its quality, right? So, there are certain video generation metrics that are available. There is fresh air, inception distance, video distance, there is brisk which is based on multiple things bringing coming together. But, all of these metrics again still lack autonomous driving specific information which is again coming back to the physics of it or real world effects. How do you measure how well that the model is generating a particular weather scenario or like what does heavy rain mean to the model and what does it mean to you? All of these kinds of aspects are still not have still not been considered or sort of coded into a metric as of now. Right? So, there is still some work that needs to be done on this side, but there are certain ways line things which we can use to establish uh certain uh quality which are like FID and FCD as I mentioned. Right? So, now that you have this uh huge amount of data, how do you validate your end-to-end AI stack, right? So, this is an example of an end-to-end AI stack consuming this six views of information and certain CAN information. CAN is the vehicle uh uh sort of the odometry which is the uh speed, acceleration, all of those things. And uh with that and with these two inputs, the end-to-end AI model is able to predict, first of all, the trajectories of all the actors in the scene. So, if if you see on this side, uh all the blue blue to green trajectories are for uh the surrounding vehicles and the orange trajectory is for the ego vehicle. And if you see here, this is the predicted trajectory that the vehicle should take. Uh that is being predicted for each and every frame. And whenever there is a vehicle that comes in, the trajectory automatically shows the requirement for braking. Right? So, this is how you can basically provide this data as input to the end-to-end AI model and calculate not only the uh trajectory, but certain metrics that go along with it. So, you can calculate like average displacement error between the ground road trajectory and your predicted trajectory. You can calculate road completion percentage, uh infraction penalty based on if your vehicle is running a red light, if it is running a stop sign, all of these things. Uh typically in So, there is a simulator called Carla. They have this leaderboard that they run for end-to-end AI models. So, they use the multiplication of uh RCP and IP as the driving score. There's also collision frequency per kilometer, lane invasion per kilometer, how frequently are you you know per kilometer for certain set of driving kilometers, how frequently are you going into a different lane automatically. There are certain comfort metrics as well which can be considered like the longitudinal jerk is in terms of moving forward or lateral acceleration in terms of moving sideways. Are these inconsistent or uh in or within the definition that is acceptable. All of these metrics can be generated and you can essentially uh build a complete uh flywheel using two components. One is the world foundation model. The other one is the end-to-end AI stack. Essentially, it starts with gathering data. Once you gather more and more data, you can use that data to train the world foundation model. Generate specific scenario data and uh train and validate your end-to-end AI stack using this data. And then identify the failure scenarios. Identify where the uh stack is failing. Identify where particularly you want more data. And then pass this information on. If needed, gather more data. Again, train the world foundation model and the cycle keeps going, right? And as many of these as many runs as you do of this cycle, the requirement of gathered data or physically recorded data progressively goes down. Right? Because you more of like more and more you don't need recorded data anymore. You have captured most of the visual perspectives that you want to capture. And whatever dynamic perspectives you want to bring in, you can always bring that in using a simulator or using a reference video. Right? So, the core behaviors and features are retained across geographies. Uh like weather conditions. All right? And the scene level features can be fine-tuned with local data. Right? So, if you want to do this again for a new geography, you don't need to gather as much amount of data as you did for the first time. You only need a certain set of data to generalize over the geography, generalize over certain objects that are present in this particular geography as compared to the base one. And on that, you can still again do the same cycle and get up and running quickly. Right? So, this essentially helps you build a comprehensive training and validation suite for these end-to-end AI models. Right? And this is typically how your Teslas of the world are able to rapidly generalize to new geographies as well. Right? So, to summarize, Autonomy 2.0 is here. It's being fueled by end-to-end AI and world foundation models. The end-to-end AI stack has the ability to generalize over a wide range of highly variable driving environments. And synthetic data and world foundation models are acting as the fuel to this end-to-end engine. And finally, the metrics are still evolving. This is still as a whole, the stack is still evolving. And there is lot more work to be done in this space going forward. Right? As an example, I would like to show some open problems. So, first one I think I've already mentioned, how can we ensure the fidelity of the output videos So, while increasing the generation length, so essentially, there is only a certain length of video that you can generate in one inference. Typically, that can go as as high as 100, 120 frames, maybe. When you do an autoregressive implementation, the system breaks down after maybe two or three implementations, and the maximum you can generate are like roughly 500 frames. How can you go beyond that limit? Right? Uh Then how do you manage generation costs? Because these are visual generative AI models and you have to run them for thousands of scenarios for hundreds of variations. How can you realistically manage this cost when you're actually running this on high GPUs, high compute GPUs, right? Then generating data for other modalities like lidar or radar. This is still an open problem. Um then ensuring the eligibility of small objects. Generative AI is not great on you know uh generating very small objects. Like for example, if I'm driving on a road and there there's a traffic sign that I'm supposed to see and from a certain distance, the model will not be able to generate that very small traffic traffic sign, right? And it will only become clearer when I'm very close to the object or it won't even show up. Right? So there are certain problems like that which are still there and that has that have to be solved. Right? So this has been my uh uh sort of coverage on these open end-to-end AI and world foundation models. Uh and I'm open for questions. Thank you. In the data privacy slide, you talked about identifying the failure. So is that failure also in simulation or is it then is the model then deployed again on the roads and then So the deployment on the road has to happen at a stage where the maturity is well validated. Right? Um but the flywheel still keep will keep on going after deployment has also happened. That is how FSD also keeps on uh uh updating their own stack. Uh even if they have deployed a certain version of stack, that will keep on updating over certain periods of periods of time where they've identified uh certain problems and then fix them and updated them. So, you can actually do an OTA of your entire autonomous driving stack now because it's just an AI model, right? So, that's how it's typically handled.