George Hotz: 3 Problems of Autonomous Driving: Static, Dynamic, Counterfactual | AI Podcast Clips

Duration: 5:34 955 transcript words EN summary

Video summary

In this discussion on autonomous driving challenges, George Hotz outlines a framework dividing the problem into three distinct categories: static, dynamic, and counterfactual. He notes that while companies like Cruise Waymo, Aurora, and Zoox utilize similar codebases derived from DARPA Urban Challenge technology, there remains significant room for breakthrough innovation in areas such as high-definition mapping under all weather conditions or advanced simulation environments. Hotz emphasizes that the first category, known as the static driving problem, involves scenarios where a vehicle is effectively alone on the road. This challenge can theoretically be solved with 100% accuracy using precise HD maps and localization systems, which allows for deterministic planning similar to how tractors are statically scheduled in farms to avoid collisions without needing real-time reaction from other agents. However, Hotz clarifies that solving the static problem is far more difficult than it appears when considering real-world conditions like lane drifting or deteriorating road markings. He points out that current systems often fail because they cannot maintain perfect localization across varying weather and surface states; for instance, Tesla's tendency to drift out of its lane represents a failure in this fundamental static domain. While he acknowledges that building high-quality localizers using LiDAR can achieve centimeter-level precision over vast distances like 100,000 miles with fallback conditions, the goal is to reach a reliability level akin to Özil D where the vehicle never operates without knowing its exact position. This ensures that even if minor errors occur, they do not lead to catastrophic failures or situations where the car is significantly off-course and unaware of it. The second category addresses the dynamic driving problem, which arises when other agents are present on the road, such as a car stopped at a red light or moving traffic. Unlike static objects defined in maps, these entities must be detected in real-time because their presence and behavior are uncertain; one cannot predict whether an object will remain stationary or move based solely on pre-existing data. Solving this requires robust models of human behavior to anticipate how other drivers might act, including the possibility that they may step out of line with expected patterns. This layer adds significant complexity compared to static navigation because it demands continuous perception and prediction capabilities rather than just precise map matching. The third and most difficult problem is identified as the counterfactual challenge, a concept Hotz associates with Judea Pearl's work on causal inference. In this scenario, autonomous vehicles must understand not only what other agents are doing but also how their own actions influence those agents' behaviors in ways that were previously unobserved or hypothetical. Since most current traffic consists of human drivers who do not follow programmed rules perfectly, the only viable method to address counterfactuals is through reinforcement learning on a massive scale involving all vehicles simultaneously. Hotz envisions a future where these systems can explore maximum exploitation strategies once fully deployed, allowing them to learn from and adapt to the complex, unpredictable interactions of human traffic in ways that static maps or simple dynamic models cannot achieve alone.

Read the full video transcript

[Music] so the way you leapfrog right is you come up with an idea or you take a direction perhaps secretly that the other people aren't taking and so cruise way mo even Aurora no Aurora Zuke's is the same stack as well they're all the same codebase even and they're all the same DARPA urban challenge codebase it's so the question is do you think there's a room for brilliance and innovation there that will change everything like say okay so I'll give you examples it could be if revolution and mapping for example that allow you to map things do HD maps of the whole world all weather conditions somehow really well or revolution is simulation to where the the what you said before becomes incorrect that kind of thing I knew room for breakthrough innovation um what I said before about oh they actually get the whole thing well I'll say this about we divide driving into three problems and I actually haven't solved the third yet but I haven't idea how to do it so there's the static the static driving problem is assuming you are the only car on the road right right and this problem can be solved with 100% with mapping and localization this is why farms work the way they do if all you have to deal with is the static problem and you can statically scheduled your machines right it's the same as like statically scheduling processes you can statically scheduled your tractors to never hit each other on their paths all right because they're you know the speed they go at so so that's the static driving problem Maps only helps you with the static driving problem yeah the question about static driving yeah you just made it sound like it's really easy it's real easy how easy how well because the whole drifting out of lane when when Tesla drifts out of lane is failing on the fundamental static driving problem Tesla is drifting out of lane the static driving problem is not easy for the world the static driving problem is easy for one route and one route in one weather condition with one state of lane markings and like no deterioration no cracks in the road I'm assuming you have a perfect localizer so that's all for the weather condition and me the lane marking condition that's the problem is how could you how do you have a perfect you can build perfect localizers are not that hard to build okay come on now with with wood lighter why don't ya wood lighter okay why don't ya but you use lighter right like use lidar build a perfect localizer building a perfect localizer without lidar it's gonna be it's gonna be hard you can get ten centimeters without liner you can get one centimeter with light our main concern about the one or ten centimeter I'm concerned every once in a while you're just way off yeah so this is why you have to carefully make sure you're always tracking your position you want to use lidar camera fusion but you can get the reliability of that system up to a hundred thousand miles and then you write some fallback condition where it's not that bad if you're way off right I think that you can get it to the point it's like özil D that you're you're never in a case where you're way off and you don't know it yeah okay so this is brilliant so that's the static static we can especially with lidar and good HD maps you can solve that problem easy no you just the static static I'm so he's very difficult for you to say something's easy I got it it's not as challenging as the other ones okay well it's it's okay maybe it's obvious how to solve it the third ones the hardest oh where do we get and a lot of people don't even think about the third one huh and you can see it as different for the second one so the second one is dynamic the second one is like say there's a an obvious examples like a car stopped at a red light right you can't have that car in your map yeah because you don't know whether that car is gonna be there or not so you have to detect that car in real time and then you have to you know do the appropriate action right also that car is not a fixed object that car may move and you have to predict with that car will dim alright so this is the dynamic problem yeah so you have to deal with this um this involves again like you're gonna need models of other people's behavior do you are you including in that I don't to step on on the third one but are you including in that your influence and people ah that's the third okay that's the boom we call it the counterfactual yeah I believe that I just talked to Judea pearl who's obsessed with counterfactuals oh yeah yeah so the static and the dynamic yeah our approach right now for lateral will scale completely to the static a dynamic the counterfactual the only way I have to do it yet they don't give you thing that I want to do once we have all these cars is I want to do reinforcement learning on the world I'm always gonna turn the exploiter up to max I'm not gonna have them explore but the only real way to get at the counterfactual is to do reinforcement learning because the other agents are humans you