The Beginning of NavLab—2 : Early Days of Autonomous Vehicles at CMU
Watch on YouTubeVideo summary
The early days of autonomous driving research at Carnegie Mellon University marked a significant leap from slow, intermittent movements to continuous navigation capable of handling real-world challenges like finding obstacles and determining location even when wheels slipped. This capability laid the foundational groundwork for SLAM, or simultaneous localization and mapping, which allows vehicles to identify fixed points in their environment while moving and calculating how far they have traveled based on those relative positions. Around this pivotal time, Red Whitaker joined the institute from civil engineering with a passion for building rugged vehicles capable of operating outdoors in harsh conditions such as coal mines, rain, and snow. His creation, known as the Terrogator, taught the team valuable lessons about environmental variables; while obstacles like weeds or tree shadows made simple paths difficult to navigate due to changing light, the rough surfaces inside coal mines actually aided sonar systems by providing consistent reflections that allowed robots to accurately map their surroundings without getting lost in smooth hallways where signals would simply bounce back empty.
These practical experiences reinforced a critical philosophy: theoretical assumptions often fail when tested against reality, so researchers must venture into the real world rather than just theorizing indoors. Building on these successes and the ability of vehicles to move continuously every thirty seconds, DARPA initiated an ambitious ten-year program aimed at developing supercomputing capabilities with specific military applications for different branches of service, including task force management for the Navy, intelligent co-pilot systems for the Air Force involving speech understanding, and robot scout vehicles for the Army. The roadmap envisioned a progressive evolution where vehicles would start by moving slowly on roads and eventually advance to handling obstacles, navigating off-road terrain, bouncing through diverse landscapes, and interacting with other vehicles, representing a visionary plan that required substantial integration of various technological components.
To execute this vision, Carnegie Mellon bid on two specific projects: one focused on road following using cameras to detect lanes and another dedicated to integrating all the disparate systems into a cohesive whole. This integration project was crucial for ensuring seamless communication between different subsystems, such as allowing the road-following system to instruct the navigation unit, which in turn would guide the steering mechanism while coordinating with obstacle avoidance protocols. Although DARPA initially intended for software developers like Martin Marietta to handle all testing without visiting Denver frequently, they eventually agreed to provide a physical vehicle for development and validation purposes. This decision was instrumental because it allowed researchers to test their sophisticated algorithms on actual hardware rather than relying solely on simulations or remote delivery methods.
The acquisition of this dedicated test vehicle officially marked the beginning of NavLab, setting the stage for decades of innovation in autonomous driving technology. By combining the continuous movement capabilities developed during thesis work with Red Whitaker's rugged outdoor vehicles and DARPA's strategic funding goals, the team created a platform that could evolve from simple road following to complex off-road navigation. This collaborative effort bridged the gap between theoretical computer science and practical engineering challenges, proving that autonomous systems could not only perceive their environment but also adapt to it dynamically. The program established by DARPA provided the necessary resources and direction for researchers to push boundaries in speed, terrain versatility, and system integration, ultimately transforming how vehicles interact with complex environments like highways filled with obstacles or unpaved trails where traditional navigation methods would fail.
Read the full video transcript
So my thesis work
was taking
um what Hans had done 15 minutes to do a
move and speed it up so I could do 30
seconds to do a move. So I still wasn't
moving continuously but at least I could
move across uh a laboratory and even
take it outdoors and find obstacles and
not run over them and figure out where I
was. This was kind of the beginning of
SLAM, simultaneous localization and
mapping. If you can see some points in
the world and figure out where they are
and watch where they are relative to you
as you move, now you can tell how far
you've moved, even if your wheels are
slipping.
Around that time, another character
enters the story, a guy named Red
Whitaker.
Red was in civil engineering and had
been playing with robots.
And uh I went to Raj I said I think he
would be a good person to have in the
institute and Raj agreed and we found a
way to bring Red out of civil
engineering and into the institute. And
Red loved building big rugged vehicles.
So he built a vehicle called the
Terrogator which we could take outdoors
and take into a coal mine and take uh
through rain and snow. Uh, and we'd
learned a lot from the Terrain. Uh, one
of the things that we'd learned is that
problems that we thought would be easy
might be hard, like following a path up
through Shley Park because of weeds on
the path, because of shadows of trees,
because of changing illumination. But
things that we thought would be hard
might be easy. Coal mines turn out to be
a wonderful place to run a robot. If
you've got sonar on your robot and
you're running down a hallway and it's a
nice smooth hallway, the sonar hits the
hallway, it bounces and never comes
back.
There are no smooth surfaces on a coal
mine. It's this wonderful rugged rough
surface and so the sonar bounced back
and so the robot knew exactly where it
was and exactly where the walls were. So
that was another important lesson. Go
out in the real world and see what
happens out there. Don't just uh
theorize.
So on the basis of being able to move
once every 30 seconds,
DARPA then decided that um a guy named
Clint Kell at DARPA decided DARPA needed
to invent the supercomput.
Now DARPA can't just invent a computer
because they think computers are cool.
They have to have some sort of a
military story to go along with it. So
they said, um let's see. Um, we'll have
three stories. For the Navy, we'll do
task force management.
For the Air Force, we'll do an
intelligent co-pilot. So, that brings in
speech understanding and those kinds of
things. And for the Army, I know, let's
do a robot scout vehicle.
And they laid out a plan over 10 years
where the vehicle would be able to move
slowly and then faster and faster and
faster. would be able to move down roads
and then roads with obstacles and then
off-road and then bouncing through all
kinds of different uh terrain and would
be uh able to deal with other vehicles
etc etc. So it's a visionary uh program
and uh they asked us to bid on it. So we
bid on two things uh project called road
following. How do you see the road with
your cameras and go down the road? In an
integration project, how do you pull all
of the different pieces together? What
does the road following system say to
the navigation system say to the
steering system say to the obstacle
avoidance system? And we said, gosh, if
we're going to do that, we better have a
vehicle to test it on.
And they said, but you're supposed to be
delivering all of your software to
Martin Marietta, who's the integrating
contractor. He said, "Yes, yes, yes,
yes, yes. We'll de deliver all of the
software to them, and we're not going to
fly out to Denver every time we need to
test something." He said, "Okay, you can
have a vehicle." And that was the
beginning of NAVL.