Chapter 14

IN THIS CHAPTER

Seeing the path to self-driving car autonomy

Imagining the future in a world of self-driving cars

Understanding the sense-plan-act cycle

Discovering, using, and combining different sensors

A self-driving car (SD car) is an autonomous vehicle, which is a vehicle that can drive by itself from a starting point to a destination without human intervention. Autonomy implies not simply having some tasks automated (such as Automated Parking Assist, demonstrated in the video “Automatic Parking Assist | GM Fleet” at YouTube.com), but also being able to perform the right steps to achieve objectives independently. An SD car performs all required tasks on its own, with a human potentially there to observe (and do nothing else unless something completely unexpected happens). Because SD cars have been part of history for more than 100 years (yes, incredible as that might seem), this chapter begins with a short history of SD cars.

For a technology to succeed, it must provide a benefit that people see as necessary and not as easily obtained using other methods. That’s why SD cars are so exciting. They offer many things of value, other than just driving. The next section of the chapter tells you how SD cars will change mobility in significant ways and helps you understand why this is such a compelling technology.

When SD cars become a bit more common and the world comes to accept them as just a part of everyday life, they will continue to affect society. The next part of the chapter helps you understand issues surrounding acceptance and the common use of SD cars, and considers why these issues are important. It answers the question of what it will be like to get into an SD car and assume that the car will get you from one place to another without problems.

Finally, SD cars require many sensor types to perform their task. Yes, in some respects you could group these sensors into those that see, hear, and touch, but that would be an oversimplification. The final section of the chapter helps you understand how the various SD car sensors function and what they contribute to the SD car as a whole.

Getting a Short History

Developing cars that can drive by themselves has long been part of the futuristic vision provided by sci-fi narrative and film since early experiments in the 1920s with radio-operated cars. You can read more about the long, fascinating history of autonomous cars in “We’ve had driverless cars for almost a hundred years” at qz.com. The problem with these early vehicles is that they weren’t practical; someone had to follow behind them to guide them using a radio controller. Consequently, even though the dream of SD cars has been cultivated for so long, the present projects have little to share with the past other than the vision of autonomy.

The modern SD cars are deeply entrenched in projects that started in the 1980s (see “In the 1980s, the Self-Driving Van Was Born” at MIT Technology Review.com — which may require a subscription). These newer efforts leverage AI to remove the need for radio control found in earlier projects. Many universities and the military (especially the U.S. Army) fund these efforts. At one time, the goal was to win at the DARPA Grand Challenge, which ended in 2007. However, now the military and commercial concerns provide plenty of incentive for engineers and developers to continue moving forward.

The turning point in the challenge was the creation of the autonomous car called Stanley, designed by scientist and entrepreneur Sebastian Thrun and his team. They won the 2005 DARPA Grand Challenge (see the video “DARPA Grand Challenge - Stanley Wins” at YouTube.com). After the victory, Thrun started the development of SD cars at Google. Today you can see the Stanley on exhibit in the Smithsonian Institution’s National Museum of American History.

The military isn’t the only one pushing for autonomous vehicles. For a long time, the automotive industry suffered from overproduction because it could produce more cars than required by market demand (though the realities of Covid-19 intervened). Market demand can go down or up as a result of all sorts of pressures, such as car longevity. In the 1930s, car longevity averaged 6.75 years, but cars today average 10.8 or more years and allow drivers to drive 250,000 or more miles. Although circumstances changed at least temporarily during the coronavirus pandemic, the decrease in sales led some makers to exit the industry or fuse together and form larger companies. SD cars are the silver bullet for the industry, offering a way to favorably reshape market demand and convince consumers to upgrade. This necessary technology will result in an increase in the production of a large number of new vehicles.

Understanding the Future of Mobility

SD cars aren’t a disruptive invention simply because they’ll radically change how people perceive cars, but also because their introduction will have a significant impact on society, economics, and urbanization. At present, no SD cars are on the road yet — only prototypes. (You may think that SD cars are already a commercial reality, but the truth is that they’re all prototypes. Look, for example, at “Uber May Be Aflame, but Its Self-Driving Cars Are Getting Good” at Wired.com and you see phrases such as pilot projects used, which you should translate to mean prototypes that aren’t ready for prime time.) Many people believe that SD car introduction will require at least another decade, and replacing all the existing car stock with SD cars will take significantly longer. The many articles on driverless cars at The Conversation.com will help you track SD car progress (or regression in some cases). However, even if SD cars are still in the future, you can clearly expect great things from them, as described in the following sections.

Climbing the six levels of autonomy

Foretelling the shape of things to come isn’t possible, but many people have at least speculated on the characteristics of SD cars. For clarity, Society of Automotive Engineers (SAE) International (http://www.sae.org/), an automotive standardization body, published a classification standard for autonomous cars (see the J3016 standard at https://tinyurl.com/2vpnxctt). Having a standard creates car automation milestones. Here are the five levels of autonomy specified by the SAE standard:

· Level 1 – driver assistance: Control is still in the hands of the driver, yet the car can perform simple support activities such as controlling the speed. This level of automation includes cruise control, when you set your car to go a certain speed, the stability control, and precharged brakes.

· Level 2 – partial automation: The car can act more often in lieu of the driver, dealing with acceleration, breaking, and steering if required. The driver’s responsibility is to remain alert and maintain control of the car. A partial automation example is the automatic braking that certain car models execute if they spot a possible collision ahead (a pedestrian crossing the road or another car suddenly stopping). Other examples are adaptive cruise control (which doesn’t just control car speed, but also adapts speed to situations such when a car is in front of you), and lane centering. This level has been available on commercial cars since 2013.

· Level 3 – conditional automation: Most automakers are working on this level as of the writing of this book. Conditional automation means that a car can drive by itself in certain contexts (for instance, only on highways or on unidirectional roads), under speed limits, and under vigilant human control. The automation could prompt the human to resume driving control. One example of this level of automation is recent car models that drive themselves when on a highway and automatically brake when traffic slows because of jams (or gridlock).

· Level 4 – high automation: The car performs all the driving tasks (steering, throttle, and brake) and monitors any changes in road conditions from departure to destination. This level of automation doesn’t require human intervention to operate, but it’s accessible only in certain locations and situations, so the driver must be available to take over as required. Vendors had originally expected to introduce this level of automation around 2020, but a quick read only will tell you that they’re still a long way to seeing this level as a reality.

· Level 5 – full automation: The car can drive from departure to destination with no human intervention, with a level of ability comparable or superior to a human driver. Level-5 automated cars won’t have a steering wheel. This level of automation is expected five or more years after Level 4 cars become a reality. You can read about what’s taking a while in “What’s Holding Back Fully Autonomous Driving?” at Thomasnet.com.

Even when SD cars achieve level-5 autonomy, you won’t see them roaming every road. Such cars are still far in the future, and there could be difficulties ahead. The “Overcoming Uncertainty of Perceptions” section, later in this chapter, discusses some of the obstacles that an AI will encounter when driving a car. The SD car won’t happen overnight; it’ll probably come about through a progressive mutation, starting with the gradual introduction of more and more automatic car models. Humans will keep holding the wheel for a long time. What you can expect to see is an AI that assists in both ordinary driving and dangerous conditions to make the driving experience safer. Even when vendors commercialize SD cars, replacing actual stock may take years. The process of revolutionizing road use in urban settings with SD cars may take 30 years.

This section contains a lot of dates, and some people are prone to thinking that any date appearing in a book must be precise. All sorts of things could happen to speed or retard adoption of SD cars. For example, the insurance industry is currently suspicious of SD cars because it is afraid that its motor insurance products will be dismissed in the future as the risk of having a car accident becomes rarer. (The McKinsey consulting firm predicts that SD cars will reduce accidents by 90 percent; see “Ten ways autonomous driving could redefine the automotive world” at McKinesey.com). Lobbying by the insurance industry could retard acceptance of SD cars. Also, consumers might put up some resistance because of lack of openness to the new technology (some consumers look for gradual product improvements, not for radical changes, as described in “Consumers Don't Really Want Self-Driving Cars, MIT Study Finds” at wbur.org). On the other hand, people who have suffered the loss of a loved one to an accident are likely to support anything that will reduce traffic accidents. They might be equally successful in speeding acceptance of SD cars. Consequently, given the vast number of ways in which social pressures change history, predicting a precise date for acceptance of SD cars isn’t possible.

Rethinking the role of cars in our lives

Mobility is inextricably tied to civilization. It’s not just the transportation of people and goods, but also ideas flowing around to and from distant places. When cars first hit the roads, few believed that they would soon replace horses and carriages. Yet, cars have many advantages over horses: They’re more practical to keep, offer faster speeds, and run longer distances. Cars also require more control and attention by humans, because horses are aware of the road and react when obstacles or possible collisions arise, but humans accept this requirement for obtaining greater mobility.

Today, car use molds both the urban fabric and economic life. Cars allow people to commute long distances from home to work each day (making suburban real estate development possible). Businesses easily send goods farther distances; cars create new businesses and jobs; and factory workers in the car industry have long since become the main actors in a new redistribution of riches. The car is the first real mass-market product, made by workers for other workers. When the car business flourishes, so do the communities that support it; when it perishes, catastrophe can ensue. Trains and airplanes are bound to predetermined journeys, whereas cars are not. Cars have opened and freed mobility on a large scale, revolutionizing, more than other long-range means of transportation, the daily life of people. As Henry Ford, the founder of the Ford Motor Company, stated, “cars freed common people from the limitations of their geography.”

As when cars first appeared, civilization is on the brink of a new revolution brought about by SD cars. When vendors introduce autonomous driving-level 5 and SD cars become mainstream, you can expect significant new emphasis on how humans design cities and suburbs, on economics, and on everyone’s lifestyle. There are obvious and less obvious ways that SD cars will change life. The most obvious and often noted ways are the following:

· Fewer accidents: Fewer accidents will occur because AI will respect road rules and conditions; it’s a smarter driver than humans are. Accident reduction will deeply affect the way vendors build cars, which are now more secure than in the past because of structural passive protections. In the future, given their absolute safety, SD cars could be lighter because of fewer protections than now. They may even be made of plastic. As a result, cars will consume fewer resources than today. In addition, the lowered accident rate will mean reduced insurance costs, creating a major impact on the insurance industry, which deals with the economics of accidents.

· Fewer jobs involving driving: Many driving jobs will disappear or require fewer workers. That will bring about cheaper transportation labor costs, thus making the transportation of goods and people even more accessible than now. It will also raise problems for finding new jobs for people. (In the United States alone, 3 million people are estimated to work in transportation.)

· More time: SD cars will help humans obtain more of the most precious things in life, such as time. SD cars won’t help people to go farther, but it will help them put the time they would have spent driving to use in other ways (because the AI will be driving). Moreover, even if traffic increases (because of smaller transportation costs and other factors), traffic will become smoother, with little or no traffic congestion. In addition, the transportation capacity of existing roads will increase. It may sound like a paradox, but this is the power of an AI when humans remain out of the picture, as illustrated by this video “The Simple Solution to Traffic” at YouTube.com.

· There are always opposing points of view when it comes to technology, and it’s important to maintain an open mind when hearing them. For example, instead of reducing traffic congestion, some people say that SD cars will actually increase traffic congestion because more people will opt to drive, rather than carpool, take a train, or rely on a bus. In addition, given the nature of people, you might see weird behavior like having a car continue driving in circles around a block while the owner eats at a restaurant when parking spaces are limited.

Apart from these immediate effects are the subtle implications that no one can determine immediately, but which can appear evident after reflection. Benedict Evans points outs a few of them in his blog post “Cars and second order consequences” at ben-evans.com). This insightful article looks deeper into consequences of the introduction of both electric cars and level-5 autonomy for SD cars on the market. As one example, SD cars could make the dystopian Panopticon a reality (see “What does the panopticon mean in the age of digital surveillance?” at The Guardian.com. The Panopticon is the institutional building theorized by the English philosopher Jeremy Bentham at the end of the eighteenth century, where everyone is under surveillance without being aware of it. When SD cars roam the streets in large number, car cameras will appear everywhere, watching and possibly reporting everything they happen to witness. Your car may spy on you and others when you least expect it.

Thinking of the future isn’t an easy exercise because it’s not simply a matter of cause and effect. Even looking into more remote orders of effects could prove ineffective when the context changes from the expected. For instance, a future Panopticon may never happen because the legal system could force SD cars not to communicate the images they capture. For this reason, prognosticators rely on scenarios that are approximate descriptions of a possible future; these scenarios may or may not be capable of happening, depending on different circumstances. Experts speculate that a car enabled with autonomous driving capabilities could engage in four different scenarios, each one redefining how humans use or even own a car:

· Autonomous driving on long journeys on highways: When drivers can voluntarily allow the AI to do the driving and take them to their destination, the driver can devote attention to other activities. Many consider this first scenario as a possible introductory scenario for autonomous cars. However, given the high speeds on highways, giving up control to an AI isn’t completely risk-free because other cars, guided by humans, could cause a crash. People have to consider consequences such as the current inattentive driving laws found in most locations. The question is one of whether the legal system would see a driver using an AI as inattentive. This is clearly a level-3 autonomy scenario. Fatalities occur (see “2 Killed in Driverless Tesla Car Crash, Officials Say” at the nytimes.com).

· Acting as a chauffeur for parking: In this scenario, the AI intervenes when the passengers have left the car, saving them the hassle of finding parking. The SD car offers a time-saving service to its occupants as it opens the possibility of both parking-lot optimization (the SD car will know where best to park) and car sharing. (After you leave the car, someone else can use it; later, you hail another car left nearby in the parking lot.) Given the limitations of autonomous driving used only for car fetching, this scenario involves a transition from level-3 to level-4 autonomy.

· Acting as a chauffeur for any journey, except those locations where SD cars remain illegal: This advanced scenario allows the AI to drive in any areas but ones that aren’t permitted for safety reasons (such as new road infrastructures that aren’t mapped by the mapping system used by the car). This scenario takes SD cars to near maturity (autonomy level 4).

· Playing on-demand taxi driver: This is an extension of scenario 2, when the SD cars are mature enough to drive by themselves all the time (level-5 autonomy), with or without passengers, providing a transportation service to anyone requiring it. Such a scenario will fully utilize cars (in this era, cars are parked 95 percent of the time; see “Today’s Cars Are Parked 95% of the Time” at Fortune.com) and revolutionize the idea of owning a car because you won’t need one of your own.

Taking a step back from unmet expectations

At this point, you may expect to ride an SD car soon because it can clearly bring safety and many advantages. At least, you may expect more automation in existing cars. Vendors have, in fact, made quite a few announcements in past years that raised expectations and made many hope for the introduction of autonomous vehicles on the road:

· In 2016, Elon Musk, Tesla’s CEO, announced that “by the end of 2017, one of Tesla’s cars will be able to drive from New York to Los Angeles without the driver having to do anything” (which you can read about at https://tinyurl.com/2368ccj4).

· Apart from Tesla, a lot of automakers, such as General Motors, have made bold statements (https://tinyurl.com/yf74m3h3). Audi (https://tinyurl.com/netjs52d) and Nissan (https://tinyurl.com/3uvz2xbc) have also made announcements.

· In 2016, Business Insider Intelligence forecast 10 million autonomous vehicles on roads by 2020, and it wasn’t the only business intelligence service to forward such ambitious targets (see https://tinyurl.com/k8jn9xsk).

Yet, in spite of such intense hype about SD cars between 2016 and 2017, current cars haven’t changed all that much. At this point, you may even wonder whether the technology will be commercialized anytime soon, with new car models or aftermarket kits capable of transforming your old car into a self-driving one.

Actually, the technology behind SD cars did improve in the recent years, and security issues didn’t limit such development all that much. Yet, everyone working on the technology will now tell you that things appear much more tricky and difficult than they looked back in those 2016–2017 years, and they postpone the introduction of SD cars to the end of the 2020 decade (or possibly beyond).

SD cars are being introduced today in limited parts of the United States. These vehicles have limited scope, such as the cars from Waymo. Waymo is an Alphabet company, heir of the Google Self-Driving Car Project, previously led by Sebastian Thrun, and it has opened its fully driverless service to residents in the metropolitan area of Phoenix, Arizona (see “Waymo is opening its fully driverless service to the general public in Phoenix” at blog.waymo.com). Access to this technology by the general public seems delayed, and it will probably first occur in particular areas and sectors, involving large fleets of cars such as taxis, shuttles, and truck transportation. The problems with SD cars reside in two areas:

· Perception is necessary for the car to determine where it is, and it relies on various technologies discussed by the end of the chapter. The problem with these technologies is that the more reliable they are, the more expensive, and the more maintenance and care they require.

· Prediction helps to elaborate on the perceptions and to provide the car with an idea of what will happen, which is the key to making good decisions on the road and avoiding collisions and other accidents. For instance, such skill is learned when the SD car is engaged in traffic and has to navigate across lanes of oncoming traffic or when, at a crossing, it must perform an unprotected left turn. Determining the behavior of other cars is critical. Unfortunately, what is natural for a human car driver, relying on experience and social cues between drivers, doesn’t seem to come easily for a SD car.

Believed to be the most challenging prediction task, predictions involving other cars are solved at present by forcing an overly cautious behavior on SD cars. Scientists are currently working on reinforcement learning solutions and imitation learning to solve these issues.

Imitation learning techniques try to mimic human behavior in a specific task. This is done in a similar way to reinforcement learning (which consists of an environment and a set of rewards for correct behavior in the environment). The imitation learning approach instead resembles machine learning because it’s supervised and based on data. Usually, experts curate the data for imitation learning manually, and it resembles laboratory data more than real-world data.

SD CARS AND THE TROLLEY PROBLEM

Some say that insurance liability and the trolley problem will seriously hinder SD car use. The insurance problem involves the question of who takes the blame when something goes wrong. Accidents happen now, and SD cars should cause fewer accidents than humans do, so the problem seems easily solved by automakers if the insurance industry won’t insure SD cars. (The insurance industry is wary of SD cars because SD car use could reshape its core business.) SD car automakers such as Audi, Volvo, Google, and Mercedes-Benz have already pledged to accept liability if their vehicles cause an accident (see “Potential liability ramifications of self-driving cars” at cohen-lawyers.com and “The Laws and Liabilities of Autonomous Vehicles” at Cornell Policy Review.com). This means that automakers will become insurers for the greater good of introducing SD cars to the market.

The trolley problem is a moral challenge introduced by the British philosopher Philippa Foot in 1967 (but it is an ancient dilemma). In this problem, a runaway trolley is about to kill a number of people that are on the track, but you can save them by diverting the trolley to another track, where unfortunately another person will be killed in their place. Of course, you need to choose which track to use, knowing that someone is going to die. Quite a few variants of the trolley problem exist, and there is even a Massachusetts Institute of Technology (MIT) website https://www.moralmachine.net/ that proposes alternative situations more suited to those that an SD car may experience.

The point is that situations arise in which someone will die, no matter how skilled the AI is that’s driving the car. In some cases, the choice isn’t between two unknown people, but between the driver and someone on the road. Such situations do happen even now, and humans resolve them by leaving the moral choice to the human at the steering wheel. Some people will save themselves, some will sacrifice for others, and some will choose what they see as the lesser evil or the greater good. Most of the time, it’s a matter of an instinctive reaction made under life-threatening pressure and fear, although the culture you are from plays an important role (as this MIT study points out: https://tinyurl.com/tmu4thty). Mercedes-Benz, the world’s oldest car maker, has stated that it will give priority to passengers’ lives (see https://tinyurl.com/kfybmkes). Car makers might consider that a trolley-problem type of catastrophic situation is already so rare — and SD cars will make it even rarer — and that self-protection is something so innate in us that most SD car buyers will agree upon this choice.

Getting into a Self-Driving Car

Creating an SD car, contrary to what people imagine, doesn’t consist of putting a robot into the front seat and letting it drive the car. Humans perform myriad tasks to drive a car that a robot wouldn’t know how to perform. To create a human-like intelligence requires many systems connecting to each other and working harmoniously together to define a proper and safe driving environment. Some efforts are under way to obtain an end-to-end solution, rather than rely on separate AI solutions for each need. The problem of developing an SD car requires solving many single problems and having the individual solutions work effectively together. For example, recognizing traffic signs and changing lanes require separate systems.

End-to-end solution is something you often hear when discussing deep learning’s role in AI. This means that a single solution will provide an answer to an entire problem, rather than some aspect of a problem. Given the power of learning from examples, many problems don’t require separate solutions, which are essentially a combination of many minor problems, with each one solved by a different AI solution. Deep learning can solve the problem as a whole by solving examples and providing a unique solution that encompasses all the problems that required separate AI solutions in the past.

NVIDIA, the deep learning GPU producer, is working on end-to-end solutions. Check out the video at https://tinyurl.com/5a3zu5vd, which shows the effectiveness of the solution as an example. Yet, as is true for any deep learning application, the goodness of the solution depends heavily on the exhaustiveness and number of examples used. To have an SD car function as an end-to-end deep learning solution requires a dataset that teaches the car to drive in an enormous number of contexts and situations, which aren’t available yet but could be in the future.

Nevertheless, hope exists that end-to-end solutions will simplify the structure of SD cars. The article at https://tinyurl.com/kuar48td explains how the deep learning process works. You may also want to read the original NVIDIA paper on how end-to-end learning helps steer a car at https://tinyurl.com/3enk2f82.

Putting all the tech together

Under the hood of an SD car are systems working together according to the robotic paradigm of sensing, planning, and acting. Everything starts at the sensing level, with many different sensors telling the car different pieces of information:

· The GPS tells where the car is in the world (with the help of a map system), which translates into latitude, longitude, and altitude coordinates.

· The radar, ultrasound, and lidar devices spot objects and provide data about their location and movements in terms of changing coordinates in space.

· The cameras inform the car about its surroundings by providing image snapshots in digital format.

Many specialized sensors appear in an SD car. The “Overcoming Uncertainty of Perceptions” section, later in this chapter, describes them at length and discloses how the system combines their output. The system must combine and process the sensor data before the perceptions necessary for a car to operate become useful. Combining sensor data therefore defines different perspectives of the world around the car.

Localization is knowing where the car is in the world, a task mainly done by processing the data from the GPS device. GPS is a space-based satellite navigation system originally created for military purposes. When used for civilian purposes, it has some inaccuracy embedded (so that only authorized personal can use it to its full precision). The same inaccuracies also appear in other systems, such as GLONASS (the Russian navigation system), GALILEO (or GNSS, the European system), or the BeiDou (or BDS, the Chinese system). Consequently, no matter what satellite constellation you use, the car can tell that it’s on a certain road, but it can miss the lane it’s using (or even end up running on a parallel road). In addition to the rough location provided by GPS, the system processes the GPS data with lidar sensor data to determine the exact position based on the details of the surroundings.

The detection system determines what is around the car. This system requires many subsystems, with each one carrying out a specific purpose by using a unique mix of sensor data and processing analysis:

· Lane detection is achieved by processing camera images using image data analysis or deep learning specialized networks for image segmentation, in which an image is partitioned into separated areas labeled by type (that is, road, cars, and pedestrians).

· Traffic signs and traffic lights detection and classification are achieved by processing images from cameras using deep learning networks that first spot the image area containing the sign or light and then labeling them with the right type (the type of sign or the color of lights). This NVIDIA article helps you understand how an SD car sees: https://tinyurl.com/ph5kdm.

· Combined data from radar, lidar, ultrasound, and cameras help locate external objects and track their movements in terms of direction, speed, and acceleration.

· Lidar data is mainly used for detecting free space on the road (an unobstructed lane or parking space).

Letting AI into the scene

After the sensing phase, which involves helping the SD car determine where it is and what’s going on around it, the planning phase begins. AI fully enters the scene at this point. Planning for an SD car boils down to solving these specific planning tasks:

· Route: Determines the path that the car should take. Because you’re in the car to go somewhere specific (well, that’s not always true, but it’s an assumption that holds true most of the time), you want to reach your destination in the fastest and safest way. In some cases, you also must consider cost. Routing algorithms, which are classic algorithms, are there to help.

· Environment prediction: Helps the car to project itself into the future because it takes time to perceive a situation, decide on a maneuver, and complete it. During the time necessary for the maneuver to take place, other cars could decide to change their position or initiate their own maneuvers, too. When driving, you also try to determine what other drivers intend to do to avoid possible collisions. An SD car does the same thing using machine learning prediction to estimate what will happen next and take the future into account.

· Behavior planning: Provides the car’s core intelligence. It incorporates the practices necessary to stay on the road successfully: lane keeping; lane changing; merging or entering into a road; keeping distance; handling traffic lights, stop signs and yield signs; avoiding obstacles; and much more. All these tasks are performed using AI, such as an expert system that incorporates many drivers’ expertise, or a probabilistic model, such as a Bayesian network, or even a simpler machine learning model.

· Trajectory planning: Determines how the car will actually carry out the required tasks, given that usually more than one way exists to achieve a goal. For example, when the car decides to change lanes, you’ll want it to do so without harsh acceleration or by getting too near other cars, and instead to move in an acceptable, safe, and pleasant way.

Understanding that it’s not just AI

After sensing and planning, it’s time for the SD car to act. Sensing, planning, and acting are all part of a cycle that repeats until the car reaches its destination and stops after parking. Acting involves the core actions of acceleration, braking, and steering. The instructions are decided during the planning phase, and the car simply executes the actions with controller system aid, such as the Proportional-Integral-Derivative (PID) controller or Model Predictive Control (MPC), which are algorithms that check whether prescribed actions execute correctly and, if not, immediately prescribe suitable countermeasures.

It may sound a bit complicated, but it’s just three systems acting, one after the other, from start to end at destination. Each system contains subsystems that solve a single driving problem, as depicted in Figure 14-1, using the fastest and most reliable algorithms.

FIGURE 14-1: An overall, schematic view of the systems working in an SD car.

At the time of writing, this framework is the state of the art. SD cars will likely continue as a bundle of software and hardware systems housing different functions and operations. In some cases, the systems will provide redundant functionality, such as using multiple sensors to track the same external object, or relying on multiple perception processing systems to ensure that you’re in the right lane. Redundancy helps to ensure zero errors and therefore reduce fatalities. For instance, even when a system like a deep learning traffic-sign detector fails or is tricked (see https://tinyurl.com/ufn9ephc), other systems can back it up and minimize or nullify the consequences for the car.

Overcoming Uncertainty of Perceptions

Steven Pinker, professor in the Department of Psychology at Harvard University, says in his book The Language Instinct: How the Mind Creates Language that “in robotics, the easy problems are hard and the hard problems are easy.” In fact, an AI playing chess against a master of the game is incredibly successful; however, more mundane activities, such as picking up an object from the table, avoiding a collision with a pedestrian, recognizing a face, or properly answering a question over the phone, can prove quite hard for an AI.

The Moravec paradox says that what is easy for humans is hard for AI (and vice versa), as explained in the 1980s by robotics and cognitive scientists Hans Moravec, Rodney Brooks, and Marvin Minsky. Humans have had a long time to develop skills such as walking, running, picking up an object, talking, and seeing; these skills developed through evolution and natural selection over millions of years. To survive in this world, humans do what all living beings have done since life has existed on earth — they develop skills that enhance their ability to interact with the world as a whole based on species goals. Conversely, high abstraction and mathematics are relatively new discoveries for humans when you consider the first humans appeared on earth around 315,000 years ago and most experts agree that mathematics only appears 2,500 years go. We aren’t naturally adapted for either mathematics or abstractions; it’s something that an AI can often perform better than we can.

Cars have some advantages over robots, which have to make their way in buildings and on outside terrain. Cars operate on roads specifically created for them, usually well-mapped ones, and cars already have working mechanical solutions for moving on road surfaces.

Actuators aren’t the greatest problem for SD cars. Planning and sensing are what pose serious hurdles. Planning is at a higher level (what AI generally excels in). When it comes to general planning, SD cars can already rely on GPS navigators, a type of AI specialized in providing directions. Sensing is the real bottleneck for SD cars because without it, no planning and actuation are possible. Drivers sense the road all the time to keep the car in its lane, to watch out for obstacles, and to respect the required rules.

Sensing hardware is updated continuously at this stage of the evolution of SD cars to find more reliable, accurate, and less costly solutions. On the other hand, both processing sensor data and using it effectively rely on robust algorithms, such as the Kalman filter (see https://tinyurl.com/2ken4zjx), which have already been around a few decades.

Introducing the car’s senses

Sensors are the key components for perceiving the environment, and an SD car can sense in two directions, internal and external:

· Proprioceptive sensors: Responsible for sensing vehicle state, such as systems status (engine, transmission, braking, and steering) and the vehicle’s position in the world by using GPS localization, rotation of the wheels, the speed of the vehicle, and its acceleration

· Exteroceptive sensors: Responsible for sensing the surrounding environment by using sensors such as camera, lidar, radar, and ultrasonic sensors

Both proprioceptive and exteroceptive sensors contribute to SD car autonomy. GPS localization, in particular, provides a guess (possibly viewed as a rough estimate) as to the SD car’s location, which is useful at a high level for planning directions and actions aimed at getting the SD car to its destination successfully. The GPS helps an SD car in the way it helps any human driver: by providing the right directions.

The exteroceptive sensors (shown in Figure 14-2) help the car specifically in driving. They replace or enhance human senses in a given situation. Each of them offers a different perspective of the environment; each suffers specific limitations; and each excels at different capabilities.

FIGURE 14-2: A schematic representation of exteroceptive sensors in an SD car.

Limitations come in a number of forms. As you explore what sensors do for an SD car, you must consider cost, sensitivity to light, sensitivity to weather, noisy recording (which means that sensitivity of the sensor changes, affecting accuracy), range, and resolution. On the other hand, capabilities involve the capability to track the velocity, position, height, and distance of objects accurately, as well as the skill to detect what those objects are and how to classify them.

Camera

Cameras are passive, vision-based sensors. They can provide mono or stereo vision. Given their low cost, you can place plenty of them on the front windshield, as well as on front grilles, side mirrors, the rear door, and the rear windshield. Commonly, stereo vision cameras mimic human perception and retrieve information on the road and from nearby vehicles, whereas mono vision cameras are usually specialized in detecting traffic signs and traffic lights. The data they capture is processed by algorithms for image processing or by deep learning neural networks to provide detection and classification information (for instance, spotting a red light or a speed-limit traffic signal). Cameras can have high resolution (they can spot small details) but are sensitive to light and weather conditions (night, fog, or snow).

Lidar (LIght Detection And Ranging)

Lidar uses infrared beams (about 900 nanometer wavelength, invisible to human eyes) that can estimate the distance between the sensor and the hit object. They use a rotating swivel to project the beam around and then return estimations in the form of a cloud of collision points, which helps estimate shapes and distances. Depending on price (with higher generally meaning better), lidar can have higher resolution than radar. However, lidar is frailer and easier to get dirty than radar because it’s exposed outside the car. (Lidar is the rotating device you see on top of the Google car in this CBS report: https://tinyurl.com/42rmwvrj.)

Radar (RAdio Detection And Ranging)

Based on radio waves that hit a target and bounce back, and whose time of flight defines distance and speed, radar can be located in the front and rear bumper, as well as on the sides of the car. Vendors have used it for years in cars to provide adaptive cruise control, blind-spot warning, collision warning, and avoidance. In contrast to other sensors that need multiple successive measurements, radar can detect an object’s speed after a single ping because of the Doppler effect (see https://tinyurl.com/4a567s23). Radar comes in short-range and long-range versions, and can both create a blueprint of surroundings and be used for localization purposes. Radar is least affected by weather conditions when compared to other types of detection, especially rain or fog, and has 150 degrees of sight and 30–200 meters of range. Its main weakness is the lack of resolution (radar doesn’t provide much detail) and inability to detect static objects properly.

Ultrasonic sensors

Ultrasonic sensors are similar to radar but use high-frequency sounds (ultrasounds, inaudible by humans, but audible by certain animals) instead of microwaves. The main weakness of ultrasonic sensors (used by manufacturers instead of the frailer and more costly lidars) is their short range.

Putting together what you perceive

When it comes to sensing what is around an SD car, you can rely on a host of different measurements, depending on the sensors installed on the car. Yet, each sensor has different resolution, range, and noise sensitivity, resulting in different measures for the same situation. In other words, none of them is perfect, and their sensory weaknesses sometimes hinder proper detection. Sonar and radar signals might be absorbed; lidar’s rays might pass through transparent solids. In addition, it’s possible to fool cameras with reflections or bad light, as described by this article at MIT Technology Review.com at https://tinyurl.com/yfudnv9c.

SD cars are here to improve our mobility, which means preserving our lives and those of others. An SD car can’t be permitted to fail to detect a pedestrian who suddenly appears in front of it. For safety reasons, vendors focus much effort on sensor fusion, which combines data from different sensors to obtain a unified measurement that’s better than any single measurement. Sensor fusion is most commonly the result of using Kalman filter variants (such as the Extended Kalman Filter or the even more complex Unscented Kalman Filter). Rudolf E. Kálmán was a Hungarian electrical engineer and an inventor who immigrated to the United States during World War II. Because of his invention, which found so many applications in guidance, navigation, and vehicle control, from cars to aircraft to spacecraft, Kálmán received the National Medal of Science in 2009 from U.S. President Barack Obama.

A Kalman filter algorithm works by filtering multiple and different measurements taken over time into a single sequence of measurements that provide a real estimate (the previous measurements were inexact manifestations). It operates by first taking all the measurements of a detected object and processing them (the state prediction phase) to estimate the current object position. Then, as new measurements flow in, it uses the new results it obtains and updates the previous ones to obtain a more reliable estimate of the position and velocity of the object (the measurement update phase), as shown in Figure 14-3.

FIGURE 14-3: A Kalman filter estimates the trajectory of a bike by fusing radar and lidar data.

In this way, an SD car can feed the algorithm the sensor measurements and use them to obtain a resulting estimate of the surrounding objects. The estimate combines all the strengths of each sensor and avoids their weaknesses. This is possible because the filter works using a more sophisticated version of probabilities and Bayes’ theorem, which are described in Chapter 10.