Part 3

Working with Software-Based AI Applications

IN THIS PART …

Perform data analysis.

Consider the relationship between AI and machine learning.

Consider the relationship between AI and deep learning.

Chapter 9

IN THIS CHAPTER

Understanding how data analysis works

Using data analysis effectively with machine learning

Determining what machine learning can achieve

Discovering the different kinds of machine learning algorithms

Amassing data isn’t a modern phenomenon; people have amassed data for centuries. No matter whether the information appears in text or numeric form, people have always appreciated how data describes the surrounding world, and among other things, they use it to move civilization forward. Data has a value in itself. By using its content, humanity can learn, transmit critical information to descendants (no need to reinvent the wheel), and effectively act in the world.

People have recently learned that data contains more than surface information. If data is in an appropriate numerical form, you can apply special techniques devised by mathematicians and statisticians, called data analysis techniques, and extract even more knowledge from it. In addition, starting from simple data analysis, you can extract meaningful information and subject data to more advanced analytics using machine learning algorithms capable of predicting the future, classifying information, and effectively helping to make optimal decisions.

Data analysis and machine learning enable people to push data usage beyond previous limits to develop a smarter AI. This chapter introduces you to data analysis. It shows how to use data as a learning tool and starting point to solve challenging AI problems, such as by suggesting the right product to a customer, understanding spoken language, translating English into German, automating car driving, and more.

Defining Data Analysis

The current era is called the Information Age not simply because we have become so data rich but also because society has reached a certain maturity in analyzing and extracting information from that data. Companies such as Alphabet (Google), Amazon, Apple, Facebook, and Microsoft, which have built their businesses on data, are ranked among the most valuable companies in the world. Such companies don’t simply gather and keep stored data that’s provided by their digital processes; they also know how to make it as valuable as oil by employing precise and elaborate data analysis. Google, for instance, records data from the web in general and from its own search engine, among other things, and in order to support its business, it has built a plurality of machine learning models that are continuously updated based on that data.

You may have encountered the “data is the new oil” mantra in the news, in magazines, or at conferences. The statement implies both that data can make a company rich and that it takes skill and hard work to make this happen. Though many have employed the concept and made it incredibly successful, it was Clive Humbly, a British mathematician, who first equated data to oil, given his experience with consumers’ data in the retail sector. Humbly is known for being among the founders of Dunnhumby, a UK marketing company, and the mind behind Tesco’s fidelity card program. In 2006, Humbly also emphasized that data is not just money that rains from the sky; it requires effort to make it useful. Just as you can’t immediately use unrefined oil because it has to be changed into something else by chemical processes that turn it into gas, plastics, or other chemicals, so data must undergo significant transformations to acquire value.

The most basic data transformations are provided by data analysis, and you liken them to the basic chemical transformations that oil undergoes in a refinery before becoming valuable fuel or plastic products. Using just data analysis, you can lay down the foundation for more advanced data analysis processes that you can apply to data. Data analysis, depending on the context, refers to a large body of possible data operations, sometimes specific to certain industries or tasks. You can categorize all these transformations in four large and general families that provide an idea of what happens in data analysis:

· Transforming: Changes the data’s structure. The term transforming refers to different processes, though the most common is putting data into ordered rows and columns in a matrix format (also called flat-file transformation). For instance, you can’t effectively process data of goods bought in a supermarket until you’ve placed each customer in a single row and added products purchased to single columns within that row. You add those products as numeric entries that contain quantities or monetary value. Transforming can also involve specialized numeric transformations such as scaling and shifting, through which you change the mean (the average) and the dispersion (the way a numeric series is spread around its mean values) of the data. These processes make the data suitable for an algorithm.

· Cleansing: Fixes imperfect data. Depending on the means of acquiring the data, you may find different problems because of missing information, extremes in range, or simply wrong values. For instance, data in a supermarket may present errors when goods are labeled with incorrect prices. Some data is adversarial, which means that it has been created to spoil any conclusions. For instance, a product may have fake reviews on the Internet that change its rank. Cleansing helps to remove adversarial examples from data and to make conclusions reliable.

· Inspecting: Validates the data. Data analysis is mainly a human job, though software plays a big role. Humans can easily recognize patterns and spot strange data elements. For this reason, data analysis produces many data statistics and provides useful and insightful visualization, such as Health InfoScape by MIT Senseable Cities and General Electric, which helps grasp informative content at a glance. For example, you can see how diseases connect to one another based on data processed from 72 million records.

· Modeling: Grasps the relationship between the elements present in data. To perform this task, you need tools taken from statistics, such as correlations, chi-square tests, linear regression, and many others that can reveal whether some values truly are different from others or just related. For instance, when analyzing expenditures in a supermarket, you can determine that people buying diapers also tend to buy beer. Statistical analysis finds these two products associated many times in the same baskets. (This study is quite a legend in data analytics; see the short story in this Forbes article, “Birth of a legend.”)

Data analysis isn’t magic. You perform transformations, cleansing, inspecting, and modeling by using mass summation and multiplication based on matrix calculus (which is nothing more than the long sequences of summation and multiplication that many people learn in school). The data analysis arsenal also provides basic statistical tools, such as mean and variance, that describe data distribution, or sophisticated tools, such as correlation and linear regression analysis, that reveal whether you can relate events or phenomena to one another (like buying diapers and beer) based on the evidence. To discover more about such data techniques, both Machine Learning For Dummies, 2nd Edition, and Python for Data Science For Dummies, 2nd Edition, by John Paul Mueller and Luca Massaron (Wiley), offer a practical overview and explanation of each of them.

What makes data analysis hard in the age of big data is the large volume of data that requires special computing tools, such as Hadoop (http://hadoop.apache.org/) and Apache Spark (https://spark.apache.org/), which are two software tools used to perform massive data operations. In spite of such advanced tools, it’s still a matter of perspiration to manually prepare up to 80 percent of the data. The interesting New York Times interview in “For Big-Data Scientists, ‘Janitor Work’ Is Key Hurdle to Insights” with Monica Rogati, who is an expert in the field and an AI advisor to many companies, discusses this issue in more detail.

Understanding why analysis is important

Data analysis is essential to AI. In fact, no modern AI is possible without visualizing, cleansing, transforming, and modeling data before advanced algorithms enter the process and turn it into information of even higher value than before.

In the beginning, when AI consisted of purely algorithmic solutions and expert systems, scientists and experts carefully prepared the data to feed them. Therefore, for instance, if someone wanted an algorithm to process information, a data expert placed the correct data into lists (ordered sequences of data elements) or in other data structures that could appropriately contain the information and allow its desired manipulation. At such a time, data experts gathered and organized the data so that its content and form were exactly as expected, because it was created or selected for that specific purpose. Manipulating known data into a specific form posed a serious limitation because crafting data required a lot of time and energy; consequently, algorithms received less information than is available today.

Today, the attention has shifted from data production to data preparation by using data analysis. The idea is that various sources already produce data in such large quantities that you can find what you need without having to create special data for the task. For instance, imagine wanting an AI to control your pet door to let cats and dogs in but keep other animals out. Modern AI algorithms learn from task-specific data, which means processing a large number of images showing examples of dogs, cats, and other animals. Most likely, such a huge set of images will arrive from the Internet, maybe from social sites or image searches. Previously, accomplishing a similar task meant that algorithms would use just a few specific inputs about shapes, sizes, and distinctive characteristics of the animals, for example. The paucity of data meant that they could accomplish only a few limited tasks. In fact, no examples exist of an AI that can power a pet door using classic algorithms or expert systems.

Data analysis comes to the rescue of modern algorithms by providing information about the images retrieved from the Internet. Using data analysis enables AI to discover the image sizes, variety, number of colors, words used in the image titles, and so on. This is part of inspecting the data and, in this case, that’s necessary to cleanse and transform it. For instance, data analysis can help you spot a photo of an animal erroneously labeled a cat (you don’t want to confuse your AI) and help you transform the images to use the same color format (for example, shades of gray) and the same size.

Reconsidering the value of data

With the explosion of data availability on digital devices (as discussed in Chapter 2), data assumes new nuances of value and usefulness beyond its initial scope of instructing (teaching) and transmitting knowledge (transferring data). The abundance of data, when provided to data analysis, acquires new functions that distinguish it from the informative ones:

· Data describes the world better by presenting a wide variety of facts, and in more detail by providing nuances for each fact. It has become so abundant that it covers every aspect of reality. You can use it to unveil how even apparently unrelated things and facts actually relate to each other.

· Data shows how facts associate with events. You can derive general rules and learn how the world will change or transform, given enough data to dig out the rules you need.

In some respects, data provides us with new super-powers. Chris Anderson, Wired’s previous editor-in-chief, discusses how large amounts of data can help scientific discoveries outside the scientific method (see “The End of Theory: The Data Deluge Makes the Scientific Method Obsolete” at Wired.com). The author relies on the example of achievements of Google in the advertising and translation business sectors, in which Google achieved prominence not by using specific models or theories but rather by applying algorithms to learn directly from data.

DISCOVERING SMARTER AI DEPENDS ON DATA

More than simply powering AI, data makes AI possible. Some people would say that AI is the output of sophisticated algorithms of elevated mathematical complexity, and that’s certainly true. Activities like vision and language understanding require algorithms that aren’t easily explained in layman’s terms and necessitate millions of computations to work. (Hardware plays a role here, too.)

Yet there’s more to AI than algorithms. Dr. Alexander Wissner-Gross, an American research scientist, entrepreneur, and fellow at the Institute for Applied Computation Science at Harvard, provided his insights in an earlier interview at Edge.org (“Datasets Over Algorithms”). The interview reflects on why AI technology took so long to take off, and Wissner-Gross concludes that it might have been a matter of the quality and availability of data rather than algorithmic capabilities.

Wissner-Gross reviews the timing of most breakthrough AI achievements in preceding years, showing how data and algorithms contribute to the success of each breakthrough and highlighting how each of them was fresh at the time the milestone was reached. Wissner-Gross shows how data is relatively new and always updated, whereas algorithms aren’t new discoveries, but rather rely on consolidation of older technology.

The conclusions of Wissner-Gross’s reflections are that, on average, the algorithm is usually 15 years older than the data. He points out that data is pushing AI’s achievements forward and leaves the reader wondering what could happen if feeding the presently available algorithms with better data in terms of quality and quantity were possible.

As in advertising, scientific data (such as from physics, chemistry or biology) can support innovation that allows scientists to approach problems without hypotheses, instead considering the variations found in large amounts of data and using discovery algorithms. In the past, scientists took uncountable observations and a multitude of experiments to gather enough deductions to describe the physics of the universe using the scientific method. This manual process allowed scientists to find many underlying laws of the world.

The ability to innovate using data alone is a major breakthrough in the scientific quest to understand the world. AI achivements such AlphaFold (described in “DeepMind solves 50-year-old ‘grand challenge’ with protein folding A.I.” at CNBC.com) allow scientists to figure out how proteins fold in space and how they function without the need for long experimentation. For many other scientific tasks data analysis pairs observations expressed as inputs and outputs. This technique makes it possible to determine how things work and to define, thanks to machine learning, approximate rules (laws) of our world without having to resort to using manual observations and deductions. Many aspects of the scientific process are now faster and more automatic.

Defining Machine Learning

The pinnacle of data analysis is machine learning. You can successfully apply machine learning only after data analysis provides correctly prepared input. However, only machine learning can associate a series of outputs and inputs, as well as determine the working rules behind the output in an effective way. Data analysis concentrates on understanding and manipulating the data so that it can become more useful and provide insights on the world, whereas machine learning strictly focuses on taking inputs from data and elaborating a working, internal representation of the world that you can use for practical purposes. Machine learning enables people to perform such tasks as predicting the future, classifying things in a meaningful way, and making the best rational decision in a given context.

The central idea behind machine learning is that you can represent reality by using a mathematical function that the algorithm doesn’t know in advance, but which it can guess after seeing some data. You can express reality and all its challenging complexity in terms of unknown mathematical functions that machine learning algorithms find and make actionable. This concept is the core idea for all kinds of machine learning algorithms.

Learning in machine learning is purely mathematical, and it ends by associating certain inputs to certain outputs. It has nothing to do with understanding what the algorithm has learned (data analysis builds understanding to a certain extent), thus the learning process is often described as training because the algorithm is trained to match the correct answer (the output) to every question offered (the input). (Machine Learning For Dummies, 2nd Edition, by John Paul Mueller and Luca Massaron, describes in detail how this process works.)

In spite of lacking deliberate understanding and being simply a mathematical process, machine learning can prove useful in many tasks. It provides the AI application the power of doing the most rational thing given a certain context when learning occurs by using the right data. The following sections help describe how machine learning works in more detail, what benefits you can hope to obtain, and the limits of using machine learning within an application.

Understanding how machine learning works

Many people are used to the idea that applications start with a function, accept data as input, and then provide a result. For example, a programmer might create a function called Add() that accepts two values as input, such as 1 and 2, and provide the result, which is 3. The output of this process is a value. In the past, writing a program meant understanding the function used to manipulate data to create a given result with certain inputs. Machine learning turns this process around. In this case, you know that you have inputs, such as 1 and 2. You also know that the desired result is 3. However, you don't know what function to apply to create the desired result. Training provides a learner algorithm with all sorts of examples of the desired inputs and results expected from those inputs. The learner then uses this input to create a function. In other words, training is the process whereby the learner algorithm maps a flexible function to the data. The output is typically the probability of a certain class or a numeric value.

To give an idea of what happens in the training process, imagine a child learning to distinguish trees from other objects. Before the child can do so in an independent fashion, a teacher presents the child with a certain number of tree images, complete with all the facts that make a tree distinguishable from other objects of the world. Such facts could be features such as the tree's material (wood), its parts (trunk, branches, leaves or needles, roots), and location (planted into the soil). The child produces an idea of what a tree looks like by contrasting the display of tree features with the images of other, different objects, such as pieces of furniture that are made of wood but do not share other characteristics with a tree.

A machine learning classifier works the same. It builds its cognitive capabilities by creating a mathematical formulation that includes all the given features in a way that creates a function that can distinguish one class from another. Pretend that a mathematical formulation, also called target function, exists to express the characteristics of a tree. In such a case, a machine learning classifier can look for its representation as a replica or an approximation (a different function that works alike). Being able to express such mathematical formulation is the representation capability of the machine learning algorithm.

From a mathematical perspective, you can express the representation process in machine learning by using the equivalent term mapping. Mapping happens when you discover the construction of a function by observing its outputs. A successful mapping in machine learning is similar to a child internalizing the idea of an object. The child understands the abstract rules derived from the facts of the world in an effective way so that when the child sees a tree, for example, the child immediately recognizes it.

Such a representation (abstract rules derived from real-world facts) is possible because the learning algorithm has many internal parameters (consisting of vectors and matrices of values), which equate to the algorithm’s memory for ideas that are suitable for its mapping activity that connects features to response classes. The dimensions and type of internal parameters delimit the kind of target functions that an algorithm can learn. An optimization engine in the algorithm changes parameters from their initial values during learning to represent the target’s hidden function.

During optimization, the algorithm searches the possible variants of its parameter combinations to find one that allows correct mapping between features and classes during training. This process evaluates many potential candidate target functions from among those that the learning algorithm can guess. The set of all the potential functions that the learning algorithm can discover is the hypothesis space. You can call the resulting classifier with its set parameters a hypothesis, a way in machine learning to say that the algorithm has set parameters to replicate the target function and is now ready to define correct classifications (a fact demonstrated later).

The hypothesis space must contain all the parameter variants of all the machine learning algorithms that you want to try to map to an unknown function when solving a classification problem. Different algorithms can have different hypothesis spaces. What really matters is that the hypothesis space contains the target function (or its approximation, which is a different but similar function, because in the end all you need is something that works).

You can imagine this phase as the time when a child experiments with many different creative ideas by assembling knowledge and experiences (an analogy for the given features) in an effort to create a visualization of a tree. Naturally, the parents are involved in this phase, and they provide relevant environmental inputs. In machine learning, someone has to provide the right learning algorithms, supply some nonlearnable parameters (called hyperparameters), choose a set of examples to learn from, and select the features that accompany the examples. Just as a child can’t always learn to distinguish between right and wrong if left alone in the world, so machine learning algorithms need guidance from human beings to learn successfully.

Understanding the benefits of machine learning

You find AI and machine learning used in a great many applications today. The only problem is that the technology works so well that you don’t know that it even exists. In fact, you might be surprised to find that many devices in your home already make use of both technologies. Both technologies definitely appear in your car and the workplace. In fact, the uses for both AI and machine learning number in the millions — all safely out of sight even when they’re quite dramatic in nature. Chapter 1 lists a few of the ways in which you might see AI used (fraud detection, resource scheduling, and others; see “Considering AI Uses” in that chapter), but that list doesn’t even begin to scratch the surface. You can find AI used in many other ways. However, it’s also useful to view uses of machine learning outside the normal realm that many consider the domain of AI. Here are a few uses for machine learning that you might not associate with an AI:

· Access control: In many cases, access control is a yes-or-no proposition. An employee smartcard grants access to a resource in much the same way that people have used keys for centuries. Some locks do offer the capability to set times and dates that access is allowed, but such coarse-grained control doesn’t really answer every need. By using machine learning, you can determine whether an employee should gain access to a resource based on role and need. For example, an employee can gain access to a training room when the training reflects an employee role.

· Animal protection: The ocean might seem large enough to allow animals and ships to cohabitate without problem. Unfortunately, many animals get hit by ships each year. A machine learning algorithm could allow ships to avoid animals by learning the sounds and characteristics of both the animal and the ship.

· Predicting wait times: Most people don’t like waiting when they have no idea how long the wait will be. Machine learning allows an application to determine waiting times based on staffing levels, staffing load, complexity of the problems the staff is trying to solve, availability of resources, and so on.

Being useful; being mundane

Even though the movies suggest that AI is sure to make a huge splash, and you do occasionally see incredible uses for AI in real life, most uses for AI are mundane and even boring. For example, Hilary Mason, general manager of machine learning at Cloudera, cites how machine learning is used in an international accounting firm to automatically fill in accounting questionnaires (see “Make AI Boring: The Road from Experimental to Practical” at InformationWeek.com). The act of performing this analysis is dull when compared to other sorts of AI activities, but the benefits are that the accounting firm saves money, and the results are better as well.

Specifying the limits of machine learning

Machine learning relies on algorithms to analyze huge datasets. Currently, machine learning can’t provide the sort of AI that the movies present. Even the best algorithms can’t think, feel, display any form of self-awareness, or exercise free will. What machine learning can do is perform predictive analytics far faster than any human can. As a result, machine learning can help humans work more efficiently. The current state of AI, then, is one of performing analysis, but humans must still consider the implications of that analysis and make the required moral and ethical decisions. Essentially, machine learning provides just the learning part of AI, and that part is nowhere near ready to create an AI of the sort you see in films.

The main point of confusion between learning and intelligence is people’s assumption that simply because a machine gets better at its job (learning), it’s also aware (intelligence). Nothing supports this view of machine learning. The same phenomenon occurs when people assume that a computer is purposely causing problems for them. The computer can’t assign emotions and therefore acts only upon the input provided and the instruction contained within an application to process that input. A true AI will eventually occur when computers can finally emulate the clever combination used by nature:

· Genetics: Slow learning from one generation to the next

· Teaching: Fast learning from organized sources

· Exploration: Spontaneous learning through media and interactions with others

Apart from the fact that machine learning consists of mathematical functions optimized for a certain purpose, other weaknesses expose the limits of machine learning. You need to consider three important limits:

· Representation: Representing some problems using mathematical functions isn’t easy, especially with complex problems like mimicking a human brain. At the moment, machine learning can solve single, specific problems that answer simple questions, such as “What is this?” and “How much is it?” and “What comes next?”

· Overfitting: Machine learning algorithms can seem to learn what you care about, but they actually often don’t. Therefore, their internal functions mostly memorize the data without learning from the data. Overfitting occurs when your algorithm learns too much from your data, up to the point of creating functions and rules that don’t exist in reality.

· Lack of effective generalization because of limited data: The algorithm learns what you teach it. If you provide the algorithm with bad or weird data, it behaves in an unexpected way.

As for representation, a simple-learner algorithm can learn many different things, but not every algorithm is suited for certain tasks. Some algorithms are general enough that they can play chess, recognize faces on Facebook, and diagnose cancer in patients. An algorithm reduces the data inputs and the expected results of those inputs to a function in every case, but the function is specific to the kind of task you want the algorithm to perform.

The secret to machine learning is generalization. However, with generalization come the problems of overfitting and biased data (data that when viewed using various statistical measures is skewed in one direction or the other). The goal is to generalize the output function so that it works on data beyond the training examples. For example, consider a spam filter. Say that your dictionary contains 100,000 words (a small dictionary). A limited training dataset of 4,000 or 5,000 word combinations (as you would see them in a real sentence) must create a generalized function that can then find spam in the 2^100,000 combinations that the function will see when working with actual data. In such conditions, the algorithm will seem to learn the rules of the language, but in reality it won’t do well. The algorithm may respond correctly to situations similar to those used to train it, but it will be clueless in completely new situations. Or, it can show biases in unexpected ways because of the kind of data used to train it.

For instance, Microsoft trained its AI, Tay, to chat with human beings on Twitter and learn from their answers. Unfortunately, the interactions went haywire because users exposed Tay to hate speech, raising concerns about the goodness of any AI powered by machine learning technology. (You can read some of the story at https://tinyurl.com/4bfakpac.) The problem was that the machine learning algorithm was fed bad, unfiltered data (Microsoft didn’t use appropriate data analysis to clean and balance the input appropriately), which overfitted the result. The overfitting selected the wrong set of functions to represent the world in a general way as needed to avoid providing nonconforming output, such as hate speech. Of course, even if the output wasn’t undesirable, it could still be nonconforming, such as giving wrong answers to straightforward questions. Other AI trained to chat with humans, such as the award-winning Kuki (https://www.kuki.ai/), aren’t exposed to the same risks as Tay because their learning is strictly controlled and supervised by data analysis and human evaluation.

Considering How to Learn from Data

Everything in machine learning revolves around algorithms. An algorithm is a procedure or formula used to solve a problem. The problem domain affects the kind of algorithm needed, but the basic premise is always the same: to solve some sort of problem, such as driving a car or playing dominoes. In the first case, the problems are complex and many, but the ultimate problem is one of getting a passenger from one place to another without crashing the car. Likewise, the goal of playing dominoes is to win.

Learning comes in many different flavors, depending on the algorithm and its objectives. You can divide machine learning algorithms into three main groups, based on their purpose:

· Supervised learning

· Unsupervised learning

· Reinforcement learning

The following sections discuss what different kinds of algorithms are exploited by machine learning in more detail.

Supervised learning

Supervised learning occurs when an algorithm learns from example data and associated target responses that can consist of numeric values or string labels, such as classes or tags, in order to later predict the correct response when given new examples. The supervised approach is similar to human learning under the supervision of a teacher. The teacher provides good examples for the student to memorize, and the student then derives general rules from these specific examples.

You need to distinguish between regression problems, whose target is a numeric value, and classification problems, whose target is a qualitative variable, such as a class or a tag. A regression task could determine the average prices of houses in the Boston area, while an example of a classification task is distinguishing between kinds of iris flowers based on their sepal and petal measures. Here are some examples of supervised learning with important applications in AI described by their data input, their data output, and the real-world application they can solve:

Data Input (X)	Data Output (y)	Real-World Application
History of customers’ purchases	A list of products that customers have never bought	Recommender system
Images	A list of boxes labeled with an object name	Image detection and recognition
English text in the form of questions	English text in the form of answers	Chatbot, a software application that can converse
English text	German text	Machine language translation
Audio	Text transcript	Speech recognition
Image, sensor data	Steering, braking, or accelerating	Behavioral planning for autonomous driving

Unsupervised learning

Unsupervised learning occurs when an algorithm learns from plain examples without any associated response, leaving the algorithm to determine the data patterns on its own. This type of algorithm tends to restructure the data into something else, such as new features that may represent a class or a new series of uncorrelated values. The resulting data are quite useful in providing humans with insights into the meaning of the original data and new useful inputs to supervised machine learning algorithms.

Unsupervised learning resembles methods used by humans to determine that certain objects or events are from the same class, such as observing the degree of similarity between objects. Some recommender systems that you find on the web in the form of marketing automation are based on this type of learning. The marketing automation algorithm derives its suggestions from what you’ve bought in the past. The recommendations are based on an estimation of what group of customers you resemble the most and then inferring your likely preferences based on that group.

Reinforcement learning

Reinforcement learning occurs when you present the algorithm with examples that lack labels, as in unsupervised learning. However, you can accompany an example with positive or negative feedback according to the consequences of the solution that the algorithm proposes.

Reinforcement learning is connected to applications for which the algorithm must make decisions (so the product is prescriptive, not just descriptive, as in unsupervised learning), and the decisions bear consequences. In the human world, it is just like learning by trial and error. Errors help you learn because they have a penalty added (cost, loss of time, regret, pain, and so on), teaching you that a certain course of action is less likely to succeed than others. An interesting example of reinforcement learning occurs when computers learn to play video games by themselves.

In this case, an application presents the algorithm with examples of specific situations, such as having the gamer stuck in a maze while avoiding an enemy. The application lets the algorithm know the outcome of actions it takes, and learning occurs while trying to avoid what it discovers to be dangerous and to pursue survival. You can see how Google DeepMind created a reinforcement learning program that plays old Atari video games on YouTube (“Google DeepMind's Deep Q-learning playing Atari Breakout”). When watching the video, notice how the program is initially clumsy and unskilled but steadily improves with training until it becomes a champion. The process is described as having strong and weak points by Raia Hadsell, a senior research scientist on the Deep Learning team at DeepMind, in an enlightening video from TEDx Talks, “Artificial intelligence, video games and the mysteries of the mind,” on YouTube.