Deep Learning

Lecturer :Ya-Mei Chang

Office: Room 446 ext 66117
[email protected]
Textbook
Title: Deep Learning with Python
Authors: F. Chollet

Reference Book
Title: Deep Learning深度學習必讀: Keras大神帶你用Python實作
Authors: F. Chollet, 葉欣睿 譯

Grading:
●​ Attendance 10% (含點名成績與課堂參與，上課使用手機或進行與課堂無關
的活動扣1分)
●​ Mark of usual 30%
●​ Midterm Exam 30%
●​ Final Report 30%

Office hours:
Tue. And Thr. 10:00~11:00

Rules:
1.​ Late homework is not accepted
2.​ Copying other people's homework is not allowed.

What is deep learning?

1.1​Artificial intelligence, machine learning, deep learning

1
1.1.1​ Artificial intelligence
●​ was born in the 1950s
●​ definition:

●​ : For a fairly long time, many experts believed that human-level

artificial intelligence could be achieved by having programmers handcraft a
sufficiently large set of explicit rules for manipulating knowledge.
●​ Symbolic AI was the dominant paradigm in AI from the 1950s to the late
1980s. It reached its peak popularity during the expert systems boom of
the 1980s.
●​ Although symbolic AI proved suitable to solve
, such as , it turned out to be intractable to figure out
explicit rules for solving more complex, fuzzy problems, such as
. A new approach arose to take symbolic AI’s place:
.
Note:

1.1.2​ Machine learning

●​ In 1843, Ada Lovelace remarked: The Analytical Engine has no pretensions
whatever to originate anything. It can do whatever we know how to order it
to perform.... Its province is to
.
●​ This remark was later quoted by AI pioneer as “Lady Lovelace’s
objection” : introduced the Turing test as well as key concepts that would
come to shape AI. Turing was pondering whether general-purpose
computers could be capable of and , and he came to the
conclusion that .
●​ A machine-learning system is rather than explicitly programmed. It’s

2
 presented with many examples relevant to a task, and
in these examples that eventually allows the system to
come up with rules for .

●​ For instance, if you wished to automate the task of tagging your vacation
pictures, you could present a machine-learning system with many examples
of pictures already tagged by humans, and the system would learn statistical
rules for associating specific pictures to specific tags.
Note:

●​ started to flourish in the 1990s, it has quickly become the most popular and
most successful subfield of AI, a trend driven by the availability of faster
and .
●​ is tightly related to , but it differs from statistics in several
important ways. Unlike statistics, machine learning tends to deal with
(such as a dataset of millions of images, each consisting of tens of thousands
of pixels) for which classical statistical analysis such as would be
impractical.
●​ As a result, machine learning, and especially deep learning, exhibits
comparatively little mathematical theory—maybe too little—and is
.
●​ are proven more often than .
1.1.3​ Learning representations from data
●​ Three things to do in machine learning
■​ — files of people speaking (speech recognition),
pictures (image tagging).
■​ — In a speech-recognition task, these could be
human-generated transcripts of sound files. In an image task,
3
 expected outputs could be tags such as “dog,” “cat,” and so on.
■​

Note:

●​ Therefore, the central problem in machine learning and deep learning is

to : in other words, to learn of
the input data at hand—representations that get us closer to the
expected output.
■​ to or data.
■​ For instance, a color image can be encoded in the
(red-green-blue) or in the format (hue-saturation-value): these
are two different representations of the same data.
■​ In this case,
1.​ The inputs are the .
2.​ The expected outputs are .
3.​ A way to measure whether our algorithm is doing a good job
could be, for instance,
.

With this representation, the black/white problem can be

expressed as a simple rule: “Black points are such that x > 0,” or
“White points are such that x < 0.” This new representation basically
solves the classification problem.
4
 ●​ , in the context of machine learning, describes an
.
●​ Machine-learning algorithms aren’t usually creative in finding these
transformations; they’re merely searching through a predefined set of
operations, called a hypothesis space.
Note:

1.1.4​ The “deep” in deep learning

●​ Deep learning is a specific : a new take on learning
representations from data that puts an emphasis on learning successive
of increasingly meaningful representations.
●​ The deep in deep learning isn’t a reference to any kind of deeper
understanding achieved by the approach; rather, it stands for this idea of
.
●​ How many layers contribute to a model of the data is called the of
the model.
●​ Other appropriate names for the field could have been
and
.
●​ Other approaches to machine learning tend to focus on learning only
one or two layers of representations of the data; hence, they’re
sometimes called .
●​ In deep learning, these layered representations are (almost always)
learned via models called ., structured in literal layers stacked
on top of each other.
●​ The term neural network is a reference to neurobiology, but although
some of the central concepts in deep learning were developed in part by
drawing inspiration from our understanding of the brain,
.
●​ You may come across pop-science articles proclaiming that deep learning
works like the brain or was modeled after the brain, but that isn’t the
case.
●​ For our purposes, deep learning is a
.

5
1.1.5​ Understanding how deep learning works, in three figures

●​ The specification of what a layer does to its input data is stored in the layer’s
, which in essence are a bunch of numbers.
●​ In technical terms, we’d say that the transformation implemented by a layer
is by its weights (see figure 1.7). (Weights are also sometimes
called the of a layer.)
●​ In this context, of all layers in a network, such that
the network will correctly map example inputs to their associated targets.

Note:

6
●​ To control the output of a neural network, you need to be able to measure how
far this output is from what you expected. This is the job of the of the
network, also called the .
●​ The fundamental trick in deep learning is to use this score as a feedback signal to
adjust the value of the weights a little, in a direction that will lower the loss score
for the current example (see figure 1.9). This adjustment is the job of the
, which implements what’s called the algorithm: the central
algorithm in deep learning.

Note:

7
 ●​ Initially, the weights of the network are assigned random values, so the
network merely implements a series of random transformations.
●​ The weights are adjusted a little in the correct direction, and the loss score
decreases. This is the , which, repeated a sufficient number of
times (typically tens of iterations over thousands of examples), yields weight
values that minimize the loss function.
Note:

1.1.6​ What deep learning has achieved so far

●​ In particular, deep learning has achieved the following breakthroughs, all in
historically difficult areas of machine learning:
■​ Near-human-level image classification
■​ Near-human-level speech recognition
■​ Near-human-level handwriting transcription
■​ Improved machine translation
■​ Improved text-to-speech conversion
■​ Digital assistants such as Google Now and Amazon Alexa
■​ Near-human-level autonomous driving
■​ Improved ad targeting, as used by Google, Baidu, and Bing

8
 ■​ Improved search results on the web
■​ Ability to answer natural-language questions
■​ Superhuman Go playing
1.1.7​ Don’t
●​ One of the best-known pioneers and proponents of the symbolic AI
approach was Marvin Minsky, who claimed in 1967, “Within a generation
... the problem of creating ‘artificial intelligence’ will substantially be
solved.” Three years later, in 1970, he made a more precisely quantified
prediction: “In from three to eight years we will have a machine with the
general intelligence of an average human being.” In 2016, such an
achievement still appears to be far in the future—so far that we have no
way to predict how long it will take—but in the 1960s and early 1970s,
several experts believed it to be right around the corner (as do many
people today). A few years later, as these high expectations failed to
materialize, researchers and government funds turned away from the field,
marking the start of the first AI winter (a reference to a nuclear winter,
because this was shortly after the height of the Cold War).
1.1.8​ The promise of AI
●​ Although we may have unrealistic short-term expectations for AI, the
long-term picture is looking bright.
●​ Your doctor doesn’t yet use AI, and neither does your accountant.
●​ Of course, you can ask your smartphone simple questions and get
reasonable answers, you can get fairly useful product recommendations on
Amazon.com, and you can search for “birthday” on Google Photos and
instantly find those pictures of your daughter’s birthday party from last
month.
●​ On the way, we may face a few setbacks and maybe a new AI winter—in
much the same way the internet industry was overhyped in 1998–1999 and
suffered from a crash that dried up investment throughout the early 2000s.
●​ AI will end up being applied to nearly every process that makes up our
society and our daily lives, much like the internet is today.
●​ Don’t believe the short-term hype, but do .