VS 265: Neural Computation Fall 2016

Linear time-invariant systems and convolution

Redwood Center for Theoretical Neuroscience

1 Linear, time-invariant systems

• Let us consider a dynamical system with input x(t) and output y(t):

• Such a system is said to be a linear, time-invariant system if it obeys the laws of superposition
and scaling over time. That is, if you observe an output signal y1 (t) in response to an input
signal x1 (t), and later observe an output y2 (t) in response to input x2 (t), then the response
to the combination αx1 (t) + βx2 (t) is just αy1 (t) + βy2 (t).

• One way to characterize a linear, time-invariant system is by measuring its impulse response
function. This is the response you would obtain to a small pulse of unit amplitude, δ(t),
where
(
1, t = 0
δ(t) =
6 0
0, t =

Let’s call the output signal we measure in response to such an input h(t).

• Once we have measured the impulse response function, h(t), we can compute the output y(t)
in response to any complex input x(t). The first step is to consider the signal x(t) to be
composed of a superposition of unit impulses of different amplitudes:
X
x(t) = δ(t − τj )x(τj )
j

Now since the system is also time-invariant (i.e., it does not change its behavior over time),
then we know that the response to a shifted impulse δ(t − τj ) is just h(t − τj ). And so the
response to a weighted sum of such shifted impulses is just a weighted sum of the resulting
shifted impulse response functions. Or in the language of mathematics:

1
 2

δ(t) → h(t)

δ(t − τj ) → h(t − τj )

x(τj )δ(t − τj ) → x(τj )h(t − τj )

X X
x(τj )δ(t − τj ) → x(τj )h(t − τj )
j j

• Thus, the output y(t) in response to the input signal x(t) may be written as
X
y(t) = x(τj )h(t − τj ) (1)
j

And in the limit that the spacing between time samples becomes infinitesimally small, this
relation becomes exact and the sum turns into an integral:
Z ∞
y(t) = x(τ )h(t − τ )dτ (2)
−∞

2 Convolution

• The expression above is known as the convolution sum (1) or convolution integral (2). It tells
us how to predict the output of a linear, time-invariant system in response to any arbitrary
input signal.

• The mathematical shorthand notation for the convolution operation is to use the symbol as
follows:

y(t) = h(t) ∗ x(t)

• One way of interpreting the convolution sum is just as we developed it above - i.e., it is simply
a linear superposition of impulse response functions h(t − τj ) each of which is multiplied by
x(τj ).

• The other (more common way) of interpreting the convolution sum is that it tells us that the
output is computed by taking a weighted sum of the present and past input values. We can
see this by writing out the sum in (1) above:

y(t) = h(0)x(t) + h(1)x(t − 1) + h(2)x(t − 2) + . . .

where we have assumed here for now that the times τj , τj+1 , . . . are spaced by one unit of
time. Note also that we do not sum over values of τj for which t − τj < 0. The reason is
that for any physical system, h(t) is defined only for t > 0. This indeed makes sense, because
otherwise we would need to know future values of the input in order to compute the present
output.
 3

• Typically the impulse response function decays away with time, and there is a point at which
we can consider it to be essentially zero and so we can truncate the expansion above at a
certain number of taps (discrete samples).

• The convolution operation may also be thought of as a filtering operation on the signal x(t),
where the impulse response function h(t) is acting as the filter. The shape of h(t) determines
which properties of the original signal x(t) are filtered out. The design of filters is usually
best thought of in the frequency domain, which we turn to next ...