Connectionist models and their properties

1982, Cognitive Science

Abstract

Much of the progress in the fields constituting cognitive science has been based upon the use of explicit information processing models, almost exclusively patterned after conventional serial computers. An extension of these ideas to massively parallel, connectianist models appears to offer a number of advantages. After a preliminary discussion, this paper introduces a general connectionist model and considers how it might be used in cognitive science. Among the issues addressed are: stability and noise-sensitivity, distributed decisionmaking, time and sequence problems, and the representation of complex concepts.

Figures (23)

Figure 2. Connectionism vs. symbolic encoding. => Assumes some general encoding —» Assumes individual connections

Figure 3. Conjunctive connections and disjunctive input sites.

lation and abstraction of a subnetwork by an individual unit. For example, a system that had separate motor abilities for turning left and turning right (e.g., fins) could use two start-stop units to model a turn-unit, as shown in Figure 4. Note that the compound unit here has two distinct outputs, where basic units have only one (which can branch, of course). In general, com- pound units will differ from basic ones only in that they can have several distinct outputs.

imum of the sums of the confidences of its two inputs. A major problem with function tables (and with CM in general) is the potential combinatoria! explosion in the number of units required for a computation. A naive ap- proach would demand N? units to represent all products of numbers from | to N. The network of Figure 7 requires many fewer units because each prod- uct is represented only once, another advantage of conjunctive connections, We could use even fewer units by exploiting positional notation and replac. ing each output connection with a conjunction of outputs from units repre- senting multiples of 1, 10, 100, etc. The question of efficient ways of building connection networks is treated in detail in Section 4 (cf. also Hin- ton, 1981la; 1981b). imum of the sums of the confidences of its two inputs. A major problem with function tables (and with CM in general) is the potential combinatorial explosion in the number of units required for a computation. A naive ap- proach would demand N? units to represent all products of numbers from 1 to N. The network of Figure 7 requires many fewer units because each prod- uct is represented only once, another advantage of conjunctive comnections. We could use even fewer units by exploiting positional notation and replac- ing each output connection with a conjunction of outputs from units repre- senting multiples of 1, 10, 100, etc. The question of efficient ways of building connection networks is treated in detail in Section 4 (cf. also Hin- ton, 198la; 1981b).

Figure 8. Grass is Green connection modified by California.

Figure 14. Modified Multiplication Table using Less Units. In the multiplication example z= xy, the number of z units required is pro- portional to N,N, even when redundant value units are eliminated, and in general the number of units could grow exponentially with the number of arguments. However, there are several refinements which can drastically reduce the number of required units. One way to do this is to fix the number of units at the precision required for the computation. Figure 14 shows the network of Figure 7 modified when less computational accuracy is required. This is the same principle that is incorporated in integer calculations in a sequential computer: computations are rounded to within the machine’s accuracy. Accuracy is related to the number of bits and the number repre- sentation. The main difference is that since the sequential computer is general purpose, the number representations are conservative, involving large number of bits. The neural units need only represent sufficient ac- curacy for the problem at hand. This will generally vary from network to network, and may involve very inhomogeneous, special purpose number representations. This is the same principle that is incorporated in integer calculations i: a sequential computer: computations are rounded to within the machine’ accuracy. Accuracy is related to the number of bits and the number repre sentation. The main difference is that since the sequential computer i general purpose, the number representations are conservative, involvin, large number of bits. The neural units need only represent sufficient ac curacy for the problem at hand. This will generally vary from network t network, and may involve very inhomogeneous, special purpose numbe representations.

Figure 15b. Coarse angle—fine position and coarse position—fine angle units combine tc yield precise values of all five parameters.

Figure 17. Spatial conerence on inputs can represent complex concepts without cross-talk Solid lines show active inputs and dashed lines (some of the) inactive inputs.

Figure 18. Spatial focus unit can gate only input from attended positions. area would compete to index objects. The obvious way to focus attention on one area of the visual field is with eye movements, but there is evidence that focus can also be done within a fixation. The general idea of internal spatial focus is shown in Figure 18. In this network, the general ‘‘baby-blue’’ unit is configured to have separate conjunctive inputs for each point in space, like the blue- square units of Figure 17. The difference is that the second input to the con- junction comes from a ‘‘focus’’ unit, and this makes a much more general network. The idea of making a unit (e.g., baby blue) more responsive to in- puts from a given spatial position can be implemented in different ways. The conjunctive connection at the x =7 lobe of the baby-blue unit is the most direct way. But treating this conjunct as a strict AND would mean that all spatial units would have to be active when there was no focus. An alter-

Figure 20. Distributed Control of Eye Fixations