Questions about connectionist models of natural language

Behavioral and Brain Sciences, 1988

A set of hypotheses is formulated for a connectionist approach to cognitive modeling. These hypotheses are shown to be incompatible with the hypotheses underlying traditional cognitive models. The connectionist models considered are massively parallel numerical computational systems that are a kind of continuous dynamical system. The numerical variables in the system correspond semantically to fine-grained features below the level of the concepts consciously used to describe the task domain. The level of analysis is intermediate between those of symbolic cognitive models and neural models. The explanations of behavior provided are like those traditional in the physical sciences, unlike the explanations provided by symbolic models.Higher-level analyses of these connectionist models reveal subtle relations to symbolic models. Parallel connectionist memory and linguistic processes are hypothesized to give rise to processes that are describable at a higher level as sequential rule appli...

Physical Review Letters

Learning a task induces connectivity changes in neural circuits, thereby changing their dynamics. To elucidate task related neural dynamics we study trained Recurrent Neural Networks. We develop a Mean Field Theory for Reservoir Computing networks trained to have multiple fixed point attractors. Our main result is that the dynamics of the network's output in the vicinity of attractors is governed by a low order linear Ordinary Differential Equation. Stability of the resulting ODE can be assessed, predicting training success or failure. As a consequence, networks of Rectified Linear (RLU) and of sigmoidal nonlinearities are shown to have diametrically different properties when it comes to learning attractors. Furthermore, a characteristic time constant, which remains finite at the edge of chaos, offers an explanation of the network's output robustness in the presence of variability of the internal neural dynamics. Finally, the proposed theory predicts state dependent frequency selectivity in network response.

Behavioral and Brain Sciences, 1992

1999

2014

Some argue the common practice of inferring multiple processes or systems from a dissociation is flawed . One proposed solution is state-trace analysis , which involves plotting, across two or more conditions of interest, performance measured by either two dependent variables, or two conditions of the same dependent measure. The resulting analysis is considered to provide evidence that either (a) a single process underlies performance (one function is produced) or (b) there is evidence for more than one process (more than one function is produced). This article reports simulations using the simple recurrent network (SRN; Elman, 1990) in which changes to the learning rate produced state-trace plots with multiple functions. We also report simulations using a single-layer error-correcting network that generate plots with a single function. We argue that the presence of different functions on a state-trace plot does not necessarily support a dual-system account, at least as typically defined (e.g. two separate autonomous systems competing to control responding); it can also indicate variation in a single parameter within theories generally considered to be single-system accounts.

International Encyclopedia of the Social & Behavioral Sciences, 2001

Connectionist approaches to cognitive modeling make use of large networks of simple computational units, which communicate by means of simple quantitative signals. Higher-level information processing emerges from the massivelyparallel interaction of these units by means of their connections, and a network may adapt its behavior by means of local changes in the strength of the connections. Connectionist approaches are related to neural networks and provide a distinct alternative to cognitive models inspired by the digital computer. To facilitate the following discussion, it will be helpful to de ne some terms. A typical connectionist network comprises a (potentially large) number of simple processing units. The units are often called (arti cial) neurons, but that terminology begs the question of their relation to biological neurons, so it will be avoided here (Sect. 5.4). In the most common case, the units form a weighted sum of their (quantitative) inputs and pass the result through Int. Encyc. Social and Behavioral Sciences

I discuss a connectionist model, based on Elman's (1990, 1991) Simple Recurrent Network, of the acquisition of complex syntactic structure. While not intended as a detailed model of the process children go through in acquiring natural languages, the model helps clarify concepts that may be useful for understanding the development of complex abilities. It provides evidence that connectionist learning can produce stage-wise development emergently. It is consistent with prior work on connectionist models emphasizing their capability of computing in ways that are not possible within the symbolic paradigm (Siegelmann, 1999). On the other hand, it suggests that one mechanism of the symbolic paradigm (a pushdown automaton) may be identified with an attractor of the learning process. Thus, the model provides a concrete example of what might be called "emergence of new conceptual structure during development" and suggests that we need to use both dynamical systems theory and symbolic computation theory to make sense of it.

Frontiers in Systems Neuroscience, 2020

The present report examines the coinciding results of two study groups each presenting a power-of-two function to describe network structures underlying perceptual processes in one case and word production during verbal fluency tasks in the other. The former is theorized as neural cliques organized according to the function N = 2 i − 1, whereas the latter assumes word conglomerations thinkable as tuples following the function N = 2 i. Both theories assume the innate optimization of energy efficiency to cause the specific connectivity structure. The vast resemblance between both formulae motivated the development of a common formulation. This was obtained by using a vector space model, in which the configuration of neural cliques or connected words is represented by a N-dimensional state vector. A further analysis of the model showed that the entire time course of word production could be derived using basically one single minimal transformation-matrix. This again seems in line with the principle of maximum energy efficiency.

1993

\Articial neural networks" provide an appealing model of computation. Such networks consist of an interconnection of a number of parallel agents, or \neurons." Each of these receives certain signals as inputs, computes some simple function, and produces a signal as output, which is in turn broadcast to the successive neurons involved in a given computation. Some of the signals originate from outside the network, and act as inputs to the whole system, while some of the output signals are communicated back to the environment and are used to encode the end result of computation. In this dissertation we focus on the \recurrent network" model, in which the underlying graph is not subject to any constraints. We investigate the computational power of neural nets, taking a classical computer science point of view. We characterize the language recognition power of networks in terms of the types of numbers (constants) utilized as weights. From a mathematical viewpoint, it is natural to consider integer, rational, and real numbers. From the standpoint of computer science, natural classes of formal languages are regular, recursive, and \all languages." We establish a precise correspondence between the mathematical and computing choices. Furthermore, when the computation time of the network is constrained to be polynomial in the input size, the classes recognized by the respective networks are: regular, P, and Analog-P, i.e. P/poly. Among other results described in this thesis are a proper hierarchy of networks using Kolmogorov-complexity characterizations, the imposition of space constraints, and a proposed \Church's thesis of analog computing."