2007 Special IssueDecoupled echo state networks with lateral inhibition
Introduction
The echo state network (ESN) (Jaeger, 2001, Jaeger, 2004a) and its spiking counterpart, liquid state network (LSN) (Maass, Natschläger, & Markram, 2002) are new recurrent neural network (RNN) structures. Both were motivated by recent neurophysiological experiments (see Jaeger (2004a) and the references therein). The kernel part of ESN is a single reservoir with a large number of neurons that are randomly inter-connected and/or self-connected. The reservoir itself is fixed, once it is picked. Moreover, during the training process of ESN, only the output connections are changed through offline linear regression or online methods, such as the recursive least square (RLS) (Jaeger, 2001, Jaeger, 2003, Jaeger, 2004a). The ESN has been successfully applied in chaotic and nonlinear dynamic systems modeling, identification and control (Ishii et al., 2004, Jaeger, 2004a, Ploger¨ et al., 2003).
Despite the simplicity of the training strategies for ESN, the construction of ESN, especially the reservoir and input/feedback connections, seems to always require numerous trials and even luck. This is because the free parameters, such as the size, spectral radius, sparsity of the reservoir weight matrix, the number of inputs/feedbacks, if needed, and all the connection weights must be appropriately selected for the network to generate satisfactory performance. Sometimes, the resulting ESN for a practical problem may have a heavy computation load for real-time and embedded applications (Prokhorov, 2005); this drawback is mainly due to handling a reservoir of large size in either learning or exploitation. Moreover, ESN with a single reservoir is not omnipotent. As shown in Jaeger (2004b), it was not possible for an ESN to be trained to work as a multiple superimposed oscillator (MSO), even when the function consists of only two sine waves, for example, . In Wiestra, Gomez, and Schmidhuber (2005), it was pointed out that the reason for the failure in the MSO problem is that all the neurons in the same reservoir are actually coupled, while the task requires the simultaneous existence of multiple decoupled internal states. To partially solve the above two problems, an evolutionary-based reservoir design method, called Evolino, was proposed in Schmidhuber, Gagliolo, Wierstra, and Gomez (2005) and Wiestra et al. (2005). It has been shown that Evolino is able to automatically generate task-dependent RNN with MSO capability, at the cost of significantly increased complexity imposed by the sub-network evolution routine.
In this paper, we propose a novel ESN structure, termed the decoupled echo state network (DESN) involving the use of lateral inhibition, to do the following: provide an alternative way of solving the MSO problem, reduce the computational complexity of the traditional ESN, and improve the probability of obtaining a satisfactory network under the condition of ‘random’ parameter selections. Unlike the evolutionary-based approach, the new structure makes use of multiple reservoirs that are competing with each other through inhibitory connections, and combines the internal states of all the reservoirs to form the output action potentials. In this way, the coupling effects between neurons within a single reservoir, as well as the correlation of neural dynamics between different reservoirs, are reduced, and multiple decoupled internal states are thereby generated to cooperatively accomplish a variety of tasks. Besides, owing to the coexistence of competitive and cooperative mechanisms, the problem of large performance variations caused by different choices of free parameters may be mitigated. In this paper, we develop two implementation schemes for DESN with lateral inhibition: one is termed DESN with reservoir prediction (DESN + RP), and the other is termed DESN with maximum available information (DESN + MaxInfo). Through experiments with computer-generated data and real-life noisy data, we observe that compared with a single-reservoir ESN, these two new structures are able to provide low-complexity but efficient solutions to the MSO problem and the predictive modeling task. Moreover, the performance of DESN with lateral inhibition is shown to be highly robust with respect to various reservoir configurations and randomly generated connection weights.
The rest of the paper is organized as follows. In Section 2, we mathematically formulate the multiple reservoir ESN problem after a short review of the echo state network, setting the stage for the development of DESN with lateral inhibition. Section 3 describes in depth the DESN with lateral inhibition, including the basic ideas and implementations of DESN + RP and DESN + MaxInfo. Section 4 presents three computer experiments: the MSO problem, a noiseless predictive modeling task and real-life sea clutter prediction; the results obtained therein demonstrate the performance and robustness improvements of our new approaches. Section 5 summarizes the paper and discusses future research topics.
Section snippets
ESN with multiple reservoirs
The proposed ESN structure with multiple reservoirs has been inspired by comparing the MSO modeling task with the classic kernel-based signal representation. For example, in the Fourier transform and wavelet transform, the individual kernel (i.e. basis function) could represent an almost unique portion of the target signal. Moreover, as shown in Jaeger (2001), an ESN with a single reservoir can be trained to be a sine-wave generator. Therefore, a natural idea to solve the MSO modeling problem
DESN with lateral inhibition
Lateral inhibition is a well-known mechanism, which was originally discovered in neurobiological systems including the retina, cochlear, and pressure sensitive nerves in skin (Arbib, 1989, Fischler and Firschein, 1987, Haykin, 2004). Specifically, when a neuron is stimulated, the neural activity of its surrounding region is suppressed via lateral inhibitory synapses. In this way, different neurons would possess various receptive fields, i.e., they will respond to stimulations of different
Computer experiments
In this section, we study the performance of DESN + RP and DESN + MaxInfo for three scenarios: the classic MSO problem, a challenging nonlinear prediction task, and a real-life sea clutter prediction task. The parameters pertaining to the network structure and training/testing strategy are shown in Table 1. Other parameters, along with the ones listed in Table 1, will be described before presenting simulation results. For all three scenarios, the training strategy used is the classic offline
Summary and discussion
In the newly proposed DESN with lateral inhibition structure, the harmonious competition and cooperation between the multiple reservoirs offer us a novel MSO problem-solving capability, low computational complexity and a strong robust behavior. The MSO problem is solved through the incorporation of a lateral inhibition mechanism in DESN as a simple but efficient method to reduce the coupling effects between neurons within the same reservoir and the correlation of the reservoirs’ activities. The
Acknowledgements
The authors would like to express their gratitude to two anonymous reviewers for their valuable suggestions and comments, which have greatly improved the presentation of this paper. This work was supported by National Science and Engineering Research Council of Canada (NSERC) and Canadian Institute of Health Research (CIHR).
References (16)
The metaphorical brain
(1989)- Bush, K., & Tsendjav, B. (2005). Improving the richness of echo state features using next ascent local search. In Proc....
- et al.
Principal component neural network: Theory and application
(1996) - et al.
Intelligence: The eye, the brain and the computer
(1987) Neural networks: A comprehensive foundation
(2004)- Ishii, K., van der Zant, T., Becanovic, V., & Plöger, P. (2004). Identification of motion with echo state network. In...
- Jaeger, H. (2001). The echo state approach to analysing and training recurrent neural networks. GMD report 148....
Adaptive nonlinear system identification with echo state network
Cited by (189)
An echo state network model with the protein structure for time series prediction
2024, Applied Soft ComputingEcho state network structure optimization algorithm based on correlation analysis
2024, Applied Soft ComputingPrediction for nonlinear time series by improved deep echo state network based on reservoir states reconstruction
2024, Autonomous Intelligent Systems
- 1
These authors contributed equally to this paper.