Elsevier

Neural Networks

Volume 20, Issue 3, April 2007, Pages 365-376
Neural Networks

2007 Special Issue
Decoupled echo state networks with lateral inhibition

https://doi.org/10.1016/j.neunet.2007.04.014Get rights and content

Abstract

Building on some prior work, in this paper we describe a novel structure termed the decoupled echo state network (DESN) involving the use of lateral inhibition. Two low-complexity implementation schemes, namely, the DESN with reservoir prediction (DESN + RP) and DESN with maximum available information (DESN + MaxInfo), are developed: (1) In the multiple superimposed oscillator (MSO) problem, DESN + MaxInfo exhibits three important attributes: lower generalization mean-square error (MSE), better robustness with respect to the random generation of reservoir weight matrix and feedback connections, and robustness to variations in the sparseness of reservoir weight matrix, compared to DESN + RP. (2) For a noiseless nonlinear prediction task, DESN + RP outperforms the DESN + MaxInfo and single reservoir-based ESN approach in terms of lower prediction MSE and better robustness to a change in the number of inputs and sparsity of the reservoir weight matrix.

Finally, in a real-life prediction task using noisy sea clutter data, both schemes exhibit higher prediction accuracy and successful design ratio than a conventional ESN with a single reservoir.

Introduction

The echo state network (ESN) (Jaeger, 2001, Jaeger, 2004a) and its spiking counterpart, liquid state network (LSN) (Maass, Natschläger, & Markram, 2002) are new recurrent neural network (RNN) structures. Both were motivated by recent neurophysiological experiments (see Jaeger (2004a) and the references therein). The kernel part of ESN is a single reservoir with a large number of neurons that are randomly inter-connected and/or self-connected. The reservoir itself is fixed, once it is picked. Moreover, during the training process of ESN, only the output connections are changed through offline linear regression or online methods, such as the recursive least square (RLS) (Jaeger, 2001, Jaeger, 2003, Jaeger, 2004a). The ESN has been successfully applied in chaotic and nonlinear dynamic systems modeling, identification and control (Ishii et al., 2004, Jaeger, 2004a, Ploger¨ et al., 2003).

Despite the simplicity of the training strategies for ESN, the construction of ESN, especially the reservoir and input/feedback connections, seems to always require numerous trials and even luck. This is because the free parameters, such as the size, spectral radius, sparsity of the reservoir weight matrix, the number of inputs/feedbacks, if needed, and all the connection weights must be appropriately selected for the network to generate satisfactory performance. Sometimes, the resulting ESN for a practical problem may have a heavy computation load for real-time and embedded applications (Prokhorov, 2005); this drawback is mainly due to handling a reservoir of large size in either learning or exploitation. Moreover, ESN with a single reservoir is not omnipotent. As shown in Jaeger (2004b), it was not possible for an ESN to be trained to work as a multiple superimposed oscillator (MSO), even when the function consists of only two sine waves, for example, sin(0.2n)+sin(0.311n). In Wiestra, Gomez, and Schmidhuber (2005), it was pointed out that the reason for the failure in the MSO problem is that all the neurons in the same reservoir are actually coupled, while the task requires the simultaneous existence of multiple decoupled internal states. To partially solve the above two problems, an evolutionary-based reservoir design method, called Evolino, was proposed in Schmidhuber, Gagliolo, Wierstra, and Gomez (2005) and Wiestra et al. (2005). It has been shown that Evolino is able to automatically generate task-dependent RNN with MSO capability, at the cost of significantly increased complexity imposed by the sub-network evolution routine.

In this paper, we propose a novel ESN structure, termed the decoupled echo state network (DESN) involving the use of lateral inhibition, to do the following: provide an alternative way of solving the MSO problem, reduce the computational complexity of the traditional ESN, and improve the probability of obtaining a satisfactory network under the condition of ‘random’ parameter selections. Unlike the evolutionary-based approach, the new structure makes use of multiple reservoirs that are competing with each other through inhibitory connections, and combines the internal states of all the reservoirs to form the output action potentials. In this way, the coupling effects between neurons within a single reservoir, as well as the correlation of neural dynamics between different reservoirs, are reduced, and multiple decoupled internal states are thereby generated to cooperatively accomplish a variety of tasks. Besides, owing to the coexistence of competitive and cooperative mechanisms, the problem of large performance variations caused by different choices of free parameters may be mitigated. In this paper, we develop two implementation schemes for DESN with lateral inhibition: one is termed DESN with reservoir prediction (DESN + RP), and the other is termed DESN with maximum available information (DESN + MaxInfo). Through experiments with computer-generated data and real-life noisy data, we observe that compared with a single-reservoir ESN, these two new structures are able to provide low-complexity but efficient solutions to the MSO problem and the predictive modeling task. Moreover, the performance of DESN with lateral inhibition is shown to be highly robust with respect to various reservoir configurations and randomly generated connection weights.

The rest of the paper is organized as follows. In Section 2, we mathematically formulate the multiple reservoir ESN problem after a short review of the echo state network, setting the stage for the development of DESN with lateral inhibition. Section 3 describes in depth the DESN with lateral inhibition, including the basic ideas and implementations of DESN + RP and DESN + MaxInfo. Section 4 presents three computer experiments: the MSO problem, a noiseless predictive modeling task and real-life sea clutter prediction; the results obtained therein demonstrate the performance and robustness improvements of our new approaches. Section 5 summarizes the paper and discusses future research topics.

Section snippets

ESN with multiple reservoirs

The proposed ESN structure with multiple reservoirs has been inspired by comparing the MSO modeling task with the classic kernel-based signal representation. For example, in the Fourier transform and wavelet transform, the individual kernel (i.e. basis function) could represent an almost unique portion of the target signal. Moreover, as shown in Jaeger (2001), an ESN with a single reservoir can be trained to be a sine-wave generator. Therefore, a natural idea to solve the MSO modeling problem

DESN with lateral inhibition

Lateral inhibition is a well-known mechanism, which was originally discovered in neurobiological systems including the retina, cochlear, and pressure sensitive nerves in skin (Arbib, 1989, Fischler and Firschein, 1987, Haykin, 2004). Specifically, when a neuron is stimulated, the neural activity of its surrounding region is suppressed via lateral inhibitory synapses. In this way, different neurons would possess various receptive fields, i.e., they will respond to stimulations of different

Computer experiments

In this section, we study the performance of DESN + RP and DESN + MaxInfo for three scenarios: the classic MSO problem, a challenging nonlinear prediction task, and a real-life sea clutter prediction task. The parameters pertaining to the network structure and training/testing strategy are shown in Table 1. Other parameters, along with the ones listed in Table 1, will be described before presenting simulation results. For all three scenarios, the training strategy used is the classic offline

Summary and discussion

In the newly proposed DESN with lateral inhibition structure, the harmonious competition and cooperation between the multiple reservoirs offer us a novel MSO problem-solving capability, low computational complexity and a strong robust behavior. The MSO problem is solved through the incorporation of a lateral inhibition mechanism in DESN as a simple but efficient method to reduce the coupling effects between neurons within the same reservoir and the correlation of the reservoirs’ activities. The

Acknowledgements

The authors would like to express their gratitude to two anonymous reviewers for their valuable suggestions and comments, which have greatly improved the presentation of this paper. This work was supported by National Science and Engineering Research Council of Canada (NSERC) and Canadian Institute of Health Research (CIHR).

References (16)

  • M.A. Arbib

    The metaphorical brain

    (1989)
  • Bush, K., & Tsendjav, B. (2005). Improving the richness of echo state features using next ascent local search. In Proc....
  • K.I. Diamantaras et al.

    Principal component neural network: Theory and application

    (1996)
  • M.A. Fischler et al.

    Intelligence: The eye, the brain and the computer

    (1987)
  • S. Haykin

    Neural networks: A comprehensive foundation

    (2004)
  • Ishii, K., van der Zant, T., Becanovic, V., & Plöger, P. (2004). Identification of motion with echo state network. In...
  • Jaeger, H. (2001). The echo state approach to analysing and training recurrent neural networks. GMD report 148....
  • H. Jaeger

    Adaptive nonlinear system identification with echo state network

There are more references available in the full text version of this article.

Cited by (189)

View all citing articles on Scopus
1

These authors contributed equally to this paper.

View full text