Neural Networking of Robotics

Introduction:

The mysterious biological intelligence of living creatures has long attracted us to explore their capabilities from perceiving, memorizing, to thinking, and then resulting in languages and Behaviors. To be specific, traditional model based control methods via numerical techniques, kinematics and dynamics approaches often fail to adapt to unknown situations (Ijspeert, 2008; Yu et al., 2014; Bing et al., 2017).

Theoretical Background:

Before studying in deep of the robotics control based on SNNs, it is worth briefly summarizing the biological mechanisms taking place in human nervous system. Since the initial discovery of neurons as the basic structure of the nervous system by Santiago Ramón y Cajal at the beginning of the twentieth century, a rough concept of how neurons might work has been Developed.

The structure of a typical neuron of the human brain embedded in a salty extra-cellular fluid 1A. Incoming signals from multiple dendrites alter the voltage of the neuronal membrane. Once the membrane potential threshold has been reached and the neuron fires, the generated output spike is transmitted via the axon of a neuron 1B.

At the presynaptic membrane, the triggered vesicle will fuse with the membrane and release its stored neurotransmitters into the synaptic cleft filled with the extra-cellular fluid. From McCulloch Pitts to Backpropagation. In 1943, neurophysiologist Warren McCulloch and mathematician Walter Pitts wrote a theoretical paper on how neurons might work describing a simple neural network model using electrical circuits (McCulloch and Pitts, 1943).

Spiking Neural Networks:

Following its biological counterpart, a third generation of neural networks (Maass, 1997, 2001) has been introduced that directly communicates by individual sequences of spikes.

A major Drawback are as follow, First, training artificial neural networks is time consuming (Krogh and Vedels by, 1995) and can easily take multiple days for state-of-the-art architectures (Lee C. S. et al., 2016). Training large-scale networks is computationally expensive [AlphaGo 1,202 CPUs and 176 GPUs (Silver et al., 2016)], and running them typically produces high response latencies (Dong et al., 2009). Second, performing computations with large networks on traditional hardware usually consumes a lot of energy as well.

Biological Plausibility :

Experimental evidence accumulated during the last few years has indicated that many biological neural systems use the timing of single-action potentials (or “spikes”) to encode information (Maass, 1997), rather than the traditional rate-based Models.

Speed and Energy Efficiency Despite the hardware upgrades that make large neural networks applicable to real-world problems, it usually does not apply to robotics platforms with limited energy and computing resources. Neurobiologists used weekly electric fish as a model to study the processing from stimulus encoding to feature extraction (Gabbiani et al., 1996; Metzner et al., 1998).

Research Directions:

To be specific, first, the architecture and mathematical model of an SNN should be determined including the neuron and synapse. Neurons are known to be a major signaling unit of the nervous system, and synapses can be seen as signal transmitters that communicate among neurons. The basic task is to determine the general topological structure of the SNN, as well as the neuron models in each layer of the SNN.

Neuron Models:

The most influential models used for SNNs are the Hodgkin-Huxley model (Hodgkin and Huxley, 1952) as well as the Integrate-and-Fire model and its variants (Burkitt, 2006)

Information Encoding and Decoding:

A number of neural information encoding methods have been proposed, such as binary coding (Gütig and Sompolinsky, 2006), population coding (Probst et al., 2012), temporal coding, and the most commonly used rate coding (Urbanczik and Senn , 2009; Wade et al., 2010). For binary coding, neurons are only modeled to take two values on/off , but it ignores the timed nature and multiplicity of spikes Altogether. For rate coding, it is inspired by the observation that neurons tend to fire more often for stronger (sensory or artificial) stimulus. For temporal coding, it is motivated by the evidence founded in neuroscience that spike-timing can be remarkably precise and reproducible Gerstner et al. (1996).Based on an input-output relationship between neuronal activity and synaptic plasticity, they are roughly classified into two types, which are rate-based and spike based, that differ in the type of their input Variables. The rate-based model is a popular approach for converting conventional ANNs into a spiking neural network that can still be trained by Backpropagation. Spike-based learning rules were developed in Gerstner et al. (1993), Ruf and Schmitt (1997), Senn et al. (1997), Kempter et al. (1999), and Roberts (1999). Experiments showed that the synaptic plasticity is influenced by the exact timing of individual spikes, in particular, by their order (Markram et al., 1997; Bi and Poo, 1998).

Network Models :

The SNN network model resembles the synapse model in that it simulates synaptic interactions among neurons.

Feed-Forward Networks:

As the first and simplest type of network topology, information in feed forward networks always travels from the input nodes, through hidden nodes (if any), to the output nodes and never goes Backwards. Different from the feed-forward networks, recurrent neural networks (RNNs) transmit their information with a directed cycle and exhibit dynamic temporal Behaviors. Living organisms seem to use this mechanism to process arbitrary sequences of inputs with their internal memory stored inside RNNs.

Learning and Robotics Applications:

Initially, solving simulated control tasks was done by manually setting network weights, e.g., in Lewis et al. (2000) and Ambrosano et al. (2016). However, this approach is limited to solving simple behavioral tasks such as wall following (Wang et al., 2009) or lane following (Kaiser et al., 2016), it is usually only feasible for very small network architectures with few weights. Finally, some alternative methods on how to train and implement spiking neural networks are discussed.

Hebbian-Based Learning:

Hebbian-based learning rule that rely on the precise timing of pre and post synaptic spikes play a crucial part in the emergence of highly non-linear functions in SNNs. Several models have been proposed on how this might work, either by using activity templates to be reproduced (Miall and Wolpert, 1996) or error signals to be minimized (Kawato and Gomi, 1992; Montgomery et al., 2002).One of these models that is primarily suitable for single-layer networks is called supervised Hebbian learning (SHL).

Supervised Learning:

This type of learning, where a neural network mimics a known outcome from given data is called supervised learning (Hastie et al., 2001). A variety of different neuroscientific studies has shown that this type of learning can also be found in the human brain (Knudsen, 1994), e.g., in motor control and motor learning (Thach, 1996; Montgomery et al., 2002).This can be a very useful property for robot control, because it might simplify the requirements of an external training signal leading to more complex tasks.

Temporal Difference:

The learning rule in which one looks at one or more steps forward in time was introduced as temporal difference (TD) learning. Hereby, Potjans et al. (2009) and Frémaux et al. (2013) used place cells to represent the state space in an MDP and single-layer SNNs for state evaluations and policies.??

Evolutionary Algorithms:

Based on these ideas, a class of algorithms has been developed for finding problem solutions by mimicking elementary natural processes called evolutionary algorithms (Michalewicz, 1996). Floreano and Mattiussi (2001) showed a visionbased controller in an irregularly textured environment that navigated without hitting obstacles. Hagras et al. (2004) later extended this approach to evolving SNN weights as well using adaptive crossover and mutation probabilities.

Simulators and Platforms:

Meanwhile, a growing number of dynamic simulators has been developed to assist robotic research (Ivaldi et al., 2014), such as Gazebo (Koenig and Howard, 2004), ODE (Wikipedia, 2017c), and V-Rep (Rohmer et al., 2013). Those simulators greatly facilitate the research process that involving mechanical design, virtual sensors simulation, and control architecture.

Designing and Training SNNs:

There is no general design framework that could offer the functionalities of modeling and training, as well as those substantial tools for the conventional ANNs do, for instance, Tensorflow (Allaire et al., 2016), Theano (Theano Development Team, 2016), and Torch (Collobert et al., 2011).Training should strengthen the combination with the burgeoning technologies of reinforcement learning, for instance, extending SNN into deep architecture or generating continuous action space (Lillicrap et al., 2015).Interdisciplinary Research of Neuroscience and Robotics. Roboticists often use a simplified brain model in a virtual robot to make a real-time simulation, while neuro-scientists develop detailed brain models that are not possible to be embedded into the real world due to their high complexity.

Conclusion:

Therefore in this article, we seek to offer readers a comprehensive review of the literature about solving robotic control tasks based on SNNs as well as the related modeling and training approaches, and meanwhile offer inspiration to researchers. Finally, some popular interfaces or platforms for simulating SNNs for robotics are preliminarily investigated’ Therefore, more knowledge and interactions from the fields of neuroscience and robotics are needed to explore this area in the future.?