Recently, the research findings entitled “Mammalian-brain-inspired neuromorphic motion-cognition nerve achieves cross-modal perceptual enhancement” from the College of Electronic Information and Optical Engineering at Nankai University were published in the world-renowned academic journal Nature Communications. According to this paper, flexible artificial synapse devices are used to develop a neuromorphic motion perception system, which realizes the brain’s multi-sensory functions at hardware level and obtains excellent motion perception performance. Dr. Jiang Chengpeng, from the College of Electronic Information and Optical Engineering at Nankai University, is the first author, and Professor Xu Wentao, from the College of Electronic Information and Optical Engineering at Nankai University, is the only corresponding author.
The design of the neuromorphic motion perception system is inspired by the macaque’s multisensory integration and spatial perception mechanisms (Fig. 1). A macaque’s self-motion will stimulate motion information such as inertial signals and optical flow signals in the vestibule and the retina. The specific areas of the cerebral cortex will process and identify the motion information encoded as spike pulse, and then realize spatial perception by integrating information from different sensory modalities. Accelerometers and gyroscopes in the neuromorphic motion perception system acquire acceleration and angular velocity signals, respectively, which are encoded into two spike trains that are transmitted to high-performance synaptic transistors for processing. The correlation between the two pulse sequences as well as their temporal relationship affect the synaptic plasticity of the device, thereby affecting the device output. The motion signals are classified and identified by evaluating the average firing rate of the pulse and the output current of the synaptic device.
Fig. 1. a. Mammalian’s ocular/vestibular cross-modal motion sensory system; b. neuromorphic motion perception system based on flexible synaptic transistors.
Furthermore, optical flow sensor, vibrotactile sensor, and inertial sensor are used to construct a sensing unit, which can detect sensory information of visual, tactile, and acceleration modes. Information from different types of sensors can be effectively integrated, thereby significantly enhancing the accuracy of motion recognition (higher than 94%). The experimental results are consistent with the perceptual enhancement effect of the brain. Moreover, the system can be attached to human skin or equipped on small drones to perform complex tasks such as recognition of human motion and drone flight pattern thanks to its features of wearability, high integration and low power consumption.
Essentially, the system mimics the process of sensory cue integration in the mammalian brain, and realizes brain-like motion perception by employing the spike coding strategy for sensory signals, the spike-integration characteristics of synaptic devices, and the spatiotemporal recognition method of synaptic current signals. This work combines neuromorphic cognitive intelligence with the brain’s multimodal perception mechanism, and is of guiding significance for the development of brain-inspired devices and biomimetic electronics. It can be applied to various fields such as mobile robots, intelligent wearable devices, and human-machine interface.
Nankai University is the first affiliation of the authors.