A Neural Attention Models Survey for Deep Learning

Attention is a fundamental component of all perceptual and cognitive processes in humans.This mechanism select, modify, and concentrate on the information that is most important to behaviour because of our limited capacity to interpret competing sources. For many years, researchers in the fields of philosophy, psychology, neuroscience, and computing have examined the notions and functions of attention.This characteristic has been thoroughly investigated in deep neural networks over the past six years. In many application domains, neural attention models currently represent the cutting edge of deep learning. A thorough overview and analysis of recent advancements in neural attention models are provided in this survey. In order to identify and examine the architectures where attention has had a notable impact, we thoroughly reviewed hundreds of them in the region. Additionally, we created and made available an automated approach to aid in the growth of reviews in the field. We discuss the main applications of attention in convolutional, recurrent networks, and generative models. We also identify common subgroups of uses and applications. Additionally, we discuss the effects of attention across various application areas and how they affect.