Learning Combination of Activation Functions and Various Optimizers with Loss Function for Improved Bi-LSTM Lexicon Embedding Model

Deep Learnings has got a lot of prominence due to its advance results in various fields like Computer Vision, Natural Language Processing, Time Series Analysis, Health Care etc. In the last decade, an active area of research has been devoted to design novel activation functions that are able to help deep neural networks to converge, obtaining better performance. The training procedure of these architectures usually involves optimization of the weights of their layers only, while non-linearity’s are generally pre-specified and their (possible) parameters are usually considered as hyper-parameters to be tuned manually. Earlier, the Deep Learning was implemented using the batch and stochastic gradient descent algorithms and some optimizers which lead to very less performance of the models. But today, lot of work is going on for the enhancement of the performance of Deep Learning using various optimization techniques. So, in this context, it is proposed to build a Deep Learning model using various Optimizers (Adagrad, RmsProp, Adam, Adamax, SGD), Loss functions (mean squared error, binary cross entropy) and various activation functions (Relu, Elu, Softmax, Sigmoid, Exponential, Tanh) with different input-size and no. of epochs, for the Convolutional neural networks and Recurrent neural networks and verify the performance such as Accuracy and Loss of the model. The proposed model has achieved maximum Accuracy when Adam optimizer and binary cross entropy loss function and Relu activation function with an input size of 250 and 20 epochs are applied on Recurrent neural networks. This paper reports a behavioural analysis and accuracy comparison of improved Bi-LSTM (Long-Short Term Memory) and LSTM models. The objective is to explore to what extend additional layers of training of data would be beneficial to tune the involved parameters. Here we have Proposed comparison between different algorithms and the experiment results are shown the recurrent neural network using LSTM for the trained data model that is scored the highest accuracy of 97%, 86%, and 89% for the Amazon Cell review, IMDB and Yelp dataset respectively. Experimental result of three public benchmark datasets show our proposed model yields obvious performance improvement.