Optimization of Neural Network Weights with Nature Inspired Algorithm

Journal: Journal of Machine Learning and Deep Learning (JMLDL), Volume:1, Issue:1, Pages: 1-9 Download pdf

Authors: Yusra Shereen

Date: 8-2024

Abstract: Neural networks are machine learning algorithms inspired by the human brain regarding structure and function. The artificial neural network (ANN) performs well at tasks on which other conventional approaches fail. They play a crucial role in knowledge representation and learning. The strength of the connection between neurons is determined by weights. Weight optimization is critical to neural networks due to several reasons, more specifically it enables more accurate predictions and reduces loss. A variety of optimization procedures are used for weight optimization. Gradient-based algorithms are the widely used method for optimization of neural network weights, but they are unable to tackle non-differentiable functions. Moreover, gradient-based algorithms may trap in local minima for non-convex functions. Many significant real-world problems have non-convex characteristics, so utilizing gradient descent can cause algorithms to be stuck in local minima. In this paper, we proposed a novel gradient-free approach for optimizing neural network weights utilizing a genetic algorithm. The genetic algorithm is a meta-heuristic algorithm based on the natural evolution process. It can be used to solve both constrained and unconstrained optimization problems. Hence it can solve the problem of convergence for non-differentiable functions and can lead solutions towards global optima. Additionally, we proposed an algorithm to optimize neural network weights utilizing a genetic algorithm. We proved the correctness of our algorithm using the loop invariant technique. Moreover, computational cost analysis is presented for the proposed algorithm. Lastly, we utilized the MNIST dataset for demonstration of our proposed approach. Genetic algorithm capabilities of global search can overcome issues of local minima trapping of non-convex functions. 

 Keywords: artificial neural network; optimization; weights; gradient free; Neural Network; Genetic Algorithm; neural networks; non convex functions; gradient descent

References:

[1]Datacamp, (March 2022), “Gradient Descent Tutorial” https://www.datacamp.com/tutorial/tutorialgradient-descent

[2]  Datajobs, “Gradient Descent”, https://datajobs.com/datasciencerepo/Gradient-Descent-[RIT].pdf

[3]      Junxiang Wang, Hongyi Li, Liang Zhao, ”Accelerated Gradient-free Neural Network Training by Multi-convex Alternating Optimization”, Neurocomputing, Volume 487, 2022, Pages 130-143, ISSN 0925-2312, https://doi.org/10.1016/j.neucom.2022.02.03 9

[4]      A. Aly, G. Guadagni and J. B. Dugan, ”Derivative-Free Optimization of Neural Networks using Local Search,” 2019 IEEE 10th Annual Ubiquitous Computing, Electronics & Mobile Communication Conference (UEMCON), 2019, pp. 0293- 0299, doi: 10.1109/UEMCON47517.2019.8993007

[5]      S. Chaudhury and T. Yamasaki, ”Robustness of Adaptive Neural Network Optimization Under Training Noise,” in IEEE Access, vol. 9, pp. 37039-37053, 2021, doi: 10.1109/ACCESS.2021.3062990

[6]      Kumar Chandar, S. “Grey Wolf optimization-Elman neural network model for stock price prediction,” in Soft Computing 25, pp. 649–658, 2021, doi: https://doi.org/10.1007/s00500-020-05174-2

[7]       Moayedi, Hossein & Nguyen, Hoang & Foong, Loke. “Nonlinear evolutionary swarm intelligence of grasshopper optimization algorithm and gray wolf optimization for weight adjustment of neural network.” Engineering with Computers. 2021, 37. 10.1007/s00366-019-00882-2.

[8]      Utkarsh Agrawal, Jatin Arora, Rahul Singh, Deepak Gupta, Ashish Khanna, and Aditya Khamparia. 2020. Hybrid Wolf-Bat Algorithm for Optimization of Connection Weights in Multi-layer Perceptron. ACM Trans. Multimedia Comput. Commun. Appl. 16, 1s, Article 37 (January 2020), 20 pages. https://doi.org/10.1145/3350532

[9]      Tripathi, R., & Singh, B. (2020). “RSO: A Gradient Free Sampling Based Approach For Training Deep Neural Networks.” arXiv. https://doi.org/10.48550/arXiv.2005.05955

[10]   Kannappan, K.; Spector, L.; Sipper, M.; Helmuth, T.; La Cava, W.; Wisdom, J.; Bernstein, O. Analyzing a decade of humancompetitive (“HUMIE”) winners: What can we learn? In Genetic Programming Theory and Practice XII; Riolo, R., Worzel, W.P., Kotanchek, M., Eds.; Springer International Publishing: Cham, Switzerland, 2015; pp. 149–166

[11]   Sipper, M.; Olson, R.S.; Moore, J.H. Evolutionary computation: The next major transition of artificial intelligence? BioData Min. 2017, 10, 26.

[12] Towards data science, “Various optimization algorithms for neural networks” https://towardsdatascience.com/optimizersfor-training-neuralnetwork-59450d71caf6

[13]   Clara Marina Mart´ınez, Dongpu Cao, “Integrated energy management for electrified vehicles,” in Ihorizon-Enabled Energy Management for Electrified Vehicles, Butterworth-Heinemann, 2019, pp 15-75, ISBN 9780128150108, https://doi.org/10.1016/B978-0-12-815010- 8.00002-8

[14]   Mazaheri, P., Rahnamayan, S., & Bidgoli, A. A. (2022). Designing Artificial Neural Network Using Particle Swarm Optimization: A Survey. In (Ed.), Swarm Intelligence - Recent Advances and Current Applications [Working Title]. IntechOpen. https://doi.org/10.5772/intechopen.106139

[15]   Lin, Cheng-Jian, Garro, Beatriz A., Vazquez, Roberto A. (2015), ´ “Designing Artificial Neural Networks Using Particle Swarm Optimization Algorithms”, in Computational Intelligence and Neuroscience, Hindawi Publishing Corporation, ISSN - 1687-5265, https://doi.org/10.1155/2015/369298

[16]   Francisco Erivaldo Fernandes Junior, Gary G. Yen, “Particle swarm optimization of deep neural networks architectures for image classification,” in Swarm and Evolutionary Computation, Volume 49, 2019, pp 62-74, ISSN 2210-6502, https://doi.org/10.1016/j.swevo.2019.05.010

[17]   Lapid, R., Haramaty, Z., & Sipper, M. (2022). An Evolutionary, Gradient-Free, Query-Efficient, Black-Box Algorithm for Generating Adversarial Instances in Deep Convolutional Neural Networks. Algorithms, 15(11), 407. https://doi.org/10.3390/a15110407  

[18]   Malik, S., Tahir, M., Sardaraz, M., & Alourani, A. (2022). A Resource Utilization Prediction Model for Cloud Data Centers Using Evolutionary Algorithms and Machine Learning Techniques. Applied Sciences, 12(4), 2160. https://doi.org/10.3390/app12042160

[19]   Hong Zhang, Hoang Nguyen, Xuan-Nam Bui, Trung Nguyen-Thoi, Thu-Thuy Bui, Nga Nguyen, Diep-Anh Vu, Vinyas Mahesh, Hossein Moayedi, Developing a novel artificial intelligence model to estimate the capital cost of mining projects using deep neural network-based ant colony optimization algorithm, Volume 66, 2020, ISSN 0301-4207, https://doi.org/10.1016/j.resourpol.2020.101 604.

[20]   Solomon Netsanet, Dehua Zheng, Wei Zhang, Girmaw Teshager, Shortterm PV power forecasting using variational mode decomposition integrated with Ant colony optimization and neural network, Energy Reports, Volume 8, 2022, Pages 2022-2035, ISSN 2352-4847, https://doi.org/10.1016/j.egyr.2022.01.120

[21]   Xiaobo Zhao, Dongji Xuan, Kaiye Zhao, Zhenzhe Li, Elman neural network using ant colony optimization algorithm for estimating of state of charge of lithium-ion battery, Journal of Energy Storage, Volume 32, 2020, 101789, ISSN 2352-152X, https://doi.org/10.1016/j.est.2020.101789