Evolving Convolutional Neural Networks

The following is an unpublished research paper written by myself, Neelank Swamy, Conner Bissell, Bryan Smith, and Gary Barnes, back in 2020. The project was done as part of our Evolutionary Computation course at the University of Georgia. I primarily was the one writing the code and doing the experimentation, especially related to the architectural evolution, but I discussed often with the others, and they were an enormous help with writing the paper.

Abstract

As biological imaging technologies become increasingly precise, and computation continues to grow exponentially in both hardware and software methods, using machine learning to analyze this growing data will become vital to the future of medicine. A large time-sink for creating machine learning models, particularly neural networks, can be tuning hyper parameters to somewhat optimal values for it to learn efficiently and effectively. Likewise, the supervised learning process can be lengthy. We propose that using an evolutionary computational approach to tune the hyper parameters of the Convolutional Neural Networks (CNNs) will increase both the learning speed and the final performance of the algorithm. The process of using evolutionary algorithms to generate and tune neural networks is known as neuroevolution. Thus, in this paper, we explore using evolutionary algorithms to traverse the search space of hyper parameters for CNNs that classify MRI brain scans as containing a brain tumor or not. Evolutionary Algorithms have been used with strong effects both on CNNs and other types of neural networks. We found them to be successful here as well.

Introduction

Artificial Intelligence applications in the medical field have been growing at an explosive rate over the last several years. One of the most promising applications is the use of Convolutional Neural Networks (CNNs) to aid diagnosis via the analysis of Magnetic Resonance Imaging scans and other medical imaging technologies. CNNs are neural networks which are particularly designed (and very effective) for image analysis. The success of this structure lies in its ability to learn and apply multiple image filters as well as other weights to image classification[1]. CNNs have proven integral to the improvements in computer vision we have been able to achieve.

If Artificial Neural Networks are meant to mimic the brain, then it can be fairly said that CNNs are designed to mimic the eyes and the brain's visual cortex. We propose that, like the eye, it could be improved through exposure to selection pressure which would result in adaptation. Our proposal is to use multiple optimizers from the Keras package as their own populations that evolve utilizing combined aspects of evolutionary algorithms such as Evolutionary Strategies and Genetic Algorithms. The input to each CNN would be the training dataset of MRI brain scans while the output would be a probability prediction that an MRI brain scan contains a brain tumor. We utilize several different Evolutionary Computation methods to see which yields the best results and how well LeNet can be effectively adapted to the purpose of MRI image analysis for brain tumor diagnosis through evolutionary computation.

This process is known as neuroevolution and is an alternative the typical approaches of manual or automated parameter tuning. Neuroevolution is essentially a form of parameter control, as it creates the ability to let the algorithm tune itself through selective pressure. An important difference between neuroevolution and simple parameter tuning is that we maintain and evolve a population of solutions, which in turn allows an automated and thorough exploration of the hyperparameter solution space.[6]

Figure 1: A scan with a tumor (left) and one without (right). Images taken from our Data Set

As Kelly notes, early detection is integral to effective treatment for brain tumor patients. And the faster treatment is administered, the more likely a patient is to recover[2]. Diagnosis is not always as trivial as the above image, and it often requires careful and time-consuming analysis. When in the business of treating life-threatening conditions, every hour matters. If CNNs can be evolved to make them even more impactful in the early detection of any sort of tumor, then we can help to save lives and improve patient outcomes. There has already been some research into the application of CNNs for brain scan classification, as well as the use of genetic algorithms on the CNNs[3]. Anaraki et al. investigated using the well-known CNN architecture AlexNet. Due to computational limitations, we used a simpler LeNet CNN architecture.

We were able to adapt the LeNet architecture to this problem space quite well using our approach, and consistently achieved fitnesses (accuracies) in our models of 85% or better for both implemented optimizers (Adam and Stochastic Gradient Descent). We noticed some runs had extremely fast convergence and others took quite some time. We attribute this to both the inherent stochasticity of our structure and the relatively small size of our generations. We also saw several individuals score perfectly on the training sets, but attribute this to overfitting as we had no individuals with perfect final fitnesses.

Background and Methodology

Dataset

Our dataset came from Kaggle.com and consists of 253 images of MRI brain scans. 98 of these images are classified by the source as “no” indicating that no tumor is present in the scan, and the remainder are classified as “yes” indicating the presence of a tumor. These images are formatted as .jpgs, and were aggregated by Navoneel Chakrabarty pulling images from Google images.

Equipment

Our algorithms were executed on two PC's, and were configured to take advantage of the Nvidia CUDA architecture via the Python Tensorflow package. The CUDA model runs on Graphics Processing Units (GPUs) and takes advantage of the GPUs optimized architecture for graphics processing, which allows it to excel at floating point calculations and parallel computation. The first PC utilized a Nvidia GeForce GTX 1060 GPU with 6 GB of DDR5 RAM, a clock speed of 1632 MHz, and 1280 CUDA cores. This machine also contained an Intel Core i7-7700K CPU with four cores running at 4.2Ghz and capable of support 8 threads. The second PC made use of an Nvidia GeForce GTX 1080 GPU with 8 GB of GDDR5X RAM, a clock speed of 1733 MHz, and 2560 CUDA cores. Additionally, this PC contained an Intel Core i7-8700K with six cores clocked to 5.2Ghz and the ability to support 12 threads. The CUDA functionality allowed for greatly enhanced model training times over CPU- based methods, even compared to a relatively powerful CPU such as the one in our machine. Even with a strong GPU, this was a limiting factor in our research, as our models consistently outgrew the available memory on our configuration for populations of μ>6, and λ>18.

Evolutionary Algorithms

Using the Distributed Evolutionary Algorithm in Python (DEAP) package, we implemented an evolutionary algorithm using several approaches. Our main strategy adopted an Evolution Strategies approach with gaussian perturbation as the mutation operator for the optimizer hyperparameters. Additionally, two individuals were able to become “parents” and create an “offspring” through crossover operators that either weight averaged each parent's object variables (with random weights), or selected a random parent's value for each object variable. Mutation had a 50% chance of occurring while weighted averaging crossover had a 25% chance, and select crossover had a 25% chance.

Other mutations were possible however, as we did allow for architectural mutations (in addition to the optimizer mutation/crossover) for the CNNs in a few runs. In these runs, we allowed for a low probability of inserting a random new layer (2.5% chance), copying an existing layer into a new position (2.5%), moving an existing layer (2.5%), and removing a random layer (5%). Furthermore, each CNN layer had chances of slightly mutating its hyperparameters (such as a Convolutional layer's kernel size or a Dense layer's number of nodes). These architectural mutations required careful attention to ensure the CNNs did not become un-trainable (such as by removing the Flatten layer, increasing the kernel size to above the input shape, etc.)

As a result of our limited computational resources, we decided to limit our initial population to six classifiers (μ = 6), and to generate eighteen offspring for each generation (λ = 18). These values were based on manual experimentation done by tweaking the values until we found a set that would run consistently on our hardware configuration.

Each individual was represented as a Python Dictionary containing an optimizer with corresponding parameters, strategy variables and a CNN architecture (either evolved or LeNet). This is visualized in figure 2 below.

Figure 2: a sample individual representation, where x represents a parameter for the optimizer, σ represents a strategy variable, and θ represents a CNN architecture.

We used two different optimizers for our CNN. The first was Adaptive Movement Estimation (Adam). The second was Stochastic Gradient Descent (SGD). Each of these optimizers works to optimize the CNN's learning process. Our algorithm optimizes the parameters of the optimizers. Adam is a method that leverages adaptive learning rates for every parameter. It stores a decaying average of previous gradients to avoid getting stuck in local optima/minima[5]. In this context, an algorithm's ability to avoid local optima is referred to as “momentum”. SDG by contrast, is simply a gradient descent algorithm with a momentum optimizer.

The overarching approach we adopted is referred to as neuroevolution. This process works to enhance the already impressive performance of CNNs by evolving the hyperparameters in the CNN optimizers as well as the underlying architecture. This allows evolutionary pressure to guide the model's development more effectively than traditional reinforcement learning strategies, as it enables a wider range of approaches to optimizing the CNN. Where reinforcement learning typically uses some form of gradient descent algorithm and a fixed architecture, our algorithm will make online changes to both the hyperparameters and architecture allowing for the individuals to better explore the search space[6].

We performed several variations on our experiment for comparison. We did 50 runs each with a (μ+λ), (μ, λ), and Simple Genetic Algorithm (SGA) using the Adam optimizer, and 50 runs on a (μ+λ) model with the SGD optimizer. The selection method used in the SGA was Tournament Selection with k = 5, and the initial population was set at 10. For each of our runs we did 20 epochs of training time for each individual and, depending on the algorithm used, each run took from four to ten minutes to complete. Our fitness was the accuracy of our model, with a perfect classifier being assigned a score of 1, and the scores being floating point values in the range [0,1].

Convolutional Neural Networks

Convolutional Neural Networks are a type of deep neural network which specializes in image analysis. To perform classification on the images in our dataset we used LeNet CNNs. CNNs are a species of feed-forward neural network particularly well suited to image processing. The LeNet algorithm is well known and established in the field. This structures were built using library functions from Tensorflow and Keras in Python. LeNet, a predecessor to AlexNet, contains seven layers. The basic architecture is laid out in Figure 3 below and illustrates the broad strokes of data processing in a CNN quite well.

Figure 3: A sample LeNet Architecture with image input on the left, illustrating passing through the convolutional and pooling layers in the center, and finally the fully connected and output layers on the right.(Page 250, [4])

In our LeNet architecture, layers one and three are convolution layers, and in our implementation we the layers have 6 and 16 filters respectively, each with kernel size 3×3. The second and fourth layers are pooling layers using Average Pooling. Finally, the fifth, sixth, and seventh layers are dense, fully connected layers which ultimately provide the classification output. Images were formatted to a 224×224 resolution for processing. LeNet was originally designed for character recognition, but we were able to adapt it with good results to our purposes, as discussed later in the paper.

Results and Analysis

Through the course of our research, we determined that not only can neuroevolution be applied the space of image classification for MRI scans, it can also be done quite effectively. For each generation, we captured the best and worst individuals, the standard deviation of individual fitnesses in the final generation, and the rate of improvement.

Figure 4: The mean accuracy (fitness) of each generation across our 50 runs for each algorithm.

As illustrated above, we determined that a (μ+λ) strategy combined our evolved optimizer provided the best results. The (μ, λ) and SGA strategies achieved nearly identical results, but the (μ, λ) typically began with much worse individuals than did the SGA. Our poorest performer was the evolved Stochastic Gradient Descent optimizer with a (μ+λ) strategy. All algorithms with the exception of SGA typically saw the largest jumps in fitness over the first 5 or so generations before beginning to slow down. The SGA also saw more improvement in the first few generations, but started at a higher mean fitness level than did its competitors, so the improvement is more gradual overall, and relatively flat from about generation seven onwards, on average.

Table 1: The overall best and mean fitness for each approach implemented

Table 1 shows the supports the observations drawn from figure 4 and shows something interesting: each approach hit the same upper limit for fitness. The inherent stochasticity of evolutionary computation makes this all but inevitable. The main difference in the approaches was with the speed and consistency of hitting that upper bound. Further research will be required to see if we get the same results on a different dataset.

Another way to measure the success of an algorithm is to measure its improvement over the course of its run. We captured that data and display it below in Figure 5.

An interesting observation is that while the SGD models did not even come close to the best accuracy, they do show the most improvement over our 16 generation runs. Further research with greater computational power will be required to determine what happens with more generations. Figure 5 mirrors closely the observations made in Figure 4, that the SGA's development is rather stagnant. Perhaps other mutation or selection operators would yield different results. We leave this for future research. We can also see that not only does our (μ+λ) strategy yield the best mean accuracy, but it also yields the most improvement in the smallest time. This is useful for computationally restricted researchers like ourselves.

In conclusion, we were able to achieve promising results for the application of neuroevolution to the problem of classifying MRI brain scans. Our evolved Adam optimizer was able to consistently outperform the evolved SGD optimizer for this domain.

Literature Review and Related Work

Considering the success of CNN's in artificial intelligence tasks such as image recognition, and the ability for GA's to find good solutions among a large range of multi-objective optimization problems, it should come as no surprise that these two have been used in conjunction in recent research. In [7], the authors go a step further than us, and use a Genetic Algorithm with individuals of variable-length. In the context of this problem, this means that the number of network layers in the CNN is not fixed but is varied in each individual during the run. The authors justify this by asserting that there is no reliable way to know how many layers is needed for a particular problem. Too few layers, and one could have issues with underfitting; too many layers, and one would almost certainly run into the issue of overfitting. We also had this idea of having individuals with evolved architectures and implemented it in our project. Due to technical limitations, we were able to collect little data with evolved architectures as many algorithm runs failed to complete. Regarding this, the authors stated that it took 250 computers running in parallel and 250 hours to complete a run. If we had more time and computing power, we would have further explored the results of these architectural evolutions. At the end of this paper, the authors mentioned that in the future, they would like to eliminate individuals that perform poorly in the first few epochs in order to save time and computing power. This is a great idea, and something we would also like to experiment with in the future.

In [8], the authors justify the use of GA's for this problem in the same manner that we do. Namely, the fact that optimizing the hyperparameters of a CNN manually is a time-consuming process that is not intuitive in the slightest without in-depth knowledge of both the CNN architecture being used and the problem at hand. Like the previously mentioned paper, these authors allow individuals to be of variable lengths. These authors also use metaheuristics to deal with the issue of dealing with training hyperparameters and how children inherit network structure during crossover. In particular, they use the GA to find the optimal number of layers and the optimal kernel size for the given dataset, and also implement a novel crossover operator that takes into account the structure of each individual before crossover, and seeks to preserve the filter number and filter sizes of the more fit parent. These authors also recognized the fact that optimizing hyperparameters can be seen as a multi-objective problem and used an incremental selection pressure to favor diversity in early generations.

In [9], the authors analyze the different ways in which CNN's perceive the visual illusions that also fool humans, in hopes to gain a better understanding of the differences between the human and CNN approach to vision. Interestingly enough, these authors find that CNN's are fooled by many of the same illusions that humans are fooled by. However, upon further research, it appears that the underlying mechanism that fools the CNN's is completely different from the underlying reasons that humans are fooled by these illusions. More specifically, as the size of the CNN's grew, they became increasingly unlike humans. This suggests that we should not use the heuristics about vision that we are familiar with to develop the architecture for image-recognizing CNN's, which is all the more reason to use GA's based on their capabilities as generalized problem solvers. In this paper, the authors were studying CNN's that were trained for the tasks of denoising, deblurring, or some combination of the two. These methods are also commonly used on medical imaging and would be an interesting extension of our work. If we were to choose to pursue this for future work, then the findings of the authors in this paper would likely prove to be extremely useful for us. Finally, these authors used a method called “simulated psychophysics,” in which they were able to compare the responses of CNN's and humans to the same images. This would be an interesting avenue for us to explore. In particular, we could compare the responses of trained medical professionals and CNN's to images of MRI's to both gain insight to how useful our CNN's are, and see if there are differences in the types of MRI's that humans and CNN's have difficulties working with.

In [10]*, *the authors seek to complete the task of medical image denoising by evolving CNN's with the use of GA's. Like us, the authors evaluate the of each CNN with an objective fitness function. In this case, the signal-to-noise ratio. This work is interesting to us, because the task of denoising is often the precursor to a more advanced task such as tumor recognition. Through this paper, the authors make a few interesting contributions to this area of research that we find particularly useful. In particular, they use an “experience based greedy strategy on the initialization stage to enrich high-performance genes and suppress defective ones in the following generation” (2). This experience-based greedy strategy is useful when computing resources are limited, which was certainly the case during our project. For this reason, we would like to explore this strategy during future work.

In [3], the authors seek to complete almost exactly the same task as us. In particular, they use a GA to optimize the hyperparameters for CNN's in order to classify Gliomas, a common type of tumor. Like us, these authors used shallow and fixed individuals, consisting of two max pooling, two fully connected, and two convolutional layers. Interestingly, when each run ended with the most fit individual, the authors did not simply take these individuals at face value. Instead, an ensemble method called bagging was used on the best individual at the end of the run in order to decrease the variance of the individual's final diagnose. Specifically, bagging (bootstrap aggregating) is a method used to reduce generalization error by combining multiple models, and is a subset of the general method of model averaging in Machine Learning. The motivation follows from the fact that different models do not usually create similar errors on the training set. In general, this concept of taking the best individual from the final result of the GA and using more specific methods based on our knowledge of CNN's and the problem at hand to further optimize the individual is an interesting prospect, and one we would be interested in exploring for future work.

Future Work

During this project, we were primarily limited by time and computing resources. Each run of 16 generations took between 4 and 12 minutes on a GPU depending on the algorithm used, and two to three times longer on CPUs. Limited memory on the GPUs forced us to limit population sizes accordingly or cause our algorithms to crash when the GPU's memory was completely allocated. Because of these limitations and relatively shallow networks, if we were to continue work on this project in the future, our first step would certainly be to increase the computing power available to us. This is further justified by [7], in which the authors required 250 parallel computers and 250 hours to complete one run with individuals. The idea of having CNN's with evolved architectures is something we would be interested in exploring during future research, which is one of the many reasons we would need to start by acquiring more computing power and a dedicated machine to perform our runs on. In addition to this, we would like to explore ways of making our runs less computationally expensive. In [7], the authors terminate individuals that perform poorly in the first few epochs. We could do something like this and just replace them with a copy of one of the more-fit individuals, or develop some intelligent repair operator that seeks to move individuals that do poorly towards some local optimum.

In [9], the authors seek to evaluate the performance of their CNN's with a method called “simulated psychophysics”, in which the responses of the CNN's to a set of inputs are compared against humans. We would like to explore this option, because it would allow us to gain insight into what images deceive our CNN's and could even allow us to optimize our CNN's so that they could supplement the weak points in the way that humans identify tumors.

In [3], the authors use a GA to optimize the hyperparameters of their CNN's, then take the best individual of each run and further optimize it. This is an interesting idea, and one we did not even consider since this semester has consisted entirely of looking at ways to optimize problems solely with GA's. This hybrid approach may be useful in this case and is one we would like to explore. In particular, we could go through a large number of runs, save the most fit individual from each one, and then optimize each individual in this population with techniques more specific to CNN's and whatever the problem is that we are working on. While doing this, we could also treat the optimization of these hyperparameters as a multi-objective function and take care to distribute our population of the best individuals of each run along the Pareto Front.

One hypothesis as to why our evolved architecture did not perform as well as only evolving optimizer hyper-parameters is that evolving architecture simultaneously with evolving optimizer hyper-parameters could be flawed by nature. If the optimizer parameters reach an ideal state with a given architecture and that architecture is then mutated, the search space for ideal optimizer hyper-parameters will, in theory, also have changed. A potential solution for this issue is to create sub-populations (consisting of optimizer hyper-parameter individuals) for each static architecture. After this sub-population finishes its evolutionary run, the architecture's fitness will be based off the best sub-individual's accuracy score and selection will resume. This process would continue until an optimal architecture and optimizer were found.

To further improve our evolution, further work could include basing fitness off individuals' accuracy on multiple datasets. This could take the form of island models in which each island corresponds to one or a few datasets and allow for cross-pollination between islands. This would theoretically allow for more generalized architectures to evolve.

Works Cited

[1] Ian H. Witten, Eibe Frank, Mark A. Hall, and Christopher J. Pal. 2017. Data mining: practical machine learning tools and techniques, Amsterdam, MA: Morgan Kaufmann.

[2] Patrick Kelly. 2010. Gliomas: Survival, origin and early detection. *Surgical Neurology International *1, 1 (2010), 96. DOI:http://dx.doi.org/10.4103/2152- 7806.74243

[3] Amin Kabir Anaraki, Moosa Ayati, and Foad Kazemi. 2019. Magnetic resonance imaging-based brain tumor grades classification and grading via convolutional neural networks and genetic algorithms. *Biocybernetics and Biomedical Engineering *39, 1 (2019), 63–74. DOI:http://dx.doi.org/10.1016/j.bbe.2018 .10.004

[4] Zhang, A., Lipton, Z. C., Li, M., & Smola, A. J. Dive into Deep Learning. https://d2l.ai/

[5] Ruder, S. (2020, March 20). An overview of gradient descent optimization algorithms. Retrieved from https://ruder.io/optimizing-gradient- descent/

[6] Stanley, K.O., Clune, J., Lehman, J. et *al. *Designing neural networks through neuroevolution. *Nat Mach Intell ***1, **24–35 (2019). https://doi.org/10.1038/s42256-018-0006-z

[7] Xueli Xiao and Ming Yan and Sunitha Basodi and Chunyan Ji and Yi Pan. Efficient Hyperparameter Optimization in Deep Learning Using a Variable Length Genetic Algorithm 2020.https://arxiv.org/pdf/2006.12703.pdf

[8] F. Johnson, A. Valderrama, C. Valle, B. Crawford, R. Soto, and R. Nanculef, “Automating Configuration of Convolutional Neural Network Hyperparameters Using Genetic Algorithm,” in IEEE Access, vol. 8, pp.156139-156152, 2020, doi: 0.1109/ACCESS.2020.3019245.

[9] A. Gomez-Villa, A. Martín, J. Vazquez-Corral, M. Bertalmío, J. Malo, Color illusions also deceive CNNs for low-level vision tasks: Analysis and implications, Vision Research, Volume 176, 2020, Pages 156-174, ISSN 0042-6989, https://doi.org/10.1016/j.visres.2020.07.0.10

[10] Liu P, El Basha MD, Li Y, Xiao Y, Sanelli PC, Fang R. Deep Evolutionary Networks with Expedited Genetic Algorithms for Medical Image Denoising. Med Image Anal. 2019 May;54:306-315. doi: 10.1016/j.media.2019.03.004. Epub 2019 Mar 21. PMID: 30981133; PMCID: PMC6527091.