4/2/24. Hypertuning the Hypersensitive Hyperparameters
Frank L -
This past week, aside from my own studying of deep learning and figuring out how to use the programming library TensorFlow, I’ve mostly just been working on running new and updated GAN models for the ASU research project. So mostly just more of me tuning hyperparameters (configuration variables that affect the AI learning process) and having the model run in the background of my computer for hours upon end. One thing that I’ve noticed along the way is that GAN models can actually be very difficult to work with.
First of all, GAN models are incredibly sensitive to changes in some of their hyperparameters. For example, one of the hyperparameters that I tweak the most often is the learning rate. Both the generator and discriminator parts have their own separate learning rates. What these learning rate values actually influence is something I don’t understand well enough to explain, but it’s basically just the step size that each part of the model takes it optimize its performance. Anyways, the difference between a 0.0001 and a 0.0002 learning rate literally determines whether the model produces actual high quality images or just blank ones like the picture below.
Additionally, we’ve been gradually scaling up the GANs that we’re working with, which can cause them to sometimes become increasingly unstable and take much longer to run. The images that we’re working with are becoming bigger in pixel size, which means a lot more data to analyze, and GAN models are also running through more training iterations to make the results as accurate as possible. These models can now take more than 3 hours to finish running. Recently, I also accidently ran a model on my computer’s CPU instead of a GPU. I didn’t even realize anything was wrong until I checked up on the progress half an hour later and found that it hadn’t even gotten close to finish processing 1% of the whole dataset yet. The GPUs are just so many times more efficient that its not even comparable.
Comments:
All viewpoints are welcome but profane, threatening, disrespectful, or harassing comments will not be tolerated and are subject to moderation up to, and including, full deletion.