My Model Is Stuck in a Loop—And So Am I
Ethan w -
Hi everyone! Welcome back to week 6 of my blog.
Last week, I took a step back from experimentation and caught up on reading while my lab focused on a major conference deadline. I read Deep Imbalanced Regression via Hierarchical Classification Adjustment, which introduced a multi-level classification structure as an alternative to smoothing approaches. I also continued Deep Learning by Goodfellow, Bengio, and Courville which went into probability, information theory, and optimization.
This week, I implemented the file name change solution that my mentor recommended last week that would prevent the file access errors from parallel processes. Instead of overwriting the folder to update it, the code would create new checkpoint folders for updates. However, I ran into an unexpected issue. It seems like an empty folder is created at the start of training before any actual training is done and the parallel process overlaps on that initial folder. This creates a loop where they overlap and then a new folder is created and they overlap again. Now, I am more confident that it is an issue with specifically how Python on Windows handles file access and process management compared to Linux. Specifically how Windows locks files that are open and how python uses spawn() instead of fork() to generate new processes. Since there is no simple way to get Windows to behave like Linux, I will either run a VM in Linux or look to adjust the code to work well in windows.
Besides troubleshooting, I also began coding the extension of LDS and FDS to 2D. The code creates a 2D density histogram and uses a 2D gaussian kernel to smoothing the distributions. While I haven’t completed the implementation or obtained results, I’ll experiment in upcoming weeks to optimize the model.
Next week, I hope to fully resolve this parallel processing issue and if it doesn’t work, I will move towards moving into a Linux VM. Stay tuned!
Comments:
All viewpoints are welcome but profane, threatening, disrespectful, or harassing comments will not be tolerated and are subject to moderation up to, and including, full deletion.