The following report summarizes the work done during Google Summer of Code 2023 along with the results, scope for improvements and future work. This also serves as the final project report with all the contributions.
- Name: Saurabh Suresh Powar
- Email: sspowar25@gmail.com
- Github Username: Spnetic-5
- LinkedIn: https://www.linkedin.com/in/psau/
- University: VJTI, Mumbai
- Organization: Fortran-lang
- Project Title: Optimizers in a deep learning framework
- Project Link: neural-fortran
- Project Tracker: Fortran-Discourse
This project aims to enhance the capabilities of neural-fortran, a parallel Fortran framework for deep learning, by implementing additional optimization algorithms. While the framework currently supports training and inference of dense and convolutional neural networks using stochastic and mini-batch gradient descent optimizers, more versatile learning and training from a broader range of problems and datasets require the use of other optimization algorithms such as RMSprop, Adam, and others commonly used in popular deep learning frameworks like Keras.
The successful implementation of these optimization algorithms will provide users with more flexibility and choice when training their models, ultimately improving the performance and capabilities of neural-fortran.
- Design and implement several commonly used optimization algorithms, such as RMSprop, Adam, and similar, into the neural-fortran framework.
- Test the newly implemented optimizers thoroughly to ensure their correctness and performance in various scenarios and datasets.
- Integrate the newly implemented optimizers into the neural-fortran framework and ensure their compatibility with existing functionality.
Throughout the GSoC project, my approach was systematic and goal-oriented. I began by researching and planning the implementation of various optimization algorithms, including Momentum, RMSprop, Adam, and others. As a foundation, I added an example program for fitting a quadratic function and gradually integrated each optimizer algorithm separately, refining and testing their subroutines.
Regular collaboration with my mentor ensured accurate access to layer parameters, and extensive testing verified the correctness and performance of the implemented optimizers. By following this approach, I successfully expanded the capabilities of the neural-fortran framework, empowering users with a diverse set of optimization choices for improved model training and performance.
- Weekly video conferences over Zoom were used as the primary mode of communication with the mentors
- The Github Issues and PRs were used to resolve any doubts, suggestions and comments.
- Weekly blogs and updates were provided in the community discourse to ensure everyone was updated with the project.
- ✔ #123: Added Leaky ReLU activation function (1D & 3D)
- ✔ #134: Example program to fit a quadratic function, SGD optimizer implementation.
- ✔ #144: Added RMSProp Optimizer subroutine.
- ✔ #148: Added Momentum and Nesterov modifications for SGD.
- ✔ #150: Added Adam optimizer implementation.
- ✔ #154: Adagrad Optimizer Implementation.
- ⊙ #157: Added Batch Normalization Layer modules
The future scope of this project lies in further expanding the optimization algorithm offerings within the neural-fortran framework. While I have successfully integrated several key algorithms, there are still many additional optimization techniques commonly used in deep learning that can be implemented. These include variants of existing algorithms with modifications specific to certain tasks, as well as entirely new optimization strategies that emerge from ongoing research.
- Variants of Existing Optimizers: Consider implementing variants of the existing optimization algorithms with specific modifications tailored to different types of neural networks and datasets.
- Custom Objective Functions: Allow users to define custom objective functions and loss functions for their specific tasks. This could involve implementing hooks or callbacks that users can provide to the optimizers.
- Learning Rate Schedulers: Allow users to define learning rate schedules that adapt the learning rate during training. This can be useful for tasks where the optimal learning rate changes over time.
- Early Stopping Criteria: Implement mechanisms for early stopping based on certain convergence criteria. This would help users avoid overfitting and save training time.
- Documentation and Tutorials: Provide comprehensive documentation and tutorials for each optimizer and overall framework code structure. Explain the theory behind the algorithms, their advantages, and practical use cases.
Throughout the duration of the project, I have grown a lot. Participating in GSoC has definitely improved my interpersonal and technical skills. Here are some of the prominent ones:
- Gained a solid understanding of deep learning concepts, including neural network architecture, activation functions, and optimization algorithms.
- Hands-on experience in implementing various optimization algorithms & understand their inner workings.
- As I encountered challenges during implementation, I developed skills in debugging and refactoring code.
- Gained experience in collaborating with mentors and receiving feedback through code reviews.
- Managing a long-term project within the GSoC timeline taught me valuable time management skills.
- Understanding and reviewing code written by someone else and modifying it.
- Learned effective testing strategies to ensure the correctness and robustness of implemented optimizers.
The project would not have been possible without the ongoing support of my mentors. I would especially like to thank
- Milan Curcic for being a such a encouraging mentor and coding together with me.
- Jeremie Vandenplas, Arteev Raina, Federico Perini, and Henil Panchal for discussing, suggesting improvements and patiently answering my questions about the project.
- Fortran community for discussing the project during community calls and being so appreciative about it.