You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Using Git and GitHub for collaboration on writing scientific papers
Getting started
First you'll have to install the Git command line tool on your machine, following these instructions. Then find the repository that you want to contribute to, copy its address from the green "Clone or Download" button, and on your local machine run e.g.
The practical development of deep learning and its associated infrastructure has initiated a broad re-examination of the practice of computer programming. In this document we briefly survey how this discussion has evolved over the past few years, and then describe our point of view on the underlying mathematics.
Program synthesis
We begin with some appeals to authority, in the form of the following references:
The optimisation algorithm used in most of DeepMind's deep RL papers is RMSProp (e.g in the Mnih et al Atari paper, in the IMPALA paper, in the RL experiments of the PBT paper, in the Zambaldi et al paper). I have seen speculation online that this is because RMSProp may be well-suited to deep learning on non-stationary distributions. In this note I try to examine the RMSProp algorithm and specifically the significance of the epsilon hyperparameter. The references are
Often in the literature RMSProp is presented as a variation of AdaGrad (e.g. in the deep learning textbook and in Karpathy's class). However, I think this is misleading, and that the explanation in Hinton's lecture is (not surprisingl
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The Melbourne Deep Learning Group (MDLG) is in the first place a research group, but given the broader importance of these technologies and urgency of Australia adopting them, we also take on a responsibility for helping to educate students at the University of Melbourne, and the broader Australian community. There are many free or low-cost introductory courses on deep learning, e.g. deeplearning.ai and fast.ai and there is no point in us reproducing some slight variation on this content. However, while there is plenty of content available, that doesn't mean it is trivial to learn in a vacuum (that's what classes are for!).
We therefore focus our attention on facilitating a thriving local community and runnning short events that help to motivate members of this community to deepen their understanding of these technologies and their applications, and to meet collaborators (e.g. we
It is still unclear what the long-term impacts of this technology will be. Large changes in productivity have occurred in history, and the potential of deep learning is comparable to other general purpose technologies (steam, electricity, chemical manufacturing, etc) responsible for those changes. While there are many real-world applications of today's deep learning in computer vision, natural language, and perhaps soon in robotics, these impacts would have to increase by several orders of magnitude to be reasonably compared with the general purpose technologies which drove previous industrial revolutions. However, as anybody familiar with the history of the industrial revolutions knows, once it is obvious to everybody that things are working you may not have time to catch up.
It is therefore worth noting that rich governments (US, China) and corporations (Google, Facebook, Amazon, Microsoft, Baidu, Alib