Gaussian processes (GPs) are a challenging area of Bayesian machine learning to get started with – from wrapping your head around dealing with infinite dimensional Gaussian distributions, to understanding kernel functions and how to choose the right one for the right task, all on top of having solid knowledge of Bayesian inference.
While primarily used as a powerful regression model with the ability to estimate uncertainty in predictions, GPs can also be used for classification, and have a very wide range of applications.
These are some of the resources I have used, or are planning to use in my on-going process of learning about GPs.
- Iain Murray & Arno Onken (University of Edinburgh)
"Machine Learning and Pattern Recognition" – requires a University of Edinburgh login
This is a general course on machine learning, but focusing heavily on Bayesian aspects in the second half of the course, including GPs. This was my first introduction to GPs and Bayesian statistics, and both lecturers are great!
Accompanying notes: Gaussian Processes, Gaussian Processes and Kernels - Richard Turner (Universtiy of Cambridge)
"Machine Learning Tutorial Series: Gaussian Processes"
This is a great introduction, providing a unique way to visualize GPs that really helps with building intuition. - Mauricio Alvarez (University of Sheffield)
"Multi Output Gaussian Processes"
An interesting look into the uses and challenges of multi-output GPs, and why to avoid independence assumptions.
- Mutual Information – Duane Rich (YouTube)
"Gaussian Processes"
Another great introductory resource. This video offers a detailed yet higher level overview of GPs.
- Kevin P. Murphy
"Machine Learning: A Probabilistic Perspective", chapters 14 (Kernels) and 15 (Gaussian processes) - Carl Edward Rasmussen & Christopher K. I. Williams
"Gaussian Processes for Machine Learning"
Note: There are some new books by Kevin Murphy that also have some GP content in them.
- Marc Peter Deisenroth, Mark van der Wilk, Yicheng Luo
"A Practical Guide to Gaussian Processes"
A detailed GP guide with interactive tools for visualizing the effect of kernel hyper-parameters. - Jochen Görtler, Rebecca Kehlbeck, Oliver Deussen
"A Visual Exploration of Gaussian Processes"
Another interactive guide, though goes into less detail about the mathematical theory of GPs than the above article. - Yuge Shi
"Gaussian Process, not quite for dummies"
This blog post is essentially a written summary of the above linked GP tutorial lecture by Richard Turner. - Martin Krasser
"Gaussian processes"
One of the few posts I could find that do a great job translating the theory of GPs into code examples without relying on existing GP frameworks, and instead building up from fundamental linear algebra using just NumPy. - David Duvenaud
"The Kernel Cookbook: Advice on Covariance functions"
An extremely useful resource for deciding on kernel functions, and understanding how to combine them to model different combinations of effects including linear, exponential and periodic behaviour.
- Seth Flaxman, Andrew Gelman, Daniel Neill, Alex Smola, Aki Vehtari, Andrew Gordon Wilson
"Fast hierarchical Gaussian processes"
While the main focus of this paper is on improving the computational efficiency of MCMC, the first two sections of the paper also provide a nice brief introduction on hierarchical Gaussian processes, where a hyper-prior is placed on the hyper-parameters of the kernel function.
These are just a few example applications involving GPs – there are many more areas where they are used!
- Carl Edward Rasmussen & Christopher K. I. Williams
"Model Selection and Adaptation of Hyperparameters" in Gaussian Processes for Machine Learning
Bayesian optimization is an increasingly popular approach for hyper-parameter tuning, as it explores the hyper-parameter search space in a more sophisticated and efficient way than grid/random searches. - Shrihari Vasudevan, Fabio Ramos, Eric Nettleton, Hugh Durrant-Whyte, Allan Blair
"Gaussian Process Modeling of Large Scale Terrain"
This paper looks into how GPs can be used to estimate elevation information in unknown or inaccessible areas, based on recorded data from surrounding locations. The authors focus on field robotics applications such as mine exploration. - Niya Chen, Zheng Qian, Xiaofeng Meng, Ian T. Nabney
"Short-Term Wind Power Forecasting Using Gaussian Processes"
Accurate forecasting for wind power can be difficult due to how erratic wind behaves. The authors use GPs to make one-day forecasts of wind power, with the aim of improving controlling strategies for wind turbines. - Laura Martínez-Ferrer, María Piles, Gustau Camps-Valls
"Crop Yield Estimation and Interpretability with Gaussian Processes"
This paper looks at how satellite sensor observations and weather data can be combined with GPs to estimate the development and yield of crops. - Elizabeth Buckingham-Jeffery, Valery Isham, Thomas House
"Gaussian process approximations for fast inference from infectious disease data"
With the recent global focus on epidemiology, it is interesting to see how GPs can be applied for deriving MLE population estimates for the number of infectious individuals with a disease at a given time.
Below is a list of some well-maintained Python packages that are commonly used to implement GPs.
- GPflow: A purpose built library just for GPs with a TensorFlow Probability back-end, focusing on making GPs more applicable to large-scale datasets.
- GPyTorch: Similar to GPflow, but PyTorch-based.
- PyMC3 and Pyro: General probabilistic machine learning packages which also have some GP functionality.
There are other packages such as GPy which are quite feature rich, but seem to be far less maintained.
- Gaussian Process Summer Schools: Workshops for researchers interested in GP theory and application, normally held in Sheffield, UK. There are also a bunch of great GP resources on this website.
- Gaussian Processes Cambridge: An interest group for developing deeper understanding of GPs, also holding regular meetings (although it appears inactive since January 2021).