Some machine learning algorithms report the uncertainty in their predictions, in order to improve trust. But can we trust the uncertainties? This project addresses this fundamental question. The research is to be carried out within the Artificial Intelligence Research Group at Ulster University, and methods tested using a high impact application from the physical sciences, under active development at the University of Manchester.
Recent failures in deep learning, such as the first fatality from driverless cars, have highlighted the importance of reporting predictive uncertainty, i.e. the uncertainty in predictions made by machine learning algorithms . The question “can we trust our model’s predictions?” is increasingly being asked in a diverse range of applications, from medical decision making to the prediction of atomic properties. Although considering predictive uncertainty is preferable to blindly trusting predictions, a new question of trust is raised: “can we trust the uncertainty in our model’s predictions?” As more applications make use of predictive uncertainty, this problem is beginning to receive increased attention within the machine learning community , mainly for classification problems. This project will focus on predictive uncertainty in regression problems, focusing on the use of Bayesian methods such as Gaussian Process regression.
The methods developed will be generic, but tested using a state-of-the-art application from computational chemistry that has been developed and updated over the past decade at the University of Manchester [3,4].
The exact research challenges to be addressed in this project can be tailored to the interests and experience of the PhD candidate, but example problems include:
*Ensembles: A number of open-source Gaussian Process packages exist, such as GPy, GPyTorch, GPFlow, and george. Could models from different packages be combined using ensemble methods to calibrate, or reduce, predictive uncertainty?
*Explanations and uncertainty: Methods from Explainable AI (XAI) aim to explain predictions made by machine learning models. Can explanations of predictions be used to help calibrate or evaluate predictive uncertainty?
*Transfer learning and uncertainty: How can we transfer uncertainties from a problem for which we have a large number of training data, to a related problem for which we have few training data?
*Features and uncertainty: What combination of features leads to reliable predictive uncertainty?
*Multi-objective formulation: A small number of measures exist for evaluating predictive uncertainty in regression, e.g. . The trade-offs between these measures could be investigated to attempt to identify a “best-of-all-worlds” approach. All of these problems are of course not mutually exclusive, but overlapping.
 Kendall, A. (2018). Deep learning is not good enough, we need Bayesian Deep Learning for safe AI: Available at: https://alexgkendall.com/computer_vision/bayesian_deep_learning_for_safe_ai/
 Ovadia, Y. et al. (2019). Can You Trust Your Model’s Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift, arxiv: 1906.02530
 Handley, C. M., Hawe, G. I., Kell, D. B., Popelier, P. (2009) Optimal construction of a fast and accurate polarisable water potential based on multipole moments trained by machine learning, Physical Chemistry Chemical Physics 11 6365-6376.
 McDonagh, J., et al. (2018). Machine Learning of Dynamic Electron Correlation Energies from Topological Atoms, Journal of Chemical Theory and Computation, 14 (1) 216–224.
 Moukari, M. et al. (2019). n-MeRCI: A new metric to evaluate the correlation between predictive uncertainty and true error, arxiv: 1908.07253.
If the University receives a large number of applicants for the project, the following desirable criteria may be applied to shortlist applicants for interview.
The University offers the following awards to support PhD study and applications are invited from UK, EU and overseas for the following levels of support:
Full award (full-time PhD fees + DfE level of maintenance grant + RTSG for 3 years).
This scholarship will cover full-time PhD tuition fees and provide the recipient with £15,000 maintenance grant per annum for three years (subject to satisfactory academic performance). This scholarship also comes with £900 per annum for three years as a research training studentship grant (RTSG) allocation to help support the PhD researcher.
Part award (full-time PhD fees + 50% DfE level of maintenance grant + RTSG for 3 years).
This scholarship will cover full-time PhD tuition fees and provide the recipient with £7,500 maintenance grant per annum for three years (subject to satisfactory academic performance). This scholarship also comes with £900 per annum for three years as a research training studentship grant (RTSG) allocation to help support the PhD researcher.
Fees only award (PhD fees + RTSG for 3 years).
This scholarship will cover full-time PhD tuition fees for three years (subject to satisfactory academic performance). This scholarship also comes with £900 per annum for three years as a research training studentship grant (RTSG) allocation to help support the PhD researcher.
The scholarship will cover tuition fees at the Home rate and a maintenance allowance of £ 15,009 per annum for three years. EU applicants will only be eligible for the fee’s component of the studentship (no maintenance award is provided). For Non-EU nationals the candidate must be "settled" in the UK. This scholarship also comes with £900 per annum for three years as a research training studentship grant (RTSG) allocation to help support the PhD researcher.
Due consideration should be given to financing your studies; for further information on cost of living etc. please refer to: www.ulster.ac.uk/doctoralcollege/postgraduate-research/fees-and-funding/financing-your-studies
As Senior Engineering Manager of Analytics at Seagate Technology I utilise the learning from my PhD ever day
Adrian Johnston - PhD in InformaticsWatch Video
Friday 7 February 2020
Late March 2020
The largest of Ulster's campuses
Monday 2 December 2019
Subject: Computer Science and Informatics
When applying for this PhD opportunity please quote reference number: