PhD Study : Trusting the uncertainty in machine learning predictions

Summary

Overview

Some machine learning algorithms report the uncertainty in their predictions, in order to improve trust. But can we trust the uncertainties? This project addresses this fundamental question. The research is to be carried out within the Artificial Intelligence Research Group at Ulster University, and methods tested using a high impact application from the physical sciences, under active development at the University of Manchester.

Background

Recent failures in deep learning, such as the first fatality from driverless cars, have highlighted the importance of reporting predictive uncertainty, i.e. the uncertainty in predictions made by machine learning algorithms [1]. The question “can we trust our model’s predictions?” is increasingly being asked in a diverse range of applications, from medical decision making to the prediction of atomic properties. Although considering predictive uncertainty is preferable to blindly trusting predictions, a new question of trust is raised: “can we trust the uncertainty in our model’s predictions?” As more applications make use of predictive uncertainty, this problem is beginning to receive increased attention within the machine learning community [2], mainly for classification problems. This project will focus on predictive uncertainty in regression problems, focusing on the use of Bayesian methods such as Gaussian Process regression.

Proposed research

The methods developed will be generic, but tested using a state-of-the-art application from computational chemistry that has been developed and updated over the past decade at the University of Manchester [3,4].

The exact research challenges to be addressed in this project can be tailored to the interests and experience of the PhD candidate, but example problems include:

*Ensembles: A number of open-source Gaussian Process packages exist, such as GPy, GPyTorch, GPFlow, and george. Could models from different packages be combined using ensemble methods to calibrate, or reduce, predictive uncertainty?

*Explanations and uncertainty: Methods from Explainable AI (XAI) aim to explain predictions made by machine learning models. Can explanations of predictions be used to help calibrate or evaluate predictive uncertainty?

*Transfer learning and uncertainty: How can we transfer uncertainties from a problem for which we have a large number of training data, to a related problem for which we have few training data?

*Features and uncertainty: What combination of features leads to reliable predictive uncertainty?

*Multi-objective formulation: A small number of measures exist for evaluating predictive uncertainty in regression, e.g. [5]. The trade-offs between these measures could be investigated to attempt to identify a “best-of-all-worlds” approach. All of these problems are of course not mutually exclusive, but overlapping.

[1] Kendall, A. (2018). Deep learning is not good enough, we need Bayesian Deep Learning for safe AI: Available at: https://alexgkendall.com/computer_vision/bayesian_deep_learning_for_safe_ai/

[2] Ovadia, Y. et al. (2019). Can You Trust Your Model’s Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift, arxiv: 1906.02530

[3] Handley, C. M., Hawe, G. I., Kell, D. B., Popelier, P. (2009) Optimal construction of a fast and accurate polarisable water potential based on multipole moments trained by machine learning, Physical Chemistry Chemical Physics 11 6365-6376.

[4] McDonagh, J., et al. (2018). Machine Learning of Dynamic Electron Correlation Energies from Topological Atoms, Journal of Chemical Theory and Computation, 14 (1) 216–224.

[5] Moukari, M. et al. (2019). n-MeRCI: A new metric to evaluate the correlation between predictive uncertainty and true error, arxiv: 1908.07253.

Essential criteria

Applicants should hold, or expect to obtain, a First or Upper Second Class Honours Degree in a subject relevant to the proposed area of study.

We may also consider applications from those who hold equivalent qualifications, for example, a Lower Second Class Honours Degree plus a Master’s Degree with Distinction.

In exceptional circumstances, the University may consider a portfolio of evidence from applicants who have appropriate professional experience which is equivalent to the learning outcomes of an Honours degree in lieu of academic qualifications.

• Sound understanding of subject area as evidenced by a comprehensive research proposal

Desirable Criteria

If the University receives a large number of applicants for the project, the following desirable criteria may be applied to shortlist applicants for interview.

• Publications record appropriate to career stage
• A comprehensive and articulate personal statement
• Applicants will be shortlisted if they have an average of 75% or greater in a first (honours) degree (or a GPA of 8.75/10). For applicants with a first degree average in the range of 70% to 74% (GPA 3.3): If they are undertaking an Masters, then the average of their first degree marks and their Masters marks will be used for shortlisting.

Funding and eligibility

The University offers the following levels of support:

Vice Chancellors Research Studentship (VCRS)

The following scholarship options are available to applicants worldwide:

• Full Award: (full-time tuition fees + £19,000 (tbc))
• Part Award: (full-time tuition fees + £9,500)
• Fees Only Award: (full-time tuition fees)

These scholarships will cover full-time PhD tuition fees for three years (subject to satisfactory academic performance) and will provide a £900 per annum research training support grant (RTSG) to help support the PhD researcher.

Applicants who already hold a doctoral degree or who have been registered on a programme of research leading to the award of a doctoral degree on a full-time basis for more than one year (or part-time equivalent) are NOT eligible to apply for an award.

Please note: you will automatically be entered into the competition for the Full Award, unless you state otherwise in your application.

Department for the Economy (DFE)

The scholarship will cover tuition fees at the Home rate and a maintenance allowance of £19,000 (tbc) per annum for three years (subject to satisfactory academic performance).

This scholarship also comes with £900 per annum for three years as a research training support grant (RTSG) allocation to help support the PhD researcher.

• Candidates with pre-settled or settled status under the EU Settlement Scheme, who also satisfy a three year residency requirement in the UK prior to the start of the course for which a Studentship is held MAY receive a Studentship covering fees and maintenance.
• Republic of Ireland (ROI) nationals who satisfy three years’ residency in the UK prior to the start of the course MAY receive a Studentship covering fees and maintenance (ROI nationals don’t need to have pre-settled or settled status under the EU Settlement Scheme to qualify).
• Other non-ROI EU applicants are ‘International’ are not eligible for this source of funding.
• Applicants who already hold a doctoral degree or who have been registered on a programme of research leading to the award of a doctoral degree on a full-time basis for more than one year (or part-time equivalent) are NOT eligible to apply for an award.

Due consideration should be given to financing your studies. Further information on cost of living