Summary

Overview

Some machine learning algorithms report the uncertainty in their predictions, in order to improve trust. But can we trust the uncertainties? This project addresses this fundamental question. The research is to be carried out within the Artificial Intelligence Research Group at Ulster University, and methods tested using a high impact application from the physical sciences, under active development at the University of Manchester.

Background

Recent failures in deep learning, such as the first fatality from driverless cars, have highlighted the importance of reporting predictive uncertainty, i.e. the uncertainty in predictions made by machine learning algorithms [1]. The question “can we trust our model’s predictions?” is increasingly being asked in a diverse range of applications, from medical decision making to the prediction of atomic properties. Although considering predictive uncertainty is preferable to blindly trusting predictions, a new question of trust is raised: “can we trust the uncertainty in our model’s predictions?” As more applications make use of predictive uncertainty, this problem is beginning to receive increased attention within the machine learning community [2], mainly for classification problems. This project will focus on predictive uncertainty in regression problems, focusing on the use of Bayesian methods such as Gaussian Process regression.

Proposed research

The methods developed will be generic, but tested using a state-of-the-art application from computational chemistry that has been developed and updated over the past decade at the University of Manchester [3,4].

The exact research challenges to be addressed in this project can be tailored to the interests and experience of the PhD candidate, but example problems include:

*Ensembles: A number of open-source Gaussian Process packages exist, such as GPy, GPyTorch, GPFlow, and george. Could models from different packages be combined using ensemble methods to calibrate, or reduce, predictive uncertainty?

*Explanations and uncertainty: Methods from Explainable AI (XAI) aim to explain predictions made by machine learning models. Can explanations of predictions be used to help calibrate or evaluate predictive uncertainty?

*Transfer learning and uncertainty: How can we transfer uncertainties from a problem for which we have a large number of training data, to a related problem for which we have few training data?

*Features and uncertainty: What combination of features leads to reliable predictive uncertainty?

*Multi-objective formulation: A small number of measures exist for evaluating predictive uncertainty in regression, e.g. [5]. The trade-offs between these measures could be investigated to attempt to identify a “best-of-all-worlds” approach. All of these problems are of course not mutually exclusive, but overlapping.

[1] Kendall, A. (2018). Deep learning is not good enough, we need Bayesian Deep Learning for safe AI: Available at: https://alexgkendall.com/computer_vision/bayesian_deep_learning_for_safe_ai/

[2] Ovadia, Y. et al. (2019). Can You Trust Your Model’s Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift, arxiv: 1906.02530

[3] Handley, C. M., Hawe, G. I., Kell, D. B., Popelier, P. (2009) Optimal construction of a fast and accurate polarisable water potential based on multipole moments trained by machine learning, Physical Chemistry Chemical Physics 11 6365-6376.

[4] McDonagh, J., et al. (2018). Machine Learning of Dynamic Electron Correlation Energies from Topological Atoms, Journal of Chemical Theory and Computation, 14 (1) 216–224.

[5] Moukari, M. et al. (2019). n-MeRCI: A new metric to evaluate the correlation between predictive uncertainty and true error, arxiv: 1908.07253.


Essential criteria

  • To hold, or expect to achieve by 15 August, an Upper Second Class Honours (2:1) Degree or equivalent from a UK institution (or overseas award deemed to be equivalent via UK NARIC) in a related or cognate field.
  • Sound understanding of subject area as evidenced by a comprehensive research proposal

Desirable Criteria

If the University receives a large number of applicants for the project, the following desirable criteria may be applied to shortlist applicants for interview.

  • Publications record appropriate to career stage
  • A comprehensive and articulate personal statement
  • Applicants will be shortlisted if they have an average of 75% or greater in a first (honours) degree (or a GPA of 8.75/10). For applicants with a first degree average in the range of 70% to 74% (GPA 3.3): If they are undertaking an Masters, then the average of their first degree marks and their Masters marks will be used for shortlisting.

Funding

    The University offers the following awards to support PhD study and applications are invited from UK, EU and overseas for the following levels of support:

    Vice Chancellors Research Studentship (VCRS)

    Full award (full-time PhD fees + DfE level of maintenance grant + RTSG for 3 years).

    This scholarship will cover full-time PhD tuition fees and provide the recipient with £15,000 maintenance grant per annum for three years (subject to satisfactory academic performance). This scholarship also comes with £900 per annum for three years as a research training support grant (RTSG) allocation to help support the PhD researcher.

    Vice-Chancellor’s Research Bursary (VCRB)

    Part award (full-time PhD fees + 50% DfE level of maintenance grant + RTSG for 3 years).

    This scholarship will cover full-time PhD tuition fees and provide the recipient with £7,500 maintenance grant per annum for three years (subject to satisfactory academic performance). This scholarship also comes with £900 per annum for three years as a research training support grant (RTSG) allocation to help support the PhD researcher.

    Vice-Chancellor’s Research Fees Bursary (VCRFB)

    Fees only award (PhD fees + RTSG for 3 years).

    This scholarship will cover full-time PhD tuition fees for three years (subject to satisfactory academic performance). This scholarship also comes with £900 per annum for three years as a research training support grant (RTSG) allocation to help support the PhD researcher.

    Department for the Economy (DFE)

    The scholarship will cover tuition fees at the Home rate and a maintenance allowance of £15,285 per annum for three years. EU applicants will only be eligible for the fee’s component of the studentship (no maintenance award is provided). For Non-EU nationals the candidate must be "settled" in the UK. This scholarship also comes with £900 per annum for three years as a research training support grant (RTSG) allocation to help support the PhD researcher.

    Due consideration should be given to financing your studies; for further information on cost of living etc. please refer to: www.ulster.ac.uk/doctoralcollege/postgraduate-research/fees-and-funding/financing-your-studies


Other information


The Doctoral College at Ulster University


Reviews

Profile picture of Adrian Johnston

As Senior Engineering Manager of Analytics at Seagate Technology I utilise the learning from my PhD ever day

Adrian Johnston - PhD in Informatics

Watch Video  

Profile picture of Xin Wei

I received the bachelor’s of engineering degree in computer science and technology from Shangrao Normal University, Jiangxi, China, in 2013; and the master’s degree in computer application and technology from the School of Mathematics and Computer Science, Fujian Normal University, China. When I was pursuing a PhD degree at Ulster University, I continued my research on face recognition and image representation.This long journey has only been possible due to the constant support and encouragement of my first supervisor. I also like to thank my second supervisor for his patience, support and guidance during my research studies. My favourite memory was the days of exercising, gathering and playing with my friends here. If I could speak to myself at the start of my PhD, the best piece of advice I would give myself would be "submit more papers to Journals instead of conferences".

Xin Wei - PhD in Computer Science and Informatics


Profile picture of Jyotsna Talreja Wassan

In the whole PhD ordeal, my supervisory team played a tremendous role:- they are three in a million. They are perfect supervisors who perfectly know which milestones or pathways to be taken during research initiatives, and they understand the roles of virtually all stages in the journey of PhD. They showcased superior abilities in managing and motivating me evoking high standards; demonstrating a commitment to excellence. Jane and Haiying guided me as their daughter and Fiona turned out to be the best of friends.I heard from “Eleanor Roosevelt” that “The future belongs to those who believe in the beauty of their dreams.” The dream with which I grew up to become a Doctor one day, has finally come true. In the journey of PhD, I embraced that a PhD is not just the highest degree in Education but rather it is a life experience where perseverance is the key. I can never forget words from my external examiner Prof Yike Guo, from Imperial College London. His words

Jyotsna Talreja Wassan - PhD in Computer Science and Informatics