PhD Study : Deep Learning Approaches for Multimodal Emotion Recognition

Summary

Emotion recognition is the task of identifying human emotion, and is an increasingly important factor in the field of human computer interaction (HCI), with applications in healthcare, well-being, gaming, safety and security. Emotion recognition can be achieved using a variety of input modalities, ranging from video, images, voice, and text. The combination of these different input modalities such as vocal signal, vision, and verbal content have been shown to have great potential in representing a wide variety of different emotions. Traditional approaches have used feature-level fusion in combination with machine learning algorithms to achieve emotion recognition with some success, however in recent years deep learning has shown remarkable capabilities in learning detailed representations of high-dimensional image and video data, as well as audio spectral features, to achieve better classification performance. Unlike tasks that focus on a single input modality to determine emotion recognition, multimodal input techniques take into consideration a range of different input data types, such as visual and audio, without the need for more intrusive physiological data signal recordings (electrocardiogram (ECG), galvanic skin response (GSR), etc). Recent studies utilising multimodal deep learning approaches have shown promising results, however due to the complex nature of multimodal emotion recognition further challenges remain.

This project aims to explore deep learning architectures and to propose, design, and develop a multimodal deep learning pipeline for emotion recognition. This project will address the following research questions:

What is the state-of-the-art deep learning models in image and audio emotion recognition?

How can we utilise audio representative learning using deep learning models?

What deep learning models can be used to extract features from high-dimensionality data such as video and image data to generalise emotional states?

What model-level fusion strategies can be used with image and audio feature representations to achieve accurate emotion recognition?

How can we incorporate representative data sources in the development and evaluation of deep learning emotion recognition models?

In answering these research challenges, the aim of the project is to advance the state-of-the-art in deep learning models for multimodal emotion recognition. Outputs from this research project could have potential impact in the application of emotion recognition within different fields, including healthcare, mental health and well-being.

Essential criteria

Applicants should hold, or expect to obtain, a First or Upper Second Class Honours Degree in a subject relevant to the proposed area of study.

We may also consider applications from those who hold equivalent qualifications, for example, a Lower Second Class Honours Degree plus a Master’s Degree with Distinction.

In exceptional circumstances, the University may consider a portfolio of evidence from applicants who have appropriate professional experience which is equivalent to the learning outcomes of an Honours degree in lieu of academic qualifications.

Desirable Criteria

If the University receives a large number of applicants for the project, the following desirable criteria may be applied to shortlist applicants for interview.

• First Class Honours (1st) Degree
• Masters at 70%
• Experience using research methods or other approaches relevant to the subject domain
• Work experience relevant to the proposed project
• Publications - peer-reviewed

Funding and eligibility

The University offers the following levels of support:

Vice Chancellors Research Studentship (VCRS)

The following scholarship options are available to applicants worldwide:

• Full Award: (full-time tuition fees + £19,000 (tbc))
• Part Award: (full-time tuition fees + £9,500)
• Fees Only Award: (full-time tuition fees)

These scholarships will cover full-time PhD tuition fees for three years (subject to satisfactory academic performance) and will provide a £900 per annum research training support grant (RTSG) to help support the PhD researcher.

Applicants who already hold a doctoral degree or who have been registered on a programme of research leading to the award of a doctoral degree on a full-time basis for more than one year (or part-time equivalent) are NOT eligible to apply for an award.

Please note: you will automatically be entered into the competition for the Full Award, unless you state otherwise in your application.

Department for the Economy (DFE)

The scholarship will cover tuition fees at the Home rate and a maintenance allowance of £19,000 (tbc) per annum for three years (subject to satisfactory academic performance).

This scholarship also comes with £900 per annum for three years as a research training support grant (RTSG) allocation to help support the PhD researcher.

• Candidates with pre-settled or settled status under the EU Settlement Scheme, who also satisfy a three year residency requirement in the UK prior to the start of the course for which a Studentship is held MAY receive a Studentship covering fees and maintenance.
• Republic of Ireland (ROI) nationals who satisfy three years’ residency in the UK prior to the start of the course MAY receive a Studentship covering fees and maintenance (ROI nationals don’t need to have pre-settled or settled status under the EU Settlement Scheme to qualify).
• Other non-ROI EU applicants are ‘International’ are not eligible for this source of funding.
• Applicants who already hold a doctoral degree or who have been registered on a programme of research leading to the award of a doctoral degree on a full-time basis for more than one year (or part-time equivalent) are NOT eligible to apply for an award.

Due consideration should be given to financing your studies. Further information on cost of living