Deep Learning Approaches for Multimodal Emotion Recognition

Summary

Emotion recognition is the task of identifying human emotion, and is an increasingly important factor in the field of human computer interaction (HCI), with applications in healthcare, well-being, gaming, safety and security. Emotion recognition can be achieved using a variety of input modalities, ranging from video, images, voice, and text. The combination of these different input modalities such as vocal signal, vision, and verbal content have been shown to have great potential in representing a wide variety of different emotions. Traditional approaches have used feature-level fusion in combination with machine learning algorithms to achieve emotion recognition with some success, however in recent years deep learning has shown remarkable capabilities in learning detailed representations of high-dimensional image and video data, as well as audio spectral features, to achieve better classification performance. Unlike tasks that focus on a single input modality to determine emotion recognition, multimodal input techniques take into consideration a range of different input data types, such as visual and audio, without the need for more intrusive physiological data signal recordings (electrocardiogram (ECG), galvanic skin response (GSR), etc). Recent studies utilising multimodal deep learning approaches have shown promising results, however due to the complex nature of multimodal emotion recognition further challenges remain.

This project aims to explore deep learning architectures and to propose, design, and develop a multimodal deep learning pipeline for emotion recognition. This project will address the following research questions:

What is the state-of-the-art deep learning models in image and audio emotion recognition?

How can we utilise audio representative learning using deep learning models?

What deep learning models can be used to extract features from high-dimensionality data such as video and image data to generalise emotional states?

What model-level fusion strategies can be used with image and audio feature representations to achieve accurate emotion recognition?

How can we incorporate representative data sources in the development and evaluation of deep learning emotion recognition models?

In answering these research challenges, the aim of the project is to advance the state-of-the-art in deep learning models for multimodal emotion recognition. Outputs from this research project could have potential impact in the application of emotion recognition within different fields, including healthcare, mental health and well-being.

Essential criteria

Applicants should hold, or expect to obtain, a First or Upper Second Class Honours Degree in a subject relevant to the proposed area of study.

We may also consider applications from those who hold equivalent qualifications, for example, a Lower Second Class Honours Degree plus a Master’s Degree with Distinction.

In exceptional circumstances, the University may consider a portfolio of evidence from applicants who have appropriate professional experience which is equivalent to the learning outcomes of an Honours degree in lieu of academic qualifications.

Desirable Criteria

If the University receives a large number of applicants for the project, the following desirable criteria may be applied to shortlist applicants for interview.

  • First Class Honours (1st) Degree
  • Masters at 70%
  • Experience using research methods or other approaches relevant to the subject domain
  • Work experience relevant to the proposed project
  • Publications - peer-reviewed

Funding and eligibility

The University offers the following levels of support:

Vice Chancellors Research Studentship (VCRS)

Full award (full-time PhD fees + DfE level of maintenance grant + RTSG for 3 years).

This scholarship will cover full-time PhD tuition fees and provide the recipient with £18,000 (tbc) maintenance grant per annum for three years (subject to satisfactory academic performance).

This scholarship also comes with £900 per annum for three years as a research training support grant (RTSG) allocation to help support the PhD researcher.

Applicants who already hold a doctoral degree or who have been registered on a programme of research leading to the award of a doctoral degree on a full-time basis for more than one year (or part-time equivalent) are NOT eligible to apply for an award.

Applicants who already hold a doctoral degree or who have been registered on a programme of research leading to the award of a doctoral degree on a full-time basis for more than one year (or part-time equivalent) are NOT eligible to apply for an award.

Vice-Chancellor’s Research Bursary (VCRB)

Part award (full-time PhD fees + 50% DfE level of maintenance grant + RTSG for 3 years).

This scholarship will cover full-time PhD tuition fees and provide the recipient with £8,000 maintenance grant per annum for three years (subject to satisfactory academic performance). This scholarship also comes with £900 per annum for three years as a research training support grant (RTSG) allocation to help support the PhD researcher.

Applicants who already hold a doctoral degree or who have been registered on a programme of research leading to the award of a doctoral degree on a full-time basis for more than one year (or part-time equivalent) are NOT eligible to apply for an award.

Vice-Chancellor’s Research Fees Bursary (VCRFB)

Fees only award (PhD fees + RTSG for 3 years).

This scholarship will cover full-time PhD tuition fees for three years (subject to satisfactory academic performance). This scholarship also comes with £900 per annum for three years as a research training support grant (RTSG) allocation to help support the PhD researcher.

Applicants who already hold a doctoral degree or who have been registered on a programme of research leading to the award of a doctoral degree on a full-time basis for more than one year (or part-time equivalent) are NOT eligible to apply for an award.

Department for the Economy (DFE)

The scholarship will cover tuition fees at the Home rate and a maintenance allowance of £18,000 (tbc) per annum for three years (subject to satisfactory academic performance).

This scholarship also comes with £900 per annum for three years as a research training support grant (RTSG) allocation to help support the PhD researcher.

  • Candidates with pre-settled or settled status under the EU Settlement Scheme, who also satisfy a three year residency requirement in the UK prior to the start of the course for which a Studentship is held MAY receive a Studentship covering fees and maintenance.
  • Republic of Ireland (ROI) nationals who satisfy three years’ residency in the UK prior to the start of the course MAY receive a Studentship covering fees and maintenance (ROI nationals don’t need to have pre-settled or settled status under the EU Settlement Scheme to qualify).
  • Other non-ROI EU applicants are ‘International’ are not eligible for this source of funding.
  • Applicants who already hold a doctoral degree or who have been registered on a programme of research leading to the award of a doctoral degree on a full-time basis for more than one year (or part-time equivalent) are NOT eligible to apply for an award.

Due consideration should be given to financing your studies. Further information on cost of living

Recommended reading

W. Liu, J. -L. Qiu, W. -L. Zheng and B. -L. Lu, "Comparing Recognition Performance and Robustness of Multimodal Deep Learning Models for Multimodal Emotion Recognition," in IEEE Transactions on Cognitive and Developmental Systems, vol. 14, no. 2, pp. 715-729, June 2022, doi: 10.1109/TCDS.2021.3071170.

Summaira, J., Li, X., Shoib, A. M., Li, S., & Abdul, J. (2021). Recent Advances and Trends in Multimodal Deep Learning: A Review. arXiv preprint arXiv:2105.11087.

L. Schoneveld, A. Othmani, and H. Abdelkawy, “Leveraging recent advances in deep learning for audio-visual emotion recognition,” Pattern Recognition Letters, 2021. ​

Koromilas, P.; Giannakopoulos, T. Deep Multimodal Emotion Recognition on Human Speech: A Review. Appl. Sci. 2021, 11, 7962

The Doctoral College at Ulster University

Key dates

Submission deadline
Monday 27 February 2023
04:00PM

Interview Date
week commencing 17 April 2023

Preferred student start date
18 Sept 2023

Applying

Apply Online  

Contact supervisor

Dr George Moore

Other supervisors

Back to Top