Impact For REF

The Research Excellence Framework (REF) assesses the quality of research in UK Higher Education Institutions.  Results of this exercise are used to inform funding for future research.

Impact was introduced in the REF2014 assessment and is defined as the effect on, change or benefit to the economy, society, culture, public policy or services, health, the environment or quality of life, beyond academia.

Impact was considered a beneficial addition to REF and weighting increased to 25% in the REF2021 assessment.

Ulster University’s most recent REF submission was made in March 2021.

REF 2029

REF 2021 Impact Element versus Proposed REF 2029 Iteration

The briefing below contains important updates regarding the proposed impact element of the 2029 Research Excellence Framework (REF). These updates have been outlined in the 'Initial Decisions' document produced by the 4 UK higher education funding bodies.

The document was a result of extensive analysis and consultation conducted by the Future Research Assessment Programme.

Key Guidance  Documents

• REF 2029 This section sets out the latest available information/updated for REF 2029
• REF 2029 Initial Decisions Report

If you are considering developing an impact case study (ICS) from your research for the next REF, please discuss with your Research Director and the Impact Team at the earliest opportunity so that you can receive appropriate support to develop your impact. Please remember that all ICSs must be underpinned by quality research produced at Ulster, that the impacts must have a benefit beyond academia, and that the impacts claimed must be backed up by verifiable available evidence.

The following information has been developed based on the guidance applied to REF2021 and will be useful for future submissions. This information will be updated once the new guidance for REF2029 is published.

• What is the definition of IMPACT for REF?

  Definition of impact for the REF

  For the purposes of REF, impact is defined as an effect on, change or benefit to the economy, society, culture, public policy or services, health, the environment or quality of life, beyond academia.

  Impact includes, but is not limited to, an effect on, change or benefit to:

  • the activity, attitude, awareness, behaviour, capacity, opportunity, performance, policy, practice, process or understanding
  • of an audience, beneficiary, community, constituency, organisation or individuals
  • in any geographic location whether locally, regionally, nationally or internationally

  Impact includes the reduction or prevention of harm, risk, cost or other negative effects.

  Impacts on research or the advancement of academic knowledge within the HE sector (whether in the UK or internally) are excluded – the submitted unit’s contribution to academic research and knowledge is assessed within the ‘outputs’ and ‘environment’ elements of REF.

  Impacts on students, teaching or other activities both within and beyond the submitting HEI are included.

• What is the definition of RESEARCH for REF?

  Definition of research for the REF

  For the purposes of REF, research is defined as a process of investigation leading to new insights effectively shared.

  It includes:

  • work of direct relevance to the needs of commerce, industry, culture, society, and to the public and voluntary sectors;
  • scholarship;
  • the invention and generation of ideas, images, performances, artefacts including design, where these lead to new or substantially improved insights;
  • the use of existing knowledge in experimental development to produce new or substantially improved materials, devices, products and processes, including design and construction.
  • Research that is published, disseminated or made publicly available in the form of assessable research outputs, and confidential reports.

  It excludes:

  • Routine testing and routine analysis of materials, components and processes such as for the maintenance of national standards, as distinct from the development of new analytical techniques;
  • The development of teaching materials that do not embody original research.
• How are the starred levels for impact defined?

  The definitions in the starred levels in the impact sub-profile are:

  Impact  sup-profile star definitions

  Star level	Definition
  Four star	Outstanding impacts in terms of their reach and significance
  Three star	Very considerable impacts in terms of their reach and significance
  Two star	Considerable impacts in terms of their reach and significance
  One star	Recognised but modest impacts in terms of their reach and significance
  Unclassified	The impact is of little or no reach and significance; or the impact was not eligible; or the impact was not underpinned by excellent research produced by the submitted unit

• What are the weightings for REF?

  REF Weighting: Outputs, Impact and Environment

  In forming their overall quality judgements, the sub-panels will assess three distinct elements of each submission – outputs, impact and environment – against the following generic criteria:

  Outputs: The sub-panels will assess the quality of submitted research outputs in terms of their ‘originality, significance and rigour, with reference to international research quality standards.  This element will carry a weighting of 60% in the overall outcome awarded to each submission.

  Impact: The sub-panels will assess the ‘reach and significance’ of impacts on the economy, society, culture, public policy or services, health, the environment or quality of life that were underpinned by excellent research conducted in the submitted unit.  This element will carry a weighting of 25%.

  Environment: The sub-panels will assess the research environment in terms of its ‘vitality and sustainability’, including the approach to enabling impact from its research, and its contribution to the vitality and sustainability of the wider discipline or research base.  This element will carry a weighting of 15%.

• How is Research Impact judged for REF?

  Impact Criteria – Reach and Significance

  Reach – the extent and/or diversity of the beneficiaries of the impact, as relevant to the nature of the impact.

  Significance – the degree to which impact has enabled, enriched, influenced, informed or changed the performance, policies, practices, products, services, understanding, awareness or well-being of the beneficiaries.

  • The sub-panels will make an overall judgement about the reach and significance of impacts, rather than assessing them separately.
  • While ICS need to demonstrate both, the balance between them may vary.
  • Sub-panels will exercise their judgement without privileging or disadvantaging either reach or significance.
  • Reach will be assessed in terms of the extent to which beneficiaries have been affected – not in geographic terms or absolute numbers.
  • Criteria will be applied wherever the impact has been felt, regardless of geography or location.
  • No account will be taken of anticipated or future potential impact – nor impact that occurred outside the assessment period.
• How many Impact Case Studies did my Unit of Assessment need to submit?

  Number of Impact Case Studies per UOA

  FTE of Category A submitted staff	Number of case studies
  Up to 19.99	2
  20 to 34.99	3
  35 to 49.99	4
  50 to 64.99	5
  65 to 79.99	6
  80 to 94.99	7
  95 to 109.99	8
  110 to 159.99	9
  160 or more	10, plus one further case study per additional 50 FTE

• What is a continued Impact Case Study?

  Impact case studies continued from REF 2014

  Case studies will be considered to be continued if:

  • The body of underpinning research is the same as described in a 2014 case study.  This should not be understood solely in relation to the referenced outputs, but means that the continued case study does not describe any new research having taken place since the previous case study that has made a distinct and material contribution to the impact and
  • There is significant overlap in the impact described, so that the impact types and beneficiaries are broadly the same as described in the 2014 case study.

• Impacts on Health and Wellbeing

  Areas of Impact

  Impacts where the beneficiaries are individuals/groups/animals whose health outcomes have been improved, quality of life has been enhanced (or potential harm mitigated) or rights/interests been protected or advocated through enhanced policy/practice.

  Types of Impact - Examples

  • Research contributes to improved provision, access, take-up or use of health services or enhancement of patient/user experience.
  • Research contributes to change to clinical, dietary, health or social care guidelines, health service decision-making, policy or practice, legislation, medical ethics or the development/adoption of new clinical/lifestyle interventions.

  Corroborating Evidence - Examples

  Documented evidence of take-up and use of new processes that improve care practice/quality of life

  • UoA3a ICS: Making person-centred practice and policy a reality
  • Associated evidence: influence on DHSSPSNI care framework

  Documented change to clinical or health guidelines

  • UoA3b ICS: Dietary mis- reporting
  • Associated evidence: citation in UK dietary recommendations
• Impact on Social Welfare

  Areas of Impact

  Impacts where the beneficiaries include individuals, groups, organisations or communities whose rights, duties, behaviours, opportunities, inclusion or quality of life have been influenced.

  Types of Impact - Examples

  • Research contributes to campaigns for social, economic, political and/or legal change through engagement with civil society groups.
  • Research contributes to change to social policy that improves social welfare, equality, social inclusion or justice outcomes, or enhanced understanding of victims’ needs in post-conflict states.

  Corroborating Evidence - Examples

  Documented evidence of research influence on post-conflict public debate

  • UoA22 ICS: Historical enquires: holding the UK authorities and Police Service of Northern Ireland to account - informing stakeholders, changing policy and influencing public debate
  • Testimonial Evidence (Factual Statement) From Civil Society Groups:  Sample testimonial
• Impacts on Public Policy, Law and Services

  Areas of Impact

  Impacts where the beneficiaries include government, NGOs, charities and public sector organisations and society through the implementation/non implementation of policies, systems or reforms.

  Types of Impact - Examples

  • Research contributes to legislative scrutiny, policy debate, parliamentary activity, government analysis or change to public policy, legislation, regulations, guidelines or public services.
  • Research influences the policy and practice of international agencies and institutions or the policy-facing work of NGOs.

  Corroborating Evidence - Examples

  Documented evidence of research use in policy debate (e.g. direct citation in parliamentary, regulatory or policy documents)

  • UoA20 ICS: Amnesty, accountability and victims’ rights in peace processes
  • Associated Evidence:  influence on amnesty report for Ugandan government
  • UoA20 ICS: Framing transitional justice practice: dealing with the past in Northern Ireland
  • Associated Evidence: citation in government-commissioned report on dealing with the past in NI

  Documented evidence of research influence on policy/practice of international agencies/institutions

  • UoA20 ICS: Gender, conflict and transition
  • Associated Evidence: citation in UN report on women’s participation in peace building
  • Testimonial Evidence (Factual Statement) corroborating influence on policy, law and services: Sample testimonial
• Impacts on Practitioners and Delivery of Professional Services, Enhanced Performance or Ethical Practice

  Areas of Impact

  Impacts where beneficiaries include organisations or individuals/service users involved in the development and/or delivery of professional services/ethics.

  Types of Impact - Examples

  • Research influences professional standards, guidelines, training, methods, processes, ethics, practice, development, services, workforce planning, expert systems (e.g. human resources, accounting, and financial services) or the work of professional bodies and learned societies.

  Corroborating Evidence - Examples

  Documented evidence of organisational adoption of guidelines or best practice

  • UoA19 ICS: Advancing HR practice through employee wellbeing strategies
  • Associated Evidence: influence on Northern Ireland Civil Service HR practice (WELL programme) (NB: evidence encompassed WELL programme guide and NICS corroborating statement)
• Impacts on the Environment

  Areas of Impact

  Impacts where the beneficiaries are the natural, historical and/or built environment, and societies, individuals or groups who benefit as a result.

  Types of Impact - Examples

  • Research contributes to change in biodiversity practices or policies.
  • Research informs environmental or climate change policy or planning debate/decisions or the enhancement of environmental strategy, regulatory provision, operations or management practices.
  • Research contributes to environmental improvements by affecting new or enhanced products, processes or services, methods, models, monitoring or techniques.

  Corroborating Evidence - Examples

  Documented evidence of research influence on processes that bring environmental benefits

  • UoA7 ICS: The development of evidence based biodiversity policy and practice in Northern Ireland
  • Associated Evidence:  influence on NI biodiversity management policy debate
• Impacts on Understanding, Learning and Participation

  Areas of Impact

  Impacts where the beneficiaries are individuals, communities and organisations whose awareness, understanding, participation or engagement have been enhanced as a result of the research.

  Types of Impact - Examples

  • Research contributes to shaping or informing public or policy debate, challenging established norms, modes of thought or practices.
  • Research contributes to the awareness, attitudes or understanding of (sections of) the public being informed and their ability to make informed decisions on issues improved.

  Corroborating Evidence - Examples

  Citation in public or policy discussion document

  • UoA3a ICS: Addressing the health inequalities of people with intellectual disabilities
  • Associated Evidence: citation in independent review of NI health policy/services

  Documented evidence of influence on civil society campaigning to inform public or policy debate (e.g. in parliament, the media, material produced by NGOs)

  • UoA22 ICS: Historical enquires: holding the UK authorities and Police Service of Northern Ireland to account - informing stakeholders, changing policy and influencing public debate
  • Associated Evidence: contribution to NI NGO justice campaign
• Impacts on Production

  Areas of Impact

  Impacts where the beneficiaries are individuals (including groups of individuals) whose production has been enhanced.

  Types of Impact - Examples

  • Research contributes to improved productivity - enhanced production yields, quality or resource-use efficiency
  • Research creates routes to international innovation and market impact

  Corroborating Evidence - Examples

  Documented evidence of new product adoption or improved working practices, sustainability or level of production

  • UoA13 ICS: Bringing nano materials research to industrial production
  • Associated Evidence: evidence of improved production efficiency processes at company (NB: evidence encompassed business performance measures and corroborating statements both from the company and Invest NI)
• Impacts on Commerce and the Economy

  Areas of Impact

  Impacts where the beneficiaries may include businesses, either new or established, the NHS, private health and social care, agriculture or other type of organisation which may undertake activity that may create wealth.

  Types of Impact - Examples

  • Research contributes to a spin-out or new business has been created, established its viability or generated revenue/profits or business performance/productivity/governance or regulation improves in established enterprise.
  • Research contributes to innovation/ entrepreneurialism, economic prosperity or change to workplace policy/practice/strategy/operations through the design/delivery/adoption of new products, services, processes or technology or through regulatory/governance or legal change.

  Corroborating Evidence - Examples

  Documented commercialisation indicators or business performance measures (e.g. turnover/profits, sales, job creation, market authorisation or investment funding uplift)

  • UoA13 ICS: Integrated healthcare sensors underpin global connected health
  • Associated Evidence: evidence of spin-out performance (NB: evidence encompassed business performance measures, corroborating statements and media coverage)

  Documented influence on regulatory technological configuration

  • UoA16 ICS: Advancing safety of hydrogen and fuel cell technologies: HySAFETY
  • Associated Evidence: influence on technological configuration within EU automotive industry regulation
• Impacts on Creativity, Culture and Society

  Areas of Impact

  Impacts where the beneficiaries may include individuals, groups of individuals, organisations or communities whose behaviours, creative practices, rights, duties and other activity have been influenced.

  Types of Impact - Examples

  • Research contributes to inspiring, co- creating and supporting new forms of artistic, literary, linguistic, social, economic, religious and other expression or influences creative practice, its artistic quality or its audience reach or collaborations with public arts venues and programming professionals.
  • Research-led engagement with marginalised, under-engaged and/or diverse audiences leads to increased cultural participation.

  Corroborating Evidence - Examples

  Testimonials (factual statements) from creative practitioners, curators, media professionals or audience, visitor or participant feedback and figures

  • Sample testimonial 

  Documented evidence of influence on creative practice and engagement

  • UoA35 ICS: Including disabled musicians using technology
  • Associated Evidence: influence on creative inclusivity (NB: evidence item was supplemented by corroborating statements from leading arts organisations)

  Professional evaluations of exhibitions, performances or other outputs or evaluative reviews of artistic or cultural expression/output in the media

  • UoA34 ICS: Video, landscape and memory
  • Associated Evidence: 
    • media review of artistic expression 1
    • media review of artistic expression 2