Our IMPACT-SC5 project in a nutshell

The objective of our IMPACT‐SC5 project is to identify and analyse the impact that the Horizon 2020 Societal Challenge ‘Climate action, environment, resource efficiency and raw materials’ (referred to as SC5) has had on science, society, the environment and the economy. Specifically, IMPACT-SC5 reviews the 87 Research and Innovation Actions (RIA) and Innovation Actions (IA) projects resulting from the 2014 and 2015 calls. The aim is to understand why certain impacts have or have not been achieved.

In a series of items, we will report on the approach and findings of the IMPACT-SC5 project. This first article describes the approach to the development and definition of the indicators used in the evaluation of the 87 projects under review.

The novelty is that IMPACT-SC5 developed a methodology and indicators for impact pathways, which is being used to investigate the individual projects as well as project portfolios. The impact pathways approach looks at the performance and achievements of research and innovation policies and aims to overcome limitations to conventional tools to predict non-linear research and innovation impact trajectories embedded in the context dependent system. Impact pathways are characterized by learning, backward and forward feedback loops, trial and error.

Horizon 2020 differentiated from its predecessors by its focus on innovation and increased orientation to impacts. It established performance indicators aimed to facilitate and measure the overall impacts of the funded actions. While a step in the direction of being able to assess the performance and effectiveness of European research and innovation policy, the indicators are not considered enough to fully capture the outcomes and impacts of projects. Reasons for this are for example that the time between the funding of activities and achieving impact can be several years and the difficulty of attributing impact to a single funding source (as most research teams receive funding from multiple and different sources). Development of the right indicators and access to quantity and quality of data are also issues impacting the quality of the evaluation.

IMPACT-SC5 can be considered a test case and pilot project for monitoring and evaluation in the next EU Research and Innovation investment programme, Horizon Europe. This new programme, running from 2021-2027, introduces key impact pathways to track progress. The evolution in impact pathways from Horizon 2020 to Horizon Europe is shown in the figure below.

IMPACT-SC5 evaluates the 87 projects along three impact pathways to assess performance (project outputs creating short-term impact) and effectiveness (the outcomes in the medium-term and the impacts in the long-term upon completion of the projects):

• Scientific impact.
• Societal and environmental impact.
• Economic impact.

To assess the impact along these three dimensions, the following set of indicators was developed. The full report on the indicator development is available in the Results section of this website (Deliverable D1-2) while in parallel you may find individually the scientific, societal as well as economic indicators by clicking the respective hyperlink.