Three categories or dimensions of impact

Based on the review of the Horizon 2020 Societal Challenge 5 projects, we have developed a set of indicators that as much as possible follows the rules of SMART (Specific, Measurable, Achievable and Attributable, Relevant and Timely) and PATHS (Proximity, Attribution, Traceability, Holism, Stability). The objective is to assess performance and effectiveness over the short to longer term, and the indicators therefore address all analysis concepts from the intervention logic. In line with the intervention logic, the indicators have been built in three dimensions:

• Scientific Impact
• Societal and Environmental Impact
• Economic Impact

A quick history of the development of indicators

The analysis of the different sources shows an evolution in the development of indicators, from looking at inputs (budget, number of participants, etc.) and immediate outputs (reports and publications) to the relationship between inputs, outputs, outcomes (or results) and impacts and from single indicators to impact pathways. The resulting set of indicators is a ‘fusion’ between those used in FP7 and Horizon 2020 and those proposed for Horizon Europe, taking into consideration also the analysis and proposals set forth in the Expert report ‘Monitoring the Impact of EU Framework Programmes’. Even though tracking of impacts on these three dimensions (scientific, societal, economic) is in its early stages, the results of IMPACT-SC5 will help validate the approach and deliver data to evaluate Horizon 2020 Societal Challenge 5 (SC5) as well as support the evolution to Horizon Europe, which will incorporate the impact pathways methodology in a structured way.

Time and Reach

The indicators show that two aspects are important to assess the programme’s performance:

• Time: What happens during and directly following the action and the uptake over a longer period of time?
• Audience reach: Also measured as the sphere of influence. Who is affected during and directly following the action and over a longer period of time? Ideally there is a ripple effect of diffusion, reaching more people who are becoming more pervasive to the point where a transition has taken place.

Scientific Impact

When looking at the impact pathways of research and innovation projects it is obvious to first look at the impact on the science itself first. Our study identifies five key subjects of evaluating the scientific impact of H2020 SC5 projects.

For each of these five evaluation subjects we defined short-term impact pathways (outputs), medium-term impact pathways (outcomes) and long-term impact pathways (impacts).

You can see the full list of impact pathways on science in this document here.

Societal and Environmental Impact

The second subject of evaluation of Horizon 2020 SC5 projects is the impact on society and the environment. Here we identify two subjects of evaluation:

You can see the full list of impact pathways on society in this document here.

Economic Impact

Thirdly there is the impact on the economy. Here we identify three subjects of evaluation:

Please find the full list of impact pathways on the economy in this document here.