Measuring impact
Measuring impact helps to clarify whether the objectives are being met, what is working, and what isn't. Funders or stakeholders may require updates and reports on the project's progress, including progress towards the impact goals. Grant funding and academic promotion applications require evidence to back up impact claims.
Impact is measured in two main ways:
Significance - the degree to which the impact has enabled, enriched, influenced, informed or changed the performance of policies, practices, products, services, culture, understanding, awareness or well-being of the beneficiaries.
Reach - is the extent, spread, breadth, and/or diversity of the beneficiaries of the impact, relative to the type of impact.
(Definitions from NHMRC Investigator Grants 2021 Guidelines, via GrantConnect)
Of these two aspects, significance is arguably the most important, the premise being research that reaches millions but has no notable benefit is not significant nor impactful.
Alongside determining significance and reach, it is important to provide evidence of the causal links between the research and the impacts. While impact from research is seldom linear (your research may be one of many factors that contribute to change in society), wherever possible think about how to measure the 'before' and 'after', to demonstrate any differences that has been achieved and how the changes are attributed to the research.
Capturing evidence of impact
Indicators of influence
While impact can take a long time to occur, indicators of future impact can accrue along the way. The following table outlines the 'impact continuum' pathway; each step along the way builds momentum towards the ultimate goal of effecting change or creating impact.
Influence |
|
---|---|
Adoption |
|
Change |
|
Adapted from content developed by Tim Haydon, The Write Media Network.
Examples of evidence
The following table outlines some of the many different ways evidence of impact can occur. Evidence does not have to be scholarly: personal feedback, appreciation or testimonials from end-users or industry partners is evidence that demonstrates how the research is creating change. Capture a record of these activities, testimonials, events and opportunities as they arise, so that you remember these instances down the track.
Stakeholder Information | Testimonials | Online Traffic | Positions of Responsibility |
---|---|---|---|
|
|
|
|
Sources of evidence
Potential resources for gathering evidence of research impact claims include:
Surveys and Testimonials
- Qualtrics is a simple to use web-survey tool to conduct surveys and evaluations. Anyone with a WSU email address can register for a WSU Qualtrics account. Check the Research Calendar for upcoming Qualtrics training.
- Testimonials can be an excellent way to evidence direct causation between research and impact. Encourage end-users to provide you with feedback, both good and constructive, in the form of written testimonials.
'Mentions' outside of academia
- Factiva and Australia & New Zealand Newsstream: Search across keywords, title and abstract, author and subject fields for mentions in Australian and international newspapers.
- Analysis & Policy Observatory (APO): search open access policy and practice research for mentions by name or publication.
- Search Hansard for mentions by name in the Parliament of Australia (Commonwealth), includes records of invited speeches in Parliament.
- Australian Clinical Practice Guidelines: search for clinical guidelines developed for use in Australian Healthcare settings.
- Australia & NZ Clinical Trials Register: search for Australian clinical trials underway or completed.
- WHO International Clinical Trials Registry Platform: search international trials by topic, country or region.
- The Lens: search for mentions by name across patents for over 100 jurisdictions; create alerts for future updates. Lens help resources.
- United Nations iLibrary: search digital content created by the UN, for mentions by name or title. UN iLibrary User Guide.
- Organisation for Economic Co-operation and Development (OECD): search for mentions by name or publication across economic, environmental and social policy. OECD iLibrary User Guide.
Recording evidence
- Identify areas of your research where data collection is possible and capture data as you go.
- Collate statistical and narrative information on the research activities, outcomes and any resulting impacts, as they occur during the project.
- Having baseline data to compare along the way will help monitor the performance of your work.
- It is much easier to keep track of this information as it occurs, rather than gather it retrospectively.
Email folder Set up an email folder to email yourself reminders of things that have occurred, or to file any relevant correspondence, feedback, or any of the examples mentioned in the previous tab.
Google Alert If your project has a specific name, or is likely to be capturing media attention with distinctive keywords (or your name is likely to be mentioned), set up a Google alert to capture mentions of your name or the research project.
Spreadsheet/log Create an activities/outcomes/impact log, using a spreadsheet with relevant tabs to capture the data; or use a template such as this one from Fast Track Impact: editable word version or PDF.
Shared notebook Enable team members and collaborators from any institution to collate impacts in a shared online notebook, such as Evernote.
Social media and online statistics Record downloads/viewing statistics and social media comments. Explore Altmetrics to discover who is engaging with the research online. Altmetric Explorer includes mentions of policy documents and patents.
Consultation with a School Librarian via WesternNow for further assistance