Measuring Impact

Home
Impact
Measuring Impact

Measuring impact

Measuring impact helps to clarify whether the objectives are being met, what is working, and what isn't. Funders or stakeholders may require updates and reports on the project's progress, including progress towards the impact goals. Grant funding and academic promotion applications require evidence to back up impact claims.

Impact is measured in two main ways:

Significance - the degree to which the impact has enabled, enriched, influenced, informed or changed the performance of policies, practices, products, services, culture, understanding, awareness or well-being of the beneficiaries.
Reach - is the extent, spread, breadth, and/or diversity of the beneficiaries of the impact, relative to the type of impact.
(Definitions from NHMRC Investigator Grants 2021 Guidelines, via GrantConnect)

Of these two aspects, significance is arguably the most important, the premise being research that reaches millions but has no notable benefit is not significant nor impactful.

Alongside determining significance and reach, it is important to provide evidence of the causal links between the research and the impacts. While impact from research is seldom linear (your research may be one of many factors that contribute to change in society), wherever possible think about how to measure the 'before' and 'after', to demonstrate any differences that has been achieved and how the changes are attributed to the research.

Capturing evidence of impact

Indicators of influence

While impact can take a long time to occur, indicators of future impact can accrue along the way. The following table outlines the 'impact continuum' pathway; each step along the way builds momentum towards the ultimate goal of effecting change or creating impact.

Influence	End-users become aware of your research through: ad hoc and informal channels (e.g. social media post or LinkedIn feed). targeted/formal/industry specific channels (e.g. a poster at a scientific conference). Indicates active engagement with end-users and dissemination of research.
Adoption	End-users adopted or applied your research in an evaluative, preliminary or tentative way, e.g. cited in policy documents, Hansard etc as a basis for decision making. end-users trialling or piloting some aspect of the research. IP licensed for R&D purposes. End-users formally endorsed or incorporated the results, outcomes, methodologies, technologies and/or recommendations from your research, e.g. incorporated into policy, guidelines, procedures, standards, legislation, regulations, strategies, educational curricular, training programs etc. new service or product arising from the research successfully passing the testing stage and entering commercialisation.
Change	End-users applied the results, outcomes, methodologies, technologies and/or recommendations from the research, e.g. implemented new programs, policies, practices, methods, tools etc. new or improved services or products enter the market. Beneficiaries modified their activities, practices, behaviours, beliefs or understandings or in some way benefited from the new or improved practices, services, products, activities, behaviours of end-users, as a result of the research, e.g. citizens, community, public demonstrably changed their behaviours, beliefs or understanding. services or products were used or purchased by consumers. students’ learning outcomes improved demonstrably. patients’ health outcomes improved demonstrably.

Adapted from content developed by Tim Haydon, The Write Media Network.

Examples of evidence

The following table outlines some of the many different ways evidence of impact can occur. Evidence does not have to be scholarly: personal feedback, appreciation or testimonials from end-users or industry partners is evidence that demonstrates how the research is creating change. Capture a record of these activities, testimonials, events and opportunities as they arise, so that you remember these instances down the track.

Stakeholder Information	Testimonials	Online Traffic	Positions of Responsibility
Annual reports from companies, government, NGOs, institutes Press coverage Work cited in further funding applications by community/ voluntary groups Surveys Dialogue through public events Newsletters Community meetings and minutes Company websites and press releases Citation by international charity and governments National statistics (ABS data)	Letters of support One-to-one testimonials Focus groups Event feedback Personal letter from individuals at third party organisations Surveys: Paper and online On-going testimonials from community party Awards	Comments on TV programmes/ News articles and websites Social media website hits, tweets	Steering group positions Secondments – offer letters Dialogue through public facing (recorded) events HANSARD

Sources of evidence

Potential resources for gathering evidence of research impact claims include:

Surveys and Testimonials

Qualtrics is a simple to use web-survey tool to conduct surveys and evaluations. Anyone with a WSU email address can register for a WSU Qualtrics account. Check the Research Calendar for upcoming Qualtrics training.
Testimonials can be an excellent way to evidence direct causation between research and impact. Encourage end-users to provide you with feedback, both good and constructive, in the form of written testimonials.

'Mentions' outside of academia

Factiva and Australia & New Zealand Newsstream: Search across keywords, title and abstract, author and subject fields for mentions in Australian and international newspapers.
Analysis & Policy Observatory (APO): search open access policy and practice research for mentions by name or publication.
Search Hansard for mentions by name in the Parliament of Australia (Commonwealth), includes records of invited speeches in Parliament.
Australian Clinical Practice Guidelines: search for clinical guidelines developed for use in Australian Healthcare settings.
Australia & NZ Clinical Trials Register: search for Australian clinical trials underway or completed.
WHO International Clinical Trials Registry Platform: search international trials by topic, country or region.
The Lens: search for mentions by name across patents for over 100 jurisdictions; create alerts for future updates. Lens help resources.
United Nations iLibrary: search digital content created by the UN, for mentions by name or title. UN iLibrary User Guide.
Organisation for Economic Co-operation and Development (OECD): search for mentions by name or publication across economic, environmental and social policy. OECD iLibrary User Guide.

Recording evidence

Identify areas of your research where data collection is possible and capture data as you go.
Collate statistical and narrative information on the research activities, outcomes and any resulting impacts, as they occur during the project.
Having baseline data to compare along the way will help monitor the performance of your work.
It is much easier to keep track of this information as it occurs, rather than gather it retrospectively.

Email folder Set up an email folder to email yourself reminders of things that have occurred, or to file any relevant correspondence, feedback, or any of the examples mentioned in the previous tab.

Google Alert If your project has a specific name, or is likely to be capturing media attention with distinctive keywords (or your name is likely to be mentioned), set up a Google alert to capture mentions of your name or the research project.

Spreadsheet/log Create an activities/outcomes/impact log, using a spreadsheet with relevant tabs to capture the data; or use a template such as this one from Fast Track Impact: editable word version or PDF.

Shared notebook Enable team members and collaborators from any institution to collate impacts in a shared online notebook, such as Evernote.

Social media and online statistics Record downloads/viewing statistics and social media comments. Explore Altmetrics to discover who is engaging with the research online. Altmetric Explorer includes mentions of policy documents and patents.

Consultation with a School Librarian via WesternNow for further assistance