The measurement tail should not be wagging the impact dog

Helen Sowey, smilingHelen Sowey was Senior Research Support Officer at the Faculty of Arts & Social Sciences, UNSW, from August 2017 to October 2018. Prior to this, she spent 20 years working as a practitioner in the health, justice, and social services sectors. Contact helen.sowey@gmail.com.

This is an edited version of a paper presented at the Australasian Research Management Society Conference, Hobart, 20 September 2018.


A pop art representation of a puppy dog, mostly in different shades of blue

‘Blue Dog’ by Romero Britto. Photo by Jonathan O’Donnell.

Australia’s Engagement and Impact Assessment encourages universities to ensure that their research is of benefit to the world beyond academia.

Or does it?

Having spent more than a year in a dedicated “engagement and impact” research support role, I am concerned to see that institutions tend to be narrowly focused on the task of showing evidence of engagement and impact, rather than thinking about what kind of impact their work might have and what kinds of engagement would allow that to happen.

This is problematic, because knowing what kind of impact is intended is a logically prior step to collecting evidence of it! If you don’t know what you are aiming for, you can hardly hope to achieve it, much less document it.

The focus on showing evidence is also problematic because it shifts attention away from creating impact, which is something grand, visionary and inspiring, towards creating only the kind of impact that is measurable – a smaller, more individualistic, and potentially less relevant endeavour.

Whether it’s the Engagement and Impact Assessment in Australia, the Research Excellence Framework (REF) in the UK, or any similar exercises in other countries, the indicators that we are asked to report against are only ever going to be proxies for more complex real-world outcomes. The impact case studies that we are asked to provide are simplifications of more complex stories of how research creates societal benefit. A generic measurement system, designed to assess progress against a very high level goal, is never going to serve as a constructive guide for what an individual’s engagement and impact goals should be.

I would like to imagine a world in which the engagement and impact agenda motivates us to think more about having impact, and less about proving it.  Here are three ideas for moving towards this vision.

1. Researchers need to define impact for themselves.

Researchers need to look at the government’s very high-level statements (e.g. “impact is the economic, environmental, social, and other benefits of research”) and define what, specifically, is the impact that they are aiming for.

If researchers are accustomed to thinking about the outcome of their research as being a publication, then they’re going to get stuck. But if they start with the idea that the scholarly outcome of their work is “new knowledge”, or “a new perspective”, then already the path to impact starts to open up. Researchers could consider:

  • Who might be interested in this new knowledge or new perspective?
  • Who might be affected by it?
  • If this new knowledge or new perspective took wings and landed outside the academy, what benefit might it have on society?
  • And a more challenging question: might it benefit some, at the expense of others?

2. Engagement and impact need to remain conceptually connected.

Impact is what you hope to achieve. Engagement is what you do to achieve it.

Engagement activities should not be random – engagement is a strategic approach to achieving an intended impact. Before you can “do” engagement, you need to decide who you want to reach, and how you might effectively reach them.

While institutions remain focused on assessment exercises in which “engagement” and “impact” are measured separately, then the conceptual link between them can get lost.

3. Institutions need to create reflective spaces.

I am advocating that universities create reflective spaces where researchers can think about engagement and impact in a way which is not framed and limited by the parameters of the assessment exercise.  Researchers need space to think about what impact means for them, what types of engagement it would entail, and whether they wish to pursue it.

Researchers need reflective spaces to grapple with challenging questions, such as:

  • Will there be a trade-off between research quality and research impact?
  • Will partnership funding compromise independence and academic freedom?

Reflective spaces can be hard to come by in our highly pressured world. But if we can’t create them in our scholarly institutions, then where can we hope to find them?

The measurement tail should not be wagging the impact dog. When we think about engagement and impact, and even as we go about the necessary task of collecting data for assessment exercises, we should not lose sight of the fact that the purpose is to benefit the world beyond academia.

Advertisements

6 Responses to The measurement tail should not be wagging the impact dog

  1. Reblogged this on Digital learning PD Dr Ann Lawless and commented:
    Real impact and engagement are connected

    Liked by 1 person

  2. Andrew Wray says:

    I enjoyed this article and don’t think it’s too late at all.
    Yes, when the UK is close to REF 2021 and Department X seems to be one case study short, then all minds turn to filling that gap. And as our academics submit draft case study narratives and evidence this week for internal assessment, then we are all concerned with the quality and quantity of the impacts being claimed.

    However, the conversations are quite different when I talk to medical or social science researchers who want to address child obesity or improve dementia care. They are searching for long term partnerships with charities, businesses and our National Health Service and will think later about measuring success (or use the measures that these partners adopt).

    Our Hispanic Studies department are working in Latin America on peace processes and untold community history projects. Their concerns are to improve understanding between ethnic and political groups, with results that may or may not be measurable.

    The advice that our Commercialisation and Impact Development team gives academic colleagues is “Do the right thing, with the right partners”. Then the impacts will surely come.

    And many researchers take exactly this approach.

    Liked by 2 people

  3. Helen Sowey says:

    Thanks for your comments, Andrew. I think the way that administrative staff frame conversations on impact can certainly shape the way that academics think about it. I’m optimistic when I hear how your Commercialisation and Impact Development team is talking.

    Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.

%d bloggers like this: