The effect of impact

This article first appeared in Funding Insight on 20 September 2018 under the title “Don’t fear the bogeyman”. It is reproduced with permission of Research Professional. For more articles like this, visit www.researchprofessional.com.


“I … have made impactful contributions to industry and practice…”

As a research whisperer, I spend my life helping people to refine their grant applications. An important part of that is wielding the ‘big red pen of clarity’, and editing their material to help express their ideas more clearly.

You can imagine my reaction when I read ‘impactful’ in a grant application recently. I was appalled. In the Australian vernacular, I nearly choked on my Weeties. This horror appeared in an otherwise excellent application, written by an otherwise excellent applicant. We had words…

At about the same time, Tim Sherratt tweeted:

André Brett replied,

The corrosive effect of the impact agenda

A marriage bed, ruined by a caved-in roof, in a forgotten building in Southern China.

The effect of time (detail), by Jonathan O’Donnell on Flickr.

These tweets helped me to reflect on the corrosive effect that the impact agenda is having on the researchers with whom I work.

Like a lot of things, the impact agenda seems quite reasonable on the surface. The government provides funds for research, and they want to know that those funds are helping society in some way.

They’ve asked universities to describe what contribution their research makes to the economy, society, environment or culture, beyond the contribution to academia. All well and good, in theory. Unfortunately, some of my researchers are getting mixed messages about the impact agenda.

We already know that researchers are angry because they feel that they are being asked to lie. Jennifer Chubb and Richard Watermeyer asked 50 senior academics about the ‘impact’ section in Australian and UK grant applications. One respondent, a professor, said that “it’s really virtually impossible to write an (Australian Research Council) grant now without lying and this is the kind of issue that they should be looking at”.

However, it goes further than that. Because the air is so thick with an expectation of ‘impact’, I’m seeing more and more statements like the one that opened this article. They represent an attitude on the part of some researchers, especially early career researchers, that troubles me.

These researchers write as though they were trying to convince politicians that their research is worthwhile. They are afraid that, in the five or six years since they have finished their PhD, their research will not have made an impact. Of course, it won’t. You are lucky to be cited in the early years of your career, much less have demonstrated effect on industry or practice.

Don’t fear the bogeyman

These researchers seem to be writing for a nebulous audience inspired by newspaper headlines, policy statements and university expectations. The impact bogeyman, in other words.

Hot tip: The impact bogeyman won’t be reviewing your grant application.

If you are applying for funding from a national research council, you’ll be assessed by other researchers. Other researchers who understand what can, and can’t, be expected from a three-year research project. If the funding agency includes ‘impact’ in their criteria, they will spell that out in their guidelines. Read the guidelines.

Like all childhood fears, the impact monster disappears when you look directly at it. For example, in the latest Rules (2018) for the Australian Research Council’s (ARC) Discovery Program (the scheme that I am most familiar with) ‘impact’ is not mentioned at all, in the whole 80 page document.

Elsewhere, the Instructions for Applicants for Discovery Project (DP19) grants only mentions research impact in four places (over 42 pages):

  • When it talks about the ‘benefit and impact statement’, which appears on the front page of the application.
  • When talking about career interruptions it asks if career interruptions have “impacted… your academic record”.
  • When asking for details of your academic career and opportunities for research it mentions the word ‘impact’ once (in 600 words of instructions).
  • When you list your ten best academic career outputs, it asks you to explain and justify the impact or significance of each research output.

Similarly, in its Assessor Handbook, it only mentions research impact in two places:

  • As one of five important factors to consider when assessing the feasibility and benefit section of proposals. The other four factors are: objectives and selection criteria; research opportunity and performance evidence; interdisciplinary research; and data management.
  • As one element that informs the assessment of the ‘benefit’ section of the proposal, for just one of the six schemes that they run: Linkage Infrastructure, Equipment and Facilities (LIEF) scheme.

There you have it. In 150-odd pages, over three key documents, the Australian Research Council talks about the impact agenda six times. I’m not saying it isn’t important. I’m just saying that you need to keep it in proportion when writing your grant application.

To keep it in proportion, keep in mind Andre’s point about the difference between ‘effect’ and ‘impact’. Most research programs will have, at best, an effect on the issue at hand. Effect is defined by the OED as ‘a change which is a result or consequence of an action or other cause.’

To have an impact, the effect would need to be marked. That is, it would need to clearly noticeable. Once again turning to the OED, impact is defined as ‘the action of one object coming forcibly into contact with another’, or ‘a marked effect or influence’.

Most research projects don’t have a noticeable effect on the outside world because research is cumulative. Research accretes over time, with each project providing another small piece of the puzzle. To try to disentangle which thread of research actually made a difference is enormously difficult. To try to predict what effect your tiny thread may have in the future is almost impossible.

Practical steps

Here are some practical ways to keep things in perspective.

  1. Read the guidelines to see how they talk about the impact agenda. This will provide an antidote to the unstated assumptions and expectations that you, or your colleagues, may be carrying.
  2. Understand your audience. If your application is going to be assessed by academics, write for academics. If your application is going to be assessed by industry, write for industry. Write for your actual audience, not the impact bogeyman. How do you know who will be assessing your application? Read the guidelines.
  3. Understand how you can help. If you are seeking to help a target population, or influence policy or contribute to the solution of a problem, talk to the people who might implement your solution. Find out what their expectations are. Understand what part of those expectations you might be able to fulfil. Be realistic.
  4. Think about change over time. What change is realistic during the project? What change is realistic within the first year after the project? How about 2-3 years after the project? Don’t go too much further than that – it is unrealistic. (Hinton et al. 2011)
  5. Think about who might change. How might things change for research participants? Who might be early adopters? Who could initiate systemic change? Have a look at Tilly Hinton’s IMPEL ladder to help with this.

Mapping out all those factors might help you to develop a sense of what the pathway to impact might be in this area, and what your contribution to that pathway might be. More importantly, it may help you to see that impact is more mouse than Gruffalo.

Advertisements

About Jonathan O'Donnell
Jonathan O'Donnell helps people get funding for their research. To be specific, he helps the people in the College of Design and Social Context at RMIT University in Melbourne, Australia. He has been doing that, on and off, since the 1990's (with varying degrees of success). He loves his job. He loves it so much that he has enrolled in a PhD to look at crowdfunding for research. With Tseen Khoo, he runs the Research Whisperer blog and @ResearchWhisper Twitter stream, about doing research in academia.

11 Responses to The effect of impact

  1. Nian (Jenny) Jiang says:

    Hi Jonathan,
    Thank you very much for sharing your thoughts just at the door of ARC Major Grants Round. What do you think of the idea of planning impact at the application stage? While the impact is not emphasised that much in the funding guidelines, it is on the agenda of government and many universities. As such, it forces us to think about impact at the application stage.
    I think it would be good to view the application in the background of the bigger environment.

    Like

    • Jonathan O'Donnell says:

      Hi Nian

      I think that it is really important to think about possible end results of your project right from the planning stages, especially if you can do it in conjunction with the end users of your research.

      I just think that you should be realistic about what can be achieved within the scope of your project, and not overestimate the effect that your work will have. That just creates false expectations in the mind of the end users, the funders and possibly yourself.

      Liked by 1 person

  2. In the 2016 Discovery ROPE instructions it says “Describe your Research Impact relative to opportunity and in the context of discipline/end user benefits. Outline significant achievements and outcomes that have contributed to a tangible impact for end users” Has that changed? This was the toughest thing for me to do in the whole proposal.

    Like

    • Jonathan O'Donnell says:

      Hi David. The ARC Discovery Project (DP20) instructions to applicants aren’t out yet.

      Two documents that were released recently are the Laureate instructions and the Centre of Excellence instructions.

      The Laureate instructions say:
      “Details of the Australian Laureate Fellowship candidate’s academic career and opportunities for research, evidence of research impact and contributions to the field, including those most relevant to this application.”

      The Centre of Excellence instructions say:
      “Please provide a statement of about 1000 words outlining further evidence of your research impact and contributions to the field, from 2005 onwards.

      Describe your research impact relative to opportunity and the context of discipline/end-user expectations. Outline significant achievements and outcomes which have contributed to the impact for end-users. The Research Impact Pathway Table may provide some help in answering this point.”

      So it isn’t in the Laureate instructions, but it is in the Centre of Excellence instructions. Take from that what you will.

      Like

  3. I agree which much of what you are saying but I then wonder what we are to make of the Research Impact Pathway Table which is explicitly connected to grant applications on the ARC page at: https://www.arc.gov.au/policies-strategies/strategy/research-impact-principles-framework/?

    Like

    • Jonathan O'Donnell says:

      Hi Rachel. I quite like what the ARC say on their Research Impact Principles and Framework page. There is really nothing there to argue with – it is all good, sound, basic advice.

      However, I find the Research Impact Pathway Table to be almost totally useless. There seems to be a clear disconnect from what they say on the page and what appears in the table.

      That is why I point people to Tilly Hinton’s work. It is much more practical, and much more useful. I believe that drawing on Tilly’s work (or pretty much any work done in this area) will give a stronger narrative than drawing on the Research Impact Pathway Table.

      Like

  4. Hi Jonathan, now that I’m in the UK system (and didn’t write ARC/NHMRC grants in Aus, so not super familiar with the impact agenda there), I’m wondering whether you can comment at all on “impact” over here, from speaking to your international Research Whisperers? I find “impact” the scariest part of REF, and “Pathways to Impact” on grants should be easy for me given my research field, but on last grant rejection was told that my ideas for getting to impact (improving toilets) were not “novel” enough.

    Like

    • i.e. not only do they want us to have impact, but they want us to achieve it in new and exciting ways!

      Like

    • Jonathan O'Donnell says:

      Hi Dani

      What can I say… You are addressing a long-standing problem, you have a ‘boots on the ground’ attitude, and your research is vital to people around the world.

      I hope that there were other reviewers that understood the real impact of your research. That is, don’t just concentrate on one reviewer (either good or bad).

      We know that researchers in both the UK and Australia are angry about having to predict the impact of their work in their applications:
      Chubb, Jennifer, and Richard Watermeyer. ‘Impact Sensationalism: A Means to an End?’ The Research Whisperer (blog), 29 August 2016.

      I’d like to say ‘get a better funder’ – look to philanthropics who do understand that making real change is worthwhile for it’s own sake. Unfortunately, system-wide reviews like the UK Research Excellence Framework and the Australian Engagement and Impact Assessment leave nowhere to hide. You can change your funding, but these all-pervasive measures will scoop you up, no matter what.

      In the end, all I can say is stay true to your research. You know that you are doing the right thing. The people that you help understand the value of what you do. It is hard work, but it is worthwhile. Take your satisfaction from that. You can’t be all things to all people – it draws you too thin. Don’t try to please all aspects of the system, and especially don’t try to please the aspect of the system that doesn’t value what you do. Be a ratbag researcher.
      O’Donnell, Jonathan. ‘Ratbag Research’. The Research Whisperer (blog), 9 April 2018.

      Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.

%d bloggers like this: