Don’t be submissive
24 September 2013 9 Comments
Repeat after me:
Submitting a grant application is not a valid Key Performance Indicator (KPI).
If you manage research staff, write out the following line fifty times:
I will not measure ‘dollars won’ as a Key Performance Indicator (KPI).
After Tseen posted her plea for “no more half-baked applications“, one of our twitterati pointed out that sometimes applications are half-baked because submitting the application satisfies a key performance indicator in a work-plan.
I know that there might be a strong temptation to list either ‘grants submitted’ or ‘dollars won’ (or both) as key performance indicators. As a Research Whisperer, this drives me crazy! In the first case, it pushes people to write grant applications that they can’t win. In the second, it adds completely unpredictable factors like government funding policy into the performance review process. In both cases, it measures the wrong thing.
Measuring research performance, like any performance measurement, is fraught with difficulty. In essence, you are trying to measure the quality or value of the research completed. This is a subjective process. The true value of research may not emerge for many years. It would be great to look at the contribution made to a field of knowledge over time in terms of new and powerful concepts, ideas, and methods, but generally that isn’t practical.
‘Indicators’ are meant to stand in for other quality measures. Generally, indicators are objective measures that can be ‘counted’ easily and quickly.
Some indicators that I like include:
- Final reports, which indicate that a research project has been satisfactorily completed.
- Research outputs such as papers, patents, and art works, which indicate that a research project has produced new knowledge.
Indicators that drive me crazy include:
- Applications submitted.
- Number of grants received or total funding received.
This is especially true when a rider is added that the application must be to a ‘top tier’ funding body (usually the largest national funding bodies and research councils in a country).
For a small organisation, an emerging research area, or a harried academic, getting a grant application across the line might be a major achievement. However, it is not an indicator of ‘key performance’, nor is it an indicator of research undertaken.
A grant application is an input, not an output. When people apply for funding, they are describing research that they might like to do in the future.
I understand why some managers might like to count applications as a KPI. In a non-traditional research area, applications might seem like the first step towards research outputs. They certainly represent a lot of hard work on the part of the applicant. However, it is a false measure. All too often it leads to half-baked applications, written in haste to ‘tick the box’.
I can also understand the attraction of counting ‘dollars won’ as a KPI. If someone wins a grant, that is an external measure of their excellence. In addition, many governments provide infrastructure funding to universities based, in part, on the amount of research funding they receive. Dollars are easy to count; there is a whole accounting system charged with counting the number of dollars coming into (and going out of) a university.
As I stated earlier, ‘dollars won’ or ‘grants won’ are false measures of research performance.
Research funding precedes research. Yes, it is an external vote of confidence, but the amount of money available, and the competitiveness of the funding ecosystem is completely beyond the control of the applicant, research manager, and university. Governments can reduce their research funding envelope, or even close whole schemes down. Industry partners change their strategic direction, or eliminate their research spend completely. This means that you don’t have any control over whether the KPI is achieved, which makes for an unrealistic, unpredictable KPI.
And don’t get me started on managers who only want to count applications written or grants won from particular ‘top tier’ funding schemes. Not only are they counting the wrong things, but they are pushing onto their staff a particular vision of how some management thinks the world works. If you get $20,000 of research funding, I don’t care if it comes from an international, Federal, or State research fund, a philanthropic fund, or an industry supporter. It counts as $20,000 in research funding. Go and do $20,000 worth of excellent research with it. Create new knowledge. Solve problems. There will be some outputs from that process. Let’s measure those outputs.
So, if you are a researcher, don’t be submissive. When it comes time to set your work-plan and your key performance indicators, make the case for KPIs that count. Then write the best applications that you can.
If you are a manager, measure the things that matter. Count the outputs, not the inputs. After all, you want your staff to be research active, not research submissive.