Mixed-up methods

A street sign leading to five different destinations, in Chinese and English.

Where to?, by Jonathan O’Donnell on Flickr.

I’ve been seeing a lot of applications lately where the methods section starts something like this:

In this project, we adopt a mixed methods approach…

It is a statement that I’m coming to loathe, because the author is generally saying:

Warning: muddled thinking ahead.
In the following section, we are going to provide a grab-bag of methods. Their connection to the research question will be weak, at best. Hell, they probably won’t even be well linked to each other…

There are no mixed methods

In a grant application, the purpose of the methods section is to show how you’re going to answer your research question. Not explore the question. Not interrogate the space. Not flail about without a clue.

Your methods should present a road-map from not-knowing to knowing. If you are using more than one method (and, let’s face it, who isn’t?), you need to show that your methods will:

  1. Work (get you to your goal); and
  2. Link together (be greater than the sum of their parts).

You need to show me both of those things, not just one of them (or, as is sometimes the case, neither of them).

My methods will work

You must demonstrate feasibility. ‘Had we but world enough, and time’, anything is possible. But you don’t have unlimited time and resources. You need to be able to show that your team, with your data, using your methods, can solve your problem, within the time allowed.

You need to be able to make a very convincing argument of how that will happen. So, for example, it isn’t enough to say:

Our analysis of this data will advance our understanding of this problem.

You need to explicitly say:

  1. We are using this particular analytic technique because…
  2. This technique unlocks this particular aspect of the data…
  3. Unlocking that is important to solving the problem because…
  4. Once that part of the puzzle is solved, we can…

That last point – about part of the puzzle leading to the next stage – is really, really important. It isn’t enough to show that the thing is feasible. You need to be able to explain that each step leads to the next step.

Sometimes, this is straightforward. For example: The workshop informs the questionnaire, which forms the basis of the interviews. Even if it is straightforward, you still need explain it.

My methods link together

I see four main criticisms of methods sections:

  1. Not enough detail (most common criticism eva!).
  2. Don’t see how the different parts relate.
  3. Don’t see why a particular method is being used.
  4. Too complex (not often, but it’s a killer).

When thinking about mixed methods, you need to concentrate on #2 & #3. They are both failures to describe how the different methods link together. If the reviewer doesn’t see how the different parts relate to one another, that isn’t their failure to understand. It is your failure to explain.

You have failed at one of several key points. Either you haven’t thought this through carefully enough, or you haven’t planned it out in enough detail, or you haven’t explained it well enough.

Applicants can seem confident things will work out, but they aren’t sure how. Essentially, they are saying ‘trust us, we will muddle through’. You can imagine how well that goes down with assessors.

Sometimes, applicants know that their techniques will work (often because they have done it before), but they haven’t clearly mapped out the process. This generally comes through as a lack of detail (see point #1). The methods section is squishy, like jelly. It looks solid, but when you poke it, it turns out to be soft. It doesn’t stand up to a rigorous review. So, you get something like:

We’ll do method one.
[Hand waving]
Then we’ll do method two.

There is no clear link between the methods.

Then there are the applicants who know exactly what they are going to do, and why, but just don’t write it down clearly enough. This can sometimes read like a very confident version of the ‘trust us’ problem. The method section skates along, gliding from point to point, assuming that the reader has all the same experience as the writer. This works well when your application falls into the hands of other experienced researchers, but it can come unstuck when your reviewer is less experienced, or from outside your field, and can’t skate along with you. They’ll be looking for the detail, and it isn’t there.

A simple example

I often see applications where people are going to undertake a survey, and then they are going to do some interviews. This seems straightforward until I start to look for the details.

  • Will you draw your interviewees from your survey respondents, or are they two completely different cohorts?
  • If they are linked, how will you identify them and contact them?
  • If they are independent cohorts, how will you link your two data-sets? Will they be combined, or will they be analysed separately?
  • If you want to combine the analysis, how will that work since the data-points are so different?
  • If you are going to analyse them separately, when and how will you bring it all together?

If you aren’t going to synthesise your findings, doesn’t that mean that you are doing two different projects concurrently?

To avoid this problem, give your draft to a critical friend, preferably in another discipline, or your local research whisperer. Ask them to read it, and read it hard. Then sit down with them and answer every single question they have about the methods section. Debate with them the merits of the actual methods that you are going to use (without descending into a flaming war). Be open to their suggestions, even the ones that make you feel pedestrian or inexperienced. Explore that discomfort. By the end, you should have a really clear idea of how another reader sees your project.

Now, go back and look at your methods section again. Be ruthless. If you are just using the same methods that you always use, think about how well they are working. Are you using them because they are the best methods to use, or because you are comfortable with them?

If you are using four different methods because there are four different people in the team, have a long discussion about what you are collectively planning to do and why. This will be a difficult discussion, but it is much better to have it before you submit your application or start your project, than when you are trying to answer hostile criticisms, or when you are in the middle of the research mess.

Finally, you will be ready to go back to your application and delete the statement, ‘We have adopted a mixed methods approach.’ There are some statements that are so general that they don’t mean anything (thanks, @ThomsonPat). This is one of those statements. You don’t need it. Your methods description will be detailed and thorough and complete. It won’t need this useless label because, in the end, all methods are mixed.


About Jonathan O'Donnell
Jonathan O'Donnell helps people get funding for their research. To be specific, he helps the people in the College of Design and Social Context at RMIT University in Melbourne, Australia. He loves his job. One day a week he does his own research into privacy, identity and transactions on the Internet. He likes that day, too, even when it makes his brain hurt. At the moment, he is spending a bunch of time looking at crowdfunding for research. In fact, he has enrolled in a Masters by Research to do just that. He'll let you know how it goes.

8 Responses to Mixed-up methods

  1. Kerry London says:

    Great article and humorous as always. regards Kerry


  2. Life Mutated says:

    In a way, I envy those applying for a grant because of the rigorous vetting it has to go through. A grant application gives structure and direction to the project, something the inexperienced don’t quite realize the value of, all they see is the need for funds?

    Although it’s not the exact same thing, the principle still applies. Not once in my 17 year long career have I worked for a company with proper planning procedures. You would think it to be a given, a necessity no software house in their right mind would forego but here’s the shocker, it’s considered ‘taboo’ to boot … if your found doing anything other than splashing code on a page that makes a button magically appear, you are in for a reprimand. You are expected to work on a wing and a prayer … or maybe wish for a miracle and have the ceo suddenly realize what goes into making a proper software package.

    Having tried to exhaustion to make companies like these see my way I have resigned my self to the belief that I have the knack of finding employment with the worst of the worst, seeing as most fellow developers I come across, swear high and low that they don’t do anything but plan to ad nauseam.

    For the work you do, I doff my hat to you sir.

    I just happened to read your about section and one of my pride and joy’s were a transaction gateway for internet transactions. This was some years ago and back then it was already a pain staking process of black-lists, grey-lists, bans on credit cards from other countries, all just to have some semblance of card holder protection. The fundamental rule when it came to card not present transactions, however, always remained one of ‘card holder priority’, one that worked most of the time but has also left many a virtual merchant out of pocket and out of product meaning it was more a dilemma of merchant security than the card holders as the bank would almost always refund the holder as a first step. But as I say, this was many years ago and things may have changed?


    • Jonathan O'Donnell says:

      Ta. I’ve worked in places without proper planning (or who substitute interminable meetings for ‘planning’). I agree – no fun at all.

      My mates ran one of the early ISPs in Australia. Credit card fraud drove them to despair until they added a real-world check into their system. When you registered your domain name, someone would ring you and check that you really existed. They might have lost a couple of real customers (who wanted their domain now and couldn’t wait for a call), but it was worth it to have the fraudsters drop away completely.

      Liked by 1 person

  3. Pingback: The Mock Panel – Decision time…. – Funding your research

  4. kari says:

    Hm, research vetted in this way must be ground breaking. If I understand exactly a) what I am going to find, b) how to get there, c) what I need to get there – well that is not research imho. That is basically production: the problem has been solved already (intellectually), all I need is some money to get there. If all research was done that way, we wouldn’t learn anything new. The future is created, not predicted, maybe we should keep that in mind.
    Btw, this is not to advocate non-rigorous research, but if I can foresee everything already, there is no risk. And to paraphrase Latour: “if it’s not risky it’s not research”.


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: