Lazy sampling

An old wooden post with grass growing out the top.

Old post, by Jonathan O’Donnell on Flickr

There are lots of different sorts of sampling techniques (both random and non-random) and a myriad of books that explain the best one to use for any given methodology. ‘Snowball sampling‘ is my favourite – you start with one person and ‘sample’ them, then you ask them who you should talk to next, and so on. I like the convenience (when it is used well), and I love the name.

It seems to me that lots of people use a technique that I don’t like at all.

Let’s call it “Lazy sampling”.

“Participants in the study were 35 undergraduate students (24 women, 11 men) aged 18 to 26, recruited from a large university in [the area where the authors work]. We recruited participants in the [sociology] lab at the main campus and many received extra credit in their courses for participating in the study.”
[Information obscured so as not to embarrass the authors]

Just to be clear – this was not an educational research paper. It wasn’t talking about pedagogy or course development. It wasn’t a study about tertiary education. They were looking at a general social issue, using university students as their sample.

In selecting their sample, the researchers made the following decisions:

  • They drew their sample from the University where they work, or (at best) a university nearby.
  • They drew their sample from one laboratory on one campus.
  • They either constrained the age of their sample so that only students would be selected, or they defined their age range after they had seen the ages of the students.
  • They either constrained their sample to students or they worked within a reward structure that resulted in only students applying. No admin staff, no faculty staff, no visitors to the laboratory…
  • They either didn’t care about gender or they didn’t try to recruit for even numbers.

Why? Why would anybody constrain their sample so tightly? One laboratory – what is that about ? Why would anyone exclude staff or visitors to the university? As a researcher, why wouldn’t you step outside the university and recruit from the local town or city? Wouldn’t it make your study stronger? Any expansion of the sample would have made this paper more interesting.

I don’t like this lazy sampling. I don’t like it at all. It really disappoints me when I open a promising article, only to find that the sample was this small and constrained.

Putting aside my own personal disappointment, lazy sampling is poor research.

It is lazy

Research is about discipline. It is hard work. Solutions are elusive and clear understanding does not come easily. It can take years to accrete the necessary knowledge that leads to a new development, to the point where there is some new consensus. Every good paper can help researchers come to the point where they can make a statement with confidence. Bad papers, on the other hand, just muddy the waters.

These is no rigour in lazy sampling. If you take the lazy way out, you will get lazier. If you flex your research muscles, you will get fitter. Aim to get fitter, and improve the strength of your research results. A strong sample won’t always produce a strong result, but a weak sample will always weaken your results.

Don’t take the easy way out. It is bad for your soul, and it is bad for research generally.

It isn’t sampling

This isn’t how sampling works. In most disciplines, you define your problem space, then you create a hypothesis, then you select a methodology to disprove the hypothesis, then you work out your population, and then you select from within that population. You aim for reproducability, generalisability and rigour.

In many lazy sampling studies, it seems like the whole study has been constructed to make the sampling as simple as possible. It seems to be a convenience sample taken to some sort of absurd extreme. Very, very, very convenient sampling, perhaps.

That isn’t sampling; it is crippling. A sample that’s small and unrepresentative makes it almost impossible to be able to say anything with integrity or confidence at the end of the study.

It is poor research

What is your research question? If your answer is something like “What are the attitudes of students in my university’s psychology department to X?” then I respectfully suggest that you are not trying hard enough. Don’t shape your research question to fit your sample – shape your sample to fit your research question.

If you constrain your sample to students, then what sort of diversity will you encounter? By definition, you are limiting yourself to a narrow educational range. Beyond that, how much diversity of ages, life experience, socio-economic status will you find in a sample of university students? Overall, the student population probably ranges across the full gamet of these social indicators. However, I’m willing to bet that the standard deviation is pretty narrow for any given university.

Will anyone be able to generalise from your results? Will they be useful to others? Universities are not paper factories. If you haven’t produced anything useful, don’t write a paper. If you end up with a study like this, think of it as a trial. Expand your horizons and wait until you do have something useful to say.

It is poor teaching

What example are you setting for these students? Through your example, they are learning that research should be conducted on whoever is handy and that rigour doesn’t matter. They are learning to take the easy way out. Is that really what you want to demonstrate to them?

I do understand that some of these studies start out as exercises for Honours or even undergraduate students, exercises that demonstrate the mechanics of research. If that is the case, don’t publish the paper. If you really want to show students how research works, construct a stronger teaching methodology – provide an environment where the mechanics of reserach are demonstrated by working with strong researchers. Let the students see them struggling with all the complexity that comes along with good research.

Take the pledge: No Lazy Sampling.

Strengthen your sample. Look outside your usual boundaries. You might be surprised by what you find.

About these ads

About Jonathan O'Donnell
Jonathan O'Donnell helps people get funding for their research. To be specific, he helps the people in the College of Design and Social Context at RMIT University in Melbourne, Australia. He loves his job. One day a week he does his own research into privacy, identity and transactions on the Internet. He likes that day, too, even when it makes his brain hurt.

8 Responses to Lazy sampling

  1. Jean says:

    This type of ‘lazy sampling’ is all too evident in my own discipline (management), where staff members (mainly psychologists) are supposedly doing research by sampling large numbers of first year undergrad students (17 and 18 year olds), many of whom have never worked a day in their life!!! But as long as the ‘sample size’ is sufficiently big enough to run some sort of statistical analysis on it, then it makes for easy publishing and makes them look like ‘stars’ due to their publication record

    • Jonathan O'Donnell says:

      Thanks, Jean. I know that some management researchers spend a great deal of time developing links with companies so that they can do qualitatitive interviews with senior managers.

      I’d never really thought about doing management research on non-managers.

      Jonathan

  2. guest says:

    That post sounds very jaded. Have you ever tried to recruit people for sociological or psychological studies? It’s hard! I have a friend whose (natural science) PhD is 3 years overdue because she can’t get her sample size large enough. (and she is certainly not lazy) I agree, you need to make sampling depended on your question but sampling from within the University isn’t lazy.

    • Jonathan O'Donnell says:

      I must admit, I haven’t had to recruit large numbers of people since very early on in my career, when I rang hundreds of accounants for a study. Even then, it wasn’t a sociological or psychologial study.

      I have taken part in a study like this, when I was at university. It was looking at how the brain recognised letters. A study like that, which is looking at brain functions, can take any human as part of a useful sample. Seems fair enough to sample from a university then.

  3. Guest2 says:

    I agree with the post on lazy sampling – it is bad practice and including a few caveats about it in the paper does not excuse this type of sampling. I would also consder “snowballing” sampling a type of lazy sampling; it is highy likely to produce a homogeneous sample similar to th one that you could get from one class, one university.

    • Jonathan O'Donnell says:

      I do tend to agree, particularly since ‘snowball’ sampling is often not used very well. Mostly, I just like the name. :-)

  4. Linda Brennan says:

    Hi I agree that the sample should match the information need and lazy sampling gets you more of the same. Same Same but [not] different as we say in Vietnam.

    However, in defense of the snowball, sometimes I need to know about a group of people, especially when I need an affinity group. I am not always looking for objectivity or multiple perspectives and sometimes I need to get a focused look at a small group of people. In these cases, a snowball technique is perfectly adapted to the method. It all depends on what you need to know. In the case of HIV transmission, a snowball technique is the only way to find out who is ‘talking’ to who.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 15,272 other followers

%d bloggers like this: