Skip to content

Science: not the exclusive domain of the select few

December 23, 2013

Fat Science

When I first started working as a research student in a proper research facility, I had to make up some gel plates my first day on the job. These are small trays, a bit smaller than a CD case, Gel Tray 2that are filled with a jelly-like material. You put your substance of interest into little wells (holes) at one end, run an electric current through it, and see how quickly it travels through the gel, from which you can make some assumptions about it’s size and shape. Or something like that. It’s been a long time since I’ve done one; I gave up bench science years ago. I was completely rubbish at it. Very good at understanding and explaining the science, but I don’t think I ever did an experiment that worked the way it was supposed to. Ever.

Case in point: that day, as I heated up the stuff in the microwave so it was the right consistency to pour into the plate, I loosened the lid on the big jar it was in, stuck it in the microwave and turned it on — for a little bit too long, apparently. The pressure built up suddenly and the lid shot off and made a dent in the roof of the machine. Horrified and embarrassed that this was my first act as a “proper scientist, I cleared up the hot gelatinous mess as best I could and started over. Then I did exactly the same thing again. To make me feel better, other people in the lab thought this was very amusing and were quick to share their own embarrassing experiences from when they first started out.

Not long after, I was at a party and somebody asked what I did. I was a research scientist, I told them. Ooooooooh. I could feel the awe emanating from them, their entire attitude toward me shifting. I learned something at that moment: the pay may have been crap and the work frequently tedious, but people were really really impressed by “science.”

Scientific scandals notwithstanding, people still seem to hold scientists up as a special breed of very clever people who do something totally impenetrable to “normal” folks, which is a shame. And not true, for the most part. Certainly, some fields are a bit like that, but the areas I now work in (and the areas usually of interest to the readers of this blog), psychology and health sciences, don’t really require much of a technical vocabulary or an advanced degree to at least follow along.

But people are so overwhelmed by the alien environment of a scientific paper that they think “critical analysis skills” are something that other people do, but which they will never be capable of. I see this all the time with new graduate students. It’s not enough to learn and regurgitate the way you may have gotten away with as an undergraduate. As a postgraduate, you’re expected to think, analyse, and critique. I teach a writing skills course for postgraduate students, and almost to a man (and woman) they are terrified of producing their own take on what they are reading. How could they, a mere student, comment on the work of experienced, qualified people, who must know what they are doing? Not to mention that it’s a published paper, which presumably got past being peer reviewed by other intelligent trained people. If it’s published, it’s obviously good, right? Wrong.

And it’s often not as simple as good or bad. As Atchka and I have both written previously, sometimes the science is good, but the interpretations of it are biased by preconceived ideas about what the authors hoped to find. Or what would produce a more publishable paper. Or secure a nice bit of grant funding. It would be nice to think these things don’t happen among the supposed seekers of truth, but that would be naive. So it is never wise to take a paper at face value, to read it as you would a magazine. And it’s particularly unwise to make assumptions about the findings of a paper simply from its title or its abstract, especially in the field of “obesity science.”

I have worked in science for over 20 years, in one guise or another. In addition to my ill-fated career as a bench scientist, I have edited and written for medical journals, completed two undergraduate degrees, a postgraduate diploma and a masters, and am currently in my second year of a PhD. My most recent job was as a systematic reviewer for a public health intelligence unit. The government and the health service paid us large sums of money to analyse scientific evidence from around the world, evaluate it and write reports of our findings. I became particularly good at taking a fine toothcomb to published papers. And I learned, much to my surprise, that a significant proportion of peer-reviewed published papers are just not very good. But nothing — NOTHING — prepared me for an examination of the “obesity” literature. I try to explain this to people, but they can’t quite grasp it. They think I’m exaggerating. I don’t blame them. Sometimes it just beggars belief.

Which brings us back to critical analysis. Let me give you an example. Imagine you opened up the morning paper and saw this headline: “British men are the best lovers in the world.” Do you think, “Wow, really? Maybe I’ll up and move to the UK?” Or, if you’re already here, do you think, harumph (we in Britain like to harumph), “Obviously false”? Or do other questions pop into your mind? I’d like you to take a moment here, if you haven’t already, to ponder that headline. Think of at least three questions that might come to mind on reading that one sentence. Go on, I’ll wait. Really, this only works if you take part.

So, got three? Maybe more? What you just did, that was critical analysis. It’s something you do every day. It’s not mysterious and unattainable. Hey, guess what, you’re a scientist. You have a questioning mind and you seek the truth, seek to understand, and ask the right questions.

statistics_vadloI have no time for people who think that science is and should remain the purview of the select few. Sure, I wouldn’t want my plumber to do my root canal — of course there is a need for training and specialisation. But when it comes to grasping the meaning of a summary of a person’s research, written predominantly in plain English (or whatever language you speak), any reasonably intelligent person who is prepared to ask the right questions can, and should, be able to do it.

What is more, I have learned that even those individuals who do have years of training and expertise are unable to do exactly this. Of course, you may not be able to understand all (or any) of the statistical techniques and analysis used. In fact, you probably won’t. But in this field, the bulk of the work should be written in a language that is understandable to most intelligent readers, or at least to those who don’t mind looking up the odd word or concept on the internet.

What’s more, it doesn’t matter that much. You will rarely see a really well-conducted study that is brought down by inappropriate use of statistical analysis. If the authors have taken the time and effort to do something really well, likely they will have either known how to do the correct analysis or will have gotten the help of somebody who has. And while the scientific literature is rife with papers whose stats are NOT appropriate and correct, the warning signs are generally there in the rest of the text. Likewise, a truly rubbish study will not be saved by good stats, although careful manipulation of the numbers will often be used to try and make a bad study look good, or at least so incomprehensible to other people that they think it MUST be good.

So do not be cowed by letters after a person’s name. Absolutely give them the respect they deserve for putting in the years, acknowledge their learning and their expertise. But realise, they are not gods. They are not infallible. And science without critical thought is unworthy of the name.

In my next blog post in the new year, I will actually take you through a scientific paper, show you what to look for, what questions to ask. If anybody has a particular paper they would like to suggest, please make a note in the comments. It would have to be one that is freely available or there would be copyright issues. But we aim to please around here, some of the time, so let me know what you need help with.

Never Diet Again Sigs

Advertisements
7 Comments leave one →
  1. December 23, 2013 11:40 am

    Great post, Angie. I don’t have a background in biostatistics or research. But I heard lots of claims being made from both mainstream sources and from Fat Acceptance people, and i wasn’t satisfied to just accept what either side claimed. So, I began downloading studies and reading them, which hugely sucked at first. But the more I read, the more I realized what you said is true: research papers are largely written in language that moderately intelligent people can comprehend if they choose to. But the key is that you have to read the research. You can’t just read a synopsis or an article and call it a day because you are reading someone else’s interpretation. If you want to understand the science, you have to get your hands dirty and read the science! And you would be surprised how quickly you are able to comprehend the scientific process by simply observing how data is collected, arranged and analyzed by professionals. It’s amazing!

    I look forward to your future posts on reading research. This will be a great resource for people who want to flex their analytical skills. Thank you!

    Peace,
    Shannon

  2. December 23, 2013 7:04 pm

    I look forward to your next post : )

  3. Leila Haddad permalink
    December 24, 2013 3:23 am

    Great post! So needed!

  4. Mich permalink
    December 25, 2013 2:32 pm

    Fantastic.

  5. Judybat25 permalink
    January 24, 2014 5:50 am

    It’s probably way too late, but I’d love to talk about this one:

    http://www.nejm.org/doi/full/10.1056/NEJMoa0708681#t=article

    It’s from the New England Journal of Medicine, entitled “Weight Loss with a Low-Carbohydrate, Mediterranean, or Low-Fat Diet”. It’s relatively old (2008) but it’s more than one diet and it’s full text.

    I have many questions about this study.

    • January 24, 2014 2:14 pm

      Hi JudyBat. I’m glad you liked this post and the next one, and for suggesting a study. Since this is exactly the methodology I talk about in my subsequent post, why not have a go at working through it using that blog as a guide. If you have any questions or problems with it, drop me an email at info at neverdietagain dot co dot uk, and I’ll try and help out. Ang 🙂

      • January 24, 2014 11:22 pm

        A agree with the consensus. More this.

        Peace,
        Shannon

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: