Science: not the exclusive domain of the select few
When I first started working as a research student in a proper research facility, I had to make up some gel plates my first day on the job. These are small trays, a bit smaller than a CD case, that are filled with a jelly-like material. You put your substance of interest into little wells (holes) at one end, run an electric current through it, and see how quickly it travels through the gel, from which you can make some assumptions about it’s size and shape. Or something like that. It’s been a long time since I’ve done one; I gave up bench science years ago. I was completely rubbish at it. Very good at understanding and explaining the science, but I don’t think I ever did an experiment that worked the way it was supposed to. Ever.
Case in point: that day, as I heated up the stuff in the microwave so it was the right consistency to pour into the plate, I loosened the lid on the big jar it was in, stuck it in the microwave and turned it on — for a little bit too long, apparently. The pressure built up suddenly and the lid shot off and made a dent in the roof of the machine. Horrified and embarrassed that this was my first act as a “proper scientist, I cleared up the hot gelatinous mess as best I could and started over. Then I did exactly the same thing again. To make me feel better, other people in the lab thought this was very amusing and were quick to share their own embarrassing experiences from when they first started out.
Not long after, I was at a party and somebody asked what I did. I was a research scientist, I told them. Ooooooooh. I could feel the awe emanating from them, their entire attitude toward me shifting. I learned something at that moment: the pay may have been crap and the work frequently tedious, but people were really really impressed by “science.”
Scientific scandals notwithstanding, people still seem to hold scientists up as a special breed of very clever people who do something totally impenetrable to “normal” folks, which is a shame. And not true, for the most part. Certainly, some fields are a bit like that, but the areas I now work in (and the areas usually of interest to the readers of this blog), psychology and health sciences, don’t really require much of a technical vocabulary or an advanced degree to at least follow along.
But people are so overwhelmed by the alien environment of a scientific paper that they think “critical analysis skills” are something that other people do, but which they will never be capable of. I see this all the time with new graduate students. It’s not enough to learn and regurgitate the way you may have gotten away with as an undergraduate. As a postgraduate, you’re expected to think, analyse, and critique. I teach a writing skills course for postgraduate students, and almost to a man (and woman) they are terrified of producing their own take on what they are reading. How could they, a mere student, comment on the work of experienced, qualified people, who must know what they are doing? Not to mention that it’s a published paper, which presumably got past being peer reviewed by other intelligent trained people. If it’s published, it’s obviously good, right? Wrong.
And it’s often not as simple as good or bad. As Atchka and I have both written previously, sometimes the science is good, but the interpretations of it are biased by preconceived ideas about what the authors hoped to find. Or what would produce a more publishable paper. Or secure a nice bit of grant funding. It would be nice to think these things don’t happen among the supposed seekers of truth, but that would be naive. So it is never wise to take a paper at face value, to read it as you would a magazine. And it’s particularly unwise to make assumptions about the findings of a paper simply from its title or its abstract, especially in the field of “obesity science.”
I have worked in science for over 20 years, in one guise or another. In addition to my ill-fated career as a bench scientist, I have edited and written for medical journals, completed two undergraduate degrees, a postgraduate diploma and a masters, and am currently in my second year of a PhD. My most recent job was as a systematic reviewer for a public health intelligence unit. The government and the health service paid us large sums of money to analyse scientific evidence from around the world, evaluate it and write reports of our findings. I became particularly good at taking a fine toothcomb to published papers. And I learned, much to my surprise, that a significant proportion of peer-reviewed published papers are just not very good. But nothing — NOTHING — prepared me for an examination of the “obesity” literature. I try to explain this to people, but they can’t quite grasp it. They think I’m exaggerating. I don’t blame them. Sometimes it just beggars belief.
Which brings us back to critical analysis. Let me give you an example. Imagine you opened up the morning paper and saw this headline: “British men are the best lovers in the world.” Do you think, “Wow, really? Maybe I’ll up and move to the UK?” Or, if you’re already here, do you think, harumph (we in Britain like to harumph), “Obviously false”? Or do other questions pop into your mind? I’d like you to take a moment here, if you haven’t already, to ponder that headline. Think of at least three questions that might come to mind on reading that one sentence. Go on, I’ll wait. Really, this only works if you take part.
So, got three? Maybe more? What you just did, that was critical analysis. It’s something you do every day. It’s not mysterious and unattainable. Hey, guess what, you’re a scientist. You have a questioning mind and you seek the truth, seek to understand, and ask the right questions.
I have no time for people who think that science is and should remain the purview of the select few. Sure, I wouldn’t want my plumber to do my root canal — of course there is a need for training and specialisation. But when it comes to grasping the meaning of a summary of a person’s research, written predominantly in plain English (or whatever language you speak), any reasonably intelligent person who is prepared to ask the right questions can, and should, be able to do it.
What is more, I have learned that even those individuals who do have years of training and expertise are unable to do exactly this. Of course, you may not be able to understand all (or any) of the statistical techniques and analysis used. In fact, you probably won’t. But in this field, the bulk of the work should be written in a language that is understandable to most intelligent readers, or at least to those who don’t mind looking up the odd word or concept on the internet.
What’s more, it doesn’t matter that much. You will rarely see a really well-conducted study that is brought down by inappropriate use of statistical analysis. If the authors have taken the time and effort to do something really well, likely they will have either known how to do the correct analysis or will have gotten the help of somebody who has. And while the scientific literature is rife with papers whose stats are NOT appropriate and correct, the warning signs are generally there in the rest of the text. Likewise, a truly rubbish study will not be saved by good stats, although careful manipulation of the numbers will often be used to try and make a bad study look good, or at least so incomprehensible to other people that they think it MUST be good.
So do not be cowed by letters after a person’s name. Absolutely give them the respect they deserve for putting in the years, acknowledge their learning and their expertise. But realise, they are not gods. They are not infallible. And science without critical thought is unworthy of the name.
In my next blog post in the new year, I will actually take you through a scientific paper, show you what to look for, what questions to ask. If anybody has a particular paper they would like to suggest, please make a note in the comments. It would have to be one that is freely available or there would be copyright issues. But we aim to please around here, some of the time, so let me know what you need help with.