1
How Science Works
Such subjects of thought furnish not sufficient employment in solitude, but require the company and conversation of our fellow-creatures, to render them a proper exercise for the mind.
David Hume, ‘Of Essay-Writing’ (1777)
Science is a social construct.
Before that statement makes you toss the book across the room, let me explain what I mean. I don’t mean it in the sense used by extreme relativists, post-modernists, anti-science crusaders, and others who suggest that there’s no real world out there, that science is only one not-particularly-special way of knowing about it, or even that science is just one ‘myth’ among many that we could choose to believe.1 Science has cured diseases, mapped the brain, forecasted the climate, and split the atom; it’s the best method we have of figuring out how the universe works and of bending it to our will. It is, in other words, our best way of moving towards the truth. Of course, we might never fully get there – a glance at history shows how hubristic it is to claim any facts as absolute or unchanging. For ratcheting our way towards better knowledge about the world, though, the methods of science are as good as it gets.
But we can’t make progress with those methods alone. It’s not enough to make a solitary observation in your lab; you must also convince other scientists that you’ve discovered something real. This is where the social part comes in. Philosophers have long discussed how important it is for scientists to show their fellow researchers how they came to their conclusions. John Stuart Mill puts it this way:
In natural philosophy, there is always some other explanation possible of the same facts; some geocentric theory instead of heliocentric, some phlogiston instead of oxygen; and it has to be shown why that other theory cannot be the true one: and until this is shown, and until we know how it is shown, we do not understand the grounds of our opinion.2
And so, scientists work together in teams, travel the world to give lectures and conference speeches, debate each other in seminars, form scientific societies to share research and, perhaps most importantly, publish their results in peer-reviewed journals. These social aspects aren’t just a perk of the job, nor mere camaraderie. They’re the process of science in action: an ongoing march of collective scrutiny, questioning, revision, refinement and consensus. Although it might sound paradoxical at first, the subjective process of science is what provides it with its unmatched degree of objectivity.3
It’s in this sense, then, that science is a social construct. Any claim about the world can only be described as scientific knowledge after it’s been through this communal process, which is designed to sieve out errors and faults and allow other scientists to say whether they judge a new finding to be reliable, robust and important. That each discovery has to run such a gauntlet imbues the eventual products of the scientific process – the published, peer-reviewed studies – with a great deal of power in society. This is no mere cant, rhetoric, or opinion, we say: this is science.
Science’s social nature does come with weaknesses, however. Because scientists focus so much on trying to persuade their peers, which is the way they get those studies through peer review and onward to publication, it’s all too easy for them to disregard the real object of science: getting us closer to the truth. And because scientists are human beings, the ways that they try to persuade each other aren’t always fully rational or objective.4 If we don’t take great care, our scientific process can become permeated by very human flaws.
This book is about how we haven’t taken enough care of our precious scientific process. It’s about how we ended up with a scientific system that doesn’t just overlook our human foibles, but amplifies them. In recent years, it’s become increasingly, painfully obvious that peer review is far from the guarantee of accuracy and reliability it’s cracked up to be, while the system of publication that’s supposed to be a crucial strength of science has become its Achilles’ heel.
To understand how the scientific publication system has gone so wrong, though, we first need to know how it’s supposed to work when it goes right.
* * *
Let’s imagine you want to do some science. The first step is to read the scientific literature. This consists of a vast library of journals, the specialist magazines that are the main outlets for new scientific knowledge. The idea of a periodical where scientists could share their work dates back to 1665, when Henry Oldenburg of the UK’s Royal Society published the first issue of, to give it its full title, Philosophical Transactions: Giving Some Accompt of the Present Undertakings, Studies, and Labours of the Ingenious in Many Considerable Parts of the World.5 The intention was that those ingenious scientists could send in letters describing their exploits, for the perusal of other interested readers. Before that, scientists either laboured alone in the courts of wealthy rulers or for private patrons or guilds (where their science was often seen as more akin to a parlour trick than an effort to discover the truth), published standalone books, or formed letter-writing circles with like-minded peers. Indeed, this latter kind of correspondence club is where institutions like the Royal Society originated.6
The initial issues of Oldenburg’s journal were more like a newsletter, with descriptions of recent experiments and discoveries. For example, Volume 1, Issue 1 described the first ever observation of what was probably the Great Red Spot of Jupiter, by the natural philosopher and polymath Robert Hooke. The entire entry read:
The Ingenious Mr. Hook did, some months since, intimate to a friend of his, that he had, with an excellent twelve foot Telescope, observed, some days before, he than spoke of it, (videl. on the ninth of May, 1664, about 9 of the clock at night) a small Spot in the biggest of the 3 obscurer Belts of Jupiter, and that, observing it from time to time, he found, that within 2 hours after, the said Spot had moved from East to West, about half the length of the Diameter of Jupiter.7
The journal still exists to this day, with the somewhat easier-to-remember title of Philosophical Transactions of the Royal Society.8 As time went on, the brief news items were replaced with longer articles containing detailed descriptions of experiments and studies. It’s now part of a global ecosystem of over 30,000 journals, ranging from the very general (like the highly prestigious journals Nature and Science, which aim to publish the world’s most noteworthy research from any scientific field) to the very specific (like the American Journal of Potato Research, which is only interested in papers about one tuberous topic in particular).9 Some journals, like Philosophical Transactions, are still run by scientific societies, but most are owned by commercial outfits such as Elsevier, Wiley and Springer Nature.10 A recent advancement is that scientific journals are all online, allowing anyone who can afford to pay the publisher’s subscription fees – or have their university library do so on their behalf – to have the world’s scientific knowledge at their fingertips.11
After reading the journals relevant to your field, you might alight on a research question. Maybe there’s a scientific theory that makes a prediction – an hypothesis – that you can test in some clever way; maybe there’s a gap in our existing knowledge that you know just how to plug; maybe you’ve had a spark of inspiration and have come up with an experiment that tests something entirely new. Before you can do any of this, though, you’ll normally need some money to fund the study: for instance, to buy new equipment or materials, to recruit participants, or to pay the salaries of the scientists you’ll hire to do the legwork. Unless you happen to be, say, a pharmaceutical company that can afford to run its own laboratories, the main way to get that all-important funding is to apply for a grant. This might come from your government, a business, an endowment fund, a non-profit, a charity, or even a wealthy individual. You might apply to the National Institutes of Health or the National Science Foundation (both of which are taxpayer-funded agencies in the United States), or to a science-funding charity like the Wellcome Trust or the Bill & Melinda Gates Foundation.12
Funding is by no means assured, and any scientist will tell you that one of the most gruelling parts of the job is trying to get their latest research ideas funded, with failure grindingly common. This grasping for cash has important knock-on effects on the science itself, and we’ll return to them later in the book. But for now, let’s imagine you’re successful in securing a grant. You can then get to work. Collecting the data might involve smashing particles together in an underground collider, finding fossils in the rocks of the Canadian Arctic, setting up the precise environment for bacterial growth in a petri dish, organising hundreds of people to come to a lab and fill in questionnaires, or running a complex computer model; it can take days, months, decades.
Once the data are in, you’ll normally have a set of numbers that you, or a more mathematically minded colleague, can analyse using some variety of statistics (another minefield to which we’ll return). Then you need to write it all up in the form of a scientific paper. The typical paper starts with an Introduction, where you summarise what’s known on the topic and what your study adds. There follows a Method section, where you describe exactly what you did – in enough detail so that anyone could, in theory, run exactly the same experiment again. You’ll then move on to a Results section, where you present the numbers, tables, graphs and statistical analyses that document your findings, and you’ll end with a Discussion section where you speculate wildly – er, I mean, provide thoughtful, informed consideration – about what it all means. You’ll top the whole thing with an Abstract: a brief statement, usually of around 150 words, that summarises the whole study and its results. The Abstract is always available for anyone to read, even if the full paper is behind the journal’s subscription paywall, so you’ll want to use it to make your results sound compelling. Papers come in all lengths and sizes, and sometimes mix up the above order, but in general your paper will end up along these lines.13
When the paper is ready, you enter the world of scientific journals, and the competition for publication. Until recently, submitting a paper to a journal meant printing out several hard copies and mailing them to the editor, but nowadays everything is handled online – though many journals still use such archaic, buggy websites that you might as well send your paper by carrier pigeon. The journal’s editor, often a senior academic, will read the paper (or, let’s be honest, probably just the Abstract) and decide whether it might be worth publishing. Most journals, especially the highly prestigious ones, pride themselves on their exclusivity and thus their low acceptance rate (Science, for example, accepts less than 7 per cent of submissions), so the majority of papers will be bounced back to the authors at this point, in what’s called a ‘desk rejection’.14 This is the initial step in quality control: a sorting by the editor of the papers into those that match the theme of the journal and have potential in terms of their scientific interest or quality, and those that aren’t worth a second look. For the fraction of articles that do take the editor’s fancy, now comes the moment of peer review. The editor will find two or three scientists who are experts in your field of research and ask them whether they’d like to evaluate your manuscript. They’ll probably decline because they’re too busy, so the editor will keep going down the list of possible reviewers until a few agree. And so begins the nail-biting wait to see if your work will receive their endorsement.
Most people, including scientists, assume peer review has always been a crucial feature of scientific publication, but its history is more complicated. Although in the seventeenth century the Royal Society tended to ask some of its members whether they thought a paper was interesting enough to publish in Philosophical Transactions, requiring them to provide a written evaluation of each study wasn’t tried until at least 1831.15 Even then, the formal peer review system we know today didn’t become universal until well into the twentieth century (as you can tell from a letter Albert Einstein sent in 1936 to the editors of Physical Review, huffily announcing that he was withdrawing his paper from consideration at their journal because they had dared to send it to another physicist for comment).16 It took until the 1970s for all journals to adopt the modern model of sending out submissions to independent experts for peer review, giving them the gatekeeping role they have today.17
Peer reviewers are usually anonymous, which is both a blessing and a curse: a blessing because it allows them to speak their minds without concern about repercussions from the scientists whose work they’re criticising (a junior scientist can be truly honest about the flaws of a big-name professor’s work), but a curse because, well, it allows them to speak their minds without concern about repercussions from the scientists whose work they’re criticising. The following are genuine excerpts from peer reviews:
• ‘Some papers are a pleasure to read. This is not one of them.’
• ‘The results are as weak as a wet noodle.’
• ‘The manuscript makes three claims: The first we’ve known for years, the second for decades, the third for centuries.’
• ‘I am afraid this manuscript may contribute not so much towards the field’s advancement as much as towards its eventual demise.’
• ‘Did you have a seizure while writing this sentence? Because I feel like I had one while reading it.’18
If the reviewers’ evaluations look like this, the editor will probably reject your paper. At that point you might want to give up, or start the whole process again by submitting to a different journal, and if that fails a different one, and if that fails a different one, and so on – it’s not uncommon for papers to go through half a dozen or more journals, usually of ever-lower prestige, before they get accepted for publication. If, on the other hand, the reviewers are more impressed, you might get the opportunity to revise your paper to respond to their critiques – perhaps running new analyses or new experiments, or rewriting certain sections – and submit it to the editor again. The back-and-forth revising process can go through multiple rounds, and often takes months. Eventually, if the reviewers are satisfied, the editor gives the go-ahead and the paper is published. If the journal still prints hard copies, you’ll get to see your precious work in print; otherwise, you’ll have to settle for the thrill of seeing it on the journal’s official website. That’s it. You’ve made your mark on the scientific literature, and you have a publication that you can add to your CV and that can be cited by other researchers. Congratulations – take the rest of the day off.
The above summary is all too brief and general, but essentially every scientific field follows that process in some form. We might ask ourselves whether, after being put through the mangle of peer review, the eventual publication still provides a faithful representation of what was done in the study. We’ll get to that in later chapters. For now, we need to consider something else. What ensures that the participants in the process just described – the researcher who submits the paper, the editor at the journal, the peers who review it – all conduct themselves with the honesty and integrity that trustworthy science requires? There’s no law requiring that everyone acts fairly and rationally when evaluating science, so what’s needed is a shared ethos, a set of values that aligns the scientists’ behaviour.19 The best-known attempt to write down these unwritten rules is that of the sociologist Robert Merton.
In 1942, Merton set out four scientific values, now known as the ‘Mertonian Norms’. None of them have snappy names, but all of them are good aspirations for scientists. First, universalism: scientific knowledge is scientific knowledge, no matter who comes up with it – so long as their methods for finding that knowledge are sound. The race, sex, age, gender, sexuality, income, social background, nationality, popularity, or any other status of a scientist should have no bearing on how their factual claims are assessed. You also can’t judge someone’s research based on what a pleasant or unpleasant person they are – which should come as a relief for some of my more disagreeable colleagues. Second, and relatedly, disinterestedness: scientists aren’t in it for the money, for political or ideological reasons, or to enhance their own ego or reputation (or the reputation of their university, country, or anything else). They’re in it to advance our understanding of the universe by discovering things and making things – full stop.20 As Charles Darwin once wrote, a scientist ‘ought to have no wishes, no affections, – a mere heart of stone.’21
The next two norms remind us of the social nature of science. The third is communality: scientists should share knowledge with each other.22 This principle underlies the whole idea of publishing your results in a journal for others to see – we’re all in this together; we have to know the details of other scientists’ work so that we can assess and build on it.23 Lastly, there’s organised scepticism: nothing is sacred, and a scientific claim should never be accepted at face value. We should suspend judgement on any given finding until we’ve properly checked all the data and methodology. The most obvious embodiment of the norm of organised scepticism is peer review itself.
* * *
It looks good in theory: by following the four Mertonian Norms, we should end up with a scientific literature we can trust – the shoulders of giants, as in Newton’s famous phrase, on which we stand to see farther. Of course, those giants often had it wrong: just to take the two examples mentioned above by John Stuart Mill, we used to believe that the Sun orbited the Earth, and that flammable objects were full of a special element called phlogiston that was released when they burned.24 But we eventually consigned these theories to the scrapheap as better data came in. Indeed, it’s a virtue for a scientist to change their mind. The biologist Richard Dawkins recounts his experience of ‘a respected elder statesman of the Zoology Department at Oxford’ who for years had:
passionately believed, and taught, that the Golgi Apparatus (a microscopic feature of the interior of cells) was not real: an artefact, an illusion. Every Monday afternoon it was the custom for the whole department to listen to a research talk by a visiting lecturer. One Monday, the visitor was an American cell biologist who presented completely convincing evidence that the Golgi Apparatus was real. At the end of the lecture, the old man strode to the front of the hall, shook the American by the hand and said – with passion – “My dear fellow, I wish to thank you. I have been wrong these fifteen years.” We clapped our hands red … In practice, not all scientists would [say that]. But all scientists pay lip service to it as an ideal – unlike, say, politicians who would probably condemn it as flip-flopping. The memory of the incident I have described still brings a lump to my throat.25
This is what people mean when they talk about science being ‘self-correcting’. Eventually, even if it takes many years or decades, older, incorrect ideas are overturned by data (or sometimes, as was rather morbidly noted by the physicist Max Planck, by all their stubborn proponents dying and leaving science to the next generation).26 Again, that’s the theory. In practice, though, the publication system described earlier in this chapter sits awkwardly with the Mertonian Norms, in many ways obstructing the process of self-correction. The specifics of this contradiction – between the competition for grants and clamour for prestigious publications on the one hand, and the open, dispassionate, sceptical appraisal of science on the other – will become increasingly clear as we progress through the book.
For now, though, notice what it was that changed the mind of Dawkins’s elder statesman: ‘completely convincing evidence’. There’s little point in trying to correct and update our scientific theories with data if the data themselves aren’t convincing – or worse, aren’t even accurate. This brings us back to the idea we discussed in the Preface: for results to warrant our trust, they need to be replicable. As the philosopher of science Sir Karl Popper puts it:
Only when certain events recur in accordance with rules or regularities, as is the case with repeatable experiments, can our observations be tested – in principle – by anyone. We do not take even our own observations quite seriously, or accept them as scientific observations, until we have repeated and tested them. Only by such repetitions can we convince ourselves that we are not dealing with a mere isolated ‘coincidence’.27
It’s not as if this is a revolutionary idea – or one that was new to Popper, writing in the 1950s. If we return to the early days of Philosophical Transactions in the seventeenth century, we find the co-founder of the Royal Society, the chemist Robert Boyle, going to extraordinary lengths to ensure the replicability of his findings. He would repeatedly demonstrate his experiments, which used his famous air pump to show various properties of air and the vacuum to groups of observers, before having them sign sworn testimony that they’d witnessed the phenomena in question.28 He would ensure that his writings were detailed enough ‘that the person I addressed them to might, without mistake, and with as little trouble as possible, be able to repeat such unusual experiments.’29 And despite the great difficulty of building the complex apparatus, he encouraged and assisted other natural philosophers to replicate his air-pump experiments in different parts of Britain and Europe.30
Replication, then, has long been a key part of how science is supposed to work – and incidentally, it’s another of its social aspects, with results only being taken seriously after they’ve been corroborated by multiple observers. But somewhere along the way, between Boyle and modern academia, a great many scientists forgot about the importance of replication. In the collision of our Mertonian ideals with the realities of the scientific publication system – not to mention the realities of human nature – the ideals have proven the more fragile, leaving us with a scientific literature full of untrustworthy, unreliable, unreplicable studies that often do more to confuse than enlighten.
In the next chapter, we’ll see just how untrustworthy, unreliable and unreplicable the scientific literature has become.
Copyright © 2020 by Stuart Ritchie