The Scientific Paper isn't Dead, Sorry

I ran across an ambitiously titled article in the Atlantic, "The Scientific Paper Is Obsolete (Here’s what’s next)". The title felt a bit like click-bait, but as an academic, that kind of bold pronouncement is hard to resist.

The TL;DR version of the paper is this: Static papers will be replaced by papers with interactive figures.

The article is worth reading, if only because he does a good job highlighting the work of some of the folks pushing hard on the vision and tools of interactive documents, from Bret Victor to Stephen Wolfram to Fernando Perez. He neatly summarizes the approaches to live interactive documents, giving some compelling examples. It's clear the technology is getting better, and those lucky enough to collaborate with folks like Victor can create compelling, one-off examples^[1].

But aside from that, there's not a lot new offered here, and certainly no insight as to "what's next." The article (and quoted experts) focus on technology, but ignore the practical obstacles to widespread adoption of interactive articles: serious issues like "how do authors avoid misleading readers?" and "how can simulations and interactive figures be reviewed and vetted?"

Let's step back and remember why we still publish papers. At the center of the publication process is providing the information needed to advance science; verifiable nuggets of information that have been reviewed, vetted and then shared. That's it. The author says that up front, when talking about how journals started.

The scientific paper—the actual form of it—was one of the enabling inventions of modernity. Before it was developed in the 1600s, results were communicated privately in letters, ephemerally in lectures, or all at once in books. There was no public forum for incremental advances. By making room for reports of single experiments or minor technical advances, journals made the chaos of science accretive. Scientists from that point forward became like the social insects: They made their progress steadily, as a buzzing mass.

There is an implication here, though. For something to be reviewed and vetted, it has to be possible for reasonable experts to review and vet it. Interactive content raises fundamental questions for the reviewing process. For example, how can reviewers know that they have explored all the possible configurations of a figure, to give their stamp of approval to what is being communicated?

Reviewing is time consuming, and hard work as it is. With interactive content, any reviewing process would now have to include vetting of the simulation code underlying the figures, if we are to trust the results. Will we expect reviewers to debug interactive papers now? Consider the infamous excel bug that led to a faulty economics paper that, in turn, was used to justify the disastrous austerity policies the have crippled some countries in recent decades (not to mention inspired Randian policies by politicians like Paul Ryan). What if every figure in every scientific paper might contain bugs leading to misleading results? Finding errors in printed text and figures is hard enough; finding it in hidden code underlying the visible results would be intractable. It's naive to assume errors like this will not happen.

Reviewing interactive documents is just the tip of the iceberg, and a minor problem compared with the effect of simulation on readers. The implicit argument in this article is that somehow interactivity is "better" and gives us more insight^[2].

But, simulations are simplifications, not reality. The illusion that they are a valid simplication makes them more insidious when it comes to convincing readers of something. Creating interactive content that gives insight, rather than misleading or confusing, is not a trivial task.

Bogost writes convincingly about procedural rhetoric and the need for procedural literacy; but, in a world where people aren't educated about basic facts, and can't tell the difference between a news site and one pushing propaganda and lies, interactive content would need to be created with great care and skill if it is not going to be misleading. The above excel bug would probably have been much more compelling if the figures had sliders to change the inputs to the simulations, or if they'd been illustrated with mesmerizing dynamic graphs^[3].

None of these questions feature prominently in the article. Rather, the author seems to believe that if academics just had the right incentives to create and share interactive comment, they would.

Until journals require scientists to submit notebooks, and until sharing your work and your data becomes the way to earn prestige, or funding, people will likely just keep doing what they’re doing.

I'm a big fan of interactive content in documents; the ability for anyone to create and share whatever they want is one of the things that draws me to the web. But there are serious roadblocks to widespread adoption of interactive content in scientific publications, and we'd be better off attacking those problems than assuming it's academic culture and incentives that are the problem.