Sign Up for Fundamentals

Stay up-to-date with the latest research findings from the Institute for Basic Biomedical Sciences.

Please enter a valid email address.
Fundamentals Topics+

The Science of Scientific Images

More Advancements in Research

The Science of Scientific Images

Artwork by Tim Phelps, Associate Professor, Medical Illustrator, Dept. of Art as Applied to Medicine, copyright 2012 Johns Hopkins, created for Dr. James Black, Dept. of Surgery. 

The Science of Scientific Images

In communicating science, words do not always suffice.

By Melissa Hendricks

August 2012--A picture is worth a thousand words.

This aphorism may be particularly relevant in science, where pictures—illustrations, diagrams and other images—can illuminate the esoteric and the complex.

But how is this so? What can a picture accomplish that words cannot?

One key factor appears to be the limitation of our working memory.

“What vision buys you over language is the ability to see relationships all at once rather than unfolding over time,” says Steve Yantis, chair of Psychological and Brain Sciences and professor of neuroscience. When someone describes a process or series of events, the listener must hold in working memory the first concept described while listening to the next, and so on, building up a conceptual structure over time. The same goes for reading a sentence or paragraph. The reader must retain the ideas discussed in a first clause while reading a second. Alas, our working memory does not have infinite capacity; so we can take in only so much through reading or listening.

Another constraint is language’s imprecision. As an example, Yantis suggests imagining the desktop screen on a computer monitor. “In principle and given enough time, I could describe the pixels on the desktop or the spatial relationships of the different objects,” he says. “But the amount of verbal information required to reproduce a drawing of my desktop would be huge and take a really, really long time.” Even simple images can be saturated with information (in the lexicon of cognitive science, they have a large amount of “signal”) to a degree that words, spoken or written, cannot easily convey.

Great science illustrations can exploit this principle. Consider this award-winning illustration here of HIV commandeering an immune cell, which won first place in the 2010 International Science and Engineering Visualization Challenge sponsored by Science and the National Science Foundation.

To construct it, designers and graphics specialists at the company Visual Science consulted more than 100 scientific publications. The illustration incorporates an astounding amount of information: the precise three-dimensional position of 17 viral and cellular proteins in HIV, and eight different types of lipid molecules, shown in the precise proportions they are found in HIV (a total of 160,000 molecules). But in addition to visually representing data, the image—stunning and frightening—tells a story that transcends chemistry and biology. It’s as if disease has been splashed in your face. 

In response to the world’s extraordinarily rich palette of light and shapes and colors, organisms evolved mechanisms that can utilize this “data.” For humans, the visual cortex and eye are virtuosic at extracting information from a scene, object or picture. Language is a product of evolution, as well. But the visual system has been the subject of evolution much longer. The eye itself has been an evolutionary motif for at least 500 million years.

We’re so good at interpreting images, in fact, that even computers cannot yet duplicate our skill, says Professor of Neuroscience Ed Connor. “We can find, identify and evaluate the objects in a visual image extremely fast and extremely accurately,” he says. This is why “captchas”—distorted strings of letters and numbers—are used to distinguish human Internet users from malicious software.

In his own research involving animal studies and mathematical models, Connor has found that the high-level visual cortex contains neurons that represent configurations of object parts, including 3-D surface fragments and elongated shape features. For example, one neuron might respond to a spherical surface in the center of an object. Another might respond to an S-shape on the right, and another might respond to a C-shape on the left. Together, those neurons would participate in representing a teapot.

In a study published in Nature Neuroscience, Connor and postdoctoral fellow Yukako Yamane illustrated this principle by diagramming how neurons could represent a Henry Moore sculpture as a spatial construct of 3-D surface curvatures. A photograph of the sculpture graced the journal’s cover.

“This explicit, information-dense system for representing object structure is what makes our experience of 3-D shape so rich, accessible and vivid,” says Connor. “So vivid, in fact, that artists like Moore spend their lives creating objects to engage us visually, in ways that words cannot.”

But even though the human brain may have its visual biases, it’s only fair to say that there are times when language surpasses the visual in communicating information. “The modalities have their strengths and weaknesses,” says Brenda Rapp, chair of Cognitive Science who studies the cognition of spoken and written language. Although a picture might best relate what a person’s face looks like, words excel at communicating abstract ideas or complex thoughts. “With written words, you can make sure your reader thinks about things the way you want them to,” says Rapp. “Pictures can’t really control what your reader is thinking.”

It should also be noted that language processing and image processing are not entirely independent, adds Yantis. This point was underscored for Yantis himself many years ago when he was walking down a street in Ann Arbor.

“As I turned a corner,” he recalls, “I obliquely glanced at a poster on a storefront. And I thought I saw an American Indian chief wearing a headdress, standing and paddling a canoe.” Intrigued, Yantis turned back to get a closer look. “It turned out to be an advertisement for a musical group, The Chieftains, which showed a collection of musical instruments.” Why did his brain misinterpret the image? Yantis figures it was because the instruments had roughly the same contour as an Indian paddling a canoe. But the key factor, he says, was probably the word “chieftains”—which must have tripped his brain into thinking of things Native American.

“Your mind is always trying to make sense of the information it receives, to build up a single coherent impression,” says Yantis.

If images have such sway over our thinking, it’s also true that some have more power than others. As any scientist who has spent time choosing slides for a presentation or publication knows, only select pictures are truly worth a thousand words.

Tim Phelps, assistant director of Art as Applied to Medicine, employs another aphorism to address this point: “Often, less is more,” he says. Many of the most informative illustrations leave out a lot. When Phelps illustrated the repair of an abdominal aortic aneurysm in a pair of drawings, he showed only the most relevant features of the complicated surgery: the bulging artery in the pre-operation rendering and the dacron graft repair in the post-op image. “I left out the musculature, liver, heart, lung and most other organs,” he says, details that would have detracted from the central story of the procedure.

A scientific and medical illustrator “is not a camera,” says Phelps. “We’re better than a camera.” He says those in his profession serve as “visual editors” and “storytellers.”

There’s one more thing, says Phelps: Medical artists strive to create pictures that are pleasing. Illustrations with aesthetic appeal are more likely to captivate viewers.

DaVinci drawing of Vitruvian Man Vitruvian Man by Leonardo da Vinci, Galleria dell' Accademia, Venice (1485-90); Photograph of original work captured by Luc Viatour.

Indeed, in a discussion of their educational value, it would be easy to overlook the fact that some scientific renderings are exquisite. Leonardo da Vinci’s “Vitruvian Man” is just one example. The work is intended to instruct, demonstrating the proportionality of the human form. But it is also extraordinarily beautiful.

Are such works art? That’s a topic for another article, although it bears noting that “Vitruvian Man” and many of da Vinci’s drawings of human anatomy and mechanical inventions hang in museums.

What can be said is that a lot of us feel at home in the realm of pictures. If you do, you’re in good company. Einstein famously thought out his ideas in pictures: The equation E=mc2, for instance, emerged from a daydream about racing a beam of light.

And as writer and artist Johann Wolfgang von Goethe noted, “People should talk less and draw more.”