Bringing Data to Life 4: Vision

"Mine eyes are made the fools o’ the other senses, or else worth all the rest."

Macbeth, Act II, Scene I by William Shakespeare

The second act of Macbeth opens with one of the most studied soliloquies in all of Shakespeare. It begins with Macbeth seeing a vision of a dagger and questioning whether it’s real or a hallucination. Either his eyesight is in conflict with his other senses—is being made a fool of—or else it’s worth the rest of his senses put together, and he should trust what he sees.

It turns out he does trust what he sees. He goes on to murder Duncan in his bed.

In fact, vision is supreme; it trumps all other senses for learning. It is our most dominant sense, taking up nearly half of our brains’ resources. Here we’ll learn how to use that to our advantage as data communicators.

We See with Our Brains

The internal mechanics of seeing seem easy to understand. Light enters the eye, is focused by the lens, and reaches the retina. The neurons in the retina respond by sending impulses deep into the brain via the optic nerve. The brain interprets these impulses, and we become visually aware. The entire process feels effortless, trustworthy and capable of providing a true representation of the world.

And nothing in the last sentence is true.

The process is extremely complex, seldom provides a completely accurate representation of the world, and is not one hundred percent trustworthy.

We Only See What Our Brains Want Us to See

Right now, your brain is actively deconstructing the information it’s getting from your eyes, filtering it, and reconstructing what it thinks it sees. It is interpolating, calculating and giving you its best guess. Why? Because it’s forced to solve a complex problem in an instant: We live in a three-dimensional world, but your brain receives two-dimensional information. Even worse, this information comes from two separate visual fields, and they project their images upside-down and backwards.

We don’t see with our eyes. We see with our brains. Our brains dedicate nearly one-half of their thinking resources to this experience.

This is why vision is supreme. It trumps all our other senses for learning. The more visual a presentation is, the more likely it is to be recognized and recalled. Much of this comes about because of the way seeing works. So, let’s quickly review what happens in the visual system when we see something.

The Visual Pathway, Pre-attentive and Attentive Processing

If I lived in the bloody world of Macbeth and cut your head off, I might flip it upside down to see this inside:

(From Visual Thinking for Design by Colin Ware.)

This diagram highlights the visual pathway through the brain. Information processing takes place at each stage along the way. The entire process from detection to interpretation is called visual perception. Receptors and neurons early in the pathway can detect a limited set of simple visual attributes. We perceive these attributes without conscious effort, which is called pre-attentive processing. Our brains process more complex attributes and relationships among attributes and their semantic meaning later. It requires conscious attention and effort, such as directing and focusing our eyes. This is attentive processing.

So, rather than a simple camera, the visual system acts more like a complex digital camera with a computer that modifies, filters and compresses the scene before presenting you with a complete and polished image. Along the way, it makes choices that you have no direct control over. In other words, we experience our environment as a fully analyzed opinion about what our brain thinks is really there.

We can use this knowledge to our advantage as data communicators. The task of looking for something by sight is called a visual query. We can leverage pre-attentive processing to enable extremely rapid visual queries. Length, width, and position in two-dimensional space are pre-attentive attributes that convey quantitative information. Size, shape, colour intensity and colour hue are pre-attentive attributes that convey qualitative or categorical information.

A Picture Is Worth a Thousand Words

We can quibble about the actual worth of a picture in word count. But psychologists have known for a long time that information presented with pictures is way more memorable than when presented with words alone. Three days from now, you might recall ten percent of the information I’m presenting only with words. But where I add pictures, that goes up to about 65 percent.

Why are written words so inefficient? One reason is the way text is perceived, processed and understood. Through the visual pathways I described earlier, our brains first see words as lots of tiny pictures—think of these as letters. These tiny pictures are then perceived as groups of tiny pictures—the words. If you’re reading an unfamiliar alphabet—for some of you, that might be Cyrillic, Chinese or Korean—that’s where the process stops. If you recognize the alphabet, the next step is to perceive these picture groups as word shapes, something with meaning. Finally, the words must be interpreted as language, strung together into grammatical sequences, and understood.

That’s a lot of mental processing. It should be no surprise that a written thought can take several seconds to comprehend. But thanks to pre-attentive processing, a simple picture takes only one-tenth of a second.

Gestalt Principles of Perception

"The whole is other than the sum of its parts."

—Kurt Koffka, German psychologist

There are ways to make pictures even more efficient. Gestalt psychology is a school of thought that emerged in Austria and Germany in the early twentieth century. The German word gestalt [ɡ???talt] is taken here to mean pattern or configuration. A central feature of gestalt psychology is that we perceive entire patterns or configurations, not simple components. Much of the Gestaltists’ work dealt with visual perception—seeing—and it left us with many concepts useful in presenting information visually.

I’m about to show you some of the Gestalt Principles of Perception that came from this work. Think of them as visual heuristics, shortcuts that your brain takes to quickly make sense of the visual world. They aren’t just theoretical abstractions. They’re used throughout the world of graphic design. The principles that follow are most useful to the design of data stories.

Proximity

We perceive objects that are close to each other as belonging to a group.

Use position and space deliberately to group some objects and separate others. It’s a powerful way to direct your audience to view certain things together without the use of borders. Your audience will immediately perceive your intended organization and structure when you put things that belong together close together.

Symmetry

We tend to group together objects that are similar in colour, shape, or orientation.

We perceive this grouping regardless of proximity. Similarity can tie elements together that might not be right next to each other in your design. You can also exploit similarity to make elements in your design stand out; simply make them dissimilar to most other elements.

Enclosure

We perceive objects as belonging together when they are enclosed by anything that forms a visual boundary around them.

The enclosure causes the objects to appear as set apart in a region distinct from the rest of what we see. You can easily accomplish enclosure with thin border lines or lightly coloured backgrounds.

Closure

When we are faced with objects that can be perceived as incomplete versus whole, we will perceive them as a whole.

When you look at this image, you are most likely to see a zebra even though the image is just a collection of black shapes on a white background. Your brain automatically fills in the missing information to create a recognizable image based on your past experience.

Continuity

We perceive objects as belonging together if they are aligned with one another or appear to form a continuation of one another.

In this example, you’re likely to perceive the red dots in the curved line as being more related to the black dots in the curved line rather than the red dots in the straight horizontal line. Your visual system naturally follows a line or a curve. This example also shows that continuity is a stronger signal of relatedness than colour.

Connection

We perceive objects that are connected as part of the same group.

It’s almost impossible not to see this figure as representing two different data series. The lines create a clear sense of connection between the data points and bring to light the overall shape of the data and its underlying trends. The perceptual strength of connection makes lines useful in connecting items in a data series and distinguishing them from other data series in the same chart.

Pr?gnanz (Symmetry and Order)

We perceive ambiguous shapes as simply as possible.

The German word pr?gnanz [p??ɡnɑnz] can be translated as "good figure." The principle of pr?gnanz is an embodiment of your brain’s laziness and preference for simplicity. It will interpret the image on the left as a rectangle, circle and triangle even when the outlines of each are hidden or incomplete. Those shapes are simpler than the overall image.

We perceive symmetrical compositions as simpler and more stable than asymmetrical compositions, but we also perceive them to be less interesting. As a designer, favour symmetrical compositions when your priority is efficiency of use and asymmetrical compositions when your priority is interestingness.

Perceiving Colour

"Colour! What a deep and mysterious language, the language of dreams."

Paul Gauguin, French painter

We all like to see some colour in our presentations. It can add much-needed life and drama to technical content. But its effective use in data visualization isn’t widely understood. Here I’ll present some background and rules of thumb.

How We See Colour

Our retinas have two basic kinds of light receptors. Rods are specialized for low light levels. They’re pretty much wasted on us modern urban dwellers because our nights aren’t usually dark enough to need them. And they have little impact on colour vision. Cones are the basis for daytime colour vision. There are three types of cones: those sensitive to short wavelengths of light, those sensitive to middle wavelengths and those sensitive to long wavelengths. These correspond roughly to the visible spectrum’s blue, green, and red areas.

There are far fewer blue-sensitive cones than the other types, and they're less sensitive to light. This chart shows the relative sensitivity of the three types of cones across the visible spectrum.

(From Visual Thinking for Design by Colin Ware.)

You can see the poor sensitivity of the blue-sensitive cones. Also, notice the large overlap in the sensitivities of the red and green-sensitive cones. Our brains use the difference between these two signals to get useful hue information. These two facts have important consequences for our use of colour in visualizations.

Because we have few and weak blue-sensitive cones, it’s a mistake to show text or any other detailed visualization using blue on a black background. The result is usually illegible.

Pure yellow exhibits the opposite problem. We see it as the brightest hue because it excites both the green- and red-sensitive cones. So, it’s a mistake to show text or detailed information in yellow on a white background. However, pure yellow is very distinct on a black background.

The colour information from the cones undergoes a major transformation along the visual pathway. Neural networks transform the signals into three colour-opponent channels: red-green, yellow-blue and luminance. They are called colour-opponent because they encode the differences between opposites rather than absolute values. That is, they pick up on contrast, not brightness. The red-green channel represents the difference between the signals from the green- and red-sensitive cones. The luminance channel represents the sum of the green- and red-sensitive cones. The yellow-blue channel represents the difference between the luminance channel and the blue-sensitive cone signals.

Three Quantitative Dimensions of Colour

We often use colour to convey quantitative information. When we use it to convey quantity or ordinality, three dimensions of colour readily convey intensity or position.

One of these is hue and is familiar to us as describing the colours of the rainbow or light refracted through a prism.

We sometimes refer to the rainbow arrangement of hues as colour temperature. We perceive blue as the coolest, green in the middle, and red as the hottest.

We often use hues to represent categorical data. Traditionally we code the lowest data categories in blue and the highest in red. Although these hues do, in fact, correspond to physical quantities of light, we don’t naturally think of them that way. We don’t automatically see hues as quantitatively different. We have to learn this association, and visualizations that rely on it should always include a legend linking colours to data ranges.

The second dimension is saturation, ranging from pure white to full saturation (the pure colour).

The third dimension is luminance (or brightness) which ranges from pure black to full luminance (again, the pure colour).

When you use these schemas, choose only a single hue. Traditionally we code the lowest values as pure black or white and the highest as pure colour. We naturally perceive differences in saturation or luminance as quantitative.

Colour For Qualitative Properties

We face two challenges when using colour to indicate a nominal or qualitative property. First, we must avoid an implied ordinality by varying only saturation or luminance. We don’t want differences in these to imply "more" or "less" of the property. Second, we need a palette sufficiently diverse that neighbouring colours are easily distinguished. Our perception of a colour is influenced by the colours surrounding it, which can confuse us and mask small differences in the data.

Fortunately, several palettes have been designed to overcome these problems. You can find examples in Tableau’s software, and Stephen Few’s book Show Me the Numbers. Alternatively, you can generate a palette using the interactive Color Brewer tool at https://colorbrewer2.org. Because colours have no inherent meaning, you should include a legend that links the colours to the data values.

Accessibility Challenges with Colour

One of our biggest challenges when using colour to convey information is the variability in human colour perception. Some 10% of the population – most male – have some form of colour vision deficiency (CVD), commonly known as "colour blindness." Furthermore, our ability to distinguish small variations in colour declines with age and can also be affected by disease, medication and diet. It also gets worse as the viewing environment gets darker.

Colour vision deficiency most commonly occurs in the red-green colour channel. You can mitigate this by being especially careful with red and green hues. They should not appear together in the same visualization. If you must, you should also use a redundant way of encoding the values in your data. For example, leverage saturation to choose light versus dark hues. Or use different shapes or shadings (i.e., hash-patterns) in addition to colour. See 5 Tips on Designing Colorblind-Friendly Visualizations on Tableau’s website for more tips. Tableau’s software and the Color Brewer tool also have options for generating CVD-friendly palettes.

Different display devices, whether smartphones, laptop screens, projectors or printers, render the same colours differently. The solution to this problem is like that for CVD: Use well-differentiated palettes and do not rely on subtle colour distinctions. If a hard copy is your goal, Color Brewer has options for printer-friendly and photocopier-friendly palettes.

Also, be aware that colours sometimes carry cultural or psychological meanings that can interfere with or enhance your audience’s engagement with your data story. Often a colour has more than one meaning depending on its context. We associate red, the colour of blood, with danger or warning signs. In some Asian cultures, though, red symbolizes prosperity; this is especially salient during the Lunar New Year festivities. Green indicates safety and permission to move forward, but it can also indicate concern for the environment. Strong, bright colours demand attention and are useful for highlighting important information. The muted colours often found in nature are considered soothing and easy on our eyes; they are useful for styling contextual content that you don’t wish to stand out.

Finally, it’s important to be aware that we don’t see colours in an absolute sense. We perceive differences in colour and luminance relative to one another. The exact same colour, described by hue, saturation and luminance, can appear quite different depending on the colours around it. Here is an example of the local contrast effect.

The small grey squares in this image are all exactly the same, but they look quite different because of their different contrasts with the background gradient.

You should represent topography on maps exclusively with differences in luminance. Luminance is the only colour channel that can process shape-from-shading information. Choropleth maps have their own colour challenges. Because our perception of a colour is affected by its context, having light colours next to dark colours can lead to false conclusions. Or suppose the regions being mapped differ greatly in size, such as the Census Divisions in Ontario. In that case, larger areas will take on more visual prominence and appear more important, even if the stronger effects are in smaller areas.

Consistency

Above all, be consistent. If your story relies on multiple visualizations of the same data series, for example, a table and a bar chart, it’s important to use the same colour scheme for each. Your audience will find it easy to relate the visualizations and tie them together in your data story. Remember to apply the Gestalt principle of similarity.

These are some of the properties of colour that you need to consider for visual clarity. There is, of course, a lot more to colour than this; colour design is subtle and can be a source of much beauty and pleasure. My goal here has been more modest: to find the grounds for effective data presentation in the wiring of the human brain.

Perceiving the Third Dimension

"Effective is not the same as beautiful."

Naomi Robbins, data visualization consultant

You should never use 3D unless the added depth dimension represents actual data. Otherwise, it adds meaningless visual content that your audience must take in and process. It adds cognitive load. It wastes their time, and even at best, it doesn’t help their understanding. At worst, optical illusions introduced by the third dimension can impair understanding. Consider this 3D pie chart. Which has the greater market share: Company A or Company G? Which has the lower share: Company B or Company H?

The optical illusions found in 3D graphics can even be used to deliberately misrepresent trends in the data. Here, the devious use of perspective hides a rising trend.

Even when the third dimension does represent data, 3D representations on a 2D surface have several problems which argue against their use. First and foremost, our perception of depth is weaker than our perception of width and height. Colin Ware suggests that we only have "2.5D vision." Depth perception is not pre-attentive and so requires some conscious effort to process.

Second, there are problems inherent in the projection of three dimensions down to two. The resulting display tends to be cluttered, and at least one dimension cannot be represented accurately. A common outcome is occlusion, in which foreground objects obscure background objects. Consider "South" in the figure below. What are its Q1 sales?

Interactivity can mitigate this if your user interface allows the viewer to manipulate and rotate the image in three dimensions. If the third dimension represents categorical data, using 2D small multiples is a better choice of visualization. This story would best be told with four 2D bar charts side by side, each showing one of the four regions.

Summing Up Seeing

"Can such things be, and overcome us like a summer’s cloud, without our special wonder?"

Macbeth, Act III, Scene IV by William Shakespeare

Congratulations! You’ve completed the second psychology article. This was the longest and most technical, with its pictures of brains and discussions of neurological processes.

We learned that we don’t see with our eyes; we see with our brains. Seeing is a complex operation with many steps and processes happening along the way. Many of these happen even before we are aware of seeing something; they are pre-attentive. Others happen within the realm of conscious thought, attention, and therefore learning.

We’ve seen how pictures convey information much more efficiently than mere words. We’ve learned some principles of perception to organize those pictures to tell visual stories even more effectively. We examined colour: how we see it and how we can use it. And we touched on the optical illusions that can get in our way when we project three dimensions onto two.

By now, you should have a good understanding of the limitations in our hardware—or is that wetware?—that constrain your abilities as educators and storytellers. We have one more psychology article ahead. But first, we’ll put our newfound knowledge to the test as we design data visualizations and dashboards.