In all acts of perception and communication, emotion comes first, and thought comes second. An intriguing scientific argument for this proposition has been put forth in a recent book titled The First Idea, by Drs. Stanley Greenspan and Stuart Shanker, experts in child development and evolutionary science. Newborn babies experience the world emotionally before they learn any words, ideas or rational concepts. According to these authors, the heart of the mysterious processes by which we humans learn to think and communicate can be found in that intimate space between mother and child, where a baby looks up into mother’s face and the mother looks down into baby’s face. In this loving space the first gifts of thought and reason, of language and culture, are given and received. An example: a baby a few weeks old learns the laws of cause and effect by discovering that he can make mommy smile by smiling first.

Cause and Effect

In that interface, mutual readings take place where the newborn begins to learn what those inchoate feelings mean and how to think about them. This pre-verbal search behavior is driven by un-named and unconscious emotions and sensations. This visual learning process between an infant and a caregiver is described as a “negotiated process of co-creation.”

Screen-to-face communication is very much like face-to-face communication, particularly for close-up computer or television screens. That is why this idea of the co-creative process is relevant to understanding how advertising works. From our very beginning, we humans have an innate, biological urge to read other human faces. It’s why most magazine covers have faces on them.

One of the most important discoveries in the past ten years in cognitive neuroscience is the existence of mirror neurons in the brain. Basically, a mirror neuron is a neuron that fires both when an animal observes an action performed by another animal and when an animal performs an action. Overwhelmingly visual in nature, these neurons “mirror” the behavior of another animal. Mirror neurons are the biological basis of how we learn by watching others.


Through the play of mimicry the developing child learns to first read and then express the inner life of emotions. Non-verbal body language and specifically the sensitive movements of the face are our first and most important language for communicating emotions. At the other end of the reproductive cycle our desire for emotional communication explains why, unlike other animals, we humans make love face-to-face.

The complex musculature of the human face provides the machinery for expressing six basic human emotions from which all others are derived, like the primary colors on a color wheel. Scott McCloud in his book Making Comics shows us in cartoon form how graphic artists mix different facial expressions to move us emotionally with a few strokes of the pen. McCloud also goes on to also show how, through carefully chosen sequences of just a few frames, emotions can be turned into stories that give us new insights into the human condition and how the world works.


In his book on how to tell a story in film Robert McKee describes the “beat” as the “fundamental unit of film structure.” McKee is an actor as well as a writer and his definition is given from an actor’s perspective. From his standpoint the beat refers both to the verbal and the non-verbal give and take between actors on the screen, within the frame or adjacent frames. With live actors on the stage this back and forth communication is what we refer to when one actor flubs a line and another improvises a new response “without missing a beat.”

But there is another way to think about the beat of film storytelling, one that is perpendicular to the screen, a second dimension to McKee’s definition. The back and forth rhythm of expectation and actuality, of anticipation and surprise that emerges from the complex interplay between the imaginings of the director and the imagination of the audience determines the beat of visual imaging, not the actors’ dialogue.

Psychic Energy

A musician performing before a live audience can feel directly the rhythm and flow of psychic energy back and forth as he connects with his foot-tapping audience. But for a film director the connection with the pulse of her audience is time-shifted, like TIVO in reverse, so that the performance of shooting and editing a movie is separated by a long period of time from the mental applause of her viewers. The best directors are those who have the intuitive ability to project themselves outside their own heads in order to see the work through the eyes of the audience.

The idea of a co-creative process implies a transaction-based model of communication. This is different than the historical view of visual communication as being one-directional, with an active advertiser sending messages to a passive couch potato consumer. We now understand that all forms of human communication are two-directional, involving the active mental participation of both a sender and a receiver. Interactivity is not just a recent invention of the Internet.

As advertising researchers, we too can look at television commercials through the eyes of viewer audiences with our Picture Sorts® techniques. We can actually measure and visualize the structure and beat of visual storytelling in film or video with a Flow of Attention® graph—a moment-by-moment map of audience attention that looks like a music score annotated with pictures instead of notes. Mathematically, the tempo of peak visuals in these graphs, the beat of the film narrative, has been shown to correlate strongly with the overall attention-getting power and long-term memorability of television commercials.

A peak in the Flow of Attention is where the audience’s moving visual field of interest is centered. The pre-conscious brain operates like an advanced digital auto-focus camera that knows not only that it’s important to focus on the face, but also calculates just the right time to take the picture. The baby is smiling now—quick, click the shutter! These special moments in time are where the critical transactions of emotionally charged information take place between the visual storyteller and the audience.

Transactional Exchanges

All business is based on the concept of a transaction, the exchange between a seller and a buyer of goods and services for money or something of equivalent value. Advertising also involves a transaction, with the advertiser offering relevant information or meaningful emotional experiences in return for the limited attention and memory resources of the consumer.

When we laugh or cry at an image we see on the television screen we are performing a form of emotional mimicry, evoking interior emotional states that resonate with the emotion states conveyed by pixels of light. Mirror neurons may be the biological basis for our ability to empathize, our neural dendrites plucked like the strings of one musical instrument being tuned to another. Emotional resonance is another way to conceptualize an emotional transaction between the two end points of a communication.

The complex emotional transactions that take place when two human beings interact has been described simply in an interesting theory known as Transactional Analysis (TA) which was developed by the psychologist Eric Berne. This psychoanalytic theory was quite popular in the nineteen seventies in part because of his insight that our emotions need to be understood in the larger context of the roles we act out in our day-to-day lives.

Berne thought of roles at the most basic level in terms of three ego-states (not to be confused with Freud’s trinity of the Ego, Superego and Id) with which we are all intimately familiar: those of Parent, Adult and Child. The emotions which we associate with the role of Parent—caring, nurturing, protecting for example—are quite different than those we associate with our inner Child—joy, anger, fear—or those we express when we are trying hard to stay in our Adult role—pleasure, sadness, disgust. Of course, the differences in the degree and quality of many emotions may not have a word in English sufficiently nuanced to capture it—the anger of a disappointed but loving parent should be very different from the anger of a frustrated out-of-control child.

The importance of roles is that they determine our expectations in different social situations. For example, in a dangerous situation involving real children we expect a man’s fear (inner child) to be trumped by his sense of protectiveness (inner parent.) Setting and then occasionally intentionally violating expectations is, of course, an essential part of storytelling.

Brand Roles

Brands play roles in our lives as consumers and it is easy to see how many brands can fit with Berne’s three types. Cake mixes from Betty Crocker and pain relievers endorsed by authority figures in white lab coats appeal to the parent inside our heads. Fast cars and fast food are usually sold to our inner child. And it’s the adult who buys perfume or shops at Victoria’s Secret.


Transactional Analysis shifts the focus of attention of communication analysts from trying to understand the internal emotional dynamics of the individual to understanding the emotional interactions between the roles being expressed by two people trying to communicate with each other. According to Wikipedia, “transactions are the flow of communication and more specifically the unspoken psychological flow of communication that runs in parallel.”

There are, of course, a number of different types of transactions that can be used by a brand in advertising to consumers. There are simple, reciprocal transactions that take place when both parties are addressing the same role the other is in. McDonald’s, expressing its adult side, might simply be informing your adult side that there is a special price offer this week on Big Macs. There are also complementary transactions when the brand, operating in one role, appropriately addresses a different role of the consumer. Showing cute kids having fun eating Happy Meals at McDonalds is a way of evoking the nurturing emotions of the adult-as-parent.

Crossed transactions can also happen and these are frequently a cause of communication failure. A few years ago we tested a Wendy’s commercial featuring the late Dave Thomas running around telling his breakfast customers not to eat a big breakfast because he had a great new bacon cheeseburger to sell them for lunch. Typically in commercials that used him, Dave appeared to communicate with the consumer inner-child to inner-child, performing the role of the lovable big kid in town who knew where to get the best hamburgers. But in this one commercial he was mis-cast in the psychological role of a parent figure, talking down to your inner child, telling you what to do—and consumers hated the ad.

At the time we had performance scores that showed the ad wasn’t working, but we weren’t sure why. We had this hypothesis, which seemed to provide some insight, but no data to prove the point. At the time the non-verbal tools which we needed to measure the specific emotions that a consumer experiences when watching a commercial had not yet been developed.

Three-dimensional View

At Ameritest we now use three different Picture Sorts® to provide a three-dimensional view of the consumer’s emotional response to ad film. The first sort, the Flow of Attention that was described previously, captures the workings of unconscious emotions directing the eye as it scans and sorts through visual stimuli, deciding what’s important enough to let into the conscious mind. The second sort, the Flow of Emotion, captures the levels of positive and negative emotion which the consumer becomes conscious of feeling in watching an ad—a response curve that is mathematically independent of the first curve but which has been strongly correlated with purchase intent or overall motivation. The third sort, the Flow of Meaning, overlays the color of fleshy nuance to the black and white sketch of positives and negatives provided by the second sort, by measuring precisely the levels of specific emotions evoked by each image in an ad; think of this as the emotional palette an ad has been painted with—which is key to understanding whether or not the emotional communication of an ad is on strategy.

The measurement of these three modes of emotional involvement is quite simple. All three Picture Sorts use the same deck of still photos taken from the ad as a non-verbal emotional “vocabulary” for describing audience response to the ad. It works because, in terms of storing and retrieving human emotions, nothing works quite so well as a photograph—which is why we all keep albums full of pictures of friends and family. For the first of the three sorts, respondents the photographs into two groups—those they remember and those they don’t. For the second sort, they place the pictures they remember into one of five groups representing a scale from very positive to very negative. And for the third sort, they sort the photos into different buckets of meaning, described by a customized set of emotional words or categories that might be used to describe the particular emotion (or positioning ideas) they felt as they watched each image in the commercial.

A Soup Opera

The usefulness of picture-sorting information for understanding the emotions generated by a commercial can be illustrated with a brief case history of a Campbell’s soup commercial recently tested as part of an Advertising Research Foundation study of emotion in advertising.

This sixty-second commercial tells the story of a recently orphaned girl who is brought to a new foster home. In the opening shots we can see in her face the sadness and fear she feels as she stands with the social worker on the porch of her new home. A tight close-up shows the girl gripping her suitcase nervously. The middle-aged woman who opens the door and brings her inside smiles and tries to be friendly, but the girl is downcast and homesick. Then, with a flash of insight, the woman goes to the kitchen, opens the cupboard and reaches for a can of Campbell’s soup. When the woman serves the soup to her the girl recognizes it as the same brand of soup that her real mother used to serve her. So, by the end of the ad, we see on her face that the girl’s feelings toward her new mom have changed completely.

For advertisers, what’s important about emotion is the motion in emotion. Advertisers use advertising to drive consumer-purchasing behavior or to move consumers to identify more closely with their brands. In storytelling, the emotion-state the audience is in at the end of a story should be quite different from the state they were in at the beginning—otherwise, emotionally, nothing really happened in the ad. As McKee correctly points out, the purpose of stories in our lives is to explain how and why life changes.

If we look at the Flow of Meaning for this commercial, shown in Exhibit 1, we see high levels of fear registering in the critical images falling on the first few beats of the story—the orphan’s face (50% of the audience) and the white-knuckled grip on her suitcase (67%). At the turning point of the story, when the familiar Campbell’s soup can and the preparation and serving shots arrive on the middle beats of the story, we see new emotions being introduced—e.g. caring (14%) and nurturing (35%). Finally, on the ending beats of the story we see the emotional change registering on the girl’s face, with the strong levels of trust (79%) she is feeling toward her new mom.

A transactional analysis, shown in Exhibit 2, of the movement in emotional states evoked by this commercial provides additional insight into the strength of this ad. On the screen, the non-verbal back and forth looks between the girl and the woman can be analyzed in terms of the emotional transactions taking place between two actors. In the beginning, the emotional transaction is child-to-adult as the girl reacts to the woman with the fear and wariness that is natural for an older child meeting a stranger for the first time. But then the woman does something unexpected—she serves the girl Campbell’s soup. As a result, by the end of the commercial the emotional transaction shifts to child-to-parent as the woman makes the transition to her role as new mom in the eyes of the child.


The emotions the audience is feeling mirror the emotions shown in the ad. In the beginning we watch the commercial in our default role as adults, merely observing the fear in the child’s face, so the emotional transaction taking place between the commercial and the viewer is also child-to-adult. But as we are drawn deeper into the story and begin to care about what happens next, we shift out of the adult role into own parenting ego-state. In the end the new-found trust we see in the face of the girl validates the behavior we observed in the ad—that serving Campbell’s would make you feel like a good mom. The insight of the ad agency is that Campbell’s does not sell soup to adult women—it sells soup to women in their role as home-makers and mothers and advertising must shift the consumer’s emotional adult-state into the parent-state in order to make a sale.

This case history is, of course, only a specific example of a more general point that applies not just to television but also to all advertising media. Whether or not it’s more rational messaging or an attempt to create an emotional connection with the brand, we should never lose sight of the fact that in every transaction that a viewer has with a piece of advertising, only half of what’s being put into the communication is put there by the advertiser; the other half is what’s put there by the consumer. The idea of Co-creativity is quite simple: it takes two to communicate.


Berne, E., Games People Play, Ballantine Books, 1996

Greenspan, S. and S. Shanker, The First Idea, Da Capo Press, 2004

McCloud, S., Making Comics, Harper, 2006

McKee, R., Story, Harper Collins, 1997