It would be three decades after the motion picture industry was born before audiences heard the words and the voices of the actors in a film, but silent movies were never really silent. From the earliest days of the art form movie makers understood that movies without sound can be deadly boring, so a piano player always accompanied the action on the screen to help audiences emotionally enter into the world of the moving pictures. Film makers have always intuitively understood how music and movies can work together in partnership. More recently, researchers in cognitive neuroscience have been discovering that the way the human brain processes music and the way it processes movies is a lot more alike than we realized.

Music is one of the three channels of communication that can be woven together in a piece of advertising film—with words and the visuals. A recent analysis by WPP of a research database of 3500 commercials from the Asia Pacific region found that slightly more than half (56%) used music in some way in the advertising. Creatives understand the role that music can play in attracting the attention of consumers and in generating the kinds of emotions that build brands.

Advertising research sometimes produces results that are counter-intuitive to creative people. Perhaps one of the more perplexing was a paper published a few years ago by Ipsos-ASI, one of the world’s leading copytesting firms, which analyzed a large database of ads and found that music, “particularly loud music at the level of the spoken word,” resulted in lower recall scores for television commercials. In contrast, a recent academic paper by Steve Oakes of the University of Liverpool found, in a review of the literature based on non-copytesting experiments, that likeable music increased memorability, purchase intent and attitudes and preferences for brands. Key to his analysis was the idea of “congruence”, or the appropriate use of music in advertising.

MindingMusic_gettingitright

The importance of music

Just how important is this creative variable in determining the overall effectiveness of the advertising—or, perhaps more to the point, how important is it to get it right? To answer this question, we recently did our own analysis of a database of commercials that had been tested in the last few years in the Ameritest system. We looked at a sample of 1400 commercials that used music as a creative element and ranked the ads according to just the one diagnostic variable measuring the likeability of the music as it was used in the ad.

MindingMusic_figure1

Comparing the top quartile to the bottom quartile of ads in terms of music liking (See figure 1) we find that music appears to make a big difference. Ads with the best-liked music have nearly twice the attention-getting power of the ads with the worst-liked music. Moreover, the best-liked music appears to double the motivation to buy the brand. The impact of likeable music on brand linkage seems less clear, however—though ads with strong music score at norm on this measure. (As suggested above, the impact of music on branding may be more a function of the congruence of the music with the brand rather than simple likeability.) But overall, these correlations suggest that music is a non-linear creative variable: it doesn’t just add to the advertising experience, it can actually multiply the performance of a commercial.

What can we say about commercials that get the music right? We think the answer goes beyond simply finding a piece of music that your audience likes. By itself, you might like a piece of music very much, but you might not like it as much in the context of a particular piece of film. In the best ads, music and video move together in a complex creative dance, without tripping over each other’s feet.

Sensory integration

Recently, neuroscientists have discovered the existence of distinct, multi-sensory neurons that are involved in the integration of the different sensory channels of the brain. In our day-to-day lives perception almost always involves multiple senses. Most of the time, our senses interact to make each one work better.You can hear someone more clearly across a crowded room if you focus your visual attention on them. Experiments have shown that you will see something more quickly if it’s accompanied with a sound. Multi-sensory neurons are at work when, as we will see, audiences process more of the images in a film accompanied by music.

At a fundamental level of perception, music and movies are both art forms that encode information in time and thus are processed by the human brain in surprisingly similar ways. Indeed, the sense of the subjective passage of time that swells and contracts as different emotions are engaged is a primary characteristic of the experience we have when we enter either the sonic world of a piece of music or the visual world of a piece of film. In a recent Admap article, we described how perceived differences in the flow of subjective time are related to the effectiveness of ad film, both in terms of the number of peak moments of attention and purchase intent. Neuroscientists have found that one of the functions of the oldest parts of the brain, the cerebellum, is involved in setting our internal clock, a mental metronome that can synchronize our thoughts and feelings with events that occur outside the brain. There is a rhythm and beat to how our minds make sense of the world.

An outdated model of the brain is that it works something like a computer, but a more up-to-date view is that the brain works more like the internet where processing is very distributed. Perception, thinking, memory formation and imagination are all complicated processes of deconstruction and reconstruction. Timing is critical to how different neural circuits in the brain are synchronized to produce a single, unified experience.

In his new best-seller, This Is Your Brain On Music, the neuroscientist and musician Daniel Levitin reported an interesting finding about how the temporal processing of music and of visual communication are related:

“Most astonishing was that the left-hemisphere regions that we found were active in tracking musical structure were the very same ones that are active when deaf people are communicating by sign language….We were now looking at a region that responded to sight—to the visual organization of words conveyed through American Sign Language. We found evidence for the existence of a brain region that processes structure in general, when that structure is conveyed over time.”

Getting the beat

The beat is the fundamental unit of structure not only of music but also of film. The beat of how an audience cognitively processes film, from pre-conscious filtering to conscious awareness of the visual information in moving pictures, is related to what neuroscientists call the “attentional blink”. The attentional beat of film is a consequence of the limited capacity of the conscious mind to hold on to very much information at one time. To measure and visualize the beat of advertising film we use a picture sorting technique, the Flow of Attention®, which is a graph that looks like a musical score with pictures from the commercial substituting for musical notes. In other published studies we have shown how the frequency and content of the video beat is highly correlated with the attention-getting power and memorability of television commercials.

The tempo of the beat in a piece of music is also related to our emotions. Music with fast tempos tend to be regarded as happy, and songs with slow tempos as serious. Beat plays a key role in storytelling, whether in a film or in a symphony. When we tap our feet to a piece of music our brains are making predictions about what is going to happen next. And, as with all good storytelling, composers continuously set expectations and then manipulate them with surprising turns in the music. For example, musicians sometimes add excitement to a piece with syncopation—when a note comes a bit earlier or later than the strict beat calls for—which surprises us and generates emotion in the gap that opens up between our expectations and the actual performance. The challenge for creative people regardless of their medium of expression is to build excitement by giving audiences (and clients) what they want, but not in the way that they expect it.

Emotion and importance

Brain researchers have firmly established that we tend to encode as important things that carry with them a lot of emotion, either positive or negative. Memory of music and memory of film both involve hierarchical encoding of content. Not all notes in a musical phrase and not all images in a film sequence are equally important. Experiments with musicians have shown that they cannot start playing a piece of music they have learned at an arbitrary location, but rather there are certain entry notes for recall. In another example, Levitin reports that musicians are faster and more accurate at recalling whether a certain note appears in a musical piece if that note is at the beginning of a phrase or falls on a strong beat, rather than being in the middle of a phrase. Similarly, the early Russian director Eisenstein, in writing about his theory of film, talked about the “privileged instants” in film where pathos is created. The images of a television commercial that fall on the beat of an attention curve—which are the images that stand out most clearly in audience memory minutes after seeing the commercial—are the ones most important in the hierarchy of the commercial’s structure, which is why they are the most predictive of clutter break-through and long term advertising memorability.

Three types of music

Leopold Stokowski, who was the conductor of the Philadelphia Orchestra and the narrator, begins the 1940 Disney film, Fantasia—which is arguably the world’s first music video—by identifying three different types of music. First, there is the type of music that exists just for its own sake. Second is the type of music that has no specific plot, but paints a definite picture in the mind of the listener. Third, is the type of music that tells a story.

Advertising makes use of the second and third types of music. A montage commercial that is cut to the beat of a popular song would be an example of the second type. Anthem commercials like these are used to celebrate a brand by associating emotionally-charged imagery with the brand name. In these types of commercials the music is in the foreground and the pictures are the background of the advertising experience. They are typically devoid of semantic content, since they are not designed to communicate rational information about a product. For that reason these types of commercials do not recall test well—because recall testing measures the semantic content of ads.

Storytelling

It is commercials of the third type, where music plays a role in the storytelling, that are the more interesting.

Stories are essential to how we make meaning out of our complex experience in order to communicate it to others. The writer uses words, the cinematographer uses pictures, the musician uses sound to tell their stories. The scientist uses numbers to tell us his story about how the world works, though he calls his stories “theories” in order to make them sound more important. Regardless of the medium of communication that is used, the structure and substance of storytelling seems to be fundamental to how our minds process information that is sequential over time.

To study how music and moving pictures interact to tell a story we performed a simple experiment with the “Sorcerer’s Apprentice,” a chapter of Fantasia that used Mickey Mouse in a classic example of dramatic storytelling without words. We recruited two matched samples of 100 consumers who had not seen the movie before and showed them the nine minute clip—with one important difference. One group saw the full music video—the other group saw the clip with the sound turned off. Afterwards, we picture sorted the visual images that each group remembered seeing in the movie.

MindingMusic_figure2

In general, we found that musical accompaniment increased the amount of visual information processed by the audience. (See figure 2.) The average difference in picture recall was small—64% of images were recalled with music compared to 58% of images recalled without music. But importantly, the number of visual peaks—the beat in the flow of visual attention—nearly doubled as a result of the interaction between the music and the visuals. Given the statistically large importance of peak moments of attention in driving commercial break-through and memorability, this finding is consistent with the multiplying effect of music on ad performance.

A musical description of the film goes something like this. The scene begins slowly with Mickey working as an apprentice in a medieval setting. Music is piano, using a legato temp. As the sorcerer leaves and Mickey puts on the sorcerer’s hat, the music begins to show signs of dissonance as the strings play a little faster. A cesura brings the broom to life.

As Mickey uses magic to gain control of the brooms, the brass and woodwind sections take the shape of the brooms’ movement. The brooms march in time with the staccato, alla marcia musical cues.

Next Mickey has an out-of-body experience featuring many sforandzos as Mickey dreams of mastering the waves in the sea and then the stars in the sky. Cymbal crashes mimic the crashing of the waves and his arm movements reflect the sforandzos that are used. Mickey then begins to lose control, and begins to drown as the speed of the music increases as does the dissonance. The glissando of the strings heighten the tension.

Finally, the appearance of the sorcerer near the end of the scene brings the musical crescendo to a sudden halt by using loud crashes from the percussion section, followed by a cesura. Sforandzos are the musical cues that are followed by the movements of the sorcerer’s arms.

As the scene draws to a close, piano consonance reappears as Mickey sheepishly reassumes the role of apprentice. A quick forte sequence and a cymbal crash end the musical piece as the sorcerer sends Mickey out of the room with a mighty sweep of the broom.

MindingMusic_dj

Interaction: music and visual

A number of general observations can be made about the interaction between the music and the visuals in this film. The use of staccato notes is prevalent in images and scenes that are better recalled. The quickly forgotten images use more of a legato tempo and are often piano/pianissimo in nature. Visual peaks often follow musical peaks. For example, when music reaches a crescendo or sforandzo that is used in a forte/fortissimo manner, important changes in action follow. Dissonance in the music often increases as the situation of the story worsens. Consonance appears when the story reaches a resolution.

MindingMusic_figure3

For many of us, talking about music in words is a lot like translating a foreign language. But a simple content analysis of the specific images that were remembered at peak levels only when accompanied by music suggests several mechanisms by which the music was working to focus audience attention on the visuals. (See figure 3.)

a) During the sequence when the animated broom empties buckets of water into the basin the ominous melody is established for the first time, foreshadowing trouble ahead, cueing viewers to pay closer attention because something is going to happen.

b) When Mickey’s dream self escapes from his body, the loud music stops abruptly, creating musical “white space” which again focuses audience attention on what is happening.

c) As brooms throw increasing amounts of water into the basin the music is extremely tension-filled, but then brass instruments are introduced to control the melody and providing a musical resolution to the preceding sequence of confusion and heightening attention to the visual.

d) An onslaught of brooms tramples Mickey accompanied by an influx of loud instruments which amplify the action.

MindingMusic_theresults

e) A whirlpool engulfs Mickey as a horn’s melody is singled out which, in its rhythm, speed and texture, mirrors Mickey’s whirling motion. Because the images and music are synchronized, the audio movement accents the visual movement and focuses audience attention.

f) As Mickey leaps aboard the Sorcerer’s spell book, thumbing through the pages for help while the brooms rage out of control, the music trills down the scale which emotionally signals, in its rapid descent, that a crisis is occurring.

g) The sorcerer parts the water with his magical powers, as clashing cymbals mirror the movements of the sorcerer’s arms. And the music stops—again creating musical white space which heightens attention to the visual.

h) Mickey Mouse abashedly tries to make amends by indicating his willingness to work while a triangle chimes lightly over subdued music. Again, the change to quiet, almost inaudible music increases viewer attention.

The rhythm of storytelling connects the past of our experience with our brain’s moment-by-moment predictions about the future. As this film example illustrates, music working together with moving pictures plays a game with our expectations. Foreshadowing, amplifying, creating empty dramatic space or “air” between one moment and the next, and mirroring, or sometimes contradicting, what’s on the screen—music draws out not just our emotions but also puts our imagination to work to find a more pointed interpretation of what’s really going on with what we see. And, in the case of advertising, this helps audiences find richer meaning in the brand’s story.

References

Levitin, D., This is Your Brain on Music, Dutton, New York, NY: 2006

Oakes, Steve, “Evaluating Empirical Research into Music in Advertising: A Congruity Perspective”, Journal of Advertising Research,vol. 47, March, 2007

Walker, D. and von Gonten, M., “Explaining Related Recall Outcomes,” Journal of Advertising Research, vol. 29 (June/July), 1989

MindingMusic_end