The Holy Grail of advertising research might look something like a cross between a blender and a calculator. It would take all the variables in the mix and magically calculate an exact Return on Investment for advertising. For now, the measure many large advertisers rely on is in-market awareness of their advertising. The measurements provided by tracking studies are key performance indicators which senior management uses to assess the success or failure of advertising campaigns. Consequently, the size of the advertising budget, and even job performance reviews, are tied to these numbers. The accuracy of these advertising measurements, therefore, is of critical concern to advertising and brand managers and their agencies.

For many companies, continuous advertising trackers, such as those offered by Millward Brown, Ipsos, TNS and other large research suppliers are frequently one of the most expensive items in the research budget. Data collection for many of these is now being moved from the telephone to the internet.

There are two main reasons why ad trackers are being moved online. The first, of course, is the ability to obtain significant economies in the cost of data collection. But the second is the ability to switch from a recall-based to a recognition-based form of advertising awareness measurement.

There has been a long running debate in advertising research circles as to the relative merits of recall versus recognition as the best way of tapping into consumer memories of advertising.

Recall was the first report card measure used in commercial pre-testing, and it is still widely used today. Those who favor a recall approach use the common sense argument that for advertising to be effective it must first lodge itself in memory—a necessary, if not sufficient, condition for advertising to work. This is particularly relevant given the standard argument that the real payout for the advertising investment is to be found in its long-term effects.

But there is more than one way to tap into consumer memories of advertising. Those who favor recognition argue that a consumer’s inability to recall or actively retrieve the memory of a television commercial days or weeks after seeing it is not a valid indicator of whether or not the consumer has in fact seen the ad or whether that ad has indeed made a lasting impact on their stored impressions of the brand. Indeed, recognition-based awareness measures are usually quite a bit higher than those produced by recall.

In commercial pre-testing, for example, Ipsos-ASI, the category leader, uses recognition as a complementary measure to day after recall, and finds, on average, that consumers will report roughly twice the level of ad awareness with recognition as opposed to a recall approach.

As a nod to recognitionists, therefore, telephone-based trackers have sometimes fallen back on the idea of using verbal descriptions of an ad to cue recognition.

However, verbal measurements do not reliably capture visual experience. Imagine if you had to find someone in a crowd that you had never met before. Think of how much more difficult it would be to recognize that person if all you had to go on was a written paragraph describing the person you’re looking for…as opposed to having a picture of the person. Verbal recognition is simply not the same as visual recognition.

The main reason verbal methods have been used in the past to track television advertising has been simple economics: for the past generation, the telephone was cheapest way to collect tracking data–but you cannot show consumers pictures over the telephone. With the internet, you can combine telephone’s low cost with the ability to show images.

Most research companies now tracking advertising over the internet use a visual approach to measuring television advertising awareness. The most common method is to use a storyboard, typically comprised of either four or six de-branded still photos taken from the commercial, to stimulate recognition.

Initially, for early adopters of internet tracking, the reason for using storyboards rather than playing full video was based on bandwidth restrictions. But now the main rationale for storyboards as the preferred approach is based on interview length. You can gather more data on more advertising executions using still photographs, which the respondent might spend a few seconds looking at, than you can if you have to download ad films.

Quite useful

This ability to measure ad awareness by individual execution can be quite useful in our age of fragmented media and integrated marketing. Advertising managers are beginning to demand more insightful information from their expensive tracking studies. They want to know more than simply, “How many consumers have seen the ad campaign?” Increasingly, they are beginning to ask more precise questions such as, “Which of my ads has the consumer seen?” In other words, which are the strong performers in my advertising mix, and which are the weak performers? These are questions that recognition-based measurement—but not recall-based measurement—can now begin to address, time permitting.

A previous paper I published in Quirks, “The Eye is Not a Camera”, (Quirk’s, March 2003) confirmed the validity of using storyboards as a substitute for full video in commercial awareness tracking. This paper described a method for choosing the right still photos to use in constructing the storyboard for the recognition measurement. But choosing the wrong photos, as I will show below, can lead to a considerable understatement of the true consumer awareness of your advertising.

When most research tracking companies construct the storyboard stimulus for on-line ad tracking the typical approach is for the research analyst to use “judgment,” sometimes in “consultation with the client and agency,” to select the four to six still photographs to represent the commercial. Yet in our considerable experience from pre-testing the visual component of television commercials using our Picture Sorts® methodology, we know that up to thirty pictures may actually be needed to accurately capture the visual information content of the average commercial execution.


You can get a very different impression of the commercial depending on your choice of visuals. Exhibit 1 shows two storyboards constructed from the same commercial. While there is some overlap, it looks like the storyboards are describing two different commercials. If this was your advertising being tracked in the marketplace—and next year’s ad budget depends on how well it performs—which of these two storyboards would you use to cue recognition of your ad?

One of these two storyboards was constructed using the judgments of an ad hoc panel of twenty experienced researchers (with a minimum of two years experience in the ad research business) who voted on which images to use in the storyboard. The other was constructed using consumer feedback from a commercial pre-test.


The consumer feedback was in the form of an Ameritest Flow of Attention® graph, a key diagnostic in our on-line pre-testing system. (See Exhibit 2) In the pre-test interview consumers “sort” all the pictures used to represent the visual information content of the ad, usually from ten to thirty pictures for a typical 30-second ad (depending on the visual complexity of the ad). The sort is done by having the images brought up randomly on the screen fifteen to twenty minutes after a consumer’s exposure to the ad and asking them whether or not they remember seeing those images in the ad. The graph shows all the visuals from beginning to the end of the ad, with the height of each picture in the graph showing the percentage of the commercial audience remembering that image.

“Peak” moments

In analyzing these attention flows the “peak” moments are particularly useful. By definition, these are the images the consumers remember best after their experience of the ad has been transferred from short term to long term memory—so it should not be surprising that these images are related to the long term memory effects of the advertising. A number of papers we have published (which can be found on our website) show how content analysis of these peaks can be used to predict the attention-getting power and the day-after-recall score of a television commercial.

In the experiment described above we were trying to understand how well researchers could predict in advance which visual moments in the advertising the consumer was likely to focus on. We looked at ten commercials, like the one shown in Exhibit One, representing a range of categories from packaged goods to high tech. We asked the panel of researchers to predict which ones would be the peak images of the commercial from the viewpoint of the consumer. We then compared the results to actual consumer data from pre-testing.

What we found was that experienced researchers were not very good at predicting what consumers would focus on in the ad. Using judgment, researchers were correct only 44% of the time in predicting what the peak visuals in the commercial would be for the target audience consumer.

Of course, it’s likely that creatives might be better at this prediction business, but our pre-testing experience suggests that they are frequently surprised by what the consumer focuses on as well. But more to the point, creatives are infrequently asked to provide input as to which visuals to include in tracking studies.

Does it matter?

The central question is this: Does the choice of pictures used in the storyboard stimulus really matter? To answer this question, we performed a separate experiment recently with one of our major clients, Unilever. (This experiment was reported in a different form at an ARF conference.) In this experiment we had two cells of matched samples of 100 target consumers each for one of their successful personal care brands. We took visual samples from eight commercials that had aired over the past several years and measured ad awareness using a recognition approach.

(As an aside, it should be noted that, unlike recall, recognition memory decays very slowly, also making recognition the better approach for studying the long-term effects of advertising.)


This difference between the two cells was this: in the first cell the recognition stimuli were only peak images that were identified from the pre-test Flow of Attention, while in the second cell the stimuli that we used were entirely non-peak visuals. Ad awareness results from the two cells are shown in Exhibit 3.

As you can see, for the very same set of commercials, advertising awareness is 40% higher for the stimulus using peak visuals compared to the one using non-peak visuals!

This magnitude of difference would be sufficient for researchers to reach entirely different conclusions about the success of the advertising campaign.

Peak visual moments can be easily identified in a pre-test within twenty minutes of commercial exposure. These turn out to be “branding moments”—the small number of images lodged most strongly in consumer memories long after the advertising has aired in the marketplace. These branding moments are one of the keys for identifying the long term effects of advertising. And they are also key to understanding the metrics that, for now, stand in for advertising’s ROI.

Of course, this is the extreme case, because in the real world a storyboard constructed using “judgment” is likely to contain a mix of both peak and non-peak visuals, as demonstrated by our first experiment. Nonetheless, this underscores the importance of identifying the right images to use in recognition-based internet ad tracking in order to narrow the range and improve the accuracy of awareness measurement.

New risk

So, while moving ad tracking studies from the telephone to the internet represents a substantial gain for researchers working on the problem of measuring advertising’s impact in the marketplace, there is a new risk that internet trackers who are not careful may still understate true ad awareness. The solution is to build a bridge between two critical pieces of internet advertising research, pre-testing and post-testing. By feeding consumer input from pre-testing into the post-testing design, you can achieve a critical methodological improvement in the accuracy of your online ad tracking.


Young, Charles, “The Eye is Not a Camera,” Quirks Marketing Research Review, March 2003.

Young, Charles, , “Researcher as Teacher”, QuirksMarketing Research Review, March 2001.