Affdex, a facial recognition software made by Affectiva, is being used by advertisers to measure audiences’ emotional reactions to ads. Capturing minute facial changes, Affdex tracks viewers’ fleeting emotional reactions, allowing analysts to identify, through proprietary algorithms based on “283 facial frames,” their positive and negative responses to an ad. Affectiva hopes to sell this technology to the TV industry so that “smart” TVs could, on the basis of one’s previous emotional responses to programs, auto-program one’s TV set.
With a webcam, according to Affectiva CEA David Berman, Affdex could eventually be used as a “more scientific version of Facebook’s Like button.”
In the next two years, Berman envisions Affdex becoming a complement to “smart” televisions that can understand people’s preferences. “If my wife and I both like to watch the same show, it will fine-tune the algorithm,” he says. “It puts the emotion back into viewing.”
Advertisers anxious about audience response have always sought quantifiable measuring techniques. Beginning in the 1930s, Daniel Starch pioneered techniques for measuring readers’ views of magazine advertisements; his Starch Reports claimed to identify effective headlines through empirical studies of the number of seconds readers spent on an ad. Today, Betsy Frank at Time Warner uses “biometrics,” such as mini video cameras and belts measuring heart rates and breathing, to track users’ responses to media. Back in the late 1930s, Paul Lazarsfeld and Frank Stanton invented a “Program Analyzer“ to measure radio audiences’ responses. Today, visitors to Television City in Las Vegas may participate in television audience research by using an updated version of the Program Analyzer, turning a dial to indicate their positive or negative response to each moment of a program.
In most of these approaches, researchers have to rely on some form of audience self-reporting, which is inherently variable. While participating a “dial trace analysis” in Television City, I noticed my neighbor laughing heartily at punch lines of a tested program but not moving his dial up past 30 (out of a top score of 100). So, to make up for his relative lack of cooperation, I moved my dial up every time he laughed. Afterward, I asked him why he didn’t move his dial much and he said that a 30 score was a reasonable score to reflect his mild amusement. Clearly, we were operating on different rating scales since I assumed that 70 was a reasonable score for anything mildly amusing and listening to him laugh heartily I assigned his behavior 80-95 scores. How useful would our data be for program analysis since we used such different standards?
Affdex removes the problem of audience self-reporting by measuring and defining facial expressions. But the application of algorithms cannot remove the problem of how to interpret that data. Why or how the algorithms determine that a facial expression indicates positive or negative responses is locked inside the black box of proprietary software and is reflective of the biases of the makers of those algorithms. Does the software categorize human emotional response beyond the simple binaries of “positive” and “negative”? How can it account for other variables, such as context, social viewing, or multitasking? What if the viewer’s responses result from hunger or sleepiness rather than the viewed program or ad? Even more problematic, does identifying a viewer’s facial expression as “positive” actually have any predictive power as to the viewer’s likelihood to buy or watch in the future?
Audience measurement continues to rely on proxy information: viewership, as measured by Nielsen people meters, and now emotional engagement, as might be measured by Affdex. (For an excellent analysis of television audience measurement, see Philip M. Napoli’s Audience Evolution).
Yet, exposure and engagement are only proximal to what advertisers really want to know: what ads actually convince consumers to buy? Perhaps Affdex will help advertisers eliminate ads that generate negative responses, but will it be able to reliably account for the multifarious variables that affect consumer behavior? Will viewers prefer that their TV pre-select programs based on Affdex’s algorithms? Or will that actually remove some of the “emotion” from viewing?