Multi-Institution Study Replicates Net Promoter Methodology; Results Indicate Concept Has Been Vastly Oversold to Companies Worldwide
NEW YORK, December 4, 2006 – A widely used customer loyalty metric billed as the “single most reliable indicator” of a company’s ability to grow has been vastly oversold to companies worldwide, according to a new study. In a first-of-its-kind academic analysis of the metric known as Net Promoter, researchers have discovered that claims of the measure’s superiority in predicting growth are false. The findings have significant implications for companies that use the metric – including GE, Allianz, Schwab, Intuit and American Express – to manage their businesses, with some even citing it to analysts and investors.
In a forthcoming study in the Journal of Marketing, a team of researchers from the Vanderbilt Owen Graduate School of Management, Norwegian School of Management, Koç University and global market research firm IPSOS have replicated the methodologies and even recreated original data used to support Net Promoter. The investigation found that Net Promoter was actually, in many cases, a less accurate predictor of growth than other common customer satisfaction or loyalty metrics. “Based on the evidence we’ve compiled, it’s hard to imagine a scenario where Net Promoter would be classified as the superior metric,” said Tim Keiningham, Senior Vice President at IPSOS Loyalty and co-author of the study.
The concept of Net Promoter, developed and advocated by noted loyalty consultant Fred Reichheld, is based on the principle that growth rates of firms within an industry can be accurately predicted based on a “score” tabulated from whether customers would recommend the company to someone else. The score reflects a ratio of “promoters” (those who would recommend) to “detractors” (those who would not recommend), and has gained widespread adoption by leading companies worldwide since its debut in 2003. This is despite the fact that the evidence in support of Net Promoter has never been subjected to rigorous scientific scrutiny or peer review and, to date, no researchers have attempted to replicate the research methodologies used by Reichheld.
Putting Net Promoter to the test In the study, a team of industry and academic experts in loyalty, statistical analysis and modeling conducted two experiments to test the effectiveness of Net Promoter. In the first, they used a comparable dataset representing over 15,000 consumers and 21 firms for five industries from the Norwegian Customer Satisfaction Barometer (NCSB) to calculate each firm’s Net Promoter score using Reichheld’s own methodology. After calculating and comparing to several other leading customer satisfaction and loyalty metrics and analyzing the links to actual revenue growth performance, the findings indicate no significant correlation between any of the satisfaction metrics – including Net Promoter – and relative changes in firm revenue in any of the industries analyzed, including banking, gas stations, home furnishings retailers, security systems and transportation.
In the second experiment, researchers replicated a subset of Reichheld’s own data for the three different U.S. industries – airlines, personal computers, and life insurance – he cites as exemplars of Net Promoter in his best-selling book The Ultimate Question. “We painstakingly replicated the scatter-plot diagrams for these three industries so that we could directly compare the relative performance of Net Promoter with that of the American Customer Satisfaction Index (ASCI),” said Prof. Bruce Cooil of the Vanderbilt Owen Graduate School of Management, a co-author of the study and an expert in statistics.
The researchers compared Reichheld’s recreated data with that from the ACSI, a national economic indicator of customer evaluations of the quality of products and services available to household consumers in the U.S. Reichheld specifically states that their examination of the ACSI showed that it had no correlation whatsoever with firm growth. Despite claims of Net Promoter’s superiority, however, the analysis actually netted strikingly similar predictive results (see example) between Net Promoter and the ACSI.
According to Cooil, “When making an apples-to-apples comparison using Reichheld’s own data, the ACSI actually showed a stronger relationship to growth in two of the three cases. And in all cases, the similarity between the Net Promoter-ACSI charts was striking. We had expected dramatically different results, given that these industries were presented as prime examples of the relationship between Net Promoter and growth, and the ACSI was specifically singled out as having no connection to a firm’s growth.”
Significant implications for companies
In addition to demonstrating the need for rigorous analysis before widespread adoption of such predictive metrics, the study’s findings have significant implications for companies and managers that have adopted Net Promoter. “Companies that utilize Net Promoter as a management tool have the potential for misallocation of company resources to manage firm performance, company value and shareholder wealth,” Keiningham said.
If assessing true loyalty among customers and its impact is the goal, then a more accurate metric would consider both customers’ feelings toward a brand or firm and customers’ actual buying behavior, according to Keiningham. “There are many factors that influence why people buy,” he said, “and there is no silver bullet for predicting growth based on customer loyalty, despite Net Promoter’s claims to the contrary.”
A summary of the research findings is available for download at www.LoyaltyMyths.com, and the study is forthcoming in the Journal of Marketing. The research team, in addition to Keiningham and Cooil, also included Prof. Tor Wallin Andreassen of the Norwegian School of Management and Prof. Lerzan Aksoy of Koç University in Turkey.
