It’s not quite as big as a phonebook, but, for some reason (lack of sleep? excessive doses of Sudafed?), this line from The Jerk popped into my head when I opened the mailbox this morning and saw the Internet Retailer Top 400 Guide stuffed in there. I referred to this report in a previous post; particularly the reference to a customer satisfaction survey conducted by ForeSee. As promised, I have taken a closer look at the methodology of the study and the claims made about the results. (As an aside, the study occupies the last 4 of the report’s 224 pages. The remainder of the tome is dedicated to a directory-style listing of the top 400 online retailers, ranked by 2004 revenue, along with various charts and graphs sorting this list by industry, prior year’s results, etc. So, no one should mistake this for a report on e-commerce customer satisfaction.)

I continue to be irritated by the inclusion of Netflix - with their subscription business model - among 39 other straight retail players. It wouldn’t be so bad, if Netflix hadn’t come in at #1 on the list, beating out Amazon by just one customer satisfaction index point. (Aside #2: Honestly, I was a little surprised that Amazon came in so high. So kill me, I just don’t think that Amazon’s customer experience is that great these days.) Apparently, ForeSee surveyed 11,000 people about their recent online retail experiences. Survey questions covered “content, functionality, look and feel, navigation, product browsing, product information, search, site performance, merchandise and price.” In theory this should have produced some compelling data. But, as hard as I try, I just can’t find anything interesting here. I keep getting distracted by statements like:

“Most e-retailers know customer satisfaction is important but very few can accurately quantify the contribution their web site is making to their overall financial success.”

Huh? I think this would be news to the army of business analysts, usability specialists, and researchers who work at these 40 companies. And this insight:

“…[B]rowsers are not only influenced by the site experience itself. They are also influenced by things like price, merchandise selection, and other factors in the retail environment… How the mix of satisfaction elements affects browsers is unique to each company’s web site and must be evaluated on a case-by-case basis.”

Did I need to spend $50 bucks to find this out? I would have at least appreciated a more detailed breakdown of the results. I know this is a (forgivable) attempt to showcase the power of their survey methodology and thereby attract new clients; and these guys are clearly experts at what they do. But generalizing “customer satisfaction” to a single numerical score is just not useful — although it may be extremely press-worthy. In my opinion, the combination of design, technology, and merchandising factors makes it very difficult to really learn anything from these results.

NG

This entry was posted on Friday, June 17th, 2005 at 4:20 pm and is filed under E-commerce, Usability Research.