Special Report
Before you rely on an online ratings service, get to know the source — and how it makes money
When you’re shopping online, whom can you trust to give you the best shopping experience? That’s the quandary facing consumers and small-business owners alike who are buying goods from unfamiliar Web stores. And with so many online companies vying for your hard-earned dollars — not to mention the fact that so many of them are in questionable financial condition — it’s natural to want to look to a trusted third party for a recommendation. But before you rely on an online ratings service like BizRate.com or Gomez.com, you’d better read the fine print.
Consider the experience of Jehremy Foster, who was in the market for a camcorder this past December. He thought the camera would make a good Christmas present for his wife, who was then six months pregnant. An experienced Web shopper, Foster, 30, browsed consumer comments in an online camera forum until he settled on the model that seemed best for his soon-to-be-growing family: a Canon ZR10.
Foster’s next stop: MySimon.com’s comparison-shopping service, where he sorted by price a listing of online merchants that sold the Canon camcorder. Noting that the lowest-priced vendors were listed at the top, Foster scanned down the list until he found vendors that rated at least two out of three possible stars on Gomez.com, an online ratings service. Family Photo and Video — a company that Foster had never heard of — had a Gomez rating of two stars and was selling the camcorder at the then-attractive price of $697. The prices other vendors listed for the same item ranged from $735 to $1,069.
Foster says that he then searched for Family Photo and Video on BizRate .com, which rates 2,000 online merchants based on customer-survey information. The photo store rated an 8.3 out of 10, which sounded pretty good to Foster, so he placed the order online and crossed that item off his gift list.
But the next day, Foster says, he got a call from a salesperson at Family Photo and Video that he found disturbing. The caller was pushing him to buy accessories and extra batteries for the camcorder he had just ordered. “It was totally bizarre,” says Foster. “Once the salesman realized I wasn’t going to buy anything extra, he was very rude and short with me.” (Family Photo and Video customer-service supervisor Mike Tate responds that sales reps follow up with customers primarily to confirm orders that might be bogus, and to offer promotional discounts. “There’s no pressure to buy,” he says, noting that Family Photo’s salespeople don’t work on commission.)
Unsettled, Foster tracked down Family Photo and Video’s off-line address. When he found out that the business was located in New York City, he logged on to the site for the Better Business Bureau of Metropolitan New York. There, to his mounting dismay, he found numerous complaints about Family Photo and Video — although the store was listed under another name, Abe’s of Maine.
A general Web search for Abe’s of Maine led Foster to Photo.net, an online community of amateur and professional photographers, where he discovered some 70 complaints from people who said they had been burned by Abe’s of Maine. The complaints alleged everything from getting billed for $400 shipping charges to the old bait-and-switch routine. (A favorite ploy of certain camera merchants is to say a hot camera model is in stock and then ship a different model.) Foster wondered how Family Photo and Video had managed to get such seemingly favorable ratings from both BizRate.com and Gomez.com.
Foster’s unfortunate experience notwithstanding, Tate says the ratings accurately reflect his company’s level of service. “We have a toll-free number, and whatever the problem, we will handle it,” says Tate. More than half of Family Photo and Video’s business comes from repeat customers, he adds. And given the site’s typical daily volume of about 200 orders, he maintains that receiving roughly 70 negative remarks on Photo.net during what was actually a five-year period isn’t a bad record at all.
Still, Jehremy Foster’s story begs the question: How is it that the online ratings systems don’t reflect the bad experiences of people like Foster and visitors who post messages on Photo.net?
The short answer is that ratings services are designed to spotlight good online merchants, not to warn consumers about bad ones. After all, the vast majority of online ratings sites are operated by companies that need to make money in order to keep providing their services. Most of the companies receive some part of their revenues from the merchants they are rating. Such a relationship between the ratings services and the companies they report on presents at least the appearance of compromising the quality of those ratings.
San Francisco-based CNET Networks has considered posting consumer feedback regarding online merchants but has so far steered clear. “We don’t have the ability to verify whether a particular poster is truly a buyer from that merchant or whether it’s another merchant looking to slam their competitor,” says Dan Miller, director of CNET’s Shopper.com, a comparison-shopping service. CNET tracks feedback from consumers about resellers but doesn’t publish the individual comments. The feedback is offered to the merchants, and Miller says that most consumer complaints reported to CNET are resolved within two days. To help authenticate the complaints, consumers are asked to submit a customer-order number with a complaint about a vendor. In the future, Miller hopes to add to CNET’s merchant information the number of complaints the site has received about each vendor, as well as how many of those complaints have been satisfactorily resolved.
Ratings services are designed to spotlight good online merchants, not to warn consumers about bad ones.
If the situation is complicated for ratings purveyors, it’s even more befuddling for the average consumer, who’s likely to simply take the ratings at face value. But the fact is, using ratings sites effectively requires knowledge of the raters’ business model and of the degree to which the company’s economic interest could skew the way the information is presented.
Without that knowledge, consumers might find themselves in trouble, especially, it seems, when they’re in the market for camera equipment. In May 2000 a computer-industry worker in Benicia, Calif. (who asked to be known only by her online alias, Cneber), ordered an Olympus C-3030 Zoom camera from Best Stop Digital, an online merchant she had found listed on CNET. Best Stop Digital’s site said that the highly in-demand camera was in stock at the attractive price of $850, while other merchants were listing it about $200 higher.
A few days later, when Cneber hadn’t received the camera but found that her credit card had been charged for the purchase, she called Best Stop Digital to ask for a tracking number. She learned that the camera she’d ordered wasn’t in stock and probably wouldn’t ship for several weeks. When she asked the customer-service representative to cancel the order, she was put on hold and then disconnected. Further calls to Best Stop Digital were transferred, put on hold, or disconnected, and Cneber says that eventually no one answered the phone at all.
Cneber, whose account is corroborated by documents and correspondence she shared with Inc. Technology, then sent an E-mail message to CNET to alert its product reviewers that a questionable merchant was included in its listings. She also recommended that CNET drop Best Stop Digital from its vendor list. She says that CNET responded with a message that the company made no guarantees regarding the quality of merchants that were listed on its site but not “CNET certified.”
CNET’s Miller says he’s sorry about Cneber’s experience and confirms that CNET had received earlier complaints about Best Stop Digital. Miller says he suggested to Cneber that she contact her credit-card company for help with the charge. (Her account was credited for the amount of the purchase a few weeks later.) In this case, Miller says, CNET couldn’t mediate between the company and the consumer, because CNET’s best information indicated that Best Stop Digital had gone out of business and had been dropped from CNET’s ratings. (Best Stop Digital is, in fact, still in business, but the company didn’t respond to Inc. Technology‘s repeated requests for comment.)
Miller says that CNET is well aware of cyberspace’s bad apples, who “are hurting the credibility of online retailers everywhere,” he says — not to mention the credibility of online information sites. To be a CNET-certified vendor, a merchant must supply CNET with up-to-date information on its pricing, shipping and handling, and inventory, as well as its site’s customer-service policy. The company must also honor the prices advertised on its Web site, conduct its transactions on a secure server, process orders promptly, deliver merchandise in professional packaging, and respond to all E-mail messages and phone calls from customers within two business days. According to Miller, CNET recently added a “code of conduct” that applies to all its listed vendors, whether they are CNET certified or not. The code prohibits certain bad practices, such as making orders contingent on the purchase of additional materials.
CNET is also aware of the complaints about formerly CNET-certified merchant Family Photo and Video/Abe’s of Maine that are posted on Photo.net. “We can’t verify those complaints. We can only act on what we know,” explains Erik Kokkonen, a CNET vice-president. If a consumer complains to CNET about one of its vendors, a CNET staffer will contact the vendor and give it a certain amount of time to fix the situation. If the merchant is repeatedly unresponsive, CNET will kick it off the list for 5 days for the first serious offense and for 15 days for the second serious offense. If there’s a third offense, the merchant’s contract with CNET as a certified vendor is terminated, but the company can still remain on the site as a regular listed vendor. “We are in the business of bringing buyers and sellers together,” says Kokkonen. “But the reality is, we have a responsibility to the sellers to make sure we’re treating them fairly as well.”
Where to Turn? Cameras and electronics are product categories that are especially rife with consumer complaints, especially with the prevalence of so-called “gray market” goods (“off-brand” versions of products that manufacturers produce to sell inexpensively abroad without warranties) that make their way into the U.S. market. But consumers need to be aware of the economic imperatives of the ratings services, no matter what kinds of goods they are looking to buy. Performing that level of research can be a big job, since there are so many different kinds of online services that aim to improve the Web shopping experience. (See “The Top Ratings and Comparison-Shopping Sites at a Glance,” below.)
BizRate.com is one of the most visible comparison-shopping and vendor-rating sites on the Net. The site’s ratings section ranks 2,000 “partner” merchants on a scale of 1 to 10 based on data from consumer surveys. Using 10 shopping criteria, BizRate.com surveys shoppers, once at the point of sale and then again after goods have been shipped, in order to get feedback on the buyers’ fulfillment experience. BizRate.com calls its partner merchants “Gold Stores,” although that somewhat lofty designation may often be misleading, as it would seem to imply that the stores offer unusually high-quality products or service.
According to BizRate.com president and CEO Chuck Davis, the company, located in Marina Del Ray, Calif., aims to include all online merchants in its surveys. It doesn’t charge vendors for a basic listing. However, vendors can pay for “preferred” placement in the marketplace section, which means that the merchant’s listing will appear at the top of any consumer search of that section. BizRate.com provides a daily feedback service to Gold Store merchants and sells additional detailed reports for about $3,000 a month. BizRate.com also makes money if buyers click through to the seller’s site to make a purchase. “We do no rating ourselves,” says Davis. “All we do is set up the infrastructure for unbiased buyer ratings.”
BizRate.com’s ratings may seem free of commercial taint, since companies can’t pay to influence the surveys, but there is still room for murkiness. For instance, a merchant’s rating is based half on its front-end store and half on its back-end fulfillment, so stores with questionable practices can still receive a rating of 5.0 and above if they merely have a nice storefront.
“It can be difficult to tell exactly what the numbers mean,” says Preston Gralla, author of The Complete Idiot’s Guide to Online Shopping. “What’s an 8.4 versus a 9.5? Those numbers might be based on criteria that aren’t important to you.” Gralla advises that shoppers take advantage of a BizRate.com feature that allows consumers to rank merchants according to specific criteria, such as on-time delivery. “If a merchant has 91% on-time delivery, that means something,” says Gralla.
On Gomez.com, another online ratings site, it’s especially difficult for consumers to assess whether a merchant’s score reflects variables that are important to them, and for a very good reason: Gomez.com doesn’t disclose all of the details about the criteria it uses to judge sites. “That would be like Coke giving away its special formula,” says Julio Gomez, chairman and CEO of Gomez Inc., in Waltham, Mass. “For each of the 75 industries that we cover, we have about a dozen criteria that we use to judge each site.” Gomez engages 30 industry analysts to generate the ratings rather than relying on consumers. (Gomez.com does publish basic information about its overall methodology, however.) In addition to running its own ratings site, Gomez.com provides merchant-ratings information to online comparison-shopping and consumer-information sites like MySimon.com, DealTime.com, and CNET.com.
What online shoppers who come across a Gomez.com rating may not realize is that the Gomez score doesn’t take into account any aspect of the fulfillment experience. According to Julio Gomez, sites are scored only on the quality of the online experience they present to shoppers and on whether they have certain policies, such as accepting returned merchandise. So under Gomez.com’s system, it’s possible for a merchant with a slick online presence but questionable or even shabby business practices to receive a rating of two or even three stars out of three possible stars, as Jehremy Foster discovered when he encountered Family Photo and Video’s two-star Gomez rating.
CEO Gomez answers such criticism by saying that his company concentrates on rating the consumer’s online experience, period. “We’re very focused on what we do. We are just about the customer experience on the Internet,” he says. His company doesn’t rate a merchant’s fulfillment performance, he says, because it wouldn’t be possible for his analysts to order products from all the 6,500 online merchants that are Gomez.com certified. “If someone out there would like to do that, they’re welcome to it,” he says.
As it happens, Julio Gomez’s former employer, Forrester Research Inc., a Cambridge, Mass., market-research firm, appears to be doing just that. Forrester offers a free online consumer service called PowerRankings, which combines extensive consumer-survey data with expert analysis to rank the top E-commerce sites on the Web. According to Tom Rhinelander, research director at Forrester, no company can pay to get on (or off) the PowerRankings list.
Here’s how the ranking system works: Forrester commissions a survey provider and research house (currently the NPD Group) to poll 20,000 customers of online E-tail and travel sites. From that data, Forrester analysts identify E-commerce sites that contain enough consumer data to justify inclusion in the PowerRankings. Since its resources are limited, Forrester ranks no more than 10 companies in each of the 13 industries that its rankings cover, which means that shoppers are unlikely to find unfamiliar merchants on the list. Forrester’s analysts open accounts with, and shop at, the sites in question, placing real orders for products; they then return all the merchandise.
Based on those responses, Forrester prepares the rankings, giving the consumer-opinion data two-thirds weight and the in-house expert opinion one-third. “We believe the consumer data should carry more weight than expert opinion. That’s part of our methodology,” Rhinelander says.
The PowerRankings list isn’t wholly free of the whiff of commercialism, since some of the companies on the list are Forrester’s clients. But Rhinelander says the company’s clients aren’t able to influence the ratings, even if they want to. Clients sometimes “call up and moan about the rankings,” he says. “The only way that will get any traction is if we get something wrong, and that can happen.”
For consumers who are looking for a beacon of brand-name credibility on the Internet, there’s always the Reliability Seal of the Better Business Bureau OnLine (BBBOnLine). BBBOnLine is drawing on the bureau’s nearly 90 years of off-line credibility, making it a name to believe in, if not blindly trust. Consumers can search BBBOnLine to see if a particular merchant has received the organization’s certification. In order to achieve the Reliability Seal, online businesses must have been in business for at least a year and be members of their local bureau.
The difference between BBBOnLine and other ratings services is that the companies that get the Reliability Seal must have already met certain qualifications. With straight sponsorship sites, virtually any company can pay to be listed. BBBOnLine has credibility because local bureau chapters are well situated to vouch for the legitimacy of businesses in their areas, according to Steve Salter, director of operations and administration for BBBOnLine, in Arlington, Va. Usually, local bureaus make an in-person visit to the companies that apply for membership, he says, which weeds out many of the worst types of operations.
Checking with the Better Business Bureau sooner might have saved Jehremy Foster some headaches, although in the end he did receive the camera that he had ordered from Family Photo and Video (a.k.a. Abe’s of Maine). But Foster says his faith has been shaken. “I don’t trust any of the online ratings systems anymore,” he adds. “From now on, I’m going to either have a personal recommendation or stick with a known entity.”
Lauren Gibbons Paul is a freelance writer based in Waban, Mass.
The Top Ratings and Comparison-Shopping Sites at a Glance
BBBOnLine What it is A service for certifying online merchants’ reliability and security and for finding out about consumer complaints lodged against merchants.
How it works Consumers can search the site for companies that have received the BBBOnLine Reliability and Privacy Seals. Consumers can also file a complaint against a vendor online.
How it makes money Like regional BBBs, BBBOnline charges companies membership dues that range from $100 to $10,000 annually (depending on the member company’s size).
What you may not know BBBOnline doesn’t yet offer complete merchant data from all its 130 member organizations, so link to the bureau’s regional sites to investigate companies.
BizRate.com What it is An online comparison-shopping and merchant-ratings service based on consumer-survey information.
How it works Consumers can search for particular products in a variety of categories and see the rankings of merchants that sell the products online. In the Store Rating section, users can search for a particular store name and read detailed profiles.
How it makes money Merchants pay BizRate.com to advertise their promotions. BizRate.com collects referral fees from merchants that get business off its site. BizRate.com also sells companies detailed market research culled from user surveys.
What you may not know Although BizRate.com sites are ranked on a scale of 1 to 10, consumers are unlikely to find any site with a rating lower than 5. The designation “Gold Store” on BizRate.com means only that the merchant has agreed to put BizRate.com’s surveys on its site.
CNET What it is An online news, reviews, and comparison-shopping portal.
How it works Consumers can search CNET for product reviews and price comparisons. Consumers can also view comments from other shoppers in unmonitored message boards regarding a particular product or vendor.
How it makes money CNET sells banner ads and also receives a referral fee from vendors when users go to their sites. All merchants in CNET Networks’ price listings pay to be listed. Some merchants pay a premium for order of placement or prominence on the list.
What you may not know CNET makes no guarantees regarding the quality of merchants that are listed on the site but aren’t “CNET certified,” although it will investigate complaints about any listed vendor.
Gomez.com What it is An online merchant-ratings service based on expert analysis.
How it works Users can view ratings of the top vendors for the quality of the online shopping experience in a number of categories.
How it makes money Gomez.com receives fees when consumers purchase products from certified vendors. It also sells consulting and in-depth reports to vendors.
What you may not know Gomez’s ratings don’t take into account the consumer’s experience after an order has been placed. So the ratings don’t reflect either order fulfillment or customer service.
MySimon.com What it is A comparison-shopping service.
How it works MySimon.com allows consumers to compare prices on items offered by online vendors. It also lists Gomez.com’s ratings information for some vendors.
How it makes money MySimon.com sells advertising to vendors.
What you may not know MySimon.com aims to list as many merchants as possible that offer a particular product, but it lets vendors pay for preferential placement on its lists.
PowerRankings of Forrester Research Inc. What it is An online merchant-ratings system based on consumer-survey data and expert analysis.
How it works Visitors can view rankings of 10 merchants in 13 categories that include apparel, toys, banks, and electronics.
How it makes money Forrester makes no revenues associated with its PowerRankings list. The company makes money through its primary business — market research and consulting.
What you may not know The PowerRankings list rates no more than 10 companies in any single category.
Please e-mail your comments to editors@inc.com.
