TOP 10 INSIGHTS

Online Surveys & Usability Testing, San Francisco, June 19-21, 2002

1. Online research’s short, bizarre history is full of irony, humor and hope, at least to David Pring, CLT Research, the opening speaker. He described cyber-research as sharing in both "e-phoria" and the decline of the digital revolution. He derided tools we have to use as potholes in the information superhighway, saying if PCs were cars, streets would be littered from all the crashes. He read off names of old computers and referred to their demise as digital Darwinism. The early exponential growth in online research has slowed down everywhere. Internet-only research companies are in trouble. Three web-based research procurement portals, where clients could post RFPs and watch bids from hopeful suppliers come in, have all closed down. Still, online research is growing an impressive 50% a year. It’s maturing, and now, it’s just one more way to conduct a survey. Others added details like an Acorn study for IIR showing 75% of research decision makers in the U.S. have now tried online research and expect to use it again in the future. And Information Week reported this year P&G will conduct 50% of its U.S. consumer surveys online.

2. Taco Bell’s use also shows online research has reached the mainstream. Tanya Hovanesian of Taco Bell and Donna Wydra of Market Facts described how a firm that might be considered the epitome of low-tech has benefited from moving a significant portion of its survey work online during the past year. They found the online segment consumes a lot of tacos. They use an online panel to provide coverage of less frequent customers in concept tests to evaluate the breadth of appeal of new items for their menu. It is often supplemented by in store research on heavier consumers.

3. Online research shown to be a natural for testing the usability of web sites. Bill MacElroy, Socratic Technologies’ CEO, did this by describing research his firm conducted on Carrier’s web-based information systems. Starting with focus groups to understand why consumers were using the site, they used one-on-one lab testing to see how they use the site, and finally remote tracking of people as they use the site at work and at home. Changes were made at each stage and the iterative process led to improvements in usefulness, satisfaction, speed and the main objective: willingness to use the site as a substitute for printed and human information sources. Whitney Quesenbery of Cognetics described an approach rating site usability in five areas: effective, efficient, engaging, error tolerant and easy to learn.

4. Speakers agreed on the benefits of online research. Although there was debate about online surveys being better and cheaper, there was widespread agreement on 4 key advantages:

You can show things to respondents. A number mentioned graphics make online surveys more interesting, and concept tests more realistic. Dupont used online picture sorting to learn how to get across a visual impression of "freshness". I described how our firm shows pictures from commercials to provide the more accurate form of ad tracking based on recognition, at a relatively low cost. It can be overdone. Socratic found engineers looking for information complained about too many pictures.

Larger samples add little to the cost. One benefit of sample size was mentioned in the Mondavi case history. There is no need to prescreen for qualified respondents. Everyone takes the whole survey. You can decide later which segments of the sample provide the most relevant results.

It’s more convenient for the respondent. They can reply when they have the time.

Results are available fast. It appears almost everyone provides initial results within 24 hours.

Stagger mailing out invitations. Both C&R Research and our firm mentioned this one. Send out too many invitations at once and those who reply immediately are likely to jam up any system.

Combine research methods. Several found hybrid methods best. Condé Nast only had home addresses of subscribers to their Spanish language magazines so they mailed invitations to take their online survey. Results from calling first and asking if the respondent would prefer answering online or by other means were reported by Fidelity’s John Allison and in my talk. All three approaches worked.

Keep subject matter interesting. Mondavi conducted label and pricing studies by surveying people who left their name while surfing wine sites. Our firm surveys all types of people about Super Bowl commercials. Both benefit greatly from subject matter of greater than usual interest to respondents.

Avoid overused panel members. Rich Nadler, founder of Perseus, the popular online survey software, after reviewing a matrix for fitting a firm’s survey requirements to software capabilities, was the first of several to express concern about panelists taking too many surveys and becoming professional respondents. What to do about it was one of the first questions asked of a panel of experts assembled on Day Two. There was talk about big panels being better, and limits various panels placed on the number of surveys panelists could take per year. But the basic poverty of their answers to this fundamental question was highlighted by the last "expert" to comment on the issue: "Is a little professionalism really a problem?"

PostMasterDirect.com was mentioned by the Mondavi researchers as an excellent source for online wine drinkers at 10¢ to 15¢ a name where 25%-30% would accept the invitation and start the survey.

SSI Lite was mentioned by several as a key source for non-panel samples. When asked, they will assume the response rate risk and send out invitations until a fixed number of replies are received.

Panels were discussed in the Mondavi, BRC and Oxygen cases as samples where a professional sample-provider will assume the risk of predicting response rates. Balancing the panel demo-graphically makes it less of a convenience sample. Negatives are the cost and the overuse of panel members described previously. SSI and Greenfield were frequently mentioned as sources open to all.

Bruzzone Research Company · 2515 Santa Clara Avenue · Alameda, CA 94501-4692 · (510) 523-5505 www.Bruzzone-Research.com