Online Research - Not Quite The Holy Grail

By David Kay

There is as much buzz about online research as there is about e-business/e-commerce, at least among market researchers. First, everyone is doing it. And they should - but not always.

The benefits of online research are obvious — cost savings above all and the ability under some circumstances — to reach difficult to find populations, such as university students and some professional groups. Because the cost is not driven by sample size, large samples can be obtained for less money.

The downside is the reliability or unreliability of the sample.

Market research requires samples that accurately represent the populations being surveyed. Obtaining survey results from a sample that doesn't represent the target population is like asking people who never owned a dog to discuss the merits of different brands of dog food. A pointless, misleading exercise, yielding results that are questionable at best. A proper sample is a random sample; best achieved through random dial telephone surveys. When this method is not possible, for example, when products have to be shown in a mall, then interviewers can screen out obviously inappropriate respondents. If we are looking for women age 30-50, interviewers can avoid asking questions of teen-age males. It's not perfect, but it is far better than allowing anyone to answer the questions regardless of qualifications, which often happens with online surveys.

The most trustworthy online surveys are those where the sample is known and controlled. For example, a survey can be conducted among e-customers, where the entire e-customer list is known, and where each potential respondent is e-mailed an invitation to take part in a password protected online survey. This is still not perfect, because the sample is self-selected. There is no interviewer begging the respondent to please take part in the survey. But every respondent qualifies, and this satisfies most researchers. This type of online survey research will grow because partnership relationships between suppliers and customers are growing and because suppliers will want more and more feedback from their customers.

Recruiting the sample by traditional means, such as by telephone and having the survey completed online is also useful. The cost benefit is partially lost, but this hybrid research will also become more common because it is convenient for the respondents. Hybrid research, in various iterations may become the most common and the most effective form of e-research.

In an acceptable variation, Research Dimensions is about to conduct a university student survey online. We know that the entire universe qualifies, since virtually all of the students use the Internet. The survey will be very well publicised on campus, and only on campus, so that we are reasonably assured that respondents will be qualified university students.

Internet research panels are also commonly used. Research companies collect names and addresses of Internet users who agree to take part in Internet surveys, usually to win prizes. These panel members are then invited to take part in various surveys. The usefulness of the approach depends on the reliability of the panel, i.e. how it is recruited and how it is controlled. Some are excellent. But in others, the "fool-around" factor can be overwhelming. Out of boredom, mischief or curiosity, inappropriate respondents complete the surveys, sometimes several at a time. For example, teen-age boys can answer surveys on panty-liners or gin for the heck of it. All they have to do is click the answers. It is too easy to cheat and too tempting, especially when prizes are offered.

Pop up surveys are also becoming common, in which visitors to a site are asked questions - during their visit. Or a button is placed on the web site inviting visitors to take a survey. We don't really know who the respondents are, but at least we know that they looked at the site. The same sample issues apply. Research Dimensions has been conducting web design surveys using this method. The don't know who the respondents are we don't charge for the service. I assume other research companies do the same.

What is the future of online research? In the short term it will be divided between good research, such as surveys b y e-mail invitation, and poor research, where any misqualified respondent can take part. In the long term, we will overcome the difficulties of sampling and the field will blossom.