Online focus groups come in a poor 2nd to telephone focus groups for hard-to-get, dispersed respondents
By George Silverman
President and Founder
Market Navigation, Inc.
What should online focus groups be compared with?
Online focus groups are usually compared favorably with face-to-face focus groups. I believe that this comparison is misleading: The main idea of online focus groups is to bring together widely dispersed and other hard-to-recruit people. For this reason, online focus groups should be compared to telephone focus groups, the other methodology designed for hard-to-recruit and/or geographically dispersed people.
Below is a comparison of all three, on all of the important variables that I could find. The “winner” for each has the green background and bold type (for printers that cannot shade the background).
|Issue|| Face to Face
||Online Focus Groups|
|Rationale||When the richness of group interaction is desired with people who can be brought into a central location.||When the richness of group interaction is desired with people who cannot easily be brought together face to face (geographically dispersed, hard to recruit, low incidence) or who you want in their own natural environment. Higher degree of openness makes this the preferred medium for remote groups.||Same as telephone focus groups, plus people you want to be online during the group.|
|First started||In the 1950’s||First viable in 1969.||Unknown, but probably as a research methodology in the early 1980’s. Didn’t become widespread until the mid- to late- 90’s.|
|Volume of groups||All estimates extremely unreliable, but seems to be upwards of 10,000 per year.||A couple of thousand a year.||No idea, but rapidly growing.|
|Acceptance of methodology||Face to face almost totally accepted as a valid qualitative methodology.||The standard methodology in some industries, such as pharmaceutical and agriculture, increasingly in financial services and others.I have conducted over 7000 telephone groups in the last 30 years. Telephone still not well accepted, or even heard of, in many industries.||Mostly used in high-tech applications. Rejected by most other industries, but acceptance growing.|
|Richest expression, greatest cues for interpretation. (This is probably the most important issue)||Here, face-to-face groups have the clear advantage. You can see body language and facial expressions. More modes to express, more data to interpret.||
Can’t see body language or facial expressions, but most are translated into the tremendous richness of the human voice. When people are on the phone, their voices become even more expressive because they know subconsciously that they can’t be seen.
Think about the voice, with its ironic sarcasm, slight hesitancies in answers (how do you spot a person typing hesitantly?), chuckling that is unintentionally expressed on the phone but not expressed in text unless the person wants to call attention to it, strong vs. meek answers (in other words what would be in bold type if one could use it), tentative vs. energized ideas, speed, inflection, tone, strength, wavering, stress etc.)
|Onlne is o distant third. All of this is lost in online text only groups. Emoticons e.g., 🙂 or :-(, don’t even begin to express the broad range of information, particularly emotion, that can be heard in the voice.|
|Visual element.||Has visual element.||Non Visual, but has the richness of the human voice. If this is the only way to get the participants, the lack of visual is not a high price to pay.||Non visual.|
|Stimulus materials||A virtually unlimited range of stimulus materials are possible||Stimulus materials can be overnighted, faxed or presented on the Web.||May look different in different browsers.|
|Backroom interaction||Easiest to interact behind the one-way mirror. Sometimes it’s too easy for the clients to interact and ignore the group.||Both telephone and online groups have virtual back rooms. There are no M&Ms available, but your favorite drinks are only as far away as your own refrigerator|
|Following the thread of the conversation||Not a problem.||Not a problem.`||May be difficult to tell what respondents are responding to online, since they are often typing in parallel.|
|Amount of information. Number of words per hour.||About the same for F2F and phone.||About 1/3 less words per unit time for online.|
|Respondents can get their thoughts out without interruption in online..||You want interaction, not thoughts running on without interruption. It’s the very purpose of a focus group.||People in online groups are responding much more in isolation and in parallel than they are in telephone or face-to-face groups.|
|Ability to stick to the guide.||Allows greater flexibility, at least the way they are presently conducted.||Easiest to stick with guide but this is not necessarily an advantage. Present practice for online focus groups is often to send the guide in advance, or build it into the software. This is usually bad practice because it will tend to bias the responses. How? Good moderators have a strong sense of the priorities. They will deeply probe areas in which new ideas are emerging, with a variety of techniques, some of which are pulled out of a vast repertoire on the spot when unanticipated issues come to light. This is usually done in a subtle way so that the participants do not know that they have just said something very important. If they knew, it would be given undue prominence and distort the group dynamics. There is nothing inherent in doing online groups that requires this practice.|
|Technology bias||None.||None.||One person’s bias is another ones sample. If you are selecting for technology sophistication, it is not a bias. If you are looking for a full range of technology attitudes represented, this method is extremely biased.|
|Rational, impersonal responses vs. full range of human interaction: emotional, cognitive, etc.||People tend to be somewhat inhibited, but get over it with expert moderation. Tremendous richness of expression is possible.||People get extremely emotional and personal on the phone, since the anonymity, lack of visual element and naturalness of talking on the phone creates a great deal of psychological safety..||
While people feel safe online for the same reasons they do on the telephone, they cannot express themselves as well using text as they can using their voices.
This essentially eliminates online groups for many sophisticated applications.
|More honest, open, outspoken, less swayed by group, less reticent to speak.||Participants somewhat inhibited.||The openness of people in telephone groups is legendary. The pull to participate, extraordinary. It is much harder to sit on the phone without talking than it is to sit at a computer without typing. People have compared the same groups of teenagers on the phone vs. face to face, and found that the teenagers were much more comfortable talking on the phone. The production was much higher, gender groups could be mixed, and the phone groups were much superior in many other ways. I’m not aware of any direct comparisons between telephone and online groups on these issues. However, the argument that young people are more comfortable on the computer than they are face to face may be true. But is anyone ready to argue that they are more comfortable on the computer than they are on the telephone?||No one has shown that online has any superiority to telephone on this cluster of attributes.|
|Whether people tend to tell the truth. Ability for moderator to detect lying.||High degree of visual cues makes it easier to detect lying, but people more tempted to please each other or the moderator, play eye contact games, etc.||Telephone has been validated in individual interviews as being the medium in which people tend to tell the truth more, but this has not been validated for phone groups.||Awaiting data.|
|Setting up the groups.||Hardest. People have to travel to attend, so you usually have to identify people in the local area (exception: f2f at conferences.)||Easiest. People can be anywhere, everyone has a phone. Groups can be worldwide, or as narrow is a single building. High-level people want to participate to talk with old friends or just hear what people have to say across the country/world.||Not everyone can type, has access to a computer or wants to participate via online computer with other people.|
|Acceptance rates||About half to a quarter of telephone.||Telephone groups lead online and F2F. Everyone is comfortable with the telephone, even people who have never participated on a conference call. People can participate from any phone: even those in hotel rooms, cars, etc. About 10 people in 50-100.||Acceptance rates are very low. Can be 20 times lower than telephone. At one meeting of the QRCA in New York they were quoted as typically 10 people in 2000 from a closed list provided by the client.|
|Show up rates||50-80%||90-100%||<50%|
|Ability to reach difficult-to-recruit participants.||Pretty poor. That’s mostly why telephone and online groups were invented!||Best, for reasons already stated.||Online better than f2f, but not nearly as good as telephone, judging by acceptance and show-up rates.|
|Opportunity for dominators to sabotage group.||Can be hard to control dominators. Hard to kick people out of group, without them getting a “message,” which fools nobody and can destroy the dynamics of the group.||Much easier to control dominators in telephone groups than in face-to-face groups because people are more easily interruptible on the phone. Can kick someone out of group privately and invisibly.||The fastest typist wins. 50 words per minute is a pretty respectable typing speed. However, people who use voice dictation can type at up to 160+ words per minute. Someone like that can overwhelm an entire group easily. Easy to kick someone out.|
|Turnaround for recruiting, executing and reporting on groups.||Usually much slower than telephone and online.||Both telephone and online are usually superior to face to face. Telephone probably has the edge because of its flexibility.||Online superior to F2F, not as good as telephone.|
|Bias Issues||Lower potential for bias than than online, higher than telephone. Validated long ago as being roughly representative enough to generalize (if not, why are you running the groups?), although not statistically representative.||Highest recruitment rates mean telephone is the most representative. Like F2F, validated long ago as being roughly representative enough to generalize, although not statistically representative||Low recruitment rates online provides a much greater opportunity for bias. 10 people out of 2000, if that continues to be the rate, is unacceptable to many, unless it is proved that these ten people are not unusual. Online needs validation. Jury still out.|
|Greater participation from the client team.||Difficult to get key people to participate in different cities.||Telephone has a slight edge here because it is easier to get access to a telephone than to a computer. If the client is stranded, he/she can dial in on a cell phone.||Computers aren’t quite as ubiquitous as phones, but they are getting there.|
|Evaluation of websites||Face to face is a distant third because people have to be given computers in a usability lab. These are not their own computers, so they are not set up optimally for the person using it. On the other hand, if you need to observe them, face to face is the way to go.||Telephone is superior. People participate in a telephone group while using their computer. A variety of software and website services are able to direct them to websites, PowerPoint presentations, software screens or anything else that can be viewed on a computer screen. This can be under the control of the moderator or the participants.||When this is done as part of an online group, it is simply too unwieldy to expect people to click through and view websites while also typing their responses.|
|Fast turnaround with equal cost||About the same price as online, about 10% more than telephone. Can be more if a lot of clients are travelling.||Telephone groups are about 10 percent cheaper that F2F and online.||Online groups are reported to be approximately the same cost as face to face.|
|Cost-effectiveness.||Least cost-effective.||Most cost-effective.||As of this writing, approximately the same cost as F2F.|
|Personal questions can be addressed, anonymity protected.||Anonymity lowest F2F. This can make people clam up.||Anonymity can be protected just as well on the phone as it can be online||Unfortunately, authenticity can also be masked much better online. Is the person who he/she says he/she is? Hardest to verify online, although there are abuses in all modes.|
|Sampling advantages, such as better mixture of people within the group, mixing heterogeneous and conflicting participants, and people who would be reluctant to participate together because they are competitors in a local geographical region.||These are notorious problems with face to face groups.||
Telephone groups have all of these advantages as well. Since the acceptance and show-up rates are higher for telephone, they probably has a more representative sample.
|Online has all of these advantages over face to face, but equal to telephone. But telephone usually has more representative sample.|
|Availability of the technology to the participants..||Traffic, bad weather, etc. can ruin show-up rate. Not everyone has a car available, ruling out non-drivers. This can be a problem with the elderly, infirm, some handicapped people, etc.||Everyone has a phone (many several). Worldwide: As long as they can get to a phone where they can be reached from the United States, they can be conferenced.||Not everyone has a computer. Not everyone is comfortable chatting on a computer.|
|Conversation flow||Usually natural, but can be stilted, easy to break into side conversations, or feel ignored.||Extremely smooth. conversation is extremely natural, with no side conversations possible. The illusion is created that the speaker is speaking to each person individually, because each person hears the speaker’s voice coming into his/her ear. each person feels like he/she is being spoken to personally. it’s extremely hard to talk while someone else is talking on the phone, but easy to interrupt.||If people are typing at the same time in parallel then trying to react to the burst of text, it’s not surprising that the flow is disjointed. Parallel play is disjointed by its very nature.|
|Possible recruiting bias to self-selected participants.||Usually does not use participant self-selection, but some facilities do. These should be avoided whenever possible.||Rarely uses participant self-selection because acceptance rates and show up rates are so high.||Often, participants are self-selected by signing up on a website. This can introduce a serious recruiting bias. Although this is common practice, it is not an inherent deficiency of the method because other recruitment procedures are available.|
|Catching cheaters, repeaters and impersonators.||Easy to verify identities face to face in the same facility, difficult when there are many facilities in the same city.||Because of the high recruitment and show up rates, usually uses a closed list supplied by clients. This greatly reduces cheaters, repeaters and phonies.||Online is extremely vulnerable to abuses.|
|Difficulty of getting in-depth information.||Face to face is widely acknowledged to be an extremely effective modality for getting in-depth information.||Telephone is even better.||Widely acknowledged, even by its practitioners, to be the least effective way of getting in-depth information.|
|Participation issues.||Show up rates are extremely unpredictable. Usually, after the warm up period, participants are extremely involved.||The modal show-up rate for telephone (the one with greatest frequency, for us non-statistical types), is 100%! It’s extremely rare for someone to either hang up or get disconnected. participants need virtually no warm up period. They get extremely involved.||Respondents can lose interest and drop out or get lost in the flow, especially in bulletin board forums. No-shows are high.|
|Group control issues.||Groups can get out of hand, with side conversations, everyone talking at once, etc.||Extremely easy to control.||Amount of text streaming may overwhelm moderator and respondents.|
|Skills needed to participate.||Usually need to drive (or have driver), speak clearly enough to participate, understand the language of the session.||Ability to use phone, speak and understand language of session. No other requirements.||Almost completely dependent on typing skills. Ability to follow simple log-in and participation instructions on a computer.|
|Novelty effect on recruitment rates.||The oldest method. Some populations are sick of going to facilities to participate in groups.||People at all levels get a kick out of participating in a nationwide group, and most have never participated in a phone group. Recruitment/show-up rate highest.||Also a novelty. One would expect higher acceptance rates.|
|Client novelty effect.||“Old” tried and true method.||Very easy to try. Only requires that the client dial into a phone number.||A little harder to try. Client must learn some simple protocols.|
|Use for older people||Hard for them to get to session.||Easiest for elderly people.||Most are not computer adept.|
|Travel time and expense.||Another reason why telephone and online were invented.||
Both telephone and online eliminate participant travel time and expense equally.
|Can be used to corroborate findings from in person groups that were done in only one or two markets.||You have to go to many locations in order to get a cross-section of the nation.||Both telephone and online groups can be conducted in as many locations as there are participants. So a group of 9 participants can be from 9 different towns, or even countries. Or, they can be as narrow as from one office building in Los Angeles.|
|‘Sensitive’ topics.||Not as easy to get people to open up on sensitive topics as telephone and online.||Both telephone and online are ideal to create the psychological safety for sensitive topics. In some ways, they are better than individual interviews because of the group support effect.|
|Participation on respondents’ schedule.||Impossible.||Impossible.||
Bulletin board or list group style allows this.
This is a truly exciting breakthrough. It is the only mode of running a focus group so far that not only repeals geography, but also repeals time. Theoretically, this could also be done on the telephone with a voice mail type system, but to my knowledge has never been tried. It is particularly suited for ongoing advisory groups, user groups, expert groups, especially groups that you want to conduct on a worldwide basis. So, if you wanted to have an ongoing advisory group of the world’s leading experts in something, this would be the method. This is a very limited, but very important application.
|Online groups easier to moderate||F2F slightly easier to moderate than telephone.||Telephone groups are probably the hardest to moderate.||Online groups are reported by same online moderators to be a little easier to moderate because they do not require the moderator to think on his/her feet quite as quickly, especially in asynchronous groups, although an online moderator does have to process a lot of information. This whole issue is probably a non issue for experienced moderators, but could be a considerable advantage for inexperienced moderators. Asynchronous groups might even be a good training method, since the moderator could discuss possible probes with a teacher before actually committing to an intervention.|
|Psychological safety of participants (respondents)||Lowest of the three. People are easily intimidated by the other people’s dress, manner, facial expressions, body language, etc. Even skilled moderators have to work hard to make people open up.||Almost as high as online. People can’t see each other, and they are from different parts of the country, so they tend to open up much more than in F2F.||Highest. People can’t even hear each others’ tone of voice. (There is a famous cartoon that shows a dog in front of a computer saying to another dog, “On the Internet, they don’t know you’re a dog.” Come to think of it, maybe it’s a disadvantage!)|
|Facility availability.||Very expensive and long lead times to build new focus group rooms. If they’re booked, you can use hotel rooms with the clients in the room, or watching though a monitor.||
Usually available, even on short notice.
Update: Phone focus groups have caught up to online with the proliferation of specialty conference call services, such as CiviCom and Premiere Global, who all have people who specialize in qualitative research.
|Online facilities were much more easily scalable then high-quality specialized telephone focus group facilities.|
|Immediate transcripts||Transcripts can be ready the next day or two.||There are several services that can remotely record telephone groups and have the transcripts ready the next morning. I have rarely found it necessary to have the transcripts sooner than a couple of days later. If the rush is that great, I use a notetaker. Transcripts are unlikely to be looked at anyway. For telephone, clients prefer the richness of the human voice on tape or CD.||Transcripts are available even during the session.|