Focus Groups Should Not Be Used to Measure Attitudes of the Population | Myki Case Study

Victoria is in the process of adopting a new public transport ticketing system called Myki based on an electronic card. It's delivery is late and the project is over budget. The system has become a political issue. It was interesting to read the following statement:

"[MyKi project’s new spokeswoman] said that Government focus groups had shown that Melburnians were looking forward to using the new card." - TheAge

I just wanted to make a few comments about problems in the reasoning of the above quote.

1) Surveys are better than focus groups at measuring attitudes of a population.

• Focus groups are an inefficient and less reliable means of measuring attitudes of a population than a representative survey. 
• Focus groups are expensive relative to surveys on a per participant basis. Focus group research, partially because of the cost, typically involves small samples. Having small samples increases the randomness in the results.
• Focus groups involve a semi-structured set of questions that are administered to a group of people. It is textbook 101 market research that focus groups are good at many things, but these good things do not include generalisability and reliability (see Riger; MarketResearchWorld; or for a much better textbook discussion see the chapter on focus groups in Lukas et al).
• Interpreting  the results of a focus group is inherently subjective. The task of analysing focus groups involves synthesising the unstructured comments of multiple respondents across multiple groups. Not everyone in the group is asked the same question. People in the group influence each other. Dominant people can emerge. The questions are not asked in the same way. There are many things that analysts can do to accurately reflect the thoughts and opinions of the focus group. However, there are limits to this process.
• Designing a reliable and generalisable survey is challenging also. Designing the survey instrument, sampling in a representative way, and getting a sufficiently large sample, is a challenging task. But it is do-able. A good survey, even if not perfect, will provide a reasonable estimate of population attitudes. A perfect focus group will not.

2) People differ in their opinions.

• On almost any issue (the Myki ticketing system included) people differ in the degree to which their attitude is positive. Every dataset on student satisfaction, employee satisfaction and product satisfaction that I have analysed has shown variability in satisfaction. In most cases there is a sizable percentage (perhaps 5, 10 or more percent) that is dissatisfied. 
• In the case of Myki, what percentage of Melburnians are "looking forward" to using the new card? What percentage of Melburnians have even heard about Myki? Because it is focus group research we have no idea. A survey could have asked about attitudes to Myki and could have provided an estimate of the degree to which Melburnians had heard about Myki  and the degree to which they looked forward to the new system. A survey would have given a balanced explanation. It could say 60% or 70% or whatever are "looking forward to it". Instead, we are left with the statement "Melburnians were looking forward" to it, which taken literally can be refuted by finding a single Melburnian who is not looking forward to the new system. I've spoken to a couple.

3) Subjective methods + vested interest = bad.

• The combination of data collection methods involving high subjectivity of interpretation (e.g., focus groups) and a financier with a vested interest in the conclusions is not good. The stakeholders (i.e., the public) that are meant to be persuaded may doubt the research. And the stakeholders in general are more likely to be deceived.
• I don't claim that the researchers who conducted the focus groups were biased in their interpretation of results. I have no information to say either way. I am saying that there is a conflict of interests (see NSW Government website) for the government. There is a conflict between getting accurate information and getting politically favourable information. Whether the government acted ethically or not is irrelevant to the issue of whether a conflict of interests exists. In addition conflict of interests are more problematic when subjective methods, such as focus groups, are applied, because there is greater scope for the bias to be applied in ways detrimental to the public good.

None of the above is meant as an attack of the new ticketing system. Rather my aim was to critique the ways that governments and organisations use research to inappropriately justify ideas, whether this be caused by ignorance or political agendas.