Human-AI hybrid approach can create more efficient marketing research

Consider a business context in which a brand manager collaborates with a consumer insights manager to formulate the problem the research is trying to address, and come up with a set of research questions. The two may collaboratively agree on a research design that, for example, begins with exploratory research (for example, in-depth interviews) followed by descriptive research (e.g., a survey). 

These first two steps of the research process are largely led by humans. Although the brand and insight managers could consult an LLM to gather secondary research on the topic and explore use cases that could help inform the research questions or research design, they would still largely rely on their knowledge of the business context to formulate the research problem, questions and design. 

The central premise is that a human–LLM hybrid approach can lead to efficiency and effectiveness gains in the marketing research process.  

Researchers replicated two studies that used an LLM. 

To test this premise, researchers partnered with a Fortune 500 food company and replicated two studies the company had conducted using an LLM. The first study was qualitative and centred around business questions relating to a popular public holiday celebration. The second study focused on testing a new refrigerated dog food. 

Research paper co-author Neeraj Arora explains that “for each study we treated the original human studies as the ‘ground truth’ and benchmarked the LLM-generated studies against them. This approach allowed us to objectively evaluate the quality of synthetic data and investigate the role LLMs could play in knowledge generation.” 

For qualitative research, the study indicates that LLMs are excellent assistants for data generation and analysis. 

On the data-generation front, LLMs effectively create desirable sample characteristics, generate synthetic respondents that match those characteristics, and conduct and moderate in-depth interviews. Results show that LLM-generated responses are superior in terms of depth and insightfulness. 

On the analysis front, the research team found that LLMs perform well, matching human experts in identifying key ideas, grouping them into themes, and summarising information.  

Although LLMs missed some themes that humans detected, they generated some that humans did not. Expert judges find that human-LLM hybrids outperformed their human-only or LLM-only counterparts. 

“The upshot is that LLMs and humans bring unique, complementary insights that managers should leverage,” says Chakraborty. 

You can find out more about the research here.