OK, so synthetic users aren't perfect. But aren't they better than *nothing*? Probably not. ?? Synthetic research is not the same as real research. ?? If you aren't doing any real research, you won't catch inaccuracies. ?? Your stakeholders may start to see fake research as "good enough". Maria Rosala and I particularly recommend that you AVOID synthetic research if: ? Your stakeholders may see synthetic research as a replacement for real research. ? Your user population is niche or specialized (and thus less likely to be documented in training data.) You CAN use synthetic research responsibly, if you: ? Realize that LLM-generated data will have some hallucinations, misinterpretations, or misrepresentations. ? Treat synthetic research as desk research that needs to be investigated. ? Understand that people's documented thinking and behaviors are likely different from real life. Get all the details in our article: ?? https://lnkd.in/eC67Mht4 #ux #aiux #userresearch #uxresearch Nielsen Norman Group
Great points, especially the last slide! I will always worry that the drive to paint one's self and work in a good light will mean an AI-driven research will have a bias toward a rosy picture of the current state of any situation when it comes to documentation of current work. We publish our best successes way more often than our mundane, more representative work.
Usually people don’t document the real way they would react or behave. If they would, then there would be a lot less psychologists in the world.
While it makes sense to be cautious with synthetic research, I struggle to see its practical applications. Who exactly is supposed to use it, and when specifically? For UX beginners, it’s challenging to discern valuable insights from misleading ones. For senior UX professionals, it often feels too generic and less precise compared to the advanced techniques they’re familiar with. New products cannot be built based on synthetic research; mature products must have certain access to real users. So, in my opinion, synthetic research is hardly useful for anyone ?? Probably the only use case (mentioned in your article) is starting work in a completely new domain. But then, instead of creating synthetic personas or CJMs, it suffices to just ask ChatGPT about the topic in a free format and learn the key terminology before conducting proper research. And what do you think about other use cases?
Sigh. "Synthetic Users" is an oxymoron.
There is no such thing as synthetic research, there is merely synthetic scenario generation and estimation through simulation. Research focuses at thruth findings. In this we hope to have both confirmation of theory but also have unexpected outcomes to reveal themselves. If we would have used synthetic research to investigate what happens at absolute zero with conductivity of electricity, there wouldn’t have been a discovery of superconductivity. Generative A.I. is 100% based on -what is known- and what is status quo from a scientific stance; theories and laws. You will not find a higgs-boson particle in synthetic research. You also cannot “learn” from a possible lie if you don’t know the truth. You can get generated scenarios that might be worth attesting though. Note that you -can- learn from simulations though it will remain questionnable if the outcome will be similar to the real world. E.g. a slippery road due to ice or oil can be “missed” when simulating an EV that is estimating the break path of a car.
I personally don't see any benefit. It's like playing pretend. If you're doing pretend research you might just as well, go all in and not do any user interviews at all and just base everything on assumptions or SMEs. The results might even be better then if you base your hypothesis on simulated responses based on response patterns that aren't based in personality just large format text. There's just no correlation between a large chunk of text and the behavior, predictable or not, of a human being. I find it disappointing that you had to adjust your message just to keep up with the AI hype. ??
The answer is like most research it depends on the variables and conditions you place it in. Synthetic research has a place, it does not replace real research on real people, but it can offer insight on where to focus etc. That being said SR's current state is developing. There is so much that goes into synthetic information there are plenty of places it can be used poorly and ways it can fail. Conversely, there are ways synthetic research can be used that were not previously possible. Starting from good information is always key.
Head of Digital | NN/g UX Ambassador | UX Master Certified | UX, Usability, Baymard & Analytics Certified
8 个月If folks conduct research with generic status quo information, then they risk arriving at a generic status quo solution. I'm having similar issues with various AI and ML Attention Mapping tools at the moment. Though some are better than others if you have the knowledge to understand the limitations of their application – the concern as you detail here, is that low-maturity influencers and decision-makers are propagating the myth that these tools are good enough. But as for synthetic users, you only need brief experience of moderated research and trying techniques like '5 whys' to understand how shallow and inaccurate synthetic users will be! ... I have so many usability lab comments and insights floating around my head from previous years that a synthetic user would never be able to generate ??