Generative artificial intelligence – some research ideas

Intro

In the last 9 months, I have (together with Luka Tomat, PhD and other colleagues), run over 40 workshops for participants from practice, in-company, and focused workshops on using generative artificial intelligence (GAI). If you need a tailor-made, customized workshop, feel free to contact me. Alternatively, register for the GAI for research CPOEF workshop on October 15 or the general GAI workshop on November, 12. The workshops are informed by practical experience with many practical examples (also see this post).

Although I also closely follow the existing research and incorporate it in our workshops, I have deliberately decided not to do any research on GAI myself. The rest of this post is about some research ideas of mine that I do not plan to investigate any further.

Disclaimer

Notwithstanding that I have put considerable thinking into the development of the ideas, I have not carried out a systematic literature review on these specific research questions. I make no claims concerning the correctness, completeness or quality of these ideas.

Still, all of the ideas are very doable. The methodology is straightforward, clear, with interesting outcomes. Several possibilities for follow-ups are offered. Feel free to use them in any way you wish.

Broad introduction

The six ideas broadly fall into the following domains. The mentioned ideas touch on the broader issues that will be affected by GAI. These issues are connected to process redesign, ethics, education, decision-making, trust, and value proposition in the digital world.

1. Replacement of tasks by GAI (in certain professions): GAI will not completely replace employees. It has replaced certain tasks though, and will continue to do so. The core question is if/what is the value of human interaction in particular tasks.
2. Ethics in GAI: GAI offers numerous opportunities for real-life uses. Its abilities raise many ethical issues.
3. Education in the GAI era: Education will be transformed in multiple ways. How does GAI affect learning?
4. GAI optimization and the value of cheap content: GAI allows the creation of content. What is its value?
5. Believability in the GAI era: do users’ perceptions of the trustfulness of GAI content vary from how they view search engine results?
6. Decision-making in the GAI era: how do the ready availability of cheap content and changes in believability affect users’ decision-making?

1.GAI replacing tasks (in medicine)

Why now?

GAI will not replace jobs. It can replace several tasks entailed in a certain job and allow experts to focus on their skillset. The core question is the value proposition for the user of executing the task.

Medicine is especially important and has often been used as an example of a field to be drastically transformed by IT. We were considering this as a case when I was an undergraduate business student back in 1995! But many AI startups in healthcare have failed (e.g., IBM Watson).

Explanation

What does a medical doctor basically do? Apart from various forms of ‘data collection’ (e.g., examining the patient), their work can be divided into:

• emotional support role: for the patient or family members. The result is a comforted/reconciled patient/family member;
• informational role: conveying the diagnosis (information) that can assist the patient with dealing with the disease. The result is a better educated patient who will behave differently; and
• decision-making role: interaction with the patient and ‘independent work’ that leads to a decision, which profoundly affects the further process (i.e., clinical pathway).

In addition, there is:

• ‘physical’ work, e.g., an operation where technology can help (e.g., robotic operations), but where (G)AI has limited use;
• administrative/paper-work: which (G)AI should eliminate as much as possible; and
• many other things I forgot to mention.

The fundamental question is to what extent can a large language model (LLM) help with each of the above three types of tasks.

There are fascinating differences between the cases of “your mother-in law has Huntington's disease” and "your son has a headache":

• Huntington’s disease: From the point of view of a "severe disease", it is awful. In terms of the complexity of the explanation, it is extremely simple, even I can tell you almost the same as a doctor. An LLM is much cheaper and has much more time than a doctor. The informational role is key here, but perhaps even more emotionally. There is nothing special about decision-making, the simple answer is "nothing can be done, but we will alleviate the symptoms".
• "I have a headache" is tricky. When a child says "I have a headache", you can: a) tell him drink a glass of water and go to school, b) ok, stay at home, c) schedule a doctor’s appointment, d) take him to the emergency room, or e) immediately call an ambulance. Although the condition is likely not serious, many heuristic decisions are involved in the process.

Proposed methodology of the research

A quasi-experiment test on whether/to what extent GAI can replace the medical doctor in a given case.

For example, ask participants: "imagine your new-born/1-year-old son has been diagnosed with Klinefelter/Marfan/Down syndrome” or “your mother-in-law has Huntington’s disease”. You are very upset about this news. Try "to learn as much as possible" or "try to comfort as much as possible."

Or: "imagine you have just been diagnosed with Type 2 Diabetes. Which changes to your lifestyle do you need to make?"

You give participants a limited time to investigate, and then mainly measure user satisfaction/perception; the time of their interaction; the ‘course’ of their interaction; the acquired knowledge, and planned behavior change. Of course, the ideal sample would be "real patients", yet this is challenging due to ethics and obtaining access to the patients.

You can play with different types of diagnoses. The core point is classifying a diagnosis based on its severity and ‘difficulty of explanation/decision-making’.

2. Ethics in digital cloning

Why now?

It is now possible to make a realistic "digital twin" of yourself in such a way that the other person (e.g., your grandmother experiencing a slight cognitive decline) will not recognize it is in fact a digital twin. How does the general population view such an action?

Explanation

The scenario: 30-year-old John has a 90-year-old grandmother in a nursing home. The grandmother is lonely, experiencing a slight cognitive decline but mostly OK, as much as is possible at her age.

John visits her once a week, they chat casually, mainly about the same topics.

However, due to his children, work, and hobbies, John has less and less time. The last time he had to decide between attending his daughter's violin performance at the music school and visiting his grandmother.

Therefore, he develops a fully customized voice-bot: fills it with all the history about the grandmother, himself, daily important news, family events…. He prepares a clone of his voice.

The bot talks over the phone to the grandmother for 3 hours every day. John doesn't tell his grandma that it's not him. Grandma never gets to knows this.

The bot is trained by John every week with current events (partly from the newsfeed, his social media, partly in person). Another AI tool has access to John's photos and e-mails from the previous week, so it learns what to say on its own (e.g., about the daughter's first communion and his son's athletics competition). The third tool monitors what grandma is watching on TV, and adds a bit of discussion about her favorite TV shows.

Now John only visits his grandmother once a month. Before each visit, he uses another GAI tool, which prepares a 1-page summary of 80 hours of conversations. John uses it to prepare himself (e.g., by baking cookies, which the grandmother had mentioned several times in the calls).

Grandma is happy. Very happy.

Would this be perceived as ethically OK?

You can also test sub-scenarios:

? John gives his sister access to the combination of tools, who then also ‘talks’ with the grandmother.

? John gives access to a combination of tools to a friend who has a grandfather who is living alone.

? Since using the combination of GAI tools is difficult, John develops a new integrated software that enables digital cloning for users without advanced computer skills.

? John offers this software free of charge to everyone on the Internet.

? John sells this software. A subscription costs €50 per month.

? John gives the software away for free, but monetizes it through ads. For example, John’s digital clone sells nutritional supplements to the grandmother.

? John makes an IPO and sells his company for $1 billion.

What are the users’ perceptions? Which factors affect these perceptions? How can you change them?

Proposed methodology of the research

Create scenarios below, test people's attitude, their behavioral intention, and ethical attitude towards these scenarios. Measure the influence of various variables (personality, IT skills, demographics etc.).

You can introduce an additional twist. E.g., present the questionnaire as market research not a scholarly investigation. At the end, invite the respondents to participate in your start-up (e.g., as a tester) and compare their real reaction to such an offer.

3. GAI in education: learning vs doing

Why now?

IT has been changing education for the last 50 years. Today, GAI is able to do most homework/pass most exams on all levels. It will change how we learn and what we need to learn for our lives/careers.

Explanation

The core question is if/how the widespread availability and ease of use of GAI tools should/could change education. I am not just talking about the evaluation of knowledge (see my post on how I changed students’ evaluation techniques due to GAI) but the general positive/negative effect of GAI on learning.

Proposed methodology of the research

Recruit participants (e.g., students from a course). Give them some bonus points or financial compensation. Place them in a controlled environment.

Give the participants questions on a certain topic that they did not know much before. The topic can be either related to the course or a very specific one.

One group can use any (G)AI tool they wish. The other group can only use a “standard” search engine. Perhaps you can even have a third group that is only allowed to use scholarly literature/textbooks or to consult companies’ reports.

Check their written answers. Code the quality of the answers without knowing to which group the answers belong.

After the supposed end of the experiment, surprise the participants – test them again with an unexpected oral exam. Check how much they have truly learnt. This test can be held immediately after the experiment or with a 1-week delay.

My main hypothesis: the quality, level of knowledge, and reasoning of written answers will be much higher in a GAI group (GAI group “performs better”). When tested later, the non-GAI group will do better (non-GAI group “learns more”). This difference will be even larger with a 1-week time lag.

You can play with various settings, e.g., by giving financial rewards to the best performers; by telling them about the oral exam in advance; by giving either exactly the same or just slightly different questions in the oral exam etc.

4. Believability in GAI

Why now?

The core issue in many fields is trust and who people perceive the credibility of online content. This is important in various domains, not just marketing but also political science, child education, social sciences, conspiracy theories, medical treatments etc. How much do we trust the LLM content? The way in which LLMs respond to queries is considerably different than for search engines. LLMs write in a much more authoritative way.

Explanation

We are all familiar with how to use a search engine (e.g., Google). Over time, users have formed certain heuristics regarding how to evaluate the credibility of the content (e.g., you would likely believe a ranking on Tripadvisor more than self-bragging content on a restaurant’s website). LLMs are a new form of communication which already summarize various sources in ‘human-like’ text.

Proposed methodology of the research

Either prepare materials in advance or ask the participants to search for them online.

Ask them to evaluate how much they believe/trust in a certain piece of information.

Measure this trust in different ways; for example, one aspect is the level of believability (on a 1–7 scale) per se. Another aspect is how likely they would double-check the fact on another source. You can tie their financial reward to the answers at the end.

5. GAI optimization

Why now?

One thing is clear: people will use GAI tools more and more when investigating a particular topic.

Various GAI tools are already included in search engines and will be included even more in the future; Open AI is also developing its own search engine.

Explanation

This is more of practical relevance than pure basic research. The core question is how to increase the likelihood that a GAI tool will list as a top result your company/organization/yourself when somebody asks e.g.: “which is the best washing machine producer for a family of 5?”

See my LinkedIn post with further elaboration.

Proposed methodology of the research

You can use experiments. Prepare fake profiles of individuals, companies, or products. Upload into the knowledge base of an LLM, preferably an open source one. Play around with different sentences to see which one works better.

Ask GAI to select the best candidate for the job/the best products.

You can also ask both professional recruiters and domain experts to evaluate the candidates.

Alternatively, you can try real-field work. Prepare content, publish it online, and check how it proliferates in various LLM tools.

6. Decision-making in the GAI era

Why now?

This idea concerns integration of the previous two. The core question is how users will decide when making a purchase of products or services.

Explanation

My suggestion is the application to tourism where users constantly make decisions on what to visit.

Proposed methodology of the research

• Offer a predefined list of destinations that participants should choose from. Some get links to websites, others to Tripadvisor, and the rest to GPT-generated content. The main content is similar.
• Compare the impact of them.
• Alternatively, use a real interaction with a search engine or LLMs. This is a little tricky from the feasibility perception – you can ask participants to provide screenshots/a transcript of the inquiries.
• Ask users to explain why they chose the particular option. Use one open-ended question and a checklist with parameters to decide which (e.g., reliability of information, safety, sustainability, quality/cost ratio) influenced the decision.
• The use of real-life experiments (with actual decisions) would of course be even better,albeit?challenging to arrange.

The way forward

For each of these ideas, not just domain-specific literature should be examined. They can be properly placed within one of the ‘grand theories’ and thus used to provide relevant findings not just for GAI. Some ideas (e.g., GAI optimization) depend greatly on the tools available and are ‘time-sensitive’. Others (e.g., where emphatic GAI can replace a human or a medical doctor, and how do we feel about that) are crucial for the future of humanity.

Feel free to use any of the ideas. Implement them in your context. Search for research questions that will investigate the long-term changes, not the current hype.