Tech & Experience Design #4

In 2023, generative AI was a hot topic at the time. It is used extensively for tasks such as summarizing books, writing reviews, generating automatic email replies, providing exemplary sales pitches, and sharing marketing tips through natural language generation. 

Similar technologies have also emerged, blurring the line between AI-generated content and human-created content. AI-generated content is often considered superior and more accurate. ChatGPT, as one of the generative AI, is capable of generating dialogue through text, there are also efforts to apply it to voice conversations. This means that the user’s spoken voice is converted to text through speech recognition, inputted into ChatGPT, and the resulting output is converted back to speech through text-to-speech synthesis (TTS). When combined with a CG character, it creates an experience where it feels like you are having a conversation with that character. 

Voice dialogue experience

The generative AI is astonishing in its text-based language generation capabilities, but can it truly be utilized in voice dialogue (Voice UI, Voice Conversation)?

Have you ever been disappointed by traditional robots or AI conversation services that respond with phrases like “I don’t understand, please repeat” or “I’m sorry, I can’t do that” even though you made an effort to engage in conversation? Or, have you ever gotten tired of repetitive responses after having multiple conversations, resulting in a lack of engagement? The generative AI can create various responses that align with the given instructions or prompts, skillfully adapting to the questions or conditions provided.

Now, the question remains: Can generative AI become the savior of AI voice conversations?

My personal thoughts are: 

As of March 2023, there are still many challenges in directly applying dialogue-based AI models like ChatGPT to voice conversations. It heavily relies on prompts and instructions. However, when it comes to generating dialogue content (scenarios), it excels, and its future development is highly anticipated.

Let’s explain based on actual verification cases using ChatGPT. (Please note that this verification was conducted in March 2023 when this article was written. AI services evolve daily, and the outputs may differ if you input the prompts and instructions mentioned here. Please keep that in mind.)

Now, let’s explore the verification from the perspective of voice conversation experiences (Voice UX Design).

Verification Case: The AI system asks the user whether they prefer meat or fish dishes and provides meal advice, including the benefits of both, based on the user’s answer, using voice responses.

[ChatGPT Input]
You are someone who can provide dietary advice. In response to your question about whether the user prefers meat or fish dishes, the user answered that they like meat. Please respond, including the benefits of meat dishes.

[ChatGPT Response]
Meat dishes are a delicious and satisfying meal for many people. Meat is rich in high-quality protein, providing the necessary amino acids for muscle and bone formation, repair, and maintenance. Additionally, meat contains nutrients such as iron, zinc, B vitamins, and vitamin D, allowing you to obtain the necessary nutrients for maintaining a healthy body.

Meat dishes offer a variety of cooking methods, such as grilling, roasting, stewing, frying, and smoking, allowing you to enjoy various flavors. Different types of meat also have distinct tastes and textures. For example, beef has a robust flavor, while pork is tender and juicy.

Furthermore, meat dishes can serve as a source of energy. Due to the presence of fat in meat, it can be a valuable energy source. However, caution should be exercised with fatty cuts or processed meat products to avoid overconsumption.

Lastly, meat dishes are perfect for social gatherings and special events. For instance, steak and roast beef are popular dishes for birthday parties or Christmas dinners.

As mentioned above, meat dishes have many appealing qualities. However, it is important to consume them in moderation to maintain a healthy diet. It is crucial to enjoy meat dishes while striving for a balanced diet.

With very helpful information and while conveying the benefits of meat dishes, it also covers important points to note. That’s amazing, isn’t it?

However, what if this response were in a voice conversation? Try speaking out the above response. It’s long, preachy, and hard to remember the whole conversation… It’s too lengthy for a simple question about whether you prefer meat or fish.

[ChatGPT Input]
Please make the above response into a concise, conversational tone of about three lines.

[ChatGPT Response]
Meat dishes are delicious meals that are rich in protein and nutrients. You can enjoy them in various cooking styles. They're popular for social gatherings, but it's important to consume them in moderation. Let's aim for a balanced diet.

It has become shorter. It’s concise, including both positive aspects and important points to note; but phrases like “delicious meals” and “various” that are commonly used in formal writing stand out. Let’s assume a more casual conversation and ask for a shorter, colloquial response.

[ChatGPT Input]
Please provide a friendlier response.

[ChatGPT Response]
Meat dishes are seriously tasty and appealing, right? They're packed with protein and nutrients, and it's great that we can enjoy them in different ways. Plus, they're popular for special events and social gatherings, so there are plenty of ways to enjoy them. Just remember to have a balanced diet while savoring those delicious meat dishes!

It’s closer to colloquial language now. Next, let’s drop the formal tone.

[ChatGPT Input]
Please talk to me as if you're speaking to a friend, not using formal language.

[ChatGPT Response]
So, you like meat dishes, huh? That means you must really enjoy meat dishes like steak, hamburgers, and yakiniku! I'm also a fan of meat! It's rich in nutrients, and there are so many variations in cooking methods and seasonings, which can make it easy to overindulge sometimes. But let's remember to eat in moderation and strive for a balanced diet!

Now it sounds more like a friend’s response. Try speaking it out loud. How does it sound? 

Next, imagine listening to it as a voice response. You can input it into a text-to-speech software or service to listen to it. It’s quite well done, but the last line may feel like an unnecessary remark and exclamation marks might feel exaggerated and feel unnatural to some people. Maybe it’s because you instructed to engage the user’s interest, but did you find this AI response interesting? Did you feel like listening to it?

If I were to write an AI dialogue scenario for this case, I would create a voice scenario like this:

"Oh,  [User's Name] you really like meat, huh!” Meat is packed with minerals and vitamins like B, which can help your immune system. It seems to be effective during this season when the temperature fluctuates, and people tend to catch colds easily. But be careful not to overeat."

What are the differences between this and the previous ChatGPT response?

Here are the aspects I focused on:

– Functional filler phrases: Use phrases like “heh,” “oh,” or “huh” to show reactions, and awareness, and fill in gaps (to eliminate silence after speech recognition).

– Addressing: Calling the user’s name to grab their interest. It shows that you’re talking to them and increases familiarity.

– Parroting: Repeating the user’s words to show that you’re actively listening to their response.

– Incorporating seasonal or personalized content: In this case, connecting it to the chilly April weather to pique interest in the conversation.

Interestingly, these are techniques of active listening. Active listening is a communication skill that involves listening to the other person, aiming for mutual understanding and trust. It emphasizes elements such as “empathy, empathetic understanding,” “unconditional positive regard,” and “congruence” as three essential elements for the listener.

In AI voice conversations, the human interlocutor is a system (robot, agent). It becomes difficult for the human speaker to talk if no response shows the listener is truly listening. Whether both parties in a conversation can have an experience of being “listened to” rather than just “heard” becomes an important element in response scenarios and agent behavior.

Non-mechanical communication

To create communication that feels less mechanical, let’s consider another example. Imagine a voice dialogue agent (robot) at the entrance of a store or company that performs temperature checks.

Verification case: User responds with a temperature of 36.5℃ in an AI dialogue system.

[Example 1] Thank you for the temperature check. No fever, you’re all good.
(*Note: This is not the output of ChatGPT.)

What do you think of this response? It may seem fine when spoken out loud or imagined in your mind. However, it can be improved further.

[Example 2] Great! No fever at all. Your voice sounds energetic too (and your expression looks good).

First, let’s analyze the word order and meaning of the original response.

In example 1, the sentence follows the order of “gratitude (thank you for the temperature check) → result (no fever) → impression (that’s good).”

In this scene, the key is to convey that the user’s temperature is fine. Therefore, it’s important to communicate that as quickly as possible. However, if we only use a single word like “no fever,” “you may enter,” or “OK,” it can sound robotic and impersonal. The challenge with voice dialogue is that psychological impressions can be felt more strongly than through text or visuals.

Next, by changing the impression from “that’s good” to a simple reaction like “great!” and moving it to the beginning of the sentence, we create a more natural conversational feel. It creates a rhythm in the dialogue similar to filler words like “oh” or “huh.” You may have experienced situations where it’s difficult to talk to someone or the rhythm of the conversation feels off. Conversations have a rhythm, so it’s important to consider that when creating scenarios.

Finally, by adding supporting evidence like “your voice sounds energetic too,” we reinforce the result and create a sense of speaking to you and expressing thoughts. On the other hand, using phrases like “thank you” or “sorry” too frequently can make the response sound more like a machine/system, so I removed “thank you.”

I tried various prompts with ChatGPT to create an ideal response for this scene, but I haven’t obtained the desired results yet (as of March 2023. However, I have a feeling that if I provide more specific examples and instructions based on the knowledge I’ve shared here, we might be able to generate responses that are closer to the desired outcome).

Can generative AI create “words specified for you”?

In the process of verifying whether ChatGPT’s responses are useful for voice dialogue, it was found that with clear instructions (prompts), the responses can be quite good. However, even in human-to-human conversations, active listening is necessary for natural mutual understanding. If we don’t incorporate active listening into scenarios and actions, the experiential value of AI dialogue may not improve.

Above all, in dialogue, the words need to be “for you” to be “listened to” by the user. When we say things that are “said to anyone,” it tends to become a state of just “hearing” when written in kanji. Instead of listing ways to overcome a dislike of certain foods using words from ChatGPT or a search, wouldn’t it be more interesting for children to listen if we told a story like “When Shohei Ohtani was a child…”?

In my previous column, Are Voice UIs Making Natural Conversations?, I explained the difficulty of AI dialogue. Even if the dialogue response scenarios are well-crafted, they will only result in a good conversational experience if they match the appearance and behavior of the AI character.

There are plenty of important elements to improve voice dialogue beyond just scenarios. Among them, how to generate scenarios, how to create responses that are not monotonous and have variation, and how to create human-like responses have been long-standing challenges. Natural language processing models like ChatGPT, which can be generated automatically, will be one of the solutions to this challenge.

To enhance the experiential value of AI voice dialogue systems, a deeper understanding of human communication is still necessary. Within this context, I will continue to explore and verify while maintaining expectations for the ongoing evolution of AI models.

・・・

In Technology and Experience Design, we will not talk about textbook UX design, but we will talk about design drawing from a wide range of knowledge. Regardless of whether it is digital or analog, we will dig deeper into what it means to design an experience by interweaving practical development sites, world affairs, and familiar perspectives such as our living spaces and human sensibilities and emotions.