Tech & Experience Design

I’ve long said, “If you want to learn UI design, UX design, and interaction design (IxD), you should try playing games.” Here, the term “games” refers to video games that run on platforms such as PlayStation, Nintendo Switch, PC, and smartphones.

Some people might argue that instead of spending time playing games, they could be reading great books on UI or UX design, or taking online courses. This is of course a valid approach. However, when it comes to UI or UX, the techniques and designs developed in gaming have been applied to various experiences in modern digital devices, including entertainment, business, and education. Moreover, games have players of all ages worldwide, and sometimes occur in challenging environments. To meet these demands, intuitive and systemically sound designs.

Playing games gives you a chance to experience great UI/UX design and the latest technologies, while also helping you quickly learn about their strengths and weaknesses.  In this article, I will elaborate on the four following key elements of UI/UX design that can be learned from gaming below.

1. UI and IxD are the Core of Games
2. Strict Demands for Interaction
3. The Compatibility of Next-Generation Technologies with Games  
4. Pioneers of Communication Platforms in the Internet era.

UI and IxD are the Core of Games

Game hardware interfaces, such as controllers, gamepads, keyboards, and mice, are used in demanding situations. Compared to devices like TV remotes or household appliance buttons, these interfaces are operated much more frequently and often handled roughly. In the past, games were simply controlled with a joystick (or D-pad) and two buttons. Today, game controllers have evolved to include analog sticks, L1/L2/R1/R2 trigger buttons on the top, touch pads, and even motion sensors, allowing players to use both hands and fingers in more complex ways.

Similarly, in games, the UI tends to be more complex compared to typical apps or social media platforms, requiring a variety of interactions. Long gaming sessions often involve repeating the same actions multiple times. Therefore, it’s essential for the UI to meet several strict conditions. It shouldn’t cause fatigue during extended play, must be user-friendly (easy to understand and use), and shouldn’t interfere with the core actions players need to perform.

No matter how well-crafted the story or how engaging the game mechanics and world-building are, insufficient UI can lead to a game being labeled as “boring.” This highlights the critical importance of UI in gaming. Compared to UI design for other products and services, game UI design can be more challenging due to the several factors involved. As a result, understanding game UI is valuable for all digital UI designers.

From my experience working at Sony and SIE (formerly SCE), I’ve noticed that Japanese users tend to be particularly sensitive and critical when it comes to the UI of games and systems. Observing and experiencing the UI of games that have successfully passed such strict user criteria can be a great source of inspiration.

High Demands for Interaction

Games operate in highly competitive environments where decisions are made in less than a second. In first-person shooters (FPS) and third-person shooters (TPS), as well as fighting games like Capcom’s Street Fighter and Namco’s Tekken, players need to make quick decisions and actions within a few frames. For example, a well-known player in Sega’s Virtua Fighter tournament was able to recognize changes in just one frame, which translates to 16.6 milliseconds (at 60 frames per second). This demonstrates how crucial timing is and how developers must minimize any delay.

Modern TVs often have a gaming mode, which is designed not only to provide the best picture quality, but primarily to output images with low latency. This mode processes the images from gaming consoles as quickly as possible to display them with minimal delay. The sensitivity and transmission speed of controllers also impact gameplay. Even a slight delay in system processing can impact the outcome of a match, so gamers need to react quickly and accurately, requiring performance that’s precise down to the millisecond. This is why wired LAN and wired controllers are preferred in FPS and fighting games.

As a result, game interactions are designed with strict requirements, ensuring that visuals, sounds, and haptics are processed with low latency and high accuracy, especially during fast-paced and intense exchanges.

Compatibility with Next-Generation Technologies

Games and Evolving UI

VR (virtual reality), AR (augmented reality), and MR (mixed reality) are collectively referred to as XR (extended reality) these days. Hardware in the XR industry, such as Meta’s Quest series, PlayStation VR head-mounted displays, and the Apple Vision Pro, which was released in the U.S. in 2023, is gaining significant attention. Games are often the first type of software to adapt to this hardware. While these devices also allow for movie watching and web browsing, many developers are focusing on creating games that leverage the unique characteristics of virtual spaces.

Natural UI (NUI) is also a UI that fits well with games. You may have seen scenes in sci-fi movies where devices are controlled by voice, hand gestures, or gaze. These interactions are called Natural UI, representing a next-generation UI where natural and intuitive human actions serve as the interface. A notable example of NUI is the Kinect for Microsoft Xbox, which uses hand and arm gestures for game control.

Since the PlayStation®4, there has been a growing number of systems and games that support voice control through Voice UI (VUI). For instance, the Nintendo Switch allows players to swing the controller like a tennis racket, throw a bowling ball, or toss a volleyball, with the in-game character responding accordingly. This makes it easy for those unfamiliar with traditional button-heavy controllers to quickly get into the game. 

Additionally, steering controllers for racing games often feature force feedback, which realistically conveys sensations such as road bumps or skidding on the tires. This is one application of haptic technology.

AI Utilized in Games

Generative AI is also gaining attention in the gaming industry. Recently, open-world games have become a trend. An open world is a vast virtual space where players can freely explore and accomplish objectives, similar to the real world. In the past, games often limited player movement to specific areas, which became irrelevant once completed. Now, players can move around freely, but the non-player characters (NPCs) within these worlds still perform actions according to pre-programmed scripts.

As open worlds expand, the number of NPCs increases significantly, leading to an overwhelming amount of time required to design their dialogue and actions. This is where generative AI comes into play.

At GDC 2024, Ubisoft announced a technology called “NEO NPC”, which enables natural dialogue with in-game NPCs. Similarly, NVIDIA unveiled a digital avatar development toolkit called “NVIDIA ACE” at CES 2024. These technologies not only generate dialogue, but also include aspects of expression and behavior. Additionally, starting in 2023, there have been attempts to input prompt information like scenarios, backgrounds, and attributes into generative AI tools like ChatGPT to automatically create dialogue in real time.

Whether hardware or software, new technologies are often first implemented in game UI and engines before being adopted across various industries.

Pioneers of Communication Platforms

In today’s world, many people communicate with family and friends through messaging apps like LINE, WhatsApp, and Messenger. These services have become essential, allowing individuals to chat in groups, send images, and more. Similar communication features were implemented in massively multiplayer online role-playing games (MMORPGs) over 20 years ago, around the year 2000.

In these games, players could add friends to their lists, send direct messages, gift items, and chat within teams or alliances, much like today’s messaging apps. However, back then, there were no sticker functions, so players instead used letters to express emotions, such as 🙂 or ^_^ for a smile.

While game developers consider features and services to gather players, they also need to address the potential downsides of increased interaction, such as dealing with disruptive behavior. This has led to the implementation of features like blacklists, blocking functions, and oversight by Game Masters (GMs), who monitor users and can suspend accounts when necessary.

Through online gaming, players can connect with one another, purchase in-game items, and give them as birthday gifts. These moments often lead to joyful reactions that can be shared on social media. With just a smartphone, anyone can enjoy these experiences from anywhere in the world. Game platforms have made this a common feature for years, highlighting how games have shaped the way we build online communities and communicate. It’s important to recognize the valuable insights gained from these pioneers.

Summary

The metaverse and Web 4.0, which extend beyond the current web, are anticipated to provide new experiences in human interaction within virtual communities, the trading of digital content, and the blending of reality and virtual environments. As mentioned earlier, modern games are highly compatible with the XR domain and are actively integrating generative AI. Moreover, online games have been evolving community and communication platforms for over 20 years.

Games traditionally present specific goals, such as defeating a final boss or completing challenges. However, many games have now evolved into open worlds, focusing on how players can enjoyably engage and live within these virtual spaces over the long term. This experience is what ultimately determines the quality of a game. The concept of “gamification,” used in services and human management, also applies game design principles and elements to enhance user engagement.

If you are interested in UI/UX design, I encourage you to approach gaming from this perspective. You’ll be amazed by how much the medium has evolved.

Now then, I’ll be returning to Eorzea!  

(Note: Eorzea is the name of the virtual world in Square Enix’s MMORPG “FINAL FANTASY XIV.”)