New Trends in AI Interfaces

In recent years, the advancement of generative AI and Large Language Models (LLMs) has significantly transformed our software experience. In particular, User Interface (UI) design is shifting from the traditional focus on "what to click" to a perspective centered on "how AI can support and automate the actions intended by the user." In this article, based on actual product examples and conversations, we will provide an easy-to-understand explanation of the forefront of future AI interfaces, including voice interfaces, agent-based workflows, and design utilizing generative AI. In the original discussion, Rafael Shad, the developer of Notion Calendar, mentioned that "software until now was designed to handle 'nouns' on the screen, but with the advent of AI, it becomes important to handle 'verbs'." Below, we will organize the key points of this new UI paradigm while citing specific examples.

1. New Trends in AI Interfaces

1-1. UI Design Focused on "Verbs"

Traditional software design was dominated by a format where users operated by clicking or inputting "noun-like elements" such as buttons and text forms arranged on the screen. However, with the evolution of LLMs and generative AI, there is an increasing number of forms where the AI automatically performs processing and makes suggestions when the user provides instructions on "what they want to do (verbs)" via text or voice.
For example, simply by the user giving a simple instruction like "reply to the email," "generate an image," or "auto-fill the form," the software side autonomously executes multiple processes in the background—this flow is a major turning point in recent UI/UX design.

1-2. Gaining User Trust and Control

While AI performs processing autonomously, it is also true that users are prone to feeling anxious about whether the information is truly correct or if the AI is acting in unintended ways. Therefore, it is considered important to design features such as "source citations" that clarify the basis for AI output, and "intermediate check functions" that allow tasks to be interrupted or corrected. In actual products, efforts are being made to ensure users can use them with peace of mind, such as displaying URLs that serve as the basis for generated results or visualizing the processing steps.

2. The Forefront of Voice Interfaces

2-1. Vapy - An AI Assistant for Developers Operated by Voice

The first service featured is "Vapy." Vapy is a voice AI for developers, aimed at integrating voice assistant functions into apps and phone numbers. In the actual demo, the following conversation took place.

Natural voice interaction like this is possible, and it has a feature that displays the response speed of the conversation (in milliseconds). In the case of voice UI, how to handle real-time performance and user interruptions is a major point. In fact, in the demo, there was a scene where the AI continued the conversation even when the user interrupted before finishing speaking, and it could not handle the interruption well, but this is an area where improvements are expected in the future.

2-2. Retell AI - Automated Phone Response and Advanced Conversational Branching

Another example, "Retell AI," is a service that automates call center operations and the like using voice AI. When a user registers a phone number, the system actually calls them, and the AI handles tasks such as listening and payment reminders.
In the actual demo, we tested a scenario where "I called Aaron, but someone else named Steve answered instead of the person himself." The AI responded, "I'm sorry, could you put Aaron on the line?" and when the other party replied, "He's not here anymore," it ended the conversation naturally.
On the other hand, there were some points where the conversation delay and the simplification of responses were slightly unnatural, suggesting that challenges remain in maintaining human-likeness. Even so, it is a major step forward that it realizes advanced context understanding, such as understanding name corrections and changing how it addresses the other party. In call center operations, flows such as having the AI handle initial responses first and then escalating to a human operator when requirements become complex will likely become common.

3. AI Agents and Workflow Design

3-1. GumLoop - Visualization with Canvas-based Workflows

“GumLoop” is a platform that visualizes complex flows executed by agents and allows users to build them without code. For example, procedures for web scraping are placed and connected as blocks on a canvas in the form of “get input → generate URL → collect data with AI agent → organize data → output results.”
Conventionally, such processing flows were described in text or source code, but the canvas-type UI makes it easier to get an overview of the entire process. On the other hand, since visibility may decrease as the number of blocks increases, improvements such as automatic folding based on zoom level and displaying legends for color-coding are expected.

3-2. AnswerGrid - Fusion of Spreadsheets and AI Agents

“AnswerGrid” is a service that aggregates information on the web based on free-input prompts and displays it in a structured data format.
For example, if you input the prompt “Create a list of AI companies in San Francisco,” it will automatically extract company names, locations, etc., and output them onto a spreadsheet. Furthermore, if you add a new column called “Funding Amount,” the AI will perform a web search for each cell and input the corresponding amount.
What is interesting is that it attaches reference links to each cell. This allows users to verify the reliability of the data. Conventional search engines only returned a list of links, but the flow where “AI summarizes and structures information while also showing the basis,” like AnswerGrid, is likely to spread in various fields in the future.

4. Generative AI and Design

4-1. Polymet - Generating UI Designs with Prompts

"Polymet (also known as Polyat)" is a service that automatically generates Web UI designs from inputs such as text or image sketches. For example, by simply prompting, "Make the background a deep orange gradient and add a glassmorphism-style sidebar," the UI is generated instantly to reflect those instructions.
In the actual demo, it was shown that one can click on the completed UI elements and re-prompt, "Change this part of the sidebar to blue," to partially modify the design. This is a way of using the tool where you "only instruct the AI on the differences for a design that has already been generated." Such incremental modification flows save time and can be said to be an example of maximizing the strengths of generative AI.

4-2. Zuni - Suggesting email replies based on context

"Zuni" is an AI integrated into email apps that displays automatic reply candidates based on the sender's content and context. While traditional email clients have fixed buttons, with Zuni, the reply buttons change "dynamically" depending on the email content, such as "Reschedule meeting," "Apologize," or "Suggest next date."
The design is simple to use, allowing users to operate it with keyboard shortcuts, such as pressing "Y" to "Reschedule" or "N" to "Decline." While it aims for high-performance chatbot-like replies, there are still issues to be improved in the future, such as measures for when a user accidentally presses a key and the UI uncertainty caused by reply candidates constantly changing.

4-3. Argil - "Digital twins" through video generation

"Argil" creates realistic "deep learning avatars" by training on a user's voice and a few minutes of video data, generating videos that move according to arbitrary scripts.
In the demo, an avatar reciting "Hello my name is Aaron..." was created in a few minutes, and the lip movements were synchronized with the text. Ingenuity is also applied to help the user understand that the initial preview shows a deliberately blurred image and that the final generation takes over ten minutes. This two-stage approach of preview and final generation makes it easier to perform necessary corrections and reduces wasteful re-generation. In the future, it will likely become possible to smoothly perform operations such as modifying only a part of the video or integrating different movements and gestures.

The examples introduced here are just a glimpse of the AI interfaces that have changed drastically in just a few years. In various fields such as voice, agents, generative design, and video, UI/UX that responds directly to "what the user wants to do" is emerging.

• Voice support: As seen in the examples of Vapy and Retell AI, natural conversational experiences, such as response speed and interrupt handling, are key.

• Agent-based workflows: As shown by GumLoop and AnswerGrid, complex processes are automatically executed while being visualized on canvases or spreadsheets.

• Generative design: Like Polymet, an incremental generation method where UIs are built based on prompts, making partial modifications easy.

• Context-aware UI: Like Zuni's email reply candidates, UIs that change dynamically according to content significantly reduce user effort.

• Video generation: Video generation by Argil's AI avatars has the potential to fundamentally change the content production flow itself.

On the other hand, there are many challenges such as "stress due to latency," "AI incorrect answers and reliability of data sources," and "mismatches in operation prediction," and how to resolve these will be a major theme for UI/UX in the future.

During the conversation, Raphael Schaad stated that unlike software that "handles nouns on a screen" as before, AI is designed to focus on "how to make the user's verbs (actions) smoother." An era where users instruct the "desired results" and the AI performs complex processing and suggestions in the background is just around the corner. In creating new interfaces, how can humans and AI collaborate and grow while receiving mutual feedback? The key can be said to lie in a design philosophy that "successfully visualizes uncertain parts and leaves final control to the user."

Related articles