A Text-to-Game Engine for UGC-Based Role-Playing Games

Role-playing Games (RPGs) have become a prominent genre within the gaming industry, offering players a distinctive experience that relies on the interplay between participants and Non-Player Characters (NPCs). These games enable players to immerse themselves in diverse roles, collaboratively shaping a unique narrative through their actions. To elevate this experience, there is a critical need for an intelligent Role-playing System capable of enabling NPCs to authentically perform their roles and actively contribute to advancing the game narrative. Such a system requires NPCs to autonomously generate context-appropriate actions, moving beyond pre-programmed behaviors.

Despite advancements in RPG systems, a major challenge persists: the predominantly reactive nature and constrained response capabilities of current NPCs. While significant progress has been made in developing systems that allow NPCs to respond dynamically to player interactions[21][8][19][20], empowering these characters to take proactive, goal-driven actions based on their objectives and the evolving game context remains an open-ended and complex problem.

To address this limitation, we draw inspiration from the LLM-Powered Autonomous Agents framework[22], which has demonstrated exceptional potential for human-like decision-making in intricate environments. Building on this foundation, we propose a novel Role-playing framework comprising four core components: Perception, Memory, Thinking, and Action (PMTA), as illustrated in Fig. 5. This framework aims to equip NPCs with the ability to perceive their surroundings, retain and utilize contextual information, engage in complex reasoning, and execute appropriate actions autonomously.

The Perception module functions as the character’s sensory system, detecting and interpreting all changes within the game world. This includes observing external behaviors, tracking alterations in the game world state, and monitoring the progression of the game narrative. These inputs are processed by the Perception module and converted into structured data that can be utilized by Large Language Models (LLMs) or Multimodal Large Language Models (MLLMs). This processed data serves as the foundation for character decision-making.

The Memory module, another critical component, is responsible for storing essential information relevant to role-playing. This includes the character’s predefined role settings, in-game objectives, self-awareness, and memory fragments generated during role-play interactions. During the decision-making process, the Memory module dynamically retrieves relevant historical memories—represented in natural language text—based on the information currently perceived by the character. This mechanism ensures that characters have access to sufficient contextual information for reasoning and decision-making, thereby maintaining long-term consistency in their behavior. Furthermore, the results of the character’s reasoning and actions are carefully analyzed, and any significant information deemed necessary for future interactions is extracted and stored in memory, enabling real-time updates.

The Thinking module processes information received from the Perception module, along with memory fragments retrieved from the Memory module. By employing reasoning, planning, and reflection techniques[22], it generates action decisions while simultaneously updating the character’s memory. Unlike traditional game AI systems, which primarily rely on methods such as rule-based logic, Finite State Machines, Behavior Trees, Markov Decision Processes (MDP), or Reinforcement Learning for decision-making[23], our framework leverages the advanced reasoning capabilities of Large Language Models (LLMs) to handle this complex task. To ensure the Thinking module has access to comprehensive contextual information, we adopt the widely recognized Retrieval-Augmented Generation (RAG)[24][25] approach. This approach provides essential data, including the character’s role-setting details, memory fragments, real-time game progress, and knowledge about the game world. Additionally, we utilize a dynamic prompt generation module to create personalized role-playing prompt templates for each character. This ensures that characters across various game genres can deliver highly immersive and contextually appropriate role-playing performances.

The Action module interprets and executes the action decisions generated by the Thinking module. These decisions consist of a sequence of action elements, each representing either dialogue or a physical action within the game world. The Action module translates these elements into executable atomic actions within the game engine, rendering them in the game environment. By executing these actions, the module triggers subsequent responses from players or other NPCs and induces changes in the state of objects within the game world.

By integrating the PMTA framework with the reasoning capabilities of LLMs, our system enables characters to autonomously think and make decisions. This not only enhances the quality of the Role-playing System but also has the potential to revolutionize the RPG gaming experience by allowing NPCs to actively participate in and shape the game narrative.

The Game Status Manager module in our framework plays a critical role in tracking game progression and enabling the seamless introduction of new plot elements. This module is integral to the gameplay experience, as it determines the appropriate timing for assigning new tasks to players, revealing new clues or plots, and transitioning to subsequent game chapters.

The Game Status Manager is designed to fulfill three key functions:

• •

  It analyzes the latest interactions among all game characters, assesses their impact on the game environment, and monitors essential game states. The updated states are then displayed through the user interface (UI), providing players with immediate feedback.

• •

  It evaluates whether specific goals or objectives have been achieved based on the current game status.

• •

  It identifies when new plot advancements are required by detecting goal completion. In response, the module assigns new tasks to players or NPCs, delivers updated clues or plot information, or concludes the current chapter to initiate the transition to the next one, thereby enhancing the overall playability of the game.

During the game development phase, the Game Building Copilot assists creators in identifying critical game statuses that require monitoring. Creators define game goals and establish the criteria for their achievement. The Copilot then determines key performance indicators that align with these goals, enabling continuous tracking throughout gameplay.

Key status details are recorded using numerical values or concise textual descriptions. For example, in emotional companion games, player and character emotions are monitored using intimacy metrics. In Dungeons & Dragons, the health of players and monsters is tracked, while in adventure games, the player’s current location within the overall map is recorded.

To address complex goals, the Copilot breaks them down into multiple sub-goals based on logical dependencies or relationships. These sub-goals are presented in a structured format to ensure clarity and enable accurate interpretation by the LLMs. An example illustrating this process is provided in Table-I.

The diversity of game goals necessitates a flexible goal-checking module capable of adapting its prompt templates to suit each unique game scenario. This flexibility is crucial, as the module operates continuously throughout game dialogues, requiring both speed and accuracy to ensure seamless gameplay. However, the limited reasoning capabilities of lightweight LLMs can hinder the effectiveness of goal assessments. To mitigate this challenge, we employ two complementary modules that leverage state-of-the-art (SOTA) models:

• •

  Cold Start: Before gameplay begins, the SOTA model processes and comprehends the complete scope of the game’s information and objectives. It generates essential considerations for goal validation, which are subsequently incorporated into the goal-checking module’s prompt templates. These considerations serve as guidance for the lightweight LLM during gameplay.

• •

  Real-Time Assessment: During gameplay, the lightweight LLM’s evaluations are periodically sampled and compared with assessments generated by the SOTA model using identical inputs. This comparative analysis enables the SOTA model to identify and highlight discrepancies or shortcomings in the lightweight LLM’s evaluations, facilitating necessary adjustments to improve accuracy.

The Game Status Manager plays a pivotal role in our pursuit of open-ended text-to-game rendering and remains a key focus of our ongoing research.

Our objective is to revolutionize gaming narratives by creating dynamic, real-time storylines that adapt seamlessly to player actions and the evolving game state. This approach aspires to achieve a ”thousand different endings” phenomenon, delivering a distinctive and personalized gaming experience with every playthrough.

Unlike traditional methods that rely on static scripts or predefined storylines, our approach ensures that the gameplay experience remains closely intertwined with the evolving narrative and progression. The Emergent Narrative Generation System is defined by two key features: Real-time Narrative Generation and Interactive Narrative Consumption.

A complete gaming experience is composed of a combination of multi-modal content including visuals, sound, background music, and sound effects. Building on the foundation of the text-based rendering capabilities, however, unfolding an evolving multi-modal content expression that dynamically responds to player interactions and narrative progress ion throughout the game poses significant challenges on the output consistency and continuous coherent evolution.

The Multi-modal Rendering System utilizes LLMs for memory retrieval, status management, and information orchestration. Through various adapters, it transforms the RPG gaming experience into corresponding multi-modal descriptions to trigger real-time content generation by large multi-modal models. The produced multi-modal content serves as part of the session memory, ensuring consistency throughout the game’s evolutionary process.

This study’s innovative nature introduces unique challenges in designing experiments akin to traditional academic research. Specifically, two primary difficulties arise: the lack of publicly available datasets tailored to our research needs and the absence of accessible baseline systems for benchmarking. These limitations have hindered our ability to conduct direct comparisons with related studies, representing a notable constraint of this paper.

To address these challenges, we undertook significant efforts to collect experimental data and validate the proposed solution. An experimental system was developed, integrating the Game Building Copilot and the Zagii Engine, both accessible via our official website¹¹1https://rpggo.ai/. The Game Building Copilot empowers users to rapidly create games by leveraging the capabilities of large language models, while the Zagii Engine functions as an AI-driven game engine, enabling interactive gameplay responses and multimodal rendering. Screenshots of the experimental system for game building and gameplay are shown in Fig. 6 and Fig. 7.

We collected and analyzed user data from our platform over the period spanning February 1, 2024, to May 31, 2024. This dataset includes contributions from 325 game developers and over 60,000 game players. During this testing phase, developers utilized the Game Building Copilot to create a variety of games. Developers were provided with several game templates that could be customized and expanded upon, or they could choose to build entirely original games. With the assistance of the Copilot, most developers published the first version of their games within an hour, enabling rapid playtesting and debugging. The resulting games covered diverse genres, including Adventure, Role-Playing, Mystery, Simulation, and Strategy. Over the study period, a total of 168 games were developed and publicly released, while some developers opted to keep their games private.

During the experimental period, the 168 published games collectively recorded a total of 60,301 gameplay sessions. A gameplay session is defined as an instance where a player begins a game and continues until they either complete it or exit midway. Among these, one exceptionally well-designed game accounted for 35,407 gameplay sessions, showcasing its remarkable success and underscoring the potential of the text-to-game framework to create highly engaging content. However, to ensure the generalizability of the data, this outlier was excluded from subsequent analysis. Instead, we focused on the remaining 167 games, which collectively accumulated 24,894 gameplay sessions.

The distribution of gameplay sessions across these 167 games is illustrated in Fig. 8. As shown, a small subset of games attracted the majority of gameplay sessions, aligning with the Pareto principle (80/20 rule) commonly observed in traffic distribution. Specifically, 29 games achieved over 100 gameplay sessions each, while 6 games exceeded 500 sessions.

Fig. 9 presents the distribution of User-NPC (Non-Player Character) interaction rounds across the 24,894 gameplay sessions. The majority of interaction rounds fell within the range of 5 to 30. Notably, more than 200 gameplay sessions featured interaction rounds exceeding 50. A significant portion of gameplay sessions, however, involved fewer than 5 interaction rounds, reflecting instances where players initiated the game but exited shortly thereafter. These shorter sessions can be attributed to various factors, including technical issues with the game’s front-end display or the game failing to meet player expectations.

The experimental data validate the effectiveness of the proposed text-to-game framework: games developed rapidly using the Game Building Copilot and the Zagii Engine demonstrate substantial playability and user engagement. Furthermore, the ability to generate highly popular RPG games was evident. However, challenges such as cold start issues and limited gameplay engagement highlight the need for further optimization of the Zagii Engine and Game Building Copilot to enhance player experience, improve narrative generation, and enable the creation of higher-quality games.

While the Zagii Engine has demonstrated its ability to support real-time gameplay and multi-modal rendering, further refinements are required to enhance its role-playing system, emergent narrative system, game status manager, and multi-modal rendering capabilities. These improvements aim to deliver richer, more immersive gameplay experiences and enable the generation of more dynamic and complex storylines. For instance, refining the emergent narrative system could enable more adaptive and responsive storytelling that reacts to player actions in real time. Similarly, enhancements to the multi-modal rendering system could improve visual fidelity and interactivity, further immersing players in the game world.

The Game Building Copilot has proven effective in simplifying the game creation process, allowing developers to quickly publish playable games. However, further optimization is necessary to improve the quality of the inputs provided to the Zagii Engine. Enhancing the Copilot’s ability to interpret user intent and generate high-quality game scripts would significantly elevate the overall quality of the games. Additionally, incorporating advanced natural language processing (NLP) techniques could enable the Copilot to better understand complex user instructions and offer more tailored suggestions for game design.

Generative AI has driven significant advancements in the creation of 2D and 3D game assets. Industry tools such as NVIDIA’s GauGAN and Artbreeder are widely utilized to generate detailed and diverse 2D textures and character designs. In the domain of 3D asset generation, innovations like NVIDIA’s DLSS and Unreal Engine’s MetaHuman Creator have enabled the production of lifelike character models and immersive environments. Academic research has further demonstrated the potential of generative AI to produce high-quality assets that seamlessly integrate into game worlds. Despite these advancements, a key challenge persists: the lack of real-time rendering and optimization algorithms to ensure that generated assets are not only visually impressive but also optimized for performance within game environments.

Another crucial area for future work is the establishment of a robust A/B testing framework tailored for AI-native RPGs. Such a framework would enable large-scale evaluation of different model versions and game features by capturing detailed data points from player interactions. The process involves creating parallel testing environments where players are randomly assigned to different versions of the game engine or specific features. By analyzing player responses and performance across these environments, developers can identify which changes most effectively enhance the gaming experience. To maximize its utility, the A/B testing framework should be both flexible and scalable, allowing for iterative, data-driven improvements to the game engine and features.