GenSim: A General Social Simulation Platform with Large Language Model based Agents

Our framework consists of three modules focusing on single agent, multi-agents, and environments (see Figure 1). In the single agent module, users can flexibly configure the agent’s profile, memory and action components. The profile includes both public information, such as gender, name, and birthplace, as well as private attributes like income and health condition. To enable the agent to retain behaviors in various ways, users can assemble different memory components—short-term memory, long-term memory, and the reflection mechanism—to build the agent’s memory. The actions of the agents are driven by LLM prompts, where users can flexibly configure them to include agent profiles, memories and so on.

In the multi-agents module, we design two strategies for generating agent interactions: script mode and agent mode. In script mode, all interactions are treated as a whole and generated in a single call to the LLM. For example, one can directly prompt LLMs to generate a dialogue between a doctor and a teacher in one step. In this strategy, the LLM acts as a meta-agent, producing the dialogue from a third-person perspective. In agent mode, interactions are generated by different agents, each representing a distinct role, and each agent generates outputs from a first-person perspective. This interaction generation process requires multiple calls to the LLM. In the example above, two agents are deployed, with each agent’s output based on the entire history of interactions between them.

In the environment module, we store all the information beyond the agents necessary for running the simulation, such as the recommendation algorithm used in a web user simulator [6]. Additionally, we allow users to globally intervene in the platform, which is useful for counterfactual inferences. We also provide essential functions to facilitate interviewing, searching, and storing different agents.

Based on the above general framework, users can easily create customized simulations. To provide additional references, we offer three default scenarios: job market, recommender system, and group discussion. These scenarios can not only facilitate related research but also can provide code bases, enabling users to construct new scenarios with minimal effort.

While there are many previous studies on leveraging LLMs to simulate human social behaviors [2], the number of agents in their simulators are usually very small. In such cases, the users need to sample a small set of individuals from the real-world large-scale populations, and then leverage agents to simulate the sampled individuals, assuming that these samples can accurately approximate real-world populations. However, the sampled small number of individuals may lead to very large fluctuations of the simulation results. To verify this statement, we conduct a preliminary experiment by simulating the user-item rating behaviors on a movie website. In specific, we base our experiment on the well-known dataset MovieLens-32M¹¹1https://grouplens.org/datasets/movielens/, which consists of 200,948 users’ 32M ratings on 87,585 movies. For each user-movie pair, we use LLMs to simulate the user’s rating on the movie in the range of $\mathbf{R}=[0.5,1.0,...,5.0]$. To study the fluctuation of the simulation results with different agent scales, we first sample 3.2K, 32K, 320K and 3.2M user-item pairs from the complete dataset, and then, for each case, we repeat the simulation of predicting user-item ratings for 10 times. Formally, suppose $\bm{p}_{i}$ represents the rating distribution of the $i$th experiment, where $i=1,2,\dots,10$. For each rating $r\in\mathbf{R}$, we compute the standard deviation across all experiments as

$$v(r)=\sigma(\bm{p}_{1}(r),\bm{p}_{2}(r),\dots,\bm{p}_{10}(r))$$

where $\sigma(\cdot)$ denotes the standard deviation operation. We use the sum of the standard deviations for all possible ratings to measure the fluctuation of the simulation results. The experiment results are presented in Figure 2(a), where we can see: as the number of samples becomes larger, the fluctuation of the simulation results is greatly lowered. This result suggests that if we only have a small number of agents, then the simulation results can be not reliable, since it can be hardly reproduced due to the large simulation fluctuation.

To solve the above problem, in our platform, we support up to one hundred thousand agents to better simulate real-world scenarios. To accelerate the simulation speed, we adapt a series of techniques including distributed parallel computing and so on. Using our platform, we evaluate the simulation speed in the job market and recommendation scenarios, where we run our simulator for one round for both settings²²2For all experiments in this paper, we used a server with a 192-core CPU, eight A100-40G GPUs, and 440 GB of memory.. The results are presented in Figure 2(b), from which we can see: as the number of agents becomes larger, the time cost increases, and when we have 10w agents, they cost 15492 and 3024 seconds for running one round in the job market and recommendation scenarios, respectively.

In addition, we also evaluate the acceleration effects of distributed parallel computing in our platform. In specific, we measure the time costs of running our platform for one round with different numbers of GPUs. The results are presented in Figure 3(a), where we can see, as the number of GPUs becomes larger, the time cost decreases, which suggests that, with the help of distributed parallel computing, our platform can effectively take the advantages of more GPUs.

Most previous LLM-agent-based simulation platforms lack error-correction mechanisms, which means that if unexpected results occur during the simulation process, they can be accumulated and amplified as the simulation progresses (see Figure 3(b)). To solve this problem, in our platform, we provide two strategies for correcting the simulation errors. The first one is based on LLMs, where we leverage GPT-4o to score on or revise the simulated results. The second one is based on real humans, where we provide interfaces for the users to score or revise the simulated agent behaviors. Between these two approaches, the first is more efficient and requires no human intervention, though it may be less accurate due to the inherent biases of LLM. The second approach is more aligned with real humans but can be labor-intensive and less efficient. Suppose the simulation result is represented by a $(q,a)$ pair, where $q$ is the prompt for driving an agent action, and $a$ is the action. For each of the above strategies, there are two forms of feedback provided by LLMs or real humans. Let the score for $(q,a)$ be $s$, and $a^{\prime}$ be the revised results³³3It should be noted that $s$ and $a^{\prime}$ may not co-exist for the same $(q,a)$ pair.. Then, we use $(q,a,s)$ and $(q,a^{\prime})$ to fine-tune the backbone LLMs using PPO and SFT, respectively.

To evaluate the effectiveness of our designed error-correction mechanisms, we conduct experiments based on the job market scenario with LLMs as the feedback provider. To begin with, we evaluate whether PPO and SFT can improve the simulation results in a single round. In the experiments, we select different numbers of samples for labeling, and use GPT-4o to measure the reasonableness of the simulation results. From Figure 4(a), we can see: for both PPO and SFT, they can improve the simulation performance across different numbers of labeled samples. Compared with PPO, the results of SFT are better, which is reasonable, since the revised action $a^{\prime}$ used in SFT may include more effective and comprehensive information.

Next, we evaluate the effectiveness of the error-correction mechanisms in a multi-round setting. Specifically, we fine-tune the backbone LLMs in the earlier round and use the updated models to simulate the results in the subsequent round. We present the results in Figure 4(b). We can see: if we do not conduct error-correction (the yellow line), the simulation performance is unsatisfactory. When using PPO (the blue line) or SFT (the red line), the simulation performance improves significantly, and these improvements continue to increase as the number of simulation rounds grows.