When LLMs Play the Telephone Game: Cumulative Changes and Attractors in Iterated Cultural Transmissions

This repo contains code for the paper "When LLMs Play the Telephone Game: Cumulative Changes and Attractors in Iterated Cultural Transmissions"

The paper is also accompanied by a website that features a Data Explorer tool, allowing to look at the simulated data used in the paper.

How to use

conda env create -f telephone_llm.yml
conda activate telephone_llm

Reproducing the simulations

Set-up environment variables

• To use OpenAI models with the OpenAI API, set the OPENAI_API_KEY env variable (e.g. in your .bashrc):

export OPENAI_API_KEY="<your_key>"

• To use OpenAI model with the Azure API, set the variables for each model, for example:

export AZURE_OPENAI_ENDPOINT_gpt_35_turbo_0125="<your_endpoint>"
export AZURE_OPENAI_KEY_gpt_35_turbo_0125="<your_key>"
export AZURE_OPENAI_API_VERSION_gpt_35_turbo_0125="<your_version>"

• To use huggingface models, set the HF_HOME env variable to define your cache directory:

export HF_HOME="$HOME/.cache/huggingface"

• To use huggingface gated models, set the HF_TOKEN env variable

export HF_TOKEN="<your_token>"

Run the simulations

The script slurm_script.sh takes in argument a model name and a prompt name. The model name is the name of LLM that you want to use (e.g. meta-llama/Meta-Llama-3-8B). Prompt names are either "rephrase", "inspiration" or "continue".

• Regular machine:

for i in {0..99}; do SLURM_ARRAY_TASK_ID=$i bash slurm_script.sh <model_name> <prompt_name> ; done

This will sequentially launch 100 chains (20 initial texts * 5 seeds) for the specified <model_name> and <prompt_name>

• Slurm-based machine:

sbatch slurm_script.sh <model_name> <prompt_name>

Generated texts will be stored in Results/<model_name>/<prompt_name>

Reproducing the analysis and figures

• Perform the analyses:

  After running simulations for several models, you can compare them by running:

  python3 run_all_analyses.py -m <_model_name1_> <_model_name2_> <_model_name3_> <...> -sn <saving_name>
  

• Generate the figures

cd plots
python3 plots/plot_all_figures.py

Figures will be stored in Figures/figures_<current_date>

Cite as

@misc{perez2024llmsplaytelephonegame,
     title={When LLMs Play the Telephone Game: Cumulative Changes and Attractors in Iterated Cultural Transmissions}, 
     author={Jérémy Perez and Corentin Léger and Grgur Kovač and Cédric Colas and Gaia Molinaro and Maxime Derex and Pierre-Yves Oudeyer and Clément Moulin-Frier},
     year={2024},
     eprint={2407.04503},
     archivePrefix={arXiv},
     primaryClass={physics.soc-ph},
     url={https://arxiv.org/abs/2407.04503}, 
}