inzva AI Projects #2 - Fake Academic Paper Generation Project
In this project, we aim to use the LaTeX source files of open access papers on arXiv as a dataset and feed it into a neural network to be able to generate realistic looking academic papers. We chose the character based recurrent neural network (RNN) model used by Andrej Karpathy in his blog post as our baseline [1]. We will try to improve the baseline results of the char-RNN model by applying transformers and attention mechanism [2]. We also want to try GANs to generate realistic LaTeX code. [3]
To the best of our knowledge there was no available dataset compiled from academic papers. Therefore we decided to prepare a dataset from academic papers on arxiv.org.
All scripts related to the dataset preparation can be found in the dataset_generation directory.
We selected Computer Vision as the topic of interest for the dataset. Therefore, we crawled arxiv.org to find papers tagged as Computer Vision between 2015 - 2018. (BeautifulSoup is used as html parser)
related scripts:
- dataset_generation/crawler.py (crawles arxiv.org as specified and writes the result to paperlinks.txt)
- dataset_generation/random_paper_sampler.py (samples examples from paperlinks.txt and writes the result to selected_papers.txt)
We downloaded the source files as tar files for the selected papers and untar/unzip them.
related script: dataset_generation/downloader.py (reads selected papers from selected_papers.txt, downloads the source files and untar/unzip them)
We resolved \include, \input kind of import statements in latex source files in order to compile each paper into one latex file and wrote a latex file for each paper.
related script: dataset_generation/latex_input_resolver.py (Finds the latex files from the source files, reads the content using TexSoup, finds the root files(files including documentclass statement), recursively replaces the import statements with the content of the imported file, and writes a latex file for each paper.)
- dataset_generation/complete_dataset.py (kind of combination of all these scripts which finds problematic source files and replaces them with other papers from the paperlinks.txt)
- dataset_generation/renumber_paper.py (renames the papers like 0.tex, 1.tex, 2.tex so on)
Using this specified process, we downloaded 4-5 GB source files for papers since source files include images etc. which are not need for our purpose. At the end, we have 799 latex files each for an academic paper. Before preprocessing, this is approximately equal to 46 MB of latex.
- Papers are licensed under one of Creative Common licenses. For details: https://arxiv.org/help/license
- The papers in the dataset are listed in dataset_generation/selected_papers.txt. The list can be used to give credit to the papers in the dataset.
Dataset is needed to be preprocessed because of noise such as created by comments and non-UTF characters. Therefore, we used preprocess_char.py to delete comments and characters that used below a certain threshold, in our experiments it is 100.
For our baseline model, we decided to use character level embedding. The details of the preprocessed char-based dataset is given below.
| Feature | Value |
|---|---|
| Number of Unique Token | 102 |
| Number of Token | 37,921,928 |
| Lower-case to Upper-case Ratio | 23.95 |
| Word to Non-word Ratio | 3.17 |
- Tensorflow 1.12
- NumPy
- TexSoup (for dataset preparation)
- BeautifulSoup (for dataset preparation)
The rnn model described in the blog post "The Unreasonable Effectiveness of Recurrent Neural Networks"[1]
After preparing the dataset, run char-rnn.py to train the model.
When training is over, run generate_text.py. This script will load the last checkpoint and generate a number of characters using the learned parameters.
From Transformer Model [2], the parts related to translation problem is deleted.
After preparing the dataset, run simplified_transformer/simplified_transformer.py to train the model.
When training is over, run simplified_transformer/generate_text.py. This script will load the last checkpoint and generate a number of characters using the learned parameters.
[1] The Unreasonable Effectiveness of Recurrent Neural Networks http://karpathy.github.io/2015/05/21/rnn-effectiveness/
[2] Vaswani, Ashish, et al. "Attention is all you need." Advances in Neural Information Processing Systems. 2017.
[3] Nie, Weili, Nina Narodytska, and Ankit Patel. "RelGAN: Relational Generative Adversarial Networks for Text Generation." (2018).