All of the tools are free to use. In order to use our app you need a google account. The google account is required to upload a notebook to Google Colab. We also use Google Drive to permanently store the file for our annotated data (see "Installation"). To create our website we used Anvil. It is NOT required to have an Anvil account to use our website.
To install our system Crypto-Answer please follow these instructions.
-
Follow the installation instructions from before
-
Enable GPU runtime in your Google Colab notebook
-
Run all cells from top to bottom (follow instructions at code cell 3 to mount to google drive for fine-tuning)
-
Our website can be used only after all cells were executed
A video of the execution of our Crypto-Answer QA-system can be seen here.
Crypto-Answer is a Question-Answering (QA) system for cryptocurrency based on Haystack. We fine-tuned the roberta-base-squad2 model from Huggingface and used Haystack's Webcrawler to create our document collection. Additionally we created a customized Pipeline with a combination of the Dense Passage Retriever and Elasticsearch Retriever. To test our QA-system Crypto-Answer we created a Website with Anvil. The Website includes two functions, the running of a query ("ASK") and an update of the data for real time information on cryptocurrencies ("UPDATE CURRENT DATA").
You can read more about it in our paper.
The Webcrawler as an illustration:
An Overview of our Website created with Anvil:
We used probands to create a ground truth of 100 gold-standard question-answer pairs.
With the ground truth we calculated the exact match values. An answer was classified as an exact match, if it included the gold standard answer precisely.
For the answer distribution, the probands had to assess if the answers returned by our QA-system to their questions, satisfied their information need and at what rank.
In total we calculated the exact match values, the answer distribution, the top-n-accuracy as well as the Mean reciprocal rank of our QA-systems paired with different settings for the reader and retriever. All metrics can be viewed here.
You can read more about it in our paper.
For our Question-Answering system a combination of libraries is required. The libraries are free to use and automatically downloaded and installed in our Google Colab notebook. Libraries and tools include: