Transformers-for-NLP-2nd-Edition

@copyright 2022, 2023, Packt Publishing, Denis Rothman

Contact me for any question you have on LinkedIn
Get the book on Amazon

Under the hood working of transformers, fine-tuning GPT-3 models, DeBERTa, vision models, and the start of Metaverse, using a variety of NLP platforms: Hugging Face, OpenAI API, Trax, and AllenNLP. A BONUS directory containing OpenAI API notebooks with ChatGPT with GPT-3.5-turbo/GPT-4 and image generation with DALL-E.

Getting started

You can run these notebooks on cloud platforms like Google Colab or your local machine. Note that some chapters require a GPU to run in a reasonable amount of time, so we recommend one of the cloud platforms as they come pre-installed with CUDA.

Running on a cloud platform

To run these notebooks on a cloud platform, just click on one of the badges in the table below:

Chapter	Colab	Kaggle	Gradient	StudioLab
Getting Started with the Architecture of the Transformer Model Multi_Head_Attention_Sub_Layer.ipynb positional_encoding.ipynb
Fine-Tuning BERT Models BERT_Fine_Tuning_Sentence_Classification_GPU.ipynb
Pretraining a RoBERTa Model from Scratch KantaiBERT.ipynb
Downstream NLP Tasks with Transformers Transformer_tasks.ipynb
Machine Translation with the Transformer Trax_translation.ipynb
The Rise of Suprahuman Transformers with GPT-3 Engines Fine_tuning_GPT_3.ipynb Getting_Started_GPT_3.ipynb
Applying Transformers to Legal and Financial Documents for AI Text Summarization Summerizing_Text_with_T5.ipynb
Matching Tokenizers and Datasets Tokenizers.ipynb Training_OpenAI_GPT_2_CH09.ipynb Summarizing_with_ChatGPT.ipynb
Semantic Role Labeling SRL.ipynb Semantic_Role_Labeling_with_ChatGPT.ipynb
Let Your Data Do the Talking: Story, Questions, and Answers QA.ipynb 01_Basic_QA_Pipeline.ipynb Haystack_QA_Pipeline.ipynb
Detecting Customer Emotions to Make Predictions SentimentAnalysis.ipynb
Analyzing Fake News with Transformers Fake_News.ipynb Fake_News_Analysis_with_ChatGPT.ipynb
Interpreting Black Box Transformer Models BertViz.ipynb Understanding_GPT_2_models_with_Ecco.ipynb
From NLP to Task-Agnostic Transformer Models Vision_Transformers.ipynb DALL_E.ipynb
The Emergence of Transformer-Driven Copilots Domain_Specific_GPT_3_Functionality.ipynb KantaiBERT_Recommender.ipynb Vision_Transformer_MLP_Mixer.ipynb Compact_Convolutional_Transformers.ipynb
Appendix III: Generic Text Completion with GPT-2 OpenAI_GPT_2.ipynb
Appendix IV: Custom Text Completion with GPT-2 Training_OpenAI_GPT_2.ipynb

OpenAI Bonus directory including ChatGPT-3.5-Turbo, GPT-4, and DALL-E

Enjoy the present, explore the future!

Here are some tips to get started.

Step 1: Install OpenAI: Install Openai. Make sure you have the latest version of pip:

```python
!pip install --upgrade pip
!pip install openai.
'''

Step 2.Make sure to obtain an API key from OpenAI

Step 3.March 2023. ChatGPT is available in the follwoing notebooks as GPT-3.5-turbo and GPT-4 You can try either version by specifyling the name of model directly in your code. as in notebook #4, Dialog_with_ChatGPT Or you can modify the notebooks for your project and add the model as a parameter to call the dialog function:

```python
#1.Choose a model that you will use for dialog function in your admin user interface:
#convmodel="gpt-3.5-turbo**"
#convmodel="gpt-4"
#2.Call the dialog function with the model you need for a specific task

def dialog(iprompt,convmodel):
    response = openai.ChatCompletion.create(
        model=convmodel,
        messages=iprompt
    )
    return response
 '''

You can also run GPT-3, GPT-3.5-turbo, and GPT-4 in the same notebook to compare the outputs and see which model is the best for a project. For that go to notebook #11, Exploring_GPT_4_API

Bonus	Colab	Kaggle	Gradient	SageMaker Studio Lab
1. Jump-start ChatGPT with the OpenAI API in a few lines of code
Jump_Starting_ChatGPT_with_the_Openai_API.ipynb
2.ChatGPT Plus GPT-4, your coding partner and personal trainer
Getting_Started_with_GPT_4.ipynb
3.Implement the OpenAI API with GPT-3, GPT-3.5-turbo(ChatGPT), and GPT-4
Exploring_GPT_4_API.ipynb
4.Advanced prompt engineering ChatGPT: GPT-3.5-turbo and GPT-4
Prompt_Engineering_as_an_alternative_to_fine_tuning.ipynb
5.Adding speech-to-text and text-to-speech to ChatGPT
Speaking_with_ChatGPT.ipynb
6.A dialog with to speech-to-text and text-to-speech with ChatGPT
Dialog_with_ChatGPT.ipynb
7.OpenAI Whisper and Google gtts for speech-to-text and text-to-speech with ChatGPT
Whisper_and_gttS.ipynb
8.Let's go back to the origins with GPT-2 and ChatGPT
GPT_2_and_ChatGPT_the_Origins
9.ChatGPT or davinin_instruct? What is best for your project?
ChatGPT_as_a_Cobot_ChatGPT_versus_davinci_instruct.ipynb
10.Build an application that can create, modify and create variations of an image With DALL-E
Generating_images_with_the_OpenAI_DALL_E_API.ipynb

Key Features

Implement models, such as BERT, Reformer, and T5, that outperform classical language models
Compare NLP applications using GPT-3, GPT-2, and other transformers
Analyze advanced use cases, including polysemy, cross-lingual learning, and computer vision. A GitHub BONUS directory with SOA ChatGPT and DALL-E notebooks.

Book Description

Transformers are a game-changer for natural language understanding (NLU) and have become one of the pillars of artificial intelligence.

Transformers for Natural Language Processing, 2nd Edition, investigates deep learning for machine translations, language modeling, question-answering, and many more NLP domains with transformers.

An Industry 4.0 AI specialist needs to be adaptable; knowing just one NLP platform is not enough anymore. Different platforms have different benefits depending on the application, whether it's cost, flexibility, ease of implementation, results, or performance. In this book, we analyze numerous use cases with Hugging Face, Google Trax, OpenAI, and AllenNLP.

This book takes transformers' capabilities further by combining multiple NLP techniques, such as sentiment analysis, named entity recognition, and semantic role labeling, to analyze complex use cases, such as dissecting fake news on Twitter. Also, see how transformers can create code using just a brief description.

By the end of this NLP book, you will understand transformers from a cognitive science perspective and be proficient in applying pretrained transformer models to various datasets.

What you will learn

Discover new ways of performing NLP techniques with the latest pretrained transformers
Grasp the workings of the original Transformer, GPT-3, BERT, T5, DeBERTa, and Reformer
Create language understanding Python programs using concepts that outperform classical deep learning models
Apply Python, TensorFlow, and PyTorch programs to sentiment analysis, text summarization, speech recognition, machine translations, and more
Measure the productivity of key transformers to define their scope, potential, and limits in production

Who This Book Is For

If you want to learn about and apply transformers to your natural language (and image) data, this book is for you.

A good understanding of NLP, Python, and deep learning is required to benefit most from this book. Many platforms covered in this book provide interactive user interfaces, which allow readers with a general interest in NLP and AI to follow several chapters of this book.

Table of Contents

1.What are Transformers?
2.Getting Started with the Architecture of the Transformer Model
3.Fine-Tuning BERT models
4.Pretraining a RoBERTa Model from Scratch
5.Downstream NLP Tasks with Transformers
6.Machine Translation with the Transformer
7.The Rise of Suprahuman Transformers with GPT-3 Engines
8.Applying Transformers to Legal and Financial Documents for AI Text Summarization
9.Matching Tokenizers and Datasets
10.Semantic Role Labeling with BERT-Based Transformers
11.Let Your Data Do the Talking: Story, Questions, and Answers
12.Detecting Customer Emotions to Make Predictions
13.Analyzing Fake News with Transformers
14.Interpreting Black Box Transformer Models
15.From NLP to Task-Agnostic Transformer Models
16.The Emergence of Transformer-Driven Copilots
Appendix I: Terminology of Transformer Models
Appendix II: Hardware Constraints for Transformer Models
And more!