NeuralNetwork_final.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "00000-ad7125dd-03e3-4496-a7ea-1015e88f475f",
    "deepnote_cell_type": "text-cell-h1",
    "is_collapsed": false,
    "tags": []
   },
   "source": [
    "# Neural Network Notebook"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "00008-05969483-c321-492c-b4b5-7c9769189e4f",
    "deepnote_cell_type": "markdown",
    "tags": []
   },
   "source": [
    "Implementation of a Feed Forward Neural Network for MultiClass Classification from scratch"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "00002-29edc4b4-e78a-45c7-b3f5-79a69718b7fc",
    "deepnote_cell_type": "markdown",
    "tags": []
   },
   "source": [
    "#### Key points from ISL chapter 10.1/10.2\n",
    "- A neural network takes an input vector of p variables X = (X1,X2, . . . ,Xp)\n",
    "- Each node calculates a linear combination of the weights, x_1-->x_n values + the bias term and we call this calculation z\n",
    "\n",
    "- Each node has an activation function. Here we can use the sigmoid function known from Logistic Regression, but the best and most commonly used is the ReLu function.\n",
    "- The activation function takes in z (linear combination) and spits out a value h.\n",
    "- h is the output of our activation function that will be used to calculate a new z value in the upcoming layer.\n",
    "- In the output layer we have the number of nodes equal to the number of classes we want to predict. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "00003-07d2a557-7521-47bf-818b-f6d1b5a23b97",
    "deepnote_cell_type": "markdown",
    "tags": []
   },
   "source": [
    "## Hyperparamters we can change\n",
    "\n",
    "- Number of Layers and Nodes (structure)\n",
    "- Epochs \n",
    "- Learning Rate \n",
    "- Batches\n",
    "- Activation Function\n",
    "- Weight Initialisation\n",
    "- Biases"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "00004-2f74eec8-485c-4354-a57c-0ce81045b648",
    "deepnote_cell_type": "markdown",
    "tags": []
   },
   "source": [
    "### Imports:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "cell_id": "00002-4a2070b5-0747-4b74-b9d7-316158b237af",
    "deepnote_cell_type": "code",
    "deepnote_to_be_reexecuted": false,
    "execution_millis": 2490,
    "execution_start": 1640699725332,
    "source_hash": "17776391",
    "tags": []
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "import matplotlib.colors\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.metrics import accuracy_score\n",
    "from tqdm import tqdm\n",
    "from tqdm import tqdm_notebook\n",
    "from sklearn import preprocessing\n",
    "from sklearn.utils import shuffle\n",
    "import math\n",
    "from sklearn.decomposition import PCA "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "00006-c18c9be9-217d-40fd-a388-9d186328e314",
    "deepnote_cell_type": "markdown",
    "tags": []
   },
   "source": [
    "#### Test Data set - Sklearn Iris Data set "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "cell_id": "00004-3efad64d-34b6-4d4f-9400-c88edbb806a2",
    "deepnote_cell_type": "code",
    "deepnote_output_heights": [
     117.125
    ],
    "deepnote_to_be_reexecuted": false,
    "execution_millis": 1,
    "execution_start": 1640699727829,
    "source_hash": "1e56039a",
    "tags": []
   },
   "outputs": [],
   "source": [
    "# ###### IRIS #######\n",
    "# from sklearn import datasets\n",
    "# iris = datasets.load_iris()\n",
    "# X_raw = iris.data\n",
    "# y = iris.target\n",
    "# X = preprocessing.normalize(X_raw)\n",
    "# X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=69)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "00008-85912dee-641f-4e0c-b18a-8b20bf6817aa",
    "deepnote_cell_type": "markdown",
    "tags": []
   },
   "source": [
    "### Final Data Preprocessing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "cell_id": "00008-286b10e5-c245-4076-8e71-77a9d69a5fca",
    "deepnote_cell_type": "code",
    "deepnote_output_heights": [
     251.4375
    ],
    "deepnote_to_be_reexecuted": false,
    "execution_millis": 21,
    "execution_start": 1640699727857,
    "source_hash": "470f0877",
    "tags": []
   },
   "outputs": [],
   "source": [
    "# training sets\n",
    "X_train = pd.read_csv('data/X_train.csv')\n",
    "X_train_norm = pd.read_csv('data/X_train_normalised.csv')\n",
    "X_train_synth = pd.read_csv('data/X_train_synth.csv')\n",
    "\n",
    "X_train = X_train.to_numpy()\n",
    "X_train_norm = X_train_norm.to_numpy()\n",
    "X_train_synth = X_train_synth.to_numpy()\n",
    "\n",
    "y_train = pd.read_csv('data/y_train.csv')\n",
    "y_train_synth = pd.read_csv('data/y_train_synth.csv')\n",
    "y_train = y_train.to_numpy()\n",
    "y_train_synth = y_train_synth.to_numpy()\n",
    "y_train = y_train.flatten()\n",
    "y_train_synth = y_train_synth.flatten()\n",
    "\n",
    "\n",
    "# testing sets\n",
    "X_test = pd.read_csv('data/X_test.csv')\n",
    "X_test_norm = pd.read_csv('data/X_test_normalised.csv')\n",
    "\n",
    "X_test = X_test.to_numpy()\n",
    "X_test_norm = X_test_norm.to_numpy()\n",
    "\n",
    "y_test = pd.read_csv('data/y_test.csv')\n",
    "y_test = y_test.to_numpy()\n",
    "y_test = y_test.flatten()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "### PCA\n",
    "\n",
    "pca = PCA(n_components=6)\n",
    "pca_youngboy = pca.fit(X_train_norm)\n",
    "X_train_pca = pca_youngboy.transform(X_train_norm)\n",
    "X_test_pca = pca_youngboy.transform(X_test_norm)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "cell_id": "00002-c56195f8-26c5-4e65-9ca3-4bb81c272cdc",
    "deepnote_cell_type": "code",
    "deepnote_output_heights": [
     78.75
    ],
    "deepnote_to_be_reexecuted": false,
    "execution_millis": 20,
    "execution_start": 1640699727950,
    "source_hash": "20c9915c",
    "tags": []
   },
   "outputs": [],
   "source": [
    "class NN: \n",
    "\n",
    "    def __init__(self,hidden_layers, nodes_in_hidden_layers, verbose:bool):\n",
    "        self.structure = [nodes_in_hidden_layers] * hidden_layers\n",
    "        self.verbose = verbose\n",
    "\n",
    "        # print flags for debugging \n",
    "        self.forprop_print0 = True \n",
    "        self.forprop_print1 = True \n",
    "        self.backprop_print = True \n",
    "\n",
    "    def activation_function(self,z,choice:str): # z is a 2d vector\n",
    "        if choice == 'relu':\n",
    "            a = []\n",
    "            for _ in z:\n",
    "                rel = [max(0.0,val) for val in _]\n",
    "                a.append(rel)   \n",
    "            return np.atleast_2d(a) \n",
    "        if choice == 'leakyrelu':\n",
    "            a = []\n",
    "            for _ in z:\n",
    "                rel = [max(0.01*val,val) for val in _]\n",
    "                a.append(rel)   \n",
    "            return np.atleast_2d(a) \n",
    "        if choice == 'sigmoid':\n",
    "            return 1.0/(1.0+ np.exp(-z))\n",
    "        if choice == 'softmax':\n",
    "            a = []\n",
    "            for _ in z:\n",
    "                e = np.exp(z[0])\n",
    "                a.append(e/e.sum())   \n",
    "            return np.atleast_2d(a) \n",
    "            \n",
    "        \n",
    "    def deriv_activation(self,a, choice): # a is a  2d vector \n",
    "        if choice == 'relu':\n",
    "            deriv = []\n",
    "            for _ in a:\n",
    "                rel = [1 if val >= 0 else 0 for val in _]\n",
    "                deriv.append(rel)   \n",
    "            return np.atleast_2d(deriv) \n",
    "        if choice == 'leakyrelu':\n",
    "            deriv = []\n",
    "            for _ in a:\n",
    "                rel = [1 if val >= 0 else 0.001 for val in _]\n",
    "                deriv.append(rel)   \n",
    "            return np.atleast_2d(deriv)  \n",
    "        if choice == 'sigmoid':\n",
    "            ones = np.ones(len(a))\n",
    "            return (a*(ones-a))\n",
    "        if choice == 'softmax':\n",
    "            return np.diag(a[0]) - (a * a.T) \n",
    "\n",
    "    def loss_function(self,p,y, choice): # takes p:vector , y:vector of ohe\n",
    "        #assert np.shape(p) == np.shape(y),f'p {np.shape(p)} and y {np.shape(y)} must be same dimention '\n",
    "        loss = 0\n",
    "        if choice == 'ce':               # use ce if softmax\n",
    "            for index in range(len(p)):\n",
    "                tru = np.argmax(y[index])\n",
    "                loss += -np.log(p[index][tru]) \n",
    "            return loss\n",
    "        if choice == 'mse':\n",
    "            return #mse\n",
    "\n",
    "    def one_hot_encode(self,y):\n",
    "        vector = np.zeros(self.classes)     # a 0 vector with 1 in the position of the true class\n",
    "        vector[y] = 1\n",
    "        return vector                           # (1,9)\n",
    "\n",
    "    def map_ys(self,y):\n",
    "        y_map = {1:0,2:1,3:2,5:3,6:4,7:5}\n",
    "        mapped = []\n",
    "        for c in y:\n",
    "            mapped.append(y_map[c])\n",
    "        return mapped\n",
    "\n",
    "    def unmap_ys(p):\n",
    "        p_unmap = {0:1,1:2,2:3,3:5,4:6,5:7}\n",
    "        for index, c in enumerate(p):\n",
    "            p[index] = p_unmap[c]\n",
    "        return p\n",
    "    \n",
    "    def load_train(self, X_train, y_train, bias:bool, initialise:bool):\n",
    "        self.classes = len(np.unique(y_train))\n",
    "        self.X_train = X_train\n",
    "        y_train = self.map_ys(y_train)\n",
    "        self.y_train = [self.one_hot_encode(y) for y in y_train] #self.map_ys(y_train)\n",
    "        self.structure.insert(0,self.X_train.shape[1])\n",
    "        self.structure.append(self.classes)            # structure = number of nodes in each layer [features, hidden,..., hidden, outputs]\n",
    "        \n",
    "\n",
    "        if initialise:                                 # Initialise weights\n",
    "            self.neurons = [None]*len(self.structure)  # Stores A values from most recent forward prop\n",
    "            self.weights = []                          # Stores all weights\n",
    "            for layer in range(len(self.structure)-1): \n",
    "                # 2d array with (9,n) for layer 1\n",
    "                w = (2 * (np.random.random([self.structure[layer],self.structure[layer+1]]))) - 1\n",
    "                self.weights.append(w) \n",
    "\n",
    "        ##############################################\n",
    "        if self.verbose: print(f'INITIAL: training data: X {np.shape(self.X_train)} y {np.shape(self.y_train)}, structure: {self.structure}, weights {[np.shape(w) for w in self.weights]}')\n",
    "        ##############################################\n",
    "\n",
    "    def fit(self, alpha, epochs,batch_size):\n",
    "        prev_loss = 1000000000000\n",
    "        self.losses = []\n",
    "        self.accuracies = []\n",
    "        \n",
    "        for epoch in range(epochs):\n",
    "            self.X_train, self.y_train= shuffle(self.X_train, self.y_train)\n",
    "            \n",
    "            loss = 0 # cLosst\n",
    "            n_c = 0 # num correct predictions\n",
    "            n_w = 0 # num wrong predictions\n",
    "\n",
    "            batch_start = 0\n",
    "            batch_num = 0\n",
    "            batch_end = batch_start + batch_size\n",
    "            \n",
    "            while batch_start < self.X_train.shape[0]:            # iterate through a the dataset in batches\n",
    "                \n",
    "                # set x and y to be propagated \n",
    "                x = self.X_train[batch_start:batch_end]         # (10,4)\n",
    "                y = self.y_train[batch_start:batch_end]         # list of ohe vectors\n",
    "\n",
    "                # single forward pass for one data point\n",
    "                self.forward_propagate(x)\n",
    "                \n",
    "                # prediction vector is last layer of 'a's\n",
    "                p = self.neurons[-1]\n",
    "\n",
    "                # increment the loss of the epoch\n",
    "                loss += self.loss_function(p,y,'ce')\n",
    "                \n",
    "                # make a prediction in order to calculate accuracy of epoch\n",
    "                for index in range(len(p)):\n",
    "                    if np.argmax(p[index]) == np.argmax(y[index]):\n",
    "                        n_c += 1\n",
    "                    else: n_w += 1 \n",
    "\n",
    "                # deltas reveal how much to update weights by\n",
    "                deltas = self.back_propagate(y)\n",
    "                for i in range(len(self.weights)): # 0,1,2                    \n",
    "                    self.weights[i] -= (alpha * deltas[i])                                                            # print(f'delta {i} {deltas[i]} {np.shape(deltas[i])}')   # print(f'weights {i} {self.weights[i]} {np.shape(self.weights[i])}')      # print(f'weights {i} {self.weights[i]} {np.shape(self.weights[i])}')\n",
    "                    \n",
    "                ##################################################################################\n",
    "                if self.verbose and batch_size == 5: print(f'BATCH 5: inputs = {np.shape(x)}, y = {y}{np.shape(y)}, p = {p}{np.shape(p)}, deltas = {np.shape(deltas)}')\n",
    "                ##################################################################################\n",
    "\n",
    "                batch_start += batch_size\n",
    "                batch_end += batch_size\n",
    "                batch_num += 1\n",
    "                \n",
    "                if batch_end > self.X_train.shape[0]: batch_end = self.X_train.shape[0]\n",
    "\n",
    "            # give overview and update loss + accuracy \n",
    "            loss = loss/self.X_train.shape[0]               #normalise loss to X size \n",
    "            accuracy = n_c/self.X_train.shape[0]\n",
    "            \n",
    "            ########################################################################################\n",
    "            if self.verbose: print(f'Epoch {epoch}: Loss = {loss}, Accuracy = {n_c}/{n_c+n_w} {n_w} {accuracy}, {np.shape(self.neurons[-1])}, {np.shape(y)}')\n",
    "            ########################################################################################\n",
    "\n",
    "            self.losses.append(loss)\n",
    "            self.accuracies.append(accuracy)\n",
    "            \n",
    "            # early stopping \n",
    "            if epoch > 200:\n",
    "                if loss > (prev_loss*1.1) or math.isnan(loss):\n",
    "                    break   \n",
    "                \n",
    "            prev_loss = loss    \n",
    "        \n",
    "        \n",
    "\n",
    "    def forward_propagate(self, x):\n",
    "        # store input features \n",
    "        self.neurons[0] = x  # (10,4)\n",
    "         \n",
    "        # propagate each layer in the network\n",
    "        for layer in range(len(self.structure)-1): # 0,1,2\n",
    "            \n",
    "            # calculate 'z' as linear combination of previous 'z'/features and weights\n",
    "            z = self.neurons[layer] @ self.weights[layer]    #(10,4) @ (4,2) = (10,2)\n",
    "            \n",
    "            # compute 'a' and store with alternative activation for last layer\n",
    "            if layer != len(self.structure)-2: \n",
    "                a = self.activation_function(z,'relu')\n",
    "            else:\n",
    "                a = self.activation_function(z,'softmax')\n",
    "            self.neurons[layer+1] = a \n",
    "\n",
    "            #######################################################\n",
    "            if self.verbose and self.forprop_print0:\n",
    "                print(f'FP First Layer z: {np.shape(self.neurons[layer])} @ {np.shape(self.weights[layer])} = {np.shape(z)}, a: {np.shape(a)}')\n",
    "                self.forprop_print0 = False\n",
    "            #######################################################\n",
    "        if self.verbose and self.forprop_print1:\n",
    "            print(f'FP Last Layer z: {np.shape(self.neurons[len(self.structure)-2])} @ {np.shape(self.weights[len(self.structure)-2])} = {np.shape(z)}, a: {np.shape(a)}')\n",
    "            self.forprop_print1 = False\n",
    "        #######################################################\n",
    "\n",
    "\n",
    "    def back_propagate(self, y):\n",
    "        \n",
    "        # compute p, y and last layer delta\n",
    "        p = self.neurons[-1]\n",
    "        errors = [p - y] \n",
    "        deltas = [self.neurons[-2].T @ (p - y)] \n",
    "        \n",
    "        # shortcut for computing each layer's delta without needing Jacobian Matrix Multiplication -- output->input        \n",
    "        for l in range(len(self.neurons) - 2, 0, -1): #2,1,2,1,2,1\n",
    "            # gradient of previous layer deltas with respect to Loss and current layer A\n",
    "            dXdA = np.atleast_2d(errors[-1])  #(10,3)\n",
    "            \n",
    "            # current layer delta\n",
    "            dAdW = dXdA @ self.weights[l].T                    #(10,2)          \n",
    "            dZ = self.deriv_activation(dAdW,'relu')       #(10,2)            \n",
    "            dLdW = dZ * dAdW                                   #(10,2)                            \n",
    "\n",
    "            # append deltas to be used by next layer and for weight update\n",
    "            errors.append(dLdW)\n",
    "            deltas.append(self.neurons[l-1].T @ dLdW)                       \n",
    "\n",
    "            #######################################################\n",
    "            if self.verbose and self.backprop_print:\n",
    "                print(f'BP: p - y = {np.shape(p-y)}, dXdA {np.shape(dXdA)}, dAdW {np.shape(dAdW)}, dZ {np.shape(dZ)}, dLdW {np.shape(dLdW)}')\n",
    "        self.backprop_print = False \n",
    "            #######################################################\n",
    "\n",
    "        # return gradients needed for weight updates in order from layer input->output\n",
    "        deltas.reverse()\n",
    "        return deltas\n",
    "        \n",
    "    def predict(self,X,Y):\n",
    "        Y = self.map_ys(Y)\n",
    "        n_c = 0 # number of correct predictions\n",
    "        c = 0   # total loss\n",
    "\n",
    "        for index, datapoint in enumerate(X):\n",
    "            \n",
    "            x = np.atleast_2d(datapoint)      # features (1,4)\n",
    "            y = Y[index]                      # single ohe vector (1,3)\n",
    "            y_ohe = self.one_hot_encode(y)\n",
    "            \n",
    "            # forward propagate\n",
    "            self.forward_propagate(x)\n",
    "            \n",
    "            # update loss\n",
    "            p = self.neurons[-1]\n",
    "\n",
    "            loss = self.loss_function(p,y_ohe,'ce')\n",
    "            c += loss\n",
    "            \n",
    "            if np.argmax(p)== y:\n",
    "                n_c+=1\n",
    "            \n",
    "            #print(f'Datapoint {index}: class: {y}, pred = {np.argmax(p)}, loss = {loss}, confidence = {self.neurons[-1]}')\n",
    "        c = c/len(X)    \n",
    "        accuracy = n_c/len(X)\n",
    "        print(f'Total Test Loss: {c}, Accuracy: {accuracy}')\n",
    "\n",
    "    def plot(self,l,n,a,b,d):\n",
    "        fig, ax = plt.subplots(1,2,figsize = (10,5))\n",
    "        \n",
    "        ax[0] = ax[0] \n",
    "        ax[1] = ax[1]\n",
    "        \n",
    "        ax[0].plot(range(len(self.losses)),self.losses)\n",
    "        ax[0].set_title(f'Training Loss Plot on {d} with alpha: {a}')\n",
    "        ax[0].set_xlabel('epochs')\n",
    "        ax[0].set_ylabel('loss')\n",
    "\n",
    "        ax[1].plot(range(len(self.accuracies)),self.accuracies)\n",
    "        ax[1].set_title(f'Training Accuracy Plot ')\n",
    "        ax[1].set_xlabel('epochs')\n",
    "        ax[1].set_ylabel('accuracy')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "cell_id": "00000-8b5d91e3-78e3-4a1a-86ac-4ee169c7a638",
    "deepnote_cell_type": "code",
    "deepnote_output_heights": [
     78.75,
     375.25,
     375.25,
     498,
     498,
     498,
     498,
     498,
     498,
     498,
     498,
     498,
     498,
     498,
     498,
     498,
     498,
     498,
     498,
     498,
     498
    ],
    "deepnote_to_be_reexecuted": false,
    "execution_millis": 132991,
    "execution_start": 1640700025594,
    "source_hash": "d956ac3",
    "tags": []
   },
   "outputs": [],
   "source": [
    "np.random.seed(9)\n",
    "\n",
    "Xs = [X_train_synth, X_train_norm, X_train_pca]\n",
    "ys = [y_train_synth, y_train, y_train]\n",
    "a = 0.0001\n",
    "l = 1\n",
    "n = 25\n",
    "b = 1\n",
    "d = 1\n",
    "\n",
    "Model = NN(l,n,verbose=False)\n",
    "Model.load_train(Xs[d], ys[d], bias = False, initialise = True)\n",
    "Model.fit(alpha = a, epochs = 3000, batch_size=b)\n",
    "Model.plot(l,n,a,b,d)\n",
    "\n",
    "Model.predict(X_test_norm,y_test)\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Model.predict(X_test_norm,y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# ls =[1,2]\n",
    "# ns =[5,15,25,30]\n",
    "# alphas = [0.01,0.001,0.0001,0.00001]\n",
    "# batches = [1,3,6,10,50,149]\n",
    "# for l in ls:\n",
    "#     for n in ns:\n",
    "#         for a in alphas:\n",
    "#             for b in batches:\n",
    "#                 for d in range(len(Xs)):\n",
    "#                     np.random.seed(9)\n",
    "#                     Model = NN(l,n,verbose=False)\n",
    "#                     Model.load_train(Xs[d], ys[d], bias = False, initialise = True)\n",
    "#                     Model.fit(alpha = a, epochs = 300, batch_size=b)\n",
    "#                     if d == 2: Model.predict(X_test_pca,y_test)\n",
    "#                     else: Model.predict(X_test_norm,y_test)\n",
    "#                     Model.plot(l,n,a,b,d)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "created_in_deepnote_cell": true,
    "deepnote_cell_type": "markdown",
    "tags": []
   },
   "source": []
  }
 ],
 "metadata": {
  "deepnote": {
   "is_reactive": false
  },
  "deepnote_execution_queue": [
   {
    "cellId": "00000-8b5d91e3-78e3-4a1a-86ac-4ee169c7a638",
    "msgId": "07301496-0fd9-44e9-b4b1-b0b4edec4000",
    "sessionId": "f3296ff2-14a1-4fab-bad3-3e8daa167ebe"
   }
  ],
  "deepnote_notebook_id": "40f4777c-531b-42da-949b-3145361d869c",
  "interpreter": {
   "hash": "b3ba2566441a7c06988d0923437866b63cedc61552a5af99d1f4fb67d367b25f"
  },
  "kernelspec": {
   "display_name": "Python 3.8.5 64-bit ('base': conda)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  },
  "orig_nbformat": 2
 },
 "nbformat": 4,
 "nbformat_minor": 2
}