1. data_cleaning.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "# Loan Payment Status Prediction\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Table of Contents\n",
    "---\n",
    "1. [Introduction](#introduction)\n",
    "2. [Key Questions](#keyquestions)\n",
    "3. [Methods](#methods)\n",
    "4. [Feature Overview](#featureoverview)\n",
    "5. [Loading and Setup](#loadingandsetup)\n",
    "6. [Assessment](#assessment)\n",
    "7. [Handling Missing Values](#missing)\n",
    "8. [Handling Features](#features)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction <a class=\"anchor\" id=\"introduction\"></a>\n",
    "---\n",
    "\n",
    "This project aims to build a machine learning model to accurately predict the future status of a loan payment. The goal is to proactively manage risk, optimize intervention strategies throughout the loan cycle, and minimize potential financial losses."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Key Questions <a class=\"anchor\" id=\"keyquestions\"></a>\n",
    "---\n",
    "\n",
    "#### The analysis aims to answer this one question:\n",
    "\n",
    " Using machine learning, how might we accurately predict the future status of a loan payment (paid or default), to improve risk management and reduce financial losses?\n",
    "\n",
    " This question serves as the foundation for the analysis and provide a framework to explore the relationships and insights of the loan dataset derived from LendingClub Website: https://www.lendingclub.com/."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Methods <a class=\"anchor\" id=\"methods\"></a>\n",
    "---\n",
    "\n",
    "We used Python programming language and its libraries such as pandas for data manipulation, matplotlib for data visualization, and sklearn for machine learning and data preprocessing. We also used Jupyter notebook as our integrated development environment (IDE) which allows for interactive programming and visualization."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Feature Overview <a class=\"anchor\" id=\"featureoverview\"></a>\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<table border=\"1\" class=\"dataframe\">\n",
    "  <thead>\n",
    "    <tr style=\"text-align: right;\">\n",
    "      <th></th>\n",
    "      <th>Columns</th>\n",
    "      <th>Description</th>\n",
    "    </tr>\n",
    "  </thead>\n",
    "  <tbody>\n",
    "    <tr>\n",
    "      <th>1</th>\n",
    "      <td>acc_now_delinq</td>\n",
    "      <td>The number of accounts on which the borrower is now delinquent.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>2</th>\n",
    "      <td>acc_open_past_24mths</td>\n",
    "      <td>Number of trades opened in the past 24 months.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>3</th>\n",
    "      <td>addr_state</td>\n",
    "      <td>The state provided by the borrower in the loan application.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>4</th>\n",
    "      <td>all_util</td>\n",
    "      <td>Balance to credit limit on all trades.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>5</th>\n",
    "      <td>annual_inc</td>\n",
    "      <td>The self-reported annual income provided by the borrower during registration.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>6</th>\n",
    "      <td>annual_inc_joint</td>\n",
    "      <td>The combined self-reported annual income provided by the co-borrowers during registration.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>7</th>\n",
    "      <td>application_type</td>\n",
    "      <td>Indicates whether the loan is an individual application or a joint application with two co-borrowers.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>8</th>\n",
    "      <td>avg_cur_bal</td>\n",
    "      <td>Average current balance of all accounts.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>9</th>\n",
    "      <td>bc_open_to_buy</td>\n",
    "      <td>Total open to buy on revolving bank cards.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>10</th>\n",
    "      <td>bc_util</td>\n",
    "      <td>Ratio of total current balance to high credit/credit limit for all bank card accounts.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>11</th>\n",
    "      <td>chargeoff_within_12_mths</td>\n",
    "      <td>Number of charge-offs within 12 months.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>12</th>\n",
    "      <td>collection_recovery_fee</td>\n",
    "      <td>Post charge-off collection fee.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>13</th>\n",
    "      <td>collections_12_mths_ex_med</td>\n",
    "      <td>Number of collections in 12 months excluding medical collections.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>14</th>\n",
    "      <td>delinq_2yrs</td>\n",
    "      <td>The number of 30+ days past-due incidences of delinquency in the borrower's credit file for the past 2 years.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>15</th>\n",
    "      <td>delinq_amnt</td>\n",
    "      <td>The past-due amount owed for the accounts on which the borrower is now delinquent.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>16</th>\n",
    "      <td>desc</td>\n",
    "      <td>Loan description provided by the borrower.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>17</th>\n",
    "      <td>dti</td>\n",
    "      <td>A ratio calculated using the borrower's total monthly debt payments on the total debt obligations, excluding mortgage and the requested LC loan, divided by the borrower's self-reported monthly income.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>18</th>\n",
    "      <td>dti_joint</td>\n",
    "      <td>A ratio calculated using the co-borrowers' total monthly payments on the total debt obligations, excluding mortgages and the requested LC loan, divided by the co-borrowers' combined self-reported monthly income.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>19</th>\n",
    "      <td>earliest_cr_line</td>\n",
    "      <td>The month the borrower's earliest reported credit line was opened.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>20</th>\n",
    "      <td>emp_length</td>\n",
    "      <td>Employment length in years. Possible values are between 0 and 10 where 0 means less than one year and 10 means ten or more years.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>21</th>\n",
    "      <td>emp_title</td>\n",
    "      <td>The job title supplied by the Borrower when applying for the loan.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>22</th>\n",
    "      <td>fico_range_high</td>\n",
    "      <td>The upper boundary range the borrower's FICO at loan origination belongs to.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>23</th>\n",
    "      <td>fico_range_low</td>\n",
    "      <td>The lower boundary range the borrower's FICO at loan origination belongs to.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>24</th>\n",
    "      <td>funded_amnt</td>\n",
    "      <td>The total amount committed to that loan at that point in time.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>25</th>\n",
    "      <td>funded_amnt_inv</td>\n",
    "      <td>The total amount committed by investors for that loan at that point in time.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>26</th>\n",
    "      <td>grade</td>\n",
    "      <td>LC assigned loan grade.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>27</th>\n",
    "      <td>home_ownership</td>\n",
    "      <td>The home ownership status provided by the borrower during registration. Our values are: RENT, OWN, MORTGAGE, OTHER.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>28</th>\n",
    "      <td>id</td>\n",
    "      <td>A unique LC assigned ID for the loan listing.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>29</th>\n",
    "      <td>il_util</td>\n",
    "      <td>Ratio of total current balance to high credit/credit limit on all installment accounts.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>30</th>\n",
    "      <td>initial_list_status</td>\n",
    "      <td>The initial listing status of the loan. Possible values are - W, F.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>31</th>\n",
    "      <td>inq_fi</td>\n",
    "      <td>Number of personal finance inquiries.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>32</th>\n",
    "      <td>inq_last_12m</td>\n",
    "      <td>Number of credit inquiries in the past 12 months.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>33</th>\n",
    "      <td>inq_last_6mths</td>\n",
    "      <td>The number of inquiries in the past 6 months (excluding auto and mortgage inquiries).</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>34</th>\n",
    "      <td>installment</td>\n",
    "      <td>The monthly payment owed by the borrower if the loan originates.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>35</th>\n",
    "      <td>int_rate</td>\n",
    "      <td>Interest Rate on the loan.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>36</th>\n",
    "      <td>issue_d</td>\n",
    "      <td>The month which the loan was funded.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>37</th>\n",
    "      <td>last_credit_pull_d</td>\n",
    "      <td>The most recent month LC pulled credit for this loan.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>38</th>\n",
    "      <td>last_fico_range_high</td>\n",
    "      <td>The upper boundary range the borrower's last FICO pulled belongs to.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>39</th>\n",
    "      <td>last_fico_range_low</td>\n",
    "      <td>The lower boundary range the borrower's last FICO pulled belongs to.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>40</th>\n",
    "      <td>last_pymnt_amnt</td>\n",
    "      <td>Last total payment amount received.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>41</th>\n",
    "      <td>last_pymnt_d</td>\n",
    "      <td>Last month payment was received.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>42</th>\n",
    "      <td>loan_amnt</td>\n",
    "      <td>The listed amount of the loan applied for by the borrower. If at some point in time, the credit department reduces the loan amount, then it will be reflected in this value.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>43</th>\n",
    "      <td>loan_status</td>\n",
    "      <td>Current status of the loan.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>44</th>\n",
    "      <td>max_bal_bc</td>\n",
    "      <td>Maximum current balance owed on all revolving accounts.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>45</th>\n",
    "      <td>member_id</td>\n",
    "      <td>A unique LC assigned ID for the borrower member.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>46</th>\n",
    "      <td>mo_sin_old_il_acct</td>\n",
    "      <td>Months since oldest bank installment account opened.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>47</th>\n",
    "      <td>mo_sin_old_rev_tl_op</td>\n",
    "      <td>Months since oldest revolving account opened.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>48</th>\n",
    "      <td>mo_sin_rcnt_rev_tl_op</td>\n",
    "      <td>Months since most recent revolving account opened.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>49</th>\n",
    "      <td>mo_sin_rcnt_tl</td>\n",
    "      <td>Months since most recent account opened.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>50</th>\n",
    "      <td>mort_acc</td>\n",
    "      <td>Number of mortgage accounts.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>51</th>\n",
    "      <td>mths_since_last_delinq</td>\n",
    "      <td>The number of months since the borrower's last delinquency.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>52</th>\n",
    "      <td>mths_since_last_major_derog</td>\n",
    "      <td>Months since most recent 90-day or worse rating.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>53</th>\n",
    "      <td>mths_since_last_record</td>\n",
    "      <td>The number of months since the last public record.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>54</th>\n",
    "      <td>mths_since_rcnt_il</td>\n",
    "      <td>Months since most recent installment accounts opened.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>55</th>\n",
    "      <td>mths_since_recent_bc</td>\n",
    "      <td>Months since most recent bank card account opened.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>56</th>\n",
    "      <td>mths_since_recent_bc_dlq</td>\n",
    "      <td>Months since most recent bank card delinquency.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>57</th>\n",
    "      <td>mths_since_recent_inq</td>\n",
    "      <td>Months since most recent inquiry.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>58</th>\n",
    "      <td>mths_since_recent_revol_delinq</td>\n",
    "      <td>Months since most recent revolving delinquency.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>59</th>\n",
    "      <td>next_pymnt_d</td>\n",
    "      <td>Next scheduled payment date.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>60</th>\n",
    "      <td>num_accts_ever_120_pd</td>\n",
    "      <td>Number of accounts ever 120 or more days past due.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>61</th>\n",
    "      <td>num_actv_bc_tl</td>\n",
    "      <td>Number of currently active bank card accounts.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>62</th>\n",
    "      <td>num_actv_rev_tl</td>\n",
    "      <td>Number of currently active revolving trades.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>63</th>\n",
    "      <td>num_bc_sats</td>\n",
    "      <td>Number of satisfactory bank card accounts.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>64</th>\n",
    "      <td>num_bc_tl</td>\n",
    "      <td>Number of bank card accounts.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>65</th>\n",
    "      <td>num_il_tl</td>\n",
    "      <td>Number of installment accounts.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>66</th>\n",
    "      <td>num_op_rev_tl</td>\n",
    "      <td>Number of open revolving accounts.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>67</th>\n",
    "      <td>num_rev_accts</td>\n",
    "      <td>Number of revolving accounts.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>68</th>\n",
    "      <td>num_rev_tl_bal_gt_0</td>\n",
    "      <td>Number of revolving trades with a balance >0.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>69</th>\n",
    "      <td>num_sats</td>\n",
    "      <td>Number of satisfactory accounts.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>70</th>\n",
    "      <td>num_tl_120dpd_2m</td>\n",
    "      <td>Number of accounts currently 120 days past due (updated in the past 2 months).</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>71</th>\n",
    "      <td>num_tl_30dpd</td>\n",
    "      <td>Number of accounts currently 30 days past due (updated in the past 2 months).</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>72</th>\n",
    "      <td>num_tl_90g_dpd_24m</td>\n",
    "      <td>Number of accounts 90 or more days past due in the last 24 months.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>73</th>\n",
    "      <td>num_tl_op_past_12m</td>\n",
    "      <td>Number of accounts opened in the past 12 months.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>74</th>\n",
    "      <td>open_acc</td>\n",
    "      <td>The number of open credit lines in the borrower's credit file.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>75</th>\n",
    "      <td>open_acc_6m</td>\n",
    "      <td>Number of open trades in the last 6 months.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>76</th>\n",
    "      <td>open_il_12m</td>\n",
    "      <td>Number of installment accounts opened in the past 12 months.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>77</th>\n",
    "      <td>open_il_24m</td>\n",
    "      <td>Number of installment accounts opened in the past 24 months.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>78</th>\n",
    "      <td>open_il_6m</td>\n",
    "      <td>Number of currently active installment trades.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>79</th>\n",
    "      <td>open_rv_12m</td>\n",
    "      <td>Number of revolving trades opened in the past 12 months.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>80</th>\n",
    "      <td>open_rv_24m</td>\n",
    "      <td>Number of revolving trades opened in the past 24 months.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>81</th>\n",
    "      <td>out_prncp</td>\n",
    "      <td>Remaining outstanding principal for the total amount funded.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>82</th>\n",
    "      <td>out_prncp_inv</td>\n",
    "      <td>Remaining outstanding principal for the portion of the total amount funded by investors.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>83</th>\n",
    "      <td>pct_tl_nvr_dlq</td>\n",
    "      <td>Percent of trades never delinquent.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>84</th>\n",
    "      <td>percent_bc_gt_75</td>\n",
    "      <td>Percentage of all bank card accounts > 75% of the limit.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>85</th>\n",
    "      <td>policy_code</td>\n",
    "      <td>\"Publicly available policy_code=1\n",
    "        New products not publicly available policy_code=2\"</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>86</th>\n",
    "      <td>pub_rec</td>\n",
    "      <td>Number of derogatory public records.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>87</th>\n",
    "      <td>pub_rec_bankruptcies</td>\n",
    "      <td>Number of public record bankruptcies.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>88</th>\n",
    "      <td>purpose</td>\n",
    "      <td>A category provided by the borrower for the loan request.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>89</th>\n",
    "      <td>pymnt_plan</td>\n",
    "      <td>Indicates if a payment plan has been put in place for the loan.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>90</th>\n",
    "      <td>recoveries</td>\n",
    "      <td>Post charge-off gross recovery.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>91</th>\n",
    "      <td>revol_bal</td>\n",
    "      <td>Total credit revolving balance.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>92</th>\n",
    "      <td>revol_util</td>\n",
    "      <td>Revolving line utilization rate, or the amount of credit the borrower is using relative to all available revolving credit.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>93</th>\n",
    "      <td>sub_grade</td>\n",
    "      <td>LC assigned loan subgrade.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>94</th>\n",
    "      <td>tax_liens</td>\n",
    "      <td>Number of tax liens.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>95</th>\n",
    "      <td>term</td>\n",
    "      <td>The number of payments on the loan. Values are in months and can be either 36 or 60.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>96</th>\n",
    "      <td>title</td>\n",
    "      <td>The loan title provided by the borrower.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>97</th>\n",
    "      <td>tot_coll_amt</td>\n",
    "      <td>Total collection amounts ever owed.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>98</th>\n",
    "      <td>tot_cur_bal</td>\n",
    "      <td>Total current balance of all accounts.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>99</th>\n",
    "      <td>tot_hi_cred_lim</td>\n",
    "      <td>Total high credit/credit limit.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>100</th>\n",
    "      <td>total_acc</td>\n",
    "      <td>The total number of credit lines currently in the borrower's credit file.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>101</th>\n",
    "      <td>total_bal_ex_mort</td>\n",
    "      <td>Total credit balance excluding mortgage.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>102</th>\n",
    "      <td>total_bc_limit</td>\n",
    "      <td>Total bank card high credit/credit limit.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>103</th>\n",
    "      <td>total_cu_tl</td>\n",
    "      <td>Total current unused revolving credit lines.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>104</th>\n",
    "      <td>total_il_high_credit_limit</td>\n",
    "      <td>Total installment high credit/credit limit.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>105</th>\n",
    "      <td>total_pymnt</td>\n",
    "      <td>Payments received to date for total amount funded.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>106</th>\n",
    "      <td>total_pymnt_inv</td>\n",
    "      <td>Payments received to date for the portion of the total amount funded by investors.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>107</th>\n",
    "      <td>total_rec_int</td>\n",
    "      <td>Interest received to date.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>108</th>\n",
    "      <td>total_rec_late_fee</td>\n",
    "      <td>Late fees received to date.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>109</th>\n",
    "      <td>total_rec_prncp</td>\n",
    "      <td>Principal received to date.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>110</th>\n",
    "      <td>total_rev_hi_lim</td>\n",
    "      <td>Total revolving high credit/credit limit.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>111</th>\n",
    "      <td>url</td>\n",
    "      <td>URL for the LC page with listing data.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>112</th>\n",
    "      <td>verified_status_joint</td>\n",
    "      <td>Indicates if the co-borrowers' joint income was verified by LC.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>113</th>\n",
    "      <td>zip_code</td>\n",
    "      <td>The first 3 numbers of the zip code provided by the borrower in the loan application.</td>\n",
    "    </tr>\n",
    "  </tbody>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Loading and Setup <a class=\"anchor\" id=\"loadingandsetup\"></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/rachelliao/anaconda3/envs/ensemble/lib/python3.8/site-packages/xgboost/compat.py:93: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.\n",
      "  from pandas import MultiIndex, Int64Index\n"
     ]
    }
   ],
   "source": [
    "# Import necessary libraries\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "import scipy.stats as stats\n",
    "from sklearn.feature_selection import SelectKBest, chi2\n",
    "from sklearn.preprocessing import OneHotEncoder\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.metrics import accuracy_score, confusion_matrix, classification_report\n",
    "from sklearn.svm import SVC\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from xgboost import XGBClassifier\n",
    "# Filter warnings\n",
    "from warnings import filterwarnings\n",
    "filterwarnings('ignore')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load the dataset\n",
    "df = pd.read_parquet('dataset/LoanData_2007_to_2018.parquet')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Set up feature information for easy lookup throughout the notebook:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load the data dictionary\n",
    "data_info = pd.read_csv('dataset/data_dictionary.csv', index_col='LoanStatNew')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The self-reported annual income provided by the borrower during registration.\n"
     ]
    }
   ],
   "source": [
    "# Testing out the feature information lookup\n",
    "print(data_info.loc['annual_inc']['Description'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set up self-defined function for feature information\n",
    "def col_info(col_name):\n",
    "    print(data_info.loc[col_name]['Description'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The self-reported annual income provided by the borrower during registration.\n"
     ]
    }
   ],
   "source": [
    "# Testing out the self-defined function\n",
    "col_info('annual_inc')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Methodology:\n",
    "\n",
    "\n",
    "1. Load the data dictionary: The code starts by loading a data dictionary from a CSV file named 'data_dictionary.csv' into a Pandas DataFrame. The data dictionary likely contains information about the columns (features) present in a dataset.\n",
    "2. Explore feature information: The code then tests the data dictionary by printing out the description of a specific feature. It does this by using the Pandas DataFrame's loc method to access the row corresponding to the feature 'annual_inc' and then retrieving the value under the 'Description' column.\n",
    "3. Define a self-defined function: The code defines a function called col_info that takes a column name as input and prints out the description of that column using the data dictionary. This function allows the user to easily obtain information about any feature/column in the dataset.\n",
    "4. Test the self-defined function: The code tests the col_info function by passing the column name 'annual_inc' as an argument and prints out the corresponding description from the data dictionary.\n",
    "\n",
    "Findings:\n",
    "\n",
    "1. The feature 'annual_inc' refers to the \"self-reported annual income\" provided by the borrower during registration. It indicates the income amount reported by the borrower when they applied for a loan or some form of credit. This information is essential for LendClub to assess the borrower's ability to repay the loan and determine the creditworthiness of the individual.\n",
    "\n",
    "2. The data dictionary serves as a reference guide that provides descriptions of each column in the dataset, allowing us to understand the meaning and purpose of each feature and how it relates to the dataset as a whole. With this information, LendClub can make more informed decisions about data analysis, feature engineering, and model building."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Assessment <a class=\"anchor\" id=\"assessment\"></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set display options to show all columns and rows\n",
    "pd.set_option('display.max_columns', None)\n",
    "pd.set_option('display.max_rows', None)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(22511, 151)"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Print the shape of the dataset\n",
    "df.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are 22511 rows, and 151 columns in this dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>member_id</th>\n",
       "      <th>loan_amnt</th>\n",
       "      <th>funded_amnt</th>\n",
       "      <th>funded_amnt_inv</th>\n",
       "      <th>term</th>\n",
       "      <th>int_rate</th>\n",
       "      <th>installment</th>\n",
       "      <th>grade</th>\n",
       "      <th>sub_grade</th>\n",
       "      <th>emp_title</th>\n",
       "      <th>emp_length</th>\n",
       "      <th>home_ownership</th>\n",
       "      <th>annual_inc</th>\n",
       "      <th>verification_status</th>\n",
       "      <th>issue_d</th>\n",
       "      <th>loan_status</th>\n",
       "      <th>pymnt_plan</th>\n",
       "      <th>url</th>\n",
       "      <th>desc</th>\n",
       "      <th>purpose</th>\n",
       "      <th>title</th>\n",
       "      <th>zip_code</th>\n",
       "      <th>addr_state</th>\n",
       "      <th>dti</th>\n",
       "      <th>delinq_2yrs</th>\n",
       "      <th>earliest_cr_line</th>\n",
       "      <th>fico_range_low</th>\n",
       "      <th>fico_range_high</th>\n",
       "      <th>inq_last_6mths</th>\n",
       "      <th>mths_since_last_delinq</th>\n",
       "      <th>mths_since_last_record</th>\n",
       "      <th>open_acc</th>\n",
       "      <th>pub_rec</th>\n",
       "      <th>revol_bal</th>\n",
       "      <th>revol_util</th>\n",
       "      <th>total_acc</th>\n",
       "      <th>initial_list_status</th>\n",
       "      <th>out_prncp</th>\n",
       "      <th>out_prncp_inv</th>\n",
       "      <th>total_pymnt</th>\n",
       "      <th>total_pymnt_inv</th>\n",
       "      <th>total_rec_prncp</th>\n",
       "      <th>total_rec_int</th>\n",
       "      <th>total_rec_late_fee</th>\n",
       "      <th>recoveries</th>\n",
       "      <th>collection_recovery_fee</th>\n",
       "      <th>last_pymnt_d</th>\n",
       "      <th>last_pymnt_amnt</th>\n",
       "      <th>next_pymnt_d</th>\n",
       "      <th>last_credit_pull_d</th>\n",
       "      <th>last_fico_range_high</th>\n",
       "      <th>last_fico_range_low</th>\n",
       "      <th>collections_12_mths_ex_med</th>\n",
       "      <th>mths_since_last_major_derog</th>\n",
       "      <th>policy_code</th>\n",
       "      <th>application_type</th>\n",
       "      <th>annual_inc_joint</th>\n",
       "      <th>dti_joint</th>\n",
       "      <th>verification_status_joint</th>\n",
       "      <th>acc_now_delinq</th>\n",
       "      <th>tot_coll_amt</th>\n",
       "      <th>tot_cur_bal</th>\n",
       "      <th>open_acc_6m</th>\n",
       "      <th>open_act_il</th>\n",
       "      <th>open_il_12m</th>\n",
       "      <th>open_il_24m</th>\n",
       "      <th>mths_since_rcnt_il</th>\n",
       "      <th>total_bal_il</th>\n",
       "      <th>il_util</th>\n",
       "      <th>open_rv_12m</th>\n",
       "      <th>open_rv_24m</th>\n",
       "      <th>max_bal_bc</th>\n",
       "      <th>all_util</th>\n",
       "      <th>total_rev_hi_lim</th>\n",
       "      <th>inq_fi</th>\n",
       "      <th>total_cu_tl</th>\n",
       "      <th>inq_last_12m</th>\n",
       "      <th>acc_open_past_24mths</th>\n",
       "      <th>avg_cur_bal</th>\n",
       "      <th>bc_open_to_buy</th>\n",
       "      <th>bc_util</th>\n",
       "      <th>chargeoff_within_12_mths</th>\n",
       "      <th>delinq_amnt</th>\n",
       "      <th>mo_sin_old_il_acct</th>\n",
       "      <th>mo_sin_old_rev_tl_op</th>\n",
       "      <th>mo_sin_rcnt_rev_tl_op</th>\n",
       "      <th>mo_sin_rcnt_tl</th>\n",
       "      <th>mort_acc</th>\n",
       "      <th>mths_since_recent_bc</th>\n",
       "      <th>mths_since_recent_bc_dlq</th>\n",
       "      <th>mths_since_recent_inq</th>\n",
       "      <th>mths_since_recent_revol_delinq</th>\n",
       "      <th>num_accts_ever_120_pd</th>\n",
       "      <th>num_actv_bc_tl</th>\n",
       "      <th>num_actv_rev_tl</th>\n",
       "      <th>num_bc_sats</th>\n",
       "      <th>num_bc_tl</th>\n",
       "      <th>num_il_tl</th>\n",
       "      <th>num_op_rev_tl</th>\n",
       "      <th>num_rev_accts</th>\n",
       "      <th>num_rev_tl_bal_gt_0</th>\n",
       "      <th>num_sats</th>\n",
       "      <th>num_tl_120dpd_2m</th>\n",
       "      <th>num_tl_30dpd</th>\n",
       "      <th>num_tl_90g_dpd_24m</th>\n",
       "      <th>num_tl_op_past_12m</th>\n",
       "      <th>pct_tl_nvr_dlq</th>\n",
       "      <th>percent_bc_gt_75</th>\n",
       "      <th>pub_rec_bankruptcies</th>\n",
       "      <th>tax_liens</th>\n",
       "      <th>tot_hi_cred_lim</th>\n",
       "      <th>total_bal_ex_mort</th>\n",
       "      <th>total_bc_limit</th>\n",
       "      <th>total_il_high_credit_limit</th>\n",
       "      <th>revol_bal_joint</th>\n",
       "      <th>sec_app_fico_range_low</th>\n",
       "      <th>sec_app_fico_range_high</th>\n",
       "      <th>sec_app_earliest_cr_line</th>\n",
       "      <th>sec_app_inq_last_6mths</th>\n",
       "      <th>sec_app_mort_acc</th>\n",
       "      <th>sec_app_open_acc</th>\n",
       "      <th>sec_app_revol_util</th>\n",
       "      <th>sec_app_open_act_il</th>\n",
       "      <th>sec_app_num_rev_accts</th>\n",
       "      <th>sec_app_chargeoff_within_12_mths</th>\n",
       "      <th>sec_app_collections_12_mths_ex_med</th>\n",
       "      <th>sec_app_mths_since_last_major_derog</th>\n",
       "      <th>hardship_flag</th>\n",
       "      <th>hardship_type</th>\n",
       "      <th>hardship_reason</th>\n",
       "      <th>hardship_status</th>\n",
       "      <th>deferral_term</th>\n",
       "      <th>hardship_amount</th>\n",
       "      <th>hardship_start_date</th>\n",
       "      <th>hardship_end_date</th>\n",
       "      <th>payment_plan_start_date</th>\n",
       "      <th>hardship_length</th>\n",
       "      <th>hardship_dpd</th>\n",
       "      <th>hardship_loan_status</th>\n",
       "      <th>orig_projected_additional_accrued_interest</th>\n",
       "      <th>hardship_payoff_balance_amount</th>\n",
       "      <th>hardship_last_payment_amount</th>\n",
       "      <th>disbursement_method</th>\n",
       "      <th>debt_settlement_flag</th>\n",
       "      <th>debt_settlement_flag_date</th>\n",
       "      <th>settlement_status</th>\n",
       "      <th>settlement_date</th>\n",
       "      <th>settlement_amount</th>\n",
       "      <th>settlement_percentage</th>\n",
       "      <th>settlement_term</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>68407277</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3600.0</td>\n",
       "      <td>3600.0</td>\n",
       "      <td>3600.0</td>\n",
       "      <td>36 months</td>\n",
       "      <td>13.99</td>\n",
       "      <td>123.03</td>\n",
       "      <td>C</td>\n",
       "      <td>C4</td>\n",
       "      <td>leadman</td>\n",
       "      <td>10+ years</td>\n",
       "      <td>MORTGAGE</td>\n",
       "      <td>55000.0</td>\n",
       "      <td>Not Verified</td>\n",
       "      <td>Dec-2015</td>\n",
       "      <td>Fully Paid</td>\n",
       "      <td>n</td>\n",
       "      <td>https://lendingclub.com/browse/loanDetail.acti...</td>\n",
       "      <td>None</td>\n",
       "      <td>debt_consolidation</td>\n",
       "      <td>Debt consolidation</td>\n",
       "      <td>190xx</td>\n",
       "      <td>PA</td>\n",
       "      <td>5.91</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Aug-2003</td>\n",
       "      <td>675.0</td>\n",
       "      <td>679.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>30.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>7.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2765.0</td>\n",
       "      <td>29.7</td>\n",
       "      <td>13.0</td>\n",
       "      <td>w</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>4421.723917</td>\n",
       "      <td>4421.72</td>\n",
       "      <td>3600.00</td>\n",
       "      <td>821.72</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Jan-2019</td>\n",
       "      <td>122.67</td>\n",
       "      <td>None</td>\n",
       "      <td>Mar-2019</td>\n",
       "      <td>564.0</td>\n",
       "      <td>560.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>30.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>Individual</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>0.0</td>\n",
       "      <td>722.0</td>\n",
       "      <td>144904.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>21.0</td>\n",
       "      <td>4981.0</td>\n",
       "      <td>36.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>722.0</td>\n",
       "      <td>34.0</td>\n",
       "      <td>9300.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>20701.0</td>\n",
       "      <td>1506.0</td>\n",
       "      <td>37.2</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>148.0</td>\n",
       "      <td>128.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>69.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>69.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>76.9</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>178050.0</td>\n",
       "      <td>7746.0</td>\n",
       "      <td>2400.0</td>\n",
       "      <td>13734.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>N</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Cash</td>\n",
       "      <td>N</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>68355089</td>\n",
       "      <td>NaN</td>\n",
       "      <td>24700.0</td>\n",
       "      <td>24700.0</td>\n",
       "      <td>24700.0</td>\n",
       "      <td>36 months</td>\n",
       "      <td>11.99</td>\n",
       "      <td>820.28</td>\n",
       "      <td>C</td>\n",
       "      <td>C1</td>\n",
       "      <td>Engineer</td>\n",
       "      <td>10+ years</td>\n",
       "      <td>MORTGAGE</td>\n",
       "      <td>65000.0</td>\n",
       "      <td>Not Verified</td>\n",
       "      <td>Dec-2015</td>\n",
       "      <td>Fully Paid</td>\n",
       "      <td>n</td>\n",
       "      <td>https://lendingclub.com/browse/loanDetail.acti...</td>\n",
       "      <td>None</td>\n",
       "      <td>small_business</td>\n",
       "      <td>Business</td>\n",
       "      <td>577xx</td>\n",
       "      <td>SD</td>\n",
       "      <td>16.06</td>\n",
       "      <td>1.0</td>\n",
       "      <td>Dec-1999</td>\n",
       "      <td>715.0</td>\n",
       "      <td>719.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>22.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>21470.0</td>\n",
       "      <td>19.2</td>\n",
       "      <td>38.0</td>\n",
       "      <td>w</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>25679.660000</td>\n",
       "      <td>25679.66</td>\n",
       "      <td>24700.00</td>\n",
       "      <td>979.66</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Jun-2016</td>\n",
       "      <td>926.35</td>\n",
       "      <td>None</td>\n",
       "      <td>Mar-2019</td>\n",
       "      <td>699.0</td>\n",
       "      <td>695.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>Individual</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>204396.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>19.0</td>\n",
       "      <td>18005.0</td>\n",
       "      <td>73.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>6472.0</td>\n",
       "      <td>29.0</td>\n",
       "      <td>111800.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>9733.0</td>\n",
       "      <td>57830.0</td>\n",
       "      <td>27.1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>113.0</td>\n",
       "      <td>192.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>13.0</td>\n",
       "      <td>17.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>20.0</td>\n",
       "      <td>27.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>22.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>97.4</td>\n",
       "      <td>7.7</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>314017.0</td>\n",
       "      <td>39475.0</td>\n",
       "      <td>79300.0</td>\n",
       "      <td>24667.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>N</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Cash</td>\n",
       "      <td>N</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>68341763</td>\n",
       "      <td>NaN</td>\n",
       "      <td>20000.0</td>\n",
       "      <td>20000.0</td>\n",
       "      <td>20000.0</td>\n",
       "      <td>60 months</td>\n",
       "      <td>10.78</td>\n",
       "      <td>432.66</td>\n",
       "      <td>B</td>\n",
       "      <td>B4</td>\n",
       "      <td>truck driver</td>\n",
       "      <td>10+ years</td>\n",
       "      <td>MORTGAGE</td>\n",
       "      <td>63000.0</td>\n",
       "      <td>Not Verified</td>\n",
       "      <td>Dec-2015</td>\n",
       "      <td>Fully Paid</td>\n",
       "      <td>n</td>\n",
       "      <td>https://lendingclub.com/browse/loanDetail.acti...</td>\n",
       "      <td>None</td>\n",
       "      <td>home_improvement</td>\n",
       "      <td>None</td>\n",
       "      <td>605xx</td>\n",
       "      <td>IL</td>\n",
       "      <td>10.78</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Aug-2000</td>\n",
       "      <td>695.0</td>\n",
       "      <td>699.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>6.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>7869.0</td>\n",
       "      <td>56.2</td>\n",
       "      <td>18.0</td>\n",
       "      <td>w</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>22705.924294</td>\n",
       "      <td>22705.92</td>\n",
       "      <td>20000.00</td>\n",
       "      <td>2705.92</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Jun-2017</td>\n",
       "      <td>15813.30</td>\n",
       "      <td>None</td>\n",
       "      <td>Mar-2019</td>\n",
       "      <td>704.0</td>\n",
       "      <td>700.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>Joint App</td>\n",
       "      <td>71000.0</td>\n",
       "      <td>13.85</td>\n",
       "      <td>Not Verified</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>189699.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>19.0</td>\n",
       "      <td>10827.0</td>\n",
       "      <td>73.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2081.0</td>\n",
       "      <td>65.0</td>\n",
       "      <td>14000.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>31617.0</td>\n",
       "      <td>2737.0</td>\n",
       "      <td>55.9</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>125.0</td>\n",
       "      <td>184.0</td>\n",
       "      <td>14.0</td>\n",
       "      <td>14.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>101.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>10.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>50.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>218418.0</td>\n",
       "      <td>18696.0</td>\n",
       "      <td>6200.0</td>\n",
       "      <td>14877.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>N</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Cash</td>\n",
       "      <td>N</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>66310712</td>\n",
       "      <td>NaN</td>\n",
       "      <td>35000.0</td>\n",
       "      <td>35000.0</td>\n",
       "      <td>35000.0</td>\n",
       "      <td>60 months</td>\n",
       "      <td>14.85</td>\n",
       "      <td>829.90</td>\n",
       "      <td>C</td>\n",
       "      <td>C5</td>\n",
       "      <td>Information Systems Officer</td>\n",
       "      <td>10+ years</td>\n",
       "      <td>MORTGAGE</td>\n",
       "      <td>110000.0</td>\n",
       "      <td>Source Verified</td>\n",
       "      <td>Dec-2015</td>\n",
       "      <td>Current</td>\n",
       "      <td>n</td>\n",
       "      <td>https://lendingclub.com/browse/loanDetail.acti...</td>\n",
       "      <td>None</td>\n",
       "      <td>debt_consolidation</td>\n",
       "      <td>Debt consolidation</td>\n",
       "      <td>076xx</td>\n",
       "      <td>NJ</td>\n",
       "      <td>17.06</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Sep-2008</td>\n",
       "      <td>785.0</td>\n",
       "      <td>789.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>13.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>7802.0</td>\n",
       "      <td>11.6</td>\n",
       "      <td>17.0</td>\n",
       "      <td>w</td>\n",
       "      <td>15897.65</td>\n",
       "      <td>15897.65</td>\n",
       "      <td>31464.010000</td>\n",
       "      <td>31464.01</td>\n",
       "      <td>19102.35</td>\n",
       "      <td>12361.66</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Feb-2019</td>\n",
       "      <td>829.90</td>\n",
       "      <td>Apr-2019</td>\n",
       "      <td>Mar-2019</td>\n",
       "      <td>679.0</td>\n",
       "      <td>675.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>Individual</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>301500.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>23.0</td>\n",
       "      <td>12609.0</td>\n",
       "      <td>70.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>6987.0</td>\n",
       "      <td>45.0</td>\n",
       "      <td>67300.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>23192.0</td>\n",
       "      <td>54962.0</td>\n",
       "      <td>12.1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>36.0</td>\n",
       "      <td>87.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>13.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>13.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>381215.0</td>\n",
       "      <td>52226.0</td>\n",
       "      <td>62500.0</td>\n",
       "      <td>18000.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>N</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Cash</td>\n",
       "      <td>N</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>68476807</td>\n",
       "      <td>NaN</td>\n",
       "      <td>10400.0</td>\n",
       "      <td>10400.0</td>\n",
       "      <td>10400.0</td>\n",
       "      <td>60 months</td>\n",
       "      <td>22.45</td>\n",
       "      <td>289.91</td>\n",
       "      <td>F</td>\n",
       "      <td>F1</td>\n",
       "      <td>Contract Specialist</td>\n",
       "      <td>3 years</td>\n",
       "      <td>MORTGAGE</td>\n",
       "      <td>104433.0</td>\n",
       "      <td>Source Verified</td>\n",
       "      <td>Dec-2015</td>\n",
       "      <td>Fully Paid</td>\n",
       "      <td>n</td>\n",
       "      <td>https://lendingclub.com/browse/loanDetail.acti...</td>\n",
       "      <td>None</td>\n",
       "      <td>major_purchase</td>\n",
       "      <td>Major purchase</td>\n",
       "      <td>174xx</td>\n",
       "      <td>PA</td>\n",
       "      <td>25.37</td>\n",
       "      <td>1.0</td>\n",
       "      <td>Jun-1998</td>\n",
       "      <td>695.0</td>\n",
       "      <td>699.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>12.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>12.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>21929.0</td>\n",
       "      <td>64.5</td>\n",
       "      <td>35.0</td>\n",
       "      <td>w</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>11740.500000</td>\n",
       "      <td>11740.50</td>\n",
       "      <td>10400.00</td>\n",
       "      <td>1340.50</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Jul-2016</td>\n",
       "      <td>10128.96</td>\n",
       "      <td>None</td>\n",
       "      <td>Mar-2018</td>\n",
       "      <td>704.0</td>\n",
       "      <td>700.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>Individual</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>331730.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>14.0</td>\n",
       "      <td>73839.0</td>\n",
       "      <td>84.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>9702.0</td>\n",
       "      <td>78.0</td>\n",
       "      <td>34000.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>27644.0</td>\n",
       "      <td>4567.0</td>\n",
       "      <td>77.5</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>128.0</td>\n",
       "      <td>210.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>12.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>12.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>19.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>12.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>96.6</td>\n",
       "      <td>60.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>439570.0</td>\n",
       "      <td>95768.0</td>\n",
       "      <td>20300.0</td>\n",
       "      <td>88097.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>N</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Cash</td>\n",
       "      <td>N</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         id  member_id  loan_amnt  funded_amnt  funded_amnt_inv        term  \\\n",
       "0  68407277        NaN     3600.0       3600.0           3600.0   36 months   \n",
       "1  68355089        NaN    24700.0      24700.0          24700.0   36 months   \n",
       "2  68341763        NaN    20000.0      20000.0          20000.0   60 months   \n",
       "3  66310712        NaN    35000.0      35000.0          35000.0   60 months   \n",
       "4  68476807        NaN    10400.0      10400.0          10400.0   60 months   \n",
       "\n",
       "   int_rate  installment grade sub_grade                    emp_title  \\\n",
       "0     13.99       123.03     C        C4                      leadman   \n",
       "1     11.99       820.28     C        C1                     Engineer   \n",
       "2     10.78       432.66     B        B4                 truck driver   \n",
       "3     14.85       829.90     C        C5  Information Systems Officer   \n",
       "4     22.45       289.91     F        F1          Contract Specialist   \n",
       "\n",
       "  emp_length home_ownership  annual_inc verification_status   issue_d  \\\n",
       "0  10+ years       MORTGAGE     55000.0        Not Verified  Dec-2015   \n",
       "1  10+ years       MORTGAGE     65000.0        Not Verified  Dec-2015   \n",
       "2  10+ years       MORTGAGE     63000.0        Not Verified  Dec-2015   \n",
       "3  10+ years       MORTGAGE    110000.0     Source Verified  Dec-2015   \n",
       "4    3 years       MORTGAGE    104433.0     Source Verified  Dec-2015   \n",
       "\n",
       "  loan_status pymnt_plan                                                url  \\\n",
       "0  Fully Paid          n  https://lendingclub.com/browse/loanDetail.acti...   \n",
       "1  Fully Paid          n  https://lendingclub.com/browse/loanDetail.acti...   \n",
       "2  Fully Paid          n  https://lendingclub.com/browse/loanDetail.acti...   \n",
       "3     Current          n  https://lendingclub.com/browse/loanDetail.acti...   \n",
       "4  Fully Paid          n  https://lendingclub.com/browse/loanDetail.acti...   \n",
       "\n",
       "   desc             purpose               title zip_code addr_state    dti  \\\n",
       "0  None  debt_consolidation  Debt consolidation    190xx         PA   5.91   \n",
       "1  None      small_business            Business    577xx         SD  16.06   \n",
       "2  None    home_improvement                None    605xx         IL  10.78   \n",
       "3  None  debt_consolidation  Debt consolidation    076xx         NJ  17.06   \n",
       "4  None      major_purchase      Major purchase    174xx         PA  25.37   \n",
       "\n",
       "   delinq_2yrs earliest_cr_line  fico_range_low  fico_range_high  \\\n",
       "0          0.0         Aug-2003           675.0            679.0   \n",
       "1          1.0         Dec-1999           715.0            719.0   \n",
       "2          0.0         Aug-2000           695.0            699.0   \n",
       "3          0.0         Sep-2008           785.0            789.0   \n",
       "4          1.0         Jun-1998           695.0            699.0   \n",
       "\n",
       "   inq_last_6mths  mths_since_last_delinq  mths_since_last_record  open_acc  \\\n",
       "0             1.0                    30.0                     NaN       7.0   \n",
       "1             4.0                     6.0                     NaN      22.0   \n",
       "2             0.0                     NaN                     NaN       6.0   \n",
       "3             0.0                     NaN                     NaN      13.0   \n",
       "4             3.0                    12.0                     NaN      12.0   \n",
       "\n",
       "   pub_rec  revol_bal  revol_util  total_acc initial_list_status  out_prncp  \\\n",
       "0      0.0     2765.0        29.7       13.0                   w       0.00   \n",
       "1      0.0    21470.0        19.2       38.0                   w       0.00   \n",
       "2      0.0     7869.0        56.2       18.0                   w       0.00   \n",
       "3      0.0     7802.0        11.6       17.0                   w   15897.65   \n",
       "4      0.0    21929.0        64.5       35.0                   w       0.00   \n",
       "\n",
       "   out_prncp_inv   total_pymnt  total_pymnt_inv  total_rec_prncp  \\\n",
       "0           0.00   4421.723917          4421.72          3600.00   \n",
       "1           0.00  25679.660000         25679.66         24700.00   \n",
       "2           0.00  22705.924294         22705.92         20000.00   \n",
       "3       15897.65  31464.010000         31464.01         19102.35   \n",
       "4           0.00  11740.500000         11740.50         10400.00   \n",
       "\n",
       "   total_rec_int  total_rec_late_fee  recoveries  collection_recovery_fee  \\\n",
       "0         821.72                 0.0         0.0                      0.0   \n",
       "1         979.66                 0.0         0.0                      0.0   \n",
       "2        2705.92                 0.0         0.0                      0.0   \n",
       "3       12361.66                 0.0         0.0                      0.0   \n",
       "4        1340.50                 0.0         0.0                      0.0   \n",
       "\n",
       "  last_pymnt_d  last_pymnt_amnt next_pymnt_d last_credit_pull_d  \\\n",
       "0     Jan-2019           122.67         None           Mar-2019   \n",
       "1     Jun-2016           926.35         None           Mar-2019   \n",
       "2     Jun-2017         15813.30         None           Mar-2019   \n",
       "3     Feb-2019           829.90     Apr-2019           Mar-2019   \n",
       "4     Jul-2016         10128.96         None           Mar-2018   \n",
       "\n",
       "   last_fico_range_high  last_fico_range_low  collections_12_mths_ex_med  \\\n",
       "0                 564.0                560.0                         0.0   \n",
       "1                 699.0                695.0                         0.0   \n",
       "2                 704.0                700.0                         0.0   \n",
       "3                 679.0                675.0                         0.0   \n",
       "4                 704.0                700.0                         0.0   \n",
       "\n",
       "   mths_since_last_major_derog  policy_code application_type  \\\n",
       "0                         30.0          1.0       Individual   \n",
       "1                          NaN          1.0       Individual   \n",
       "2                          NaN          1.0        Joint App   \n",
       "3                          NaN          1.0       Individual   \n",
       "4                          NaN          1.0       Individual   \n",
       "\n",
       "   annual_inc_joint  dti_joint verification_status_joint  acc_now_delinq  \\\n",
       "0               NaN        NaN                      None             0.0   \n",
       "1               NaN        NaN                      None             0.0   \n",
       "2           71000.0      13.85              Not Verified             0.0   \n",
       "3               NaN        NaN                      None             0.0   \n",
       "4               NaN        NaN                      None             0.0   \n",
       "\n",
       "   tot_coll_amt  tot_cur_bal  open_acc_6m  open_act_il  open_il_12m  \\\n",
       "0         722.0     144904.0          2.0          2.0          0.0   \n",
       "1           0.0     204396.0          1.0          1.0          0.0   \n",
       "2           0.0     189699.0          0.0          1.0          0.0   \n",
       "3           0.0     301500.0          1.0          1.0          0.0   \n",
       "4           0.0     331730.0          1.0          3.0          0.0   \n",
       "\n",
       "   open_il_24m  mths_since_rcnt_il  total_bal_il  il_util  open_rv_12m  \\\n",
       "0          1.0                21.0        4981.0     36.0          3.0   \n",
       "1          1.0                19.0       18005.0     73.0          2.0   \n",
       "2          4.0                19.0       10827.0     73.0          0.0   \n",
       "3          1.0                23.0       12609.0     70.0          1.0   \n",
       "4          3.0                14.0       73839.0     84.0          4.0   \n",
       "\n",
       "   open_rv_24m  max_bal_bc  all_util  total_rev_hi_lim  inq_fi  total_cu_tl  \\\n",
       "0          3.0       722.0      34.0            9300.0     3.0          1.0   \n",
       "1          3.0      6472.0      29.0          111800.0     0.0          0.0   \n",
       "2          2.0      2081.0      65.0           14000.0     2.0          5.0   \n",
       "3          1.0      6987.0      45.0           67300.0     0.0          1.0   \n",
       "4          7.0      9702.0      78.0           34000.0     2.0          1.0   \n",
       "\n",
       "   inq_last_12m  acc_open_past_24mths  avg_cur_bal  bc_open_to_buy  bc_util  \\\n",
       "0           4.0                   4.0      20701.0          1506.0     37.2   \n",
       "1           6.0                   4.0       9733.0         57830.0     27.1   \n",
       "2           1.0                   6.0      31617.0          2737.0     55.9   \n",
       "3           0.0                   2.0      23192.0         54962.0     12.1   \n",
       "4           3.0                  10.0      27644.0          4567.0     77.5   \n",
       "\n",
       "   chargeoff_within_12_mths  delinq_amnt  mo_sin_old_il_acct  \\\n",
       "0                       0.0          0.0               148.0   \n",
       "1                       0.0          0.0               113.0   \n",
       "2                       0.0          0.0               125.0   \n",
       "3                       0.0          0.0                36.0   \n",
       "4                       0.0          0.0               128.0   \n",
       "\n",
       "   mo_sin_old_rev_tl_op  mo_sin_rcnt_rev_tl_op  mo_sin_rcnt_tl  mort_acc  \\\n",
       "0                 128.0                    3.0             3.0       1.0   \n",
       "1                 192.0                    2.0             2.0       4.0   \n",
       "2                 184.0                   14.0            14.0       5.0   \n",
       "3                  87.0                    2.0             2.0       1.0   \n",
       "4                 210.0                    4.0             4.0       6.0   \n",
       "\n",
       "   mths_since_recent_bc  mths_since_recent_bc_dlq  mths_since_recent_inq  \\\n",
       "0                   4.0                      69.0                    4.0   \n",
       "1                   2.0                       NaN                    0.0   \n",
       "2                 101.0                       NaN                   10.0   \n",
       "3                   2.0                       NaN                    NaN   \n",
       "4                   4.0                      12.0                    1.0   \n",
       "\n",
       "   mths_since_recent_revol_delinq  num_accts_ever_120_pd  num_actv_bc_tl  \\\n",
       "0                            69.0                    2.0             2.0   \n",
       "1                             6.0                    0.0             5.0   \n",
       "2                             NaN                    0.0             2.0   \n",
       "3                             NaN                    0.0             4.0   \n",
       "4                            12.0                    0.0             4.0   \n",
       "\n",
       "   num_actv_rev_tl  num_bc_sats  num_bc_tl  num_il_tl  num_op_rev_tl  \\\n",
       "0              4.0          2.0        5.0        3.0            4.0   \n",
       "1              5.0         13.0       17.0        6.0           20.0   \n",
       "2              3.0          2.0        4.0        6.0            4.0   \n",
       "3              5.0          8.0       10.0        2.0           10.0   \n",
       "4              6.0          5.0        9.0       10.0            7.0   \n",
       "\n",
       "   num_rev_accts  num_rev_tl_bal_gt_0  num_sats  num_tl_120dpd_2m  \\\n",
       "0            9.0                  4.0       7.0               0.0   \n",
       "1           27.0                  5.0      22.0               0.0   \n",
       "2            7.0                  3.0       6.0               0.0   \n",
       "3           13.0                  5.0      13.0               0.0   \n",
       "4           19.0                  6.0      12.0               0.0   \n",
       "\n",
       "   num_tl_30dpd  num_tl_90g_dpd_24m  num_tl_op_past_12m  pct_tl_nvr_dlq  \\\n",
       "0           0.0                 0.0                 3.0            76.9   \n",
       "1           0.0                 0.0                 2.0            97.4   \n",
       "2           0.0                 0.0                 0.0           100.0   \n",
       "3           0.0                 0.0                 1.0           100.0   \n",
       "4           0.0                 0.0                 4.0            96.6   \n",
       "\n",
       "   percent_bc_gt_75  pub_rec_bankruptcies  tax_liens  tot_hi_cred_lim  \\\n",
       "0               0.0                   0.0        0.0         178050.0   \n",
       "1               7.7                   0.0        0.0         314017.0   \n",
       "2              50.0                   0.0        0.0         218418.0   \n",
       "3               0.0                   0.0        0.0         381215.0   \n",
       "4              60.0                   0.0        0.0         439570.0   \n",
       "\n",
       "   total_bal_ex_mort  total_bc_limit  total_il_high_credit_limit  \\\n",
       "0             7746.0          2400.0                     13734.0   \n",
       "1            39475.0         79300.0                     24667.0   \n",
       "2            18696.0          6200.0                     14877.0   \n",
       "3            52226.0         62500.0                     18000.0   \n",
       "4            95768.0         20300.0                     88097.0   \n",
       "\n",
       "   revol_bal_joint  sec_app_fico_range_low  sec_app_fico_range_high  \\\n",
       "0              NaN                     NaN                      NaN   \n",
       "1              NaN                     NaN                      NaN   \n",
       "2              NaN                     NaN                      NaN   \n",
       "3              NaN                     NaN                      NaN   \n",
       "4              NaN                     NaN                      NaN   \n",
       "\n",
       "   sec_app_earliest_cr_line  sec_app_inq_last_6mths  sec_app_mort_acc  \\\n",
       "0                       NaN                     NaN               NaN   \n",
       "1                       NaN                     NaN               NaN   \n",
       "2                       NaN                     NaN               NaN   \n",
       "3                       NaN                     NaN               NaN   \n",
       "4                       NaN                     NaN               NaN   \n",
       "\n",
       "   sec_app_open_acc  sec_app_revol_util  sec_app_open_act_il  \\\n",
       "0               NaN                 NaN                  NaN   \n",
       "1               NaN                 NaN                  NaN   \n",
       "2               NaN                 NaN                  NaN   \n",
       "3               NaN                 NaN                  NaN   \n",
       "4               NaN                 NaN                  NaN   \n",
       "\n",
       "   sec_app_num_rev_accts  sec_app_chargeoff_within_12_mths  \\\n",
       "0                    NaN                               NaN   \n",
       "1                    NaN                               NaN   \n",
       "2                    NaN                               NaN   \n",
       "3                    NaN                               NaN   \n",
       "4                    NaN                               NaN   \n",
       "\n",
       "   sec_app_collections_12_mths_ex_med  sec_app_mths_since_last_major_derog  \\\n",
       "0                                 NaN                                  NaN   \n",
       "1                                 NaN                                  NaN   \n",
       "2                                 NaN                                  NaN   \n",
       "3                                 NaN                                  NaN   \n",
       "4                                 NaN                                  NaN   \n",
       "\n",
       "  hardship_flag hardship_type hardship_reason hardship_status  deferral_term  \\\n",
       "0             N          None            None            None            NaN   \n",
       "1             N          None            None            None            NaN   \n",
       "2             N          None            None            None            NaN   \n",
       "3             N          None            None            None            NaN   \n",
       "4             N          None            None            None            NaN   \n",
       "\n",
       "   hardship_amount hardship_start_date hardship_end_date  \\\n",
       "0              NaN                None              None   \n",
       "1              NaN                None              None   \n",
       "2              NaN                None              None   \n",
       "3              NaN                None              None   \n",
       "4              NaN                None              None   \n",
       "\n",
       "  payment_plan_start_date  hardship_length  hardship_dpd hardship_loan_status  \\\n",
       "0                    None              NaN           NaN                 None   \n",
       "1                    None              NaN           NaN                 None   \n",
       "2                    None              NaN           NaN                 None   \n",
       "3                    None              NaN           NaN                 None   \n",
       "4                    None              NaN           NaN                 None   \n",
       "\n",
       "   orig_projected_additional_accrued_interest  hardship_payoff_balance_amount  \\\n",
       "0                                         NaN                             NaN   \n",
       "1                                         NaN                             NaN   \n",
       "2                                         NaN                             NaN   \n",
       "3                                         NaN                             NaN   \n",
       "4                                         NaN                             NaN   \n",
       "\n",
       "   hardship_last_payment_amount disbursement_method debt_settlement_flag  \\\n",
       "0                           NaN                Cash                    N   \n",
       "1                           NaN                Cash                    N   \n",
       "2                           NaN                Cash                    N   \n",
       "3                           NaN                Cash                    N   \n",
       "4                           NaN                Cash                    N   \n",
       "\n",
       "  debt_settlement_flag_date settlement_status settlement_date  \\\n",
       "0                      None              None            None   \n",
       "1                      None              None            None   \n",
       "2                      None              None            None   \n",
       "3                      None              None            None   \n",
       "4                      None              None            None   \n",
       "\n",
       "   settlement_amount  settlement_percentage  settlement_term  \n",
       "0                NaN                    NaN              NaN  \n",
       "1                NaN                    NaN              NaN  \n",
       "2                NaN                    NaN              NaN  \n",
       "3                NaN                    NaN              NaN  \n",
       "4                NaN                    NaN              NaN  "
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Print the first 5 rows of the dataset\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 22511 entries, 0 to 22510\n",
      "Columns: 151 entries, id to settlement_term\n",
      "dtypes: float64(114), int64(1), object(36)\n",
      "memory usage: 25.9+ MB\n"
     ]
    }
   ],
   "source": [
    "# Display the info of the dataset\n",
    "df.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "id                                              int64\n",
       "member_id                                     float64\n",
       "loan_amnt                                     float64\n",
       "funded_amnt                                   float64\n",
       "funded_amnt_inv                               float64\n",
       "term                                           object\n",
       "int_rate                                      float64\n",
       "installment                                   float64\n",
       "grade                                          object\n",
       "sub_grade                                      object\n",
       "emp_title                                      object\n",
       "emp_length                                     object\n",
       "home_ownership                                 object\n",
       "annual_inc                                    float64\n",
       "verification_status                            object\n",
       "issue_d                                        object\n",
       "loan_status                                    object\n",
       "pymnt_plan                                     object\n",
       "url                                            object\n",
       "desc                                           object\n",
       "purpose                                        object\n",
       "title                                          object\n",
       "zip_code                                       object\n",
       "addr_state                                     object\n",
       "dti                                           float64\n",
       "delinq_2yrs                                   float64\n",
       "earliest_cr_line                               object\n",
       "fico_range_low                                float64\n",
       "fico_range_high                               float64\n",
       "inq_last_6mths                                float64\n",
       "mths_since_last_delinq                        float64\n",
       "mths_since_last_record                        float64\n",
       "open_acc                                      float64\n",
       "pub_rec                                       float64\n",
       "revol_bal                                     float64\n",
       "revol_util                                    float64\n",
       "total_acc                                     float64\n",
       "initial_list_status                            object\n",
       "out_prncp                                     float64\n",
       "out_prncp_inv                                 float64\n",
       "total_pymnt                                   float64\n",
       "total_pymnt_inv                               float64\n",
       "total_rec_prncp                               float64\n",
       "total_rec_int                                 float64\n",
       "total_rec_late_fee                            float64\n",
       "recoveries                                    float64\n",
       "collection_recovery_fee                       float64\n",
       "last_pymnt_d                                   object\n",
       "last_pymnt_amnt                               float64\n",
       "next_pymnt_d                                   object\n",
       "last_credit_pull_d                             object\n",
       "last_fico_range_high                          float64\n",
       "last_fico_range_low                           float64\n",
       "collections_12_mths_ex_med                    float64\n",
       "mths_since_last_major_derog                   float64\n",
       "policy_code                                   float64\n",
       "application_type                               object\n",
       "annual_inc_joint                              float64\n",
       "dti_joint                                     float64\n",
       "verification_status_joint                      object\n",
       "acc_now_delinq                                float64\n",
       "tot_coll_amt                                  float64\n",
       "tot_cur_bal                                   float64\n",
       "open_acc_6m                                   float64\n",
       "open_act_il                                   float64\n",
       "open_il_12m                                   float64\n",
       "open_il_24m                                   float64\n",
       "mths_since_rcnt_il                            float64\n",
       "total_bal_il                                  float64\n",
       "il_util                                       float64\n",
       "open_rv_12m                                   float64\n",
       "open_rv_24m                                   float64\n",
       "max_bal_bc                                    float64\n",
       "all_util                                      float64\n",
       "total_rev_hi_lim                              float64\n",
       "inq_fi                                        float64\n",
       "total_cu_tl                                   float64\n",
       "inq_last_12m                                  float64\n",
       "acc_open_past_24mths                          float64\n",
       "avg_cur_bal                                   float64\n",
       "bc_open_to_buy                                float64\n",
       "bc_util                                       float64\n",
       "chargeoff_within_12_mths                      float64\n",
       "delinq_amnt                                   float64\n",
       "mo_sin_old_il_acct                            float64\n",
       "mo_sin_old_rev_tl_op                          float64\n",
       "mo_sin_rcnt_rev_tl_op                         float64\n",
       "mo_sin_rcnt_tl                                float64\n",
       "mort_acc                                      float64\n",
       "mths_since_recent_bc                          float64\n",
       "mths_since_recent_bc_dlq                      float64\n",
       "mths_since_recent_inq                         float64\n",
       "mths_since_recent_revol_delinq                float64\n",
       "num_accts_ever_120_pd                         float64\n",
       "num_actv_bc_tl                                float64\n",
       "num_actv_rev_tl                               float64\n",
       "num_bc_sats                                   float64\n",
       "num_bc_tl                                     float64\n",
       "num_il_tl                                     float64\n",
       "num_op_rev_tl                                 float64\n",
       "num_rev_accts                                 float64\n",
       "num_rev_tl_bal_gt_0                           float64\n",
       "num_sats                                      float64\n",
       "num_tl_120dpd_2m                              float64\n",
       "num_tl_30dpd                                  float64\n",
       "num_tl_90g_dpd_24m                            float64\n",
       "num_tl_op_past_12m                            float64\n",
       "pct_tl_nvr_dlq                                float64\n",
       "percent_bc_gt_75                              float64\n",
       "pub_rec_bankruptcies                          float64\n",
       "tax_liens                                     float64\n",
       "tot_hi_cred_lim                               float64\n",
       "total_bal_ex_mort                             float64\n",
       "total_bc_limit                                float64\n",
       "total_il_high_credit_limit                    float64\n",
       "revol_bal_joint                               float64\n",
       "sec_app_fico_range_low                        float64\n",
       "sec_app_fico_range_high                       float64\n",
       "sec_app_earliest_cr_line                      float64\n",
       "sec_app_inq_last_6mths                        float64\n",
       "sec_app_mort_acc                              float64\n",
       "sec_app_open_acc                              float64\n",
       "sec_app_revol_util                            float64\n",
       "sec_app_open_act_il                           float64\n",
       "sec_app_num_rev_accts                         float64\n",
       "sec_app_chargeoff_within_12_mths              float64\n",
       "sec_app_collections_12_mths_ex_med            float64\n",
       "sec_app_mths_since_last_major_derog           float64\n",
       "hardship_flag                                  object\n",
       "hardship_type                                  object\n",
       "hardship_reason                                object\n",
       "hardship_status                                object\n",
       "deferral_term                                 float64\n",
       "hardship_amount                               float64\n",
       "hardship_start_date                            object\n",
       "hardship_end_date                              object\n",
       "payment_plan_start_date                        object\n",
       "hardship_length                               float64\n",
       "hardship_dpd                                  float64\n",
       "hardship_loan_status                           object\n",
       "orig_projected_additional_accrued_interest    float64\n",
       "hardship_payoff_balance_amount                float64\n",
       "hardship_last_payment_amount                  float64\n",
       "disbursement_method                            object\n",
       "debt_settlement_flag                           object\n",
       "debt_settlement_flag_date                      object\n",
       "settlement_status                              object\n",
       "settlement_date                                object\n",
       "settlement_amount                             float64\n",
       "settlement_percentage                         float64\n",
       "settlement_term                               float64\n",
       "dtype: object"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Display the data types of the dataset\n",
    "df.dtypes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "id                                            22511\n",
       "member_id                                         0\n",
       "loan_amnt                                      1140\n",
       "funded_amnt                                    1140\n",
       "funded_amnt_inv                                1190\n",
       "term                                              2\n",
       "int_rate                                         64\n",
       "installment                                    6915\n",
       "grade                                             7\n",
       "sub_grade                                        35\n",
       "emp_title                                     10992\n",
       "emp_length                                       11\n",
       "home_ownership                                    3\n",
       "annual_inc                                     2442\n",
       "verification_status                               3\n",
       "issue_d                                           1\n",
       "loan_status                                       6\n",
       "pymnt_plan                                        2\n",
       "url                                           22511\n",
       "desc                                              1\n",
       "purpose                                          13\n",
       "title                                            12\n",
       "zip_code                                        837\n",
       "addr_state                                       49\n",
       "dti                                            3795\n",
       "delinq_2yrs                                      15\n",
       "earliest_cr_line                                554\n",
       "fico_range_low                                   38\n",
       "fico_range_high                                  38\n",
       "inq_last_6mths                                    6\n",
       "mths_since_last_delinq                          102\n",
       "mths_since_last_record                          121\n",
       "open_acc                                         52\n",
       "pub_rec                                          14\n",
       "revol_bal                                     16490\n",
       "revol_util                                     1044\n",
       "total_acc                                        90\n",
       "initial_list_status                               2\n",
       "out_prncp                                      2030\n",
       "out_prncp_inv                                  2083\n",
       "total_pymnt                                   21566\n",
       "total_pymnt_inv                               21213\n",
       "total_rec_prncp                                6942\n",
       "total_rec_int                                 20848\n",
       "total_rec_late_fee                              704\n",
       "recoveries                                     2665\n",
       "collection_recovery_fee                        2701\n",
       "last_pymnt_d                                     40\n",
       "last_pymnt_amnt                               17314\n",
       "next_pymnt_d                                      2\n",
       "last_credit_pull_d                               40\n",
       "last_fico_range_high                             72\n",
       "last_fico_range_low                              71\n",
       "collections_12_mths_ex_med                        5\n",
       "mths_since_last_major_derog                     113\n",
       "policy_code                                       1\n",
       "application_type                                  2\n",
       "annual_inc_joint                                129\n",
       "dti_joint                                       164\n",
       "verification_status_joint                         1\n",
       "acc_now_delinq                                    3\n",
       "tot_coll_amt                                   1707\n",
       "tot_cur_bal                                   21489\n",
       "open_acc_6m                                      13\n",
       "open_act_il                                      35\n",
       "open_il_12m                                      12\n",
       "open_il_24m                                      17\n",
       "mths_since_rcnt_il                              201\n",
       "total_bal_il                                  17011\n",
       "il_util                                         160\n",
       "open_rv_12m                                      18\n",
       "open_rv_24m                                      28\n",
       "max_bal_bc                                    10701\n",
       "all_util                                        139\n",
       "total_rev_hi_lim                               2103\n",
       "inq_fi                                           18\n",
       "total_cu_tl                                      33\n",
       "inq_last_12m                                     28\n",
       "acc_open_past_24mths                             32\n",
       "avg_cur_bal                                   15009\n",
       "bc_open_to_buy                                13131\n",
       "bc_util                                        1081\n",
       "chargeoff_within_12_mths                          4\n",
       "delinq_amnt                                      82\n",
       "mo_sin_old_il_acct                              363\n",
       "mo_sin_old_rev_tl_op                            579\n",
       "mo_sin_rcnt_rev_tl_op                           156\n",
       "mo_sin_rcnt_tl                                  109\n",
       "mort_acc                                         21\n",
       "mths_since_recent_bc                            249\n",
       "mths_since_recent_bc_dlq                        107\n",
       "mths_since_recent_inq                            25\n",
       "mths_since_recent_revol_delinq                  112\n",
       "num_accts_ever_120_pd                            22\n",
       "num_actv_bc_tl                                   25\n",
       "num_actv_rev_tl                                  34\n",
       "num_bc_sats                                      32\n",
       "num_bc_tl                                        44\n",
       "num_il_tl                                        67\n",
       "num_op_rev_tl                                    49\n",
       "num_rev_accts                                    70\n",
       "num_rev_tl_bal_gt_0                              35\n",
       "num_sats                                         52\n",
       "num_tl_120dpd_2m                                  3\n",
       "num_tl_30dpd                                      3\n",
       "num_tl_90g_dpd_24m                               14\n",
       "num_tl_op_past_12m                               21\n",
       "pct_tl_nvr_dlq                                  348\n",
       "percent_bc_gt_75                                103\n",
       "pub_rec_bankruptcies                              8\n",
       "tax_liens                                        13\n",
       "tot_hi_cred_lim                               20377\n",
       "total_bal_ex_mort                             20230\n",
       "total_bc_limit                                 1457\n",
       "total_il_high_credit_limit                    16959\n",
       "revol_bal_joint                                   0\n",
       "sec_app_fico_range_low                            0\n",
       "sec_app_fico_range_high                           0\n",
       "sec_app_earliest_cr_line                          0\n",
       "sec_app_inq_last_6mths                            0\n",
       "sec_app_mort_acc                                  0\n",
       "sec_app_open_acc                                  0\n",
       "sec_app_revol_util                                0\n",
       "sec_app_open_act_il                               0\n",
       "sec_app_num_rev_accts                             0\n",
       "sec_app_chargeoff_within_12_mths                  0\n",
       "sec_app_collections_12_mths_ex_med                0\n",
       "sec_app_mths_since_last_major_derog               0\n",
       "hardship_flag                                     2\n",
       "hardship_type                                     1\n",
       "hardship_reason                                   9\n",
       "hardship_status                                   3\n",
       "deferral_term                                     1\n",
       "hardship_amount                                 180\n",
       "hardship_start_date                              25\n",
       "hardship_end_date                                25\n",
       "payment_plan_start_date                          24\n",
       "hardship_length                                   1\n",
       "hardship_dpd                                     29\n",
       "hardship_loan_status                              4\n",
       "orig_projected_additional_accrued_interest      144\n",
       "hardship_payoff_balance_amount                  180\n",
       "hardship_last_payment_amount                    177\n",
       "disbursement_method                               1\n",
       "debt_settlement_flag                              2\n",
       "debt_settlement_flag_date                        30\n",
       "settlement_status                                 3\n",
       "settlement_date                                  34\n",
       "settlement_amount                               652\n",
       "settlement_percentage                            86\n",
       "settlement_term                                  24\n",
       "dtype: int64"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Check the number of unique values in each column\n",
    "df.nunique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>member_id</th>\n",
       "      <th>loan_amnt</th>\n",
       "      <th>funded_amnt</th>\n",
       "      <th>funded_amnt_inv</th>\n",
       "      <th>int_rate</th>\n",
       "      <th>installment</th>\n",
       "      <th>annual_inc</th>\n",
       "      <th>dti</th>\n",
       "      <th>delinq_2yrs</th>\n",
       "      <th>fico_range_low</th>\n",
       "      <th>fico_range_high</th>\n",
       "      <th>inq_last_6mths</th>\n",
       "      <th>mths_since_last_delinq</th>\n",
       "      <th>mths_since_last_record</th>\n",
       "      <th>open_acc</th>\n",
       "      <th>pub_rec</th>\n",
       "      <th>revol_bal</th>\n",
       "      <th>revol_util</th>\n",
       "      <th>total_acc</th>\n",
       "      <th>out_prncp</th>\n",
       "      <th>out_prncp_inv</th>\n",
       "      <th>total_pymnt</th>\n",
       "      <th>total_pymnt_inv</th>\n",
       "      <th>total_rec_prncp</th>\n",
       "      <th>total_rec_int</th>\n",
       "      <th>total_rec_late_fee</th>\n",
       "      <th>recoveries</th>\n",
       "      <th>collection_recovery_fee</th>\n",
       "      <th>last_pymnt_amnt</th>\n",
       "      <th>last_fico_range_high</th>\n",
       "      <th>last_fico_range_low</th>\n",
       "      <th>collections_12_mths_ex_med</th>\n",
       "      <th>mths_since_last_major_derog</th>\n",
       "      <th>policy_code</th>\n",
       "      <th>annual_inc_joint</th>\n",
       "      <th>dti_joint</th>\n",
       "      <th>acc_now_delinq</th>\n",
       "      <th>tot_coll_amt</th>\n",
       "      <th>tot_cur_bal</th>\n",
       "      <th>open_acc_6m</th>\n",
       "      <th>open_act_il</th>\n",
       "      <th>open_il_12m</th>\n",
       "      <th>open_il_24m</th>\n",
       "      <th>mths_since_rcnt_il</th>\n",
       "      <th>total_bal_il</th>\n",
       "      <th>il_util</th>\n",
       "      <th>open_rv_12m</th>\n",
       "      <th>open_rv_24m</th>\n",
       "      <th>max_bal_bc</th>\n",
       "      <th>all_util</th>\n",
       "      <th>total_rev_hi_lim</th>\n",
       "      <th>inq_fi</th>\n",
       "      <th>total_cu_tl</th>\n",
       "      <th>inq_last_12m</th>\n",
       "      <th>acc_open_past_24mths</th>\n",
       "      <th>avg_cur_bal</th>\n",
       "      <th>bc_open_to_buy</th>\n",
       "      <th>bc_util</th>\n",
       "      <th>chargeoff_within_12_mths</th>\n",
       "      <th>delinq_amnt</th>\n",
       "      <th>mo_sin_old_il_acct</th>\n",
       "      <th>mo_sin_old_rev_tl_op</th>\n",
       "      <th>mo_sin_rcnt_rev_tl_op</th>\n",
       "      <th>mo_sin_rcnt_tl</th>\n",
       "      <th>mort_acc</th>\n",
       "      <th>mths_since_recent_bc</th>\n",
       "      <th>mths_since_recent_bc_dlq</th>\n",
       "      <th>mths_since_recent_inq</th>\n",
       "      <th>mths_since_recent_revol_delinq</th>\n",
       "      <th>num_accts_ever_120_pd</th>\n",
       "      <th>num_actv_bc_tl</th>\n",
       "      <th>num_actv_rev_tl</th>\n",
       "      <th>num_bc_sats</th>\n",
       "      <th>num_bc_tl</th>\n",
       "      <th>num_il_tl</th>\n",
       "      <th>num_op_rev_tl</th>\n",
       "      <th>num_rev_accts</th>\n",
       "      <th>num_rev_tl_bal_gt_0</th>\n",
       "      <th>num_sats</th>\n",
       "      <th>num_tl_120dpd_2m</th>\n",
       "      <th>num_tl_30dpd</th>\n",
       "      <th>num_tl_90g_dpd_24m</th>\n",
       "      <th>num_tl_op_past_12m</th>\n",
       "      <th>pct_tl_nvr_dlq</th>\n",
       "      <th>percent_bc_gt_75</th>\n",
       "      <th>pub_rec_bankruptcies</th>\n",
       "      <th>tax_liens</th>\n",
       "      <th>tot_hi_cred_lim</th>\n",
       "      <th>total_bal_ex_mort</th>\n",
       "      <th>total_bc_limit</th>\n",
       "      <th>total_il_high_credit_limit</th>\n",
       "      <th>revol_bal_joint</th>\n",
       "      <th>sec_app_fico_range_low</th>\n",
       "      <th>sec_app_fico_range_high</th>\n",
       "      <th>sec_app_earliest_cr_line</th>\n",
       "      <th>sec_app_inq_last_6mths</th>\n",
       "      <th>sec_app_mort_acc</th>\n",
       "      <th>sec_app_open_acc</th>\n",
       "      <th>sec_app_revol_util</th>\n",
       "      <th>sec_app_open_act_il</th>\n",
       "      <th>sec_app_num_rev_accts</th>\n",
       "      <th>sec_app_chargeoff_within_12_mths</th>\n",
       "      <th>sec_app_collections_12_mths_ex_med</th>\n",
       "      <th>sec_app_mths_since_last_major_derog</th>\n",
       "      <th>deferral_term</th>\n",
       "      <th>hardship_amount</th>\n",
       "      <th>hardship_length</th>\n",
       "      <th>hardship_dpd</th>\n",
       "      <th>orig_projected_additional_accrued_interest</th>\n",
       "      <th>hardship_payoff_balance_amount</th>\n",
       "      <th>hardship_last_payment_amount</th>\n",
       "      <th>settlement_amount</th>\n",
       "      <th>settlement_percentage</th>\n",
       "      <th>settlement_term</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>2.251100e+04</td>\n",
       "      <td>0.0</td>\n",
       "      <td>22511.000000</td>\n",
       "      <td>22511.000000</td>\n",
       "      <td>22511.000000</td>\n",
       "      <td>22511.000000</td>\n",
       "      <td>22511.000000</td>\n",
       "      <td>2.251100e+04</td>\n",
       "      <td>22509.000000</td>\n",
       "      <td>22510.000000</td>\n",
       "      <td>22510.000000</td>\n",
       "      <td>22510.000000</td>\n",
       "      <td>22510.000000</td>\n",
       "      <td>11637.000000</td>\n",
       "      <td>4038.00000</td>\n",
       "      <td>22510.000000</td>\n",
       "      <td>22510.000000</td>\n",
       "      <td>22510.000000</td>\n",
       "      <td>22499.00000</td>\n",
       "      <td>22510.000000</td>\n",
       "      <td>22510.000000</td>\n",
       "      <td>22510.000000</td>\n",
       "      <td>22510.000000</td>\n",
       "      <td>22510.000000</td>\n",
       "      <td>22510.000000</td>\n",
       "      <td>22510.000000</td>\n",
       "      <td>22510.000000</td>\n",
       "      <td>22510.000000</td>\n",
       "      <td>22510.000000</td>\n",
       "      <td>22510.000000</td>\n",
       "      <td>22510.000000</td>\n",
       "      <td>22510.000000</td>\n",
       "      <td>22510.000000</td>\n",
       "      <td>6466.000000</td>\n",
       "      <td>22510.0</td>\n",
       "      <td>169.000000</td>\n",
       "      <td>169.000000</td>\n",
       "      <td>22510.000000</td>\n",
       "      <td>22510.000000</td>\n",
       "      <td>2.251000e+04</td>\n",
       "      <td>21344.000000</td>\n",
       "      <td>21344.000000</td>\n",
       "      <td>21344.000000</td>\n",
       "      <td>21344.000000</td>\n",
       "      <td>20785.000000</td>\n",
       "      <td>21344.000000</td>\n",
       "      <td>18595.000000</td>\n",
       "      <td>21344.000000</td>\n",
       "      <td>21344.000000</td>\n",
       "      <td>21344.000000</td>\n",
       "      <td>21344.000000</td>\n",
       "      <td>22510.000000</td>\n",
       "      <td>21344.000000</td>\n",
       "      <td>21344.000000</td>\n",
       "      <td>21344.000000</td>\n",
       "      <td>22510.000000</td>\n",
       "      <td>22510.000000</td>\n",
       "      <td>22296.000000</td>\n",
       "      <td>22282.000000</td>\n",
       "      <td>22510.000000</td>\n",
       "      <td>22510.000000</td>\n",
       "      <td>21919.000000</td>\n",
       "      <td>22510.000000</td>\n",
       "      <td>22510.000000</td>\n",
       "      <td>22510.000000</td>\n",
       "      <td>22510.000000</td>\n",
       "      <td>22303.000000</td>\n",
       "      <td>5715.000000</td>\n",
       "      <td>20159.000000</td>\n",
       "      <td>8085.000000</td>\n",
       "      <td>22510.000000</td>\n",
       "      <td>22510.000000</td>\n",
       "      <td>22510.000000</td>\n",
       "      <td>22510.000000</td>\n",
       "      <td>22510.000000</td>\n",
       "      <td>22510.000000</td>\n",
       "      <td>22510.000000</td>\n",
       "      <td>22510.000000</td>\n",
       "      <td>22510.000000</td>\n",
       "      <td>22510.000000</td>\n",
       "      <td>21560.000000</td>\n",
       "      <td>22510.000000</td>\n",
       "      <td>22510.000000</td>\n",
       "      <td>22510.000000</td>\n",
       "      <td>22510.000000</td>\n",
       "      <td>22282.000000</td>\n",
       "      <td>22510.000000</td>\n",
       "      <td>22510.000000</td>\n",
       "      <td>2.251000e+04</td>\n",
       "      <td>22510.000000</td>\n",
       "      <td>22510.000000</td>\n",
       "      <td>22510.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>180.0</td>\n",
       "      <td>180.000000</td>\n",
       "      <td>180.0</td>\n",
       "      <td>180.000000</td>\n",
       "      <td>144.000000</td>\n",
       "      <td>180.000000</td>\n",
       "      <td>180.000000</td>\n",
       "      <td>662.000000</td>\n",
       "      <td>662.000000</td>\n",
       "      <td>662.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>6.756447e+07</td>\n",
       "      <td>NaN</td>\n",
       "      <td>15031.787793</td>\n",
       "      <td>15031.787793</td>\n",
       "      <td>15025.527520</td>\n",
       "      <td>12.255020</td>\n",
       "      <td>436.321286</td>\n",
       "      <td>7.820695e+04</td>\n",
       "      <td>19.423974</td>\n",
       "      <td>0.335584</td>\n",
       "      <td>695.075522</td>\n",
       "      <td>699.075611</td>\n",
       "      <td>0.608796</td>\n",
       "      <td>34.162929</td>\n",
       "      <td>65.69787</td>\n",
       "      <td>12.140116</td>\n",
       "      <td>0.238250</td>\n",
       "      <td>17388.656908</td>\n",
       "      <td>51.57675</td>\n",
       "      <td>25.538783</td>\n",
       "      <td>994.961977</td>\n",
       "      <td>994.471550</td>\n",
       "      <td>15043.269607</td>\n",
       "      <td>15037.171987</td>\n",
       "      <td>12062.974932</td>\n",
       "      <td>2748.012439</td>\n",
       "      <td>2.121175</td>\n",
       "      <td>230.161043</td>\n",
       "      <td>41.006250</td>\n",
       "      <td>4262.523270</td>\n",
       "      <td>677.663172</td>\n",
       "      <td>661.832519</td>\n",
       "      <td>0.023678</td>\n",
       "      <td>44.457934</td>\n",
       "      <td>1.0</td>\n",
       "      <td>114705.080296</td>\n",
       "      <td>18.990828</td>\n",
       "      <td>0.004798</td>\n",
       "      <td>279.812306</td>\n",
       "      <td>1.435598e+05</td>\n",
       "      <td>1.108602</td>\n",
       "      <td>2.928879</td>\n",
       "      <td>0.761525</td>\n",
       "      <td>1.674100</td>\n",
       "      <td>20.917873</td>\n",
       "      <td>36538.676068</td>\n",
       "      <td>71.576499</td>\n",
       "      <td>1.388587</td>\n",
       "      <td>2.974747</td>\n",
       "      <td>5887.961207</td>\n",
       "      <td>60.874859</td>\n",
       "      <td>34865.387161</td>\n",
       "      <td>0.943919</td>\n",
       "      <td>1.537341</td>\n",
       "      <td>2.233602</td>\n",
       "      <td>4.876499</td>\n",
       "      <td>13311.317859</td>\n",
       "      <td>10591.638366</td>\n",
       "      <td>59.810511</td>\n",
       "      <td>0.008574</td>\n",
       "      <td>14.074500</td>\n",
       "      <td>127.087823</td>\n",
       "      <td>185.524878</td>\n",
       "      <td>13.055664</td>\n",
       "      <td>7.452466</td>\n",
       "      <td>1.653354</td>\n",
       "      <td>23.932700</td>\n",
       "      <td>39.150306</td>\n",
       "      <td>6.458108</td>\n",
       "      <td>35.480891</td>\n",
       "      <td>0.529542</td>\n",
       "      <td>3.725011</td>\n",
       "      <td>5.692625</td>\n",
       "      <td>4.839627</td>\n",
       "      <td>8.000666</td>\n",
       "      <td>8.873256</td>\n",
       "      <td>8.494536</td>\n",
       "      <td>14.734651</td>\n",
       "      <td>5.740338</td>\n",
       "      <td>12.091870</td>\n",
       "      <td>0.000510</td>\n",
       "      <td>0.003199</td>\n",
       "      <td>0.088139</td>\n",
       "      <td>2.270324</td>\n",
       "      <td>93.986219</td>\n",
       "      <td>44.643582</td>\n",
       "      <td>0.132652</td>\n",
       "      <td>0.070013</td>\n",
       "      <td>1.788133e+05</td>\n",
       "      <td>53921.267614</td>\n",
       "      <td>22772.761972</td>\n",
       "      <td>45965.028698</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3.0</td>\n",
       "      <td>130.504278</td>\n",
       "      <td>3.0</td>\n",
       "      <td>14.272222</td>\n",
       "      <td>390.468542</td>\n",
       "      <td>10659.213444</td>\n",
       "      <td>183.667111</td>\n",
       "      <td>4980.598958</td>\n",
       "      <td>47.125378</td>\n",
       "      <td>13.823263</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>1.843754e+06</td>\n",
       "      <td>NaN</td>\n",
       "      <td>8734.651784</td>\n",
       "      <td>8734.651784</td>\n",
       "      <td>8729.680902</td>\n",
       "      <td>4.213027</td>\n",
       "      <td>249.411778</td>\n",
       "      <td>6.125830e+04</td>\n",
       "      <td>11.006613</td>\n",
       "      <td>0.887691</td>\n",
       "      <td>31.228687</td>\n",
       "      <td>31.229115</td>\n",
       "      <td>0.887993</td>\n",
       "      <td>21.863500</td>\n",
       "      <td>23.81295</td>\n",
       "      <td>5.776034</td>\n",
       "      <td>0.646684</td>\n",
       "      <td>22622.637023</td>\n",
       "      <td>24.22553</td>\n",
       "      <td>12.195480</td>\n",
       "      <td>3018.661632</td>\n",
       "      <td>3017.074677</td>\n",
       "      <td>9788.105024</td>\n",
       "      <td>9782.869853</td>\n",
       "      <td>8384.981147</td>\n",
       "      <td>2769.683951</td>\n",
       "      <td>14.109329</td>\n",
       "      <td>967.317167</td>\n",
       "      <td>173.583214</td>\n",
       "      <td>6493.202576</td>\n",
       "      <td>79.365744</td>\n",
       "      <td>127.447170</td>\n",
       "      <td>0.166284</td>\n",
       "      <td>21.498643</td>\n",
       "      <td>0.0</td>\n",
       "      <td>59397.290685</td>\n",
       "      <td>7.427075</td>\n",
       "      <td>0.071004</td>\n",
       "      <td>2287.920339</td>\n",
       "      <td>1.553523e+05</td>\n",
       "      <td>1.242477</td>\n",
       "      <td>3.089567</td>\n",
       "      <td>0.995896</td>\n",
       "      <td>1.687826</td>\n",
       "      <td>27.213235</td>\n",
       "      <td>43079.900165</td>\n",
       "      <td>23.018098</td>\n",
       "      <td>1.519849</td>\n",
       "      <td>2.630944</td>\n",
       "      <td>5284.130759</td>\n",
       "      <td>20.016522</td>\n",
       "      <td>34215.881650</td>\n",
       "      <td>1.446445</td>\n",
       "      <td>2.721517</td>\n",
       "      <td>2.479804</td>\n",
       "      <td>3.304034</td>\n",
       "      <td>16003.715498</td>\n",
       "      <td>15741.883996</td>\n",
       "      <td>28.130991</td>\n",
       "      <td>0.099612</td>\n",
       "      <td>815.215656</td>\n",
       "      <td>52.306451</td>\n",
       "      <td>95.945983</td>\n",
       "      <td>17.018234</td>\n",
       "      <td>8.978085</td>\n",
       "      <td>1.935176</td>\n",
       "      <td>31.672653</td>\n",
       "      <td>23.252152</td>\n",
       "      <td>5.826340</td>\n",
       "      <td>22.672464</td>\n",
       "      <td>1.371891</td>\n",
       "      <td>2.333436</td>\n",
       "      <td>3.366398</td>\n",
       "      <td>3.085935</td>\n",
       "      <td>4.805447</td>\n",
       "      <td>7.593589</td>\n",
       "      <td>4.798359</td>\n",
       "      <td>8.326798</td>\n",
       "      <td>3.342459</td>\n",
       "      <td>5.758941</td>\n",
       "      <td>0.024551</td>\n",
       "      <td>0.057248</td>\n",
       "      <td>0.493106</td>\n",
       "      <td>1.912824</td>\n",
       "      <td>8.831363</td>\n",
       "      <td>36.248053</td>\n",
       "      <td>0.381843</td>\n",
       "      <td>0.446809</td>\n",
       "      <td>1.730730e+05</td>\n",
       "      <td>49517.560009</td>\n",
       "      <td>22403.564952</td>\n",
       "      <td>45041.561576</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.0</td>\n",
       "      <td>108.186827</td>\n",
       "      <td>0.0</td>\n",
       "      <td>10.296081</td>\n",
       "      <td>320.493440</td>\n",
       "      <td>6503.722235</td>\n",
       "      <td>178.053547</td>\n",
       "      <td>3596.196186</td>\n",
       "      <td>5.255945</td>\n",
       "      <td>7.501520</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>3.617740e+05</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1000.000000</td>\n",
       "      <td>1000.000000</td>\n",
       "      <td>1000.000000</td>\n",
       "      <td>5.320000</td>\n",
       "      <td>30.540000</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>660.000000</td>\n",
       "      <td>664.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>4.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.0</td>\n",
       "      <td>28000.000000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>6.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>12.500000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>2.500000e+03</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3.0</td>\n",
       "      <td>5.280000</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>33.840000</td>\n",
       "      <td>594.070000</td>\n",
       "      <td>0.060000</td>\n",
       "      <td>250.000000</td>\n",
       "      <td>30.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>6.729818e+07</td>\n",
       "      <td>NaN</td>\n",
       "      <td>8000.000000</td>\n",
       "      <td>8000.000000</td>\n",
       "      <td>8000.000000</td>\n",
       "      <td>9.170000</td>\n",
       "      <td>255.040000</td>\n",
       "      <td>4.800000e+04</td>\n",
       "      <td>12.700000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>670.000000</td>\n",
       "      <td>674.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>15.000000</td>\n",
       "      <td>51.00000</td>\n",
       "      <td>8.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>6256.000000</td>\n",
       "      <td>33.40000</td>\n",
       "      <td>17.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>7359.465000</td>\n",
       "      <td>7357.160000</td>\n",
       "      <td>5670.825000</td>\n",
       "      <td>895.707500</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>332.152500</td>\n",
       "      <td>629.000000</td>\n",
       "      <td>625.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>27.000000</td>\n",
       "      <td>1.0</td>\n",
       "      <td>78200.000000</td>\n",
       "      <td>13.830000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>3.219325e+04</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>6.000000</td>\n",
       "      <td>10249.250000</td>\n",
       "      <td>59.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>2411.750000</td>\n",
       "      <td>48.000000</td>\n",
       "      <td>14800.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>3275.250000</td>\n",
       "      <td>1523.750000</td>\n",
       "      <td>38.100000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>100.000000</td>\n",
       "      <td>119.000000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>6.000000</td>\n",
       "      <td>20.000000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>16.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>5.000000</td>\n",
       "      <td>4.000000</td>\n",
       "      <td>5.000000</td>\n",
       "      <td>9.000000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>8.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>90.900000</td>\n",
       "      <td>9.100000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>5.385925e+04</td>\n",
       "      <td>23536.750000</td>\n",
       "      <td>8000.000000</td>\n",
       "      <td>16945.500000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3.0</td>\n",
       "      <td>48.422500</td>\n",
       "      <td>3.0</td>\n",
       "      <td>4.750000</td>\n",
       "      <td>145.267500</td>\n",
       "      <td>5622.290000</td>\n",
       "      <td>50.665000</td>\n",
       "      <td>2069.297500</td>\n",
       "      <td>45.000000</td>\n",
       "      <td>8.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>6.752576e+07</td>\n",
       "      <td>NaN</td>\n",
       "      <td>13975.000000</td>\n",
       "      <td>13975.000000</td>\n",
       "      <td>13900.000000</td>\n",
       "      <td>11.990000</td>\n",
       "      <td>379.390000</td>\n",
       "      <td>6.700000e+04</td>\n",
       "      <td>18.840000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>690.000000</td>\n",
       "      <td>694.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>31.000000</td>\n",
       "      <td>68.00000</td>\n",
       "      <td>11.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>11868.500000</td>\n",
       "      <td>51.40000</td>\n",
       "      <td>24.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>12800.504927</td>\n",
       "      <td>12797.025000</td>\n",
       "      <td>10000.000000</td>\n",
       "      <td>1781.695000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>768.950000</td>\n",
       "      <td>694.000000</td>\n",
       "      <td>690.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>44.000000</td>\n",
       "      <td>1.0</td>\n",
       "      <td>103000.000000</td>\n",
       "      <td>18.220000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>8.466000e+04</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>12.000000</td>\n",
       "      <td>24686.000000</td>\n",
       "      <td>75.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>4483.000000</td>\n",
       "      <td>62.000000</td>\n",
       "      <td>25800.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>4.000000</td>\n",
       "      <td>7606.000000</td>\n",
       "      <td>5000.000000</td>\n",
       "      <td>62.500000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>130.000000</td>\n",
       "      <td>169.000000</td>\n",
       "      <td>8.000000</td>\n",
       "      <td>5.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>13.000000</td>\n",
       "      <td>37.000000</td>\n",
       "      <td>5.000000</td>\n",
       "      <td>32.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>5.000000</td>\n",
       "      <td>4.000000</td>\n",
       "      <td>7.000000</td>\n",
       "      <td>7.000000</td>\n",
       "      <td>8.000000</td>\n",
       "      <td>13.000000</td>\n",
       "      <td>5.000000</td>\n",
       "      <td>11.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>97.400000</td>\n",
       "      <td>40.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.187960e+05</td>\n",
       "      <td>41107.500000</td>\n",
       "      <td>15900.000000</td>\n",
       "      <td>35206.000000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3.0</td>\n",
       "      <td>97.285000</td>\n",
       "      <td>3.0</td>\n",
       "      <td>15.500000</td>\n",
       "      <td>291.855000</td>\n",
       "      <td>9526.755000</td>\n",
       "      <td>122.320000</td>\n",
       "      <td>4172.365000</td>\n",
       "      <td>45.000000</td>\n",
       "      <td>14.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>6.811268e+07</td>\n",
       "      <td>NaN</td>\n",
       "      <td>20000.000000</td>\n",
       "      <td>20000.000000</td>\n",
       "      <td>20000.000000</td>\n",
       "      <td>14.480000</td>\n",
       "      <td>579.130000</td>\n",
       "      <td>9.500000e+04</td>\n",
       "      <td>25.660000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>710.000000</td>\n",
       "      <td>714.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>49.000000</td>\n",
       "      <td>82.00000</td>\n",
       "      <td>15.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>21223.250000</td>\n",
       "      <td>70.10000</td>\n",
       "      <td>32.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>20863.242500</td>\n",
       "      <td>20860.462500</td>\n",
       "      <td>16500.000000</td>\n",
       "      <td>3634.725000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>5961.932500</td>\n",
       "      <td>734.000000</td>\n",
       "      <td>730.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>63.000000</td>\n",
       "      <td>1.0</td>\n",
       "      <td>140000.000000</td>\n",
       "      <td>23.430000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>2.131185e+05</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>4.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>23.000000</td>\n",
       "      <td>47858.000000</td>\n",
       "      <td>88.000000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>4.000000</td>\n",
       "      <td>7771.500000</td>\n",
       "      <td>75.000000</td>\n",
       "      <td>43400.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>7.000000</td>\n",
       "      <td>18104.750000</td>\n",
       "      <td>13054.250000</td>\n",
       "      <td>84.600000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>153.000000</td>\n",
       "      <td>237.000000</td>\n",
       "      <td>16.000000</td>\n",
       "      <td>9.000000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>28.000000</td>\n",
       "      <td>59.000000</td>\n",
       "      <td>10.000000</td>\n",
       "      <td>51.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>5.000000</td>\n",
       "      <td>7.000000</td>\n",
       "      <td>6.000000</td>\n",
       "      <td>10.000000</td>\n",
       "      <td>12.000000</td>\n",
       "      <td>11.000000</td>\n",
       "      <td>19.000000</td>\n",
       "      <td>7.000000</td>\n",
       "      <td>15.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>100.000000</td>\n",
       "      <td>75.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>2.593810e+05</td>\n",
       "      <td>68325.000000</td>\n",
       "      <td>29700.000000</td>\n",
       "      <td>61790.750000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3.0</td>\n",
       "      <td>178.272500</td>\n",
       "      <td>3.0</td>\n",
       "      <td>24.250000</td>\n",
       "      <td>528.727500</td>\n",
       "      <td>15028.415000</td>\n",
       "      <td>269.147500</td>\n",
       "      <td>7160.005000</td>\n",
       "      <td>50.000000</td>\n",
       "      <td>18.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>6.861706e+07</td>\n",
       "      <td>NaN</td>\n",
       "      <td>35000.000000</td>\n",
       "      <td>35000.000000</td>\n",
       "      <td>35000.000000</td>\n",
       "      <td>28.990000</td>\n",
       "      <td>1354.660000</td>\n",
       "      <td>3.964280e+06</td>\n",
       "      <td>999.000000</td>\n",
       "      <td>15.000000</td>\n",
       "      <td>845.000000</td>\n",
       "      <td>850.000000</td>\n",
       "      <td>5.000000</td>\n",
       "      <td>120.000000</td>\n",
       "      <td>120.00000</td>\n",
       "      <td>65.000000</td>\n",
       "      <td>23.000000</td>\n",
       "      <td>647310.000000</td>\n",
       "      <td>134.30000</td>\n",
       "      <td>111.000000</td>\n",
       "      <td>22692.220000</td>\n",
       "      <td>22692.220000</td>\n",
       "      <td>57291.001148</td>\n",
       "      <td>57291.000000</td>\n",
       "      <td>35000.000000</td>\n",
       "      <td>23336.790000</td>\n",
       "      <td>724.800001</td>\n",
       "      <td>23876.710000</td>\n",
       "      <td>4297.807800</td>\n",
       "      <td>36127.510000</td>\n",
       "      <td>850.000000</td>\n",
       "      <td>845.000000</td>\n",
       "      <td>4.000000</td>\n",
       "      <td>130.000000</td>\n",
       "      <td>1.0</td>\n",
       "      <td>500000.000000</td>\n",
       "      <td>39.500000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>140842.000000</td>\n",
       "      <td>2.127709e+06</td>\n",
       "      <td>14.000000</td>\n",
       "      <td>40.000000</td>\n",
       "      <td>12.000000</td>\n",
       "      <td>19.000000</td>\n",
       "      <td>363.000000</td>\n",
       "      <td>878459.000000</td>\n",
       "      <td>223.000000</td>\n",
       "      <td>22.000000</td>\n",
       "      <td>43.000000</td>\n",
       "      <td>127305.000000</td>\n",
       "      <td>151.000000</td>\n",
       "      <td>755900.000000</td>\n",
       "      <td>17.000000</td>\n",
       "      <td>35.000000</td>\n",
       "      <td>32.000000</td>\n",
       "      <td>50.000000</td>\n",
       "      <td>334017.000000</td>\n",
       "      <td>263953.000000</td>\n",
       "      <td>139.500000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>88216.000000</td>\n",
       "      <td>436.000000</td>\n",
       "      <td>707.000000</td>\n",
       "      <td>192.000000</td>\n",
       "      <td>192.000000</td>\n",
       "      <td>25.000000</td>\n",
       "      <td>524.000000</td>\n",
       "      <td>189.000000</td>\n",
       "      <td>24.000000</td>\n",
       "      <td>153.000000</td>\n",
       "      <td>32.000000</td>\n",
       "      <td>25.000000</td>\n",
       "      <td>35.000000</td>\n",
       "      <td>32.000000</td>\n",
       "      <td>44.000000</td>\n",
       "      <td>89.000000</td>\n",
       "      <td>50.000000</td>\n",
       "      <td>80.000000</td>\n",
       "      <td>35.000000</td>\n",
       "      <td>65.000000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>13.000000</td>\n",
       "      <td>25.000000</td>\n",
       "      <td>100.000000</td>\n",
       "      <td>100.000000</td>\n",
       "      <td>8.000000</td>\n",
       "      <td>22.000000</td>\n",
       "      <td>2.388482e+06</td>\n",
       "      <td>878659.000000</td>\n",
       "      <td>303000.000000</td>\n",
       "      <td>591177.000000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3.0</td>\n",
       "      <td>629.700000</td>\n",
       "      <td>3.0</td>\n",
       "      <td>30.000000</td>\n",
       "      <td>1889.100000</td>\n",
       "      <td>28479.590000</td>\n",
       "      <td>780.050000</td>\n",
       "      <td>17500.000000</td>\n",
       "      <td>67.450000</td>\n",
       "      <td>36.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                 id  member_id     loan_amnt   funded_amnt  funded_amnt_inv  \\\n",
       "count  2.251100e+04        0.0  22511.000000  22511.000000     22511.000000   \n",
       "mean   6.756447e+07        NaN  15031.787793  15031.787793     15025.527520   \n",
       "std    1.843754e+06        NaN   8734.651784   8734.651784      8729.680902   \n",
       "min    3.617740e+05        NaN   1000.000000   1000.000000      1000.000000   \n",
       "25%    6.729818e+07        NaN   8000.000000   8000.000000      8000.000000   \n",
       "50%    6.752576e+07        NaN  13975.000000  13975.000000     13900.000000   \n",
       "75%    6.811268e+07        NaN  20000.000000  20000.000000     20000.000000   \n",
       "max    6.861706e+07        NaN  35000.000000  35000.000000     35000.000000   \n",
       "\n",
       "           int_rate   installment    annual_inc           dti   delinq_2yrs  \\\n",
       "count  22511.000000  22511.000000  2.251100e+04  22509.000000  22510.000000   \n",
       "mean      12.255020    436.321286  7.820695e+04     19.423974      0.335584   \n",
       "std        4.213027    249.411778  6.125830e+04     11.006613      0.887691   \n",
       "min        5.320000     30.540000  0.000000e+00      0.000000      0.000000   \n",
       "25%        9.170000    255.040000  4.800000e+04     12.700000      0.000000   \n",
       "50%       11.990000    379.390000  6.700000e+04     18.840000      0.000000   \n",
       "75%       14.480000    579.130000  9.500000e+04     25.660000      0.000000   \n",
       "max       28.990000   1354.660000  3.964280e+06    999.000000     15.000000   \n",
       "\n",
       "       fico_range_low  fico_range_high  inq_last_6mths  \\\n",
       "count    22510.000000     22510.000000    22510.000000   \n",
       "mean       695.075522       699.075611        0.608796   \n",
       "std         31.228687        31.229115        0.887993   \n",
       "min        660.000000       664.000000        0.000000   \n",
       "25%        670.000000       674.000000        0.000000   \n",
       "50%        690.000000       694.000000        0.000000   \n",
       "75%        710.000000       714.000000        1.000000   \n",
       "max        845.000000       850.000000        5.000000   \n",
       "\n",
       "       mths_since_last_delinq  mths_since_last_record      open_acc  \\\n",
       "count            11637.000000              4038.00000  22510.000000   \n",
       "mean                34.162929                65.69787     12.140116   \n",
       "std                 21.863500                23.81295      5.776034   \n",
       "min                  0.000000                 0.00000      1.000000   \n",
       "25%                 15.000000                51.00000      8.000000   \n",
       "50%                 31.000000                68.00000     11.000000   \n",
       "75%                 49.000000                82.00000     15.000000   \n",
       "max                120.000000               120.00000     65.000000   \n",
       "\n",
       "            pub_rec      revol_bal   revol_util     total_acc     out_prncp  \\\n",
       "count  22510.000000   22510.000000  22499.00000  22510.000000  22510.000000   \n",
       "mean       0.238250   17388.656908     51.57675     25.538783    994.961977   \n",
       "std        0.646684   22622.637023     24.22553     12.195480   3018.661632   \n",
       "min        0.000000       0.000000      0.00000      4.000000      0.000000   \n",
       "25%        0.000000    6256.000000     33.40000     17.000000      0.000000   \n",
       "50%        0.000000   11868.500000     51.40000     24.000000      0.000000   \n",
       "75%        0.000000   21223.250000     70.10000     32.000000      0.000000   \n",
       "max       23.000000  647310.000000    134.30000    111.000000  22692.220000   \n",
       "\n",
       "       out_prncp_inv   total_pymnt  total_pymnt_inv  total_rec_prncp  \\\n",
       "count   22510.000000  22510.000000     22510.000000     22510.000000   \n",
       "mean      994.471550  15043.269607     15037.171987     12062.974932   \n",
       "std      3017.074677   9788.105024      9782.869853      8384.981147   \n",
       "min         0.000000      0.000000         0.000000         0.000000   \n",
       "25%         0.000000   7359.465000      7357.160000      5670.825000   \n",
       "50%         0.000000  12800.504927     12797.025000     10000.000000   \n",
       "75%         0.000000  20863.242500     20860.462500     16500.000000   \n",
       "max     22692.220000  57291.001148     57291.000000     35000.000000   \n",
       "\n",
       "       total_rec_int  total_rec_late_fee    recoveries  \\\n",
       "count   22510.000000        22510.000000  22510.000000   \n",
       "mean     2748.012439            2.121175    230.161043   \n",
       "std      2769.683951           14.109329    967.317167   \n",
       "min         0.000000            0.000000      0.000000   \n",
       "25%       895.707500            0.000000      0.000000   \n",
       "50%      1781.695000            0.000000      0.000000   \n",
       "75%      3634.725000            0.000000      0.000000   \n",
       "max     23336.790000          724.800001  23876.710000   \n",
       "\n",
       "       collection_recovery_fee  last_pymnt_amnt  last_fico_range_high  \\\n",
       "count             22510.000000     22510.000000          22510.000000   \n",
       "mean                 41.006250      4262.523270            677.663172   \n",
       "std                 173.583214      6493.202576             79.365744   \n",
       "min                   0.000000         0.000000              0.000000   \n",
       "25%                   0.000000       332.152500            629.000000   \n",
       "50%                   0.000000       768.950000            694.000000   \n",
       "75%                   0.000000      5961.932500            734.000000   \n",
       "max                4297.807800     36127.510000            850.000000   \n",
       "\n",
       "       last_fico_range_low  collections_12_mths_ex_med  \\\n",
       "count         22510.000000                22510.000000   \n",
       "mean            661.832519                    0.023678   \n",
       "std             127.447170                    0.166284   \n",
       "min               0.000000                    0.000000   \n",
       "25%             625.000000                    0.000000   \n",
       "50%             690.000000                    0.000000   \n",
       "75%             730.000000                    0.000000   \n",
       "max             845.000000                    4.000000   \n",
       "\n",
       "       mths_since_last_major_derog  policy_code  annual_inc_joint   dti_joint  \\\n",
       "count                  6466.000000      22510.0        169.000000  169.000000   \n",
       "mean                     44.457934          1.0     114705.080296   18.990828   \n",
       "std                      21.498643          0.0      59397.290685    7.427075   \n",
       "min                       0.000000          1.0      28000.000000    3.000000   \n",
       "25%                      27.000000          1.0      78200.000000   13.830000   \n",
       "50%                      44.000000          1.0     103000.000000   18.220000   \n",
       "75%                      63.000000          1.0     140000.000000   23.430000   \n",
       "max                     130.000000          1.0     500000.000000   39.500000   \n",
       "\n",
       "       acc_now_delinq   tot_coll_amt   tot_cur_bal   open_acc_6m  \\\n",
       "count    22510.000000   22510.000000  2.251000e+04  21344.000000   \n",
       "mean         0.004798     279.812306  1.435598e+05      1.108602   \n",
       "std          0.071004    2287.920339  1.553523e+05      1.242477   \n",
       "min          0.000000       0.000000  0.000000e+00      0.000000   \n",
       "25%          0.000000       0.000000  3.219325e+04      0.000000   \n",
       "50%          0.000000       0.000000  8.466000e+04      1.000000   \n",
       "75%          0.000000       0.000000  2.131185e+05      2.000000   \n",
       "max          2.000000  140842.000000  2.127709e+06     14.000000   \n",
       "\n",
       "        open_act_il   open_il_12m   open_il_24m  mths_since_rcnt_il  \\\n",
       "count  21344.000000  21344.000000  21344.000000        20785.000000   \n",
       "mean       2.928879      0.761525      1.674100           20.917873   \n",
       "std        3.089567      0.995896      1.687826           27.213235   \n",
       "min        0.000000      0.000000      0.000000            0.000000   \n",
       "25%        1.000000      0.000000      0.000000            6.000000   \n",
       "50%        2.000000      0.000000      1.000000           12.000000   \n",
       "75%        4.000000      1.000000      2.000000           23.000000   \n",
       "max       40.000000     12.000000     19.000000          363.000000   \n",
       "\n",
       "        total_bal_il       il_util   open_rv_12m   open_rv_24m     max_bal_bc  \\\n",
       "count   21344.000000  18595.000000  21344.000000  21344.000000   21344.000000   \n",
       "mean    36538.676068     71.576499      1.388587      2.974747    5887.961207   \n",
       "std     43079.900165     23.018098      1.519849      2.630944    5284.130759   \n",
       "min         0.000000      0.000000      0.000000      0.000000       0.000000   \n",
       "25%     10249.250000     59.000000      0.000000      1.000000    2411.750000   \n",
       "50%     24686.000000     75.000000      1.000000      2.000000    4483.000000   \n",
       "75%     47858.000000     88.000000      2.000000      4.000000    7771.500000   \n",
       "max    878459.000000    223.000000     22.000000     43.000000  127305.000000   \n",
       "\n",
       "           all_util  total_rev_hi_lim        inq_fi   total_cu_tl  \\\n",
       "count  21344.000000      22510.000000  21344.000000  21344.000000   \n",
       "mean      60.874859      34865.387161      0.943919      1.537341   \n",
       "std       20.016522      34215.881650      1.446445      2.721517   \n",
       "min        0.000000          0.000000      0.000000      0.000000   \n",
       "25%       48.000000      14800.000000      0.000000      0.000000   \n",
       "50%       62.000000      25800.000000      0.000000      0.000000   \n",
       "75%       75.000000      43400.000000      1.000000      2.000000   \n",
       "max      151.000000     755900.000000     17.000000     35.000000   \n",
       "\n",
       "       inq_last_12m  acc_open_past_24mths    avg_cur_bal  bc_open_to_buy  \\\n",
       "count  21344.000000          22510.000000   22510.000000    22296.000000   \n",
       "mean       2.233602              4.876499   13311.317859    10591.638366   \n",
       "std        2.479804              3.304034   16003.715498    15741.883996   \n",
       "min        0.000000              0.000000       0.000000        0.000000   \n",
       "25%        0.000000              3.000000    3275.250000     1523.750000   \n",
       "50%        2.000000              4.000000    7606.000000     5000.000000   \n",
       "75%        3.000000              7.000000   18104.750000    13054.250000   \n",
       "max       32.000000             50.000000  334017.000000   263953.000000   \n",
       "\n",
       "            bc_util  chargeoff_within_12_mths   delinq_amnt  \\\n",
       "count  22282.000000              22510.000000  22510.000000   \n",
       "mean      59.810511                  0.008574     14.074500   \n",
       "std       28.130991                  0.099612    815.215656   \n",
       "min        0.000000                  0.000000      0.000000   \n",
       "25%       38.100000                  0.000000      0.000000   \n",
       "50%       62.500000                  0.000000      0.000000   \n",
       "75%       84.600000                  0.000000      0.000000   \n",
       "max      139.500000                  3.000000  88216.000000   \n",
       "\n",
       "       mo_sin_old_il_acct  mo_sin_old_rev_tl_op  mo_sin_rcnt_rev_tl_op  \\\n",
       "count        21919.000000          22510.000000           22510.000000   \n",
       "mean           127.087823            185.524878              13.055664   \n",
       "std             52.306451             95.945983              17.018234   \n",
       "min              1.000000              6.000000               0.000000   \n",
       "25%            100.000000            119.000000               3.000000   \n",
       "50%            130.000000            169.000000               8.000000   \n",
       "75%            153.000000            237.000000              16.000000   \n",
       "max            436.000000            707.000000             192.000000   \n",
       "\n",
       "       mo_sin_rcnt_tl      mort_acc  mths_since_recent_bc  \\\n",
       "count    22510.000000  22510.000000          22303.000000   \n",
       "mean         7.452466      1.653354             23.932700   \n",
       "std          8.978085      1.935176             31.672653   \n",
       "min          0.000000      0.000000              0.000000   \n",
       "25%          2.000000      0.000000              6.000000   \n",
       "50%          5.000000      1.000000             13.000000   \n",
       "75%          9.000000      3.000000             28.000000   \n",
       "max        192.000000     25.000000            524.000000   \n",
       "\n",
       "       mths_since_recent_bc_dlq  mths_since_recent_inq  \\\n",
       "count               5715.000000           20159.000000   \n",
       "mean                  39.150306               6.458108   \n",
       "std                   23.252152               5.826340   \n",
       "min                    0.000000               0.000000   \n",
       "25%                   20.000000               2.000000   \n",
       "50%                   37.000000               5.000000   \n",
       "75%                   59.000000              10.000000   \n",
       "max                  189.000000              24.000000   \n",
       "\n",
       "       mths_since_recent_revol_delinq  num_accts_ever_120_pd  num_actv_bc_tl  \\\n",
       "count                     8085.000000           22510.000000    22510.000000   \n",
       "mean                        35.480891               0.529542        3.725011   \n",
       "std                         22.672464               1.371891        2.333436   \n",
       "min                          0.000000               0.000000        0.000000   \n",
       "25%                         16.000000               0.000000        2.000000   \n",
       "50%                         32.000000               0.000000        3.000000   \n",
       "75%                         51.000000               0.000000        5.000000   \n",
       "max                        153.000000              32.000000       25.000000   \n",
       "\n",
       "       num_actv_rev_tl   num_bc_sats     num_bc_tl     num_il_tl  \\\n",
       "count     22510.000000  22510.000000  22510.000000  22510.000000   \n",
       "mean          5.692625      4.839627      8.000666      8.873256   \n",
       "std           3.366398      3.085935      4.805447      7.593589   \n",
       "min           0.000000      0.000000      0.000000      0.000000   \n",
       "25%           3.000000      3.000000      5.000000      4.000000   \n",
       "50%           5.000000      4.000000      7.000000      7.000000   \n",
       "75%           7.000000      6.000000     10.000000     12.000000   \n",
       "max          35.000000     32.000000     44.000000     89.000000   \n",
       "\n",
       "       num_op_rev_tl  num_rev_accts  num_rev_tl_bal_gt_0      num_sats  \\\n",
       "count   22510.000000   22510.000000         22510.000000  22510.000000   \n",
       "mean        8.494536      14.734651             5.740338     12.091870   \n",
       "std         4.798359       8.326798             3.342459      5.758941   \n",
       "min         0.000000       2.000000             0.000000      1.000000   \n",
       "25%         5.000000       9.000000             3.000000      8.000000   \n",
       "50%         8.000000      13.000000             5.000000     11.000000   \n",
       "75%        11.000000      19.000000             7.000000     15.000000   \n",
       "max        50.000000      80.000000            35.000000     65.000000   \n",
       "\n",
       "       num_tl_120dpd_2m  num_tl_30dpd  num_tl_90g_dpd_24m  num_tl_op_past_12m  \\\n",
       "count      21560.000000  22510.000000        22510.000000        22510.000000   \n",
       "mean           0.000510      0.003199            0.088139            2.270324   \n",
       "std            0.024551      0.057248            0.493106            1.912824   \n",
       "min            0.000000      0.000000            0.000000            0.000000   \n",
       "25%            0.000000      0.000000            0.000000            1.000000   \n",
       "50%            0.000000      0.000000            0.000000            2.000000   \n",
       "75%            0.000000      0.000000            0.000000            3.000000   \n",
       "max            2.000000      2.000000           13.000000           25.000000   \n",
       "\n",
       "       pct_tl_nvr_dlq  percent_bc_gt_75  pub_rec_bankruptcies     tax_liens  \\\n",
       "count    22510.000000      22282.000000          22510.000000  22510.000000   \n",
       "mean        93.986219         44.643582              0.132652      0.070013   \n",
       "std          8.831363         36.248053              0.381843      0.446809   \n",
       "min         12.500000          0.000000              0.000000      0.000000   \n",
       "25%         90.900000          9.100000              0.000000      0.000000   \n",
       "50%         97.400000         40.000000              0.000000      0.000000   \n",
       "75%        100.000000         75.000000              0.000000      0.000000   \n",
       "max        100.000000        100.000000              8.000000     22.000000   \n",
       "\n",
       "       tot_hi_cred_lim  total_bal_ex_mort  total_bc_limit  \\\n",
       "count     2.251000e+04       22510.000000    22510.000000   \n",
       "mean      1.788133e+05       53921.267614    22772.761972   \n",
       "std       1.730730e+05       49517.560009    22403.564952   \n",
       "min       2.500000e+03           0.000000        0.000000   \n",
       "25%       5.385925e+04       23536.750000     8000.000000   \n",
       "50%       1.187960e+05       41107.500000    15900.000000   \n",
       "75%       2.593810e+05       68325.000000    29700.000000   \n",
       "max       2.388482e+06      878659.000000   303000.000000   \n",
       "\n",
       "       total_il_high_credit_limit  revol_bal_joint  sec_app_fico_range_low  \\\n",
       "count                22510.000000              0.0                     0.0   \n",
       "mean                 45965.028698              NaN                     NaN   \n",
       "std                  45041.561576              NaN                     NaN   \n",
       "min                      0.000000              NaN                     NaN   \n",
       "25%                  16945.500000              NaN                     NaN   \n",
       "50%                  35206.000000              NaN                     NaN   \n",
       "75%                  61790.750000              NaN                     NaN   \n",
       "max                 591177.000000              NaN                     NaN   \n",
       "\n",
       "       sec_app_fico_range_high  sec_app_earliest_cr_line  \\\n",
       "count                      0.0                       0.0   \n",
       "mean                       NaN                       NaN   \n",
       "std                        NaN                       NaN   \n",
       "min                        NaN                       NaN   \n",
       "25%                        NaN                       NaN   \n",
       "50%                        NaN                       NaN   \n",
       "75%                        NaN                       NaN   \n",
       "max                        NaN                       NaN   \n",
       "\n",
       "       sec_app_inq_last_6mths  sec_app_mort_acc  sec_app_open_acc  \\\n",
       "count                     0.0               0.0               0.0   \n",
       "mean                      NaN               NaN               NaN   \n",
       "std                       NaN               NaN               NaN   \n",
       "min                       NaN               NaN               NaN   \n",
       "25%                       NaN               NaN               NaN   \n",
       "50%                       NaN               NaN               NaN   \n",
       "75%                       NaN               NaN               NaN   \n",
       "max                       NaN               NaN               NaN   \n",
       "\n",
       "       sec_app_revol_util  sec_app_open_act_il  sec_app_num_rev_accts  \\\n",
       "count                 0.0                  0.0                    0.0   \n",
       "mean                  NaN                  NaN                    NaN   \n",
       "std                   NaN                  NaN                    NaN   \n",
       "min                   NaN                  NaN                    NaN   \n",
       "25%                   NaN                  NaN                    NaN   \n",
       "50%                   NaN                  NaN                    NaN   \n",
       "75%                   NaN                  NaN                    NaN   \n",
       "max                   NaN                  NaN                    NaN   \n",
       "\n",
       "       sec_app_chargeoff_within_12_mths  sec_app_collections_12_mths_ex_med  \\\n",
       "count                               0.0                                 0.0   \n",
       "mean                                NaN                                 NaN   \n",
       "std                                 NaN                                 NaN   \n",
       "min                                 NaN                                 NaN   \n",
       "25%                                 NaN                                 NaN   \n",
       "50%                                 NaN                                 NaN   \n",
       "75%                                 NaN                                 NaN   \n",
       "max                                 NaN                                 NaN   \n",
       "\n",
       "       sec_app_mths_since_last_major_derog  deferral_term  hardship_amount  \\\n",
       "count                                  0.0          180.0       180.000000   \n",
       "mean                                   NaN            3.0       130.504278   \n",
       "std                                    NaN            0.0       108.186827   \n",
       "min                                    NaN            3.0         5.280000   \n",
       "25%                                    NaN            3.0        48.422500   \n",
       "50%                                    NaN            3.0        97.285000   \n",
       "75%                                    NaN            3.0       178.272500   \n",
       "max                                    NaN            3.0       629.700000   \n",
       "\n",
       "       hardship_length  hardship_dpd  \\\n",
       "count            180.0    180.000000   \n",
       "mean               3.0     14.272222   \n",
       "std                0.0     10.296081   \n",
       "min                3.0      0.000000   \n",
       "25%                3.0      4.750000   \n",
       "50%                3.0     15.500000   \n",
       "75%                3.0     24.250000   \n",
       "max                3.0     30.000000   \n",
       "\n",
       "       orig_projected_additional_accrued_interest  \\\n",
       "count                                  144.000000   \n",
       "mean                                   390.468542   \n",
       "std                                    320.493440   \n",
       "min                                     33.840000   \n",
       "25%                                    145.267500   \n",
       "50%                                    291.855000   \n",
       "75%                                    528.727500   \n",
       "max                                   1889.100000   \n",
       "\n",
       "       hardship_payoff_balance_amount  hardship_last_payment_amount  \\\n",
       "count                      180.000000                    180.000000   \n",
       "mean                     10659.213444                    183.667111   \n",
       "std                       6503.722235                    178.053547   \n",
       "min                        594.070000                      0.060000   \n",
       "25%                       5622.290000                     50.665000   \n",
       "50%                       9526.755000                    122.320000   \n",
       "75%                      15028.415000                    269.147500   \n",
       "max                      28479.590000                    780.050000   \n",
       "\n",
       "       settlement_amount  settlement_percentage  settlement_term  \n",
       "count         662.000000             662.000000       662.000000  \n",
       "mean         4980.598958              47.125378        13.823263  \n",
       "std          3596.196186               5.255945         7.501520  \n",
       "min           250.000000              30.000000         0.000000  \n",
       "25%          2069.297500              45.000000         8.000000  \n",
       "50%          4172.365000              45.000000        14.000000  \n",
       "75%          7160.005000              50.000000        18.000000  \n",
       "max         17500.000000              67.450000        36.000000  "
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Get a statistical summary of the dataset\n",
    "df.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here are some of the key statistics:\n",
    "\n",
    "- The average loan amount (loan_amnt) is approximately $15,031.8.\n",
    "- The minimum loan amount is $1,000, and the maximum is $35,000.\n",
    "- The average interest rate (int_rate) is approximately 12.255%.\n",
    "- The average annual income (annual_inc) of borrowers is approximately $78,206.9."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "id                                                0\n",
       "member_id                                     22511\n",
       "loan_amnt                                         0\n",
       "funded_amnt                                       0\n",
       "funded_amnt_inv                                   0\n",
       "term                                              0\n",
       "int_rate                                          0\n",
       "installment                                       0\n",
       "grade                                             0\n",
       "sub_grade                                         0\n",
       "emp_title                                      1349\n",
       "emp_length                                     1340\n",
       "home_ownership                                    0\n",
       "annual_inc                                        0\n",
       "verification_status                               0\n",
       "issue_d                                           0\n",
       "loan_status                                       0\n",
       "pymnt_plan                                        0\n",
       "url                                               0\n",
       "desc                                          22510\n",
       "purpose                                           0\n",
       "title                                           133\n",
       "zip_code                                          1\n",
       "addr_state                                        1\n",
       "dti                                               2\n",
       "delinq_2yrs                                       1\n",
       "earliest_cr_line                                  1\n",
       "fico_range_low                                    1\n",
       "fico_range_high                                   1\n",
       "inq_last_6mths                                    1\n",
       "mths_since_last_delinq                        10874\n",
       "mths_since_last_record                        18473\n",
       "open_acc                                          1\n",
       "pub_rec                                           1\n",
       "revol_bal                                         1\n",
       "revol_util                                       12\n",
       "total_acc                                         1\n",
       "initial_list_status                               1\n",
       "out_prncp                                         1\n",
       "out_prncp_inv                                     1\n",
       "total_pymnt                                       1\n",
       "total_pymnt_inv                                   1\n",
       "total_rec_prncp                                   1\n",
       "total_rec_int                                     1\n",
       "total_rec_late_fee                                1\n",
       "recoveries                                        1\n",
       "collection_recovery_fee                           1\n",
       "last_pymnt_d                                     18\n",
       "last_pymnt_amnt                                   1\n",
       "next_pymnt_d                                  19926\n",
       "last_credit_pull_d                                1\n",
       "last_fico_range_high                              1\n",
       "last_fico_range_low                               1\n",
       "collections_12_mths_ex_med                        1\n",
       "mths_since_last_major_derog                   16045\n",
       "policy_code                                       1\n",
       "application_type                                  1\n",
       "annual_inc_joint                              22342\n",
       "dti_joint                                     22342\n",
       "verification_status_joint                     22342\n",
       "acc_now_delinq                                    1\n",
       "tot_coll_amt                                      1\n",
       "tot_cur_bal                                       1\n",
       "open_acc_6m                                    1167\n",
       "open_act_il                                    1167\n",
       "open_il_12m                                    1167\n",
       "open_il_24m                                    1167\n",
       "mths_since_rcnt_il                             1726\n",
       "total_bal_il                                   1167\n",
       "il_util                                        3916\n",
       "open_rv_12m                                    1167\n",
       "open_rv_24m                                    1167\n",
       "max_bal_bc                                     1167\n",
       "all_util                                       1167\n",
       "total_rev_hi_lim                                  1\n",
       "inq_fi                                         1167\n",
       "total_cu_tl                                    1167\n",
       "inq_last_12m                                   1167\n",
       "acc_open_past_24mths                              1\n",
       "avg_cur_bal                                       1\n",
       "bc_open_to_buy                                  215\n",
       "bc_util                                         229\n",
       "chargeoff_within_12_mths                          1\n",
       "delinq_amnt                                       1\n",
       "mo_sin_old_il_acct                              592\n",
       "mo_sin_old_rev_tl_op                              1\n",
       "mo_sin_rcnt_rev_tl_op                             1\n",
       "mo_sin_rcnt_tl                                    1\n",
       "mort_acc                                          1\n",
       "mths_since_recent_bc                            208\n",
       "mths_since_recent_bc_dlq                      16796\n",
       "mths_since_recent_inq                          2352\n",
       "mths_since_recent_revol_delinq                14426\n",
       "num_accts_ever_120_pd                             1\n",
       "num_actv_bc_tl                                    1\n",
       "num_actv_rev_tl                                   1\n",
       "num_bc_sats                                       1\n",
       "num_bc_tl                                         1\n",
       "num_il_tl                                         1\n",
       "num_op_rev_tl                                     1\n",
       "num_rev_accts                                     1\n",
       "num_rev_tl_bal_gt_0                               1\n",
       "num_sats                                          1\n",
       "num_tl_120dpd_2m                                951\n",
       "num_tl_30dpd                                      1\n",
       "num_tl_90g_dpd_24m                                1\n",
       "num_tl_op_past_12m                                1\n",
       "pct_tl_nvr_dlq                                    1\n",
       "percent_bc_gt_75                                229\n",
       "pub_rec_bankruptcies                              1\n",
       "tax_liens                                         1\n",
       "tot_hi_cred_lim                                   1\n",
       "total_bal_ex_mort                                 1\n",
       "total_bc_limit                                    1\n",
       "total_il_high_credit_limit                        1\n",
       "revol_bal_joint                               22511\n",
       "sec_app_fico_range_low                        22511\n",
       "sec_app_fico_range_high                       22511\n",
       "sec_app_earliest_cr_line                      22511\n",
       "sec_app_inq_last_6mths                        22511\n",
       "sec_app_mort_acc                              22511\n",
       "sec_app_open_acc                              22511\n",
       "sec_app_revol_util                            22511\n",
       "sec_app_open_act_il                           22511\n",
       "sec_app_num_rev_accts                         22511\n",
       "sec_app_chargeoff_within_12_mths              22511\n",
       "sec_app_collections_12_mths_ex_med            22511\n",
       "sec_app_mths_since_last_major_derog           22511\n",
       "hardship_flag                                     1\n",
       "hardship_type                                 22331\n",
       "hardship_reason                               22331\n",
       "hardship_status                               22331\n",
       "deferral_term                                 22331\n",
       "hardship_amount                               22331\n",
       "hardship_start_date                           22331\n",
       "hardship_end_date                             22331\n",
       "payment_plan_start_date                       22331\n",
       "hardship_length                               22331\n",
       "hardship_dpd                                  22331\n",
       "hardship_loan_status                          22331\n",
       "orig_projected_additional_accrued_interest    22367\n",
       "hardship_payoff_balance_amount                22331\n",
       "hardship_last_payment_amount                  22331\n",
       "disbursement_method                               1\n",
       "debt_settlement_flag                              1\n",
       "debt_settlement_flag_date                     21849\n",
       "settlement_status                             21849\n",
       "settlement_date                               21849\n",
       "settlement_amount                             21849\n",
       "settlement_percentage                         21849\n",
       "settlement_term                               21849\n",
       "dtype: int64"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Check the number of null values in each column\n",
    "df.isnull().sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Check for duplicate rows\n",
    "df.duplicated().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Cleaning <a class=\"anchor\" id=\"cleaning\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Handling Missing Value"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "id                                              0.000000\n",
       "member_id                                     100.000000\n",
       "loan_amnt                                       0.000000\n",
       "funded_amnt                                     0.000000\n",
       "funded_amnt_inv                                 0.000000\n",
       "term                                            0.000000\n",
       "int_rate                                        0.000000\n",
       "installment                                     0.000000\n",
       "grade                                           0.000000\n",
       "sub_grade                                       0.000000\n",
       "emp_title                                       5.992626\n",
       "emp_length                                      5.952645\n",
       "home_ownership                                  0.000000\n",
       "annual_inc                                      0.000000\n",
       "verification_status                             0.000000\n",
       "issue_d                                         0.000000\n",
       "loan_status                                     0.000000\n",
       "pymnt_plan                                      0.000000\n",
       "url                                             0.000000\n",
       "desc                                           99.995558\n",
       "purpose                                         0.000000\n",
       "title                                           0.590822\n",
       "zip_code                                        0.004442\n",
       "addr_state                                      0.004442\n",
       "dti                                             0.008885\n",
       "delinq_2yrs                                     0.004442\n",
       "earliest_cr_line                                0.004442\n",
       "fico_range_low                                  0.004442\n",
       "fico_range_high                                 0.004442\n",
       "inq_last_6mths                                  0.004442\n",
       "mths_since_last_delinq                         48.305273\n",
       "mths_since_last_record                         82.062103\n",
       "open_acc                                        0.004442\n",
       "pub_rec                                         0.004442\n",
       "revol_bal                                       0.004442\n",
       "revol_util                                      0.053307\n",
       "total_acc                                       0.004442\n",
       "initial_list_status                             0.004442\n",
       "out_prncp                                       0.004442\n",
       "out_prncp_inv                                   0.004442\n",
       "total_pymnt                                     0.004442\n",
       "total_pymnt_inv                                 0.004442\n",
       "total_rec_prncp                                 0.004442\n",
       "total_rec_int                                   0.004442\n",
       "total_rec_late_fee                              0.004442\n",
       "recoveries                                      0.004442\n",
       "collection_recovery_fee                         0.004442\n",
       "last_pymnt_d                                    0.079961\n",
       "last_pymnt_amnt                                 0.004442\n",
       "next_pymnt_d                                   88.516725\n",
       "last_credit_pull_d                              0.004442\n",
       "last_fico_range_high                            0.004442\n",
       "last_fico_range_low                             0.004442\n",
       "collections_12_mths_ex_med                      0.004442\n",
       "mths_since_last_major_derog                    71.276265\n",
       "policy_code                                     0.004442\n",
       "application_type                                0.004442\n",
       "annual_inc_joint                               99.249256\n",
       "dti_joint                                      99.249256\n",
       "verification_status_joint                      99.249256\n",
       "acc_now_delinq                                  0.004442\n",
       "tot_coll_amt                                    0.004442\n",
       "tot_cur_bal                                     0.004442\n",
       "open_acc_6m                                     5.184132\n",
       "open_act_il                                     5.184132\n",
       "open_il_12m                                     5.184132\n",
       "open_il_24m                                     5.184132\n",
       "mths_since_rcnt_il                              7.667363\n",
       "total_bal_il                                    5.184132\n",
       "il_util                                        17.395940\n",
       "open_rv_12m                                     5.184132\n",
       "open_rv_24m                                     5.184132\n",
       "max_bal_bc                                      5.184132\n",
       "all_util                                        5.184132\n",
       "total_rev_hi_lim                                0.004442\n",
       "inq_fi                                          5.184132\n",
       "total_cu_tl                                     5.184132\n",
       "inq_last_12m                                    5.184132\n",
       "acc_open_past_24mths                            0.004442\n",
       "avg_cur_bal                                     0.004442\n",
       "bc_open_to_buy                                  0.955089\n",
       "bc_util                                         1.017280\n",
       "chargeoff_within_12_mths                        0.004442\n",
       "delinq_amnt                                     0.004442\n",
       "mo_sin_old_il_acct                              2.629825\n",
       "mo_sin_old_rev_tl_op                            0.004442\n",
       "mo_sin_rcnt_rev_tl_op                           0.004442\n",
       "mo_sin_rcnt_tl                                  0.004442\n",
       "mort_acc                                        0.004442\n",
       "mths_since_recent_bc                            0.923993\n",
       "mths_since_recent_bc_dlq                       74.612412\n",
       "mths_since_recent_inq                          10.448225\n",
       "mths_since_recent_revol_delinq                 64.084225\n",
       "num_accts_ever_120_pd                           0.004442\n",
       "num_actv_bc_tl                                  0.004442\n",
       "num_actv_rev_tl                                 0.004442\n",
       "num_bc_sats                                     0.004442\n",
       "num_bc_tl                                       0.004442\n",
       "num_il_tl                                       0.004442\n",
       "num_op_rev_tl                                   0.004442\n",
       "num_rev_accts                                   0.004442\n",
       "num_rev_tl_bal_gt_0                             0.004442\n",
       "num_sats                                        0.004442\n",
       "num_tl_120dpd_2m                                4.224601\n",
       "num_tl_30dpd                                    0.004442\n",
       "num_tl_90g_dpd_24m                              0.004442\n",
       "num_tl_op_past_12m                              0.004442\n",
       "pct_tl_nvr_dlq                                  0.004442\n",
       "percent_bc_gt_75                                1.017280\n",
       "pub_rec_bankruptcies                            0.004442\n",
       "tax_liens                                       0.004442\n",
       "tot_hi_cred_lim                                 0.004442\n",
       "total_bal_ex_mort                               0.004442\n",
       "total_bc_limit                                  0.004442\n",
       "total_il_high_credit_limit                      0.004442\n",
       "revol_bal_joint                               100.000000\n",
       "sec_app_fico_range_low                        100.000000\n",
       "sec_app_fico_range_high                       100.000000\n",
       "sec_app_earliest_cr_line                      100.000000\n",
       "sec_app_inq_last_6mths                        100.000000\n",
       "sec_app_mort_acc                              100.000000\n",
       "sec_app_open_acc                              100.000000\n",
       "sec_app_revol_util                            100.000000\n",
       "sec_app_open_act_il                           100.000000\n",
       "sec_app_num_rev_accts                         100.000000\n",
       "sec_app_chargeoff_within_12_mths              100.000000\n",
       "sec_app_collections_12_mths_ex_med            100.000000\n",
       "sec_app_mths_since_last_major_derog           100.000000\n",
       "hardship_flag                                   0.004442\n",
       "hardship_type                                  99.200391\n",
       "hardship_reason                                99.200391\n",
       "hardship_status                                99.200391\n",
       "deferral_term                                  99.200391\n",
       "hardship_amount                                99.200391\n",
       "hardship_start_date                            99.200391\n",
       "hardship_end_date                              99.200391\n",
       "payment_plan_start_date                        99.200391\n",
       "hardship_length                                99.200391\n",
       "hardship_dpd                                   99.200391\n",
       "hardship_loan_status                           99.200391\n",
       "orig_projected_additional_accrued_interest     99.360313\n",
       "hardship_payoff_balance_amount                 99.200391\n",
       "hardship_last_payment_amount                   99.200391\n",
       "disbursement_method                             0.004442\n",
       "debt_settlement_flag                            0.004442\n",
       "debt_settlement_flag_date                      97.059215\n",
       "settlement_status                              97.059215\n",
       "settlement_date                                97.059215\n",
       "settlement_amount                              97.059215\n",
       "settlement_percentage                          97.059215\n",
       "settlement_term                                97.059215\n",
       "dtype: float64"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Check the percentage of missing values in each column\n",
    "df.isnull().sum() / len(df) * 100"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a copy of the original dataframe\n",
    "df_clean = df.copy()\n",
    "\n",
    "# Get the columns with more than 20% missing values\n",
    "cols_to_drop = df_clean.columns[df_clean.isnull().sum() / len(df_clean) * 100 > 20]\n",
    "\n",
    "# Drop the columns with more than 20% missing values\n",
    "df_clean.drop(cols_to_drop, axis=1, inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(22511, 107)"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Check the shape of the dataframe\n",
    "df_clean.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Methodology:\n",
    "\n",
    "1. Check the percentage of missing values in each column: The code starts by calculating the percentage of missing values in each column of the original DataFrame (df). It uses the isnull() method to create a boolean DataFrame where True represents missing values, then sums the number of missing values for each column and divides it by the total number of rows in the DataFrame to get the percentage of missing values for each column.\n",
    "2. Create a copy of the original dataframe: The code then creates a copy of the original DataFrame df and assigns it to a new DataFrame called df_clean. This step ensures that the original DataFrame remains unchanged, and any modifications are made to the new copy.\n",
    "3. Get the columns with more than 20% missing values: The code identifies the columns in the df_clean DataFrame that have more than 20% missing values. It uses boolean indexing and comparison to filter out the column names that meet the condition (columns with missing values more than 20%).\n",
    "4. Drop the columns with more than 20% missing values: The code drops the identified columns with more than 20% missing values from the df_clean DataFrame using the drop() method along with the cols_to_drop variable containing the column names to be removed. The axis=1 parameter indicates that we are dropping columns.\n",
    "5. Check the shape of the dataframe: After dropping the columns with more than 20% missing values, the code outputs the shape of the resulting df_clean DataFrame, which indicates the number of rows and columns in the cleaned DataFrame.\n",
    "Findings:\n",
    "\n",
    "Findings:\n",
    "\n",
    "1. The original DataFrame df had 22,511 rows and a total of 151 columns.\n",
    "\n",
    "2. After performing data cleaning and dropping the columns with more than 20% missing values, the cleaned DataFrame df_clean now has 22,511 rows and 107 columns.\n",
    "    - This means that there were some columns in the original DataFrame with more than 20% missing values, and they were removed from the cleaned DataFrame to handle missing data. The new DataFrame retains the same number of rows but has fewer columns, resulting in a more manageable and complete dataset for further analysis or modeling."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "id                             0.000000\n",
       "loan_amnt                      0.000000\n",
       "funded_amnt                    0.000000\n",
       "funded_amnt_inv                0.000000\n",
       "term                           0.000000\n",
       "int_rate                       0.000000\n",
       "installment                    0.000000\n",
       "grade                          0.000000\n",
       "sub_grade                      0.000000\n",
       "emp_title                      5.992626\n",
       "emp_length                     5.952645\n",
       "home_ownership                 0.000000\n",
       "annual_inc                     0.000000\n",
       "verification_status            0.000000\n",
       "issue_d                        0.000000\n",
       "loan_status                    0.000000\n",
       "pymnt_plan                     0.000000\n",
       "url                            0.000000\n",
       "purpose                        0.000000\n",
       "title                          0.590822\n",
       "zip_code                       0.004442\n",
       "addr_state                     0.004442\n",
       "dti                            0.008885\n",
       "delinq_2yrs                    0.004442\n",
       "earliest_cr_line               0.004442\n",
       "fico_range_low                 0.004442\n",
       "fico_range_high                0.004442\n",
       "inq_last_6mths                 0.004442\n",
       "open_acc                       0.004442\n",
       "pub_rec                        0.004442\n",
       "revol_bal                      0.004442\n",
       "revol_util                     0.053307\n",
       "total_acc                      0.004442\n",
       "initial_list_status            0.004442\n",
       "out_prncp                      0.004442\n",
       "out_prncp_inv                  0.004442\n",
       "total_pymnt                    0.004442\n",
       "total_pymnt_inv                0.004442\n",
       "total_rec_prncp                0.004442\n",
       "total_rec_int                  0.004442\n",
       "total_rec_late_fee             0.004442\n",
       "recoveries                     0.004442\n",
       "collection_recovery_fee        0.004442\n",
       "last_pymnt_d                   0.079961\n",
       "last_pymnt_amnt                0.004442\n",
       "last_credit_pull_d             0.004442\n",
       "last_fico_range_high           0.004442\n",
       "last_fico_range_low            0.004442\n",
       "collections_12_mths_ex_med     0.004442\n",
       "policy_code                    0.004442\n",
       "application_type               0.004442\n",
       "acc_now_delinq                 0.004442\n",
       "tot_coll_amt                   0.004442\n",
       "tot_cur_bal                    0.004442\n",
       "open_acc_6m                    5.184132\n",
       "open_act_il                    5.184132\n",
       "open_il_12m                    5.184132\n",
       "open_il_24m                    5.184132\n",
       "mths_since_rcnt_il             7.667363\n",
       "total_bal_il                   5.184132\n",
       "il_util                       17.395940\n",
       "open_rv_12m                    5.184132\n",
       "open_rv_24m                    5.184132\n",
       "max_bal_bc                     5.184132\n",
       "all_util                       5.184132\n",
       "total_rev_hi_lim               0.004442\n",
       "inq_fi                         5.184132\n",
       "total_cu_tl                    5.184132\n",
       "inq_last_12m                   5.184132\n",
       "acc_open_past_24mths           0.004442\n",
       "avg_cur_bal                    0.004442\n",
       "bc_open_to_buy                 0.955089\n",
       "bc_util                        1.017280\n",
       "chargeoff_within_12_mths       0.004442\n",
       "delinq_amnt                    0.004442\n",
       "mo_sin_old_il_acct             2.629825\n",
       "mo_sin_old_rev_tl_op           0.004442\n",
       "mo_sin_rcnt_rev_tl_op          0.004442\n",
       "mo_sin_rcnt_tl                 0.004442\n",
       "mort_acc                       0.004442\n",
       "mths_since_recent_bc           0.923993\n",
       "mths_since_recent_inq         10.448225\n",
       "num_accts_ever_120_pd          0.004442\n",
       "num_actv_bc_tl                 0.004442\n",
       "num_actv_rev_tl                0.004442\n",
       "num_bc_sats                    0.004442\n",
       "num_bc_tl                      0.004442\n",
       "num_il_tl                      0.004442\n",
       "num_op_rev_tl                  0.004442\n",
       "num_rev_accts                  0.004442\n",
       "num_rev_tl_bal_gt_0            0.004442\n",
       "num_sats                       0.004442\n",
       "num_tl_120dpd_2m               4.224601\n",
       "num_tl_30dpd                   0.004442\n",
       "num_tl_90g_dpd_24m             0.004442\n",
       "num_tl_op_past_12m             0.004442\n",
       "pct_tl_nvr_dlq                 0.004442\n",
       "percent_bc_gt_75               1.017280\n",
       "pub_rec_bankruptcies           0.004442\n",
       "tax_liens                      0.004442\n",
       "tot_hi_cred_lim                0.004442\n",
       "total_bal_ex_mort              0.004442\n",
       "total_bc_limit                 0.004442\n",
       "total_il_high_credit_limit     0.004442\n",
       "hardship_flag                  0.004442\n",
       "disbursement_method            0.004442\n",
       "debt_settlement_flag           0.004442\n",
       "dtype: float64"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Check the percentage of missing values in each column\n",
    "df_clean.isnull().sum() / len(df_clean) * 100"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's investigate the columns with relatively high missing values comparing to the rest of the dataset:\n",
    "1. il_util (~17% of the rows are missing)\n",
    "\n",
    "2. mths_since_recent_inq (~10% of the rows are missing)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Ratio of total current balance to high credit/credit limit on all install acct\n"
     ]
    }
   ],
   "source": [
    "# Exam the il_util column\n",
    "col_info('il_util')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Interpretation of the column:**\n",
    "\n",
    "The column \"il_util\" refers to a financial metric that calculates the percentage of the total current balance owed on all installment accounts in relation to the high credit or credit limit allowed on those accounts.\n",
    "\n",
    "**Timing of Variable in the Loan Application Process:**\n",
    "\n",
    "Arises **after** Loan Disbursement.\n",
    "\n",
    "Information about the current balances on installment accounts is determined after the loan has been disbursed and the borrower starts using their installment accounts.\n",
    "\n",
    "**Decision on data handling:**\n",
    "\n",
    "Since it arises after loan disbursement, it is not useful for loan prediction analysis, let's **drop** this column.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "# drop the il_util column\n",
    "df_clean.drop('il_util', axis=1, inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Months since most recent inquiry.\n"
     ]
    }
   ],
   "source": [
    "# Exam the mths_since_recent_inq column\n",
    "col_info('mths_since_recent_inq')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Interpretation of the column:**\n",
    "\n",
    "The \"mths_since_recent_inq\" column in the LendingClub loan data tells us the time that has passed since the last hard credit inquiry was made on the borrower's credit report at the time of the loan application. \n",
    "\n",
    "It provides information about how actively the borrower has been seeking credit recently, and it can be a factor considered by LendingClub when evaluating a loan application. Generally, a longer period since the last inquiry is considered more favorable by lenders, as it suggests that the borrower is not taking on new credit obligations too frequently.\n",
    "\n",
    "**Timing of Variable in the Loan Application Process:**\n",
    "\n",
    "Arises **before** Loan Disbursement.\n",
    "\n",
    "Credit inquiries occur when LendingClub check the borrower's credit report as part of the loan application process. It reflects the borrower's recent credit activity and credit-seeking behavior at the time of applying for the loan.\n",
    "\n",
    "**Decision on data handling:**\n",
    "\n",
    "Since it arises before loan disbursement, it is useful for loan prediction analysis, let's **keep** this column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "id                            0.000000\n",
       "loan_amnt                     0.000000\n",
       "funded_amnt                   0.000000\n",
       "funded_amnt_inv               0.000000\n",
       "term                          0.000000\n",
       "int_rate                      0.000000\n",
       "installment                   0.000000\n",
       "grade                         0.000000\n",
       "sub_grade                     0.000000\n",
       "emp_title                     5.992626\n",
       "emp_length                    5.952645\n",
       "home_ownership                0.000000\n",
       "annual_inc                    0.000000\n",
       "verification_status           0.000000\n",
       "issue_d                       0.000000\n",
       "loan_status                   0.000000\n",
       "pymnt_plan                    0.000000\n",
       "url                           0.000000\n",
       "purpose                       0.000000\n",
       "title                         0.590822\n",
       "zip_code                      0.004442\n",
       "addr_state                    0.004442\n",
       "dti                           0.000000\n",
       "delinq_2yrs                   0.000000\n",
       "earliest_cr_line              0.004442\n",
       "fico_range_low                0.000000\n",
       "fico_range_high               0.000000\n",
       "inq_last_6mths                0.000000\n",
       "open_acc                      0.000000\n",
       "pub_rec                       0.000000\n",
       "revol_bal                     0.000000\n",
       "revol_util                    0.000000\n",
       "total_acc                     0.000000\n",
       "initial_list_status           0.004442\n",
       "out_prncp                     0.000000\n",
       "out_prncp_inv                 0.000000\n",
       "total_pymnt                   0.000000\n",
       "total_pymnt_inv               0.000000\n",
       "total_rec_prncp               0.000000\n",
       "total_rec_int                 0.000000\n",
       "total_rec_late_fee            0.000000\n",
       "recoveries                    0.000000\n",
       "collection_recovery_fee       0.000000\n",
       "last_pymnt_d                  0.079961\n",
       "last_pymnt_amnt               0.000000\n",
       "last_credit_pull_d            0.004442\n",
       "last_fico_range_high          0.000000\n",
       "last_fico_range_low           0.000000\n",
       "collections_12_mths_ex_med    0.000000\n",
       "policy_code                   0.000000\n",
       "application_type              0.004442\n",
       "acc_now_delinq                0.000000\n",
       "tot_coll_amt                  0.000000\n",
       "tot_cur_bal                   0.000000\n",
       "open_acc_6m                   0.000000\n",
       "open_act_il                   0.000000\n",
       "open_il_12m                   0.000000\n",
       "open_il_24m                   0.000000\n",
       "mths_since_rcnt_il            0.000000\n",
       "total_bal_il                  0.000000\n",
       "open_rv_12m                   0.000000\n",
       "open_rv_24m                   0.000000\n",
       "max_bal_bc                    0.000000\n",
       "all_util                      0.000000\n",
       "total_rev_hi_lim              0.000000\n",
       "inq_fi                        0.000000\n",
       "total_cu_tl                   0.000000\n",
       "inq_last_12m                  0.000000\n",
       "acc_open_past_24mths          0.000000\n",
       "avg_cur_bal                   0.000000\n",
       "bc_open_to_buy                0.000000\n",
       "bc_util                       0.000000\n",
       "chargeoff_within_12_mths      0.000000\n",
       "delinq_amnt                   0.000000\n",
       "mo_sin_old_il_acct            0.000000\n",
       "mo_sin_old_rev_tl_op          0.000000\n",
       "mo_sin_rcnt_rev_tl_op         0.000000\n",
       "mo_sin_rcnt_tl                0.000000\n",
       "mort_acc                      0.000000\n",
       "mths_since_recent_bc          0.000000\n",
       "mths_since_recent_inq         0.000000\n",
       "num_accts_ever_120_pd         0.000000\n",
       "num_actv_bc_tl                0.000000\n",
       "num_actv_rev_tl               0.000000\n",
       "num_bc_sats                   0.000000\n",
       "num_bc_tl                     0.000000\n",
       "num_il_tl                     0.000000\n",
       "num_op_rev_tl                 0.000000\n",
       "num_rev_accts                 0.000000\n",
       "num_rev_tl_bal_gt_0           0.000000\n",
       "num_sats                      0.000000\n",
       "num_tl_120dpd_2m              0.000000\n",
       "num_tl_30dpd                  0.000000\n",
       "num_tl_90g_dpd_24m            0.000000\n",
       "num_tl_op_past_12m            0.000000\n",
       "pct_tl_nvr_dlq                0.000000\n",
       "percent_bc_gt_75              0.000000\n",
       "pub_rec_bankruptcies          0.000000\n",
       "tax_liens                     0.000000\n",
       "tot_hi_cred_lim               0.000000\n",
       "total_bal_ex_mort             0.000000\n",
       "total_bc_limit                0.000000\n",
       "total_il_high_credit_limit    0.000000\n",
       "hardship_flag                 0.004442\n",
       "disbursement_method           0.004442\n",
       "debt_settlement_flag          0.004442\n",
       "dtype: float64"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Get the columns with less than 20% missing values\n",
    "cols_to_fill = df_clean.columns[df_clean.isnull().sum() / len(df_clean) * 100 < 20]\n",
    "\n",
    "# Fill the missing values with the mean of the respective columns for float columns\n",
    "for col in cols_to_fill:\n",
    "    if df_clean[col].dtype == 'float64':\n",
    "        df_clean[col].fillna(df_clean[col].mean(), inplace=True)\n",
    "\n",
    "# Check the percentage of missing values in each column \n",
    "df_clean.isnull().sum() / len(df_clean) * 100"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's investigate the columns with relatively high missing values comparing to the rest of the dataset:\n",
    "1. emp_title (~6% of the rows are missing)\n",
    "\n",
    "2. emp_length (~6% of the rows are missing)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The job title supplied by the Borrower when applying for the loan.*\n"
     ]
    }
   ],
   "source": [
    "# Exam the emp_title column\n",
    "col_info('emp_title')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Interpretation of the column:**\n",
    "\n",
    "The \"emp_title\" column in the LendingClub loan data refers to the job title that the borrower provides on their loan application. When individuals apply for a loan with LendingClub, they are typically asked to provide various pieces of personal and financial information, including their employment details.\n",
    "\n",
    "**Timing of Variable in the Loan Application Process:**\n",
    "\n",
    "Arises **before** Loan Disbursement.\n",
    "\n",
    "This information is used by the lender, LendingClub, as part of the loan application and underwriting process to assess the borrower's income stability, job status, and creditworthiness.\n",
    "\n",
    "However, because the job title is self-reported, it is considered less reliable than verified income data, and lenders often rely on other financial factors to make their lending decisions.\n",
    "\n",
    "**Decision on data handling:**\n",
    "\n",
    "Although it arises before loan disbursement, but it is unrealistic for us to fill the missing job title for the borrower without any insider information. Therefore, let's **drop** the missing rows for this column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Employment length in years. Possible values are between 0 and 10 where 0 means less than one year and 10 means ten or more years. \n"
     ]
    }
   ],
   "source": [
    "# Exam the emp_length column\n",
    "col_info('emp_length')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Interpretation of the column:**\n",
    "\n",
    "The \"emp_length\" column in the LendingClub loan data refers to the number of years the borrower has been employed at their current job or employer at the time of applying for the loan.\n",
    "\n",
    "**Timing of Variable in the Loan Application Process:**\n",
    "\n",
    "Arises **before** Loan Disbursement.\n",
    "\n",
    "When individuals apply for a loan with LendingClub, they are typically required to provide information about their employment history, including the length of time they have been working at their current job.\n",
    "\n",
    "However, the reported employment length is self-reported by the borrower and is not independently verified by the lender in most cases. It is just one of several factors considered during the loan application and underwriting process.\n",
    "\n",
    "**Decision on data handling:**\n",
    "\n",
    "Although it arises before loan disbursement, but it is unrealistic for us to fill the missing employment length for the borrower without any insider information. Therefore, let's **drop** the missing rows for this column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "id                            0.0\n",
       "loan_amnt                     0.0\n",
       "funded_amnt                   0.0\n",
       "funded_amnt_inv               0.0\n",
       "term                          0.0\n",
       "int_rate                      0.0\n",
       "installment                   0.0\n",
       "grade                         0.0\n",
       "sub_grade                     0.0\n",
       "emp_title                     0.0\n",
       "emp_length                    0.0\n",
       "home_ownership                0.0\n",
       "annual_inc                    0.0\n",
       "verification_status           0.0\n",
       "issue_d                       0.0\n",
       "loan_status                   0.0\n",
       "pymnt_plan                    0.0\n",
       "url                           0.0\n",
       "purpose                       0.0\n",
       "title                         0.0\n",
       "zip_code                      0.0\n",
       "addr_state                    0.0\n",
       "dti                           0.0\n",
       "delinq_2yrs                   0.0\n",
       "earliest_cr_line              0.0\n",
       "fico_range_low                0.0\n",
       "fico_range_high               0.0\n",
       "inq_last_6mths                0.0\n",
       "open_acc                      0.0\n",
       "pub_rec                       0.0\n",
       "revol_bal                     0.0\n",
       "revol_util                    0.0\n",
       "total_acc                     0.0\n",
       "initial_list_status           0.0\n",
       "out_prncp                     0.0\n",
       "out_prncp_inv                 0.0\n",
       "total_pymnt                   0.0\n",
       "total_pymnt_inv               0.0\n",
       "total_rec_prncp               0.0\n",
       "total_rec_int                 0.0\n",
       "total_rec_late_fee            0.0\n",
       "recoveries                    0.0\n",
       "collection_recovery_fee       0.0\n",
       "last_pymnt_d                  0.0\n",
       "last_pymnt_amnt               0.0\n",
       "last_credit_pull_d            0.0\n",
       "last_fico_range_high          0.0\n",
       "last_fico_range_low           0.0\n",
       "collections_12_mths_ex_med    0.0\n",
       "policy_code                   0.0\n",
       "application_type              0.0\n",
       "acc_now_delinq                0.0\n",
       "tot_coll_amt                  0.0\n",
       "tot_cur_bal                   0.0\n",
       "open_acc_6m                   0.0\n",
       "open_act_il                   0.0\n",
       "open_il_12m                   0.0\n",
       "open_il_24m                   0.0\n",
       "mths_since_rcnt_il            0.0\n",
       "total_bal_il                  0.0\n",
       "open_rv_12m                   0.0\n",
       "open_rv_24m                   0.0\n",
       "max_bal_bc                    0.0\n",
       "all_util                      0.0\n",
       "total_rev_hi_lim              0.0\n",
       "inq_fi                        0.0\n",
       "total_cu_tl                   0.0\n",
       "inq_last_12m                  0.0\n",
       "acc_open_past_24mths          0.0\n",
       "avg_cur_bal                   0.0\n",
       "bc_open_to_buy                0.0\n",
       "bc_util                       0.0\n",
       "chargeoff_within_12_mths      0.0\n",
       "delinq_amnt                   0.0\n",
       "mo_sin_old_il_acct            0.0\n",
       "mo_sin_old_rev_tl_op          0.0\n",
       "mo_sin_rcnt_rev_tl_op         0.0\n",
       "mo_sin_rcnt_tl                0.0\n",
       "mort_acc                      0.0\n",
       "mths_since_recent_bc          0.0\n",
       "mths_since_recent_inq         0.0\n",
       "num_accts_ever_120_pd         0.0\n",
       "num_actv_bc_tl                0.0\n",
       "num_actv_rev_tl               0.0\n",
       "num_bc_sats                   0.0\n",
       "num_bc_tl                     0.0\n",
       "num_il_tl                     0.0\n",
       "num_op_rev_tl                 0.0\n",
       "num_rev_accts                 0.0\n",
       "num_rev_tl_bal_gt_0           0.0\n",
       "num_sats                      0.0\n",
       "num_tl_120dpd_2m              0.0\n",
       "num_tl_30dpd                  0.0\n",
       "num_tl_90g_dpd_24m            0.0\n",
       "num_tl_op_past_12m            0.0\n",
       "pct_tl_nvr_dlq                0.0\n",
       "percent_bc_gt_75              0.0\n",
       "pub_rec_bankruptcies          0.0\n",
       "tax_liens                     0.0\n",
       "tot_hi_cred_lim               0.0\n",
       "total_bal_ex_mort             0.0\n",
       "total_bc_limit                0.0\n",
       "total_il_high_credit_limit    0.0\n",
       "hardship_flag                 0.0\n",
       "disbursement_method           0.0\n",
       "debt_settlement_flag          0.0\n",
       "dtype: float64"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Drop the missing values for object columns\n",
    "df_clean.dropna(inplace=True)\n",
    "\n",
    "# Check the percentage of missing values in each column\n",
    "df_clean.isnull().sum() / len(df_clean) * 100"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We see that we have handled all the missing values in the dataset. Now the dataset is cleaned and ready to use for EDA."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(21015, 106)"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Check the shape of the dataframe after handling missing values\n",
    "df_clean.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After handling missing values, there are 21015 rows and 106 features left in the dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Handling Features to Avoid Data Leakage <a class=\"anchor\" id=\"features\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Post-loan Features"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After carefulling examing the remaining columns in the dataset, these are the post-loan features that should be dropped since they arise after the loans are disbursed:\n",
    "\n",
    "1.\ttotal_pymnt\n",
    "2.\ttotal_pymnt_inv\n",
    "3.\ttotal_rec_prncp\n",
    "4.\ttotal_rec_int\n",
    "5.\ttotal_rec_late_fee\n",
    "6.\trecoveries\n",
    "7.\tcollection_recovery_fee\n",
    "8.\tlast_pymnt_d\n",
    "9.\tlast_pymnt_amnt\n",
    "10.\tlast_credit_pull_d\n",
    "11.\tlast_fico_range_high\n",
    "12.\tlast_fico_range_low\n",
    "13.\tcollections_12_mths_ex_med\n",
    "14.\tapplication_type\n",
    "15.\tacc_now_delinq\n",
    "16.\ttot_coll_amt\n",
    "17.\ttot_cur_bal\n",
    "18.\topen_acc_6m\n",
    "19.\topen_act_il\n",
    "20.\topen_il_12m\n",
    "21.\topen_il_24m\n",
    "22.\tmths_since_rcnt_il\n",
    "23.\ttotal_bal_il\n",
    "24.\topen_rv_12m\n",
    "25.\topen_rv_24m\n",
    "26.\tmax_bal_bc\n",
    "27.\tall_util\n",
    "28.\ttotal_rev_hi_lim\n",
    "29.\tinq_fi\n",
    "30.\ttotal_cu_tl\n",
    "31.\tinq_last_12m\n",
    "32.\tacc_open_past_24mths\n",
    "33.\tavg_cur_bal\n",
    "34.\tbc_open_to_buy\n",
    "35.\tbc_util\n",
    "36.\tchargeoff_within_12_mths\n",
    "37.\tdelinq_amnt\n",
    "38.\tmo_sin_old_il_acct\n",
    "39.\tmo_sin_old_rev_tl_op\n",
    "40.\tmo_sin_rcnt_rev_tl_op\n",
    "41.\tmo_sin_rcnt_tl\n",
    "42.\tmort_acc\n",
    "43.\tmths_since_recent_bc\n",
    "44.\tmths_since_recent_inq\n",
    "45.\tnum_accts_ever_120_pd\n",
    "46.\tnum_actv_bc_tl\n",
    "47.\tnum_actv_rev_tl\n",
    "48.\tnum_bc_sats\n",
    "49.\tnum_bc_tl\n",
    "50.\tnum_il_tl\n",
    "51.\tnum_op_rev_tl\n",
    "52.\tnum_rev_accts\n",
    "53.\tnum_rev_tl_bal_gt_0\n",
    "54.\tnum_sats\n",
    "55.\tnum_tl_120dpd_2m\n",
    "56.\tnum_tl_30dpd\n",
    "57.\tnum_tl_90g_dpd_24m\n",
    "58.\tnum_tl_op_past_12m\n",
    "59.\tpct_tl_nvr_dlq\n",
    "60.\tpercent_bc_gt_75\n",
    "61.\tpub_rec_bankruptcies\n",
    "62.\ttax_liens\n",
    "63.\ttot_hi_cred_lim\n",
    "64.\ttotal_bal_ex_mort\n",
    "65.\ttotal_bc_limit\n",
    "66.\ttotal_il_high_credit_limit\n",
    "67.\thardship_flag\n",
    "68.\tdisbursement_method\n",
    "69.\tdebt_settlement_flag\n",
    "70. issue_d\n",
    "\n",
    "These features are related to post-loan payment, recovery, credit history, and other metrics that are relevant after the loans have been disbursed."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here is the description for each post-loan feature:\n",
    "\n",
    "<table border=\"1\" class=\"dataframe\">\n",
    "  <thead>\n",
    "    <tr style=\"text-align: right;\">\n",
    "      <th></th>\n",
    "      <th>LoanStatNew</th>\n",
    "      <th>Description</th>\n",
    "    </tr>\n",
    "  </thead>\n",
    "  <tbody>\n",
    "    <tr>\n",
    "      <th>1</th>\n",
    "      <td>total_pymnt</td>\n",
    "      <td>Payments received to date for the total amount funded.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>2</th>\n",
    "      <td>total_pymnt_inv</td>\n",
    "      <td>Payments received to date for the portion of the total amount funded by investors.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>3</th>\n",
    "      <td>total_rec_prncp</td>\n",
    "      <td>Principal received to date.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>4</th>\n",
    "      <td>total_rec_int</td>\n",
    "      <td>Interest received to date.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>5</th>\n",
    "      <td>total_rec_late_fee</td>\n",
    "      <td>Late fees received to date.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>6</th>\n",
    "      <td>recoveries</td>\n",
    "      <td>Post charge-off gross recovery.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>7</th>\n",
    "      <td>collection_recovery_fee</td>\n",
    "      <td>Post charge-off collection fee.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>8</th>\n",
    "      <td>last_pymnt_d</td>\n",
    "      <td>Last month payment was received.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>9</th>\n",
    "      <td>last_pymnt_amnt</td>\n",
    "      <td>Last total payment amount received.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>10</th>\n",
    "      <td>last_credit_pull_d</td>\n",
    "      <td>The most recent month LC pulled credit for this loan.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>11</th>\n",
    "      <td>last_fico_range_high</td>\n",
    "      <td>The upper boundary range the borrower's last FICO pulled belongs to.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>12</th>\n",
    "      <td>last_fico_range_low</td>\n",
    "      <td>The lower boundary range the borrower's last FICO pulled belongs to.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>13</th>\n",
    "      <td>collections_12_mths_ex_med</td>\n",
    "      <td>Number of collections in 12 months excluding medical collections.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>14</th>\n",
    "      <td>issue_d</td>\n",
    "      <td>The month which the loan was funded.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>15</th>\n",
    "      <td>acc_now_delinq</td>\n",
    "      <td>The number of accounts on which the borrower is now delinquent.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>16</th>\n",
    "      <td>tot_coll_amt</td>\n",
    "      <td>Total collection amounts ever owed.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>17</th>\n",
    "      <td>tot_cur_bal</td>\n",
    "      <td>Total current balance of all accounts.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>18</th>\n",
    "      <td>open_acc_6m</td>\n",
    "      <td>Number of open trades in the last 6 months.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>19</th>\n",
    "      <td>open_act_il</td>\n",
    "      <td>Number of currently active installment trades.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>20</th>\n",
    "      <td>open_il_12m</td>\n",
    "      <td>Number of installment accounts opened in the past 12 months.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>21</th>\n",
    "      <td>open_il_24m</td>\n",
    "      <td>Number of installment accounts opened in the past 24 months.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>22</th>\n",
    "      <td>mths_since_rcnt_il</td>\n",
    "      <td>Months since most recent installment accounts opened.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>23</th>\n",
    "      <td>total_bal_il</td>\n",
    "      <td>Total current balance of all installment accounts.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>24</th>\n",
    "      <td>open_rv_12m</td>\n",
    "      <td>Number of revolving trades opened in the past 12 months.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>25</th>\n",
    "      <td>open_rv_24m</td>\n",
    "      <td>Number of revolving trades opened in the past 24 months.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>26</th>\n",
    "      <td>max_bal_bc</td>\n",
    "      <td>Maximum current balance owed on all revolving accounts.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>27</th>\n",
    "      <td>all_util</td>\n",
    "      <td>Balance to credit limit on all trades.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>28</th>\n",
    "      <td>total_rev_hi_lim</td>\n",
    "      <td>Total bank card high credit/credit limit.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>29</th>\n",
    "      <td>inq_fi</td>\n",
    "      <td>Number of personal finance inquiries.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>30</th>\n",
    "      <td>total_cu_tl</td>\n",
    "      <td>Number of finance trades.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>31</th>\n",
    "      <td>inq_last_12m</td>\n",
    "      <td>Number of credit inquiries in the past 12 months.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>32</th>\n",
    "      <td>acc_open_past_24mths</td>\n",
    "      <td>Number of trades opened in the past 24 months.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>33</th>\n",
    "      <td>avg_cur_bal</td>\n",
    "      <td>Average current balance of all accounts.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>34</th>\n",
    "      <td>bc_open_to_buy</td>\n",
    "      <td>Total open to buy on revolving bank cards.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>35</th>\n",
    "      <td>bc_util</td>\n",
    "      <td>Ratio of total current balance to high credit/credit limit for all bank card accounts.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>36</th>\n",
    "      <td>chargeoff_within_12_mths</td>\n",
    "      <td>Number of charge-offs within 12 months.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>37</th>\n",
    "      <td>delinq_amnt</td>\n",
    "      <td>The past-due amount owed for the accounts on which the borrower is now delinquent.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>38</th>\n",
    "      <td>mo_sin_old_il_acct</td>\n",
    "      <td>Months since oldest bank installment account opened.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>39</th>\n",
    "      <td>mo_sin_old_rev_tl_op</td>\n",
    "      <td>Months since oldest revolving account opened.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>40</th>\n",
    "      <td>mo_sin_rcnt_rev_tl_op</td>\n",
    "      <td>Months since most recent revolving account opened.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>41</th>\n",
    "      <td>mo_sin_rcnt_tl</td>\n",
    "      <td>Months since most recent account opened.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>42</th>\n",
    "      <td>mort_acc</td>\n",
    "      <td>Number of mortgage accounts.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>43</th>\n",
    "      <td>mths_since_recent_bc</td>\n",
    "      <td>Months since most recent bank card account opened.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>44</th>\n",
    "      <td>mths_since_recent_inq</td>\n",
    "      <td>Months since most recent inquiry.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>45</th>\n",
    "      <td>num_accts_ever_120_pd</td>\n",
    "      <td>Number of accounts ever 120 or more days past due.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>46</th>\n",
    "      <td>num_actv_bc_tl</td>\n",
    "      <td>Number of currently active bank card accounts.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>47</th>\n",
    "      <td>num_actv_rev_tl</td>\n",
    "      <td>Number of currently active revolving trades.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>48</th>\n",
    "      <td>num_bc_sats</td>\n",
    "      <td>Number of satisfactory bank card accounts.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>49</th>\n",
    "      <td>num_bc_tl</td>\n",
    "      <td>Number of bank card accounts.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>50</th>\n",
    "      <td>num_il_tl</td>\n",
    "      <td>Number of installment accounts.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>51</th>\n",
    "      <td>num_op_rev_tl</td>\n",
    "      <td>Number of open revolving accounts.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>52</th>\n",
    "      <td>num_rev_accts</td>\n",
    "      <td>Number of revolving accounts.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>53</th>\n",
    "      <td>num_rev_tl_bal_gt_0</td>\n",
    "      <td>Number of revolving trades with a balance >0.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>54</th>\n",
    "      <td>num_sats</td>\n",
    "      <td>Number of satisfactory accounts.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>55</th>\n",
    "      <td>num_tl_120dpd_2m</td>\n",
    "      <td>Number of accounts currently 120 days past due (updated in the past 2 months).</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>56</th>\n",
    "      <td>num_tl_30dpd</td>\n",
    "      <td>Number of accounts currently 30 days past due (updated in the past 2 months).</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>57</th>\n",
    "      <td>num_tl_90g_dpd_24m</td>\n",
    "      <td>Number of accounts 90 or more days past due in the last 24 months.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>58</th>\n",
    "      <td>num_tl_op_past_12m</td>\n",
    "      <td>Number of accounts opened in the past 12 months.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>59</th>\n",
    "      <td>pct_tl_nvr_dlq</td>\n",
    "      <td>Percent of trades never delinquent.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>60</th>\n",
    "      <td>percent_bc_gt_75</td>\n",
    "      <td>Percentage of all bank card accounts > 75% of the limit.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>61</th>\n",
    "      <td>pub_rec_bankruptcies</td>\n",
    "      <td>Number of public record bankruptcies.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>62</th>\n",
    "      <td>tax_liens</td>\n",
    "      <td>Number of tax liens.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>63</th>\n",
    "      <td>tot_hi_cred_lim</td>\n",
    "      <td>Total high credit/credit limit.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>64</th>\n",
    "      <td>total_bal_ex_mort</td>\n",
    "      <td>Total credit balance excluding mortgage.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>65</th>\n",
    "      <td>total_bc_limit</td>\n",
    "      <td>Total bank card high credit/credit limit.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>66</th>\n",
    "      <td>total_il_high_credit_limit</td>\n",
    "      <td>Total installment high credit/credit limit.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>67</th>\n",
    "      <td>hardship_flag</td>\n",
    "      <td>Flags whether or not the borrower is on a hardship plan.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>68</th>\n",
    "      <td>disbursement_method</td>\n",
    "      <td>The method by which the borrower receives their loan.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>69</th>\n",
    "      <td>debt_settlement_flag</td>\n",
    "      <td>Flags whether or not the borrower has settled their debt.</td>\n",
    "    </tr>\n",
    "  </tbody>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "# List of columns to drop (post-loan features)\n",
    "columns_to_drop = [\n",
    "    'total_pymnt', 'total_pymnt_inv', 'total_rec_prncp', 'total_rec_int',\n",
    "    'total_rec_late_fee', 'recoveries', 'collection_recovery_fee', 'last_pymnt_d',\n",
    "    'last_pymnt_amnt', 'last_credit_pull_d', 'last_fico_range_high', 'last_fico_range_low',\n",
    "    'collections_12_mths_ex_med', 'issue_d', 'acc_now_delinq', 'tot_coll_amt',\n",
    "    'tot_cur_bal', 'open_acc_6m', 'open_act_il', 'open_il_12m', 'open_il_24m',\n",
    "    'mths_since_rcnt_il', 'total_bal_il', 'open_rv_12m', 'open_rv_24m', 'max_bal_bc',\n",
    "    'all_util', 'total_rev_hi_lim', 'inq_fi', 'total_cu_tl', 'inq_last_12m',\n",
    "    'acc_open_past_24mths', 'avg_cur_bal', 'bc_open_to_buy', 'bc_util', 'chargeoff_within_12_mths',\n",
    "    'delinq_amnt', 'mo_sin_old_il_acct', 'mo_sin_old_rev_tl_op', 'mo_sin_rcnt_rev_tl_op',\n",
    "    'mo_sin_rcnt_tl', 'mort_acc', 'mths_since_recent_bc', 'mths_since_recent_inq',\n",
    "    'num_accts_ever_120_pd', 'num_actv_bc_tl', 'num_actv_rev_tl', 'num_bc_sats',\n",
    "    'num_bc_tl', 'num_il_tl', 'num_op_rev_tl', 'num_rev_accts', 'num_rev_tl_bal_gt_0',\n",
    "    'num_sats', 'num_tl_120dpd_2m', 'num_tl_30dpd', 'num_tl_90g_dpd_24m',\n",
    "    'num_tl_op_past_12m', 'pct_tl_nvr_dlq', 'percent_bc_gt_75', 'pub_rec_bankruptcies',\n",
    "    'tax_liens', 'tot_hi_cred_lim', 'total_bal_ex_mort', 'total_bc_limit',\n",
    "    'total_il_high_credit_limit', 'hardship_flag', 'disbursement_method', 'debt_settlement_flag'\n",
    "]\n",
    "\n",
    "# Drop the columns from the DataFrame\n",
    "df_clean.drop(columns=columns_to_drop, inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(21015, 37)"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Check the shape of the dataframe after dropping post-loan features\n",
    "df_clean.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to avoid data leakage, we dropped the post-loan features, resulting in 21015 rows and 37 columns left."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['id',\n",
       " 'loan_amnt',\n",
       " 'funded_amnt',\n",
       " 'funded_amnt_inv',\n",
       " 'term',\n",
       " 'int_rate',\n",
       " 'installment',\n",
       " 'grade',\n",
       " 'sub_grade',\n",
       " 'emp_title',\n",
       " 'emp_length',\n",
       " 'home_ownership',\n",
       " 'annual_inc',\n",
       " 'verification_status',\n",
       " 'loan_status',\n",
       " 'pymnt_plan',\n",
       " 'url',\n",
       " 'purpose',\n",
       " 'title',\n",
       " 'zip_code',\n",
       " 'addr_state',\n",
       " 'dti',\n",
       " 'delinq_2yrs',\n",
       " 'earliest_cr_line',\n",
       " 'fico_range_low',\n",
       " 'fico_range_high',\n",
       " 'inq_last_6mths',\n",
       " 'open_acc',\n",
       " 'pub_rec',\n",
       " 'revol_bal',\n",
       " 'revol_util',\n",
       " 'total_acc',\n",
       " 'initial_list_status',\n",
       " 'out_prncp',\n",
       " 'out_prncp_inv',\n",
       " 'policy_code',\n",
       " 'application_type']"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# List the remaining columns\n",
    "df_clean.columns.tolist()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Irrelevant/Redundant Features"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are some features that can be dropped since they do not contribute to the analysis or are considered as redundant.\n",
    "\n",
    "To determine which features do not contribute to the analysis or are redundant for predicting the loan_status, we need to consider the relevance and potential impact of each feature on the target variable. Based on the information provided, the following features seem less likely to directly contribute to the analysis:\n",
    "\n",
    "1. **id:** This variable appears to be an identifier for the loan and does not provide any meaningful information for predicting loan_status.\n",
    "\n",
    "2. **url:** This variable is likely a link to the loan's webpage and is not relevant to predicting loan_status.\n",
    "\n",
    "3. **pymnt_plan:** Since most of the data points may have the same value (e.g., \"n\" for no), this variable may not have enough variability to be useful in the analysis.\n",
    "\n",
    "4. **title:** The loan title might contain similar information as the 'purpose' variable and may not add much additional value for prediction.\n",
    "\n",
    "5. **zip_code:** The geographic location of borrowers might not have a strong correlation with loan_status.\n",
    "\n",
    "6. **out_prncp** and **out_prncp_inv:** These variables represent outstanding principal amount and are likely to have a strong correlation with the loan_status, making them less useful for prediction.\n",
    "\n",
    "7. **policy_code:** If most of the data points have the same value (e.g., \"1\"), then this variable may not provide any meaningful information for the analysis."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here is the description for the above features:\n",
    "\n",
    "<table border=\"1\" class=\"dataframe\">\n",
    "  <thead>\n",
    "    <tr style=\"text-align: right;\">\n",
    "      <th></th>\n",
    "      <th>LoanStatNew</th>\n",
    "      <th>Description</th>\n",
    "    </tr>\n",
    "  </thead>\n",
    "  <tbody>\n",
    "    <tr>\n",
    "      <th>1</th>\n",
    "      <td>id</td>\n",
    "      <td>A unique LC assigned ID for the loan listing.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>2</th>\n",
    "      <td>url</td>\n",
    "      <td>URL for the LC page with listing data.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>3</th>\n",
    "      <td>pymnt_plan</td>\n",
    "      <td>Indicates if a payment plan has been put in place for the loan.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>4</th>\n",
    "      <td>title</td>\n",
    "      <td>The loan title provided by the borrower.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>5</th>\n",
    "      <td>zip_code</td>\n",
    "      <td>The first 3 numbers of the zip code provided by the borrower in the loan application.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>6</th>\n",
    "      <td>out_prncp</td>\n",
    "      <td>Remaining outstanding principal amount for total amount funded.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>7</th>\n",
    "      <td>out_prncp_inv</td>\n",
    "      <td>Remaining outstanding principal amount for portion of total amount funded by investors.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>8</th>\n",
    "      <td>policy_code</td>\n",
    "      <td>Publicly available policy_code=1, new products not publicly available policy_code=2.</td>\n",
    "    </tr>\n",
    "  </tbody>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's further investigate some of the features mentioned above:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Counts for 'pymnt_plan':\n",
      "n    22508\n",
      "y        3\n",
      "Name: pymnt_plan, dtype: int64\n",
      "\n",
      "Counts for 'title':\n",
      "Debt consolidation         12676\n",
      "Credit card refinancing     5621\n",
      "Home improvement            1269\n",
      "Other                       1242\n",
      "Major purchase               478\n",
      "Medical expenses             255\n",
      "Business                     236\n",
      "Car financing                205\n",
      "Vacation                     146\n",
      "Moving and relocation        143\n",
      "Home buying                   93\n",
      "Green loan                    14\n",
      "Name: title, dtype: int64\n",
      "\n",
      "Counts for 'purpose':\n",
      "debt_consolidation    12742\n",
      "credit_card            5652\n",
      "home_improvement       1284\n",
      "other                  1253\n",
      "major_purchase          480\n",
      "medical                 257\n",
      "small_business          238\n",
      "car                     208\n",
      "vacation                145\n",
      "moving                  144\n",
      "house                    93\n",
      "renewable_energy         14\n",
      "debt_cons                 1\n",
      "Name: purpose, dtype: int64\n",
      "\n",
      "Counts for 'policy_code':\n",
      "1.0    22510\n",
      "Name: policy_code, dtype: int64\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Check unique values in each column\n",
    "variables_list = ['pymnt_plan', 'title', 'purpose', 'policy_code']\n",
    "\n",
    "# Initialize a dictionary to store the counts for each variable\n",
    "variable_counts = {}\n",
    "\n",
    "# Loop through each variable in the variables_list\n",
    "for var in variables_list:\n",
    "    # Use value_counts() to count the occurrences of each unique value and store it in the dictionary\n",
    "    variable_counts[var] = df[var].value_counts()\n",
    "\n",
    "# Print the counts for each variable\n",
    "for var, counts in variable_counts.items():\n",
    "    print(f\"Counts for '{var}':\")\n",
    "    print(counts)\n",
    "    print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we have confirmed with further information from the dataset that the following features are less likely to directly contribute to the analysis, so we are going to drop them:\n",
    "\n",
    "1. id\n",
    "2. url\n",
    "3. pymnt_plan\n",
    "4. title\n",
    "5. zip_code\n",
    "6. out_prncp & out_prncp_inv\n",
    "7. policy_code"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "# List of columns to drop\n",
    "cols_to_drop = ['id', 'url', 'pymnt_plan', 'title', 'zip_code', 'out_prncp', 'out_prncp_inv', 'policy_code']\n",
    "\n",
    "# Use the drop() method to remove the specified columns\n",
    "df_clean = df_clean.drop(columns=cols_to_drop)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(21015, 29)"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Check the shape of the dataframe after dropping post-loan features\n",
    "df_clean.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After dropping the redundant/irrelevant columns, we needed up with 21015 rows and 29 columns."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['loan_amnt',\n",
       " 'funded_amnt',\n",
       " 'funded_amnt_inv',\n",
       " 'term',\n",
       " 'int_rate',\n",
       " 'installment',\n",
       " 'grade',\n",
       " 'sub_grade',\n",
       " 'emp_title',\n",
       " 'emp_length',\n",
       " 'home_ownership',\n",
       " 'annual_inc',\n",
       " 'verification_status',\n",
       " 'loan_status',\n",
       " 'purpose',\n",
       " 'addr_state',\n",
       " 'dti',\n",
       " 'delinq_2yrs',\n",
       " 'earliest_cr_line',\n",
       " 'fico_range_low',\n",
       " 'fico_range_high',\n",
       " 'inq_last_6mths',\n",
       " 'open_acc',\n",
       " 'pub_rec',\n",
       " 'revol_bal',\n",
       " 'revol_util',\n",
       " 'total_acc',\n",
       " 'initial_list_status',\n",
       " 'application_type']"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# List the remaining columns\n",
    "df_clean.columns.tolist()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>loan_amnt</th>\n",
       "      <th>funded_amnt</th>\n",
       "      <th>funded_amnt_inv</th>\n",
       "      <th>term</th>\n",
       "      <th>int_rate</th>\n",
       "      <th>installment</th>\n",
       "      <th>grade</th>\n",
       "      <th>sub_grade</th>\n",
       "      <th>emp_title</th>\n",
       "      <th>emp_length</th>\n",
       "      <th>home_ownership</th>\n",
       "      <th>annual_inc</th>\n",
       "      <th>verification_status</th>\n",
       "      <th>loan_status</th>\n",
       "      <th>purpose</th>\n",
       "      <th>addr_state</th>\n",
       "      <th>dti</th>\n",
       "      <th>delinq_2yrs</th>\n",
       "      <th>earliest_cr_line</th>\n",
       "      <th>fico_range_low</th>\n",
       "      <th>fico_range_high</th>\n",
       "      <th>inq_last_6mths</th>\n",
       "      <th>open_acc</th>\n",
       "      <th>pub_rec</th>\n",
       "      <th>revol_bal</th>\n",
       "      <th>revol_util</th>\n",
       "      <th>total_acc</th>\n",
       "      <th>initial_list_status</th>\n",
       "      <th>application_type</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>3600.0</td>\n",
       "      <td>3600.0</td>\n",
       "      <td>3600.0</td>\n",
       "      <td>36 months</td>\n",
       "      <td>13.99</td>\n",
       "      <td>123.03</td>\n",
       "      <td>C</td>\n",
       "      <td>C4</td>\n",
       "      <td>leadman</td>\n",
       "      <td>10+ years</td>\n",
       "      <td>MORTGAGE</td>\n",
       "      <td>55000.0</td>\n",
       "      <td>Not Verified</td>\n",
       "      <td>Fully Paid</td>\n",
       "      <td>debt_consolidation</td>\n",
       "      <td>PA</td>\n",
       "      <td>5.91</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Aug-2003</td>\n",
       "      <td>675.0</td>\n",
       "      <td>679.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2765.0</td>\n",
       "      <td>29.7</td>\n",
       "      <td>13.0</td>\n",
       "      <td>w</td>\n",
       "      <td>Individual</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>24700.0</td>\n",
       "      <td>24700.0</td>\n",
       "      <td>24700.0</td>\n",
       "      <td>36 months</td>\n",
       "      <td>11.99</td>\n",
       "      <td>820.28</td>\n",
       "      <td>C</td>\n",
       "      <td>C1</td>\n",
       "      <td>Engineer</td>\n",
       "      <td>10+ years</td>\n",
       "      <td>MORTGAGE</td>\n",
       "      <td>65000.0</td>\n",
       "      <td>Not Verified</td>\n",
       "      <td>Fully Paid</td>\n",
       "      <td>small_business</td>\n",
       "      <td>SD</td>\n",
       "      <td>16.06</td>\n",
       "      <td>1.0</td>\n",
       "      <td>Dec-1999</td>\n",
       "      <td>715.0</td>\n",
       "      <td>719.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>22.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>21470.0</td>\n",
       "      <td>19.2</td>\n",
       "      <td>38.0</td>\n",
       "      <td>w</td>\n",
       "      <td>Individual</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>35000.0</td>\n",
       "      <td>35000.0</td>\n",
       "      <td>35000.0</td>\n",
       "      <td>60 months</td>\n",
       "      <td>14.85</td>\n",
       "      <td>829.90</td>\n",
       "      <td>C</td>\n",
       "      <td>C5</td>\n",
       "      <td>Information Systems Officer</td>\n",
       "      <td>10+ years</td>\n",
       "      <td>MORTGAGE</td>\n",
       "      <td>110000.0</td>\n",
       "      <td>Source Verified</td>\n",
       "      <td>Current</td>\n",
       "      <td>debt_consolidation</td>\n",
       "      <td>NJ</td>\n",
       "      <td>17.06</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Sep-2008</td>\n",
       "      <td>785.0</td>\n",
       "      <td>789.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>13.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>7802.0</td>\n",
       "      <td>11.6</td>\n",
       "      <td>17.0</td>\n",
       "      <td>w</td>\n",
       "      <td>Individual</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>10400.0</td>\n",
       "      <td>10400.0</td>\n",
       "      <td>10400.0</td>\n",
       "      <td>60 months</td>\n",
       "      <td>22.45</td>\n",
       "      <td>289.91</td>\n",
       "      <td>F</td>\n",
       "      <td>F1</td>\n",
       "      <td>Contract Specialist</td>\n",
       "      <td>3 years</td>\n",
       "      <td>MORTGAGE</td>\n",
       "      <td>104433.0</td>\n",
       "      <td>Source Verified</td>\n",
       "      <td>Fully Paid</td>\n",
       "      <td>major_purchase</td>\n",
       "      <td>PA</td>\n",
       "      <td>25.37</td>\n",
       "      <td>1.0</td>\n",
       "      <td>Jun-1998</td>\n",
       "      <td>695.0</td>\n",
       "      <td>699.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>12.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>21929.0</td>\n",
       "      <td>64.5</td>\n",
       "      <td>35.0</td>\n",
       "      <td>w</td>\n",
       "      <td>Individual</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>11950.0</td>\n",
       "      <td>11950.0</td>\n",
       "      <td>11950.0</td>\n",
       "      <td>36 months</td>\n",
       "      <td>13.44</td>\n",
       "      <td>405.18</td>\n",
       "      <td>C</td>\n",
       "      <td>C3</td>\n",
       "      <td>Veterinary Tecnician</td>\n",
       "      <td>4 years</td>\n",
       "      <td>RENT</td>\n",
       "      <td>34000.0</td>\n",
       "      <td>Source Verified</td>\n",
       "      <td>Fully Paid</td>\n",
       "      <td>debt_consolidation</td>\n",
       "      <td>GA</td>\n",
       "      <td>10.20</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Oct-1987</td>\n",
       "      <td>690.0</td>\n",
       "      <td>694.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>8822.0</td>\n",
       "      <td>68.4</td>\n",
       "      <td>6.0</td>\n",
       "      <td>w</td>\n",
       "      <td>Individual</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   loan_amnt  funded_amnt  funded_amnt_inv        term  int_rate  installment  \\\n",
       "0     3600.0       3600.0           3600.0   36 months     13.99       123.03   \n",
       "1    24700.0      24700.0          24700.0   36 months     11.99       820.28   \n",
       "3    35000.0      35000.0          35000.0   60 months     14.85       829.90   \n",
       "4    10400.0      10400.0          10400.0   60 months     22.45       289.91   \n",
       "5    11950.0      11950.0          11950.0   36 months     13.44       405.18   \n",
       "\n",
       "  grade sub_grade                    emp_title emp_length home_ownership  \\\n",
       "0     C        C4                      leadman  10+ years       MORTGAGE   \n",
       "1     C        C1                     Engineer  10+ years       MORTGAGE   \n",
       "3     C        C5  Information Systems Officer  10+ years       MORTGAGE   \n",
       "4     F        F1          Contract Specialist    3 years       MORTGAGE   \n",
       "5     C        C3         Veterinary Tecnician    4 years           RENT   \n",
       "\n",
       "   annual_inc verification_status loan_status             purpose addr_state  \\\n",
       "0     55000.0        Not Verified  Fully Paid  debt_consolidation         PA   \n",
       "1     65000.0        Not Verified  Fully Paid      small_business         SD   \n",
       "3    110000.0     Source Verified     Current  debt_consolidation         NJ   \n",
       "4    104433.0     Source Verified  Fully Paid      major_purchase         PA   \n",
       "5     34000.0     Source Verified  Fully Paid  debt_consolidation         GA   \n",
       "\n",
       "     dti  delinq_2yrs earliest_cr_line  fico_range_low  fico_range_high  \\\n",
       "0   5.91          0.0         Aug-2003           675.0            679.0   \n",
       "1  16.06          1.0         Dec-1999           715.0            719.0   \n",
       "3  17.06          0.0         Sep-2008           785.0            789.0   \n",
       "4  25.37          1.0         Jun-1998           695.0            699.0   \n",
       "5  10.20          0.0         Oct-1987           690.0            694.0   \n",
       "\n",
       "   inq_last_6mths  open_acc  pub_rec  revol_bal  revol_util  total_acc  \\\n",
       "0             1.0       7.0      0.0     2765.0        29.7       13.0   \n",
       "1             4.0      22.0      0.0    21470.0        19.2       38.0   \n",
       "3             0.0      13.0      0.0     7802.0        11.6       17.0   \n",
       "4             3.0      12.0      0.0    21929.0        64.5       35.0   \n",
       "5             0.0       5.0      0.0     8822.0        68.4        6.0   \n",
       "\n",
       "  initial_list_status application_type  \n",
       "0                   w       Individual  \n",
       "1                   w       Individual  \n",
       "3                   w       Individual  \n",
       "4                   w       Individual  \n",
       "5                   w       Individual  "
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Show a sample of the data\n",
    "df_clean.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are done with data cleaning, now let's save this cleaned dataset for later use:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Save the cleaned data to a parquet file\n",
    "df_clean.to_parquet('dataset/LoanData_after_DataCleaning.parquet')"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.17"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}