intermediate_notebooks/examples/cuDatashader.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "J6PB5HeksEl0"
   },
   "source": [
    "# cuDatashader vs Datashader comparison for simple points plotting\n",
    "This notebook lets you expereince cuDataShader, which is a GPU acclerated version of DataShader.  Other cuDataShader notebooks can be found in the cuDataShader [repository]().  We hope to merge cuDataShader into the DataShader project itself.  Hooray Open Source cross collaboration!  \n",
    "\n",
    "This notebook installs cuDataShader, which at the time of writing, doesn't have a conda package.  This may change in the future and the notebook will be updated.  We also have this notebook in [Colab](https://colab.research.google.com/drive/1bFIBg54zS9RmU58VwjJMAaqJ1xP27BXj) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "W-xKixoWsbwe"
   },
   "outputs": [],
   "source": [
    "!nvidia-smi"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "J4QZr986RVbh"
   },
   "source": [
    "## Install cuDataShader and other dependancies"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "9Clw343Lsu-k"
   },
   "outputs": [],
   "source": [
    "!git clone https://github.com/rapidsai/cuDataShader.git"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "cu99Lh_rs4MS"
   },
   "outputs": [],
   "source": [
    "!ls\n",
    "%cd cuDataShader\n",
    "!pip install -e ."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "g6CfmjqoMfW2"
   },
   "outputs": [],
   "source": [
    "!pip install pyproj\n",
    "!pip install datashader\n",
    "## Ignore the restart warning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "zndllU9ORQkd"
   },
   "source": [
    "## Let's get the Taxi Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Qtsw9vv6QPZt"
   },
   "outputs": [],
   "source": [
    "## You can change the data set used by changing the year and the month: \"yellow_tripdata_<YYYY>-<MM>.csv\".  See working example below\n",
    "!wget -O nyc_taxi.csv https://s3.amazonaws.com/nyc-tlc/trip+data/yellow_tripdata_2015-01.csv "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "3GBJN3IrRb17"
   },
   "source": [
    "## Let's begin!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "ZyrQznuFsEl2"
   },
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "\n",
    "import pandas as pd\n",
    "import cudf\n",
    "\n",
    "import os\n",
    "os.environ[\"CUDA_VISIBLE_DEVICES\"] = \"0\" # Choose GPU\n",
    "\n",
    "pdf = pd.read_csv('nyc_taxi.csv', usecols=['dropoff_latitude','dropoff_longitude', 'passenger_count']) # Load into Pandas DF and select rows\n",
    "pdf['passenger_count'] = pdf['passenger_count'].astype(np.float64) # Convert aggregation column\n",
    "pdf.tail()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "La_6UYDCsEl6"
   },
   "source": [
    "Then transform datapoints from GPS coordinates (longitudes, latitudes) to actually plottable 2D points :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "ACg8oiBIsEl7"
   },
   "outputs": [],
   "source": [
    "from pyproj import Proj, transform\n",
    "\n",
    "inProj = Proj(init='epsg:4326') # Latitude and longitudes\n",
    "outProj = Proj(init='epsg:3857') # 2D projected points\n",
    "\n",
    "x, y = transform(inProj, outProj, pdf['dropoff_longitude'].values, pdf['dropoff_latitude'].values) # Apply transformation\n",
    "\n",
    "pdf['dropoff_x'] = x\n",
    "pdf['dropoff_y'] = y\n",
    "\n",
    "pdf.drop(['dropoff_latitude', 'dropoff_longitude'], axis=1, inplace=True)\n",
    "pdf = pdf[~pdf.isin([np.nan, np.inf, -np.inf]).any(1)]\n",
    "\n",
    "pdf = pdf[(pdf.dropoff_x > -8239910.23) & (pdf.dropoff_x < -8229529.24) & (pdf.dropoff_y > 4968481.34) & (pdf.dropoff_y < 4983152.92)] # Filter over Manhattan\n",
    "#pdf = pdf.sample(frac=0.1) # Sample a fraction of the dataset\n",
    "\n",
    "pdf.tail()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "QXG8EYE-sEl9"
   },
   "outputs": [],
   "source": [
    "pdf.to_pickle('pdf.pkl') # Backup the dataset (prevent users from waiting above computations)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "gUCT9zzIsEl_"
   },
   "source": [
    "If you launched above computations once, you can start from there for now on"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "zVrv3_1AsEmA"
   },
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "\n",
    "import pandas as pd\n",
    "import cudf\n",
    "\n",
    "import time\n",
    "\n",
    "import os\n",
    "os.environ[\"CUDA_VISIBLE_DEVICES\"] = \"0\" # Choose GPU\n",
    "\n",
    "pdf = pd.read_pickle('pdf.pkl') # Load backup\n",
    "gdf = cudf.from_pandas(pdf) # Convert to cuDF DataFrame"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "ODl13lx9sEmC"
   },
   "outputs": [],
   "source": [
    "print(\"Dataframe has {:,} rows\".format(pdf.shape[0]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "3uiURt3_sEmE"
   },
   "source": [
    "## Render image with regular Datashader"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "_g7efGfBsEmF"
   },
   "outputs": [],
   "source": [
    "import datashader as ds\n",
    "from datashader import transfer_functions as tf\n",
    "from datashader.colors import Hot\n",
    "\n",
    "t0 = time.time() # Save start time\n",
    "cvs = ds.Canvas(plot_width=750, plot_height=625, x_range=(-8239910.23,-8229529.24), y_range=(4968481.34,4983152.92)) # Create canvas\n",
    "agg = cvs.points(pdf, 'dropoff_x', 'dropoff_y', ds.count('passenger_count')) # Perform aggregation\n",
    "img = tf.shade(agg, cmap=Hot, how='eq_hist') # Produce image from aggregation\n",
    "ds_time = time.time()-t0 # Compute elapsed time\n",
    "print(\"{} ms\".format(round(ds_time * 1000))) # Display elapsed time\n",
    "\n",
    "img # Display image"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "JnYijfVBsEmH"
   },
   "source": [
    "## Render image with GPU accelerated cuDatashader (exact same usage/syntax)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "LoCHYAhgsEmH"
   },
   "outputs": [],
   "source": [
    "# Overloading Datashader functions\n",
    "import cudatashader as ds\n",
    "from cudatashader import transfer_functions as tf\n",
    "from cudatashader.colors import Hot\n",
    "\n",
    "t0 = time.time() # Save start time\n",
    "cvs = ds.Canvas(plot_width=750, plot_height=625, x_range=(-8239910.23,-8229529.24), y_range=(4968481.34,4983152.92)) # Create canvas\n",
    "agg = cvs.points(gdf, 'dropoff_x', 'dropoff_y', ds.count('passenger_count')) # Perform aggregation\n",
    "img2 = tf.shade(agg, cmap=Hot, how='eq_hist') # Produce image from aggregation\n",
    "cuviz_time = time.time()-t0 # Compute elapsed time\n",
    "print(\"{} ms\".format(round(cuviz_time * 1000))) # Display elapsed time\n",
    "\n",
    "img2 # Display image"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "5FR-PLkksEmK"
   },
   "outputs": [],
   "source": [
    "print('GPU speedup: {:.2f}'.format(ds_time/cuviz_time))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "uFPHf9QCSH0a"
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [],
   "name": "cuDatashader.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}