IMDB_In_Keras-testing_epochs_layers_bSize.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Analyzing IMDB Data in Keras"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-04-30T22:35:27.896268Z",
     "start_time": "2018-04-30T22:35:25.180063Z"
    }
   },
   "outputs": [],
   "source": [
    "# Imports\n",
    "import numpy as np\n",
    "import keras\n",
    "from keras.datasets import imdb\n",
    "from keras.models import Sequential\n",
    "from keras.layers import Dense, Dropout, Activation\n",
    "from keras.preprocessing.text import Tokenizer\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "\n",
    "np.random.seed(42)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Loading the data\n",
    "This dataset comes preloaded with Keras, so one simple command will get us training and testing data. There is a parameter for how many words we want to look at. We've set it at 1000, but feel free to experiment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-04-30T22:35:33.135337Z",
     "start_time": "2018-04-30T22:35:29.722282Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(25000,)\n",
      "(25000,)\n"
     ]
    }
   ],
   "source": [
    "# Loading the data (it's preloaded in Keras)\n",
    "(x_train, y_train), (x_test, y_test) = imdb.load_data(num_words=1000)\n",
    "\n",
    "print(x_train.shape)\n",
    "print(x_test.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Examining the data\n",
    "Notice that the data has been already pre-processed, where all the words have numbers, and the reviews come in as a vector with the words that the review contains. For example, if the word 'the' is the first one in our dictionary, and a review contains the word 'the', then there is a 1 in the corresponding vector.\n",
    "\n",
    "The output comes as a vector of 1's and 0's, where 1 is a positive sentiment for the review, and 0 is negative."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-04-30T22:35:33.167422Z",
     "start_time": "2018-04-30T22:35:33.162409Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1, 14, 22, 16, 43, 530, 973, 2, 2, 65, 458, 2, 66, 2, 4, 173, 36, 256, 5, 25, 100, 43, 838, 112, 50, 670, 2, 9, 35, 480, 284, 5, 150, 4, 172, 112, 167, 2, 336, 385, 39, 4, 172, 2, 2, 17, 546, 38, 13, 447, 4, 192, 50, 16, 6, 147, 2, 19, 14, 22, 4, 2, 2, 469, 4, 22, 71, 87, 12, 16, 43, 530, 38, 76, 15, 13, 2, 4, 22, 17, 515, 17, 12, 16, 626, 18, 2, 5, 62, 386, 12, 8, 316, 8, 106, 5, 4, 2, 2, 16, 480, 66, 2, 33, 4, 130, 12, 16, 38, 619, 5, 25, 124, 51, 36, 135, 48, 25, 2, 33, 6, 22, 12, 215, 28, 77, 52, 5, 14, 407, 16, 82, 2, 8, 4, 107, 117, 2, 15, 256, 4, 2, 7, 2, 5, 723, 36, 71, 43, 530, 476, 26, 400, 317, 46, 7, 4, 2, 2, 13, 104, 88, 4, 381, 15, 297, 98, 32, 2, 56, 26, 141, 6, 194, 2, 18, 4, 226, 22, 21, 134, 476, 26, 480, 5, 144, 30, 2, 18, 51, 36, 28, 224, 92, 25, 104, 4, 226, 65, 16, 38, 2, 88, 12, 16, 283, 5, 16, 2, 113, 103, 32, 15, 16, 2, 19, 178, 32]\n",
      "1\n"
     ]
    }
   ],
   "source": [
    "print(x_train[0])\n",
    "print(y_train[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. One-hot encoding the output\n",
    "Here, we'll turn the input vectors into (0,1)-vectors. For example, if the pre-processed vector contains the number 14, then in the processed vector, the 14th entry will be 1."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-04-30T22:35:36.196526Z",
     "start_time": "2018-04-30T22:35:33.200510Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(25000, 1000)\n"
     ]
    }
   ],
   "source": [
    "# One-hot encoding the output into vector mode, each of length 1000\n",
    "tokenizer = Tokenizer(num_words=1000)\n",
    "x_train = tokenizer.sequences_to_matrix(x_train, mode='binary')\n",
    "x_test = tokenizer.sequences_to_matrix(x_test, mode='binary')\n",
    "print(x_train.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And we'll also one-hot encode the output."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-04-30T22:35:36.234627Z",
     "start_time": "2018-04-30T22:35:36.228612Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(25000, 2)\n",
      "(25000, 2)\n",
      "[[0. 1. 1. ... 0. 0. 0.]\n",
      " [0. 1. 1. ... 0. 0. 0.]\n",
      " [0. 1. 1. ... 0. 0. 0.]\n",
      " ...\n",
      " [0. 1. 1. ... 0. 0. 0.]\n",
      " [0. 1. 1. ... 0. 0. 0.]\n",
      " [0. 1. 1. ... 0. 0. 0.]] [[0. 1.]\n",
      " [1. 0.]\n",
      " [1. 0.]\n",
      " ...\n",
      " [1. 0.]\n",
      " [0. 1.]\n",
      " [1. 0.]]\n"
     ]
    }
   ],
   "source": [
    "# One-hot encoding the output\n",
    "num_classes = 2\n",
    "y_train = keras.utils.to_categorical(y_train, num_classes)\n",
    "y_test = keras.utils.to_categorical(y_test, num_classes)\n",
    "print(y_train.shape)\n",
    "print(y_test.shape)\n",
    "print(x_train,y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Building the  model architecture\n",
    "Build a model here using sequential. Feel free to experiment with different layers and sizes! Also, experiment adding dropout to reduce overfitting."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-04-30T17:16:02.812001Z",
     "start_time": "2018-04-30T17:16:02.809024Z"
    }
   },
   "outputs": [],
   "source": [
    "tests = {}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-04-30T17:16:03.554572Z",
     "start_time": "2018-04-30T17:16:03.410852Z"
    }
   },
   "outputs": [],
   "source": [
    "# TODO: 3 layers, 2 hidden, baselayer @16\n",
    "model16 = Sequential()\n",
    "model16.add(Dense(16, activation='relu',  input_shape=(1000,)))\n",
    "model16.add(Dropout(.1))\n",
    "model16.add(Dense(8, activation='relu'))\n",
    "model16.add(Dropout(.1))\n",
    "model16.add(Dense(4, activation='relu'))\n",
    "model16.add(Dropout(.1))\n",
    "model16.add(Dense(2, activation='softmax'))\n",
    "\n",
    "# TODO: Compile the model using a loss function and an optimizer.\n",
    "model16.compile(loss = 'categorical_crossentropy', optimizer='nadam', metrics=['accuracy'])\n",
    "model16.summary()\n",
    "tests.update({'model16': model16})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-04-30T17:16:04.415836Z",
     "start_time": "2018-04-30T17:16:04.272866Z"
    }
   },
   "outputs": [],
   "source": [
    "# TODO: 3 layers, 2 hidden, baselayer @128\n",
    "model128 = Sequential()\n",
    "model128.add(Dense(128, activation='relu',  input_shape=(1000,)))\n",
    "model128.add(Dropout(.1))\n",
    "model128.add(Dense(32, activation='relu'))\n",
    "model128.add(Dropout(.1))\n",
    "model128.add(Dense(8, activation='relu'))\n",
    "model128.add(Dropout(.1))\n",
    "model128.add(Dense(2, activation='softmax'))\n",
    "\n",
    "# TODO: Compile the model using a loss function and an optimizer.\n",
    "model128.compile(loss = 'categorical_crossentropy', optimizer='nadam', metrics=['accuracy'])\n",
    "model128.summary()\n",
    "tests.update({'model128': model128})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-04-30T17:16:05.164689Z",
     "start_time": "2018-04-30T17:16:05.050293Z"
    }
   },
   "outputs": [],
   "source": [
    "# TODO: 2 layers, 1 hidden, baselayer @128\n",
    "model128_b = Sequential()\n",
    "model128_b.add(Dense(128, activation='relu',  input_shape=(1000,)))\n",
    "model128_b.add(Dropout(.1))\n",
    "model128_b.add(Dense(16, activation='relu'))\n",
    "model128_b.add(Dropout(.1))\n",
    "model128_b.add(Dense(2, activation='softmax'))\n",
    "\n",
    "# TODO: Compile the model using a loss function and an optimizer.\n",
    "model128_b.compile(loss = 'categorical_crossentropy', optimizer='nadam', metrics=['accuracy'])\n",
    "model128_b.summary()\n",
    "tests.update({'model128_b': model128_b})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-04-30T17:16:05.842720Z",
     "start_time": "2018-04-30T17:16:05.731712Z"
    }
   },
   "outputs": [],
   "source": [
    "# TODO: 2 layers, 1 hidden, baselayer @16\n",
    "model16_b = Sequential()\n",
    "model16_b.add(Dense(16, activation='relu',  input_shape=(1000,)))\n",
    "model16_b.add(Dropout(.1))\n",
    "model16_b.add(Dense(4, activation='relu'))\n",
    "model16_b.add(Dropout(.1))\n",
    "model16_b.add(Dense(2, activation='softmax'))\n",
    "\n",
    "# TODO: Compile the model using a loss function and an optimizer.\n",
    "model16_b.compile(loss = 'categorical_crossentropy', optimizer='nadam', metrics=['accuracy'])\n",
    "model16_b.summary()\n",
    "tests.update({'model16_b': model16_b})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-04-30T17:16:06.497534Z",
     "start_time": "2018-04-30T17:16:06.494533Z"
    }
   },
   "outputs": [],
   "source": [
    "epochs = (1,16,32,64,128)\n",
    "batchsizes = (16,32,64,128)\n",
    "runstr = [(z[0], x,y,z[1])for x in epochs for y in batchsizes for z in tests.items()]\n",
    "print(runstr)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Training the model\n",
    "Run the model here. Experiment with different batch_size, and number of epochs!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "start_time": "2018-04-30T17:16:09.461Z"
    }
   },
   "outputs": [],
   "source": [
    "# TODO: Run the model. Feel free to experiment with different batch sizes and number of epochs.\n",
    "results = []\n",
    "import tensorflow as tf\n",
    "with tf.device('/device:GPU:0'):\n",
    "    for trials in runstr:\n",
    "            trials[3].fit(x_train, y_train, epochs=trials[1], batch_size=trials[2], verbose=1);\n",
    "            testscore = trials[3].evaluate(x_test, y_test, verbose=0)\n",
    "            trainscore = trials[3].evaluate(x_train, y_train, verbose=0)\n",
    "            runtitle = str(trials[0])+'_'+str(trials[1])+'_'+str(trials[2])\n",
    "            results.append((runtitle,testscore,trainscore))\n",
    "            #saving for use.\n",
    "            trials[3].save(runtitle)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6. Evaluating the model\n",
    "This will give you the accuracy of the model, as evaluated on the testing set. Can you get something over 85%?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-04-30T23:49:29.162880Z",
     "start_time": "2018-04-30T22:59:27.890184Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "25000/25000 [==============================] - 4s 178us/step\n",
      "25000/25000 [==============================] - 3s 120us/step\n",
      "25000/25000 [==============================] - 5s 192us/step\n",
      "25000/25000 [==============================] - 3s 129us/step\n",
      "25000/25000 [==============================] - 5s 189us/step\n",
      "25000/25000 [==============================] - 3s 112us/step\n",
      "25000/25000 [==============================] - 5s 184us/step\n",
      "25000/25000 [==============================] - 3s 121us/step\n",
      "25000/25000 [==============================] - 5s 185us/step\n",
      "25000/25000 [==============================] - 3s 119us/step\n",
      "25000/25000 [==============================] - 5s 191us/step\n",
      "25000/25000 [==============================] - 3s 109us/step\n",
      "25000/25000 [==============================] - 5s 212us/step\n",
      "25000/25000 [==============================] - 3s 105us/step\n",
      "25000/25000 [==============================] - 5s 188us/step\n",
      "25000/25000 [==============================] - 3s 118us/step\n",
      "25000/25000 [==============================] - 5s 191us/step\n",
      "25000/25000 [==============================] - 4s 147us/step\n",
      "25000/25000 [==============================] - 5s 205us/step\n",
      "25000/25000 [==============================] - 4s 145us/step\n",
      "25000/25000 [==============================] - 5s 207us/step\n",
      "25000/25000 [==============================] - 4s 142us/step\n",
      "25000/25000 [==============================] - 5s 203us/step\n",
      "25000/25000 [==============================] - 3s 133us/step\n",
      "25000/25000 [==============================] - 5s 202us/step\n",
      "25000/25000 [==============================] - 3s 114us/step\n",
      "25000/25000 [==============================] - 5s 190us/step\n",
      "25000/25000 [==============================] - 3s 115us/step\n",
      "25000/25000 [==============================] - 5s 202us/step\n",
      "25000/25000 [==============================] - 3s 112us/step\n",
      "25000/25000 [==============================] - 6s 260us/step\n",
      "25000/25000 [==============================] - 3s 135us/step\n",
      "25000/25000 [==============================] - 5s 198us/step\n",
      "25000/25000 [==============================] - 3s 118us/step\n",
      "25000/25000 [==============================] - 5s 198us/step\n",
      "25000/25000 [==============================] - 3s 117us/step\n",
      "25000/25000 [==============================] - 5s 207us/step\n",
      "25000/25000 [==============================] - 3s 121us/step\n",
      "25000/25000 [==============================] - 5s 196us/step\n",
      "25000/25000 [==============================] - 3s 118us/step\n",
      "25000/25000 [==============================] - 5s 216us/step\n",
      "25000/25000 [==============================] - 3s 121us/step\n",
      "25000/25000 [==============================] - 6s 234us/step\n",
      "25000/25000 [==============================] - 3s 114us/step\n",
      "25000/25000 [==============================] - 6s 230us/step\n",
      "25000/25000 [==============================] - 3s 133us/step\n",
      "25000/25000 [==============================] - 5s 201us/step\n",
      "25000/25000 [==============================] - 3s 110us/step\n",
      "25000/25000 [==============================] - 6s 229us/step\n",
      "25000/25000 [==============================] - 4s 141us/step\n",
      "25000/25000 [==============================] - 6s 225us/step\n",
      "25000/25000 [==============================] - 3s 123us/step\n",
      "25000/25000 [==============================] - 7s 266us/step\n",
      "25000/25000 [==============================] - 3s 127us/step\n",
      "25000/25000 [==============================] - 6s 231us/step\n",
      "25000/25000 [==============================] - 3s 133us/step\n",
      "25000/25000 [==============================] - 6s 224us/step\n",
      "25000/25000 [==============================] - 3s 124us/step\n",
      "25000/25000 [==============================] - 7s 278us/step\n",
      "25000/25000 [==============================] - 3s 125us/step\n",
      "25000/25000 [==============================] - 7s 277us/step\n",
      "25000/25000 [==============================] - 3s 128us/step\n",
      "25000/25000 [==============================] - 6s 233us/step\n",
      "25000/25000 [==============================] - 3s 134us/step\n",
      "25000/25000 [==============================] - 7s 293us/step\n",
      "25000/25000 [==============================] - 3s 129us/step\n",
      "25000/25000 [==============================] - 6s 255us/step\n",
      "25000/25000 [==============================] - 4s 180us/step\n",
      "25000/25000 [==============================] - 7s 298us/step\n",
      "25000/25000 [==============================] - 4s 170us/step\n",
      "25000/25000 [==============================] - 6s 238us/step\n",
      "25000/25000 [==============================] - 3s 131us/step\n",
      "25000/25000 [==============================] - 7s 267us/step\n",
      "25000/25000 [==============================] - 3s 132us/step\n",
      "25000/25000 [==============================] - 7s 295us/step\n",
      "25000/25000 [==============================] - 4s 146us/step\n",
      "25000/25000 [==============================] - 6s 259us/step\n",
      "25000/25000 [==============================] - 4s 146us/step\n",
      "25000/25000 [==============================] - 7s 267us/step\n",
      "25000/25000 [==============================] - 5s 217us/step\n",
      "25000/25000 [==============================] - 7s 268us/step\n",
      "25000/25000 [==============================] - 4s 150us/step\n",
      "25000/25000 [==============================] - 7s 277us/step\n",
      "25000/25000 [==============================] - 4s 150us/step\n",
      "25000/25000 [==============================] - 7s 286us/step\n",
      "25000/25000 [==============================] - 4s 155us/step\n",
      "25000/25000 [==============================] - 8s 317us/step\n",
      "25000/25000 [==============================] - 4s 145us/step\n",
      "25000/25000 [==============================] - 7s 271us/step\n",
      "25000/25000 [==============================] - 3s 136us/step\n",
      "25000/25000 [==============================] - 7s 270us/step\n",
      "25000/25000 [==============================] - 4s 150us/step\n",
      "25000/25000 [==============================] - 8s 315us/step\n",
      "25000/25000 [==============================] - 4s 166us/step\n",
      "25000/25000 [==============================] - 6s 260us/step\n",
      "25000/25000 [==============================] - 4s 145us/step\n",
      "25000/25000 [==============================] - 7s 290us/step\n",
      "25000/25000 [==============================] - 4s 174us/step\n",
      "25000/25000 [==============================] - 8s 302us/step\n",
      "25000/25000 [==============================] - 4s 168us/step\n",
      "25000/25000 [==============================] - 8s 336us/step\n",
      "25000/25000 [==============================] - 5s 182us/step\n",
      "25000/25000 [==============================] - 8s 310us/step\n",
      "25000/25000 [==============================] - 4s 165us/step\n",
      "25000/25000 [==============================] - 10s 401us/step\n",
      "25000/25000 [==============================] - 4s 176us/step\n",
      "25000/25000 [==============================] - 9s 358us/step\n",
      "25000/25000 [==============================] - 5s 192us/step\n",
      "25000/25000 [==============================] - 11s 456us/step\n",
      "25000/25000 [==============================] - 5s 198us/step\n",
      "25000/25000 [==============================] - 8s 326us/step\n",
      "25000/25000 [==============================] - 4s 175us/step\n",
      "25000/25000 [==============================] - 8s 309us/step\n",
      "25000/25000 [==============================] - 4s 166us/step\n",
      "25000/25000 [==============================] - 9s 367us/step\n",
      "25000/25000 [==============================] - 4s 168us/step\n",
      "25000/25000 [==============================] - 15s 584us/step\n",
      "25000/25000 [==============================] - 6s 242us/step\n",
      "25000/25000 [==============================] - 10s 408us/step\n",
      "25000/25000 [==============================] - 5s 218us/step\n",
      "25000/25000 [==============================] - 7s 299us/step\n",
      "25000/25000 [==============================] - 4s 178us/step\n",
      "25000/25000 [==============================] - 11s 421us/step\n",
      "25000/25000 [==============================] - 5s 202us/step\n",
      "25000/25000 [==============================] - 10s 395us/step\n",
      "25000/25000 [==============================] - 6s 252us/step\n",
      "25000/25000 [==============================] - 8s 325us/step\n",
      "25000/25000 [==============================] - 4s 180us/step\n",
      "25000/25000 [==============================] - 8s 332us/step\n",
      "25000/25000 [==============================] - 5s 183us/step\n",
      "25000/25000 [==============================] - 8s 334us/step\n",
      "25000/25000 [==============================] - 4s 177us/step\n",
      "25000/25000 [==============================] - 9s 351us/step\n",
      "25000/25000 [==============================] - 5s 190us/step\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "25000/25000 [==============================] - 8s 339us/step\n",
      "25000/25000 [==============================] - 4s 176us/step\n",
      "25000/25000 [==============================] - 8s 340us/step\n",
      "25000/25000 [==============================] - 5s 180us/step\n",
      "25000/25000 [==============================] - 9s 347us/step\n",
      "25000/25000 [==============================] - 5s 182us/step\n",
      "25000/25000 [==============================] - 9s 351us/step\n",
      "25000/25000 [==============================] - 5s 189us/step\n",
      "25000/25000 [==============================] - 11s 439us/step\n",
      "25000/25000 [==============================] - 5s 189us/step\n",
      "25000/25000 [==============================] - 9s 353us/step\n",
      "25000/25000 [==============================] - 4s 177us/step\n",
      "25000/25000 [==============================] - 9s 347us/step\n",
      "25000/25000 [==============================] - 4s 178us/step\n",
      "25000/25000 [==============================] - 9s 359us/step\n",
      "25000/25000 [==============================] - 6s 221us/step\n",
      "25000/25000 [==============================] - 11s 426us/step\n",
      "25000/25000 [==============================] - 6s 223us/step\n",
      "25000/25000 [==============================] - 9s 354us/step\n",
      "25000/25000 [==============================] - 4s 170us/step\n",
      "25000/25000 [==============================] - 9s 363us/step\n",
      "25000/25000 [==============================] - 4s 174us/step\n",
      "25000/25000 [==============================] - 10s 389us/step\n",
      "25000/25000 [==============================] - 4s 164us/step\n",
      "25000/25000 [==============================] - 9s 366us/step\n",
      "25000/25000 [==============================] - 5s 182us/step\n",
      "25000/25000 [==============================] - 10s 384us/step\n",
      "25000/25000 [==============================] - 4s 175us/step\n",
      "25000/25000 [==============================] - 11s 443us/step\n",
      "25000/25000 [==============================] - 5s 207us/step\n",
      "25000/25000 [==============================] - 9s 354us/step\n",
      "25000/25000 [==============================] - 5s 181us/step\n",
      "25000/25000 [==============================] - 10s 402us/step\n",
      "25000/25000 [==============================] - 5s 196us/step\n",
      "25000/25000 [==============================] - 11s 423us/step\n",
      "25000/25000 [==============================] - 7s 277us/step\n",
      "25000/25000 [==============================] - 11s 447us/step\n",
      "25000/25000 [==============================] - 10s 407us/step\n",
      "25000/25000 [==============================] - 9s 380us/step\n",
      "25000/25000 [==============================] - 7s 286us/step\n",
      "25000/25000 [==============================] - 8s 335us/step\n",
      "25000/25000 [==============================] - 4s 161us/step\n",
      "25000/25000 [==============================] - 8s 331us/step\n",
      "25000/25000 [==============================] - 4s 177us/step\n",
      "25000/25000 [==============================] - 9s 372us/step\n",
      "25000/25000 [==============================] - 5s 204us/step\n",
      "25000/25000 [==============================] - 9s 376us/step\n",
      "25000/25000 [==============================] - 4s 157us/step\n",
      "25000/25000 [==============================] - 9s 360us/step\n",
      "25000/25000 [==============================] - 4s 179us/step\n",
      "25000/25000 [==============================] - 9s 370us/step\n",
      "25000/25000 [==============================] - 7s 272us/step\n",
      "25000/25000 [==============================] - 9s 372us/step\n",
      "25000/25000 [==============================] - 5s 192us/step\n",
      "25000/25000 [==============================] - 13s 512us/step\n",
      "25000/25000 [==============================] - 5s 197us/step\n",
      "25000/25000 [==============================] - 10s 396us/step\n",
      "25000/25000 [==============================] - 7s 276us/step\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[('model128_128_128', 0.77568, 0.99976),\n",
       " ('model128_128_16', 0.79284, 0.99948),\n",
       " ('model128_128_32', 0.78716, 0.99972),\n",
       " ('model128_128_64', 0.77952, 0.99976),\n",
       " ('model128_16_128', 0.83964, 0.99996),\n",
       " ('model128_16_16', 0.8366, 0.99872),\n",
       " ('model128_16_32', 0.83788, 0.9998),\n",
       " ('model128_16_64', 0.83736, 0.99992),\n",
       " ('model128_1_128', 0.85048, 0.95844),\n",
       " ('model128_1_16', 0.8564, 0.87244),\n",
       " ('model128_1_32', 0.85588, 0.89756),\n",
       " ('model128_1_64', 0.85508, 0.92948),\n",
       " ('model128_20_128', 0.84324, 1.0),\n",
       " ('model128_20_16', 0.8388, 0.998),\n",
       " ('model128_20_32', 0.84012, 0.99976),\n",
       " ('model128_20_64', 0.84124, 0.99996),\n",
       " ('model128_32_128', 0.83632, 1.0),\n",
       " ('model128_32_16', 0.83144, 0.99796),\n",
       " ('model128_32_32', 0.83484, 0.99996),\n",
       " ('model128_32_64', 0.83504, 1.0),\n",
       " ('model128_64_128', 0.82076, 1.0),\n",
       " ('model128_64_16', 0.836, 0.99992),\n",
       " ('model128_64_32', 0.83252, 1.0),\n",
       " ('model128_64_64', 0.82692, 0.99996),\n",
       " ('model128_b_128_128', 0.82504, 1.0),\n",
       " ('model128_b_128_16', 0.83168, 1.0),\n",
       " ('model128_b_128_32', 0.82756, 1.0),\n",
       " ('model128_b_128_64', 0.82932, 0.99996),\n",
       " ('model128_b_16_128', 0.83244, 0.99976),\n",
       " ('model128_b_16_16', 0.83452, 0.99832),\n",
       " ('model128_b_16_32', 0.83992, 0.9998),\n",
       " ('model128_b_16_64', 0.83864, 0.99996),\n",
       " ('model128_b_1_128', 0.8514, 0.9636),\n",
       " ('model128_b_1_16', 0.85612, 0.87688),\n",
       " ('model128_b_1_32', 0.858, 0.90356),\n",
       " ('model128_b_1_64', 0.858, 0.93432),\n",
       " ('model128_b_20_128', 0.83888, 1.0),\n",
       " ('model128_b_20_16', 0.83732, 0.99796),\n",
       " ('model128_b_20_32', 0.83628, 0.99972),\n",
       " ('model128_b_20_64', 0.8376, 1.0),\n",
       " ('model128_b_32_128', 0.83816, 1.0),\n",
       " ('model128_b_32_16', 0.83784, 0.99932),\n",
       " ('model128_b_32_32', 0.84116, 1.0),\n",
       " ('model128_b_32_64', 0.83736, 1.0),\n",
       " ('model128_b_64_128', 0.83444, 1.0),\n",
       " ('model128_b_64_16', 0.8316, 1.0),\n",
       " ('model128_b_64_32', 0.8386, 1.0),\n",
       " ('model128_b_64_64', 0.83364, 1.0),\n",
       " ('model16_128_128', 0.81532, 0.98904),\n",
       " ('model16_128_16', 0.81792, 0.98772),\n",
       " ('model16_128_32', 0.8122, 0.98844),\n",
       " ('model16_128_64', 0.81616, 0.98876),\n",
       " ('model16_16_128', 0.8272, 0.98356),\n",
       " ('model16_16_16', 0.82864, 0.95872),\n",
       " ('model16_16_32', 0.82704, 0.97772),\n",
       " ('model16_16_64', 0.82568, 0.97996),\n",
       " ('model16_1_128', 0.85412, 0.91188),\n",
       " ('model16_1_16', 0.85096, 0.87388),\n",
       " ('model16_1_32', 0.85328, 0.8882),\n",
       " ('model16_1_64', 0.85356, 0.903),\n",
       " ('model16_20_128', 0.8316, 0.97908),\n",
       " ('model16_20_16', 0.83572, 0.96152),\n",
       " ('model16_20_32', 0.82988, 0.97512),\n",
       " ('model16_20_64', 0.83256, 0.9802),\n",
       " ('model16_32_128', 0.82464, 0.9864),\n",
       " ('model16_32_16', 0.82456, 0.98216),\n",
       " ('model16_32_32', 0.82292, 0.98564),\n",
       " ('model16_32_64', 0.82124, 0.98712),\n",
       " ('model16_40_16', 0.82928, 0.98268),\n",
       " ('model16_64_128', 0.81796, 0.98856),\n",
       " ('model16_64_16', 0.8214, 0.98524),\n",
       " ('model16_64_32', 0.82108, 0.98676),\n",
       " ('model16_64_64', 0.81888, 0.98688),\n",
       " ('model16_b_128_16', 0.82924, 0.9928),\n",
       " ('model16_b_128_32', 0.8276, 0.99388),\n",
       " ('model16_b_128_64', 0.82596, 0.99368),\n",
       " ('model16_b_16_128', 0.83012, 0.99192),\n",
       " ('model16_b_16_16', 0.8308, 0.97944),\n",
       " ('model16_b_16_32', 0.82564, 0.98812),\n",
       " ('model16_b_16_64', 0.8328, 0.99008),\n",
       " ('model16_b_1_128', 0.858, 0.90772),\n",
       " ('model16_b_1_16', 0.85344, 0.87248),\n",
       " ('model16_b_1_32', 0.85864, 0.88748),\n",
       " ('model16_b_1_64', 0.85444, 0.89364),\n",
       " ('model16_b_20_128', 0.8292, 0.98548),\n",
       " ('model16_b_20_16', 0.82868, 0.9712),\n",
       " ('model16_b_20_32', 0.82704, 0.98132),\n",
       " ('model16_b_20_64', 0.82928, 0.98488),\n",
       " ('model16_b_32_128', 0.82984, 0.99316),\n",
       " ('model16_b_32_16', 0.82836, 0.99132),\n",
       " ('model16_b_32_32', 0.82852, 0.99208),\n",
       " ('model16_b_32_64', 0.82852, 0.99228),\n",
       " ('model16_b_64_128', 0.82956, 0.9938),\n",
       " ('model16_b_64_16', 0.82744, 0.99328),\n",
       " ('model16_b_64_32', 0.8268, 0.9936),\n",
       " ('model16_b_64_64', 0.82916, 0.9938)]"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from keras.models import load_model\n",
    "import os\n",
    "\n",
    "location = os.getcwd() + '\\\\savedmodels'\n",
    "models = []\n",
    "results = []\n",
    "\n",
    "for files in os.listdir(location):\n",
    "    try:\n",
    "        models.append(str(files))\n",
    "    except Exception as e:\n",
    "        raise e\n",
    "        print(\"No files found here!\")\n",
    "        \n",
    "for model_ in models:\n",
    "    try:\n",
    "        model_loc = location + '\\\\' + model_\n",
    "        evalmodel = load_model(model_loc)\n",
    "        testscore = evalmodel.evaluate(x_test, y_test, verbose=1)[1]\n",
    "        trainscore = evalmodel.evaluate(x_train, y_train, verbose=1)[1]\n",
    "        runtitle = str(model_)\n",
    "        results.append((runtitle,testscore,trainscore))\n",
    "    except:\n",
    "        results.append('N/A')\n",
    "        \n",
    "results\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-04-30T23:57:03.331589Z",
     "start_time": "2018-04-30T23:57:03.319561Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "('model128_128_128', 0.77568, 0.99976)\n",
      "('model128_128_16', 0.79284, 0.99948)\n",
      "('model128_128_32', 0.78716, 0.99972)\n",
      "('model128_128_64', 0.77952, 0.99976)\n",
      "('model128_16_128', 0.83964, 0.99996)\n",
      "('model128_16_16', 0.8366, 0.99872)\n",
      "('model128_16_32', 0.83788, 0.9998)\n",
      "('model128_16_64', 0.83736, 0.99992)\n",
      "('model128_1_128', 0.85048, 0.95844)\n",
      "('model128_1_16', 0.8564, 0.87244)\n",
      "('model128_1_32', 0.85588, 0.89756)\n",
      "('model128_1_64', 0.85508, 0.92948)\n",
      "('model128_20_128', 0.84324, 1.0)\n",
      "('model128_20_16', 0.8388, 0.998)\n",
      "('model128_20_32', 0.84012, 0.99976)\n",
      "('model128_20_64', 0.84124, 0.99996)\n",
      "('model128_32_128', 0.83632, 1.0)\n",
      "('model128_32_16', 0.83144, 0.99796)\n",
      "('model128_32_32', 0.83484, 0.99996)\n",
      "('model128_32_64', 0.83504, 1.0)\n",
      "('model128_64_128', 0.82076, 1.0)\n",
      "('model128_64_16', 0.836, 0.99992)\n",
      "('model128_64_32', 0.83252, 1.0)\n",
      "('model128_64_64', 0.82692, 0.99996)\n",
      "('model128_b_128_128', 0.82504, 1.0)\n",
      "('model128_b_128_16', 0.83168, 1.0)\n",
      "('model128_b_128_32', 0.82756, 1.0)\n",
      "('model128_b_128_64', 0.82932, 0.99996)\n",
      "('model128_b_16_128', 0.83244, 0.99976)\n",
      "('model128_b_16_16', 0.83452, 0.99832)\n",
      "('model128_b_16_32', 0.83992, 0.9998)\n",
      "('model128_b_16_64', 0.83864, 0.99996)\n",
      "('model128_b_1_128', 0.8514, 0.9636)\n",
      "('model128_b_1_16', 0.85612, 0.87688)\n",
      "('model128_b_1_32', 0.858, 0.90356)\n",
      "('model128_b_1_64', 0.858, 0.93432)\n",
      "('model128_b_20_128', 0.83888, 1.0)\n",
      "('model128_b_20_16', 0.83732, 0.99796)\n",
      "('model128_b_20_32', 0.83628, 0.99972)\n",
      "('model128_b_20_64', 0.8376, 1.0)\n",
      "('model128_b_32_128', 0.83816, 1.0)\n",
      "('model128_b_32_16', 0.83784, 0.99932)\n",
      "('model128_b_32_32', 0.84116, 1.0)\n",
      "('model128_b_32_64', 0.83736, 1.0)\n",
      "('model128_b_64_128', 0.83444, 1.0)\n",
      "('model128_b_64_16', 0.8316, 1.0)\n",
      "('model128_b_64_32', 0.8386, 1.0)\n",
      "('model128_b_64_64', 0.83364, 1.0)\n",
      "('model16_128_128', 0.81532, 0.98904)\n",
      "('model16_128_16', 0.81792, 0.98772)\n",
      "('model16_128_32', 0.8122, 0.98844)\n",
      "('model16_128_64', 0.81616, 0.98876)\n",
      "('model16_16_128', 0.8272, 0.98356)\n",
      "('model16_16_16', 0.82864, 0.95872)\n",
      "('model16_16_32', 0.82704, 0.97772)\n",
      "('model16_16_64', 0.82568, 0.97996)\n",
      "('model16_1_128', 0.85412, 0.91188)\n",
      "('model16_1_16', 0.85096, 0.87388)\n",
      "('model16_1_32', 0.85328, 0.8882)\n",
      "('model16_1_64', 0.85356, 0.903)\n",
      "('model16_20_128', 0.8316, 0.97908)\n",
      "('model16_20_16', 0.83572, 0.96152)\n",
      "('model16_20_32', 0.82988, 0.97512)\n",
      "('model16_20_64', 0.83256, 0.9802)\n",
      "('model16_32_128', 0.82464, 0.9864)\n",
      "('model16_32_16', 0.82456, 0.98216)\n",
      "('model16_32_32', 0.82292, 0.98564)\n",
      "('model16_32_64', 0.82124, 0.98712)\n",
      "('model16_40_16', 0.82928, 0.98268)\n",
      "('model16_64_128', 0.81796, 0.98856)\n",
      "('model16_64_16', 0.8214, 0.98524)\n",
      "('model16_64_32', 0.82108, 0.98676)\n",
      "('model16_64_64', 0.81888, 0.98688)\n",
      "('model16_b_128_16', 0.82924, 0.9928)\n",
      "('model16_b_128_32', 0.8276, 0.99388)\n",
      "('model16_b_128_64', 0.82596, 0.99368)\n",
      "('model16_b_16_128', 0.83012, 0.99192)\n",
      "('model16_b_16_16', 0.8308, 0.97944)\n",
      "('model16_b_16_32', 0.82564, 0.98812)\n",
      "('model16_b_16_64', 0.8328, 0.99008)\n",
      "('model16_b_1_128', 0.858, 0.90772)\n",
      "('model16_b_1_16', 0.85344, 0.87248)\n",
      "('model16_b_1_32', 0.85864, 0.88748)\n",
      "('model16_b_1_64', 0.85444, 0.89364)\n",
      "('model16_b_20_128', 0.8292, 0.98548)\n",
      "('model16_b_20_16', 0.82868, 0.9712)\n",
      "('model16_b_20_32', 0.82704, 0.98132)\n",
      "('model16_b_20_64', 0.82928, 0.98488)\n",
      "('model16_b_32_128', 0.82984, 0.99316)\n",
      "('model16_b_32_16', 0.82836, 0.99132)\n",
      "('model16_b_32_32', 0.82852, 0.99208)\n",
      "('model16_b_32_64', 0.82852, 0.99228)\n",
      "('model16_b_64_128', 0.82956, 0.9938)\n",
      "('model16_b_64_16', 0.82744, 0.99328)\n",
      "('model16_b_64_32', 0.8268, 0.9936)\n",
      "('model16_b_64_64', 0.82916, 0.9938)\n"
     ]
    }
   ],
   "source": [
    "# Evaluating the model on the training and testing set\n",
    "import csv\n",
    "for scores in results:\n",
    "    print(scores)\n",
    "\n",
    "with open(location + '\\\\' +'IMDB_In_Keras-testing_epochs_layers_bSize.csv', 'r+t') as myfile:\n",
    "    writer = csv.writer(myfile, lineterminator='\\n')\n",
    "    for rows in results:\n",
    "        writer.writerow(x for x in rows)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#best results are for smaller epochs, as all our models like to overfit.\n",
    "\n",
    "#>85%\n",
    "\n",
    "\n",
    "* model16_b_1_32\n",
    "* model128_b_1_32\n",
    "* model128_b_1_64\n",
    "* model16_b_1_128\n",
    "* model128_1_16\n",
    "* model128_b_1_16\n",
    "* model128_1_32\n",
    "* model128_1_64\n",
    "* model16_b_1_64\n",
    "* model16_1_128\n",
    "* model16_1_64\n",
    "* model16_b_1_16\n",
    "* model16_1_32\n",
    "* model128_b_1_128\n",
    "* model16_1_16\n",
    "* model128_1_128"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-05-01T00:02:59.812359Z",
     "start_time": "2018-05-01T00:02:59.808881Z"
    }
   },
   "outputs": [],
   "source": [
    "# Next we try adjusting our dropout rates and activation functions (relu, tanh, selu, elu)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "hide_input": false,
  "kernelspec": {
   "display_name": "Python (aind)",
   "language": "python",
   "name": "aind"
  },
  "nbTranslate": {
   "displayLangs": [
    "*"
   ],
   "hotkey": "alt-t",
   "langInMainMenu": true,
   "sourceLang": "en",
   "targetLang": "fr",
   "useGoogleTranslate": true
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "position": {
    "height": "884px",
    "left": "1722.32px",
    "right": "20px",
    "top": "88.9931px",
    "width": "350px"
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}