AFML 9.1.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Hyper-Parameter Tuning with CV\n",
    "\n",
    "This notebook will cover exercise answer.\n",
    "\n",
    "* Exercise 9.1\n",
    "* Exercise 9.2\n",
    "* Exercise 9.3\n",
    "* Exercise 9.4\n",
    "* Exercise 9.5\n",
    "* Exercise 9.6\n",
    "\n",
    "As we go along, there will be some explanations.\n",
    "\n",
    "Most of the functions below can be found under Tool/metrics\n",
    "\n",
    "Contact: boyboi86@gmail.com"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Num of CPU core:  4\n",
      "Machine info:  Windows-10-10.0.18362-SP0\n",
      "Python 3.7.4 (default, Aug  9 2019, 18:34:13) [MSC v.1915 64 bit (AMD64)]\n",
      "Numpy 1.17.3\n",
      "Pandas 1.0.3\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\Wei_X\\Anaconda3\\lib\\site-packages\\statsmodels\\tools\\_testing.py:19: FutureWarning: pandas.util.testing is deprecated. Use the functions in the public API at pandas.testing instead.\n",
      "  import pandas.util.testing as tm\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<Figure size 1500x800 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "import research as rs\n",
    "\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\Wei_X\\Desktop\\Python\\research\\Util\\indicator.py:54: FutureWarning: The pandas.datetime class is deprecated and will be removed from pandas in a future version. Import from datetime module instead.\n",
      "  df0 = pd.date_range(periods=n_samples, freq=pd.tseries.offsets.BDay(), end=pd.datetime.today())\n"
     ]
    }
   ],
   "source": [
    "X, y = rs.make_data(n_features=10, n_informative=5, n_redundant= 0, n_samples= 2000)\n",
    "\n",
    "#Take note of depreciation warning Pandas 1.0.3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Using backend SequentialBackend with 1 concurrent workers.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 10 folds for each of 25 candidates, totalling 250 fits\n",
      "[CV] C=0.01, gamma=0.01, kernel=rbf ..................................\n",
      "[CV] ..... C=0.01, gamma=0.01, kernel=rbf, score=-0.287, total=   0.6s\n",
      "[CV] C=0.01, gamma=0.01, kernel=rbf ..................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Done   1 out of   1 | elapsed:    0.5s remaining:    0.0s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] ..... C=0.01, gamma=0.01, kernel=rbf, score=-0.260, total=   0.6s\n",
      "[CV] C=0.01, gamma=0.01, kernel=rbf ..................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Done   2 out of   2 | elapsed:    1.0s remaining:    0.0s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] ..... C=0.01, gamma=0.01, kernel=rbf, score=-0.198, total=   0.5s\n",
      "[CV] C=0.01, gamma=0.01, kernel=rbf ..................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Done   3 out of   3 | elapsed:    1.6s remaining:    0.0s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] ..... C=0.01, gamma=0.01, kernel=rbf, score=-0.186, total=   0.5s\n",
      "[CV] C=0.01, gamma=0.01, kernel=rbf ..................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Done   4 out of   4 | elapsed:    2.1s remaining:    0.0s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] ..... C=0.01, gamma=0.01, kernel=rbf, score=-0.271, total=   0.5s\n",
      "[CV] C=0.01, gamma=0.01, kernel=rbf ..................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Done   5 out of   5 | elapsed:    2.6s remaining:    0.0s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] ..... C=0.01, gamma=0.01, kernel=rbf, score=-0.618, total=   0.5s\n",
      "[CV] C=0.01, gamma=0.01, kernel=rbf ..................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Done   6 out of   6 | elapsed:    3.2s remaining:    0.0s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] ..... C=0.01, gamma=0.01, kernel=rbf, score=-0.515, total=   0.5s\n",
      "[CV] C=0.01, gamma=0.01, kernel=rbf ..................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Done   7 out of   7 | elapsed:    3.7s remaining:    0.0s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] ..... C=0.01, gamma=0.01, kernel=rbf, score=-0.605, total=   0.5s\n",
      "[CV] C=0.01, gamma=0.01, kernel=rbf ..................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Done   8 out of   8 | elapsed:    4.2s remaining:    0.0s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] ..... C=0.01, gamma=0.01, kernel=rbf, score=-0.507, total=   0.5s\n",
      "[CV] C=0.01, gamma=0.01, kernel=rbf ..................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Done   9 out of   9 | elapsed:    4.8s remaining:    0.0s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] ..... C=0.01, gamma=0.01, kernel=rbf, score=-0.623, total=   0.5s\n",
      "[CV] C=0.01, gamma=0.1, kernel=rbf ...................................\n",
      "[CV] ...... C=0.01, gamma=0.1, kernel=rbf, score=-0.276, total=   0.5s\n",
      "[CV] C=0.01, gamma=0.1, kernel=rbf ...................................\n",
      "[CV] ...... C=0.01, gamma=0.1, kernel=rbf, score=-0.231, total=   0.5s\n",
      "[CV] C=0.01, gamma=0.1, kernel=rbf ...................................\n",
      "[CV] ...... C=0.01, gamma=0.1, kernel=rbf, score=-0.202, total=   0.5s\n",
      "[CV] C=0.01, gamma=0.1, kernel=rbf ...................................\n",
      "[CV] ...... C=0.01, gamma=0.1, kernel=rbf, score=-0.205, total=   0.5s\n",
      "[CV] C=0.01, gamma=0.1, kernel=rbf ...................................\n",
      "[CV] ...... C=0.01, gamma=0.1, kernel=rbf, score=-0.223, total=   0.5s\n",
      "[CV] C=0.01, gamma=0.1, kernel=rbf ...................................\n",
      "[CV] ...... C=0.01, gamma=0.1, kernel=rbf, score=-0.545, total=   0.5s\n",
      "[CV] C=0.01, gamma=0.1, kernel=rbf ...................................\n",
      "[CV] ...... C=0.01, gamma=0.1, kernel=rbf, score=-0.505, total=   0.5s\n",
      "[CV] C=0.01, gamma=0.1, kernel=rbf ...................................\n",
      "[CV] ...... C=0.01, gamma=0.1, kernel=rbf, score=-0.546, total=   0.5s\n",
      "[CV] C=0.01, gamma=0.1, kernel=rbf ...................................\n",
      "[CV] ...... C=0.01, gamma=0.1, kernel=rbf, score=-0.490, total=   0.5s\n",
      "[CV] C=0.01, gamma=0.1, kernel=rbf ...................................\n",
      "[CV] ...... C=0.01, gamma=0.1, kernel=rbf, score=-0.547, total=   0.5s\n",
      "[CV] C=0.01, gamma=1, kernel=rbf .....................................\n",
      "[CV] ........ C=0.01, gamma=1, kernel=rbf, score=-0.694, total=   0.5s\n",
      "[CV] C=0.01, gamma=1, kernel=rbf .....................................\n",
      "[CV] ........ C=0.01, gamma=1, kernel=rbf, score=-0.693, total=   0.5s\n",
      "[CV] C=0.01, gamma=1, kernel=rbf .....................................\n",
      "[CV] ........ C=0.01, gamma=1, kernel=rbf, score=-0.693, total=   0.5s\n",
      "[CV] C=0.01, gamma=1, kernel=rbf .....................................\n",
      "[CV] ........ C=0.01, gamma=1, kernel=rbf, score=-0.694, total=   0.5s\n",
      "[CV] C=0.01, gamma=1, kernel=rbf .....................................\n",
      "[CV] ........ C=0.01, gamma=1, kernel=rbf, score=-0.693, total=   0.5s\n",
      "[CV] C=0.01, gamma=1, kernel=rbf .....................................\n",
      "[CV] ........ C=0.01, gamma=1, kernel=rbf, score=-0.694, total=   0.5s\n",
      "[CV] C=0.01, gamma=1, kernel=rbf .....................................\n",
      "[CV] ........ C=0.01, gamma=1, kernel=rbf, score=-0.693, total=   0.5s\n",
      "[CV] C=0.01, gamma=1, kernel=rbf .....................................\n",
      "[CV] ........ C=0.01, gamma=1, kernel=rbf, score=-0.694, total=   0.5s\n",
      "[CV] C=0.01, gamma=1, kernel=rbf .....................................\n",
      "[CV] ........ C=0.01, gamma=1, kernel=rbf, score=-0.694, total=   0.5s\n",
      "[CV] C=0.01, gamma=1, kernel=rbf .....................................\n",
      "[CV] ........ C=0.01, gamma=1, kernel=rbf, score=-0.693, total=   0.5s\n",
      "[CV] C=0.01, gamma=10, kernel=rbf ....................................\n",
      "[CV] ....... C=0.01, gamma=10, kernel=rbf, score=-0.695, total=   0.6s\n",
      "[CV] C=0.01, gamma=10, kernel=rbf ....................................\n",
      "[CV] ....... C=0.01, gamma=10, kernel=rbf, score=-0.693, total=   0.6s\n",
      "[CV] C=0.01, gamma=10, kernel=rbf ....................................\n",
      "[CV] ....... C=0.01, gamma=10, kernel=rbf, score=-0.694, total=   0.6s\n",
      "[CV] C=0.01, gamma=10, kernel=rbf ....................................\n",
      "[CV] ....... C=0.01, gamma=10, kernel=rbf, score=-0.695, total=   0.6s\n",
      "[CV] C=0.01, gamma=10, kernel=rbf ....................................\n",
      "[CV] ....... C=0.01, gamma=10, kernel=rbf, score=-0.694, total=   0.6s\n",
      "[CV] C=0.01, gamma=10, kernel=rbf ....................................\n",
      "[CV] ....... C=0.01, gamma=10, kernel=rbf, score=-0.693, total=   0.6s\n",
      "[CV] C=0.01, gamma=10, kernel=rbf ....................................\n",
      "[CV] ....... C=0.01, gamma=10, kernel=rbf, score=-0.694, total=   0.6s\n",
      "[CV] C=0.01, gamma=10, kernel=rbf ....................................\n",
      "[CV] ....... C=0.01, gamma=10, kernel=rbf, score=-0.696, total=   0.6s\n",
      "[CV] C=0.01, gamma=10, kernel=rbf ....................................\n",
      "[CV] ....... C=0.01, gamma=10, kernel=rbf, score=-0.694, total=   0.6s\n",
      "[CV] C=0.01, gamma=10, kernel=rbf ....................................\n",
      "[CV] ....... C=0.01, gamma=10, kernel=rbf, score=-0.693, total=   0.6s\n",
      "[CV] C=0.01, gamma=100, kernel=rbf ...................................\n",
      "[CV] ...... C=0.01, gamma=100, kernel=rbf, score=-0.694, total=   1.2s\n",
      "[CV] C=0.01, gamma=100, kernel=rbf ...................................\n",
      "[CV] ...... C=0.01, gamma=100, kernel=rbf, score=-0.693, total=   1.2s\n",
      "[CV] C=0.01, gamma=100, kernel=rbf ...................................\n",
      "[CV] ...... C=0.01, gamma=100, kernel=rbf, score=-0.693, total=   1.2s\n",
      "[CV] C=0.01, gamma=100, kernel=rbf ...................................\n",
      "[CV] ...... C=0.01, gamma=100, kernel=rbf, score=-0.693, total=   1.2s\n",
      "[CV] C=0.01, gamma=100, kernel=rbf ...................................\n",
      "[CV] ...... C=0.01, gamma=100, kernel=rbf, score=-0.694, total=   1.2s\n",
      "[CV] C=0.01, gamma=100, kernel=rbf ...................................\n",
      "[CV] ...... C=0.01, gamma=100, kernel=rbf, score=-0.694, total=   1.2s\n",
      "[CV] C=0.01, gamma=100, kernel=rbf ...................................\n",
      "[CV] ...... C=0.01, gamma=100, kernel=rbf, score=-0.693, total=   1.2s\n",
      "[CV] C=0.01, gamma=100, kernel=rbf ...................................\n",
      "[CV] ...... C=0.01, gamma=100, kernel=rbf, score=-0.694, total=   1.3s\n",
      "[CV] C=0.01, gamma=100, kernel=rbf ...................................\n",
      "[CV] ...... C=0.01, gamma=100, kernel=rbf, score=-0.693, total=   1.2s\n",
      "[CV] C=0.01, gamma=100, kernel=rbf ...................................\n",
      "[CV] ...... C=0.01, gamma=100, kernel=rbf, score=-0.694, total=   1.3s\n",
      "[CV] C=0.1, gamma=0.01, kernel=rbf ...................................\n",
      "[CV] ...... C=0.1, gamma=0.01, kernel=rbf, score=-0.226, total=   0.3s\n",
      "[CV] C=0.1, gamma=0.01, kernel=rbf ...................................\n",
      "[CV] ...... C=0.1, gamma=0.01, kernel=rbf, score=-0.244, total=   0.3s\n",
      "[CV] C=0.1, gamma=0.01, kernel=rbf ...................................\n",
      "[CV] ...... C=0.1, gamma=0.01, kernel=rbf, score=-0.168, total=   0.3s\n",
      "[CV] C=0.1, gamma=0.01, kernel=rbf ...................................\n",
      "[CV] ...... C=0.1, gamma=0.01, kernel=rbf, score=-0.154, total=   0.3s\n",
      "[CV] C=0.1, gamma=0.01, kernel=rbf ...................................\n",
      "[CV] ...... C=0.1, gamma=0.01, kernel=rbf, score=-0.281, total=   0.3s\n",
      "[CV] C=0.1, gamma=0.01, kernel=rbf ...................................\n",
      "[CV] ...... C=0.1, gamma=0.01, kernel=rbf, score=-0.547, total=   0.3s\n",
      "[CV] C=0.1, gamma=0.01, kernel=rbf ...................................\n",
      "[CV] ...... C=0.1, gamma=0.01, kernel=rbf, score=-0.454, total=   0.3s\n",
      "[CV] C=0.1, gamma=0.01, kernel=rbf ...................................\n",
      "[CV] ...... C=0.1, gamma=0.01, kernel=rbf, score=-0.515, total=   0.3s\n",
      "[CV] C=0.1, gamma=0.01, kernel=rbf ...................................\n",
      "[CV] ...... C=0.1, gamma=0.01, kernel=rbf, score=-0.426, total=   0.3s\n",
      "[CV] C=0.1, gamma=0.01, kernel=rbf ...................................\n",
      "[CV] ...... C=0.1, gamma=0.01, kernel=rbf, score=-0.558, total=   0.3s\n",
      "[CV] C=0.1, gamma=0.1, kernel=rbf ....................................\n",
      "[CV] ....... C=0.1, gamma=0.1, kernel=rbf, score=-0.204, total=   0.3s\n",
      "[CV] C=0.1, gamma=0.1, kernel=rbf ....................................\n",
      "[CV] ....... C=0.1, gamma=0.1, kernel=rbf, score=-0.136, total=   0.3s\n",
      "[CV] C=0.1, gamma=0.1, kernel=rbf ....................................\n",
      "[CV] ....... C=0.1, gamma=0.1, kernel=rbf, score=-0.128, total=   0.3s\n",
      "[CV] C=0.1, gamma=0.1, kernel=rbf ....................................\n",
      "[CV] ....... C=0.1, gamma=0.1, kernel=rbf, score=-0.098, total=   0.3s\n",
      "[CV] C=0.1, gamma=0.1, kernel=rbf ....................................\n",
      "[CV] ....... C=0.1, gamma=0.1, kernel=rbf, score=-0.188, total=   0.3s\n",
      "[CV] C=0.1, gamma=0.1, kernel=rbf ....................................\n",
      "[CV] ....... C=0.1, gamma=0.1, kernel=rbf, score=-0.407, total=   0.3s\n",
      "[CV] C=0.1, gamma=0.1, kernel=rbf ....................................\n",
      "[CV] ....... C=0.1, gamma=0.1, kernel=rbf, score=-0.380, total=   0.3s\n",
      "[CV] C=0.1, gamma=0.1, kernel=rbf ....................................\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] ....... C=0.1, gamma=0.1, kernel=rbf, score=-0.342, total=   0.3s\n",
      "[CV] C=0.1, gamma=0.1, kernel=rbf ....................................\n",
      "[CV] ....... C=0.1, gamma=0.1, kernel=rbf, score=-0.346, total=   0.3s\n",
      "[CV] C=0.1, gamma=0.1, kernel=rbf ....................................\n",
      "[CV] ....... C=0.1, gamma=0.1, kernel=rbf, score=-0.433, total=   0.3s\n",
      "[CV] C=0.1, gamma=1, kernel=rbf ......................................\n",
      "[CV] ......... C=0.1, gamma=1, kernel=rbf, score=-0.693, total=   0.5s\n",
      "[CV] C=0.1, gamma=1, kernel=rbf ......................................\n",
      "[CV] ......... C=0.1, gamma=1, kernel=rbf, score=-0.694, total=   0.5s\n",
      "[CV] C=0.1, gamma=1, kernel=rbf ......................................\n",
      "[CV] ......... C=0.1, gamma=1, kernel=rbf, score=-0.693, total=   0.5s\n",
      "[CV] C=0.1, gamma=1, kernel=rbf ......................................\n",
      "[CV] ......... C=0.1, gamma=1, kernel=rbf, score=-0.695, total=   0.5s\n",
      "[CV] C=0.1, gamma=1, kernel=rbf ......................................\n",
      "[CV] ......... C=0.1, gamma=1, kernel=rbf, score=-0.693, total=   0.5s\n",
      "[CV] C=0.1, gamma=1, kernel=rbf ......................................\n",
      "[CV] ......... C=0.1, gamma=1, kernel=rbf, score=-0.694, total=   0.5s\n",
      "[CV] C=0.1, gamma=1, kernel=rbf ......................................\n",
      "[CV] ......... C=0.1, gamma=1, kernel=rbf, score=-0.694, total=   0.5s\n",
      "[CV] C=0.1, gamma=1, kernel=rbf ......................................\n",
      "[CV] ......... C=0.1, gamma=1, kernel=rbf, score=-0.694, total=   0.5s\n",
      "[CV] C=0.1, gamma=1, kernel=rbf ......................................\n",
      "[CV] ......... C=0.1, gamma=1, kernel=rbf, score=-0.694, total=   0.5s\n",
      "[CV] C=0.1, gamma=1, kernel=rbf ......................................\n",
      "[CV] ......... C=0.1, gamma=1, kernel=rbf, score=-0.694, total=   0.5s\n",
      "[CV] C=0.1, gamma=10, kernel=rbf .....................................\n",
      "[CV] ........ C=0.1, gamma=10, kernel=rbf, score=-0.693, total=   0.6s\n",
      "[CV] C=0.1, gamma=10, kernel=rbf .....................................\n",
      "[CV] ........ C=0.1, gamma=10, kernel=rbf, score=-0.694, total=   0.6s\n",
      "[CV] C=0.1, gamma=10, kernel=rbf .....................................\n",
      "[CV] ........ C=0.1, gamma=10, kernel=rbf, score=-0.693, total=   0.6s\n",
      "[CV] C=0.1, gamma=10, kernel=rbf .....................................\n",
      "[CV] ........ C=0.1, gamma=10, kernel=rbf, score=-0.693, total=   0.6s\n",
      "[CV] C=0.1, gamma=10, kernel=rbf .....................................\n",
      "[CV] ........ C=0.1, gamma=10, kernel=rbf, score=-0.693, total=   0.6s\n",
      "[CV] C=0.1, gamma=10, kernel=rbf .....................................\n",
      "[CV] ........ C=0.1, gamma=10, kernel=rbf, score=-0.693, total=   0.6s\n",
      "[CV] C=0.1, gamma=10, kernel=rbf .....................................\n",
      "[CV] ........ C=0.1, gamma=10, kernel=rbf, score=-0.694, total=   0.6s\n",
      "[CV] C=0.1, gamma=10, kernel=rbf .....................................\n",
      "[CV] ........ C=0.1, gamma=10, kernel=rbf, score=-0.694, total=   0.6s\n",
      "[CV] C=0.1, gamma=10, kernel=rbf .....................................\n",
      "[CV] ........ C=0.1, gamma=10, kernel=rbf, score=-0.693, total=   0.6s\n",
      "[CV] C=0.1, gamma=10, kernel=rbf .....................................\n",
      "[CV] ........ C=0.1, gamma=10, kernel=rbf, score=-0.694, total=   0.6s\n",
      "[CV] C=0.1, gamma=100, kernel=rbf ....................................\n",
      "[CV] ....... C=0.1, gamma=100, kernel=rbf, score=-0.693, total=   1.3s\n",
      "[CV] C=0.1, gamma=100, kernel=rbf ....................................\n",
      "[CV] ....... C=0.1, gamma=100, kernel=rbf, score=-0.693, total=   1.3s\n",
      "[CV] C=0.1, gamma=100, kernel=rbf ....................................\n",
      "[CV] ....... C=0.1, gamma=100, kernel=rbf, score=-0.693, total=   1.2s\n",
      "[CV] C=0.1, gamma=100, kernel=rbf ....................................\n",
      "[CV] ....... C=0.1, gamma=100, kernel=rbf, score=-0.693, total=   1.3s\n",
      "[CV] C=0.1, gamma=100, kernel=rbf ....................................\n",
      "[CV] ....... C=0.1, gamma=100, kernel=rbf, score=-0.693, total=   1.2s\n",
      "[CV] C=0.1, gamma=100, kernel=rbf ....................................\n",
      "[CV] ....... C=0.1, gamma=100, kernel=rbf, score=-0.693, total=   1.3s\n",
      "[CV] C=0.1, gamma=100, kernel=rbf ....................................\n",
      "[CV] ....... C=0.1, gamma=100, kernel=rbf, score=-0.694, total=   1.3s\n",
      "[CV] C=0.1, gamma=100, kernel=rbf ....................................\n",
      "[CV] ....... C=0.1, gamma=100, kernel=rbf, score=-0.693, total=   1.3s\n",
      "[CV] C=0.1, gamma=100, kernel=rbf ....................................\n",
      "[CV] ....... C=0.1, gamma=100, kernel=rbf, score=-0.694, total=   1.3s\n",
      "[CV] C=0.1, gamma=100, kernel=rbf ....................................\n",
      "[CV] ....... C=0.1, gamma=100, kernel=rbf, score=-0.697, total=   1.3s\n",
      "[CV] C=1, gamma=0.01, kernel=rbf .....................................\n",
      "[CV] ........ C=1, gamma=0.01, kernel=rbf, score=-0.198, total=   0.3s\n",
      "[CV] C=1, gamma=0.01, kernel=rbf .....................................\n",
      "[CV] ........ C=1, gamma=0.01, kernel=rbf, score=-0.201, total=   0.2s\n",
      "[CV] C=1, gamma=0.01, kernel=rbf .....................................\n",
      "[CV] ........ C=1, gamma=0.01, kernel=rbf, score=-0.159, total=   0.2s\n",
      "[CV] C=1, gamma=0.01, kernel=rbf .....................................\n",
      "[CV] ........ C=1, gamma=0.01, kernel=rbf, score=-0.121, total=   0.2s\n",
      "[CV] C=1, gamma=0.01, kernel=rbf .....................................\n",
      "[CV] ........ C=1, gamma=0.01, kernel=rbf, score=-0.259, total=   0.2s\n",
      "[CV] C=1, gamma=0.01, kernel=rbf .....................................\n",
      "[CV] ........ C=1, gamma=0.01, kernel=rbf, score=-0.455, total=   0.2s\n",
      "[CV] C=1, gamma=0.01, kernel=rbf .....................................\n",
      "[CV] ........ C=1, gamma=0.01, kernel=rbf, score=-0.354, total=   0.2s\n",
      "[CV] C=1, gamma=0.01, kernel=rbf .....................................\n",
      "[CV] ........ C=1, gamma=0.01, kernel=rbf, score=-0.411, total=   0.2s\n",
      "[CV] C=1, gamma=0.01, kernel=rbf .....................................\n",
      "[CV] ........ C=1, gamma=0.01, kernel=rbf, score=-0.337, total=   0.2s\n",
      "[CV] C=1, gamma=0.01, kernel=rbf .....................................\n",
      "[CV] ........ C=1, gamma=0.01, kernel=rbf, score=-0.469, total=   0.3s\n",
      "[CV] C=1, gamma=0.1, kernel=rbf ......................................\n",
      "[CV] ......... C=1, gamma=0.1, kernel=rbf, score=-0.222, total=   0.3s\n",
      "[CV] C=1, gamma=0.1, kernel=rbf ......................................\n",
      "[CV] ......... C=1, gamma=0.1, kernel=rbf, score=-0.114, total=   0.3s\n",
      "[CV] C=1, gamma=0.1, kernel=rbf ......................................\n",
      "[CV] ......... C=1, gamma=0.1, kernel=rbf, score=-0.119, total=   0.3s\n",
      "[CV] C=1, gamma=0.1, kernel=rbf ......................................\n",
      "[CV] ......... C=1, gamma=0.1, kernel=rbf, score=-0.099, total=   0.3s\n",
      "[CV] C=1, gamma=0.1, kernel=rbf ......................................\n",
      "[CV] ......... C=1, gamma=0.1, kernel=rbf, score=-0.173, total=   0.3s\n",
      "[CV] C=1, gamma=0.1, kernel=rbf ......................................\n",
      "[CV] ......... C=1, gamma=0.1, kernel=rbf, score=-0.331, total=   0.3s\n",
      "[CV] C=1, gamma=0.1, kernel=rbf ......................................\n",
      "[CV] ......... C=1, gamma=0.1, kernel=rbf, score=-0.298, total=   0.3s\n",
      "[CV] C=1, gamma=0.1, kernel=rbf ......................................\n",
      "[CV] ......... C=1, gamma=0.1, kernel=rbf, score=-0.212, total=   0.3s\n",
      "[CV] C=1, gamma=0.1, kernel=rbf ......................................\n",
      "[CV] ......... C=1, gamma=0.1, kernel=rbf, score=-0.284, total=   0.2s\n",
      "[CV] C=1, gamma=0.1, kernel=rbf ......................................\n",
      "[CV] ......... C=1, gamma=0.1, kernel=rbf, score=-0.352, total=   0.3s\n",
      "[CV] C=1, gamma=1, kernel=rbf ........................................\n",
      "[CV] ........... C=1, gamma=1, kernel=rbf, score=-0.403, total=   0.6s\n",
      "[CV] C=1, gamma=1, kernel=rbf ........................................\n",
      "[CV] ........... C=1, gamma=1, kernel=rbf, score=-0.356, total=   0.6s\n",
      "[CV] C=1, gamma=1, kernel=rbf ........................................\n",
      "[CV] ........... C=1, gamma=1, kernel=rbf, score=-0.326, total=   0.6s\n",
      "[CV] C=1, gamma=1, kernel=rbf ........................................\n",
      "[CV] ........... C=1, gamma=1, kernel=rbf, score=-0.309, total=   0.6s\n",
      "[CV] C=1, gamma=1, kernel=rbf ........................................\n",
      "[CV] ........... C=1, gamma=1, kernel=rbf, score=-0.339, total=   0.6s\n",
      "[CV] C=1, gamma=1, kernel=rbf ........................................\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] ........... C=1, gamma=1, kernel=rbf, score=-0.614, total=   0.6s\n",
      "[CV] C=1, gamma=1, kernel=rbf ........................................\n",
      "[CV] ........... C=1, gamma=1, kernel=rbf, score=-0.522, total=   0.6s\n",
      "[CV] C=1, gamma=1, kernel=rbf ........................................\n",
      "[CV] ........... C=1, gamma=1, kernel=rbf, score=-0.527, total=   0.7s\n",
      "[CV] C=1, gamma=1, kernel=rbf ........................................\n",
      "[CV] ........... C=1, gamma=1, kernel=rbf, score=-0.675, total=   0.7s\n",
      "[CV] C=1, gamma=1, kernel=rbf ........................................\n",
      "[CV] ........... C=1, gamma=1, kernel=rbf, score=-0.551, total=   0.7s\n",
      "[CV] C=1, gamma=10, kernel=rbf .......................................\n",
      "[CV] .......... C=1, gamma=10, kernel=rbf, score=-0.693, total=   0.8s\n",
      "[CV] C=1, gamma=10, kernel=rbf .......................................\n",
      "[CV] .......... C=1, gamma=10, kernel=rbf, score=-0.693, total=   0.7s\n",
      "[CV] C=1, gamma=10, kernel=rbf .......................................\n",
      "[CV] .......... C=1, gamma=10, kernel=rbf, score=-0.693, total=   0.7s\n",
      "[CV] C=1, gamma=10, kernel=rbf .......................................\n",
      "[CV] .......... C=1, gamma=10, kernel=rbf, score=-0.693, total=   0.7s\n",
      "[CV] C=1, gamma=10, kernel=rbf .......................................\n",
      "[CV] .......... C=1, gamma=10, kernel=rbf, score=-0.693, total=   0.7s\n",
      "[CV] C=1, gamma=10, kernel=rbf .......................................\n",
      "[CV] .......... C=1, gamma=10, kernel=rbf, score=-0.693, total=   0.7s\n",
      "[CV] C=1, gamma=10, kernel=rbf .......................................\n",
      "[CV] .......... C=1, gamma=10, kernel=rbf, score=-0.693, total=   0.8s\n",
      "[CV] C=1, gamma=10, kernel=rbf .......................................\n",
      "[CV] .......... C=1, gamma=10, kernel=rbf, score=-0.693, total=   0.7s\n",
      "[CV] C=1, gamma=10, kernel=rbf .......................................\n",
      "[CV] .......... C=1, gamma=10, kernel=rbf, score=-0.693, total=   0.7s\n",
      "[CV] C=1, gamma=10, kernel=rbf .......................................\n",
      "[CV] .......... C=1, gamma=10, kernel=rbf, score=-0.693, total=   0.7s\n",
      "[CV] C=1, gamma=100, kernel=rbf ......................................\n",
      "[CV] ......... C=1, gamma=100, kernel=rbf, score=-0.693, total=   1.9s\n",
      "[CV] C=1, gamma=100, kernel=rbf ......................................\n",
      "[CV] ......... C=1, gamma=100, kernel=rbf, score=-0.693, total=   1.4s\n",
      "[CV] C=1, gamma=100, kernel=rbf ......................................\n",
      "[CV] ......... C=1, gamma=100, kernel=rbf, score=-0.693, total=   1.3s\n",
      "[CV] C=1, gamma=100, kernel=rbf ......................................\n",
      "[CV] ......... C=1, gamma=100, kernel=rbf, score=-0.693, total=   1.5s\n",
      "[CV] C=1, gamma=100, kernel=rbf ......................................\n",
      "[CV] ......... C=1, gamma=100, kernel=rbf, score=-0.693, total=   1.4s\n",
      "[CV] C=1, gamma=100, kernel=rbf ......................................\n",
      "[CV] ......... C=1, gamma=100, kernel=rbf, score=-0.693, total=   1.4s\n",
      "[CV] C=1, gamma=100, kernel=rbf ......................................\n",
      "[CV] ......... C=1, gamma=100, kernel=rbf, score=-0.693, total=   1.3s\n",
      "[CV] C=1, gamma=100, kernel=rbf ......................................\n",
      "[CV] ......... C=1, gamma=100, kernel=rbf, score=-0.693, total=   1.4s\n",
      "[CV] C=1, gamma=100, kernel=rbf ......................................\n",
      "[CV] ......... C=1, gamma=100, kernel=rbf, score=-0.693, total=   1.4s\n",
      "[CV] C=1, gamma=100, kernel=rbf ......................................\n",
      "[CV] ......... C=1, gamma=100, kernel=rbf, score=-0.693, total=   1.3s\n",
      "[CV] C=10, gamma=0.01, kernel=rbf ....................................\n",
      "[CV] ....... C=10, gamma=0.01, kernel=rbf, score=-0.200, total=   0.2s\n",
      "[CV] C=10, gamma=0.01, kernel=rbf ....................................\n",
      "[CV] ....... C=10, gamma=0.01, kernel=rbf, score=-0.143, total=   0.2s\n",
      "[CV] C=10, gamma=0.01, kernel=rbf ....................................\n",
      "[CV] ....... C=10, gamma=0.01, kernel=rbf, score=-0.145, total=   0.2s\n",
      "[CV] C=10, gamma=0.01, kernel=rbf ....................................\n",
      "[CV] ....... C=10, gamma=0.01, kernel=rbf, score=-0.106, total=   0.2s\n",
      "[CV] C=10, gamma=0.01, kernel=rbf ....................................\n",
      "[CV] ....... C=10, gamma=0.01, kernel=rbf, score=-0.398, total=   0.2s\n",
      "[CV] C=10, gamma=0.01, kernel=rbf ....................................\n",
      "[CV] ....... C=10, gamma=0.01, kernel=rbf, score=-0.340, total=   0.2s\n",
      "[CV] C=10, gamma=0.01, kernel=rbf ....................................\n",
      "[CV] ....... C=10, gamma=0.01, kernel=rbf, score=-0.289, total=   0.2s\n",
      "[CV] C=10, gamma=0.01, kernel=rbf ....................................\n",
      "[CV] ....... C=10, gamma=0.01, kernel=rbf, score=-0.268, total=   0.2s\n",
      "[CV] C=10, gamma=0.01, kernel=rbf ....................................\n",
      "[CV] ....... C=10, gamma=0.01, kernel=rbf, score=-0.286, total=   0.2s\n",
      "[CV] C=10, gamma=0.01, kernel=rbf ....................................\n",
      "[CV] ....... C=10, gamma=0.01, kernel=rbf, score=-0.356, total=   0.2s\n",
      "[CV] C=10, gamma=0.1, kernel=rbf .....................................\n",
      "[CV] ........ C=10, gamma=0.1, kernel=rbf, score=-0.279, total=   0.3s\n",
      "[CV] C=10, gamma=0.1, kernel=rbf .....................................\n",
      "[CV] ........ C=10, gamma=0.1, kernel=rbf, score=-0.190, total=   0.3s\n",
      "[CV] C=10, gamma=0.1, kernel=rbf .....................................\n",
      "[CV] ........ C=10, gamma=0.1, kernel=rbf, score=-0.150, total=   0.4s\n",
      "[CV] C=10, gamma=0.1, kernel=rbf .....................................\n",
      "[CV] ........ C=10, gamma=0.1, kernel=rbf, score=-0.128, total=   0.6s\n",
      "[CV] C=10, gamma=0.1, kernel=rbf .....................................\n",
      "[CV] ........ C=10, gamma=0.1, kernel=rbf, score=-0.168, total=   0.5s\n",
      "[CV] C=10, gamma=0.1, kernel=rbf .....................................\n",
      "[CV] ........ C=10, gamma=0.1, kernel=rbf, score=-0.359, total=   0.3s\n",
      "[CV] C=10, gamma=0.1, kernel=rbf .....................................\n",
      "[CV] ........ C=10, gamma=0.1, kernel=rbf, score=-0.288, total=   0.3s\n",
      "[CV] C=10, gamma=0.1, kernel=rbf .....................................\n",
      "[CV] ........ C=10, gamma=0.1, kernel=rbf, score=-0.263, total=   0.3s\n",
      "[CV] C=10, gamma=0.1, kernel=rbf .....................................\n",
      "[CV] ........ C=10, gamma=0.1, kernel=rbf, score=-0.315, total=   0.3s\n",
      "[CV] C=10, gamma=0.1, kernel=rbf .....................................\n",
      "[CV] ........ C=10, gamma=0.1, kernel=rbf, score=-0.314, total=   0.3s\n",
      "[CV] C=10, gamma=1, kernel=rbf .......................................\n",
      "[CV] .......... C=10, gamma=1, kernel=rbf, score=-0.399, total=   0.6s\n",
      "[CV] C=10, gamma=1, kernel=rbf .......................................\n",
      "[CV] .......... C=10, gamma=1, kernel=rbf, score=-0.354, total=   0.6s\n",
      "[CV] C=10, gamma=1, kernel=rbf .......................................\n",
      "[CV] .......... C=10, gamma=1, kernel=rbf, score=-0.324, total=   0.6s\n",
      "[CV] C=10, gamma=1, kernel=rbf .......................................\n",
      "[CV] .......... C=10, gamma=1, kernel=rbf, score=-0.316, total=   0.7s\n",
      "[CV] C=10, gamma=1, kernel=rbf .......................................\n",
      "[CV] .......... C=10, gamma=1, kernel=rbf, score=-0.341, total=   0.6s\n",
      "[CV] C=10, gamma=1, kernel=rbf .......................................\n",
      "[CV] .......... C=10, gamma=1, kernel=rbf, score=-0.610, total=   0.6s\n",
      "[CV] C=10, gamma=1, kernel=rbf .......................................\n",
      "[CV] .......... C=10, gamma=1, kernel=rbf, score=-0.527, total=   0.6s\n",
      "[CV] C=10, gamma=1, kernel=rbf .......................................\n",
      "[CV] .......... C=10, gamma=1, kernel=rbf, score=-0.515, total=   0.7s\n",
      "[CV] C=10, gamma=1, kernel=rbf .......................................\n",
      "[CV] .......... C=10, gamma=1, kernel=rbf, score=-0.682, total=   0.6s\n",
      "[CV] C=10, gamma=1, kernel=rbf .......................................\n",
      "[CV] .......... C=10, gamma=1, kernel=rbf, score=-0.539, total=   0.6s\n",
      "[CV] C=10, gamma=10, kernel=rbf ......................................\n",
      "[CV] ......... C=10, gamma=10, kernel=rbf, score=-0.693, total=   0.7s\n",
      "[CV] C=10, gamma=10, kernel=rbf ......................................\n",
      "[CV] ......... C=10, gamma=10, kernel=rbf, score=-0.693, total=   0.7s\n",
      "[CV] C=10, gamma=10, kernel=rbf ......................................\n",
      "[CV] ......... C=10, gamma=10, kernel=rbf, score=-0.693, total=   0.7s\n",
      "[CV] C=10, gamma=10, kernel=rbf ......................................\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] ......... C=10, gamma=10, kernel=rbf, score=-0.693, total=   0.8s\n",
      "[CV] C=10, gamma=10, kernel=rbf ......................................\n",
      "[CV] ......... C=10, gamma=10, kernel=rbf, score=-0.693, total=   0.7s\n",
      "[CV] C=10, gamma=10, kernel=rbf ......................................\n",
      "[CV] ......... C=10, gamma=10, kernel=rbf, score=-0.693, total=   0.8s\n",
      "[CV] C=10, gamma=10, kernel=rbf ......................................\n",
      "[CV] ......... C=10, gamma=10, kernel=rbf, score=-0.693, total=   0.7s\n",
      "[CV] C=10, gamma=10, kernel=rbf ......................................\n",
      "[CV] ......... C=10, gamma=10, kernel=rbf, score=-0.693, total=   0.8s\n",
      "[CV] C=10, gamma=10, kernel=rbf ......................................\n",
      "[CV] ......... C=10, gamma=10, kernel=rbf, score=-0.693, total=   0.9s\n",
      "[CV] C=10, gamma=10, kernel=rbf ......................................\n",
      "[CV] ......... C=10, gamma=10, kernel=rbf, score=-0.693, total=   0.8s\n",
      "[CV] C=10, gamma=100, kernel=rbf .....................................\n",
      "[CV] ........ C=10, gamma=100, kernel=rbf, score=-0.693, total=   1.8s\n",
      "[CV] C=10, gamma=100, kernel=rbf .....................................\n",
      "[CV] ........ C=10, gamma=100, kernel=rbf, score=-0.693, total=   1.9s\n",
      "[CV] C=10, gamma=100, kernel=rbf .....................................\n",
      "[CV] ........ C=10, gamma=100, kernel=rbf, score=-0.693, total=   1.5s\n",
      "[CV] C=10, gamma=100, kernel=rbf .....................................\n",
      "[CV] ........ C=10, gamma=100, kernel=rbf, score=-0.693, total=   1.7s\n",
      "[CV] C=10, gamma=100, kernel=rbf .....................................\n",
      "[CV] ........ C=10, gamma=100, kernel=rbf, score=-0.693, total=   1.4s\n",
      "[CV] C=10, gamma=100, kernel=rbf .....................................\n",
      "[CV] ........ C=10, gamma=100, kernel=rbf, score=-0.693, total=   1.3s\n",
      "[CV] C=10, gamma=100, kernel=rbf .....................................\n",
      "[CV] ........ C=10, gamma=100, kernel=rbf, score=-0.693, total=   1.4s\n",
      "[CV] C=10, gamma=100, kernel=rbf .....................................\n",
      "[CV] ........ C=10, gamma=100, kernel=rbf, score=-0.693, total=   1.6s\n",
      "[CV] C=10, gamma=100, kernel=rbf .....................................\n",
      "[CV] ........ C=10, gamma=100, kernel=rbf, score=-0.693, total=   1.4s\n",
      "[CV] C=10, gamma=100, kernel=rbf .....................................\n",
      "[CV] ........ C=10, gamma=100, kernel=rbf, score=-0.693, total=   1.3s\n",
      "[CV] C=100, gamma=0.01, kernel=rbf ...................................\n",
      "[CV] ...... C=100, gamma=0.01, kernel=rbf, score=-0.200, total=   0.4s\n",
      "[CV] C=100, gamma=0.01, kernel=rbf ...................................\n",
      "[CV] ...... C=100, gamma=0.01, kernel=rbf, score=-0.135, total=   0.4s\n",
      "[CV] C=100, gamma=0.01, kernel=rbf ...................................\n",
      "[CV] ...... C=100, gamma=0.01, kernel=rbf, score=-0.146, total=   0.4s\n",
      "[CV] C=100, gamma=0.01, kernel=rbf ...................................\n",
      "[CV] ...... C=100, gamma=0.01, kernel=rbf, score=-0.097, total=   0.4s\n",
      "[CV] C=100, gamma=0.01, kernel=rbf ...................................\n",
      "[CV] ...... C=100, gamma=0.01, kernel=rbf, score=-0.415, total=   0.3s\n",
      "[CV] C=100, gamma=0.01, kernel=rbf ...................................\n",
      "[CV] ...... C=100, gamma=0.01, kernel=rbf, score=-0.305, total=   0.3s\n",
      "[CV] C=100, gamma=0.01, kernel=rbf ...................................\n",
      "[CV] ...... C=100, gamma=0.01, kernel=rbf, score=-0.272, total=   0.3s\n",
      "[CV] C=100, gamma=0.01, kernel=rbf ...................................\n",
      "[CV] ...... C=100, gamma=0.01, kernel=rbf, score=-0.223, total=   0.3s\n",
      "[CV] C=100, gamma=0.01, kernel=rbf ...................................\n",
      "[CV] ...... C=100, gamma=0.01, kernel=rbf, score=-0.286, total=   0.3s\n",
      "[CV] C=100, gamma=0.01, kernel=rbf ...................................\n",
      "[CV] ...... C=100, gamma=0.01, kernel=rbf, score=-0.326, total=   0.3s\n",
      "[CV] C=100, gamma=0.1, kernel=rbf ....................................\n",
      "[CV] ....... C=100, gamma=0.1, kernel=rbf, score=-0.288, total=   0.3s\n",
      "[CV] C=100, gamma=0.1, kernel=rbf ....................................\n",
      "[CV] ....... C=100, gamma=0.1, kernel=rbf, score=-0.253, total=   0.3s\n",
      "[CV] C=100, gamma=0.1, kernel=rbf ....................................\n",
      "[CV] ....... C=100, gamma=0.1, kernel=rbf, score=-0.193, total=   0.3s\n",
      "[CV] C=100, gamma=0.1, kernel=rbf ....................................\n",
      "[CV] ....... C=100, gamma=0.1, kernel=rbf, score=-0.192, total=   0.4s\n",
      "[CV] C=100, gamma=0.1, kernel=rbf ....................................\n",
      "[CV] ....... C=100, gamma=0.1, kernel=rbf, score=-0.237, total=   0.4s\n",
      "[CV] C=100, gamma=0.1, kernel=rbf ....................................\n",
      "[CV] ....... C=100, gamma=0.1, kernel=rbf, score=-0.397, total=   0.3s\n",
      "[CV] C=100, gamma=0.1, kernel=rbf ....................................\n",
      "[CV] ....... C=100, gamma=0.1, kernel=rbf, score=-0.302, total=   0.3s\n",
      "[CV] C=100, gamma=0.1, kernel=rbf ....................................\n",
      "[CV] ....... C=100, gamma=0.1, kernel=rbf, score=-0.334, total=   0.3s\n",
      "[CV] C=100, gamma=0.1, kernel=rbf ....................................\n",
      "[CV] ....... C=100, gamma=0.1, kernel=rbf, score=-0.360, total=   0.3s\n",
      "[CV] C=100, gamma=0.1, kernel=rbf ....................................\n",
      "[CV] ....... C=100, gamma=0.1, kernel=rbf, score=-0.315, total=   0.3s\n",
      "[CV] C=100, gamma=1, kernel=rbf ......................................\n",
      "[CV] ......... C=100, gamma=1, kernel=rbf, score=-0.400, total=   0.6s\n",
      "[CV] C=100, gamma=1, kernel=rbf ......................................\n",
      "[CV] ......... C=100, gamma=1, kernel=rbf, score=-0.348, total=   0.6s\n",
      "[CV] C=100, gamma=1, kernel=rbf ......................................\n",
      "[CV] ......... C=100, gamma=1, kernel=rbf, score=-0.318, total=   0.7s\n",
      "[CV] C=100, gamma=1, kernel=rbf ......................................\n",
      "[CV] ......... C=100, gamma=1, kernel=rbf, score=-0.317, total=   0.7s\n",
      "[CV] C=100, gamma=1, kernel=rbf ......................................\n",
      "[CV] ......... C=100, gamma=1, kernel=rbf, score=-0.344, total=   0.6s\n",
      "[CV] C=100, gamma=1, kernel=rbf ......................................\n",
      "[CV] ......... C=100, gamma=1, kernel=rbf, score=-0.607, total=   0.6s\n",
      "[CV] C=100, gamma=1, kernel=rbf ......................................\n",
      "[CV] ......... C=100, gamma=1, kernel=rbf, score=-0.525, total=   0.6s\n",
      "[CV] C=100, gamma=1, kernel=rbf ......................................\n",
      "[CV] ......... C=100, gamma=1, kernel=rbf, score=-0.517, total=   0.6s\n",
      "[CV] C=100, gamma=1, kernel=rbf ......................................\n",
      "[CV] ......... C=100, gamma=1, kernel=rbf, score=-0.701, total=   0.6s\n",
      "[CV] C=100, gamma=1, kernel=rbf ......................................\n",
      "[CV] ......... C=100, gamma=1, kernel=rbf, score=-0.542, total=   0.6s\n",
      "[CV] C=100, gamma=10, kernel=rbf .....................................\n",
      "[CV] ........ C=100, gamma=10, kernel=rbf, score=-0.693, total=   0.7s\n",
      "[CV] C=100, gamma=10, kernel=rbf .....................................\n",
      "[CV] ........ C=100, gamma=10, kernel=rbf, score=-0.693, total=   0.7s\n",
      "[CV] C=100, gamma=10, kernel=rbf .....................................\n",
      "[CV] ........ C=100, gamma=10, kernel=rbf, score=-0.693, total=   0.7s\n",
      "[CV] C=100, gamma=10, kernel=rbf .....................................\n",
      "[CV] ........ C=100, gamma=10, kernel=rbf, score=-0.693, total=   0.7s\n",
      "[CV] C=100, gamma=10, kernel=rbf .....................................\n",
      "[CV] ........ C=100, gamma=10, kernel=rbf, score=-0.693, total=   0.7s\n",
      "[CV] C=100, gamma=10, kernel=rbf .....................................\n",
      "[CV] ........ C=100, gamma=10, kernel=rbf, score=-0.693, total=   0.7s\n",
      "[CV] C=100, gamma=10, kernel=rbf .....................................\n",
      "[CV] ........ C=100, gamma=10, kernel=rbf, score=-0.693, total=   0.7s\n",
      "[CV] C=100, gamma=10, kernel=rbf .....................................\n",
      "[CV] ........ C=100, gamma=10, kernel=rbf, score=-0.693, total=   0.7s\n",
      "[CV] C=100, gamma=10, kernel=rbf .....................................\n",
      "[CV] ........ C=100, gamma=10, kernel=rbf, score=-0.693, total=   0.7s\n",
      "[CV] C=100, gamma=10, kernel=rbf .....................................\n",
      "[CV] ........ C=100, gamma=10, kernel=rbf, score=-0.693, total=   0.6s\n",
      "[CV] C=100, gamma=100, kernel=rbf ....................................\n",
      "[CV] ....... C=100, gamma=100, kernel=rbf, score=-0.693, total=   1.4s\n",
      "[CV] C=100, gamma=100, kernel=rbf ....................................\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] ....... C=100, gamma=100, kernel=rbf, score=-0.693, total=   1.4s\n",
      "[CV] C=100, gamma=100, kernel=rbf ....................................\n",
      "[CV] ....... C=100, gamma=100, kernel=rbf, score=-0.693, total=   1.3s\n",
      "[CV] C=100, gamma=100, kernel=rbf ....................................\n",
      "[CV] ....... C=100, gamma=100, kernel=rbf, score=-0.693, total=   1.3s\n",
      "[CV] C=100, gamma=100, kernel=rbf ....................................\n",
      "[CV] ....... C=100, gamma=100, kernel=rbf, score=-0.693, total=   1.6s\n",
      "[CV] C=100, gamma=100, kernel=rbf ....................................\n",
      "[CV] ....... C=100, gamma=100, kernel=rbf, score=-0.693, total=   1.4s\n",
      "[CV] C=100, gamma=100, kernel=rbf ....................................\n",
      "[CV] ....... C=100, gamma=100, kernel=rbf, score=-0.693, total=   1.3s\n",
      "[CV] C=100, gamma=100, kernel=rbf ....................................\n",
      "[CV] ....... C=100, gamma=100, kernel=rbf, score=-0.693, total=   1.3s\n",
      "[CV] C=100, gamma=100, kernel=rbf ....................................\n",
      "[CV] ....... C=100, gamma=100, kernel=rbf, score=-0.693, total=   1.4s\n",
      "[CV] C=100, gamma=100, kernel=rbf ....................................\n",
      "[CV] ....... C=100, gamma=100, kernel=rbf, score=-0.693, total=   1.3s\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Done 250 out of 250 | elapsed:  2.8min finished\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "GridSearchCV(cv=10, estimator=SVC(probability=True),\n",
       "             param_grid={'C': [0.01, 0.1, 1, 10, 100],\n",
       "                         'gamma': [0.01, 0.1, 1, 10, 100], 'kernel': ['rbf']},\n",
       "             scoring='neg_log_loss', verbose=10)"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.model_selection import GridSearchCV, RandomizedSearchCV\n",
    "from sklearn.svm import SVC\n",
    "\n",
    "n_splits = 10\n",
    "\n",
    "param_grid = {'kernel':['rbf'], #redundant; default is already 'rbf'\n",
    "              'C':[1e-2, 1e-1, 1, 10, 100],\n",
    "              'gamma': [1e-2, 1e-1, 1, 10, 100]}\n",
    "\n",
    "clf_svm = SVC(kernel='rbf',\n",
    "              probability=True)\n",
    "\n",
    "clf = GridSearchCV(estimator = clf_svm, \n",
    "                   param_grid = param_grid,\n",
    "                   scoring='neg_log_loss', \n",
    "                   n_jobs=None, \n",
    "                   #iid='deprecated', take note of depreciation! version 0.22 warning\n",
    "                   refit=True,\n",
    "                   cv= n_splits, # 10 fold CV, but! stratified KFold\n",
    "                   verbose=10)\n",
    "\n",
    "clf.fit(X,\n",
    "        y['bin'],\n",
    "       sample_weight = None) #sample_weight here!!\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note 1**\n",
    "\n",
    "I allowed verbose = 10, so that we can see the output messages. Based on my due diligence and educated guess from these code snippets in the book, I believe Dr Marco was running sklearn version 0.14 - 0.16.\n",
    "\n",
    "The only thing I found online for sklearn node_count was for decision tree, hence the answer provided is experimental (I think that was what Dr Marco wanted?).\n",
    "\n",
    "[sklearn SVC](https://scikit-learn.org/stable/modules/generated/sklearn.svm.SVC.html?highlight=svc#sklearn.svm.SVC)\n",
    "\n",
    "[sklearn GridSearchCV](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html?highlight=gridsearchcv#sklearn.model_selection.GridSearchCV)\n",
    "\n",
    "[sklearn hyper-parameter tuning](https://scikit-learn.org/stable/modules/grid_search.html#gridsearch-scoring)\n",
    "\n",
    "The upside of using sklearn 0.23.1, most of the function found in the book can be found in their current library.\n",
    "\n",
    "* uniform log\n",
    "* sample_weight can be passed to fit method\n",
    "\n",
    "**Note 2**\n",
    "\n",
    "As for how I derived combination of param_grids as \"nodes\" in the below answers. Kindly refer to [ML Mastery](https://machinelearningmastery.com/how-to-configure-the-number-of-layers-and-nodes-in-a-neural-network/#:~:text=A%20node%2C%20also%20called%20a,layers%20to%20comprise%20a%20network.)\n",
    "\n",
    "The below is a diagram which I found from google.\n",
    "\n",
    "![logo](https://pathmind.com/images/wiki/perceptron_node.png)\n",
    "\n",
    "Although this is under Deep-ML, but given the time of writing we will have to perform an educated guess to match the author's intention as well as terms mentioned in the book (Match functionality to form connections).\n",
    "\n",
    "Essentially, nodes are perceptors (Deep-Learning) which are **inputs**. In our case, Grid Search (GS) inputs are param_grids.\n",
    "\n",
    "Hence, I believe this assumption does make sense in this GS exercise's context.\n",
    "\n",
    "**Note 3**\n",
    "\n",
    "At the same time, The results I placed below is mostly average/ mean. The reality is after splitting. Each node/ candidate were ran once on each sample.\n",
    "\n",
    "In this example, I only generated 2,000 samples."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best params for estimator: SVC(C=1, gamma=0.1, probability=True)\n",
      "Best CV Score: -0.098618\n",
      "\n",
      "A total of 87 was performed before optimal CV score found! (Under split3_test_score / 4th split)\n",
      "\n",
      "Estimated time taken for entire process: 74.261129 seconds\n",
      "Total number of candidates/ nodes: 25\n",
      "\n",
      "=======================================================\n",
      "Estimated total mean time required for optimal solution:\n",
      "\n",
      "Score Time: 0.160910s\n",
      "Fit Time: 6.813837s\n"
     ]
    }
   ],
   "source": [
    "def print_result(clf, n_splits):\n",
    "    num_nodes = len(clf.cv_results_['rank_test_score'])\n",
    "    total_time = clf.refit_time_ * num_nodes * n_splits\n",
    "\n",
    "    best_scores, best_scores_idx, mean_score_time, mean_fit_time = [], 0, .0, .0\n",
    "\n",
    "    for i in np.arange(n_splits):\n",
    "        best_scores.append(clf.cv_results_[\"split\"+ str(i) +\"_test_score\"][clf.best_index_])\n",
    "        idx = np.where(max(best_scores) == best_scores)[0][0] #always + 1 because index starts from 0 as default\n",
    "\n",
    "    best_scores_idx = (clf.best_index_ + 1) + len(clf.cv_results_['mean_score_time']) * idx\n",
    "    print(\"Best params for estimator: {0}\\nBest CV Score: {1:.6f}\\n\".format(clf.best_estimator_, max(best_scores)))\n",
    "    print(\"A total of {0} was performed before optimal CV score found! (Under split{1}_test_score / {2}th split)\\n\".format(best_scores_idx, idx, idx + 1))\n",
    "\n",
    "    print(\"Estimated time taken for entire process: {0:.6f} seconds\\nTotal number of candidates/ nodes: {1}\\n\".format(total_time, \n",
    "                                                                                                           num_nodes))\n",
    "    i = 0\n",
    "    while i < (clf.best_index_ + 1):\n",
    "        mean_score_time += clf.cv_results_['mean_score_time'][i]\n",
    "        mean_fit_time += clf.cv_results_['mean_fit_time'][i]\n",
    "        i+=1\n",
    "    print(\"=\" * 55)\n",
    "    print(\"Estimated total mean time required for optimal solution:\\n\\nScore Time: {0:.6f}s\\nFit Time: {1:.6f}s\".format(mean_score_time, mean_fit_time))\n",
    "\n",
    "print_result(clf = clf, n_splits = n_splits)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 10 folds for each of 25 candidates, totalling 250 fits\n",
      "[CV] C=0.7833260760128987, gamma=0.2994822977403476 ..................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Using backend SequentialBackend with 1 concurrent workers.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV]  C=0.7833260760128987, gamma=0.2994822977403476, score=-0.271, total=   0.5s\n",
      "[CV] C=0.7833260760128987, gamma=0.2994822977403476 ..................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Done   1 out of   1 | elapsed:    0.4s remaining:    0.0s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV]  C=0.7833260760128987, gamma=0.2994822977403476, score=-0.158, total=   0.5s\n",
      "[CV] C=0.7833260760128987, gamma=0.2994822977403476 ..................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Done   2 out of   2 | elapsed:    0.9s remaining:    0.0s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV]  C=0.7833260760128987, gamma=0.2994822977403476, score=-0.132, total=   0.5s\n",
      "[CV] C=0.7833260760128987, gamma=0.2994822977403476 ..................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Done   3 out of   3 | elapsed:    1.4s remaining:    0.0s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV]  C=0.7833260760128987, gamma=0.2994822977403476, score=-0.114, total=   0.5s\n",
      "[CV] C=0.7833260760128987, gamma=0.2994822977403476 ..................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Done   4 out of   4 | elapsed:    1.9s remaining:    0.0s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV]  C=0.7833260760128987, gamma=0.2994822977403476, score=-0.163, total=   0.5s\n",
      "[CV] C=0.7833260760128987, gamma=0.2994822977403476 ..................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Done   5 out of   5 | elapsed:    2.4s remaining:    0.0s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV]  C=0.7833260760128987, gamma=0.2994822977403476, score=-0.390, total=   0.5s\n",
      "[CV] C=0.7833260760128987, gamma=0.2994822977403476 ..................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Done   6 out of   6 | elapsed:    2.9s remaining:    0.0s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV]  C=0.7833260760128987, gamma=0.2994822977403476, score=-0.378, total=   0.5s\n",
      "[CV] C=0.7833260760128987, gamma=0.2994822977403476 ..................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Done   7 out of   7 | elapsed:    3.4s remaining:    0.0s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV]  C=0.7833260760128987, gamma=0.2994822977403476, score=-0.289, total=   0.5s\n",
      "[CV] C=0.7833260760128987, gamma=0.2994822977403476 ..................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Done   8 out of   8 | elapsed:    3.9s remaining:    0.0s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV]  C=0.7833260760128987, gamma=0.2994822977403476, score=-0.377, total=   0.5s\n",
      "[CV] C=0.7833260760128987, gamma=0.2994822977403476 ..................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Done   9 out of   9 | elapsed:    4.4s remaining:    0.0s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV]  C=0.7833260760128987, gamma=0.2994822977403476, score=-0.373, total=   0.5s\n",
      "[CV] C=0.49037818303803193, gamma=8.536817927058488 ..................\n",
      "[CV]  C=0.49037818303803193, gamma=8.536817927058488, score=-0.693, total=   0.6s\n",
      "[CV] C=0.49037818303803193, gamma=8.536817927058488 ..................\n",
      "[CV]  C=0.49037818303803193, gamma=8.536817927058488, score=-0.693, total=   0.6s\n",
      "[CV] C=0.49037818303803193, gamma=8.536817927058488 ..................\n",
      "[CV]  C=0.49037818303803193, gamma=8.536817927058488, score=-0.693, total=   0.7s\n",
      "[CV] C=0.49037818303803193, gamma=8.536817927058488 ..................\n",
      "[CV]  C=0.49037818303803193, gamma=8.536817927058488, score=-0.693, total=   0.7s\n",
      "[CV] C=0.49037818303803193, gamma=8.536817927058488 ..................\n",
      "[CV]  C=0.49037818303803193, gamma=8.536817927058488, score=-0.693, total=   0.8s\n",
      "[CV] C=0.49037818303803193, gamma=8.536817927058488 ..................\n",
      "[CV]  C=0.49037818303803193, gamma=8.536817927058488, score=-0.694, total=   0.6s\n",
      "[CV] C=0.49037818303803193, gamma=8.536817927058488 ..................\n",
      "[CV]  C=0.49037818303803193, gamma=8.536817927058488, score=-0.693, total=   0.6s\n",
      "[CV] C=0.49037818303803193, gamma=8.536817927058488 ..................\n",
      "[CV]  C=0.49037818303803193, gamma=8.536817927058488, score=-0.694, total=   0.6s\n",
      "[CV] C=0.49037818303803193, gamma=8.536817927058488 ..................\n",
      "[CV]  C=0.49037818303803193, gamma=8.536817927058488, score=-0.694, total=   0.6s\n",
      "[CV] C=0.49037818303803193, gamma=8.536817927058488 ..................\n",
      "[CV]  C=0.49037818303803193, gamma=8.536817927058488, score=-0.694, total=   0.6s\n",
      "[CV] C=17.197180707859385, gamma=0.036133428302774155 ................\n",
      "[CV]  C=17.197180707859385, gamma=0.036133428302774155, score=-0.221, total=   0.2s\n",
      "[CV] C=17.197180707859385, gamma=0.036133428302774155 ................\n",
      "[CV]  C=17.197180707859385, gamma=0.036133428302774155, score=-0.132, total=   0.2s\n",
      "[CV] C=17.197180707859385, gamma=0.036133428302774155 ................\n",
      "[CV]  C=17.197180707859385, gamma=0.036133428302774155, score=-0.124, total=   0.3s\n",
      "[CV] C=17.197180707859385, gamma=0.036133428302774155 ................\n",
      "[CV]  C=17.197180707859385, gamma=0.036133428302774155, score=-0.115, total=   0.3s\n",
      "[CV] C=17.197180707859385, gamma=0.036133428302774155 ................\n",
      "[CV]  C=17.197180707859385, gamma=0.036133428302774155, score=-0.261, total=   0.2s\n",
      "[CV] C=17.197180707859385, gamma=0.036133428302774155 ................\n",
      "[CV]  C=17.197180707859385, gamma=0.036133428302774155, score=-0.316, total=   0.2s\n",
      "[CV] C=17.197180707859385, gamma=0.036133428302774155 ................\n",
      "[CV]  C=17.197180707859385, gamma=0.036133428302774155, score=-0.274, total=   0.2s\n",
      "[CV] C=17.197180707859385, gamma=0.036133428302774155 ................\n",
      "[CV]  C=17.197180707859385, gamma=0.036133428302774155, score=-0.224, total=   0.2s\n",
      "[CV] C=17.197180707859385, gamma=0.036133428302774155 ................\n",
      "[CV]  C=17.197180707859385, gamma=0.036133428302774155, score=-0.272, total=   0.2s\n",
      "[CV] C=17.197180707859385, gamma=0.036133428302774155 ................\n",
      "[CV]  C=17.197180707859385, gamma=0.036133428302774155, score=-0.326, total=   0.2s\n",
      "[CV] C=7.942556107190451, gamma=2.264226103633606 ....................\n",
      "[CV]  C=7.942556107190451, gamma=2.264226103633606, score=-0.644, total=   0.6s\n",
      "[CV] C=7.942556107190451, gamma=2.264226103633606 ....................\n",
      "[CV]  C=7.942556107190451, gamma=2.264226103633606, score=-0.672, total=   0.6s\n",
      "[CV] C=7.942556107190451, gamma=2.264226103633606 ....................\n",
      "[CV]  C=7.942556107190451, gamma=2.264226103633606, score=-0.662, total=   0.6s\n",
      "[CV] C=7.942556107190451, gamma=2.264226103633606 ....................\n",
      "[CV]  C=7.942556107190451, gamma=2.264226103633606, score=-0.655, total=   0.6s\n",
      "[CV] C=7.942556107190451, gamma=2.264226103633606 ....................\n",
      "[CV]  C=7.942556107190451, gamma=2.264226103633606, score=-0.644, total=   0.7s\n",
      "[CV] C=7.942556107190451, gamma=2.264226103633606 ....................\n",
      "[CV]  C=7.942556107190451, gamma=2.264226103633606, score=-0.666, total=   0.7s\n",
      "[CV] C=7.942556107190451, gamma=2.264226103633606 ....................\n",
      "[CV]  C=7.942556107190451, gamma=2.264226103633606, score=-0.644, total=   0.7s\n",
      "[CV] C=7.942556107190451, gamma=2.264226103633606 ....................\n",
      "[CV]  C=7.942556107190451, gamma=2.264226103633606, score=-0.669, total=   0.9s\n",
      "[CV] C=7.942556107190451, gamma=2.264226103633606 ....................\n",
      "[CV]  C=7.942556107190451, gamma=2.264226103633606, score=-0.671, total=   0.7s\n",
      "[CV] C=7.942556107190451, gamma=2.264226103633606 ....................\n",
      "[CV]  C=7.942556107190451, gamma=2.264226103633606, score=-0.670, total=   0.7s\n",
      "[CV] C=84.23648703125728, gamma=0.010677557703921118 .................\n",
      "[CV]  C=84.23648703125728, gamma=0.010677557703921118, score=-0.202, total=   0.3s\n",
      "[CV] C=84.23648703125728, gamma=0.010677557703921118 .................\n",
      "[CV]  C=84.23648703125728, gamma=0.010677557703921118, score=-0.134, total=   0.3s\n",
      "[CV] C=84.23648703125728, gamma=0.010677557703921118 .................\n",
      "[CV]  C=84.23648703125728, gamma=0.010677557703921118, score=-0.146, total=   0.3s\n",
      "[CV] C=84.23648703125728, gamma=0.010677557703921118 .................\n",
      "[CV]  C=84.23648703125728, gamma=0.010677557703921118, score=-0.098, total=   0.4s\n",
      "[CV] C=84.23648703125728, gamma=0.010677557703921118 .................\n",
      "[CV]  C=84.23648703125728, gamma=0.010677557703921118, score=-0.410, total=   0.3s\n",
      "[CV] C=84.23648703125728, gamma=0.010677557703921118 .................\n",
      "[CV]  C=84.23648703125728, gamma=0.010677557703921118, score=-0.306, total=   0.3s\n",
      "[CV] C=84.23648703125728, gamma=0.010677557703921118 .................\n",
      "[CV]  C=84.23648703125728, gamma=0.010677557703921118, score=-0.274, total=   0.3s\n",
      "[CV] C=84.23648703125728, gamma=0.010677557703921118 .................\n",
      "[CV]  C=84.23648703125728, gamma=0.010677557703921118, score=-0.224, total=   0.3s\n",
      "[CV] C=84.23648703125728, gamma=0.010677557703921118 .................\n",
      "[CV]  C=84.23648703125728, gamma=0.010677557703921118, score=-0.286, total=   0.4s\n",
      "[CV] C=84.23648703125728, gamma=0.010677557703921118 .................\n",
      "[CV]  C=84.23648703125728, gamma=0.010677557703921118, score=-0.325, total=   0.4s\n",
      "[CV] C=73.75910503732396, gamma=65.74587879428422 ....................\n",
      "[CV]  C=73.75910503732396, gamma=65.74587879428422, score=-0.693, total=   1.6s\n",
      "[CV] C=73.75910503732396, gamma=65.74587879428422 ....................\n",
      "[CV]  C=73.75910503732396, gamma=65.74587879428422, score=-0.693, total=   1.3s\n",
      "[CV] C=73.75910503732396, gamma=65.74587879428422 ....................\n",
      "[CV]  C=73.75910503732396, gamma=65.74587879428422, score=-0.693, total=   1.4s\n",
      "[CV] C=73.75910503732396, gamma=65.74587879428422 ....................\n",
      "[CV]  C=73.75910503732396, gamma=65.74587879428422, score=-0.693, total=   1.4s\n",
      "[CV] C=73.75910503732396, gamma=65.74587879428422 ....................\n",
      "[CV]  C=73.75910503732396, gamma=65.74587879428422, score=-0.693, total=   1.4s\n",
      "[CV] C=73.75910503732396, gamma=65.74587879428422 ....................\n",
      "[CV]  C=73.75910503732396, gamma=65.74587879428422, score=-0.693, total=   1.3s\n",
      "[CV] C=73.75910503732396, gamma=65.74587879428422 ....................\n",
      "[CV]  C=73.75910503732396, gamma=65.74587879428422, score=-0.693, total=   1.3s\n",
      "[CV] C=73.75910503732396, gamma=65.74587879428422 ....................\n",
      "[CV]  C=73.75910503732396, gamma=65.74587879428422, score=-0.693, total=   1.4s\n",
      "[CV] C=73.75910503732396, gamma=65.74587879428422 ....................\n",
      "[CV]  C=73.75910503732396, gamma=65.74587879428422, score=-0.693, total=   1.3s\n",
      "[CV] C=73.75910503732396, gamma=65.74587879428422 ....................\n",
      "[CV]  C=73.75910503732396, gamma=65.74587879428422, score=-0.693, total=   1.4s\n",
      "[CV] C=35.48667662529329, gamma=4.403630972109106 ....................\n",
      "[CV]  C=35.48667662529329, gamma=4.403630972109106, score=-0.693, total=   0.7s\n",
      "[CV] C=35.48667662529329, gamma=4.403630972109106 ....................\n",
      "[CV]  C=35.48667662529329, gamma=4.403630972109106, score=-0.692, total=   0.7s\n",
      "[CV] C=35.48667662529329, gamma=4.403630972109106 ....................\n",
      "[CV]  C=35.48667662529329, gamma=4.403630972109106, score=-0.694, total=   0.6s\n",
      "[CV] C=35.48667662529329, gamma=4.403630972109106 ....................\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV]  C=35.48667662529329, gamma=4.403630972109106, score=-0.694, total=   0.7s\n",
      "[CV] C=35.48667662529329, gamma=4.403630972109106 ....................\n",
      "[CV]  C=35.48667662529329, gamma=4.403630972109106, score=-0.693, total=   0.6s\n",
      "[CV] C=35.48667662529329, gamma=4.403630972109106 ....................\n",
      "[CV]  C=35.48667662529329, gamma=4.403630972109106, score=-0.693, total=   0.6s\n",
      "[CV] C=35.48667662529329, gamma=4.403630972109106 ....................\n",
      "[CV]  C=35.48667662529329, gamma=4.403630972109106, score=-0.693, total=   0.6s\n",
      "[CV] C=35.48667662529329, gamma=4.403630972109106 ....................\n",
      "[CV]  C=35.48667662529329, gamma=4.403630972109106, score=-0.693, total=   0.6s\n",
      "[CV] C=35.48667662529329, gamma=4.403630972109106 ....................\n",
      "[CV]  C=35.48667662529329, gamma=4.403630972109106, score=-0.693, total=   0.7s\n",
      "[CV] C=35.48667662529329, gamma=4.403630972109106 ....................\n",
      "[CV]  C=35.48667662529329, gamma=4.403630972109106, score=-0.693, total=   0.6s\n",
      "[CV] C=5.978726724902645, gamma=0.019090360658614053 .................\n",
      "[CV]  C=5.978726724902645, gamma=0.019090360658614053, score=-0.201, total=   0.3s\n",
      "[CV] C=5.978726724902645, gamma=0.019090360658614053 .................\n",
      "[CV]  C=5.978726724902645, gamma=0.019090360658614053, score=-0.130, total=   0.2s\n",
      "[CV] C=5.978726724902645, gamma=0.019090360658614053 .................\n",
      "[CV]  C=5.978726724902645, gamma=0.019090360658614053, score=-0.136, total=   0.2s\n",
      "[CV] C=5.978726724902645, gamma=0.019090360658614053 .................\n",
      "[CV]  C=5.978726724902645, gamma=0.019090360658614053, score=-0.096, total=   0.2s\n",
      "[CV] C=5.978726724902645, gamma=0.019090360658614053 .................\n",
      "[CV]  C=5.978726724902645, gamma=0.019090360658614053, score=-0.366, total=   0.2s\n",
      "[CV] C=5.978726724902645, gamma=0.019090360658614053 .................\n",
      "[CV]  C=5.978726724902645, gamma=0.019090360658614053, score=-0.316, total=   0.2s\n",
      "[CV] C=5.978726724902645, gamma=0.019090360658614053 .................\n",
      "[CV]  C=5.978726724902645, gamma=0.019090360658614053, score=-0.282, total=   0.2s\n",
      "[CV] C=5.978726724902645, gamma=0.019090360658614053 .................\n",
      "[CV]  C=5.978726724902645, gamma=0.019090360658614053, score=-0.230, total=   0.2s\n",
      "[CV] C=5.978726724902645, gamma=0.019090360658614053 .................\n",
      "[CV]  C=5.978726724902645, gamma=0.019090360658614053, score=-0.276, total=   0.2s\n",
      "[CV] C=5.978726724902645, gamma=0.019090360658614053 .................\n",
      "[CV]  C=5.978726724902645, gamma=0.019090360658614053, score=-0.347, total=   0.2s\n",
      "[CV] C=1.8239315886743146, gamma=0.16735997421143464 .................\n",
      "[CV]  C=1.8239315886743146, gamma=0.16735997421143464, score=-0.256, total=   0.3s\n",
      "[CV] C=1.8239315886743146, gamma=0.16735997421143464 .................\n",
      "[CV]  C=1.8239315886743146, gamma=0.16735997421143464, score=-0.145, total=   0.3s\n",
      "[CV] C=1.8239315886743146, gamma=0.16735997421143464 .................\n",
      "[CV]  C=1.8239315886743146, gamma=0.16735997421143464, score=-0.117, total=   0.3s\n",
      "[CV] C=1.8239315886743146, gamma=0.16735997421143464 .................\n",
      "[CV]  C=1.8239315886743146, gamma=0.16735997421143464, score=-0.110, total=   0.3s\n",
      "[CV] C=1.8239315886743146, gamma=0.16735997421143464 .................\n",
      "[CV]  C=1.8239315886743146, gamma=0.16735997421143464, score=-0.155, total=   0.3s\n",
      "[CV] C=1.8239315886743146, gamma=0.16735997421143464 .................\n",
      "[CV]  C=1.8239315886743146, gamma=0.16735997421143464, score=-0.359, total=   0.3s\n",
      "[CV] C=1.8239315886743146, gamma=0.16735997421143464 .................\n",
      "[CV]  C=1.8239315886743146, gamma=0.16735997421143464, score=-0.323, total=   0.3s\n",
      "[CV] C=1.8239315886743146, gamma=0.16735997421143464 .................\n",
      "[CV]  C=1.8239315886743146, gamma=0.16735997421143464, score=-0.242, total=   0.3s\n",
      "[CV] C=1.8239315886743146, gamma=0.16735997421143464 .................\n",
      "[CV]  C=1.8239315886743146, gamma=0.16735997421143464, score=-0.326, total=   0.3s\n",
      "[CV] C=1.8239315886743146, gamma=0.16735997421143464 .................\n",
      "[CV]  C=1.8239315886743146, gamma=0.16735997421143464, score=-0.347, total=   0.4s\n",
      "[CV] C=1.308709482507712, gamma=0.06937936028678086 ..................\n",
      "[CV]  C=1.308709482507712, gamma=0.06937936028678086, score=-0.215, total=   0.2s\n",
      "[CV] C=1.308709482507712, gamma=0.06937936028678086 ..................\n",
      "[CV]  C=1.308709482507712, gamma=0.06937936028678086, score=-0.113, total=   0.2s\n",
      "[CV] C=1.308709482507712, gamma=0.06937936028678086 ..................\n",
      "[CV]  C=1.308709482507712, gamma=0.06937936028678086, score=-0.119, total=   0.2s\n",
      "[CV] C=1.308709482507712, gamma=0.06937936028678086 ..................\n",
      "[CV]  C=1.308709482507712, gamma=0.06937936028678086, score=-0.095, total=   0.2s\n",
      "[CV] C=1.308709482507712, gamma=0.06937936028678086 ..................\n",
      "[CV]  C=1.308709482507712, gamma=0.06937936028678086, score=-0.183, total=   0.2s\n",
      "[CV] C=1.308709482507712, gamma=0.06937936028678086 ..................\n",
      "[CV]  C=1.308709482507712, gamma=0.06937936028678086, score=-0.322, total=   0.2s\n",
      "[CV] C=1.308709482507712, gamma=0.06937936028678086 ..................\n",
      "[CV]  C=1.308709482507712, gamma=0.06937936028678086, score=-0.291, total=   0.2s\n",
      "[CV] C=1.308709482507712, gamma=0.06937936028678086 ..................\n",
      "[CV]  C=1.308709482507712, gamma=0.06937936028678086, score=-0.216, total=   0.2s\n",
      "[CV] C=1.308709482507712, gamma=0.06937936028678086 ..................\n",
      "[CV]  C=1.308709482507712, gamma=0.06937936028678086, score=-0.276, total=   0.2s\n",
      "[CV] C=1.308709482507712, gamma=0.06937936028678086 ..................\n",
      "[CV]  C=1.308709482507712, gamma=0.06937936028678086, score=-0.356, total=   0.2s\n",
      "[CV] C=0.04576291319920591, gamma=3.5828831431903536 .................\n",
      "[CV]  C=0.04576291319920591, gamma=3.5828831431903536, score=-0.693, total=   0.6s\n",
      "[CV] C=0.04576291319920591, gamma=3.5828831431903536 .................\n",
      "[CV]  C=0.04576291319920591, gamma=3.5828831431903536, score=-0.693, total=   0.6s\n",
      "[CV] C=0.04576291319920591, gamma=3.5828831431903536 .................\n",
      "[CV]  C=0.04576291319920591, gamma=3.5828831431903536, score=-0.693, total=   0.6s\n",
      "[CV] C=0.04576291319920591, gamma=3.5828831431903536 .................\n",
      "[CV]  C=0.04576291319920591, gamma=3.5828831431903536, score=-0.694, total=   0.6s\n",
      "[CV] C=0.04576291319920591, gamma=3.5828831431903536 .................\n",
      "[CV]  C=0.04576291319920591, gamma=3.5828831431903536, score=-0.693, total=   0.6s\n",
      "[CV] C=0.04576291319920591, gamma=3.5828831431903536 .................\n",
      "[CV]  C=0.04576291319920591, gamma=3.5828831431903536, score=-0.693, total=   0.6s\n",
      "[CV] C=0.04576291319920591, gamma=3.5828831431903536 .................\n",
      "[CV]  C=0.04576291319920591, gamma=3.5828831431903536, score=-0.694, total=   0.6s\n",
      "[CV] C=0.04576291319920591, gamma=3.5828831431903536 .................\n",
      "[CV]  C=0.04576291319920591, gamma=3.5828831431903536, score=-0.693, total=   0.6s\n",
      "[CV] C=0.04576291319920591, gamma=3.5828831431903536 .................\n",
      "[CV]  C=0.04576291319920591, gamma=3.5828831431903536, score=-0.694, total=   0.6s\n",
      "[CV] C=0.04576291319920591, gamma=3.5828831431903536 .................\n",
      "[CV]  C=0.04576291319920591, gamma=3.5828831431903536, score=-0.693, total=   0.6s\n",
      "[CV] C=0.029600079668876515, gamma=0.6614167167922481 ................\n",
      "[CV]  C=0.029600079668876515, gamma=0.6614167167922481, score=-0.694, total=   0.5s\n",
      "[CV] C=0.029600079668876515, gamma=0.6614167167922481 ................\n",
      "[CV]  C=0.029600079668876515, gamma=0.6614167167922481, score=-0.693, total=   0.5s\n",
      "[CV] C=0.029600079668876515, gamma=0.6614167167922481 ................\n",
      "[CV]  C=0.029600079668876515, gamma=0.6614167167922481, score=-0.694, total=   0.5s\n",
      "[CV] C=0.029600079668876515, gamma=0.6614167167922481 ................\n",
      "[CV]  C=0.029600079668876515, gamma=0.6614167167922481, score=-0.696, total=   0.5s\n",
      "[CV] C=0.029600079668876515, gamma=0.6614167167922481 ................\n",
      "[CV]  C=0.029600079668876515, gamma=0.6614167167922481, score=-0.693, total=   0.5s\n",
      "[CV] C=0.029600079668876515, gamma=0.6614167167922481 ................\n",
      "[CV]  C=0.029600079668876515, gamma=0.6614167167922481, score=-0.693, total=   0.5s\n",
      "[CV] C=0.029600079668876515, gamma=0.6614167167922481 ................\n",
      "[CV]  C=0.029600079668876515, gamma=0.6614167167922481, score=-0.694, total=   0.5s\n",
      "[CV] C=0.029600079668876515, gamma=0.6614167167922481 ................\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV]  C=0.029600079668876515, gamma=0.6614167167922481, score=-0.694, total=   0.5s\n",
      "[CV] C=0.029600079668876515, gamma=0.6614167167922481 ................\n",
      "[CV]  C=0.029600079668876515, gamma=0.6614167167922481, score=-0.693, total=   0.5s\n",
      "[CV] C=0.029600079668876515, gamma=0.6614167167922481 ................\n",
      "[CV]  C=0.029600079668876515, gamma=0.6614167167922481, score=-0.694, total=   0.5s\n",
      "[CV] C=0.27015114208486657, gamma=2.2135429049733886 .................\n",
      "[CV]  C=0.27015114208486657, gamma=2.2135429049733886, score=-0.694, total=   0.6s\n",
      "[CV] C=0.27015114208486657, gamma=2.2135429049733886 .................\n",
      "[CV]  C=0.27015114208486657, gamma=2.2135429049733886, score=-0.693, total=   0.6s\n",
      "[CV] C=0.27015114208486657, gamma=2.2135429049733886 .................\n",
      "[CV]  C=0.27015114208486657, gamma=2.2135429049733886, score=-0.693, total=   0.6s\n",
      "[CV] C=0.27015114208486657, gamma=2.2135429049733886 .................\n",
      "[CV]  C=0.27015114208486657, gamma=2.2135429049733886, score=-0.694, total=   0.5s\n",
      "[CV] C=0.27015114208486657, gamma=2.2135429049733886 .................\n",
      "[CV]  C=0.27015114208486657, gamma=2.2135429049733886, score=-0.694, total=   0.6s\n",
      "[CV] C=0.27015114208486657, gamma=2.2135429049733886 .................\n",
      "[CV]  C=0.27015114208486657, gamma=2.2135429049733886, score=-0.694, total=   0.6s\n",
      "[CV] C=0.27015114208486657, gamma=2.2135429049733886 .................\n",
      "[CV]  C=0.27015114208486657, gamma=2.2135429049733886, score=-0.694, total=   0.6s\n",
      "[CV] C=0.27015114208486657, gamma=2.2135429049733886 .................\n",
      "[CV]  C=0.27015114208486657, gamma=2.2135429049733886, score=-0.695, total=   0.6s\n",
      "[CV] C=0.27015114208486657, gamma=2.2135429049733886 .................\n",
      "[CV]  C=0.27015114208486657, gamma=2.2135429049733886, score=-0.694, total=   0.6s\n",
      "[CV] C=0.27015114208486657, gamma=2.2135429049733886 .................\n",
      "[CV]  C=0.27015114208486657, gamma=2.2135429049733886, score=-0.693, total=   0.6s\n",
      "[CV] C=27.634818776488856, gamma=5.217636608655088 ...................\n",
      "[CV]  C=27.634818776488856, gamma=5.217636608655088, score=-0.694, total=   0.6s\n",
      "[CV] C=27.634818776488856, gamma=5.217636608655088 ...................\n",
      "[CV]  C=27.634818776488856, gamma=5.217636608655088, score=-0.694, total=   0.6s\n",
      "[CV] C=27.634818776488856, gamma=5.217636608655088 ...................\n",
      "[CV]  C=27.634818776488856, gamma=5.217636608655088, score=-0.693, total=   0.6s\n",
      "[CV] C=27.634818776488856, gamma=5.217636608655088 ...................\n",
      "[CV]  C=27.634818776488856, gamma=5.217636608655088, score=-0.693, total=   0.7s\n",
      "[CV] C=27.634818776488856, gamma=5.217636608655088 ...................\n",
      "[CV]  C=27.634818776488856, gamma=5.217636608655088, score=-0.693, total=   0.6s\n",
      "[CV] C=27.634818776488856, gamma=5.217636608655088 ...................\n",
      "[CV]  C=27.634818776488856, gamma=5.217636608655088, score=-0.693, total=   0.7s\n",
      "[CV] C=27.634818776488856, gamma=5.217636608655088 ...................\n",
      "[CV]  C=27.634818776488856, gamma=5.217636608655088, score=-0.693, total=   0.7s\n",
      "[CV] C=27.634818776488856, gamma=5.217636608655088 ...................\n",
      "[CV]  C=27.634818776488856, gamma=5.217636608655088, score=-0.693, total=   0.7s\n",
      "[CV] C=27.634818776488856, gamma=5.217636608655088 ...................\n",
      "[CV]  C=27.634818776488856, gamma=5.217636608655088, score=-0.693, total=   0.6s\n",
      "[CV] C=27.634818776488856, gamma=5.217636608655088 ...................\n",
      "[CV]  C=27.634818776488856, gamma=5.217636608655088, score=-0.693, total=   0.7s\n",
      "[CV] C=0.036714112790199004, gamma=16.814112346672502 ................\n",
      "[CV]  C=0.036714112790199004, gamma=16.814112346672502, score=-0.693, total=   0.9s\n",
      "[CV] C=0.036714112790199004, gamma=16.814112346672502 ................\n",
      "[CV]  C=0.036714112790199004, gamma=16.814112346672502, score=-0.693, total=   1.2s\n",
      "[CV] C=0.036714112790199004, gamma=16.814112346672502 ................\n",
      "[CV]  C=0.036714112790199004, gamma=16.814112346672502, score=-0.693, total=   1.0s\n",
      "[CV] C=0.036714112790199004, gamma=16.814112346672502 ................\n",
      "[CV]  C=0.036714112790199004, gamma=16.814112346672502, score=-0.693, total=   1.0s\n",
      "[CV] C=0.036714112790199004, gamma=16.814112346672502 ................\n",
      "[CV]  C=0.036714112790199004, gamma=16.814112346672502, score=-0.695, total=   0.8s\n",
      "[CV] C=0.036714112790199004, gamma=16.814112346672502 ................\n",
      "[CV]  C=0.036714112790199004, gamma=16.814112346672502, score=-0.693, total=   0.9s\n",
      "[CV] C=0.036714112790199004, gamma=16.814112346672502 ................\n",
      "[CV]  C=0.036714112790199004, gamma=16.814112346672502, score=-0.695, total=   1.0s\n",
      "[CV] C=0.036714112790199004, gamma=16.814112346672502 ................\n",
      "[CV]  C=0.036714112790199004, gamma=16.814112346672502, score=-0.694, total=   1.0s\n",
      "[CV] C=0.036714112790199004, gamma=16.814112346672502 ................\n",
      "[CV]  C=0.036714112790199004, gamma=16.814112346672502, score=-0.694, total=   1.0s\n",
      "[CV] C=0.036714112790199004, gamma=16.814112346672502 ................\n",
      "[CV]  C=0.036714112790199004, gamma=16.814112346672502, score=-0.694, total=   0.8s\n",
      "[CV] C=32.38506442109645, gamma=2.4076439586143232 ...................\n",
      "[CV]  C=32.38506442109645, gamma=2.4076439586143232, score=-0.626, total=   0.6s\n",
      "[CV] C=32.38506442109645, gamma=2.4076439586143232 ...................\n",
      "[CV]  C=32.38506442109645, gamma=2.4076439586143232, score=-0.654, total=   0.6s\n",
      "[CV] C=32.38506442109645, gamma=2.4076439586143232 ...................\n",
      "[CV]  C=32.38506442109645, gamma=2.4076439586143232, score=-0.634, total=   0.6s\n",
      "[CV] C=32.38506442109645, gamma=2.4076439586143232 ...................\n",
      "[CV]  C=32.38506442109645, gamma=2.4076439586143232, score=-0.657, total=   0.6s\n",
      "[CV] C=32.38506442109645, gamma=2.4076439586143232 ...................\n",
      "[CV]  C=32.38506442109645, gamma=2.4076439586143232, score=-0.624, total=   0.6s\n",
      "[CV] C=32.38506442109645, gamma=2.4076439586143232 ...................\n",
      "[CV]  C=32.38506442109645, gamma=2.4076439586143232, score=-0.668, total=   0.6s\n",
      "[CV] C=32.38506442109645, gamma=2.4076439586143232 ...................\n",
      "[CV]  C=32.38506442109645, gamma=2.4076439586143232, score=-0.661, total=   0.6s\n",
      "[CV] C=32.38506442109645, gamma=2.4076439586143232 ...................\n",
      "[CV]  C=32.38506442109645, gamma=2.4076439586143232, score=-0.647, total=   0.6s\n",
      "[CV] C=32.38506442109645, gamma=2.4076439586143232 ...................\n",
      "[CV]  C=32.38506442109645, gamma=2.4076439586143232, score=-0.669, total=   0.6s\n",
      "[CV] C=32.38506442109645, gamma=2.4076439586143232 ...................\n",
      "[CV]  C=32.38506442109645, gamma=2.4076439586143232, score=-0.630, total=   0.6s\n",
      "[CV] C=0.03262650909275475, gamma=8.22184960950331 ...................\n",
      "[CV]  C=0.03262650909275475, gamma=8.22184960950331, score=-0.693, total=   0.6s\n",
      "[CV] C=0.03262650909275475, gamma=8.22184960950331 ...................\n",
      "[CV]  C=0.03262650909275475, gamma=8.22184960950331, score=-0.693, total=   0.6s\n",
      "[CV] C=0.03262650909275475, gamma=8.22184960950331 ...................\n",
      "[CV]  C=0.03262650909275475, gamma=8.22184960950331, score=-0.693, total=   0.6s\n",
      "[CV] C=0.03262650909275475, gamma=8.22184960950331 ...................\n",
      "[CV]  C=0.03262650909275475, gamma=8.22184960950331, score=-0.693, total=   0.6s\n",
      "[CV] C=0.03262650909275475, gamma=8.22184960950331 ...................\n",
      "[CV]  C=0.03262650909275475, gamma=8.22184960950331, score=-0.694, total=   0.6s\n",
      "[CV] C=0.03262650909275475, gamma=8.22184960950331 ...................\n",
      "[CV]  C=0.03262650909275475, gamma=8.22184960950331, score=-0.694, total=   0.6s\n",
      "[CV] C=0.03262650909275475, gamma=8.22184960950331 ...................\n",
      "[CV]  C=0.03262650909275475, gamma=8.22184960950331, score=-0.693, total=   0.6s\n",
      "[CV] C=0.03262650909275475, gamma=8.22184960950331 ...................\n",
      "[CV]  C=0.03262650909275475, gamma=8.22184960950331, score=-0.693, total=   0.6s\n",
      "[CV] C=0.03262650909275475, gamma=8.22184960950331 ...................\n",
      "[CV]  C=0.03262650909275475, gamma=8.22184960950331, score=-0.694, total=   0.6s\n",
      "[CV] C=0.03262650909275475, gamma=8.22184960950331 ...................\n",
      "[CV]  C=0.03262650909275475, gamma=8.22184960950331, score=-0.696, total=   0.6s\n",
      "[CV] C=0.8984964421366174, gamma=19.901842696875132 ..................\n",
      "[CV]  C=0.8984964421366174, gamma=19.901842696875132, score=-0.694, total=   1.0s\n",
      "[CV] C=0.8984964421366174, gamma=19.901842696875132 ..................\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV]  C=0.8984964421366174, gamma=19.901842696875132, score=-0.694, total=   0.9s\n",
      "[CV] C=0.8984964421366174, gamma=19.901842696875132 ..................\n",
      "[CV]  C=0.8984964421366174, gamma=19.901842696875132, score=-0.693, total=   0.9s\n",
      "[CV] C=0.8984964421366174, gamma=19.901842696875132 ..................\n",
      "[CV]  C=0.8984964421366174, gamma=19.901842696875132, score=-0.693, total=   0.9s\n",
      "[CV] C=0.8984964421366174, gamma=19.901842696875132 ..................\n",
      "[CV]  C=0.8984964421366174, gamma=19.901842696875132, score=-0.693, total=   0.9s\n",
      "[CV] C=0.8984964421366174, gamma=19.901842696875132 ..................\n",
      "[CV]  C=0.8984964421366174, gamma=19.901842696875132, score=-0.695, total=   0.9s\n",
      "[CV] C=0.8984964421366174, gamma=19.901842696875132 ..................\n",
      "[CV]  C=0.8984964421366174, gamma=19.901842696875132, score=-0.693, total=   0.9s\n",
      "[CV] C=0.8984964421366174, gamma=19.901842696875132 ..................\n",
      "[CV]  C=0.8984964421366174, gamma=19.901842696875132, score=-0.694, total=   1.0s\n",
      "[CV] C=0.8984964421366174, gamma=19.901842696875132 ..................\n",
      "[CV]  C=0.8984964421366174, gamma=19.901842696875132, score=-0.693, total=   1.0s\n",
      "[CV] C=0.8984964421366174, gamma=19.901842696875132 ..................\n",
      "[CV]  C=0.8984964421366174, gamma=19.901842696875132, score=-0.693, total=   0.9s\n",
      "[CV] C=0.6540714028171729, gamma=0.02390444043508945 .................\n",
      "[CV]  C=0.6540714028171729, gamma=0.02390444043508945, score=-0.193, total=   0.2s\n",
      "[CV] C=0.6540714028171729, gamma=0.02390444043508945 .................\n",
      "[CV]  C=0.6540714028171729, gamma=0.02390444043508945, score=-0.155, total=   0.2s\n",
      "[CV] C=0.6540714028171729, gamma=0.02390444043508945 .................\n",
      "[CV]  C=0.6540714028171729, gamma=0.02390444043508945, score=-0.143, total=   0.2s\n",
      "[CV] C=0.6540714028171729, gamma=0.02390444043508945 .................\n",
      "[CV]  C=0.6540714028171729, gamma=0.02390444043508945, score=-0.105, total=   0.2s\n",
      "[CV] C=0.6540714028171729, gamma=0.02390444043508945 .................\n",
      "[CV]  C=0.6540714028171729, gamma=0.02390444043508945, score=-0.259, total=   0.2s\n",
      "[CV] C=0.6540714028171729, gamma=0.02390444043508945 .................\n",
      "[CV]  C=0.6540714028171729, gamma=0.02390444043508945, score=-0.392, total=   0.2s\n",
      "[CV] C=0.6540714028171729, gamma=0.02390444043508945 .................\n",
      "[CV]  C=0.6540714028171729, gamma=0.02390444043508945, score=-0.329, total=   0.2s\n",
      "[CV] C=0.6540714028171729, gamma=0.02390444043508945 .................\n",
      "[CV]  C=0.6540714028171729, gamma=0.02390444043508945, score=-0.325, total=   0.2s\n",
      "[CV] C=0.6540714028171729, gamma=0.02390444043508945 .................\n",
      "[CV]  C=0.6540714028171729, gamma=0.02390444043508945, score=-0.309, total=   0.2s\n",
      "[CV] C=0.6540714028171729, gamma=0.02390444043508945 .................\n",
      "[CV]  C=0.6540714028171729, gamma=0.02390444043508945, score=-0.403, total=   0.2s\n",
      "[CV] C=0.1596177980415499, gamma=64.82991049432805 ...................\n",
      "[CV]  C=0.1596177980415499, gamma=64.82991049432805, score=-0.693, total=   1.2s\n",
      "[CV] C=0.1596177980415499, gamma=64.82991049432805 ...................\n",
      "[CV]  C=0.1596177980415499, gamma=64.82991049432805, score=-0.693, total=   1.3s\n",
      "[CV] C=0.1596177980415499, gamma=64.82991049432805 ...................\n",
      "[CV]  C=0.1596177980415499, gamma=64.82991049432805, score=-0.694, total=   1.3s\n",
      "[CV] C=0.1596177980415499, gamma=64.82991049432805 ...................\n",
      "[CV]  C=0.1596177980415499, gamma=64.82991049432805, score=-0.694, total=   1.3s\n",
      "[CV] C=0.1596177980415499, gamma=64.82991049432805 ...................\n",
      "[CV]  C=0.1596177980415499, gamma=64.82991049432805, score=-0.694, total=   1.3s\n",
      "[CV] C=0.1596177980415499, gamma=64.82991049432805 ...................\n",
      "[CV]  C=0.1596177980415499, gamma=64.82991049432805, score=-0.693, total=   1.3s\n",
      "[CV] C=0.1596177980415499, gamma=64.82991049432805 ...................\n",
      "[CV]  C=0.1596177980415499, gamma=64.82991049432805, score=-0.693, total=   1.3s\n",
      "[CV] C=0.1596177980415499, gamma=64.82991049432805 ...................\n",
      "[CV]  C=0.1596177980415499, gamma=64.82991049432805, score=-0.694, total=   1.2s\n",
      "[CV] C=0.1596177980415499, gamma=64.82991049432805 ...................\n",
      "[CV]  C=0.1596177980415499, gamma=64.82991049432805, score=-0.693, total=   1.3s\n",
      "[CV] C=0.1596177980415499, gamma=64.82991049432805 ...................\n",
      "[CV]  C=0.1596177980415499, gamma=64.82991049432805, score=-0.694, total=   1.3s\n",
      "[CV] C=0.07306551805039999, gamma=5.528921153783024 ..................\n",
      "[CV]  C=0.07306551805039999, gamma=5.528921153783024, score=-0.693, total=   0.5s\n",
      "[CV] C=0.07306551805039999, gamma=5.528921153783024 ..................\n",
      "[CV]  C=0.07306551805039999, gamma=5.528921153783024, score=-0.693, total=   0.6s\n",
      "[CV] C=0.07306551805039999, gamma=5.528921153783024 ..................\n",
      "[CV]  C=0.07306551805039999, gamma=5.528921153783024, score=-0.693, total=   0.5s\n",
      "[CV] C=0.07306551805039999, gamma=5.528921153783024 ..................\n",
      "[CV]  C=0.07306551805039999, gamma=5.528921153783024, score=-0.693, total=   0.6s\n",
      "[CV] C=0.07306551805039999, gamma=5.528921153783024 ..................\n",
      "[CV]  C=0.07306551805039999, gamma=5.528921153783024, score=-0.693, total=   0.6s\n",
      "[CV] C=0.07306551805039999, gamma=5.528921153783024 ..................\n",
      "[CV]  C=0.07306551805039999, gamma=5.528921153783024, score=-0.695, total=   0.6s\n",
      "[CV] C=0.07306551805039999, gamma=5.528921153783024 ..................\n",
      "[CV]  C=0.07306551805039999, gamma=5.528921153783024, score=-0.693, total=   0.6s\n",
      "[CV] C=0.07306551805039999, gamma=5.528921153783024 ..................\n",
      "[CV]  C=0.07306551805039999, gamma=5.528921153783024, score=-0.694, total=   0.6s\n",
      "[CV] C=0.07306551805039999, gamma=5.528921153783024 ..................\n",
      "[CV]  C=0.07306551805039999, gamma=5.528921153783024, score=-0.693, total=   0.6s\n",
      "[CV] C=0.07306551805039999, gamma=5.528921153783024 ..................\n",
      "[CV]  C=0.07306551805039999, gamma=5.528921153783024, score=-0.694, total=   0.6s\n",
      "[CV] C=0.3546879775104732, gamma=5.1124552176969225 ..................\n",
      "[CV]  C=0.3546879775104732, gamma=5.1124552176969225, score=-0.693, total=   0.6s\n",
      "[CV] C=0.3546879775104732, gamma=5.1124552176969225 ..................\n",
      "[CV]  C=0.3546879775104732, gamma=5.1124552176969225, score=-0.694, total=   0.6s\n",
      "[CV] C=0.3546879775104732, gamma=5.1124552176969225 ..................\n",
      "[CV]  C=0.3546879775104732, gamma=5.1124552176969225, score=-0.693, total=   0.6s\n",
      "[CV] C=0.3546879775104732, gamma=5.1124552176969225 ..................\n",
      "[CV]  C=0.3546879775104732, gamma=5.1124552176969225, score=-0.693, total=   0.6s\n",
      "[CV] C=0.3546879775104732, gamma=5.1124552176969225 ..................\n",
      "[CV]  C=0.3546879775104732, gamma=5.1124552176969225, score=-0.693, total=   0.6s\n",
      "[CV] C=0.3546879775104732, gamma=5.1124552176969225 ..................\n",
      "[CV]  C=0.3546879775104732, gamma=5.1124552176969225, score=-0.693, total=   0.6s\n",
      "[CV] C=0.3546879775104732, gamma=5.1124552176969225 ..................\n",
      "[CV]  C=0.3546879775104732, gamma=5.1124552176969225, score=-0.693, total=   0.6s\n",
      "[CV] C=0.3546879775104732, gamma=5.1124552176969225 ..................\n",
      "[CV]  C=0.3546879775104732, gamma=5.1124552176969225, score=-0.694, total=   0.6s\n",
      "[CV] C=0.3546879775104732, gamma=5.1124552176969225 ..................\n",
      "[CV]  C=0.3546879775104732, gamma=5.1124552176969225, score=-0.694, total=   0.6s\n",
      "[CV] C=0.3546879775104732, gamma=5.1124552176969225 ..................\n",
      "[CV]  C=0.3546879775104732, gamma=5.1124552176969225, score=-0.693, total=   0.6s\n",
      "[CV] C=87.1657109323045, gamma=10.197495561031324 ....................\n",
      "[CV]  C=87.1657109323045, gamma=10.197495561031324, score=-0.693, total=   0.7s\n",
      "[CV] C=87.1657109323045, gamma=10.197495561031324 ....................\n",
      "[CV]  C=87.1657109323045, gamma=10.197495561031324, score=-0.693, total=   0.7s\n",
      "[CV] C=87.1657109323045, gamma=10.197495561031324 ....................\n",
      "[CV]  C=87.1657109323045, gamma=10.197495561031324, score=-0.693, total=   0.7s\n",
      "[CV] C=87.1657109323045, gamma=10.197495561031324 ....................\n",
      "[CV]  C=87.1657109323045, gamma=10.197495561031324, score=-0.693, total=   0.7s\n",
      "[CV] C=87.1657109323045, gamma=10.197495561031324 ....................\n",
      "[CV]  C=87.1657109323045, gamma=10.197495561031324, score=-0.693, total=   0.7s\n",
      "[CV] C=87.1657109323045, gamma=10.197495561031324 ....................\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV]  C=87.1657109323045, gamma=10.197495561031324, score=-0.693, total=   0.7s\n",
      "[CV] C=87.1657109323045, gamma=10.197495561031324 ....................\n",
      "[CV]  C=87.1657109323045, gamma=10.197495561031324, score=-0.693, total=   0.7s\n",
      "[CV] C=87.1657109323045, gamma=10.197495561031324 ....................\n",
      "[CV]  C=87.1657109323045, gamma=10.197495561031324, score=-0.693, total=   0.7s\n",
      "[CV] C=87.1657109323045, gamma=10.197495561031324 ....................\n",
      "[CV]  C=87.1657109323045, gamma=10.197495561031324, score=-0.693, total=   0.7s\n",
      "[CV] C=87.1657109323045, gamma=10.197495561031324 ....................\n",
      "[CV]  C=87.1657109323045, gamma=10.197495561031324, score=-0.693, total=   0.7s\n",
      "[CV] C=19.930431094669313, gamma=1.4785219604311253 ..................\n",
      "[CV]  C=19.930431094669313, gamma=1.4785219604311253, score=-0.492, total=   0.6s\n",
      "[CV] C=19.930431094669313, gamma=1.4785219604311253 ..................\n",
      "[CV]  C=19.930431094669313, gamma=1.4785219604311253, score=-0.456, total=   0.6s\n",
      "[CV] C=19.930431094669313, gamma=1.4785219604311253 ..................\n",
      "[CV]  C=19.930431094669313, gamma=1.4785219604311253, score=-0.455, total=   0.6s\n",
      "[CV] C=19.930431094669313, gamma=1.4785219604311253 ..................\n",
      "[CV]  C=19.930431094669313, gamma=1.4785219604311253, score=-0.456, total=   0.6s\n",
      "[CV] C=19.930431094669313, gamma=1.4785219604311253 ..................\n",
      "[CV]  C=19.930431094669313, gamma=1.4785219604311253, score=-0.469, total=   0.6s\n",
      "[CV] C=19.930431094669313, gamma=1.4785219604311253 ..................\n",
      "[CV]  C=19.930431094669313, gamma=1.4785219604311253, score=-0.649, total=   0.6s\n",
      "[CV] C=19.930431094669313, gamma=1.4785219604311253 ..................\n",
      "[CV]  C=19.930431094669313, gamma=1.4785219604311253, score=-0.571, total=   0.6s\n",
      "[CV] C=19.930431094669313, gamma=1.4785219604311253 ..................\n",
      "[CV]  C=19.930431094669313, gamma=1.4785219604311253, score=-0.566, total=   0.6s\n",
      "[CV] C=19.930431094669313, gamma=1.4785219604311253 ..................\n",
      "[CV]  C=19.930431094669313, gamma=1.4785219604311253, score=-0.645, total=   0.6s\n",
      "[CV] C=19.930431094669313, gamma=1.4785219604311253 ..................\n",
      "[CV]  C=19.930431094669313, gamma=1.4785219604311253, score=-0.563, total=   0.6s\n",
      "[CV] C=16.473902978917007, gamma=0.294016639825569 ...................\n",
      "[CV]  C=16.473902978917007, gamma=0.294016639825569, score=-0.284, total=   0.6s\n",
      "[CV] C=16.473902978917007, gamma=0.294016639825569 ...................\n",
      "[CV]  C=16.473902978917007, gamma=0.294016639825569, score=-0.204, total=   0.6s\n",
      "[CV] C=16.473902978917007, gamma=0.294016639825569 ...................\n",
      "[CV]  C=16.473902978917007, gamma=0.294016639825569, score=-0.147, total=   0.6s\n",
      "[CV] C=16.473902978917007, gamma=0.294016639825569 ...................\n",
      "[CV]  C=16.473902978917007, gamma=0.294016639825569, score=-0.157, total=   0.6s\n",
      "[CV] C=16.473902978917007, gamma=0.294016639825569 ...................\n",
      "[CV]  C=16.473902978917007, gamma=0.294016639825569, score=-0.207, total=   0.6s\n",
      "[CV] C=16.473902978917007, gamma=0.294016639825569 ...................\n",
      "[CV]  C=16.473902978917007, gamma=0.294016639825569, score=-0.426, total=   0.5s\n",
      "[CV] C=16.473902978917007, gamma=0.294016639825569 ...................\n",
      "[CV]  C=16.473902978917007, gamma=0.294016639825569, score=-0.361, total=   0.6s\n",
      "[CV] C=16.473902978917007, gamma=0.294016639825569 ...................\n",
      "[CV]  C=16.473902978917007, gamma=0.294016639825569, score=-0.324, total=   0.5s\n",
      "[CV] C=16.473902978917007, gamma=0.294016639825569 ...................\n",
      "[CV]  C=16.473902978917007, gamma=0.294016639825569, score=-0.422, total=   0.6s\n",
      "[CV] C=16.473902978917007, gamma=0.294016639825569 ...................\n",
      "[CV]  C=16.473902978917007, gamma=0.294016639825569, score=-0.330, total=   0.5s\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Done 250 out of 250 | elapsed:  2.5min finished\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "RandomizedSearchCV(cv=10, estimator=SVC(probability=True), n_iter=25,\n",
       "                   param_distributions={'C': <scipy.stats._distn_infrastructure.rv_frozen object at 0x000001DCA53709C8>,\n",
       "                                        'gamma': <scipy.stats._distn_infrastructure.rv_frozen object at 0x000001DCA439F148>},\n",
       "                   scoring='neg_log_loss', verbose=10)"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.utils.fixes import loguniform\n",
    "\n",
    "param_distributions = {'C':loguniform(a = 1e-2, b= 1e2),\n",
    "                      'gamma':loguniform(a = 1e-2, b= 1e2)}\n",
    "\n",
    "Rclf = RandomizedSearchCV(estimator = clf_svm, \n",
    "                          param_distributions = param_distributions,\n",
    "                          n_iter = 25, # newly added as well\n",
    "                          scoring='neg_log_loss', \n",
    "                          n_jobs=None, \n",
    "                          #iid='deprecated', take note of depreciation! version 0.22 warning\n",
    "                          refit=True,\n",
    "                          cv= n_splits, # 10 fold CV, but! stratified KFold\n",
    "                          verbose=10)\n",
    "\n",
    "Rclf.fit(X,\n",
    "         y['bin'],\n",
    "         sample_weight = None) #sample_weight here!!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best params for estimator: SVC(C=1.308709482507712, gamma=0.06937936028678086, probability=True)\n",
      "Best CV Score: -0.095234\n",
      "\n",
      "A total of 85 was performed before optimal CV score found! (Under split3_test_score / 4th split)\n",
      "\n",
      "Estimated time taken for entire process: 67.476273 seconds\n",
      "Total number of candidates/ nodes: 25\n",
      "\n",
      "=======================================================\n",
      "Estimated total mean time required for optimal solution:\n",
      "\n",
      "Score Time: 0.111877s\n",
      "Fit Time: 5.103568s\n"
     ]
    }
   ],
   "source": [
    "print_result(clf = Rclf, n_splits = n_splits)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Comparison between GSCV vs RSCV:\n",
    "\n",
    "**Grid Search CV**\n",
    "\n",
    "Best params for estimator: SVC(C=1, gamma=0.1, probability=True)\n",
    "\n",
    "Best CV Score: -0.095049\n",
    "\n",
    "A total of 87 was performed before optimal CV score found! (Under split3_test_score / 4th split)\n",
    "\n",
    "Estimated time taken for entire process: 77.301800 seconds\n",
    "Total number of candidates/ nodes: 25\n",
    "\n",
    "Estimated total mean time required for optimal solution:\n",
    "\n",
    "Score Time: 0.156009s\n",
    "Fit Time: 6.964366s\n",
    "\n",
    "**Random Search CV**\n",
    "\n",
    "Best params for estimator: SVC(C=5.146914740190327, gamma=0.03550270078366642, probability=True)\n",
    "\n",
    "Best CV Score: -0.100946 (Lowest)\n",
    "\n",
    "A total of 78 was performed before optimal CV score found! (Under split3_test_score / 4th split)\n",
    "\n",
    "Estimated time taken for entire process: 62.176406 seconds\n",
    "Total number of candidates/ nodes: 25\n",
    "\n",
    "Estimated total mean time required for optimal solution:\n",
    "\n",
    "Score Time: 0.021389s\n",
    "Fit Time: 1.155873s\n",
    "\n",
    "### Conclusion\n",
    "\n",
    "Number of nodes are both 25 (RSCV: n_iter = 25), (GSCV = 5 * 5 inputs).\n",
    "\n",
    "While RSCV does perform significantly much faster for both score and fit time as well as the number of fits required.\n",
    "\n",
    "GSCV has a higher neg_log_loss score (Not optimal prediction score as compared to RSCV).\n",
    "\n",
    "Hence RSCV does seem overall better with a lower score and faster results.\n",
    "\n",
    "But as seen in previous chapters, randomness will usually cause an inflated score (more bias). Hence, GSCV may provide a more reliable outcome.\n",
    "\n",
    "**Note**\n",
    "\n",
    "For sklearn, their attributes after fit will always return the highest mean score (Worst score for neg_log_loss). Hence the above comparison is more like comparing between the worst log_loss reported, which method provides the better prediction score (lowest)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "def in_sample_sharpe_ratio(clf):\n",
    "    sharpe_ratio = []\n",
    "    for i in np.arange(len(clf.cv_results_['mean_test_score'])):\n",
    "        if clf.cv_results_['mean_test_score'][i] < 0:\n",
    "            sharpe_ratio.append(-1 * clf.cv_results_['mean_test_score'][i]/ clf.cv_results_['std_test_score'][i])\n",
    "        else:\n",
    "            sharpe_ratio.append(clf.cv_results_['mean_test_score'][i]/ clf.cv_results_['std_test_score'][i])\n",
    "    print(\"IS Best Score Sharpe Ratio: {0:.6f}\".format(sharpe_ratio[clf.best_index_]))\n",
    "    print(\"Best IS Sharpe ratio: {0:.6f}\\nLowest IS Sharpe Ratio: {1:.6f}\\nMean Sharpe Ratio: {2:.6f}\".format(max(sharpe_ratio), min(sharpe_ratio), np.mean(sharpe_ratio)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "IS Best Score Sharpe Ratio: 2.487544\n",
      "Best IS Sharpe ratio: inf\n",
      "Lowest IS Sharpe Ratio: 2.194502\n",
      "Mean Sharpe Ratio: inf\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\Wei_X\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:5: RuntimeWarning: divide by zero encountered in double_scalars\n",
      "  \"\"\"\n"
     ]
    }
   ],
   "source": [
    "in_sample_sharpe_ratio(clf)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 10 folds for each of 25 candidates, totalling 250 fits\n",
      "[CV] C=0.01, gamma=0.01, kernel=rbf ..................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Using backend SequentialBackend with 1 concurrent workers.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] ...... C=0.01, gamma=0.01, kernel=rbf, score=0.890, total=   0.5s\n",
      "[CV] C=0.01, gamma=0.01, kernel=rbf ..................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Done   1 out of   1 | elapsed:    0.4s remaining:    0.0s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] ...... C=0.01, gamma=0.01, kernel=rbf, score=0.880, total=   0.5s\n",
      "[CV] C=0.01, gamma=0.01, kernel=rbf ..................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Done   2 out of   2 | elapsed:    0.9s remaining:    0.0s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] ...... C=0.01, gamma=0.01, kernel=rbf, score=0.915, total=   0.5s\n",
      "[CV] C=0.01, gamma=0.01, kernel=rbf ..................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Done   3 out of   3 | elapsed:    1.4s remaining:    0.0s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] ...... C=0.01, gamma=0.01, kernel=rbf, score=0.945, total=   0.5s\n",
      "[CV] C=0.01, gamma=0.01, kernel=rbf ..................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Done   4 out of   4 | elapsed:    1.9s remaining:    0.0s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] ...... C=0.01, gamma=0.01, kernel=rbf, score=0.900, total=   0.5s\n",
      "[CV] C=0.01, gamma=0.01, kernel=rbf ..................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Done   5 out of   5 | elapsed:    2.4s remaining:    0.0s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] ...... C=0.01, gamma=0.01, kernel=rbf, score=0.700, total=   0.5s\n",
      "[CV] C=0.01, gamma=0.01, kernel=rbf ..................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Done   6 out of   6 | elapsed:    2.9s remaining:    0.0s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] ...... C=0.01, gamma=0.01, kernel=rbf, score=0.775, total=   0.5s\n",
      "[CV] C=0.01, gamma=0.01, kernel=rbf ..................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Done   7 out of   7 | elapsed:    3.4s remaining:    0.0s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] ...... C=0.01, gamma=0.01, kernel=rbf, score=0.685, total=   0.5s\n",
      "[CV] C=0.01, gamma=0.01, kernel=rbf ..................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Done   8 out of   8 | elapsed:    3.9s remaining:    0.0s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] ...... C=0.01, gamma=0.01, kernel=rbf, score=0.735, total=   0.5s\n",
      "[CV] C=0.01, gamma=0.01, kernel=rbf ..................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Done   9 out of   9 | elapsed:    4.4s remaining:    0.0s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] ...... C=0.01, gamma=0.01, kernel=rbf, score=0.715, total=   0.5s\n",
      "[CV] C=0.01, gamma=0.1, kernel=rbf ...................................\n",
      "[CV] ....... C=0.01, gamma=0.1, kernel=rbf, score=0.920, total=   0.5s\n",
      "[CV] C=0.01, gamma=0.1, kernel=rbf ...................................\n",
      "[CV] ....... C=0.01, gamma=0.1, kernel=rbf, score=0.920, total=   0.5s\n",
      "[CV] C=0.01, gamma=0.1, kernel=rbf ...................................\n",
      "[CV] ....... C=0.01, gamma=0.1, kernel=rbf, score=0.960, total=   0.5s\n",
      "[CV] C=0.01, gamma=0.1, kernel=rbf ...................................\n",
      "[CV] ....... C=0.01, gamma=0.1, kernel=rbf, score=0.970, total=   0.5s\n",
      "[CV] C=0.01, gamma=0.1, kernel=rbf ...................................\n",
      "[CV] ....... C=0.01, gamma=0.1, kernel=rbf, score=0.930, total=   0.5s\n",
      "[CV] C=0.01, gamma=0.1, kernel=rbf ...................................\n",
      "[CV] ....... C=0.01, gamma=0.1, kernel=rbf, score=0.795, total=   0.5s\n",
      "[CV] C=0.01, gamma=0.1, kernel=rbf ...................................\n",
      "[CV] ....... C=0.01, gamma=0.1, kernel=rbf, score=0.775, total=   0.5s\n",
      "[CV] C=0.01, gamma=0.1, kernel=rbf ...................................\n",
      "[CV] ....... C=0.01, gamma=0.1, kernel=rbf, score=0.760, total=   0.5s\n",
      "[CV] C=0.01, gamma=0.1, kernel=rbf ...................................\n",
      "[CV] ....... C=0.01, gamma=0.1, kernel=rbf, score=0.775, total=   0.5s\n",
      "[CV] C=0.01, gamma=0.1, kernel=rbf ...................................\n",
      "[CV] ....... C=0.01, gamma=0.1, kernel=rbf, score=0.750, total=   0.5s\n",
      "[CV] C=0.01, gamma=1, kernel=rbf .....................................\n",
      "[CV] ......... C=0.01, gamma=1, kernel=rbf, score=0.500, total=   0.5s\n",
      "[CV] C=0.01, gamma=1, kernel=rbf .....................................\n",
      "[CV] ......... C=0.01, gamma=1, kernel=rbf, score=0.500, total=   0.5s\n",
      "[CV] C=0.01, gamma=1, kernel=rbf .....................................\n",
      "[CV] ......... C=0.01, gamma=1, kernel=rbf, score=0.500, total=   0.5s\n",
      "[CV] C=0.01, gamma=1, kernel=rbf .....................................\n",
      "[CV] ......... C=0.01, gamma=1, kernel=rbf, score=0.500, total=   0.5s\n",
      "[CV] C=0.01, gamma=1, kernel=rbf .....................................\n",
      "[CV] ......... C=0.01, gamma=1, kernel=rbf, score=0.500, total=   0.5s\n",
      "[CV] C=0.01, gamma=1, kernel=rbf .....................................\n",
      "[CV] ......... C=0.01, gamma=1, kernel=rbf, score=0.505, total=   0.5s\n",
      "[CV] C=0.01, gamma=1, kernel=rbf .....................................\n",
      "[CV] ......... C=0.01, gamma=1, kernel=rbf, score=0.505, total=   0.5s\n",
      "[CV] C=0.01, gamma=1, kernel=rbf .....................................\n",
      "[CV] ......... C=0.01, gamma=1, kernel=rbf, score=0.505, total=   0.5s\n",
      "[CV] C=0.01, gamma=1, kernel=rbf .....................................\n",
      "[CV] ......... C=0.01, gamma=1, kernel=rbf, score=0.505, total=   0.5s\n",
      "[CV] C=0.01, gamma=1, kernel=rbf .....................................\n",
      "[CV] ......... C=0.01, gamma=1, kernel=rbf, score=0.505, total=   0.5s\n",
      "[CV] C=0.01, gamma=10, kernel=rbf ....................................\n",
      "[CV] ........ C=0.01, gamma=10, kernel=rbf, score=0.500, total=   0.6s\n",
      "[CV] C=0.01, gamma=10, kernel=rbf ....................................\n",
      "[CV] ........ C=0.01, gamma=10, kernel=rbf, score=0.500, total=   0.6s\n",
      "[CV] C=0.01, gamma=10, kernel=rbf ....................................\n",
      "[CV] ........ C=0.01, gamma=10, kernel=rbf, score=0.500, total=   0.6s\n",
      "[CV] C=0.01, gamma=10, kernel=rbf ....................................\n",
      "[CV] ........ C=0.01, gamma=10, kernel=rbf, score=0.500, total=   0.6s\n",
      "[CV] C=0.01, gamma=10, kernel=rbf ....................................\n",
      "[CV] ........ C=0.01, gamma=10, kernel=rbf, score=0.500, total=   0.6s\n",
      "[CV] C=0.01, gamma=10, kernel=rbf ....................................\n",
      "[CV] ........ C=0.01, gamma=10, kernel=rbf, score=0.505, total=   0.6s\n",
      "[CV] C=0.01, gamma=10, kernel=rbf ....................................\n",
      "[CV] ........ C=0.01, gamma=10, kernel=rbf, score=0.505, total=   0.6s\n",
      "[CV] C=0.01, gamma=10, kernel=rbf ....................................\n",
      "[CV] ........ C=0.01, gamma=10, kernel=rbf, score=0.505, total=   0.6s\n",
      "[CV] C=0.01, gamma=10, kernel=rbf ....................................\n",
      "[CV] ........ C=0.01, gamma=10, kernel=rbf, score=0.505, total=   0.6s\n",
      "[CV] C=0.01, gamma=10, kernel=rbf ....................................\n",
      "[CV] ........ C=0.01, gamma=10, kernel=rbf, score=0.505, total=   0.6s\n",
      "[CV] C=0.01, gamma=100, kernel=rbf ...................................\n",
      "[CV] ....... C=0.01, gamma=100, kernel=rbf, score=0.500, total=   1.2s\n",
      "[CV] C=0.01, gamma=100, kernel=rbf ...................................\n",
      "[CV] ....... C=0.01, gamma=100, kernel=rbf, score=0.500, total=   1.2s\n",
      "[CV] C=0.01, gamma=100, kernel=rbf ...................................\n",
      "[CV] ....... C=0.01, gamma=100, kernel=rbf, score=0.500, total=   1.4s\n",
      "[CV] C=0.01, gamma=100, kernel=rbf ...................................\n",
      "[CV] ....... C=0.01, gamma=100, kernel=rbf, score=0.500, total=   1.4s\n",
      "[CV] C=0.01, gamma=100, kernel=rbf ...................................\n",
      "[CV] ....... C=0.01, gamma=100, kernel=rbf, score=0.500, total=   1.4s\n",
      "[CV] C=0.01, gamma=100, kernel=rbf ...................................\n",
      "[CV] ....... C=0.01, gamma=100, kernel=rbf, score=0.505, total=   1.4s\n",
      "[CV] C=0.01, gamma=100, kernel=rbf ...................................\n",
      "[CV] ....... C=0.01, gamma=100, kernel=rbf, score=0.505, total=   1.4s\n",
      "[CV] C=0.01, gamma=100, kernel=rbf ...................................\n",
      "[CV] ....... C=0.01, gamma=100, kernel=rbf, score=0.505, total=   1.5s\n",
      "[CV] C=0.01, gamma=100, kernel=rbf ...................................\n",
      "[CV] ....... C=0.01, gamma=100, kernel=rbf, score=0.505, total=   1.4s\n",
      "[CV] C=0.01, gamma=100, kernel=rbf ...................................\n",
      "[CV] ....... C=0.01, gamma=100, kernel=rbf, score=0.505, total=   1.3s\n",
      "[CV] C=0.1, gamma=0.01, kernel=rbf ...................................\n",
      "[CV] ....... C=0.1, gamma=0.01, kernel=rbf, score=0.910, total=   0.4s\n",
      "[CV] C=0.1, gamma=0.01, kernel=rbf ...................................\n",
      "[CV] ....... C=0.1, gamma=0.01, kernel=rbf, score=0.920, total=   0.3s\n",
      "[CV] C=0.1, gamma=0.01, kernel=rbf ...................................\n",
      "[CV] ....... C=0.1, gamma=0.01, kernel=rbf, score=0.965, total=   0.3s\n",
      "[CV] C=0.1, gamma=0.01, kernel=rbf ...................................\n",
      "[CV] ....... C=0.1, gamma=0.01, kernel=rbf, score=0.955, total=   0.3s\n",
      "[CV] C=0.1, gamma=0.01, kernel=rbf ...................................\n",
      "[CV] ....... C=0.1, gamma=0.01, kernel=rbf, score=0.930, total=   0.3s\n",
      "[CV] C=0.1, gamma=0.01, kernel=rbf ...................................\n",
      "[CV] ....... C=0.1, gamma=0.01, kernel=rbf, score=0.765, total=   0.3s\n",
      "[CV] C=0.1, gamma=0.01, kernel=rbf ...................................\n",
      "[CV] ....... C=0.1, gamma=0.01, kernel=rbf, score=0.805, total=   0.3s\n",
      "[CV] C=0.1, gamma=0.01, kernel=rbf ...................................\n",
      "[CV] ....... C=0.1, gamma=0.01, kernel=rbf, score=0.760, total=   0.3s\n",
      "[CV] C=0.1, gamma=0.01, kernel=rbf ...................................\n",
      "[CV] ....... C=0.1, gamma=0.01, kernel=rbf, score=0.775, total=   0.3s\n",
      "[CV] C=0.1, gamma=0.01, kernel=rbf ...................................\n",
      "[CV] ....... C=0.1, gamma=0.01, kernel=rbf, score=0.755, total=   0.3s\n",
      "[CV] C=0.1, gamma=0.1, kernel=rbf ....................................\n",
      "[CV] ........ C=0.1, gamma=0.1, kernel=rbf, score=0.935, total=   0.3s\n",
      "[CV] C=0.1, gamma=0.1, kernel=rbf ....................................\n",
      "[CV] ........ C=0.1, gamma=0.1, kernel=rbf, score=0.950, total=   0.3s\n",
      "[CV] C=0.1, gamma=0.1, kernel=rbf ....................................\n",
      "[CV] ........ C=0.1, gamma=0.1, kernel=rbf, score=0.980, total=   0.4s\n",
      "[CV] C=0.1, gamma=0.1, kernel=rbf ....................................\n",
      "[CV] ........ C=0.1, gamma=0.1, kernel=rbf, score=0.985, total=   0.3s\n",
      "[CV] C=0.1, gamma=0.1, kernel=rbf ....................................\n",
      "[CV] ........ C=0.1, gamma=0.1, kernel=rbf, score=0.950, total=   0.4s\n",
      "[CV] C=0.1, gamma=0.1, kernel=rbf ....................................\n",
      "[CV] ........ C=0.1, gamma=0.1, kernel=rbf, score=0.830, total=   0.3s\n",
      "[CV] C=0.1, gamma=0.1, kernel=rbf ....................................\n",
      "[CV] ........ C=0.1, gamma=0.1, kernel=rbf, score=0.860, total=   0.3s\n",
      "[CV] C=0.1, gamma=0.1, kernel=rbf ....................................\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] ........ C=0.1, gamma=0.1, kernel=rbf, score=0.825, total=   0.3s\n",
      "[CV] C=0.1, gamma=0.1, kernel=rbf ....................................\n",
      "[CV] ........ C=0.1, gamma=0.1, kernel=rbf, score=0.860, total=   0.3s\n",
      "[CV] C=0.1, gamma=0.1, kernel=rbf ....................................\n",
      "[CV] ........ C=0.1, gamma=0.1, kernel=rbf, score=0.820, total=   0.3s\n",
      "[CV] C=0.1, gamma=1, kernel=rbf ......................................\n",
      "[CV] .......... C=0.1, gamma=1, kernel=rbf, score=0.500, total=   0.5s\n",
      "[CV] C=0.1, gamma=1, kernel=rbf ......................................\n",
      "[CV] .......... C=0.1, gamma=1, kernel=rbf, score=0.500, total=   0.5s\n",
      "[CV] C=0.1, gamma=1, kernel=rbf ......................................\n",
      "[CV] .......... C=0.1, gamma=1, kernel=rbf, score=0.500, total=   0.5s\n",
      "[CV] C=0.1, gamma=1, kernel=rbf ......................................\n",
      "[CV] .......... C=0.1, gamma=1, kernel=rbf, score=0.500, total=   0.5s\n",
      "[CV] C=0.1, gamma=1, kernel=rbf ......................................\n",
      "[CV] .......... C=0.1, gamma=1, kernel=rbf, score=0.500, total=   0.5s\n",
      "[CV] C=0.1, gamma=1, kernel=rbf ......................................\n",
      "[CV] .......... C=0.1, gamma=1, kernel=rbf, score=0.505, total=   0.5s\n",
      "[CV] C=0.1, gamma=1, kernel=rbf ......................................\n",
      "[CV] .......... C=0.1, gamma=1, kernel=rbf, score=0.505, total=   0.5s\n",
      "[CV] C=0.1, gamma=1, kernel=rbf ......................................\n",
      "[CV] .......... C=0.1, gamma=1, kernel=rbf, score=0.505, total=   0.5s\n",
      "[CV] C=0.1, gamma=1, kernel=rbf ......................................\n",
      "[CV] .......... C=0.1, gamma=1, kernel=rbf, score=0.505, total=   0.5s\n",
      "[CV] C=0.1, gamma=1, kernel=rbf ......................................\n",
      "[CV] .......... C=0.1, gamma=1, kernel=rbf, score=0.505, total=   0.5s\n",
      "[CV] C=0.1, gamma=10, kernel=rbf .....................................\n",
      "[CV] ......... C=0.1, gamma=10, kernel=rbf, score=0.500, total=   0.7s\n",
      "[CV] C=0.1, gamma=10, kernel=rbf .....................................\n",
      "[CV] ......... C=0.1, gamma=10, kernel=rbf, score=0.500, total=   0.6s\n",
      "[CV] C=0.1, gamma=10, kernel=rbf .....................................\n",
      "[CV] ......... C=0.1, gamma=10, kernel=rbf, score=0.500, total=   0.6s\n",
      "[CV] C=0.1, gamma=10, kernel=rbf .....................................\n",
      "[CV] ......... C=0.1, gamma=10, kernel=rbf, score=0.500, total=   0.6s\n",
      "[CV] C=0.1, gamma=10, kernel=rbf .....................................\n",
      "[CV] ......... C=0.1, gamma=10, kernel=rbf, score=0.500, total=   0.6s\n",
      "[CV] C=0.1, gamma=10, kernel=rbf .....................................\n",
      "[CV] ......... C=0.1, gamma=10, kernel=rbf, score=0.505, total=   0.6s\n",
      "[CV] C=0.1, gamma=10, kernel=rbf .....................................\n",
      "[CV] ......... C=0.1, gamma=10, kernel=rbf, score=0.505, total=   0.6s\n",
      "[CV] C=0.1, gamma=10, kernel=rbf .....................................\n",
      "[CV] ......... C=0.1, gamma=10, kernel=rbf, score=0.505, total=   0.6s\n",
      "[CV] C=0.1, gamma=10, kernel=rbf .....................................\n",
      "[CV] ......... C=0.1, gamma=10, kernel=rbf, score=0.505, total=   0.6s\n",
      "[CV] C=0.1, gamma=10, kernel=rbf .....................................\n",
      "[CV] ......... C=0.1, gamma=10, kernel=rbf, score=0.505, total=   0.6s\n",
      "[CV] C=0.1, gamma=100, kernel=rbf ....................................\n",
      "[CV] ........ C=0.1, gamma=100, kernel=rbf, score=0.500, total=   1.3s\n",
      "[CV] C=0.1, gamma=100, kernel=rbf ....................................\n",
      "[CV] ........ C=0.1, gamma=100, kernel=rbf, score=0.500, total=   1.3s\n",
      "[CV] C=0.1, gamma=100, kernel=rbf ....................................\n",
      "[CV] ........ C=0.1, gamma=100, kernel=rbf, score=0.500, total=   1.3s\n",
      "[CV] C=0.1, gamma=100, kernel=rbf ....................................\n",
      "[CV] ........ C=0.1, gamma=100, kernel=rbf, score=0.500, total=   1.3s\n",
      "[CV] C=0.1, gamma=100, kernel=rbf ....................................\n",
      "[CV] ........ C=0.1, gamma=100, kernel=rbf, score=0.500, total=   1.3s\n",
      "[CV] C=0.1, gamma=100, kernel=rbf ....................................\n",
      "[CV] ........ C=0.1, gamma=100, kernel=rbf, score=0.505, total=   1.3s\n",
      "[CV] C=0.1, gamma=100, kernel=rbf ....................................\n",
      "[CV] ........ C=0.1, gamma=100, kernel=rbf, score=0.505, total=   1.3s\n",
      "[CV] C=0.1, gamma=100, kernel=rbf ....................................\n",
      "[CV] ........ C=0.1, gamma=100, kernel=rbf, score=0.505, total=   1.3s\n",
      "[CV] C=0.1, gamma=100, kernel=rbf ....................................\n",
      "[CV] ........ C=0.1, gamma=100, kernel=rbf, score=0.505, total=   1.7s\n",
      "[CV] C=0.1, gamma=100, kernel=rbf ....................................\n",
      "[CV] ........ C=0.1, gamma=100, kernel=rbf, score=0.505, total=   1.4s\n",
      "[CV] C=1, gamma=0.01, kernel=rbf .....................................\n",
      "[CV] ......... C=1, gamma=0.01, kernel=rbf, score=0.925, total=   0.3s\n",
      "[CV] C=1, gamma=0.01, kernel=rbf .....................................\n",
      "[CV] ......... C=1, gamma=0.01, kernel=rbf, score=0.935, total=   0.3s\n",
      "[CV] C=1, gamma=0.01, kernel=rbf .....................................\n",
      "[CV] ......... C=1, gamma=0.01, kernel=rbf, score=0.965, total=   0.3s\n",
      "[CV] C=1, gamma=0.01, kernel=rbf .....................................\n",
      "[CV] ......... C=1, gamma=0.01, kernel=rbf, score=0.985, total=   0.3s\n",
      "[CV] C=1, gamma=0.01, kernel=rbf .....................................\n",
      "[CV] ......... C=1, gamma=0.01, kernel=rbf, score=0.945, total=   0.2s\n",
      "[CV] C=1, gamma=0.01, kernel=rbf .....................................\n",
      "[CV] ......... C=1, gamma=0.01, kernel=rbf, score=0.820, total=   0.2s\n",
      "[CV] C=1, gamma=0.01, kernel=rbf .....................................\n",
      "[CV] ......... C=1, gamma=0.01, kernel=rbf, score=0.845, total=   0.2s\n",
      "[CV] C=1, gamma=0.01, kernel=rbf .....................................\n",
      "[CV] ......... C=1, gamma=0.01, kernel=rbf, score=0.805, total=   0.2s\n",
      "[CV] C=1, gamma=0.01, kernel=rbf .....................................\n",
      "[CV] ......... C=1, gamma=0.01, kernel=rbf, score=0.840, total=   0.2s\n",
      "[CV] C=1, gamma=0.01, kernel=rbf .....................................\n",
      "[CV] ......... C=1, gamma=0.01, kernel=rbf, score=0.825, total=   0.2s\n",
      "[CV] C=1, gamma=0.1, kernel=rbf ......................................\n",
      "[CV] .......... C=1, gamma=0.1, kernel=rbf, score=0.940, total=   0.3s\n",
      "[CV] C=1, gamma=0.1, kernel=rbf ......................................\n",
      "[CV] .......... C=1, gamma=0.1, kernel=rbf, score=0.970, total=   0.2s\n",
      "[CV] C=1, gamma=0.1, kernel=rbf ......................................\n",
      "[CV] .......... C=1, gamma=0.1, kernel=rbf, score=0.975, total=   0.3s\n",
      "[CV] C=1, gamma=0.1, kernel=rbf ......................................\n",
      "[CV] .......... C=1, gamma=0.1, kernel=rbf, score=0.970, total=   0.3s\n",
      "[CV] C=1, gamma=0.1, kernel=rbf ......................................\n",
      "[CV] .......... C=1, gamma=0.1, kernel=rbf, score=0.960, total=   0.3s\n",
      "[CV] C=1, gamma=0.1, kernel=rbf ......................................\n",
      "[CV] .......... C=1, gamma=0.1, kernel=rbf, score=0.870, total=   0.2s\n",
      "[CV] C=1, gamma=0.1, kernel=rbf ......................................\n",
      "[CV] .......... C=1, gamma=0.1, kernel=rbf, score=0.910, total=   0.3s\n",
      "[CV] C=1, gamma=0.1, kernel=rbf ......................................\n",
      "[CV] .......... C=1, gamma=0.1, kernel=rbf, score=0.910, total=   0.3s\n",
      "[CV] C=1, gamma=0.1, kernel=rbf ......................................\n",
      "[CV] .......... C=1, gamma=0.1, kernel=rbf, score=0.885, total=   0.2s\n",
      "[CV] C=1, gamma=0.1, kernel=rbf ......................................\n",
      "[CV] .......... C=1, gamma=0.1, kernel=rbf, score=0.850, total=   0.3s\n",
      "[CV] C=1, gamma=1, kernel=rbf ........................................\n",
      "[CV] ............ C=1, gamma=1, kernel=rbf, score=0.910, total=   0.6s\n",
      "[CV] C=1, gamma=1, kernel=rbf ........................................\n",
      "[CV] ............ C=1, gamma=1, kernel=rbf, score=0.900, total=   0.6s\n",
      "[CV] C=1, gamma=1, kernel=rbf ........................................\n",
      "[CV] ............ C=1, gamma=1, kernel=rbf, score=0.965, total=   0.6s\n",
      "[CV] C=1, gamma=1, kernel=rbf ........................................\n",
      "[CV] ............ C=1, gamma=1, kernel=rbf, score=0.900, total=   0.6s\n",
      "[CV] C=1, gamma=1, kernel=rbf ........................................\n",
      "[CV] ............ C=1, gamma=1, kernel=rbf, score=0.880, total=   0.7s\n",
      "[CV] C=1, gamma=1, kernel=rbf ........................................\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] ............ C=1, gamma=1, kernel=rbf, score=0.700, total=   0.6s\n",
      "[CV] C=1, gamma=1, kernel=rbf ........................................\n",
      "[CV] ............ C=1, gamma=1, kernel=rbf, score=0.760, total=   0.6s\n",
      "[CV] C=1, gamma=1, kernel=rbf ........................................\n",
      "[CV] ............ C=1, gamma=1, kernel=rbf, score=0.680, total=   0.6s\n",
      "[CV] C=1, gamma=1, kernel=rbf ........................................\n",
      "[CV] ............ C=1, gamma=1, kernel=rbf, score=0.730, total=   0.6s\n",
      "[CV] C=1, gamma=1, kernel=rbf ........................................\n",
      "[CV] ............ C=1, gamma=1, kernel=rbf, score=0.715, total=   0.6s\n",
      "[CV] C=1, gamma=10, kernel=rbf .......................................\n",
      "[CV] ........... C=1, gamma=10, kernel=rbf, score=0.500, total=   0.6s\n",
      "[CV] C=1, gamma=10, kernel=rbf .......................................\n",
      "[CV] ........... C=1, gamma=10, kernel=rbf, score=0.500, total=   0.6s\n",
      "[CV] C=1, gamma=10, kernel=rbf .......................................\n",
      "[CV] ........... C=1, gamma=10, kernel=rbf, score=0.500, total=   0.6s\n",
      "[CV] C=1, gamma=10, kernel=rbf .......................................\n",
      "[CV] ........... C=1, gamma=10, kernel=rbf, score=0.500, total=   0.7s\n",
      "[CV] C=1, gamma=10, kernel=rbf .......................................\n",
      "[CV] ........... C=1, gamma=10, kernel=rbf, score=0.500, total=   0.7s\n",
      "[CV] C=1, gamma=10, kernel=rbf .......................................\n",
      "[CV] ........... C=1, gamma=10, kernel=rbf, score=0.505, total=   0.7s\n",
      "[CV] C=1, gamma=10, kernel=rbf .......................................\n",
      "[CV] ........... C=1, gamma=10, kernel=rbf, score=0.505, total=   0.7s\n",
      "[CV] C=1, gamma=10, kernel=rbf .......................................\n",
      "[CV] ........... C=1, gamma=10, kernel=rbf, score=0.505, total=   0.7s\n",
      "[CV] C=1, gamma=10, kernel=rbf .......................................\n",
      "[CV] ........... C=1, gamma=10, kernel=rbf, score=0.505, total=   0.7s\n",
      "[CV] C=1, gamma=10, kernel=rbf .......................................\n",
      "[CV] ........... C=1, gamma=10, kernel=rbf, score=0.505, total=   0.7s\n",
      "[CV] C=1, gamma=100, kernel=rbf ......................................\n",
      "[CV] .......... C=1, gamma=100, kernel=rbf, score=0.500, total=   1.4s\n",
      "[CV] C=1, gamma=100, kernel=rbf ......................................\n",
      "[CV] .......... C=1, gamma=100, kernel=rbf, score=0.500, total=   1.3s\n",
      "[CV] C=1, gamma=100, kernel=rbf ......................................\n",
      "[CV] .......... C=1, gamma=100, kernel=rbf, score=0.500, total=   1.3s\n",
      "[CV] C=1, gamma=100, kernel=rbf ......................................\n",
      "[CV] .......... C=1, gamma=100, kernel=rbf, score=0.500, total=   1.3s\n",
      "[CV] C=1, gamma=100, kernel=rbf ......................................\n",
      "[CV] .......... C=1, gamma=100, kernel=rbf, score=0.500, total=   1.4s\n",
      "[CV] C=1, gamma=100, kernel=rbf ......................................\n",
      "[CV] .......... C=1, gamma=100, kernel=rbf, score=0.505, total=   1.4s\n",
      "[CV] C=1, gamma=100, kernel=rbf ......................................\n",
      "[CV] .......... C=1, gamma=100, kernel=rbf, score=0.505, total=   1.3s\n",
      "[CV] C=1, gamma=100, kernel=rbf ......................................\n",
      "[CV] .......... C=1, gamma=100, kernel=rbf, score=0.505, total=   1.3s\n",
      "[CV] C=1, gamma=100, kernel=rbf ......................................\n",
      "[CV] .......... C=1, gamma=100, kernel=rbf, score=0.505, total=   1.4s\n",
      "[CV] C=1, gamma=100, kernel=rbf ......................................\n",
      "[CV] .......... C=1, gamma=100, kernel=rbf, score=0.505, total=   1.3s\n",
      "[CV] C=10, gamma=0.01, kernel=rbf ....................................\n",
      "[CV] ........ C=10, gamma=0.01, kernel=rbf, score=0.925, total=   0.2s\n",
      "[CV] C=10, gamma=0.01, kernel=rbf ....................................\n",
      "[CV] ........ C=10, gamma=0.01, kernel=rbf, score=0.960, total=   0.2s\n",
      "[CV] C=10, gamma=0.01, kernel=rbf ....................................\n",
      "[CV] ........ C=10, gamma=0.01, kernel=rbf, score=0.970, total=   0.2s\n",
      "[CV] C=10, gamma=0.01, kernel=rbf ....................................\n",
      "[CV] ........ C=10, gamma=0.01, kernel=rbf, score=0.970, total=   0.2s\n",
      "[CV] C=10, gamma=0.01, kernel=rbf ....................................\n",
      "[CV] ........ C=10, gamma=0.01, kernel=rbf, score=0.940, total=   0.2s\n",
      "[CV] C=10, gamma=0.01, kernel=rbf ....................................\n",
      "[CV] ........ C=10, gamma=0.01, kernel=rbf, score=0.865, total=   0.2s\n",
      "[CV] C=10, gamma=0.01, kernel=rbf ....................................\n",
      "[CV] ........ C=10, gamma=0.01, kernel=rbf, score=0.910, total=   0.2s\n",
      "[CV] C=10, gamma=0.01, kernel=rbf ....................................\n",
      "[CV] ........ C=10, gamma=0.01, kernel=rbf, score=0.850, total=   0.2s\n",
      "[CV] C=10, gamma=0.01, kernel=rbf ....................................\n",
      "[CV] ........ C=10, gamma=0.01, kernel=rbf, score=0.870, total=   0.2s\n",
      "[CV] C=10, gamma=0.01, kernel=rbf ....................................\n",
      "[CV] ........ C=10, gamma=0.01, kernel=rbf, score=0.860, total=   0.2s\n",
      "[CV] C=10, gamma=0.1, kernel=rbf .....................................\n",
      "[CV] ......... C=10, gamma=0.1, kernel=rbf, score=0.910, total=   0.3s\n",
      "[CV] C=10, gamma=0.1, kernel=rbf .....................................\n",
      "[CV] ......... C=10, gamma=0.1, kernel=rbf, score=0.920, total=   0.3s\n",
      "[CV] C=10, gamma=0.1, kernel=rbf .....................................\n",
      "[CV] ......... C=10, gamma=0.1, kernel=rbf, score=0.965, total=   0.3s\n",
      "[CV] C=10, gamma=0.1, kernel=rbf .....................................\n",
      "[CV] ......... C=10, gamma=0.1, kernel=rbf, score=0.965, total=   0.3s\n",
      "[CV] C=10, gamma=0.1, kernel=rbf .....................................\n",
      "[CV] ......... C=10, gamma=0.1, kernel=rbf, score=0.965, total=   0.3s\n",
      "[CV] C=10, gamma=0.1, kernel=rbf .....................................\n",
      "[CV] ......... C=10, gamma=0.1, kernel=rbf, score=0.860, total=   0.3s\n",
      "[CV] C=10, gamma=0.1, kernel=rbf .....................................\n",
      "[CV] ......... C=10, gamma=0.1, kernel=rbf, score=0.865, total=   0.3s\n",
      "[CV] C=10, gamma=0.1, kernel=rbf .....................................\n",
      "[CV] ......... C=10, gamma=0.1, kernel=rbf, score=0.895, total=   0.3s\n",
      "[CV] C=10, gamma=0.1, kernel=rbf .....................................\n",
      "[CV] ......... C=10, gamma=0.1, kernel=rbf, score=0.880, total=   0.3s\n",
      "[CV] C=10, gamma=0.1, kernel=rbf .....................................\n",
      "[CV] ......... C=10, gamma=0.1, kernel=rbf, score=0.875, total=   0.3s\n",
      "[CV] C=10, gamma=1, kernel=rbf .......................................\n",
      "[CV] ........... C=10, gamma=1, kernel=rbf, score=0.900, total=   0.6s\n",
      "[CV] C=10, gamma=1, kernel=rbf .......................................\n",
      "[CV] ........... C=10, gamma=1, kernel=rbf, score=0.900, total=   0.6s\n",
      "[CV] C=10, gamma=1, kernel=rbf .......................................\n",
      "[CV] ........... C=10, gamma=1, kernel=rbf, score=0.940, total=   0.6s\n",
      "[CV] C=10, gamma=1, kernel=rbf .......................................\n",
      "[CV] ........... C=10, gamma=1, kernel=rbf, score=0.900, total=   0.6s\n",
      "[CV] C=10, gamma=1, kernel=rbf .......................................\n",
      "[CV] ........... C=10, gamma=1, kernel=rbf, score=0.870, total=   0.6s\n",
      "[CV] C=10, gamma=1, kernel=rbf .......................................\n",
      "[CV] ........... C=10, gamma=1, kernel=rbf, score=0.700, total=   0.6s\n",
      "[CV] C=10, gamma=1, kernel=rbf .......................................\n",
      "[CV] ........... C=10, gamma=1, kernel=rbf, score=0.760, total=   0.6s\n",
      "[CV] C=10, gamma=1, kernel=rbf .......................................\n",
      "[CV] ........... C=10, gamma=1, kernel=rbf, score=0.680, total=   0.6s\n",
      "[CV] C=10, gamma=1, kernel=rbf .......................................\n",
      "[CV] ........... C=10, gamma=1, kernel=rbf, score=0.730, total=   0.6s\n",
      "[CV] C=10, gamma=1, kernel=rbf .......................................\n",
      "[CV] ........... C=10, gamma=1, kernel=rbf, score=0.715, total=   0.6s\n",
      "[CV] C=10, gamma=10, kernel=rbf ......................................\n",
      "[CV] .......... C=10, gamma=10, kernel=rbf, score=0.500, total=   0.7s\n",
      "[CV] C=10, gamma=10, kernel=rbf ......................................\n",
      "[CV] .......... C=10, gamma=10, kernel=rbf, score=0.500, total=   0.7s\n",
      "[CV] C=10, gamma=10, kernel=rbf ......................................\n",
      "[CV] .......... C=10, gamma=10, kernel=rbf, score=0.500, total=   0.7s\n",
      "[CV] C=10, gamma=10, kernel=rbf ......................................\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] .......... C=10, gamma=10, kernel=rbf, score=0.500, total=   0.7s\n",
      "[CV] C=10, gamma=10, kernel=rbf ......................................\n",
      "[CV] .......... C=10, gamma=10, kernel=rbf, score=0.500, total=   0.7s\n",
      "[CV] C=10, gamma=10, kernel=rbf ......................................\n",
      "[CV] .......... C=10, gamma=10, kernel=rbf, score=0.505, total=   0.7s\n",
      "[CV] C=10, gamma=10, kernel=rbf ......................................\n",
      "[CV] .......... C=10, gamma=10, kernel=rbf, score=0.505, total=   0.6s\n",
      "[CV] C=10, gamma=10, kernel=rbf ......................................\n",
      "[CV] .......... C=10, gamma=10, kernel=rbf, score=0.505, total=   0.7s\n",
      "[CV] C=10, gamma=10, kernel=rbf ......................................\n",
      "[CV] .......... C=10, gamma=10, kernel=rbf, score=0.505, total=   0.7s\n",
      "[CV] C=10, gamma=10, kernel=rbf ......................................\n",
      "[CV] .......... C=10, gamma=10, kernel=rbf, score=0.505, total=   0.7s\n",
      "[CV] C=10, gamma=100, kernel=rbf .....................................\n",
      "[CV] ......... C=10, gamma=100, kernel=rbf, score=0.500, total=   1.4s\n",
      "[CV] C=10, gamma=100, kernel=rbf .....................................\n",
      "[CV] ......... C=10, gamma=100, kernel=rbf, score=0.500, total=   1.5s\n",
      "[CV] C=10, gamma=100, kernel=rbf .....................................\n",
      "[CV] ......... C=10, gamma=100, kernel=rbf, score=0.500, total=   1.6s\n",
      "[CV] C=10, gamma=100, kernel=rbf .....................................\n",
      "[CV] ......... C=10, gamma=100, kernel=rbf, score=0.500, total=   1.3s\n",
      "[CV] C=10, gamma=100, kernel=rbf .....................................\n",
      "[CV] ......... C=10, gamma=100, kernel=rbf, score=0.500, total=   1.4s\n",
      "[CV] C=10, gamma=100, kernel=rbf .....................................\n",
      "[CV] ......... C=10, gamma=100, kernel=rbf, score=0.505, total=   1.4s\n",
      "[CV] C=10, gamma=100, kernel=rbf .....................................\n",
      "[CV] ......... C=10, gamma=100, kernel=rbf, score=0.505, total=   1.3s\n",
      "[CV] C=10, gamma=100, kernel=rbf .....................................\n",
      "[CV] ......... C=10, gamma=100, kernel=rbf, score=0.505, total=   1.5s\n",
      "[CV] C=10, gamma=100, kernel=rbf .....................................\n",
      "[CV] ......... C=10, gamma=100, kernel=rbf, score=0.505, total=   1.4s\n",
      "[CV] C=10, gamma=100, kernel=rbf .....................................\n",
      "[CV] ......... C=10, gamma=100, kernel=rbf, score=0.505, total=   1.5s\n",
      "[CV] C=100, gamma=0.01, kernel=rbf ...................................\n",
      "[CV] ....... C=100, gamma=0.01, kernel=rbf, score=0.935, total=   0.3s\n",
      "[CV] C=100, gamma=0.01, kernel=rbf ...................................\n",
      "[CV] ....... C=100, gamma=0.01, kernel=rbf, score=0.965, total=   0.3s\n",
      "[CV] C=100, gamma=0.01, kernel=rbf ...................................\n",
      "[CV] ....... C=100, gamma=0.01, kernel=rbf, score=0.970, total=   0.3s\n",
      "[CV] C=100, gamma=0.01, kernel=rbf ...................................\n",
      "[CV] ....... C=100, gamma=0.01, kernel=rbf, score=0.970, total=   0.3s\n",
      "[CV] C=100, gamma=0.01, kernel=rbf ...................................\n",
      "[CV] ....... C=100, gamma=0.01, kernel=rbf, score=0.935, total=   0.3s\n",
      "[CV] C=100, gamma=0.01, kernel=rbf ...................................\n",
      "[CV] ....... C=100, gamma=0.01, kernel=rbf, score=0.885, total=   0.3s\n",
      "[CV] C=100, gamma=0.01, kernel=rbf ...................................\n",
      "[CV] ....... C=100, gamma=0.01, kernel=rbf, score=0.905, total=   0.3s\n",
      "[CV] C=100, gamma=0.01, kernel=rbf ...................................\n",
      "[CV] ....... C=100, gamma=0.01, kernel=rbf, score=0.920, total=   0.3s\n",
      "[CV] C=100, gamma=0.01, kernel=rbf ...................................\n",
      "[CV] ....... C=100, gamma=0.01, kernel=rbf, score=0.875, total=   0.3s\n",
      "[CV] C=100, gamma=0.01, kernel=rbf ...................................\n",
      "[CV] ....... C=100, gamma=0.01, kernel=rbf, score=0.855, total=   0.3s\n",
      "[CV] C=100, gamma=0.1, kernel=rbf ....................................\n",
      "[CV] ........ C=100, gamma=0.1, kernel=rbf, score=0.900, total=   0.3s\n",
      "[CV] C=100, gamma=0.1, kernel=rbf ....................................\n",
      "[CV] ........ C=100, gamma=0.1, kernel=rbf, score=0.885, total=   0.4s\n",
      "[CV] C=100, gamma=0.1, kernel=rbf ....................................\n",
      "[CV] ........ C=100, gamma=0.1, kernel=rbf, score=0.950, total=   0.4s\n",
      "[CV] C=100, gamma=0.1, kernel=rbf ....................................\n",
      "[CV] ........ C=100, gamma=0.1, kernel=rbf, score=0.930, total=   0.4s\n",
      "[CV] C=100, gamma=0.1, kernel=rbf ....................................\n",
      "[CV] ........ C=100, gamma=0.1, kernel=rbf, score=0.900, total=   0.3s\n",
      "[CV] C=100, gamma=0.1, kernel=rbf ....................................\n",
      "[CV] ........ C=100, gamma=0.1, kernel=rbf, score=0.850, total=   0.3s\n",
      "[CV] C=100, gamma=0.1, kernel=rbf ....................................\n",
      "[CV] ........ C=100, gamma=0.1, kernel=rbf, score=0.860, total=   0.3s\n",
      "[CV] C=100, gamma=0.1, kernel=rbf ....................................\n",
      "[CV] ........ C=100, gamma=0.1, kernel=rbf, score=0.840, total=   0.3s\n",
      "[CV] C=100, gamma=0.1, kernel=rbf ....................................\n",
      "[CV] ........ C=100, gamma=0.1, kernel=rbf, score=0.835, total=   0.3s\n",
      "[CV] C=100, gamma=0.1, kernel=rbf ....................................\n",
      "[CV] ........ C=100, gamma=0.1, kernel=rbf, score=0.870, total=   0.3s\n",
      "[CV] C=100, gamma=1, kernel=rbf ......................................\n",
      "[CV] .......... C=100, gamma=1, kernel=rbf, score=0.900, total=   0.6s\n",
      "[CV] C=100, gamma=1, kernel=rbf ......................................\n",
      "[CV] .......... C=100, gamma=1, kernel=rbf, score=0.900, total=   0.6s\n",
      "[CV] C=100, gamma=1, kernel=rbf ......................................\n",
      "[CV] .......... C=100, gamma=1, kernel=rbf, score=0.940, total=   0.6s\n",
      "[CV] C=100, gamma=1, kernel=rbf ......................................\n",
      "[CV] .......... C=100, gamma=1, kernel=rbf, score=0.900, total=   0.7s\n",
      "[CV] C=100, gamma=1, kernel=rbf ......................................\n",
      "[CV] .......... C=100, gamma=1, kernel=rbf, score=0.870, total=   0.7s\n",
      "[CV] C=100, gamma=1, kernel=rbf ......................................\n",
      "[CV] .......... C=100, gamma=1, kernel=rbf, score=0.700, total=   0.7s\n",
      "[CV] C=100, gamma=1, kernel=rbf ......................................\n",
      "[CV] .......... C=100, gamma=1, kernel=rbf, score=0.760, total=   0.7s\n",
      "[CV] C=100, gamma=1, kernel=rbf ......................................\n",
      "[CV] .......... C=100, gamma=1, kernel=rbf, score=0.680, total=   0.7s\n",
      "[CV] C=100, gamma=1, kernel=rbf ......................................\n",
      "[CV] .......... C=100, gamma=1, kernel=rbf, score=0.730, total=   0.6s\n",
      "[CV] C=100, gamma=1, kernel=rbf ......................................\n",
      "[CV] .......... C=100, gamma=1, kernel=rbf, score=0.715, total=   0.7s\n",
      "[CV] C=100, gamma=10, kernel=rbf .....................................\n",
      "[CV] ......... C=100, gamma=10, kernel=rbf, score=0.500, total=   0.7s\n",
      "[CV] C=100, gamma=10, kernel=rbf .....................................\n",
      "[CV] ......... C=100, gamma=10, kernel=rbf, score=0.500, total=   0.7s\n",
      "[CV] C=100, gamma=10, kernel=rbf .....................................\n",
      "[CV] ......... C=100, gamma=10, kernel=rbf, score=0.500, total=   0.8s\n",
      "[CV] C=100, gamma=10, kernel=rbf .....................................\n",
      "[CV] ......... C=100, gamma=10, kernel=rbf, score=0.500, total=   0.8s\n",
      "[CV] C=100, gamma=10, kernel=rbf .....................................\n",
      "[CV] ......... C=100, gamma=10, kernel=rbf, score=0.500, total=   0.7s\n",
      "[CV] C=100, gamma=10, kernel=rbf .....................................\n",
      "[CV] ......... C=100, gamma=10, kernel=rbf, score=0.505, total=   0.7s\n",
      "[CV] C=100, gamma=10, kernel=rbf .....................................\n",
      "[CV] ......... C=100, gamma=10, kernel=rbf, score=0.505, total=   0.8s\n",
      "[CV] C=100, gamma=10, kernel=rbf .....................................\n",
      "[CV] ......... C=100, gamma=10, kernel=rbf, score=0.505, total=   0.7s\n",
      "[CV] C=100, gamma=10, kernel=rbf .....................................\n",
      "[CV] ......... C=100, gamma=10, kernel=rbf, score=0.505, total=   0.7s\n",
      "[CV] C=100, gamma=10, kernel=rbf .....................................\n",
      "[CV] ......... C=100, gamma=10, kernel=rbf, score=0.505, total=   0.7s\n",
      "[CV] C=100, gamma=100, kernel=rbf ....................................\n",
      "[CV] ........ C=100, gamma=100, kernel=rbf, score=0.500, total=   1.3s\n",
      "[CV] C=100, gamma=100, kernel=rbf ....................................\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] ........ C=100, gamma=100, kernel=rbf, score=0.500, total=   1.3s\n",
      "[CV] C=100, gamma=100, kernel=rbf ....................................\n",
      "[CV] ........ C=100, gamma=100, kernel=rbf, score=0.500, total=   1.3s\n",
      "[CV] C=100, gamma=100, kernel=rbf ....................................\n",
      "[CV] ........ C=100, gamma=100, kernel=rbf, score=0.500, total=   1.4s\n",
      "[CV] C=100, gamma=100, kernel=rbf ....................................\n",
      "[CV] ........ C=100, gamma=100, kernel=rbf, score=0.500, total=   1.4s\n",
      "[CV] C=100, gamma=100, kernel=rbf ....................................\n",
      "[CV] ........ C=100, gamma=100, kernel=rbf, score=0.505, total=   1.4s\n",
      "[CV] C=100, gamma=100, kernel=rbf ....................................\n",
      "[CV] ........ C=100, gamma=100, kernel=rbf, score=0.505, total=   1.4s\n",
      "[CV] C=100, gamma=100, kernel=rbf ....................................\n",
      "[CV] ........ C=100, gamma=100, kernel=rbf, score=0.505, total=   1.4s\n",
      "[CV] C=100, gamma=100, kernel=rbf ....................................\n",
      "[CV] ........ C=100, gamma=100, kernel=rbf, score=0.505, total=   1.4s\n",
      "[CV] C=100, gamma=100, kernel=rbf ....................................\n",
      "[CV] ........ C=100, gamma=100, kernel=rbf, score=0.505, total=   1.4s\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Done 250 out of 250 | elapsed:  2.7min finished\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "IS Best Score Sharpe Ratio: 21.373090\n",
      "Best IS Sharpe ratio: 201.000000\n",
      "Lowest IS Sharpe Ratio: 8.064150\n",
      "Mean Sharpe Ratio: 104.225012\n"
     ]
    }
   ],
   "source": [
    "clf1 = GridSearchCV(estimator = clf_svm, \n",
    "                   param_grid = param_grid,\n",
    "                   scoring='accuracy', #change to accuracy\n",
    "                   n_jobs=None, \n",
    "                   #iid='deprecated', take note of depreciation! version 0.22 warning\n",
    "                   refit=True,\n",
    "                   cv= n_splits, # 10 fold CV, but! stratified KFold\n",
    "                   verbose=10)\n",
    "\n",
    "clf1.fit(X,\n",
    "        y['bin'],\n",
    "       sample_weight = None) #sample_weight here!!\n",
    "\n",
    "in_sample_sharpe_ratio(clf1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "IS Best Score Sharpe Ratio: 2.509210\n",
      "Best IS Sharpe ratio: inf\n",
      "Lowest IS Sharpe Ratio: 2.459825\n",
      "Mean Sharpe Ratio: inf\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\Wei_X\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:5: RuntimeWarning: divide by zero encountered in double_scalars\n",
      "  \"\"\"\n"
     ]
    }
   ],
   "source": [
    "in_sample_sharpe_ratio(Rclf)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 10 folds for each of 25 candidates, totalling 250 fits\n",
      "[CV] C=0.15672202039716004, gamma=27.760222656384236 .................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Using backend SequentialBackend with 1 concurrent workers.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV]  C=0.15672202039716004, gamma=27.760222656384236, score=0.500, total=   1.2s\n",
      "[CV] C=0.15672202039716004, gamma=27.760222656384236 .................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Done   1 out of   1 | elapsed:    1.1s remaining:    0.0s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV]  C=0.15672202039716004, gamma=27.760222656384236, score=0.500, total=   1.2s\n",
      "[CV] C=0.15672202039716004, gamma=27.760222656384236 .................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Done   2 out of   2 | elapsed:    2.3s remaining:    0.0s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV]  C=0.15672202039716004, gamma=27.760222656384236, score=0.500, total=   1.2s\n",
      "[CV] C=0.15672202039716004, gamma=27.760222656384236 .................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Done   3 out of   3 | elapsed:    3.5s remaining:    0.0s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV]  C=0.15672202039716004, gamma=27.760222656384236, score=0.500, total=   1.2s\n",
      "[CV] C=0.15672202039716004, gamma=27.760222656384236 .................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Done   4 out of   4 | elapsed:    4.7s remaining:    0.0s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV]  C=0.15672202039716004, gamma=27.760222656384236, score=0.500, total=   1.3s\n",
      "[CV] C=0.15672202039716004, gamma=27.760222656384236 .................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Done   5 out of   5 | elapsed:    6.0s remaining:    0.0s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV]  C=0.15672202039716004, gamma=27.760222656384236, score=0.505, total=   1.2s\n",
      "[CV] C=0.15672202039716004, gamma=27.760222656384236 .................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Done   6 out of   6 | elapsed:    7.2s remaining:    0.0s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV]  C=0.15672202039716004, gamma=27.760222656384236, score=0.505, total=   1.2s\n",
      "[CV] C=0.15672202039716004, gamma=27.760222656384236 .................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Done   7 out of   7 | elapsed:    8.4s remaining:    0.0s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV]  C=0.15672202039716004, gamma=27.760222656384236, score=0.505, total=   1.1s\n",
      "[CV] C=0.15672202039716004, gamma=27.760222656384236 .................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Done   8 out of   8 | elapsed:    9.5s remaining:    0.0s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV]  C=0.15672202039716004, gamma=27.760222656384236, score=0.505, total=   1.1s\n",
      "[CV] C=0.15672202039716004, gamma=27.760222656384236 .................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Done   9 out of   9 | elapsed:   10.7s remaining:    0.0s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV]  C=0.15672202039716004, gamma=27.760222656384236, score=0.505, total=   1.1s\n",
      "[CV] C=57.55419286589449, gamma=0.43792557443306657 ..................\n",
      "[CV]  C=57.55419286589449, gamma=0.43792557443306657, score=0.925, total=   0.6s\n",
      "[CV] C=57.55419286589449, gamma=0.43792557443306657 ..................\n",
      "[CV]  C=57.55419286589449, gamma=0.43792557443306657, score=0.955, total=   0.6s\n",
      "[CV] C=57.55419286589449, gamma=0.43792557443306657 ..................\n",
      "[CV]  C=57.55419286589449, gamma=0.43792557443306657, score=0.970, total=   0.6s\n",
      "[CV] C=57.55419286589449, gamma=0.43792557443306657 ..................\n",
      "[CV]  C=57.55419286589449, gamma=0.43792557443306657, score=0.955, total=   0.6s\n",
      "[CV] C=57.55419286589449, gamma=0.43792557443306657 ..................\n",
      "[CV]  C=57.55419286589449, gamma=0.43792557443306657, score=0.930, total=   0.6s\n",
      "[CV] C=57.55419286589449, gamma=0.43792557443306657 ..................\n",
      "[CV]  C=57.55419286589449, gamma=0.43792557443306657, score=0.850, total=   0.6s\n",
      "[CV] C=57.55419286589449, gamma=0.43792557443306657 ..................\n",
      "[CV]  C=57.55419286589449, gamma=0.43792557443306657, score=0.865, total=   0.6s\n",
      "[CV] C=57.55419286589449, gamma=0.43792557443306657 ..................\n",
      "[CV]  C=57.55419286589449, gamma=0.43792557443306657, score=0.845, total=   0.6s\n",
      "[CV] C=57.55419286589449, gamma=0.43792557443306657 ..................\n",
      "[CV]  C=57.55419286589449, gamma=0.43792557443306657, score=0.865, total=   0.6s\n",
      "[CV] C=57.55419286589449, gamma=0.43792557443306657 ..................\n",
      "[CV]  C=57.55419286589449, gamma=0.43792557443306657, score=0.855, total=   0.6s\n",
      "[CV] C=69.5333640118451, gamma=0.10857742689982587 ...................\n",
      "[CV]  C=69.5333640118451, gamma=0.10857742689982587, score=0.905, total=   0.3s\n",
      "[CV] C=69.5333640118451, gamma=0.10857742689982587 ...................\n",
      "[CV]  C=69.5333640118451, gamma=0.10857742689982587, score=0.890, total=   0.4s\n",
      "[CV] C=69.5333640118451, gamma=0.10857742689982587 ...................\n",
      "[CV]  C=69.5333640118451, gamma=0.10857742689982587, score=0.955, total=   0.3s\n",
      "[CV] C=69.5333640118451, gamma=0.10857742689982587 ...................\n",
      "[CV]  C=69.5333640118451, gamma=0.10857742689982587, score=0.935, total=   0.4s\n",
      "[CV] C=69.5333640118451, gamma=0.10857742689982587 ...................\n",
      "[CV]  C=69.5333640118451, gamma=0.10857742689982587, score=0.910, total=   0.4s\n",
      "[CV] C=69.5333640118451, gamma=0.10857742689982587 ...................\n",
      "[CV]  C=69.5333640118451, gamma=0.10857742689982587, score=0.850, total=   0.3s\n",
      "[CV] C=69.5333640118451, gamma=0.10857742689982587 ...................\n",
      "[CV]  C=69.5333640118451, gamma=0.10857742689982587, score=0.865, total=   0.3s\n",
      "[CV] C=69.5333640118451, gamma=0.10857742689982587 ...................\n",
      "[CV]  C=69.5333640118451, gamma=0.10857742689982587, score=0.840, total=   0.4s\n",
      "[CV] C=69.5333640118451, gamma=0.10857742689982587 ...................\n",
      "[CV]  C=69.5333640118451, gamma=0.10857742689982587, score=0.835, total=   0.4s\n",
      "[CV] C=69.5333640118451, gamma=0.10857742689982587 ...................\n",
      "[CV]  C=69.5333640118451, gamma=0.10857742689982587, score=0.865, total=   0.4s\n",
      "[CV] C=0.015052328779232103, gamma=4.832081421972723 .................\n",
      "[CV]  C=0.015052328779232103, gamma=4.832081421972723, score=0.500, total=   0.7s\n",
      "[CV] C=0.015052328779232103, gamma=4.832081421972723 .................\n",
      "[CV]  C=0.015052328779232103, gamma=4.832081421972723, score=0.500, total=   0.6s\n",
      "[CV] C=0.015052328779232103, gamma=4.832081421972723 .................\n",
      "[CV]  C=0.015052328779232103, gamma=4.832081421972723, score=0.500, total=   0.5s\n",
      "[CV] C=0.015052328779232103, gamma=4.832081421972723 .................\n",
      "[CV]  C=0.015052328779232103, gamma=4.832081421972723, score=0.500, total=   0.6s\n",
      "[CV] C=0.015052328779232103, gamma=4.832081421972723 .................\n",
      "[CV]  C=0.015052328779232103, gamma=4.832081421972723, score=0.500, total=   0.6s\n",
      "[CV] C=0.015052328779232103, gamma=4.832081421972723 .................\n",
      "[CV]  C=0.015052328779232103, gamma=4.832081421972723, score=0.505, total=   0.5s\n",
      "[CV] C=0.015052328779232103, gamma=4.832081421972723 .................\n",
      "[CV]  C=0.015052328779232103, gamma=4.832081421972723, score=0.505, total=   0.6s\n",
      "[CV] C=0.015052328779232103, gamma=4.832081421972723 .................\n",
      "[CV]  C=0.015052328779232103, gamma=4.832081421972723, score=0.505, total=   0.6s\n",
      "[CV] C=0.015052328779232103, gamma=4.832081421972723 .................\n",
      "[CV]  C=0.015052328779232103, gamma=4.832081421972723, score=0.505, total=   0.6s\n",
      "[CV] C=0.015052328779232103, gamma=4.832081421972723 .................\n",
      "[CV]  C=0.015052328779232103, gamma=4.832081421972723, score=0.505, total=   0.7s\n",
      "[CV] C=0.13925235325726276, gamma=6.429143417227232 ..................\n",
      "[CV]  C=0.13925235325726276, gamma=6.429143417227232, score=0.500, total=   1.1s\n",
      "[CV] C=0.13925235325726276, gamma=6.429143417227232 ..................\n",
      "[CV]  C=0.13925235325726276, gamma=6.429143417227232, score=0.500, total=   1.3s\n",
      "[CV] C=0.13925235325726276, gamma=6.429143417227232 ..................\n",
      "[CV]  C=0.13925235325726276, gamma=6.429143417227232, score=0.500, total=   1.3s\n",
      "[CV] C=0.13925235325726276, gamma=6.429143417227232 ..................\n",
      "[CV]  C=0.13925235325726276, gamma=6.429143417227232, score=0.500, total=   0.7s\n",
      "[CV] C=0.13925235325726276, gamma=6.429143417227232 ..................\n",
      "[CV]  C=0.13925235325726276, gamma=6.429143417227232, score=0.500, total=   0.7s\n",
      "[CV] C=0.13925235325726276, gamma=6.429143417227232 ..................\n",
      "[CV]  C=0.13925235325726276, gamma=6.429143417227232, score=0.505, total=   0.6s\n",
      "[CV] C=0.13925235325726276, gamma=6.429143417227232 ..................\n",
      "[CV]  C=0.13925235325726276, gamma=6.429143417227232, score=0.505, total=   0.6s\n",
      "[CV] C=0.13925235325726276, gamma=6.429143417227232 ..................\n",
      "[CV]  C=0.13925235325726276, gamma=6.429143417227232, score=0.505, total=   0.6s\n",
      "[CV] C=0.13925235325726276, gamma=6.429143417227232 ..................\n",
      "[CV]  C=0.13925235325726276, gamma=6.429143417227232, score=0.505, total=   0.6s\n",
      "[CV] C=0.13925235325726276, gamma=6.429143417227232 ..................\n",
      "[CV]  C=0.13925235325726276, gamma=6.429143417227232, score=0.505, total=   0.7s\n",
      "[CV] C=1.1883911809514291, gamma=0.29172451246590325 .................\n",
      "[CV]  C=1.1883911809514291, gamma=0.29172451246590325, score=0.930, total=   0.6s\n",
      "[CV] C=1.1883911809514291, gamma=0.29172451246590325 .................\n",
      "[CV]  C=1.1883911809514291, gamma=0.29172451246590325, score=0.945, total=   0.6s\n",
      "[CV] C=1.1883911809514291, gamma=0.29172451246590325 .................\n",
      "[CV]  C=1.1883911809514291, gamma=0.29172451246590325, score=0.975, total=   0.7s\n",
      "[CV] C=1.1883911809514291, gamma=0.29172451246590325 .................\n",
      "[CV]  C=1.1883911809514291, gamma=0.29172451246590325, score=0.975, total=   0.6s\n",
      "[CV] C=1.1883911809514291, gamma=0.29172451246590325 .................\n",
      "[CV]  C=1.1883911809514291, gamma=0.29172451246590325, score=0.960, total=   0.6s\n",
      "[CV] C=1.1883911809514291, gamma=0.29172451246590325 .................\n",
      "[CV]  C=1.1883911809514291, gamma=0.29172451246590325, score=0.870, total=   0.6s\n",
      "[CV] C=1.1883911809514291, gamma=0.29172451246590325 .................\n",
      "[CV]  C=1.1883911809514291, gamma=0.29172451246590325, score=0.890, total=   0.6s\n",
      "[CV] C=1.1883911809514291, gamma=0.29172451246590325 .................\n",
      "[CV]  C=1.1883911809514291, gamma=0.29172451246590325, score=0.875, total=   0.6s\n",
      "[CV] C=1.1883911809514291, gamma=0.29172451246590325 .................\n",
      "[CV]  C=1.1883911809514291, gamma=0.29172451246590325, score=0.875, total=   0.5s\n",
      "[CV] C=1.1883911809514291, gamma=0.29172451246590325 .................\n",
      "[CV]  C=1.1883911809514291, gamma=0.29172451246590325, score=0.860, total=   0.5s\n",
      "[CV] C=2.484851414405741, gamma=0.019904226105394024 .................\n",
      "[CV]  C=2.484851414405741, gamma=0.019904226105394024, score=0.925, total=   0.2s\n",
      "[CV] C=2.484851414405741, gamma=0.019904226105394024 .................\n",
      "[CV]  C=2.484851414405741, gamma=0.019904226105394024, score=0.960, total=   0.2s\n",
      "[CV] C=2.484851414405741, gamma=0.019904226105394024 .................\n",
      "[CV]  C=2.484851414405741, gamma=0.019904226105394024, score=0.970, total=   0.2s\n",
      "[CV] C=2.484851414405741, gamma=0.019904226105394024 .................\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV]  C=2.484851414405741, gamma=0.019904226105394024, score=0.975, total=   0.3s\n",
      "[CV] C=2.484851414405741, gamma=0.019904226105394024 .................\n",
      "[CV]  C=2.484851414405741, gamma=0.019904226105394024, score=0.940, total=   0.3s\n",
      "[CV] C=2.484851414405741, gamma=0.019904226105394024 .................\n",
      "[CV]  C=2.484851414405741, gamma=0.019904226105394024, score=0.850, total=   0.2s\n",
      "[CV] C=2.484851414405741, gamma=0.019904226105394024 .................\n",
      "[CV]  C=2.484851414405741, gamma=0.019904226105394024, score=0.885, total=   0.2s\n",
      "[CV] C=2.484851414405741, gamma=0.019904226105394024 .................\n",
      "[CV]  C=2.484851414405741, gamma=0.019904226105394024, score=0.845, total=   0.2s\n",
      "[CV] C=2.484851414405741, gamma=0.019904226105394024 .................\n",
      "[CV]  C=2.484851414405741, gamma=0.019904226105394024, score=0.875, total=   0.3s\n",
      "[CV] C=2.484851414405741, gamma=0.019904226105394024 .................\n",
      "[CV]  C=2.484851414405741, gamma=0.019904226105394024, score=0.860, total=   0.2s\n",
      "[CV] C=5.518650428876807, gamma=0.17552129365098906 ..................\n",
      "[CV]  C=5.518650428876807, gamma=0.17552129365098906, score=0.915, total=   0.4s\n",
      "[CV] C=5.518650428876807, gamma=0.17552129365098906 ..................\n",
      "[CV]  C=5.518650428876807, gamma=0.17552129365098906, score=0.920, total=   0.4s\n",
      "[CV] C=5.518650428876807, gamma=0.17552129365098906 ..................\n",
      "[CV]  C=5.518650428876807, gamma=0.17552129365098906, score=0.970, total=   0.4s\n",
      "[CV] C=5.518650428876807, gamma=0.17552129365098906 ..................\n",
      "[CV]  C=5.518650428876807, gamma=0.17552129365098906, score=0.965, total=   0.4s\n",
      "[CV] C=5.518650428876807, gamma=0.17552129365098906 ..................\n",
      "[CV]  C=5.518650428876807, gamma=0.17552129365098906, score=0.950, total=   0.3s\n",
      "[CV] C=5.518650428876807, gamma=0.17552129365098906 ..................\n",
      "[CV]  C=5.518650428876807, gamma=0.17552129365098906, score=0.855, total=   0.4s\n",
      "[CV] C=5.518650428876807, gamma=0.17552129365098906 ..................\n",
      "[CV]  C=5.518650428876807, gamma=0.17552129365098906, score=0.885, total=   0.3s\n",
      "[CV] C=5.518650428876807, gamma=0.17552129365098906 ..................\n",
      "[CV]  C=5.518650428876807, gamma=0.17552129365098906, score=0.870, total=   0.4s\n",
      "[CV] C=5.518650428876807, gamma=0.17552129365098906 ..................\n",
      "[CV]  C=5.518650428876807, gamma=0.17552129365098906, score=0.870, total=   0.5s\n",
      "[CV] C=5.518650428876807, gamma=0.17552129365098906 ..................\n",
      "[CV]  C=5.518650428876807, gamma=0.17552129365098906, score=0.860, total=   0.4s\n",
      "[CV] C=0.041594627113456614, gamma=0.020209697392933386 ..............\n",
      "[CV]  C=0.041594627113456614, gamma=0.020209697392933386, score=0.910, total=   0.4s\n",
      "[CV] C=0.041594627113456614, gamma=0.020209697392933386 ..............\n",
      "[CV]  C=0.041594627113456614, gamma=0.020209697392933386, score=0.915, total=   0.4s\n",
      "[CV] C=0.041594627113456614, gamma=0.020209697392933386 ..............\n",
      "[CV]  C=0.041594627113456614, gamma=0.020209697392933386, score=0.960, total=   0.4s\n",
      "[CV] C=0.041594627113456614, gamma=0.020209697392933386 ..............\n",
      "[CV]  C=0.041594627113456614, gamma=0.020209697392933386, score=0.965, total=   0.4s\n",
      "[CV] C=0.041594627113456614, gamma=0.020209697392933386 ..............\n",
      "[CV]  C=0.041594627113456614, gamma=0.020209697392933386, score=0.925, total=   0.4s\n",
      "[CV] C=0.041594627113456614, gamma=0.020209697392933386 ..............\n",
      "[CV]  C=0.041594627113456614, gamma=0.020209697392933386, score=0.770, total=   0.4s\n",
      "[CV] C=0.041594627113456614, gamma=0.020209697392933386 ..............\n",
      "[CV]  C=0.041594627113456614, gamma=0.020209697392933386, score=0.810, total=   0.4s\n",
      "[CV] C=0.041594627113456614, gamma=0.020209697392933386 ..............\n",
      "[CV]  C=0.041594627113456614, gamma=0.020209697392933386, score=0.760, total=   0.4s\n",
      "[CV] C=0.041594627113456614, gamma=0.020209697392933386 ..............\n",
      "[CV]  C=0.041594627113456614, gamma=0.020209697392933386, score=0.780, total=   0.3s\n",
      "[CV] C=0.041594627113456614, gamma=0.020209697392933386 ..............\n",
      "[CV]  C=0.041594627113456614, gamma=0.020209697392933386, score=0.775, total=   0.4s\n",
      "[CV] C=55.218609624210195, gamma=0.3675277355380988 ..................\n",
      "[CV]  C=55.218609624210195, gamma=0.3675277355380988, score=0.925, total=   0.6s\n",
      "[CV] C=55.218609624210195, gamma=0.3675277355380988 ..................\n",
      "[CV]  C=55.218609624210195, gamma=0.3675277355380988, score=0.945, total=   0.7s\n",
      "[CV] C=55.218609624210195, gamma=0.3675277355380988 ..................\n",
      "[CV]  C=55.218609624210195, gamma=0.3675277355380988, score=0.960, total=   0.6s\n",
      "[CV] C=55.218609624210195, gamma=0.3675277355380988 ..................\n",
      "[CV]  C=55.218609624210195, gamma=0.3675277355380988, score=0.960, total=   0.6s\n",
      "[CV] C=55.218609624210195, gamma=0.3675277355380988 ..................\n",
      "[CV]  C=55.218609624210195, gamma=0.3675277355380988, score=0.930, total=   0.6s\n",
      "[CV] C=55.218609624210195, gamma=0.3675277355380988 ..................\n",
      "[CV]  C=55.218609624210195, gamma=0.3675277355380988, score=0.850, total=   0.6s\n",
      "[CV] C=55.218609624210195, gamma=0.3675277355380988 ..................\n",
      "[CV]  C=55.218609624210195, gamma=0.3675277355380988, score=0.870, total=   0.6s\n",
      "[CV] C=55.218609624210195, gamma=0.3675277355380988 ..................\n",
      "[CV]  C=55.218609624210195, gamma=0.3675277355380988, score=0.850, total=   0.6s\n",
      "[CV] C=55.218609624210195, gamma=0.3675277355380988 ..................\n",
      "[CV]  C=55.218609624210195, gamma=0.3675277355380988, score=0.870, total=   0.6s\n",
      "[CV] C=55.218609624210195, gamma=0.3675277355380988 ..................\n",
      "[CV]  C=55.218609624210195, gamma=0.3675277355380988, score=0.865, total=   0.6s\n",
      "[CV] C=5.2309068716240725, gamma=18.48512146230226 ...................\n",
      "[CV]  C=5.2309068716240725, gamma=18.48512146230226, score=0.500, total=   1.0s\n",
      "[CV] C=5.2309068716240725, gamma=18.48512146230226 ...................\n",
      "[CV]  C=5.2309068716240725, gamma=18.48512146230226, score=0.500, total=   1.1s\n",
      "[CV] C=5.2309068716240725, gamma=18.48512146230226 ...................\n",
      "[CV]  C=5.2309068716240725, gamma=18.48512146230226, score=0.500, total=   1.0s\n",
      "[CV] C=5.2309068716240725, gamma=18.48512146230226 ...................\n",
      "[CV]  C=5.2309068716240725, gamma=18.48512146230226, score=0.500, total=   1.0s\n",
      "[CV] C=5.2309068716240725, gamma=18.48512146230226 ...................\n",
      "[CV]  C=5.2309068716240725, gamma=18.48512146230226, score=0.500, total=   0.9s\n",
      "[CV] C=5.2309068716240725, gamma=18.48512146230226 ...................\n",
      "[CV]  C=5.2309068716240725, gamma=18.48512146230226, score=0.505, total=   1.2s\n",
      "[CV] C=5.2309068716240725, gamma=18.48512146230226 ...................\n",
      "[CV]  C=5.2309068716240725, gamma=18.48512146230226, score=0.505, total=   0.9s\n",
      "[CV] C=5.2309068716240725, gamma=18.48512146230226 ...................\n",
      "[CV]  C=5.2309068716240725, gamma=18.48512146230226, score=0.505, total=   0.9s\n",
      "[CV] C=5.2309068716240725, gamma=18.48512146230226 ...................\n",
      "[CV]  C=5.2309068716240725, gamma=18.48512146230226, score=0.505, total=   1.0s\n",
      "[CV] C=5.2309068716240725, gamma=18.48512146230226 ...................\n",
      "[CV]  C=5.2309068716240725, gamma=18.48512146230226, score=0.505, total=   0.9s\n",
      "[CV] C=3.998115813506231, gamma=0.6085977830008019 ...................\n",
      "[CV]  C=3.998115813506231, gamma=0.6085977830008019, score=0.935, total=   0.6s\n",
      "[CV] C=3.998115813506231, gamma=0.6085977830008019 ...................\n",
      "[CV]  C=3.998115813506231, gamma=0.6085977830008019, score=0.945, total=   0.6s\n",
      "[CV] C=3.998115813506231, gamma=0.6085977830008019 ...................\n",
      "[CV]  C=3.998115813506231, gamma=0.6085977830008019, score=0.960, total=   0.7s\n",
      "[CV] C=3.998115813506231, gamma=0.6085977830008019 ...................\n",
      "[CV]  C=3.998115813506231, gamma=0.6085977830008019, score=0.950, total=   0.7s\n",
      "[CV] C=3.998115813506231, gamma=0.6085977830008019 ...................\n",
      "[CV]  C=3.998115813506231, gamma=0.6085977830008019, score=0.920, total=   0.6s\n",
      "[CV] C=3.998115813506231, gamma=0.6085977830008019 ...................\n",
      "[CV]  C=3.998115813506231, gamma=0.6085977830008019, score=0.830, total=   0.6s\n",
      "[CV] C=3.998115813506231, gamma=0.6085977830008019 ...................\n",
      "[CV]  C=3.998115813506231, gamma=0.6085977830008019, score=0.850, total=   0.7s\n",
      "[CV] C=3.998115813506231, gamma=0.6085977830008019 ...................\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV]  C=3.998115813506231, gamma=0.6085977830008019, score=0.820, total=   0.6s\n",
      "[CV] C=3.998115813506231, gamma=0.6085977830008019 ...................\n",
      "[CV]  C=3.998115813506231, gamma=0.6085977830008019, score=0.840, total=   0.6s\n",
      "[CV] C=3.998115813506231, gamma=0.6085977830008019 ...................\n",
      "[CV]  C=3.998115813506231, gamma=0.6085977830008019, score=0.845, total=   0.7s\n",
      "[CV] C=0.42886643804112307, gamma=0.01008651146927285 ................\n",
      "[CV]  C=0.42886643804112307, gamma=0.01008651146927285, score=0.920, total=   0.3s\n",
      "[CV] C=0.42886643804112307, gamma=0.01008651146927285 ................\n",
      "[CV]  C=0.42886643804112307, gamma=0.01008651146927285, score=0.925, total=   0.3s\n",
      "[CV] C=0.42886643804112307, gamma=0.01008651146927285 ................\n",
      "[CV]  C=0.42886643804112307, gamma=0.01008651146927285, score=0.970, total=   0.3s\n",
      "[CV] C=0.42886643804112307, gamma=0.01008651146927285 ................\n",
      "[CV]  C=0.42886643804112307, gamma=0.01008651146927285, score=0.975, total=   0.3s\n",
      "[CV] C=0.42886643804112307, gamma=0.01008651146927285 ................\n",
      "[CV]  C=0.42886643804112307, gamma=0.01008651146927285, score=0.930, total=   0.3s\n",
      "[CV] C=0.42886643804112307, gamma=0.01008651146927285 ................\n",
      "[CV]  C=0.42886643804112307, gamma=0.01008651146927285, score=0.805, total=   0.3s\n",
      "[CV] C=0.42886643804112307, gamma=0.01008651146927285 ................\n",
      "[CV]  C=0.42886643804112307, gamma=0.01008651146927285, score=0.815, total=   0.3s\n",
      "[CV] C=0.42886643804112307, gamma=0.01008651146927285 ................\n",
      "[CV]  C=0.42886643804112307, gamma=0.01008651146927285, score=0.785, total=   0.3s\n",
      "[CV] C=0.42886643804112307, gamma=0.01008651146927285 ................\n",
      "[CV]  C=0.42886643804112307, gamma=0.01008651146927285, score=0.820, total=   0.3s\n",
      "[CV] C=0.42886643804112307, gamma=0.01008651146927285 ................\n",
      "[CV]  C=0.42886643804112307, gamma=0.01008651146927285, score=0.795, total=   0.2s\n",
      "[CV] C=0.29000112096988767, gamma=0.1168890911417608 .................\n",
      "[CV]  C=0.29000112096988767, gamma=0.1168890911417608, score=0.930, total=   0.3s\n",
      "[CV] C=0.29000112096988767, gamma=0.1168890911417608 .................\n",
      "[CV]  C=0.29000112096988767, gamma=0.1168890911417608, score=0.965, total=   0.3s\n",
      "[CV] C=0.29000112096988767, gamma=0.1168890911417608 .................\n",
      "[CV]  C=0.29000112096988767, gamma=0.1168890911417608, score=0.975, total=   0.3s\n",
      "[CV] C=0.29000112096988767, gamma=0.1168890911417608 .................\n",
      "[CV]  C=0.29000112096988767, gamma=0.1168890911417608, score=0.980, total=   0.3s\n",
      "[CV] C=0.29000112096988767, gamma=0.1168890911417608 .................\n",
      "[CV]  C=0.29000112096988767, gamma=0.1168890911417608, score=0.955, total=   0.3s\n",
      "[CV] C=0.29000112096988767, gamma=0.1168890911417608 .................\n",
      "[CV]  C=0.29000112096988767, gamma=0.1168890911417608, score=0.860, total=   0.3s\n",
      "[CV] C=0.29000112096988767, gamma=0.1168890911417608 .................\n",
      "[CV]  C=0.29000112096988767, gamma=0.1168890911417608, score=0.905, total=   0.3s\n",
      "[CV] C=0.29000112096988767, gamma=0.1168890911417608 .................\n",
      "[CV]  C=0.29000112096988767, gamma=0.1168890911417608, score=0.880, total=   0.3s\n",
      "[CV] C=0.29000112096988767, gamma=0.1168890911417608 .................\n",
      "[CV]  C=0.29000112096988767, gamma=0.1168890911417608, score=0.870, total=   0.4s\n",
      "[CV] C=0.29000112096988767, gamma=0.1168890911417608 .................\n",
      "[CV]  C=0.29000112096988767, gamma=0.1168890911417608, score=0.850, total=   0.3s\n",
      "[CV] C=0.018706312268733587, gamma=1.3818876145704548 ................\n",
      "[CV]  C=0.018706312268733587, gamma=1.3818876145704548, score=0.500, total=   0.5s\n",
      "[CV] C=0.018706312268733587, gamma=1.3818876145704548 ................\n",
      "[CV]  C=0.018706312268733587, gamma=1.3818876145704548, score=0.500, total=   0.6s\n",
      "[CV] C=0.018706312268733587, gamma=1.3818876145704548 ................\n",
      "[CV]  C=0.018706312268733587, gamma=1.3818876145704548, score=0.500, total=   0.5s\n",
      "[CV] C=0.018706312268733587, gamma=1.3818876145704548 ................\n",
      "[CV]  C=0.018706312268733587, gamma=1.3818876145704548, score=0.500, total=   0.5s\n",
      "[CV] C=0.018706312268733587, gamma=1.3818876145704548 ................\n",
      "[CV]  C=0.018706312268733587, gamma=1.3818876145704548, score=0.500, total=   0.5s\n",
      "[CV] C=0.018706312268733587, gamma=1.3818876145704548 ................\n",
      "[CV]  C=0.018706312268733587, gamma=1.3818876145704548, score=0.505, total=   0.5s\n",
      "[CV] C=0.018706312268733587, gamma=1.3818876145704548 ................\n",
      "[CV]  C=0.018706312268733587, gamma=1.3818876145704548, score=0.505, total=   0.5s\n",
      "[CV] C=0.018706312268733587, gamma=1.3818876145704548 ................\n",
      "[CV]  C=0.018706312268733587, gamma=1.3818876145704548, score=0.505, total=   0.5s\n",
      "[CV] C=0.018706312268733587, gamma=1.3818876145704548 ................\n",
      "[CV]  C=0.018706312268733587, gamma=1.3818876145704548, score=0.505, total=   0.5s\n",
      "[CV] C=0.018706312268733587, gamma=1.3818876145704548 ................\n",
      "[CV]  C=0.018706312268733587, gamma=1.3818876145704548, score=0.505, total=   0.5s\n",
      "[CV] C=0.08426119546444133, gamma=71.80224660735999 ..................\n",
      "[CV]  C=0.08426119546444133, gamma=71.80224660735999, score=0.500, total=   1.2s\n",
      "[CV] C=0.08426119546444133, gamma=71.80224660735999 ..................\n",
      "[CV]  C=0.08426119546444133, gamma=71.80224660735999, score=0.500, total=   1.4s\n",
      "[CV] C=0.08426119546444133, gamma=71.80224660735999 ..................\n",
      "[CV]  C=0.08426119546444133, gamma=71.80224660735999, score=0.500, total=   1.3s\n",
      "[CV] C=0.08426119546444133, gamma=71.80224660735999 ..................\n",
      "[CV]  C=0.08426119546444133, gamma=71.80224660735999, score=0.500, total=   1.3s\n",
      "[CV] C=0.08426119546444133, gamma=71.80224660735999 ..................\n",
      "[CV]  C=0.08426119546444133, gamma=71.80224660735999, score=0.500, total=   1.3s\n",
      "[CV] C=0.08426119546444133, gamma=71.80224660735999 ..................\n",
      "[CV]  C=0.08426119546444133, gamma=71.80224660735999, score=0.505, total=   1.3s\n",
      "[CV] C=0.08426119546444133, gamma=71.80224660735999 ..................\n",
      "[CV]  C=0.08426119546444133, gamma=71.80224660735999, score=0.505, total=   1.3s\n",
      "[CV] C=0.08426119546444133, gamma=71.80224660735999 ..................\n",
      "[CV]  C=0.08426119546444133, gamma=71.80224660735999, score=0.505, total=   1.3s\n",
      "[CV] C=0.08426119546444133, gamma=71.80224660735999 ..................\n",
      "[CV]  C=0.08426119546444133, gamma=71.80224660735999, score=0.505, total=   1.3s\n",
      "[CV] C=0.08426119546444133, gamma=71.80224660735999 ..................\n",
      "[CV]  C=0.08426119546444133, gamma=71.80224660735999, score=0.505, total=   1.3s\n",
      "[CV] C=27.152035452154053, gamma=50.98410456804878 ...................\n",
      "[CV]  C=27.152035452154053, gamma=50.98410456804878, score=0.500, total=   1.3s\n",
      "[CV] C=27.152035452154053, gamma=50.98410456804878 ...................\n",
      "[CV]  C=27.152035452154053, gamma=50.98410456804878, score=0.500, total=   1.3s\n",
      "[CV] C=27.152035452154053, gamma=50.98410456804878 ...................\n",
      "[CV]  C=27.152035452154053, gamma=50.98410456804878, score=0.500, total=   1.3s\n",
      "[CV] C=27.152035452154053, gamma=50.98410456804878 ...................\n",
      "[CV]  C=27.152035452154053, gamma=50.98410456804878, score=0.500, total=   1.3s\n",
      "[CV] C=27.152035452154053, gamma=50.98410456804878 ...................\n",
      "[CV]  C=27.152035452154053, gamma=50.98410456804878, score=0.500, total=   1.3s\n",
      "[CV] C=27.152035452154053, gamma=50.98410456804878 ...................\n",
      "[CV]  C=27.152035452154053, gamma=50.98410456804878, score=0.505, total=   1.3s\n",
      "[CV] C=27.152035452154053, gamma=50.98410456804878 ...................\n",
      "[CV]  C=27.152035452154053, gamma=50.98410456804878, score=0.505, total=   1.3s\n",
      "[CV] C=27.152035452154053, gamma=50.98410456804878 ...................\n",
      "[CV]  C=27.152035452154053, gamma=50.98410456804878, score=0.505, total=   1.5s\n",
      "[CV] C=27.152035452154053, gamma=50.98410456804878 ...................\n",
      "[CV]  C=27.152035452154053, gamma=50.98410456804878, score=0.505, total=   1.3s\n",
      "[CV] C=27.152035452154053, gamma=50.98410456804878 ...................\n",
      "[CV]  C=27.152035452154053, gamma=50.98410456804878, score=0.505, total=   1.3s\n",
      "[CV] C=0.3847152048096837, gamma=1.5624727057871528 ..................\n",
      "[CV]  C=0.3847152048096837, gamma=1.5624727057871528, score=0.500, total=   0.5s\n",
      "[CV] C=0.3847152048096837, gamma=1.5624727057871528 ..................\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV]  C=0.3847152048096837, gamma=1.5624727057871528, score=0.500, total=   0.5s\n",
      "[CV] C=0.3847152048096837, gamma=1.5624727057871528 ..................\n",
      "[CV]  C=0.3847152048096837, gamma=1.5624727057871528, score=0.500, total=   0.6s\n",
      "[CV] C=0.3847152048096837, gamma=1.5624727057871528 ..................\n",
      "[CV]  C=0.3847152048096837, gamma=1.5624727057871528, score=0.500, total=   0.5s\n",
      "[CV] C=0.3847152048096837, gamma=1.5624727057871528 ..................\n",
      "[CV]  C=0.3847152048096837, gamma=1.5624727057871528, score=0.500, total=   0.6s\n",
      "[CV] C=0.3847152048096837, gamma=1.5624727057871528 ..................\n",
      "[CV]  C=0.3847152048096837, gamma=1.5624727057871528, score=0.505, total=   0.6s\n",
      "[CV] C=0.3847152048096837, gamma=1.5624727057871528 ..................\n",
      "[CV]  C=0.3847152048096837, gamma=1.5624727057871528, score=0.505, total=   0.5s\n",
      "[CV] C=0.3847152048096837, gamma=1.5624727057871528 ..................\n",
      "[CV]  C=0.3847152048096837, gamma=1.5624727057871528, score=0.505, total=   0.6s\n",
      "[CV] C=0.3847152048096837, gamma=1.5624727057871528 ..................\n",
      "[CV]  C=0.3847152048096837, gamma=1.5624727057871528, score=0.505, total=   0.6s\n",
      "[CV] C=0.3847152048096837, gamma=1.5624727057871528 ..................\n",
      "[CV]  C=0.3847152048096837, gamma=1.5624727057871528, score=0.505, total=   0.5s\n",
      "[CV] C=0.11037174111819205, gamma=12.308020649384025 .................\n",
      "[CV]  C=0.11037174111819205, gamma=12.308020649384025, score=0.500, total=   0.8s\n",
      "[CV] C=0.11037174111819205, gamma=12.308020649384025 .................\n",
      "[CV]  C=0.11037174111819205, gamma=12.308020649384025, score=0.500, total=   0.7s\n",
      "[CV] C=0.11037174111819205, gamma=12.308020649384025 .................\n",
      "[CV]  C=0.11037174111819205, gamma=12.308020649384025, score=0.500, total=   0.7s\n",
      "[CV] C=0.11037174111819205, gamma=12.308020649384025 .................\n",
      "[CV]  C=0.11037174111819205, gamma=12.308020649384025, score=0.500, total=   0.6s\n",
      "[CV] C=0.11037174111819205, gamma=12.308020649384025 .................\n",
      "[CV]  C=0.11037174111819205, gamma=12.308020649384025, score=0.500, total=   0.7s\n",
      "[CV] C=0.11037174111819205, gamma=12.308020649384025 .................\n",
      "[CV]  C=0.11037174111819205, gamma=12.308020649384025, score=0.505, total=   0.7s\n",
      "[CV] C=0.11037174111819205, gamma=12.308020649384025 .................\n",
      "[CV]  C=0.11037174111819205, gamma=12.308020649384025, score=0.505, total=   0.7s\n",
      "[CV] C=0.11037174111819205, gamma=12.308020649384025 .................\n",
      "[CV]  C=0.11037174111819205, gamma=12.308020649384025, score=0.505, total=   0.7s\n",
      "[CV] C=0.11037174111819205, gamma=12.308020649384025 .................\n",
      "[CV]  C=0.11037174111819205, gamma=12.308020649384025, score=0.505, total=   0.7s\n",
      "[CV] C=0.11037174111819205, gamma=12.308020649384025 .................\n",
      "[CV]  C=0.11037174111819205, gamma=12.308020649384025, score=0.505, total=   0.7s\n",
      "[CV] C=0.22904284674804973, gamma=9.040750845541929 ..................\n",
      "[CV]  C=0.22904284674804973, gamma=9.040750845541929, score=0.500, total=   0.6s\n",
      "[CV] C=0.22904284674804973, gamma=9.040750845541929 ..................\n",
      "[CV]  C=0.22904284674804973, gamma=9.040750845541929, score=0.500, total=   0.6s\n",
      "[CV] C=0.22904284674804973, gamma=9.040750845541929 ..................\n",
      "[CV]  C=0.22904284674804973, gamma=9.040750845541929, score=0.500, total=   0.6s\n",
      "[CV] C=0.22904284674804973, gamma=9.040750845541929 ..................\n",
      "[CV]  C=0.22904284674804973, gamma=9.040750845541929, score=0.500, total=   0.7s\n",
      "[CV] C=0.22904284674804973, gamma=9.040750845541929 ..................\n",
      "[CV]  C=0.22904284674804973, gamma=9.040750845541929, score=0.500, total=   0.6s\n",
      "[CV] C=0.22904284674804973, gamma=9.040750845541929 ..................\n",
      "[CV]  C=0.22904284674804973, gamma=9.040750845541929, score=0.505, total=   0.7s\n",
      "[CV] C=0.22904284674804973, gamma=9.040750845541929 ..................\n",
      "[CV]  C=0.22904284674804973, gamma=9.040750845541929, score=0.505, total=   0.6s\n",
      "[CV] C=0.22904284674804973, gamma=9.040750845541929 ..................\n",
      "[CV]  C=0.22904284674804973, gamma=9.040750845541929, score=0.505, total=   0.6s\n",
      "[CV] C=0.22904284674804973, gamma=9.040750845541929 ..................\n",
      "[CV]  C=0.22904284674804973, gamma=9.040750845541929, score=0.505, total=   0.7s\n",
      "[CV] C=0.22904284674804973, gamma=9.040750845541929 ..................\n",
      "[CV]  C=0.22904284674804973, gamma=9.040750845541929, score=0.505, total=   0.6s\n",
      "[CV] C=23.96237684089462, gamma=5.761150299153378 ....................\n",
      "[CV]  C=23.96237684089462, gamma=5.761150299153378, score=0.505, total=   0.6s\n",
      "[CV] C=23.96237684089462, gamma=5.761150299153378 ....................\n",
      "[CV]  C=23.96237684089462, gamma=5.761150299153378, score=0.505, total=   0.6s\n",
      "[CV] C=23.96237684089462, gamma=5.761150299153378 ....................\n",
      "[CV]  C=23.96237684089462, gamma=5.761150299153378, score=0.500, total=   0.6s\n",
      "[CV] C=23.96237684089462, gamma=5.761150299153378 ....................\n",
      "[CV]  C=23.96237684089462, gamma=5.761150299153378, score=0.500, total=   0.6s\n",
      "[CV] C=23.96237684089462, gamma=5.761150299153378 ....................\n",
      "[CV]  C=23.96237684089462, gamma=5.761150299153378, score=0.500, total=   0.6s\n",
      "[CV] C=23.96237684089462, gamma=5.761150299153378 ....................\n",
      "[CV]  C=23.96237684089462, gamma=5.761150299153378, score=0.505, total=   0.7s\n",
      "[CV] C=23.96237684089462, gamma=5.761150299153378 ....................\n",
      "[CV]  C=23.96237684089462, gamma=5.761150299153378, score=0.505, total=   0.6s\n",
      "[CV] C=23.96237684089462, gamma=5.761150299153378 ....................\n",
      "[CV]  C=23.96237684089462, gamma=5.761150299153378, score=0.505, total=   0.6s\n",
      "[CV] C=23.96237684089462, gamma=5.761150299153378 ....................\n",
      "[CV]  C=23.96237684089462, gamma=5.761150299153378, score=0.505, total=   0.6s\n",
      "[CV] C=23.96237684089462, gamma=5.761150299153378 ....................\n",
      "[CV]  C=23.96237684089462, gamma=5.761150299153378, score=0.505, total=   0.6s\n",
      "[CV] C=43.706038535344085, gamma=6.064752410306777 ...................\n",
      "[CV]  C=43.706038535344085, gamma=6.064752410306777, score=0.505, total=   0.6s\n",
      "[CV] C=43.706038535344085, gamma=6.064752410306777 ...................\n",
      "[CV]  C=43.706038535344085, gamma=6.064752410306777, score=0.505, total=   0.6s\n",
      "[CV] C=43.706038535344085, gamma=6.064752410306777 ...................\n",
      "[CV]  C=43.706038535344085, gamma=6.064752410306777, score=0.500, total=   0.6s\n",
      "[CV] C=43.706038535344085, gamma=6.064752410306777 ...................\n",
      "[CV]  C=43.706038535344085, gamma=6.064752410306777, score=0.500, total=   0.6s\n",
      "[CV] C=43.706038535344085, gamma=6.064752410306777 ...................\n",
      "[CV]  C=43.706038535344085, gamma=6.064752410306777, score=0.500, total=   0.6s\n",
      "[CV] C=43.706038535344085, gamma=6.064752410306777 ...................\n",
      "[CV]  C=43.706038535344085, gamma=6.064752410306777, score=0.505, total=   0.7s\n",
      "[CV] C=43.706038535344085, gamma=6.064752410306777 ...................\n",
      "[CV]  C=43.706038535344085, gamma=6.064752410306777, score=0.505, total=   0.7s\n",
      "[CV] C=43.706038535344085, gamma=6.064752410306777 ...................\n",
      "[CV]  C=43.706038535344085, gamma=6.064752410306777, score=0.505, total=   0.7s\n",
      "[CV] C=43.706038535344085, gamma=6.064752410306777 ...................\n",
      "[CV]  C=43.706038535344085, gamma=6.064752410306777, score=0.505, total=   0.7s\n",
      "[CV] C=43.706038535344085, gamma=6.064752410306777 ...................\n",
      "[CV]  C=43.706038535344085, gamma=6.064752410306777, score=0.505, total=   0.7s\n",
      "[CV] C=0.04463639175731548, gamma=0.21957197036133624 ................\n",
      "[CV]  C=0.04463639175731548, gamma=0.21957197036133624, score=0.920, total=   0.5s\n",
      "[CV] C=0.04463639175731548, gamma=0.21957197036133624 ................\n",
      "[CV]  C=0.04463639175731548, gamma=0.21957197036133624, score=0.935, total=   0.6s\n",
      "[CV] C=0.04463639175731548, gamma=0.21957197036133624 ................\n",
      "[CV]  C=0.04463639175731548, gamma=0.21957197036133624, score=0.935, total=   0.5s\n",
      "[CV] C=0.04463639175731548, gamma=0.21957197036133624 ................\n",
      "[CV]  C=0.04463639175731548, gamma=0.21957197036133624, score=0.985, total=   0.6s\n",
      "[CV] C=0.04463639175731548, gamma=0.21957197036133624 ................\n",
      "[CV]  C=0.04463639175731548, gamma=0.21957197036133624, score=0.925, total=   0.5s\n",
      "[CV] C=0.04463639175731548, gamma=0.21957197036133624 ................\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV]  C=0.04463639175731548, gamma=0.21957197036133624, score=0.825, total=   0.5s\n",
      "[CV] C=0.04463639175731548, gamma=0.21957197036133624 ................\n",
      "[CV]  C=0.04463639175731548, gamma=0.21957197036133624, score=0.810, total=   0.5s\n",
      "[CV] C=0.04463639175731548, gamma=0.21957197036133624 ................\n",
      "[CV]  C=0.04463639175731548, gamma=0.21957197036133624, score=0.810, total=   0.5s\n",
      "[CV] C=0.04463639175731548, gamma=0.21957197036133624 ................\n",
      "[CV]  C=0.04463639175731548, gamma=0.21957197036133624, score=0.830, total=   0.5s\n",
      "[CV] C=0.04463639175731548, gamma=0.21957197036133624 ................\n",
      "[CV]  C=0.04463639175731548, gamma=0.21957197036133624, score=0.795, total=   0.5s\n",
      "[CV] C=0.047510626135955775, gamma=0.07612942514991188 ...............\n",
      "[CV]  C=0.047510626135955775, gamma=0.07612942514991188, score=0.915, total=   0.4s\n",
      "[CV] C=0.047510626135955775, gamma=0.07612942514991188 ...............\n",
      "[CV]  C=0.047510626135955775, gamma=0.07612942514991188, score=0.925, total=   0.4s\n",
      "[CV] C=0.047510626135955775, gamma=0.07612942514991188 ...............\n",
      "[CV]  C=0.047510626135955775, gamma=0.07612942514991188, score=0.970, total=   0.4s\n",
      "[CV] C=0.047510626135955775, gamma=0.07612942514991188 ...............\n",
      "[CV]  C=0.047510626135955775, gamma=0.07612942514991188, score=0.985, total=   0.4s\n",
      "[CV] C=0.047510626135955775, gamma=0.07612942514991188 ...............\n",
      "[CV]  C=0.047510626135955775, gamma=0.07612942514991188, score=0.935, total=   0.4s\n",
      "[CV] C=0.047510626135955775, gamma=0.07612942514991188 ...............\n",
      "[CV]  C=0.047510626135955775, gamma=0.07612942514991188, score=0.795, total=   0.4s\n",
      "[CV] C=0.047510626135955775, gamma=0.07612942514991188 ...............\n",
      "[CV]  C=0.047510626135955775, gamma=0.07612942514991188, score=0.825, total=   0.3s\n",
      "[CV] C=0.047510626135955775, gamma=0.07612942514991188 ...............\n",
      "[CV]  C=0.047510626135955775, gamma=0.07612942514991188, score=0.785, total=   0.4s\n",
      "[CV] C=0.047510626135955775, gamma=0.07612942514991188 ...............\n",
      "[CV]  C=0.047510626135955775, gamma=0.07612942514991188, score=0.835, total=   0.4s\n",
      "[CV] C=0.047510626135955775, gamma=0.07612942514991188 ...............\n",
      "[CV]  C=0.047510626135955775, gamma=0.07612942514991188, score=0.795, total=   0.4s\n",
      "[CV] C=0.02717296795001801, gamma=0.012391614992354834 ...............\n",
      "[CV]  C=0.02717296795001801, gamma=0.012391614992354834, score=0.905, total=   0.5s\n",
      "[CV] C=0.02717296795001801, gamma=0.012391614992354834 ...............\n",
      "[CV]  C=0.02717296795001801, gamma=0.012391614992354834, score=0.895, total=   0.4s\n",
      "[CV] C=0.02717296795001801, gamma=0.012391614992354834 ...............\n",
      "[CV]  C=0.02717296795001801, gamma=0.012391614992354834, score=0.945, total=   0.5s\n",
      "[CV] C=0.02717296795001801, gamma=0.012391614992354834 ...............\n",
      "[CV]  C=0.02717296795001801, gamma=0.012391614992354834, score=0.960, total=   0.5s\n",
      "[CV] C=0.02717296795001801, gamma=0.012391614992354834 ...............\n",
      "[CV]  C=0.02717296795001801, gamma=0.012391614992354834, score=0.915, total=   0.5s\n",
      "[CV] C=0.02717296795001801, gamma=0.012391614992354834 ...............\n",
      "[CV]  C=0.02717296795001801, gamma=0.012391614992354834, score=0.765, total=   0.5s\n",
      "[CV] C=0.02717296795001801, gamma=0.012391614992354834 ...............\n",
      "[CV]  C=0.02717296795001801, gamma=0.012391614992354834, score=0.795, total=   0.5s\n",
      "[CV] C=0.02717296795001801, gamma=0.012391614992354834 ...............\n",
      "[CV]  C=0.02717296795001801, gamma=0.012391614992354834, score=0.725, total=   0.5s\n",
      "[CV] C=0.02717296795001801, gamma=0.012391614992354834 ...............\n",
      "[CV]  C=0.02717296795001801, gamma=0.012391614992354834, score=0.735, total=   0.5s\n",
      "[CV] C=0.02717296795001801, gamma=0.012391614992354834 ...............\n",
      "[CV]  C=0.02717296795001801, gamma=0.012391614992354834, score=0.740, total=   0.5s\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Done 250 out of 250 | elapsed:  2.6min finished\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "IS Best Score Sharpe Ratio: 19.242468\n",
      "Best IS Sharpe ratio: 219.745415\n",
      "Lowest IS Sharpe Ratio: 9.359770\n",
      "Mean Sharpe Ratio: 106.661255\n"
     ]
    }
   ],
   "source": [
    "Rclf1 = RandomizedSearchCV(estimator = clf_svm, \n",
    "                          param_distributions = param_distributions,\n",
    "                          n_iter = 25, # newly added as well\n",
    "                          scoring='accuracy', \n",
    "                          n_jobs=None, \n",
    "                          #iid='deprecated', take note of depreciation! version 0.22 warning\n",
    "                          refit=True,\n",
    "                          cv= n_splits, # 10 fold CV, but! stratified KFold\n",
    "                          verbose=10)\n",
    "\n",
    "Rclf1.fit(X,\n",
    "         y['bin'],\n",
    "         sample_weight = None) #sample_weight here!!\n",
    "\n",
    "in_sample_sharpe_ratio(Rclf1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Scoring Methods Comparison\n",
    "\n",
    "To ensure fairness in comparing across both methods and scoring type. We will only take the \"best\" score given provided and lowest score that we can find.\n",
    "\n",
    "When using RSCV, remember to provide random_state. Otherwise, your answers will keep changing.\n",
    "\n",
    "GSCV (Neg_log_loss):\n",
    "\n",
    "In-Sample Best Score Sharpe Ratio: 2.487544 (Lowest)\n",
    "\n",
    "Lowest In-Sample Sharpe Ratio: 2.194502\n",
    "\n",
    "GSCV (Accuracy):\n",
    "\n",
    "In-Sample Best Score Sharpe Ratio: 21.373090\n",
    "\n",
    "Lowest In-Sample Sharpe Ratio: 8.064150\n",
    "\n",
    "RSCV (Neg_log_loss):\n",
    "\n",
    "In-Sample Best Score Sharpe Ratio: 2.509210\n",
    "\n",
    "Lowest In-Sample Sharpe Ratio: 2.175036 (Lowest)\n",
    "\n",
    "RSCV (Accuracy):\n",
    "\n",
    "In-Sample Best Score Sharpe Ratio: 23.442070\n",
    "\n",
    "Lowest In-Sample Sharpe Ratio: 6.252751\n",
    "\n",
    "**At-a-glance**\n",
    "\n",
    "GridSearchCV (Neg_log_loss) produces a lowest sharpe ratio based on the best score provided.\n",
    "\n",
    "However, when comparing with RSCV (Neg_log_loss) lowest in-sample sharpe ratio. It seems higher.\n",
    "\n",
    "This demostrates a consistence in sharpe ratio.\n",
    "\n",
    "This characteristic is carried over when using \"accuracy\" as a scoring method (RSCV has a lower IS SR: 6.252751).\n",
    "\n",
    "When comparing between \"accuracy\" and \"neg_log_loss\", it is very obvious accuracy provides a very high sharpe ratio.\n",
    "\n",
    "But it is even more obvious \"accuracy\" has a large difference between best and lowest SR.\n",
    "\n",
    "It seems that \"accuracy\" as a scoring method is not very accurate.\n",
    "\n",
    "This is also mentioned in AFML page 134, section 9.4.\n",
    "\n",
    "### Conclusion\n",
    "\n",
    "In short, strongly encourage cross-entropy (neg_log_loss) as a scoring method.\n",
    "\n",
    "Refer to the below negative log loss formula (Binary classes):\n",
    "\n",
    "![logo](http://wiki.fast.ai/images/2/28/Cross_entropy_formula.png)\n",
    "    \n",
    "For natural logarithm if p was to be < 1. The answer would always be negative. Hence to make it let less confusing and comparable to other metrics, we adopt negative sign (Minimize negative log loss).\n",
    "\n",
    "If you maximise log loss instead (Remove negative sign only/ Maximum log likelihood), your probability would be opposite of the below. Your prediction probability will keep increasing rapidly (Instead of penalizing errorrous prediction, it will end up endorsing such mistakes).\n",
    "\n",
    "![log_loss](http://wiki.fast.ai/images/4/43/Log_loss_graph.png)\n",
    "\n",
    "When you have a strategy that bets equally. Accuracy may prove to be bad choice since they attribute true prediction with erroreous prediction equally (your predicted value equals the actual value). \n",
    "\n",
    "Under Kelly's criterion, if your lose rate is 50/50 it will lead to a quick ruination.(Recall 2p-1)\n",
    "\n",
    "If sample weights were included, accuracy may be considered. In most cases cross-entropy is a better choice, since it will factor in the uncertainty based on how much it varies from actual label.\n",
    "\n",
    "Update: Up till this stage, I realised Dr Marco teaches deep technical knowledge with useful resources to develop further understanding. But he only ask key conceptual questions."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}