FrozenLake_v0_epsilon_greedy_Q_Learning.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## FrozenLake-V0\n",
    "#### $\\epsilon$-greedy $Q$-Learning approach to solve FrozenLake-v0 enivronment from OpenAI-Gym  \n",
    "\n",
    "***FrozenLake-v0 environment***\n",
    "\n",
    "The agent controls the movement of a character in a grid world. Some tiles of the grid are walkable, and others lead to the agent falling into the water. Additionally, the movement direction of the agent is uncertain and only partially depends on the chosen direction. The agent is rewarded for finding a walkable path to a goal tile.\n",
    "\n",
    "Winter is here. You and your friends were tossing around a frisbee at the park when you made a wild throw that left the frisbee out in the middle of the lake. The water is mostly frozen, but there are a few holes where the ice has melted. If you step into one of those holes, you'll fall into the freezing water. At this time, there's an international frisbee shortage, so it's absolutely imperative that you navigate across the lake and retrieve the disc. However, the ice is slippery, so you won't always move in the direction you intend.\n",
    "\n",
    "The surface is described using a grid like the following:\n",
    "```\n",
    "SFFF       (S: starting point, safe)\n",
    "FHFH       (F: frozen surface, safe)\n",
    "FFFH       (H: hole, fall to your doom)\n",
    "HFFG       (G: goal, where the frisbee is located)\n",
    "\n",
    "```\n",
    "The episode ends when you reach the goal or fall in a hole. You receive a reward of 1 if you reach the goal, and zero otherwise.  \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python2.7/site-packages/matplotlib/font_manager.py:273: UserWarning: Matplotlib is building the font cache using fc-list. This may take a moment.\n",
      "  warnings.warn('Matplotlib is building the font cache using fc-list. This may take a moment.')\n"
     ]
    }
   ],
   "source": [
    "import gym\n",
    "from gym import wrappers\n",
    "from matplotlib import pyplot as plt\n",
    "import json\n",
    "from os import listdir\n",
    "\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "http://www.cs.cmu.edu/afs/cs/project/jair/pub/volume4/kaelbling96a.pdf\n",
    "\n",
    "***$Q$-Learning :***\n",
    "\n",
    "```\n",
    "                                      - - - < -  Action  - - < - - -\n",
    "                                     |                              | \n",
    "                                     |                              | \n",
    "                                _____________                    _______\n",
    "                               | Environment |                  | Agent |\n",
    "                               |_____________|                  |_______|\n",
    "                                     |                              |\n",
    "                                     |                              |\n",
    "                                      - - - > -  State  - - > - - -  \n",
    "\n",
    "```\n",
    "Consider an environment which has a vehicle that wants to reach a known location. An agent helps the vehicle in reaching its desitnation by providing instructions. After following the instruction the vehicle sends back its current state in the environment to the agent, so that the agent can make next decision.\n",
    "Lets call an instruction as an action and vehicles current state in the environment as a state. An action could be as simple as move up, down, right or left. A state can be combination of gps coordinates, sensory information from the vehicle like pixel data from camera, lidar information etc.\n",
    "\n",
    "In order to provide actions to the vehicle the agent should either be intelligent or should fake it until it becomes intelligent, that is, may be to start with it can provide random actions, learn based on the feedback from the environment and finally refine its learning.\n",
    "There are many ways for an agent to learn, Reinforcement Learning is one of them. The figure below depicts what happens in Reinforcement learning. Everything that has been explained above stays the same about environment and agent, but there is an additional input that an agent receives from the environment, that is, Reward.  \n",
    "\n",
    "```\n",
    "                                  - - - < -  Action  - - < - - -\n",
    "                                 |                              | \n",
    "                                 |                              | \n",
    "                            _____________                    _______\n",
    "                           | Environment |                  | Agent |\n",
    "                           |_____________|                  |_______|\n",
    "                                 |  |                          |  |\n",
    "                                 |  |                          |  |\n",
    "                                 |   - - > - - Reward - - > - -   |\n",
    "                                 |                                |                                 \n",
    "                                  - - > - - - State  - - > - - - - \n",
    "\n",
    "```\n",
    "A reward can be considered as a feedback, but more of a way to motivate the agent to learn to do the right thing. For example agent would receive a positive reward if the action instructed by the agent has moved the vehicle to next state safely without causing an accident and a negative reward if it has caused an accident.\n",
    "\n",
    "```\n",
    "                      - - - < -  Action  - - < - - -\n",
    "                     |                              | \n",
    "                     |                              | \n",
    "                _____________                    _______                  _______ \n",
    "               | Environment |                  | Agent |- < - - - - > - |Q-Table|\n",
    "               |_____________|                  |_______|                |_______|\n",
    "                     |  |                          |  |\n",
    "                     |  |                          |  |\n",
    "                     |   - - > - - Reward - - > - -   |\n",
    "                     |                                |                                 \n",
    "                      - - > - - - State  - - > - - - - \n",
    "\n",
    "                                          Q-Learning \n",
    "```\n",
    "\n",
    "$Q$-Learning is a basic Reinforcement learning techinique. It is simple and easy to implement, especially if our states are discrete and the state space is small. To start with the agent provides random actions as input to the environment and bulids a table which eventually helps it in making the right decisions. The table is called as a Q table. The table would look similar to the one below.\n",
    "\n",
    "|States|Action,Q-value pairs\n",
    "|------|--------|---------\n",
    "|State-1| {'Action-1': -20, 'Action-2': 10}\n",
    "|State-2| {'Action-1': 20, 'Action-2': 0}\n",
    "\n",
    "The table has two columns, namely, states and action,Q-value pairs. Each row in the table has information about a unique state and its corresponding action Q-value pairs, for all possible actions when the vehicle is in that state. The Q values are values that are learned by the agent from the feedback provided by the environment. \n",
    "\n",
    "Assume that the agent has a Q Table as above and it wants to send an input to vehicle which is at 'state-1'. The agent first looks at all the action Q value pairs for that state, then chooses the action with highest Q value and sends it to the vehicle. The vehicle then sends back a reward and next state so that the agent can update the Q value for the action. The equation for updating the Q value is given by\n",
    "\n",
    "$$ Q(s,a) = R(s,a) + \\gamma(\\max_{a'}Q(s',a')) $$\n",
    "\n",
    "We want to learn the Q value over time, the Q value learning rule or the $Q$-Learning rule is give by :\n",
    "\n",
    "$$ Q(s,a) := Q(s,a) + \\alpha(R(s,a) + \\gamma(\\max_{a'}Q(s',a')) - Q(s,a)) $$\n",
    "\n",
    "Where   \n",
    "  $Q$ is $Q$ function  \n",
    "  $R$ is reward function   \n",
    "  $\\alpha$ is Learning rate      \n",
    "  $\\gamma$ is Discount factor  \n",
    "  $s$ is current state  \n",
    "  $s'$ is future state  \n",
    "  $a$ is current action  \n",
    "  $a'$ is future action  \n",
    "  \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import random\n",
    "import math\n",
    "\n",
    "RESULTS_PATH = 'tmp/results'\n",
    "\n",
    "class Agent(object):\n",
    "    \"\"\" An agent that learns to balance the CartPole \"\"\" \n",
    "\n",
    "    def __init__(self,env, learning, epsilon=1.0, gamma=0.5, alpha=0.5, valid_actions=[]):\n",
    "\n",
    "        # Paramters for the agent \n",
    "        self.env = env \n",
    "        self.valid_actions = valid_actions\n",
    "        self.learning = learning # Whether the agent is expected to learn\n",
    "        self.Q = dict()          # Create a Q-table which will be a dictionary of tuples\n",
    "        self.epsilon = epsilon   # Random exploration factor\n",
    "        self.alpha = alpha       # Learning factor\n",
    "        self.gamma = gamma       # Discount factor\n",
    "        self.current_state = env.reset() # Current state of the agent\n",
    "        self.done = False        # Whether it is time to reset the environment\n",
    "\n",
    "        self.epsilon_0 = epsilon # initial value of epsilon\n",
    "        self.alpha_0 = alpha     # initial value of alpha\n",
    "        self.total_trail = 1\n",
    "\n",
    "    def reset(self,testing=False):\n",
    "        \"\"\" The reset function is called at the beginning of each trial.\n",
    "            'testing' is set to True if testing trials are being used\n",
    "            once training trials have completed. \"\"\"\n",
    "\n",
    "        self.current_state = self.env.reset()\n",
    "\n",
    "        # Decay function for epsilon\n",
    "        constant = 0.0001\n",
    "\n",
    "        self.epsilon =  self.epsilon_0 * math.exp(-constant*self.total_trail)\n",
    "        #self.epsilon -= 0.0001\n",
    "        \n",
    "\n",
    "        # Decay function for alpha (not usual to implement this)\n",
    "        #self.alpha = self.alpha_0 * math.exp(-constant*self.total_trail)\n",
    "        self.alpha = max(self.alpha-0.001, 0.05 )\n",
    "\n",
    "        self.total_trail += 1\n",
    "\n",
    "        # change params for testing \n",
    "\n",
    "    def build_state(self):\n",
    "        \"\"\" The build_state function is called when the agent requests data from the \n",
    "            environment. \"\"\"\n",
    "\n",
    "        return self.current_state\n",
    "\n",
    "\n",
    "    def get_maxQ(self, state):\n",
    "        \"\"\" The get_max_Q function is called when the agent is asked to find the\n",
    "            maximum Q-value of all actions based on the 'state' the agent is in. \"\"\"\n",
    "\n",
    "        maxQ = max(self.Q[state].values())\n",
    "\n",
    "        return maxQ\n",
    "\n",
    "    def createQ(self, state):\n",
    "        \"\"\" The createQ function is called when a state is generated by the agent. \"\"\"\n",
    "        if state in self.Q:\n",
    "            pass\n",
    "        else :\n",
    "            self.Q[state] = dict(zip(self.valid_actions, [0] * len(self.valid_actions)))\n",
    "\n",
    "    def choose_action(self, state):\n",
    "        \"\"\" The choose_action function is called when the agent is asked to choose\n",
    "            which action to take, based on the 'state' the agent is in. \"\"\"\n",
    "\n",
    "\n",
    "        if not self.learning:\n",
    "            action = self.env.action_space.sample()\n",
    "            return action\n",
    "\n",
    "        if random.random() < self.epsilon:\n",
    "            action = self.env.action_space.sample()\n",
    "\n",
    "        else :\n",
    "            maxQ = self.get_maxQ(state)\n",
    "            action = random.choice([key for key in self.Q[state] if self.Q[state][key]==maxQ])\n",
    "            \n",
    "        return action\n",
    "\n",
    "\n",
    "    def learn(self, state, action, reward):\n",
    "        \"\"\" The learn function is called after the agent completes an action and\n",
    "            receives an award. This function does not consider future rewards \n",
    "            when conducting learning. \"\"\"\n",
    "\n",
    "        # Qnew = (1-alpha)*Qold + alpha*(gamma*max(Qnext))\n",
    "        try :\n",
    "            # If Q table already has current_state \n",
    "            self.Q[state][action] = self.alpha*(reward - self.Q[state][action] + \\\n",
    "                    (self.gamma*self.get_maxQ(self.current_state))) + self.Q[state][action]\n",
    "        except :\n",
    "            self.Q[state][action] = (self.alpha*reward) + ((1 - self.alpha)*self.Q[state][action])\n",
    "        \n",
    "\n",
    "\n",
    "    def update(self):\n",
    "        \"\"\" The update function is called when a time step is completed in the \n",
    "            environment for a given trial. This function will build the agent\n",
    "            state, choose an action, receive a reward, and learn if enabled. \"\"\"\n",
    "\n",
    "        state = self.build_state()          # Get current state\n",
    "        self.createQ(state)                 # Create 'state' in Q-table\n",
    "        action = self.choose_action(state)  # Choose an action\n",
    "\n",
    "        self.current_state, reward, self.done, info = self.env.step(action)\n",
    "        \n",
    "        self.learn(state, action, reward)   # Q-learn\n",
    "\n",
    "        return\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[2017-03-08 19:55:43,060] Making new env: FrozenLake-v0\n",
      "[2017-03-08 19:55:43,157] Clearing 30 monitor files from previous run (because force=True was provided)\n",
      "[2017-03-08 19:55:43,161] Starting new video recorder writing to /Users/mmuppidi/Documents/Personal/Projects/ML/reinforcement_learning/tmp/results/openaigym.video.0.19089.video000000.json\n",
      "[2017-03-08 19:55:43,164] Starting new video recorder writing to /Users/mmuppidi/Documents/Personal/Projects/ML/reinforcement_learning/tmp/results/openaigym.video.0.19089.video000001.json\n",
      "[2017-03-08 19:55:43,169] Starting new video recorder writing to /Users/mmuppidi/Documents/Personal/Projects/ML/reinforcement_learning/tmp/results/openaigym.video.0.19089.video000008.json\n",
      "[2017-03-08 19:55:43,176] Starting new video recorder writing to /Users/mmuppidi/Documents/Personal/Projects/ML/reinforcement_learning/tmp/results/openaigym.video.0.19089.video000027.json\n",
      "[2017-03-08 19:55:43,185] Starting new video recorder writing to /Users/mmuppidi/Documents/Personal/Projects/ML/reinforcement_learning/tmp/results/openaigym.video.0.19089.video000064.json\n",
      "[2017-03-08 19:55:43,198] Starting new video recorder writing to /Users/mmuppidi/Documents/Personal/Projects/ML/reinforcement_learning/tmp/results/openaigym.video.0.19089.video000125.json\n",
      "[2017-03-08 19:55:43,214] Starting new video recorder writing to /Users/mmuppidi/Documents/Personal/Projects/ML/reinforcement_learning/tmp/results/openaigym.video.0.19089.video000216.json\n",
      "[2017-03-08 19:55:43,270] Starting new video recorder writing to /Users/mmuppidi/Documents/Personal/Projects/ML/reinforcement_learning/tmp/results/openaigym.video.0.19089.video000343.json\n",
      "[2017-03-08 19:55:43,371] Starting new video recorder writing to /Users/mmuppidi/Documents/Personal/Projects/ML/reinforcement_learning/tmp/results/openaigym.video.0.19089.video000512.json\n",
      "[2017-03-08 19:55:43,516] Starting new video recorder writing to /Users/mmuppidi/Documents/Personal/Projects/ML/reinforcement_learning/tmp/results/openaigym.video.0.19089.video000729.json\n",
      "[2017-03-08 19:55:43,677] Starting new video recorder writing to /Users/mmuppidi/Documents/Personal/Projects/ML/reinforcement_learning/tmp/results/openaigym.video.0.19089.video001000.json\n",
      "[2017-03-08 19:55:44,266] Starting new video recorder writing to /Users/mmuppidi/Documents/Personal/Projects/ML/reinforcement_learning/tmp/results/openaigym.video.0.19089.video002000.json\n",
      "[2017-03-08 19:55:44,886] Starting new video recorder writing to /Users/mmuppidi/Documents/Personal/Projects/ML/reinforcement_learning/tmp/results/openaigym.video.0.19089.video003000.json\n",
      "[2017-03-08 19:55:45,495] Starting new video recorder writing to /Users/mmuppidi/Documents/Personal/Projects/ML/reinforcement_learning/tmp/results/openaigym.video.0.19089.video004000.json\n",
      "[2017-03-08 19:55:46,153] Starting new video recorder writing to /Users/mmuppidi/Documents/Personal/Projects/ML/reinforcement_learning/tmp/results/openaigym.video.0.19089.video005000.json\n",
      "[2017-03-08 19:55:46,257] Finished writing results. You can upload them to the scoreboard via gym.upload('/Users/mmuppidi/Documents/Personal/Projects/ML/reinforcement_learning/tmp/results')\n"
     ]
    }
   ],
   "source": [
    "def run():\n",
    "    # Initializing the environment \n",
    "    env = gym.make('FrozenLake-v0')\n",
    "    # Identifying the valid actions\n",
    "    valid_actions = range(env.action_space.n)\n",
    "    # Initializing the mornitoring of the environment, this saves the results and stats\n",
    "    env = wrappers.Monitor(env, RESULTS_PATH,force=True)\n",
    "    # Initializing the agent \n",
    "    agent = Agent(env=env,epsilon=0.05,alpha=0.99,gamma=0.99,learning=True,valid_actions=valid_actions)\n",
    "    # Learning process starts here\n",
    "    while agent.epsilon > 0.03:\n",
    "        agent.update()\n",
    "        if agent.done :\n",
    "            agent.reset()\n",
    "\n",
    "    env.close()\n",
    "    #gym.upload(RESULTS_PATH,api_key=<api-key>)\n",
    "    #print agent.Q\n",
    "\n",
    "    \n",
    "run()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def get_episode_batch_file_name(results_path):\n",
    "    for file_name in listdir(results_path):\n",
    "        if 'openaigym.episode_batch' in file_name:\n",
    "            return results_path+ '/'+ file_name\n",
    "        \n",
    "def element_wise_average(rewards):\n",
    "    reward_avg = [sum(rewards[:i])/(i+1) for i in range(len(rewards))]\n",
    "    return reward_avg\n",
    "\n",
    "def get_consecutive_100_episode_average_list(rewards):\n",
    "    consecutive_average = [sum(rewards[:100])/100.0]\n",
    "    for i in range(100,len(rewards)):\n",
    "        consecutive_average.append(sum(rewards[i-100 : i])/100.0)\n",
    "    return consecutive_average\n",
    "\n",
    "# Loading the results\n",
    "result = json.load(open(get_episode_batch_file_name(RESULTS_PATH)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAEACAYAAABfxaZOAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAHQRJREFUeJzt3XmUVPWZ//H3w6JsBlAxTEBQAUOiKIKg4xJLQcWIIlEU\njUE98FMzKmY0CY5OIk4SjyY5M8a4xkGTiSY9Rp0IGlFcWlyCgIAg0mzGho6sipIGUeh+fn98b9NF\nW20XdHXdqrqf1zmcu9Tl1vOF6qe+/d2uuTsiIpIMreIOQERE8kdJX0QkQZT0RUQSRElfRCRBlPRF\nRBJESV9EJEGySvpmNsLMKsxsmZlNauSa881ssZktMrOHcxumiIjkgjU1Tt/MWgHLgGHA+8AcYKy7\nV6Rd0xf4X+Bkd99sZvu7+8aWC1tERPZENjX9ocByd6909+1AGTCqwTX/D7jb3TcDKOGLiBSmbJJ+\nD2B12nFVdC7docBXzexVM3vdzE7PVYAiIpI7bbK4xjKca9gm1AboC3wD6AW8YmaH1dX8RUSkMGST\n9KsIibxOT0LbfsNr/urutcB7ZrYU6Ae8mX6RmWmhHxGRPeDumSrguy2b5p05QF8z621mewFjgakN\nrvkzcAqAme1PSPjvZrqZu5fsn5tvvjn2GFQ+lU3lK70/udRk0nf3GuBq4DlgMVDm7kvM7BYzGxld\n8yzwgZktBl4Avu/um3IaqYiINFs2zTu4+3Tgqw3O3dzg+Hrg+tyFJiIiuaYZuTmUSqXiDqFFlXL5\nSrlsoPJJvSYnZ+X0zcw8n+8nIlIKzAzPY0euiIiUCCV9EZEEUdIXEUkQJX0RkQRR0hcRSRAlfRGR\nBFHSFxFJECV9EZEEUdIXEUkQJX0RkQRR0hcRSRAlfRGRBFHSFxFJECV9EZEEUdIXEUkQJX0RkQRR\n0hcRSRAlfRGRBFHSFxFJECV9EZEEUdIXEUkQJX0RkQRpE3cAIiLNVVMDK1ZAZSV07AiLF8P06XD0\n0XDjjXFHV1iU9EWkKHzyCSxZArW1sGULbN0KL74If/sbvPYauMMhh8DHH8NBB8E558DJJ8cddeEx\nd8/fm5l5Pt9PRIqXO8ybB88+C7NnQ3k5dOsGbaKqavv2kErBoYfC8cfDgAFxRtuyzAx3t5zcS0lf\nROLkDitXwvr18PLLoTa/aBFs3w5r18Lo0fD1r8OoUaEmn0R5T/pmNgK4g9DxO8Xdb2/w+iXAL4Cq\n6NRd7v5ghvso6YskXEUFPPJISOwrV4bmGTPYf/9Qc+/fH446CrZtg5EjoZWGm+Q36ZtZK2AZMAx4\nH5gDjHX3irRrLgEGu/vEJu6lpC9SgrZtC+3qTz4Z2tm/9rWQ1EeNgg4dQlJfsCB0sG7YAGPHwqBB\n0Ls39OkDBx8crpHMcpn0s+nIHQosd/fK6M3LgFFARYPr9F8mUgKqq+Hhh0OzS/v20KkTnH56aH55\n6y1YuDBc06VLuO6jj2DTJvjqV+GEE+DWW+G99+CKK+B3v4OuXcPomlQKbrghtL23bx93KZMrm6Tf\nA1iddlxF+CJo6FtmdiLht4Lr3L0qwzUiUmCqq+Hdd2HZMli6FO69NzSx7LNPGDGzdi2MGQPt2sHQ\noaF9fb/9YNUquPPOUEvfbz/Yd9/P33v48PyXR75YNkk/Uw2+YRvNVOAP7r7dzK4AfkdoDhKRmG3a\nFNrQAXbsCDX32trQlv7qq+FPjx7Qrx985Svw2GNw7LG73mPzZvjSl/Ifu+ReNkm/CuiVdtyT0La/\nk7tvSjt8ANilozfd5MmTd+6nUilSqVQWIYhINtavh1degfvug+efh8MPDzXyE08MtfEdO2DdulAr\n79EDxo2Dv/wl1Oq/iBJ+fpWXl1NeXt4i986mI7c1sJRQc18DzAYudPcladd0d/e10f5o4AfuflyG\ne6kjVyRH/vEPmDo1dKKuWQMzZ4bO1AED4MIL4bzzQsdpr16huUaKV147ct29xsyuBp6jfsjmEjO7\nBZjj7k8BE83sbGA78CFwaS6CE0m6zz4Lo2G2bAmTkBYsCLX5VavgoYfgyCPDUMdWreAb34DHH9+1\n1t6jR3yxS2HS5CyRGH32GSxfHtrdN24MNfKVK0ONfckSmDs3dKC2aROWFzjsMNhrr9D+fsUVYWik\nlD7NyBUpYitWwPz58MYb8OCDYchjx46hJr9yZehMPe20MEqmb9+wfozGsCdbvsfpi8gecg/LCZSX\nhxUg//u/Qzt7v35h7PuMGTB4cNxRSpIo6YvkiDu8/XYY615REWrzb70Vau8HHhja16+4Ai66KDTZ\niMRBzTsie+DNN8PQx5dfDrNL6zpWISwtcPzxcMQRYamBoUND56rWkJE9pTZ9kZi4w09+AjffHI77\n9oXWrcP67aeeCtdcEzpaRXJJbfoiebZxI1x7LfzhD+F45sww4Umk2OgXTpHIhg2h0/Xvfw8P7fjx\nj+H//i8Mo+zWLawa+cc/hsXDlPClWKl5RxJp61a47bYwquaVV7742gkTwrDJiy7KS2gin6M2fZHd\ntHFjWD/mgQdCDf7DD8P5k04KtfbBg8PM1oMPDmPmO3fW2HgpHGrTF8nShx/CjTfC/ffXn7v8chg/\nHoYMUWKX5FHSl5K0dm1I7tOmheNZs0KTznHHwd57xxubSJyU9KXkVFWFZ6x27gzTp4ehlBojLxLo\nR0GK3tatcP310LNnaK458MDwEJDly8NSB0r4IvX04yBFY9GiMIRy8+aw4uT48SHJd+wI//mfIfmP\nGRPG0E+bpvZ6kUzUvCMF67PPwiP9Ro8Oywxncv314Tmud9wBbdvmNz6RYqQhm1IwampCon/7bfjF\nL+BPfwrnjzgCBg6EH/wgrCe/fn1YZ/6b31RtXpJB4/Sl5PzylyGpp7vtNjj33LC+jUiSaZy+lIzv\nfx+efjosRfzv/x6e7dqxY1ipUkRyT0lf8s4d5s0LI2s++AC6dw9rzw8cGHdkIqVPo3ckr9zDg0SO\nPhr69AlJf80aJXyRfFFNX1qce3joyJAh9efeeUcP9RaJg2r60mJqamDSpDA5asgQmDgxLE28cqUS\nvkhcVNOXnHKHHTtCE07d4wNPPhmmTAkrWIpIvJT0pdk2bw7t8mPGhFmzdW66KdT099knvthEZFdK\n+pK111+HQw+F/fYLs2D/8Q/42c/g17+uv+b++8MaOKedBm306RIpOPqxlCbV1sJLL8Hw4ZlfP/zw\n8ASqrl21uJlIoVPSly/08MPwne+E/R//GE45BRYuhGHDQs1/woR44xOR3aNlGKRRL74YkvtFF4W1\ncL7ylbgjEkmmXC7DoF/GBYCyMjjrrNCUs317qNEPGwa33gqPPKKEL1Iqsqrpm9kI4A7Cl8QUd7+9\nkevOAx4Fjnb3eRleV02/wMyfD4MGZX5twoTwIHERiVdea/pm1gq4CzgdOAy40Mz6Z7iuE3ANMCsX\ngUnLev31sCxxXcKfMQPWrQv7Rx0VljhWwhcpPdk07wwFlrt7pbtvB8qAURmu+wlwO/BpDuOTHKqt\nDZOn3MPsWIDzzw/Hw4fDAQfUL4amB5KIlKZsRu/0AFanHVcRvgh2MrOBQE93/4uZNVgVXQqBO7Ru\nXX+8996h7V5j6UWSJZsf+UztSDsb5s3MgP8CLmni7wAwefLknfupVIpUKpVFCNIcK1ZAv35h//LL\nwyMIJ05UwhcpVOXl5ZSXl7fIvZvsyDWzY4HJ7j4iOr4B8LrOXDP7ErACqCYk++7AB8DZDTtz1ZGb\nf08/DSNHhv1Fi8JEKhEpLnl9XKKZtQaWAsOANcBs4EJ3z/ioajN7CbjO3edneE1JPw+OPDJMoEq3\ndSu0bx9PPCLSPHkdvePuNcDVwHPAYqDM3ZeY2S1mNjLTX+ELmnek5SxeHEbkLFwYHlICYS0cdyV8\nEQk0I7dEfPoptGsHp54Kd99d34YvIsVPD0aXz7npprB99FHo0iXeWESkcCnpl4ArrwzNOLNmKeGL\nyBfT2jtFbubMkPDHjYNjjok7GhEpdGrTL2Lf+x786lfQowdUVcUdjYi0FLXpJ9yGDWHJBIDRo+Hx\nx+ONR0SKh2r6Raa2tn45hbIyuOCCeOMRkZan9fQT6je/gW7dwv6WLUr4IrL7VNMvEu+9BwcfHPZf\neCE85EREkkE1/YTZvDkk/EsvhR07lPBFZM+ppl/gtm6Fjh3D/nvvQe/esYYjIjFQTT9Bxo4N24UL\nlfBFpPmU9AvYuefCtGkwdy4MGBB3NCJSCpT0C9Q998ATT4SHngweHHc0IlIqNDmrAI0fDw8+CN/9\nbkj+IiK5oo7cAjNlCkyYEEbovPBC3NGISCHI65OzcklJv2lnngnHHw833hh3JCJSKJT0S9To0fDn\nP8PHH8OXvhR3NCJSKDRkswTNnBkS/g9/qIQvIi1HNf0CsGpV/Rh8/fOISENaWrmE9O4dkj7ARx/F\nG4uIlD4178RowYKQ8AcODOvrdO4cd0QiUurUvBOT116DE06AXr2gsjLuaESkkGn0TpGbOxeGDAn7\ntbVgOfmvFJFSpdE7Rezb3w4Jf9AgqK5WwheR/FJNP4/23Rc2bQr7a9ZA9+7xxiMixUE1/SK0eXNI\n+LfeGoZlKuGLSBxU08+TG28Mi6dpWKaI7C7V9ItMbS088wz87GdxRyIiSZdV0jezEWZWYWbLzGxS\nhtevMLOFZjbfzGaaWf/ch1q8br45jMn/znfijkREkq7J5h0zawUsA4YB7wNzgLHuXpF2TSd3r472\nzwL+xd3PyHCvRDbvnHginHwy/Md/xB2JiBSjfC/DMBRY7u6V0ZuXAaOAnUm/LuFHOgG1uQiuFFxz\nDbz6anjsoYhI3LJJ+j2A1WnHVYQvgl2Y2b8A1wFtgVNyEl2Rc4e77oJx46BLl7ijERHJLuln+pXi\nc2007n4PcI+ZjQV+BFya6WaTJ0/euZ9KpUilUlmEUJyuuy5sp0yJNw4RKS7l5eWUl5e3yL2zadM/\nFpjs7iOi4xsAd/fbG7negE3u/rm6bZLa9H/0I/jpT8ODze+/P+5oRKSY5XvI5hygr5n1NrO9gLHA\n1AYB9U07HEno+E2sZctCwh8zBu67L+5oRETqNdm84+41ZnY18BzhS2KKuy8xs1uAOe7+FHC1mQ0H\nPgM2AZe0ZNCFrLoarroq7D/wgNbWEZHCohm5OVaX5F94AU5Rd7aI5ICenFWg7rgjbCsrwzr5IiKF\nRjX9HDKDAw6AdevijkRESonW3ilAzz8PbdvC++/HHYmISOPUvNNMW7dCx471x61bxxeLiEhTVNNv\npoMOCtvLLoMtW2INRUSkSarpN9Mhh0BZmUbqiEhxUEduM6xdC//0T6GG36FD3NGISKnKZUeukn4z\n1I3JL6EiiUgB0uidmH32GfTsGfaffTbeWEREdodq+ntg9mw45hjYf3/YsCHuaESk1KmmH6OampDw\nR46E9evjjkZEZPco6e+m6dPD9oILtJiaiBQfJf3dNGMGDBsGF18cdyQiIrtPbfq7qW1bePRRGD06\n7khEJCk0ZDMm27ZB+/bwwQew775xRyMiSaGO3JjMng0DBijhi0jxUk0/S+7QqhV06QKbNsUdjYgk\niWr6MXjttbBdtCjeOEREmkM1/SxpyQURiYtq+nn2ySdhu2ZNvHGIiDSXkn4WFi6ETp2ge/e4IxER\naR4l/SzMmgVjxsQdhYhI8ynpN8Edvvc92HvvuCMREWk+deQ2Yc4cGDo0tOu3axd3NCKSROrIzZMJ\nE0LCByV8ESkNqul/gbphmm+/DYcdFm8sIpJcuazp68HojXj11bDdtk3t+SJSOlTTb4QmY4lIoch7\nm76ZjTCzCjNbZmaTMrz+r2a22MwWmNkMMzswF8HFZfHisH3yyXjjEBHJtSZr+mbWClgGDAPeB+YA\nY929Iu2ak4A33H2bmV0JpNx9bIZ7FUVNf599oLoaamv1dCwRiV++a/pDgeXuXunu24EyYFT6Be7+\nsrtviw5nAT1yEVxcqqvht79VwheR0pNN0u8BrE47ruKLk/p44JnmBBWnHTvC8EzNwBWRUpTN6J1M\n9d2MbTRmdjEwGDipsZtNnjx5534qlSKVSmURQv5Mnx5G7HToEHckIpJU5eXllJeXt8i9s2nTPxaY\n7O4jouMbAHf32xtcNxz4FfANd/+gkXsVfJu+Ru2ISKHJd5v+HKCvmfU2s72AscDUBgEdBdwHnN1Y\nwi8mFRVNXyMiUoyyGqdvZiMItfhWwBR3v83MbgHmuPtTZjYDOBxYQ2gOqnT3czLcp6Br+h99BF27\nQk1NeDSiiEghyGVNX5Oz0lx7Ldx5p5p2RKSwKOm3kHbt4JxzoKws7khEROpplc0WsG0bfPppqO2L\niJQqJf3I3Llh+8//HG8cIiItSUk/sngxHH983FGIiLQsJX3CaJ0rr4R+/eKORESkZSnpA4MGhe1V\nV8Ubh4hIS0v06J0dO6Bt27B/4IGwalW88YiIZKLROzly2mn1+++8E18cIiL5ktik/61vwUsvwbHH\nhslYnTrFHZGISMtLbPNO3cJqelCKiBQ6Ne80U2Vl2G7ZooQvIsmSyKQ/cWLYas18EUmaRCb9qVO1\n3IKIJFPikv4dd4StxuSLSBIlriNXT8YSkWKjjtw9tH172H7ySbxxiIjEJVFJ/6GHwrZdu3jjEBGJ\nS6Kad9q3h27dtNyCiBQXNe/soW3b4Pe/jzsKEZH4JKamX1MDbdpAdTV07BhLCCIie0Q1/T3w+uth\nq4QvIkmWmKSvh52LiCQo6d9zD1x3XdxRiIjEKxFJf8eOsD3//HjjEBGJWyI6cletgv79YevWvL+1\niEizqSN3N61eDUccEXcUIiLxS0TSf/LJ+iUYRESSLBFJf8WK0LwjIpJ0WSV9MxthZhVmtszMJmV4\n/UQze9PMtpvZt3IfZvPs2AFjxsQdhYhI/JpM+mbWCrgLOB04DLjQzBrWmyuBS4BHch5hDixYAD17\nxh2FiEj82mRxzVBgubtXAphZGTAKqKi7wN1XRa8V3Cr11dWhI7dPn7gjERGJXzbNOz2A1WnHVdG5\norBqFRx6KHTtGnckIiLxyybpZxobWnA1+sa8/DIsXx53FCIihSGb5p0qoFfacU/g/T19w8mTJ+/c\nT6VSpFKpPb1VVtauhcsvb9G3EBHJqfLycsrLy1vk3k3OyDWz1sBSYBiwBpgNXOjuSzJc+xDwlLs/\n3si98j4jt08fOO88uP32vL6tiEjO5HVGrrvXAFcDzwGLgTJ3X2Jmt5jZyCigo81sNXAecJ+ZLcpF\ncLnQoweccUbcUYiIFIaSX3unWzf461+hb9+8vq2ISM5o7Z0srV8PGzeG2r6IiJR40q+shMGDwwPR\nRUSkxJP+//wPvL/H44xEREpPSSf9u+6CM8+MOwoRkcJR0h25ZrBwIQwYkLe3FBHJOXXkZuGnPw3b\nww+PNw4RkUJSsjX9Y44JHbgtNKlNRCRvclnTz2YZhqJTXQ2zZ8O0aXFHIiJSWEqqeWfHDnjkEdhn\nn3CsTlwRkV2VVNJ/5RW4+OKwP2tW6MgVEZF6JZf0DzgAliwJbfoiIrKrkurINYPRo+GJJ1rsLURE\n8i6XHbkll/TfeguOOKLF3kJEJO+U9DPYsCE07dTWqi1fREqLJmdlsHhx2Crhi4g0riRq+tu3w/Dh\n0KYNvPBCzm8vIhIr1fSBDz8M6+UD/PrXMHMmDBwYb0wiIoWuaGv6N90E994L7drBmjXQsyesXp2T\nW4uIFBTV9Am1/PPPDwl//HhYtSruiERECl9R1vTdQ/v9o4/CuefmIDARkQKW+Jr+okVhaOaQIXFH\nIiJSXIoy6c+ZAx07Qq9ecUciIlJcimpp5dmzw/r4kybBZZfFHY2ISPEp+Db91avDQmqjRkGnTuFc\nr15hrXwttyAiSZCYh6gsXQoTJ8Jzz9XPtJ03D/r3D0/FEhGR3VPQSX/69JDwf/lLOOSQMCzzqKPi\njkpEpHgVdPPOsGHQubOWShaRZEvEkM2aGnjxxdCWLyIiuZFV0jezEWZWYWbLzGxShtf3MrMyM1tu\nZn81s6wGUy5cGGbTbtpUf+7WW0MH7d13h+MzzsjmTiIiko0mk76ZtQLuAk4HDgMuNLP+DS4bD3zo\n7v2AO4CfZ/Pm8+bBgw9Cv35hGOabb8L8+WEdnWnToG/fsEZ+sSgvL487hBZVyuUr5bKByif1sqnp\nDwWWu3ulu28HyoCGjS6jgN9F+48Bw7INYNw4eOONsLTCt78Njz0Wav8zZsDy5dnepTCU+gevlMtX\nymUDlU/qZZP0ewDp61dWRecyXuPuNcBHZrZvUzeuqQlr6PTpAz//OVRUQFUVnH12ltGLiMhuyWbI\nZqYe44ZDcBpeYxmuAeCss+r3KyvhuON2fb1Hw68TERHJmSaHbJrZscBkdx8RHd8AuLvfnnbNM9E1\nb5hZa2CNu3+uNd7M8jc+VESkhORzRu4coK+Z9QbWAGOBCxtcMw24BHgDGAO8mOlGuQpaRET2TJNJ\n391rzOxq4DlCH8AUd19iZrcAc9z9KWAK8HszWw58QPhiEBGRApPXGbkiIhKvvM3IbWqCVyEysylm\nts7MFqad62pmz5nZUjN71sw6p712ZzRBbYGZDUw7f0lU7qVmNi7f5WiMmfU0sxfN7B0zW2RmE6Pz\nJVFGM9vbzN4ws/lR+W6Ozh9kZrOiWP9oZm2i841OMjSzf4vOLzGz0+IqU0Nm1srM5pnZ1Oi4lMr2\nnpm9Ff3/zY7OlcRnE8DMOpvZn6J/98VmdkxeyufuLf6H8OWyAugNtAUWAP3z8d7NjPsEYCCwMO3c\n7cAPo/1JwG3R/hnA09H+McCsaL8rsBLoDHSp24+7bFFs3YGB0X4nYCnQv8TK2CHatgZmRXH/LzAm\nOn8vcEW0/13gnmj/AqAs2v86MJ/QHHpQ9Fm2uMsWxfavwMPA1Oi4lMr2LtC1wblS+mz+Frgs2m8T\nxdji5ctX4Y4Fnkk7vgGYFPc/epax92bXpF8BfDna7w4sifbvAy5Iu24J8GVC/8a9aefvTb+ukP4A\nfwaGl2IZgQ7AXMJkw/VAq+j8zs8mMB04JtpvDayP9nf5vALP1F0Xc5l6AjOAFPVJf0MplC2K5W/A\nfg3OlcRnE9gHWJnhfIuXL1/NO9lM8CoWB7j7OgB3XwvUDU1trIwNz/+dAiy7mR1E+K1mFuFDVxJl\njJo/5gNrCQlyJfCRu9dGl6R/FhtOMvw4mmRYqOX7L+AHRHNizGw/YFOJlA1CuZ41szlmNiE6Vyqf\nzUOAjWb2UNQ89xsz60AeypevpJ/NBK9i19gEtYIvu5l1Iiyfca27V9N4fEVXRnevdfejCLXiocDX\nMl0WbRsrR8GVz8zOBNa5+wLq4zM+H2vRlS3Nce5+NPBN4CozO5HS+Wy2AQYBd7v7IGAL4beuFi9f\nvpJ+FZC+8mZP4P08vXeurTOzLwOYWXdCUwGEMh6Ydl1dGQu67FFH32PA7939yeh0SZURwN03Ay8T\nmjy6WFhIEHaNdWf5LEwy7Ozum2i83HE6HjjbzN4F/gicQljssHMJlA3YWdPF3TcQmh6HUjqfzSpg\ntbvPjY4fJ3wJtHj58pX0d07wMrO9CO1QU/P03s3VsPY0Fbg02r8UeDLt/DjYOYv5o+jXtGeBU6Oe\n+q7AqdG5QvEg8I67/yrtXEmU0cz2rxv9YGbtCf0V7wAvESYRQphUmF6+S6L99EmGU4Gx0QiYg4G+\nwOyWL0Hj3P1Gd+/l7ocQfp5edPeLKYGyAZhZh+g3UMysI3AasIgS+WxGsa02s0OjU8OAxeSjfHns\nuBhBGB2yHLgh7o6ULGP+A+Fb81NgFXAZobf8+agsM4AuadffRRj98BYwKO38pVG5lwHj4i5XWlzH\nAzWE0VTzgXnR/9O+pVBGYEBUpgXAQuCm6PzBhNnjywijXdpG5/cGHo3KMQs4KO1e/xaVewlwWtxl\na1DOk6jvyC2JskXlqPtcLqrLGaXy2YziOpJQIV4APEEYgdPi5dPkLBGRBCnYxyWKiEjuKemLiCSI\nkr6ISIIo6YuIJIiSvohIgijpi4gkiJK+iEiCKOmLiCTI/wc+mYjTv+v1ZgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x105d6cfd0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Average of rewards till that episode vs episodes\n",
    "plt.plot(element_wise_average(result['episode_rewards']))\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAEACAYAAABfxaZOAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJztnXmYFNX1978HhplhZFFBdoFRFCIuoAY0i87rBioRd8Ek\naqKGxGh+akxwd4hZRI0rMS5BTUwMSohKVBCjjrgGUFDDjoYd2XdmeoaZ8/5x+1q3q6u6qrqrt+rz\neZ5+aunbt+/t5Vunzj33XGJmCIIgCKVBq3w3QBAEQcgdIvqCIAglhIi+IAhCCSGiLwiCUEKI6AuC\nIJQQIvqCIAglhC/RJ6LhRLSIiJYQ0ViH5w8kojeJ6GMimkdEp4ffVEEQBCFTyCtOn4haAVgC4GQA\nawHMBjCKmRcZZR4D8DEzP0ZEXwPwKjNXZ6/ZgiAIQjr4sfSHAFjKzCuYuQnAJAAjbWVaAHSI7+8L\nYE14TRQEQRDCosxHmZ4AVhnHq6EuBCbjAMwgop8BqAJwSjjNEwRBEMLEj6VPDufsPqHRAJ5i5gMB\nnAngr5k2TBAEQQgfP5b+agC9jeNeUL59k8sBDAMAZv6QiCqJqDMzbzILEZEk+hEEQUgDZnYywAPj\nx9KfDaAfEfUhonIAowBMtZVZgbhLJz6QW2EXfA0zR/Zxxx135L0N0j/pm/Qveo8w8RR9Zm4GcDWA\nGQDmA5jEzAuJaBwRjYgXuwHAlUQ0D8DfAFwaaisFQRCEUPDj3gEzTwfQ33buDmN/IYBvhds0QRAE\nIWxkRm6I1NTU5LsJWSXK/Yty3wDpn2DhOTkr1Dcj4ly+nyAIQhQgInAOB3IFQRCEiCCiLwiCUEKI\n6AuCIJQQIvqCIAglhIi+IAhCCSGiLwiCUEKI6AuCIJQQIvqCIAglhIi+IAhCCSGiLwiCUEKI6AuC\nIJQQIvqCIAglhIi+ECl27QIWLMh3K4QwmD073y2IJiL6QqS49VZg4MB8t0IIgyFDgJ07892K6CGi\nL0SK3bvz3QIhDHQG9ubm/LYjiojoC5Hi73/PdwuEMPjsM7VtbMxvO6KIiL4QKcTSjwZjxqhtLJbf\ndkQRX6JPRMOJaBERLSGisQ7P30dEc4noYyJaTERbwm+qIAilwp49aiuWfvh4ij4RtQIwAcAwAAMB\njCaiAWYZZr6emQcz89EAHgbwz2w0VhByzQcfAOefn+9WlB6ffqq2Ivrh48fSHwJgKTOvYOYmAJMA\njExRfjQA8awKkWDyZGDKlHy3onQR9074+BH9ngBWGcer4+eSIKLeAPoCeDPjlglCAUChLEUtpItY\n+uHjR/SdfvbsUnYUgH8ws9vzgpATmprCqee++9R269Zw6hOCIZZ++JT5KLMaQG/juBeAtS5lRwG4\nKlVltbW1X+3X1NSgpqbGRxMEwR8HHQR88QVQXw+0aRNevfX1wH77hVefkJqjjgJWrixdS7+urg51\ndXVZqZu8jHIiag1gMYCTAawDMAvAaGZeaCvXH8A0Zj4oRV1yEyBklfJyZeVv2AAccEDm9Wn3zscf\nA4MHZ15fobElHme3//7ZfZ8NG4AuXbzLMQOrVwNf/zpQVgZMmACcfXZ221YMEBGYORRno6d7h5mb\nAVwNYAaA+QAmMfNCIhpHRCOMoqOgBnkFIS/s3KkEv6wsfLfA0UdH0+o8/XRgxAjvcpnStSuwfLl3\nuXnzgN69gfXrgZ491cVCCBc/7h0w83QA/W3n7rAdjwuxXYIQGC3K7duHJ/oHHABs3Kj26+vVnUSU\nmDULaJWjKZr19d5ldu1S2/32UxdaScMQPjIjV4gMe/eq7T77hCP6u3ZZgg8ACxe6l80nb7wBvP12\n8NfpZGbpRihNmeIvIdq6dWrLDKxZA8yZk1yGGbjqKmD+fHVcXw+0batcPUK4iOgLkUFb+vvvH47o\n/+pXicdZGlfLmFNOAdKJh3j//cze9/zzgXfe8S53xhlqu3cvcP31yl9vZ+NG4I9/BH7yE3Xc0AC0\nbi3RO9lARF+IDLGYit6prAxHLOxWbNQESLtO0nGh6Ne0tHiX3b5dbWMxdx+9U3zHgQdG7zMvBET0\nhZwydy4wcWL49S5fDhxyCLBjh4q0GZuUISo49th8I9o4MC0twE03+RPJXJGJoOrX+hnc1mMGN97o\nfrfk9LlUVqroHaeL0m23AT/8IfDll8ANN/hqshBHRF/IKWPHAldcEX69f/qT2m7apNwIM2dmXue3\nvgV07w4ce2zmde3ZA9x1l3JbZIMyXyEZiWjBHjo0+Gt1P/xcOL7zHbV9M8U8fafJdBUVaus0bvDr\nXwNPPQXMmAH8/vfebRAsRPSFnJKtaIzWrcOvs7lZ+a3DmOSlxTGsmcJ2Djss+GsaG4Hq6vSid4KI\nfkUF0KdP6jKpPpdUUT8S3RMcEX0hp6xZo7YPPxxuvdkIO2xsVCGaRxyReV1aHC+/PPO6TLRY6qyU\nQWhsBKqqVCbRf/wj2Gs3bVLbH/wgdblhw4B//UuF0Tpx//3Addcli/6IESoKC1AuHDf06+64w72M\nkIiIvpBTKivVNhP/uBNmioQXX0wvmsWOFv0//AE45pjM6tKiH3bGzp070w+51JY+AEybFuy1fmLu\nAeV+WbBATQLT7LuvtT9+PPDAA4mif+GFwPPPA+eeC3TrlvpuQi+aY4+0EtwR0RfyQti35aalv88+\nztEgQWlsVK6JsjKgU6fM6spWFEospiaQEQUfJG5stC6WQV8bdGzCTIlhvpeT26tjRxWj36qVGpy3\nDxab362ezCX4R0S/SOnSpbjT/m7fHt4flihxkHDPHjVZKdPMmNrSB4DzzlPbdC3KW27JrC0m27ap\nPr/yinVhKi8PniYiFrNy7jz9tLUurR+Civ5//2vt79gB/Pvfan/bNrWdNct63nQ1OfVLx/IDIvrp\nIKJfpJgzRYsJ80IVZiTLhg3AuHHKCtS3/FpQ0sUU/R/9SG3feCO9ut56K7O2mOjP7bPPlHBXVKhH\n0LuJxkZ1V6Qjn1atSl3eJBZTr/1//89f+VWr1HejZ00vWmQ916OHeu7SS9WxXioRcBb9v/xFbUeN\nkjWR00FEv8hZtMj6I4VFfT3w+efh1qkxk26F2e733rNC/LSrQE8KShdT9DWmIPllxYpw8vGvWJEo\nnO+8Y4l+69YqhUFjI7DWLfG5jWXLVP/0OEsQNm8G2rVT7795s3c6Bu1+01FW8+ZZKRaIVD16sNf8\nzMvLgaVLreOWFms8obw8cZA3ignxsoGIfpHzta8Br70Wbp333AP06xdunRrT+g7bz60F+cQT1Xb9\n+szqC0v0+/bNrB1mPW+9pdYLAIBXX7VEf+tW4JvfVHc7PR3XtUvmz39WA7g6N04Qd+H27ep9GxvV\nYOtIhwVUTd/7T3+a+NzEiWrGrX7fWEz58QEVzaNZtw649lrr+N13E+vXUURAet9NKSKiHwHCnvAT\n5DY/E8IQfVNY9OBwnz4q30umdxJOop+uO6FdO5VQLNOLqU4frdGir9EhsX6pr7c+wyCD383N1iDr\n3r3OvxlzsH7YMGtfT9YyMfthRl7Z27Rjh7Xfp0/ib9+MChLcEdGPAE89FW592chhPm1acj71dGLL\n7Zihg2ZUSFkZ8OyzwKQMVniYPDnZZeDXdQIon7sWu127VFRKui4I7W6rr08Mr3zmGeB//7OO5871\nrmv6dMtC3ndfS2ydxNiNhgbVH33hXrYsuYx5cTIvTOa+5sUXrQVdTPRA7ejRKt7ftOZjMeA///Hf\nZiEOM+fsod5OCIOOHZmVHaQeYTJyZPh1AsyXXaa2N9wQXrtXrrTquvFG63yXLpm/B8B81lnW8YAB\nzBdeGOz15uN3v2Pu2jW9tvTtq+o46aTkes1Hu3befQaYf/5ztV2wgHnTJuv1LS3+2nPrrer7rK52\n/5y3bXN+7uKLE9t84IFq27o18zPPMMdiiW01H88+q7bPPcdcVqb2jzmGecoUf+0uVuLaGYoOi6Vf\npHTsmL26wx4Y1tTXq4iP3r29y/rFzbWVTi4aJ8z0DjfcYM0STYcOHdK39HW0lva79+jhXM6vX37j\nRqBzZzUmZM5B8PvdNzR498cttYKTpQ+oz/p730u9UE1Tk7p7uvBCa3D4kEPURC7BHyL6Hjz5JPDy\ny8Ffd801wf2rAHDxxep2PdW08ltuUYtGZwunW/VMGBdfU23jRiUSmU50MjFF36zXFGsdCvjww/5D\nJy+5RG27drXOVVaqlAW//nV6ba2qUgOuqYS5a1fgvvuSz3fooLY6ZNTNzaSjaLxm/r7ySuIgqI7X\n97sO8MqVapwi1W/c7YKgB6I1ejzgINfVtS127rQuGnrAupjnq+SFsG4Z/DxQhO4dgLl79/Re9/LL\n6b1u3LjUt+j6VvfBB7Pj3tG37GGh23j55Wq7dy/zCSeo40yZNYv5sMOYp05l3r3bOn/UUcnuL4B5\n6NBgbV671jo3eXKwz7t378Q2eL1+717352+/3d2lc++9zO+9l3iustK7b+b7PP98sL5dcAHzo486\n16VZvpy5Vy/m2bPd3998bNuWXEeHDskusgsuUM998IE6d/XV/tpczCDX7h0iGk5Ei4hoCRE5Zion\noguJaD4RfUZEfw3xupR3gmZG5HjEQbYzAP7wh9mpN1trpu7apW7dW7dWA3NhZa/cbz81CFlVZZ33\nG7bohZkoLGg8u77b0FttrbuRylWif1NO9OoFfOMbieeCRnQFdRc2NanUCvo7NNMsaPRsYT+pqY89\n1rkNdjed/g0BalY6oPov+Mfz701ErQBMADAMwEAAo4logK1MPwBjARzPzEcAuDapoiLGvA32w4D4\npzNyZHLEihs6fwpguXaWLVPnbr/dKmemqK2oSM8//otfAPfe6/xcnz5WpMhvfxu8bkC5c5xynD/3\nnCVs6aQNcGL1aufkX3YR1DNAg0Z7mP5nN9Ensh56stAll1hRNfrib7qKnEQ8VQhrqs/KzUcehO7d\nUz//z3+q/umQVR1iqQXY6QLe1OTsn3fK3++WGtv+Pf7mN9Y6uvoi/9FHqdsuJOLHphsCYCkzr2Dm\nJgCTANinYlwJ4A/MvAMAmDmgTBY27doFK79kibXvJ8SP2fnCokXjzjutc6Yvv6xMLdYd1Kd5773O\nfmN7/X/4Q7B6NbW13lk0y8vDi9N38gXfdFPisZlXJpXVbMe0NP1YrCtWqO0zzySeb2gAjjrKOna6\ne/QS/SFDrOMf/9i6EGjBra9Xs2P9ctpp1r5X+uhnn1VbPbPYTP8AOKdObmx0vhiY6+reemvq933x\nRbXt399aM0BfRPXdhZm3R/DGj+j3BGBOvVgdP2dyKID+RPQuEb1PRMMQITJZoCOVq4RZRUu4uY/s\nU/ftkRVEahaj9ngGwU+fMkljoPvkNj1fi75T/5m9XWo6XlvnhLdj/9zNz8erbrOseUE1I3fcslLu\n3ess3nZr3CkvkFcKYe3OAFT/tKDq9lZWWgOybph9c8tx74T+Le7erS5g69erPunPwelOxM3SNy8E\n+nm3qCH9O23f3nKP6e87GwvnlAJ+Atuc7Ei7xJQB6AfgBAC9AbxDRAO15W9Sa5iANTU1qAkj8XmW\nySQ64PbbVU5xJ2prVdbGDz90fv6ii9T2619XWyc/LZH6E7n9wdzw47fPJJmVFjA3P3Z5ucqmqDMq\nmmL0+98rF5Tbhay5WQnwl186z5p1Ys4ca3/ZstQrTblFTplidc45wEsvJZf51rfU1st91bVrcv/c\nRH/9euDxx9U4iEa/tlev5FWpqqsTJ2yZjB9v7Y8alfhcZaX7WIDOYjplipUxdO9e6+Ll9H5ulr5m\n8GDghBPUvtcdCrN14bz44sTnzKybUaGurg51bgsKZ4gf0V8NJeSaXgDsTovVAD5g5hYAy4loMYBD\nACR522rDXj2jADn8cJWKd9y41LeeevbkjqRLYyJnn6225h/SHCTWAhOG6P/sZ0o0qqtVuGq6eKUb\nSNXWBQtSv1b78LdtS91vZuuCvWsXcMopymL1ytGiv5dUd09mKufTT09ehESvGAX4X9BF37V07px4\nXrfXdDNqC9sp/cETTyjftxP6s21uTv4NbN3qPaBruiFN69xpXCGVIWL/bE86KfX7AlZdV13lXk9U\nsBvE43Tccwj4ce/MBtCPiPoQUTmAUQCm2sq8COAkACCizlCCb4vGLV62bQuWJXHtWusWdPt294Fg\nLUiTJ6eu7913VWoEU6zMP2ybNqmtynnzrAuGtqjcRH/mTHXbXFGhMlemi9ccBbsY6MG55cutfTe0\ny2jzZv8Xu3//W/Vp9+7EtL5O+PGLB0nu5fdOcc4cVa/d4tevN91YqcROJ0Kzs2aNlYTO6fvXd4yp\n6v7kE+c2xGLJr/Oy9INQVmbdVYQ18a5U8RR9Zm4GcDWAGQDmA5jEzAuJaBwRjYiXeQ3AZiKaD+AN\nADcwcwjJZAsHc9DLiy1bVMIvzfXXpy7/xBNqO2aM8/PTpqm1Vd1y6JeXp/ZTDx6sImcAy4p0++PM\nm6f+2Lt3K/ELuqKSpr4+8c7ku99Vro/vf18d2wVC3+ZXV1t3R24XSy1cmzcrsUkl+o88orZLlyox\nXLTIaoMbegnBVJjhsk6fvbbyx45NXEBFtwdIDum97DK1tYu+rl9vy8uTs1aauA2SDxvm7moE1MW+\nVavkdpnflXmHYw5sO72nX5fj/ferO0w3nnwSuPtuYPZs77oEb3xdM5l5OoD+tnN32I5/DuDn4TWt\nsPCbaKulRVlmps9YR3SkYv/9gUcfVbfwTuGOS5eqP9GQIclhh37CH+0DquaasmbbAfUn13/8PXuC\nRS9pl0rbttbdUefOwF9tMzfsA3dOybbc/Mta/BoavC39n/xEpeq1Z6NMRbdu7iGtzCrVr+lHd/PF\nz5gBnHpqcnu0eyIWcx6Ettenv1s9PnL99cCRR7q3383S97qDAqzfkmkU6PYcdZRl6V9xReKgcWWl\nKmeGtfq19K/1CPD2WnxdCIakYfDJ5s3qj/bqq6nL7dyphKFVK+uPM3Oms8U81XCSaWF1m9C1eDHw\nwgvOwpVK9LWVZndZOP0ZncQraE76vXtV3/fssZYWdIqy8OOLdRNTff6NN5Try8ua1FEvppClcqn9\n7W+pXTJr16pw1hdfVGM3ZgiiiddFxs2NVF+v0kVcdJFKAaK/W/3b8HIXZRIO6/Rb0hdf807Tnom1\nsTE52itocIGQG0T0fRKLqVjvM89MXW7LFkvAP/7YOu8WTaHROWG0Ra6tqJtvtsrcfbezkKTy6es/\nvw431HlonKJqdNnx461FMfQCG37RszABdecCJC58obEvs6fFwQyL9BL9xx4DXn/dW1y11V5Wptwt\ngErY5cb69YluDDtTp6oJbOecoyYtudGtW+p2PfZY8jntNjrpJOD559VMY/3d9u6trOJf/jJ1vW6W\nvh+cRF9/3ubdrjZYHn1ULeLToUNyQEKYPn1ARXqlu0axYCGin4J0IgNiMSsD4hFHqD+uH/QEI/2e\n+rb/mmsSyzlZTql8+voPq//IXbsqwXWKi25oUM937qyEo6YmvcW27W10iuTRqyQBavxDv8Z0W7i9\nt/28lzWpV66qrvY/NpNKrPwOJNqjcOyY7it9J6jHNkx0f1u1Uv5vr8VCMrH0nQyIhgZrlSs7Y8ao\nz7R79+TfYNiW/nnnAbfdFl59pYqIfgrSSTFs9zHrW96FCxPLuU1a0n8cLQL2qf9B3TtaWPQ2FlN3\nIk7lly5NjM0Psti2TkNw7bXBFyRfvFiFVG7dqrJYapzee82a5MU+vFb6Mr8Pv5ZnqsFcv6tfed2B\n6LasW2f5/p0mV+nvyu/KUG3bKqtcZ/S0u4PcBBxwNiA2bUr+Lg45JPFYR/7Y2x2mpS+Eg4h+CpwG\n2rysN/uAoV4dyu5DdnOb6D+2tvjNurQFbieVe8cu+o2N7qK/fj1w9NHWcRDR1+j0Aw8+qLY6IsUJ\nHdb5+edKbOwDjU7v/c47yeMeM2embtOgQWrL7JwYzAl7GgeTt99OPvf558mTutxEf80aJZrHH6+O\n33/fcieNGJFcvrFR5b23T6Zyo1Mn9dALj2u+8Q1Vz+LF7q918+l36WKllJ4yJTGtBeAs+k1NIvqF\niIh+CuwC3qNHYtIsJ+yWvrbU7XHdsRhw8MFq31w4W/u0taVvvr85K9EkHUvfyR20e3dyQrd03QS6\nT9/+tnsZcyGQAw5w/ozsOA1iBhEWp2gZO61bp7aqnSYw9eiRbBC4uYF69FBuLN0/PdDdr5/qnz2G\nvrEROPTQYNlPDz44+fNs00YNQJuuNTtOv6XGRtU33e9zz03+HbpZ+jKQW3iI6ENN5ydKnlXZuXPi\npKz27b193OZAJmBlFNTpBjRafIHETJwDB6rteeepmbjmH13HpduZOTN5YFSjsz4+/LDq45NPqguL\nUz/mzUueZekm+hs3po4i0QLhZ1zkiCOU9Wsung04LxTjNFvXre/29gwcmPjdOM17+O1v1Z1E0AlA\n5eXB5jRUVKhQ0gsvtAwDLZB2cT/7bOeUD6mYNQs47jjreM4cNdnOy+XkJvrl5Vb2WCfE0i8eRPRh\nhc7pVYns6CyXHTp4W772gczzzgOuvFLtmwLY2OhscX3ve6rchAnWhaKpKXHyUhCcLHq3OwMiyxWS\nqhzgHcqpXRfmXYwTzO4LpDtdVLQgmndcbikHTDZtUt+FKXpOs6wfesj9vU2uuy65XTpTZVOTt3FQ\nXq4udJMnWxcY/Z7pTohzex8A+O9/ldHiJz2Gm+ifcor7RdxpLEAs/cJEJjTDEnIzmsK0eLXQVFa6\n/5lbWpSwrFyZ/McwU99q94LXZCEi67a/rMyqI9WtucmKFaodTpPKysrcU/ua7o+yMm9xb252tsh1\nezOx9FItCdmhQ7A5BFpYzc/cKYuo32glp4uC+X35bQ+QPHkvTNE3AwOamvwtBGO6hbZu9SfeTuNK\nYukXJmLpwxJ904o2LfpTTlHbd95xF4V//UtZn2PGqNhxEy0Q5nT8IDNEAfcp+iY6imX+fGVhV1db\ncfkmbgO/9hmVjz/unu9cX9gGDXLOZ6/7nE6GUj2Y7JRjSte3dGnweoHEz9zMT6/RWTK9OOYYtb3g\nAutCXF3tf43ZP/3J2r/iisTnwsitpTO06u+pXbvk79eJVq2sgd4VK1Q0kV/Rz3bIphAOIvqwhNSM\n7GhoUD94Ziu1MeAu+m4hmCamP9oUfburwAk9WSrVEozacnVKaWDiZuk3NPhfElB/Zv/9b/Jz6cxv\n+NrXrP2PPrIGee112a3goO/lNRh61FH+RPfii9V7P/+8ZRkfeGDihLx0uf32zLNHmhlSzzlHXSz3\n7vW2vI880rqw6vDddEVfQjYLk8iK/vbtVpIxL3RsuDlrduVKSzxNazWTJf50mNu8eSp/jr7F97MY\nhC6T6v11Th6vRULKypzrWbfO/93HlCn+yvkVL/v7ajeTdrM1NqrPL4wlFk3Mi9aaNSosMYwVvcIm\nqMVsfp6bN6uLk5lq2o2GBivcVl9g9+zxfv+yMuDpp4Hp09Xxn/6k3lcs/QIkrBXW/TzU2+WG73xH\nrSfV0uJdVq89deWV1rlOndQ5TXU180MPudf51FNWPfZufvJJ4nm9f9NNzOPHMy9b5t3G5mb1miuu\nSH7u6acT63/77cS2nHlm4vGiRczt2yfXM3Qo8wsvWMdPPJHcF41Zn/2huf565p07vfvGzDx7tnrt\nb36T2Af9+vvuU8c//7nannaae9u8uO025/a6fX/Z4LLLkj+3gQMTy7zwgvXctGnB3yPVd+P1mpYW\n5o8+UvvjxzNfdVXq1x1/vPNv/JFHgrdbSCaunaHocGQtfZ2WN4jVZg7e2pNhffGFSomQahBUc845\nic+5ZUQ86yyVR0XH66dCuyWcBvnsfnu7NfyjHyUe9+7t3AeixHwx55zjvfxeKn7/e/8ZOo89VsmE\nzjV0wgkqE6jui3ahxGLKEvXrO3eiEPK36HGiVJx9tjVeMnx4dttjp6nJ+k3v2OFtsbvdQYilX3hE\nVvT1j7BtW/8ugVhMxTPr2HondDKrbdsSE2aZou83+iLIQK7GKzJk/XrAvsqa00Qap8/E7rutqHBP\nb5wLiIC77lL7egbzhAlWJs+weOABb5dY2Dh9j06/G6cU2LkgFrNWEPvNb7zF20wzYo4piU+/8Iis\n6JvZML1WStLEYip51KxZahq+06QfHcc8fTrw4x8nvlbj9Od1mgMQ1AqaM8cSQTs6EdVDD6noCzOd\nQrduVvrfIUPU+ABz8qCw02xiP6Lft69a5OWZZ9zX+02HLVuAe+6x2qZ5+231/djXFQiCOVnuuuvU\nurl6QN3v2gmZcO65ajUvE6dZwK+95r5mrxf6wvj442qbypjR6MidxsbEhVq8xNuc86AXkAHE0i9E\nIhunb1q3fmdXxmIq8ReghPGb30wuo0XfnozNFH2nwcvDD0+eph80QkOHCTqhVzGKxZTVeuutSlgA\n9b5HHaX2Dz1UWdB6Mo05iGyfWFZWZkV9mJ+h/WLhlTY6DOxrArdv7xxy6Rf7mqzMatbubbepjJHZ\npk0b4OSTE885ufk6dQLSXVb6zDOVAJ9+ujq++mrv1xx6qDISgmYyNTHvmsTSLzwia+nfcIO/clq8\nBw9WC1boH+wnnziL8o4dwJIlVtSHTnNghmP275/8unbt1DiDaRH5DY8MwvLlasq+Wbe5r8NSnVw8\nTqF5zc1qPMNEW9+54vjjE0MQzVXJ0sUuRlOn5mcGqRmqeuih4datZ0O3bx/sdU6pmYNknDWXuQya\ncVXIPpEVfcCynFL59HUs8u9+l3jeLYf40KEqZ4t24ejl40yXjs5GaFJVpR5aQN9/33+KXj/oMYw5\nc9S2stJa2FyL/ooVVkoJv9PmjzwyeZFznYHS6eKWDewuI3OSW7rY+3n//fmZQTprlnItrVljLfAS\nFvfeqwISdB4kM212KvS41WmnWXcHTrOXTbZtU8ncOnZUg//6bsmva1XIHb5En4iGE9EiIlpCREk/\nTSK6lIg2ENHH8ccPnerJNUOHqolVqSJ4YjHl/rCvJLVrl3Ma5V69EqepawvIFFA3C75LF/Xn6dPH\nyk0TFvqBraaxAAAY/0lEQVSuxMzOqTNmaldX795WFs9UOVZM7P01yYe/tkOHcN7XLu67d+dH9Nu1\nU8ZJjx7hf57l5YnRV34Hq7WlT2S50Lza1rEjcP751h2k3X0mFA6eok9ErQBMADAMwEAAo4nIKd/e\nJGY+Ov540uH5nKGt6ZoaYPbsxGyDdvTMWHOZPkBN2DJvUzV796qoBi2YI0YAf/mLtWxeqolWlZVK\n9NOJ2vFCh/bpVAydO1u39U5jGhs3JrptPvjAeTJNYyPwyivO7zl8uDWWkCvCmpxlDzHcvVtF8fiZ\nKFeM6IuLH7RB8Npr1liKn4g07cJcu9YK1TUT+AmFgR9LfwiApcy8gpmbAEwCMNKhXBpZVrKDjr64\n5BLvslr0nVw5Tql3Bw1SwmDePeiUzK++mtr3WVGRPdE//PDEENJDD1VWsduf9YQTEiNz9JiEXfTP\nPtv9Pe++W11Us0VLS3IunDBDK53GbPQKVlFjxw5rQNcLpzV2/YyjVFUp42PDBsuIOuusYO0Uso8f\n0e8JwFyQbnX8nJ1ziWgeET1PRL1CaV2aaGHwc7u8c6f6Qzjd1jvluamoUP5Lc/Uhne3RTzTO229n\nnlfFDSe3ktukGXuufD3AbP/MKioSffq59NESJV8gg+a5D4r9ji8qBEl819xspQzRvym/bq+2bdX/\nSX+OEr1TePj5Czn9XOyyNRXAs8zcRERjAPwZwMnJLwNqjfizmpoa1NTU+GpoEIJkeLzpJnVL6nSB\ncLJSyssTMyQCwFtvqa3XLfCWLSo/SbYIcgdh9+lr95R90tNLL6moJkANDGvLO1duHfv3kk7WzlQ8\n9hjw6KPWRCSv5TBLgdmzrTu4445Tfn2/2UcrKpToV1SoMSwR/fSoq6tDnX2WZUj4Ef3VAHobx70A\nJExfYWZzOYonAIx3q6w23aDjADQ3+1tNCbCyIuof58aNVlij01T/TFwzxx7rvYh3JmhB9HMnYbf0\n3dxSZvbQHTus/Wy6dUyy4Qoz+dGP1JKOhx2mfNhRtfTTpW/fYJPgKiuttA1B1jsQErEbxOPCyLcd\nx497ZzaAfkTUh4jKAYyCsuy/goiMjC0YCcBhUbvcsWaN/wE57bfXgun1ulQuIy+xDTtDpJ1UaZft\nVFSogds771TuMDcR1wNyv/xl+Fa2H+x3HtlIC6FdGNmYN1FqtG6tggJkJm7h4mnpM3MzEV0NYAbU\nRWIiMy8konEAZjPzywB+RkRnAWgCsAXAZVlssydbt1qpeZ98EvihzwDSX/1KhZ7dcktyCKfGtDx/\n8YvEiUpedxdhrojkRE2NGlz1Q0WFWg92zRoVygk4Rzk98ICK3rnnHn9rBoRNLt5TRD+RffdV41aZ\n5IYKMzeSEC6+hsWYeTqA/rZzdxj7NwO4OdympU9jI3DIIWr/7LO9Fynp0kVtdf4ap8lVGtOCufhi\nNQN28mR17JVRMtui3727uhD5wRyg1XH4en1YE3MC2YYNmbUvHQ4/PHklsrAR0U9Efw7XXhv8tfq/\nlK1gBSFzInk9bmy0rBQdJplqRmGQqeLmj7m8PJjLw2+a4VxgDrDpz8ZL9PRgZy7JRZZJ3e9sjx8U\nC/ozT8daD+JiFPJDZEVfW+R6++67yeW05e130gqQmFysosIKdVyyxPu1Tzyh7ibSXd81TNq0UbNt\nAf+ir2f45pL/+z9r/7nngBdfDP892rZVicnCTItRzLz5psrM6SdBm536erUVS79wiWSWzVjMyjeS\nKq5bR68ECSvTi2ADSvT1nADtTkrFfvsVxgIegMqkqDOKaveOl6Xrlo4hm3TooD63rVuBCy/M3vuM\nGJG9uouNbt2AK69M77Va7EX0C5dIir7p3tE43arqSJDDD/dfd28jeLVtW5XErBjR8dSA5cv3Clec\nNUt9jtkem7AzaJBas1gofET0C59IunfseeGPPDLRQjfLtWsXbMLUGWeo7bRpKtd5rldcCouKimTx\n7tTJ+3XnnZf7dLnTplkzRIXCRkS/8ImspW+KfpcuzjHyDQ0qPC2Ie0cP3OoJXMU6cJVuHHVZmeU6\nyxUywFo8iNgXPpG09O3uHfvsU81bb6UXB15ZaYWm5drVERbpCmmQheaF0kMv4G4u1ykUFpEUfbt7\nx030GxuBiy4KXn99vZWVs1gtfS36EyZ4r6zErBZ9AawUzoLgxLXXqt9LFlJqCSERSdHfsCFR9Nu0\nSVymUGO/OKRDsYt+VZW/W3LtAst2lktBELJLJP/CX3yRGIlC5Cz6TlE+QbnvPue6C52uXdW2rEyl\nWtDhm27okNR85N8RBCE8Iin6FRVAdbV1fNBBzlE2YSyEfdllmb0+X+hFuHv18peRNNeDt4IgZIdI\nuncaGhJnl1ZVOacQCMO9U6zohHSl2n9BKFUiK/qm26Z3b2DZsuRyYbh3ihWdQnr37vy2QxCE3BJZ\n0Tct/cMOc86TX8qWviaqC4ELguBM5ER/1SqVXsAU/fJy95DNUhd97eYRBKE0iNxA7jPPqK0p+hUV\nzjNyS9m9AwCffhos7xAg0TuCUOxETvSd8sK4Wfql7t454ojgr5Fp9oJQ3ETOveMUb15enmzpL18O\n/OUvslpSEMrK/KWQFgShcImcpX/kkcDo0YnnnNw7eiGTkSNz064osG2bjAEIQrHjy9InouFEtIiI\nlhDR2BTlzieiFiLKW7qlnTutZGia8nI1a9ZMjqYna4ml75999hGfviAUO56iT0StAEwAMAzAQACj\niWiAQ7l2AK4B8GHYjQzC8uXJfueKCnXOTCymF1URERMEoZTwY+kPAbCUmVcwcxOASQCcnCJ3AhgP\nIK/Jd9u0SUzBoM8Bicv9tWoFnHpq7tolCIJQCPgR/Z4AVhnHq+PnvoKIBgHoxcyvhti2tHCKvXey\n5jdvloyRgiCUHn5kz8kB8pUDhYgIwP0ALvV4DQCgtrb2q/2amhrUhJx4O9WEq0GDrP2dO53XzRUE\nQcg3dXV1qKury0rdfkR/NQBjOXD0ArDWOG4P5euvi18AugF4iYjOYuaP7ZWZop8N3ET/vvsSF9fe\nu9daCEUQBKGQsBvE48aNC61uP7bubAD9iKgPEZUDGAVgqn6SmXcwcxdmPoiZq6EGcr/jJPi5wE30\nW7dWeePNcqU8G1cQhNLEU/SZuRnA1QBmAJgPYBIzLySicUQ0wuklSOHeyTZuov/ZZ/7KCYIgRBlf\nQ5nMPB1Af9u5O1zKnhRCu9LGTcz1SlFe5QRBEKJM5IYy3cS8c+fE49tus0I5BUEQSoWSEf0xY6x9\nvZi5OUNXEAShFCgZ0dfpFlparBQMe/fmrl2CIAiFQMmIPpGVbfP889W5rVtz2zZBEIR8UzKiD1ii\n/8or6lhywwuCUGqUlOgzJy6mIsnWBEEoNUpO9HUefUBEXxCE0qOkRP/YYxMXUxHRFwSh1Iic6K9f\n7y76FRWJ7h1JuCYIQqkRKdljVmGY++7r/Lx9rdyf/jQ37RIEQSgUIiX6DQ0qHt/L0j/4YGDJEqB/\nf+dygiAIUSVSor9qVaL7xs7WrcCNN6oykoJBEIRSJFJrR/3vf8Ahh7g//8knwJYtyuKXZGuCIJQi\nkbL0GxpSu2x0zh2x9AVBKFUiJfqxmJVjxwlzuURZQEUQhFIkUqKvB3LdmDLF2u/QIfvtEQRBKDQi\nJ/qpLPhOndS2qio37REEQSg0IjWQO3MmUOajR3v2ZL8tgiAIhUikLP3WrYETT8x3KwRBEAoXX6JP\nRMOJaBERLSGisQ7PjyGiT4loLhHNJKIB4TfVm1gMaNs2H+8sCIJQHHiKPhG1AjABwDAAAwGMdhD1\nvzHzkcw8GMA9AO4PvaU+8PLpA6nj+AVBEKKOH0t/CIClzLyCmZsATAIw0izAzLuMw3YA8rL6rFfI\nJqDSL8jiKYIglCp+RL8ngFXG8er4uQSI6CoiWgbgLgA/C6d5/tmzB5g+3Vv0BUEQShk/0TtOWeeT\nbGVmfgTAI0Q0CsBtAC5zqqy2tvar/ZqaGtTU1Phogjc336y2BxwQSnWCIAh5o66uDnV1dVmpm9jD\n10FExwGoZebh8eMbATAzj3cpTwC2MnNSgmMiYq/3S5cxY4DHHwc2bwb23z8rbyEIgpAXiAjMHMqy\nT37cO7MB9COiPkRUDmAUgKm2BvUzDkcAWBJG44KgE6iJe0cQBMEdT/cOMzcT0dUAZkBdJCYy80Ii\nGgdgNjO/DOBqIjoFQCOArQAuzWajnTjmGLWV2baCIAjueLp3Qn2zLLp3Jk4E3nsPePLJrFQvCIKQ\nN3Lt3ikKGhslc6YgCIIXkRH9WExEXxAEwYvIiH5Dg6yGJQiC4EVkRH/ePGDv3ny3QhAEobCJjOhX\nVgJHHpnvVgiCIBQ2kRH9WEzcO4IgCF5ERvQlekcQBMGbyIi+WPqCIAjeREb0xdIXBEHwJjKiL5a+\nIAiCN5ERfbH0BUEQvImM6IulLwiC4E1kRL+xUURfEATBi8iIvuTeEQRB8CYSoj9pErBsGdCuXb5b\nIgiCUNhEQvSfekpte/TIbzsEQRAKnUiIfiyW7xYIgiAUB5EQ/YMPzncLBEEQioNIiH51NXDzzflu\nhSAIQuHjS/SJaDgRLSKiJUQ01uH564hoPhHNI6LXiejA8JvqjkTuCIIg+MNT9ImoFYAJAIYBGAhg\nNBENsBX7GMAxzDwIwBQA94Td0FSI6AuCIPjDj6U/BMBSZl7BzE0AJgEYaRZg5reZuSF++CGAnuE2\nMzUi+oIgCP7wI/o9AawyjlcjtahfDmBaJo0KAjPw8ssi+oIgCH4o81GGHM6xY0Gi7wE4BsCJbpXV\n1tZ+tV9TU4OamhofTXBn61bgiy+A4cMzqkYQBKFgqKurQ11dXVbqJmZH/bYKEB0HoJaZh8ePbwTA\nzDzeVu4UAA8COIGZN7vUxV7vF5R164Cjj1ZbQRCEKEJEYGYnAzwwftw7swH0I6I+RFQOYBSAqbYG\nDQbwKICz3AQ/WzQ0qEXRBUEQBG88RZ+ZmwFcDWAGgPkAJjHzQiIaR0Qj4sXuBrAPgMlENJeIXsxa\ni23IIK4gCIJ//Pj0wczTAfS3nbvD2D815Hb5Rix9QRAE/xT9jNwlS4CdO/PdCkEQhOKg6EW/oQEY\nOjTfrRAEQSgOilb0GxqAlhZg7Vqgfft8t0YQBKE4KFrRb9sWuPNO4J57gO7d890aQRCE4qBoRR8A\nFixQq2X94Af5bokgCEJxUNSiv2oVsHIlUFWV75YIgiAUB54zckN9sxBn5FJ8blr37kr4y3wFnwqC\nIBQfuZ6RW9DccIMIviAIgl+KUvTnzrX2xbUjCILgn6IU/e9+19qX2biCIAj+KUrR32ykdOuZ0+Va\nBEEQipuiFP127ax9mZglCILgn6IU/TPOAIYNA95/H/j61/PdGkEQhOKhKONempqAkSOB44/Pd0sE\nQRCKi6K09JctA5qb890KQRCE4qMoRb+8HOjbN9+tEARBKD6KUvSbmpTwC4IgCMEoStGXJRIFQRDS\nQ0RfEAShhPAl+kQ0nIgWEdESIhrr8Py3iegjImoionPDb6bF4sXArFki+oIgCOngKfpE1ArABADD\nAAwEMJqIBtiKrQBwKYC/hd5CG9//vtqK6AuCIATHT5z+EABLmXkFABDRJAAjASzSBZh5Zfy5rOdp\n3rRJbWUgVxAEITh+3Ds9AawyjlfHz+WU3buBu+6yRF8sfUEQhOD4sfSdEvenbdHX1tZ+tV9TU4Oa\nmhpfr3vvPeCmm4DBg4HDD5dEa4IgRJe6ujrU1dVlpW7PlbOI6DgAtcw8PH58IwBm5vEOZZ8C8C9m\n/qdLXWmvnPX668BppynRf+IJ4Jhj0qpGEASh6Ahz5Sw/lv5sAP2IqA+AdQBGARidqn1hNEzT3Ayc\ndBKwfLk6njsX6NAhzHcQBEEoHXytkUtEwwE8CDUGMJGZ7yKicQBmM/PLRHQsgBcA7AugAcCXzHyE\nQz2BLf0dO9Q6uO++Cxx9tDrX0mKtkSsIghB1wrT0C35h9E2bgAED1JZIrYfb1JSlBgqCIBQguXbv\n5BVz9u2FFyYuoCIIgiAEo6hE/7nn8tsWQRCEYqfgc+9s2gRs25bvVgiCIESDgvfpDxig8u3ksJmC\nIAgFRUkN5OooHRF9QRBKlcgP5DIDEyeqhc8FQRCE8ChI0d++HbjqKis086KL8tseQRCEqFCQA7l7\n9gCdOlnHv/pV/toiCIIQJQrK0v/d74DPPgOqqoDGRut8ZWX+2iQIghAlCmogt18/Nfnqk0+Aa65R\n4Zp//zuwfj3QpUvOmikIglBQhDmQW1DunT17VNpkAPj2t4FRo9S+5M4XBEEIh4Jx74weDWzcCPTu\nrY47dgTatlX7eisIgiBkRsG4d/r2VWkWBg8GVq8GqqvV+Y0bxbUjCEJpE8nJWZ06AQsWAF275qw5\ngiAIRUHkfPq7dgFbtiiXjiAIgpA9CkL0t29XC6VIaKYgCEJ2yetA7syZQJ8+wJo1allEQRAEIbvk\nTfQ3bQJOPFGlWOjbFzj44Hy1RBAEoXTw5d4houFEtIiIlhDRWIfny4loEhEtJaIPiKi3V527dlnb\nWAy44IKgTRcEQRCC4in6RNQKwAQAwwAMBDCaiAbYil0OYAszHwLgAQB3u9V3553qcd996njBArXo\neVVVeh0oJOrq6vLdhKwS5f5FuW+A9E+w8GPpDwGwlJlXMHMTgEkARtrKjATw5/j+PwCc7FZZY6N6\ndOwIPPIIcMklwPDhwGmnpdP8wiLqP7wo9y/KfQOkf4KFH59+TwCrjOPVUBcCxzLM3ExE24hof2be\nYq/szjvTbaogCIKQKX4sfacJAfYZVvYy5FBGEARByDOeM3KJ6DgAtcw8PH58IwBm5vFGmWnxMv8h\notYA1jFzUvIEIpILgSAIQhrkcrnE2QD6EVEfAOsAjAIw2lbmXwAuBfAfABcAeNOporAaLQiCIKSH\np+jHffRXA5gB5Q6ayMwLiWgcgNnM/DKAiQCeIaKlADZDXRgEQRCEAiOnCdcEQRCE/JKz3DteE7wK\nESKaSETriehT49x+RDSDiBYT0WtE1NF47qH4BLV5RDTIOH9pvN+LieiSXPfDDSLqRURvEtECIvqM\niH4WPx+JPhJRBRH9h4jmxvt3R/x8XyL6MN7WvxNRWfy86yRDIropfn4hERVMgDERtSKij4loavw4\nSn1bTkSfxL+/WfFzkfhtAgARdSSiyfHPfT4RDc1J/5g56w+oi8syAH0AtAEwD8CAXLx3hu3+FoBB\nAD41zo0H8Mv4/lgAd8X3TwfwSnx/KIAP4/v7AfgcQEcA++r9fPct3rZuAAbF99sBWAxgQMT6WBXf\ntgbwYbzdzwG4IH7+jwDGxPd/AuCR+P5FACbF9w8DMBfKHdo3/lumfPct3rbrAPwVwNT4cZT69gWA\n/WznovTbfBrAD+L7ZfE2Zr1/uerccQCmGcc3Ahib7w/dZ9v7IFH0FwHoGt/vBmBhfP9RABcZ5RYC\n6Ao1vvFH4/wfzXKF9ADwIoBTothHAFUA5kDNMdkAoFX8/Fe/TQDTAQyN77cGsCG+n/B7BTBNl8tz\nn3oBeB1ADSzR3xiFvsXb8j8AnWznIvHbBNAewOcO57Pev1y5d5wmePXM0XuHTRdmXg8AzPwlAB2a\n6tZH+/k1KMC+E1FfqLuaD6F+dJHoY9z9MRfAl1AC+TmAbczcEi9i/hYTJhkC2E5E+6Nw+3c/gF8g\nPieGiDoB2BqRvgGqX68R0WwiuiJ+Liq/zYMAbCKip+LuuceJqAo56F+uRN/PBK9ix22CWsH3nYja\nQaXP+D9m3gX39hVdH5m5hZkHQ1nFQwB8zalYfOvWj4LrHxGdCWA9M8+D1T5CcluLrm8G32DmYwGc\nAeCnRPRtROe3WQbgaAB/YOajAeyGuuvKev9yJfqrAZiZN3sBWJuj9w6b9UTUFQCIqBuUqwBQfTzQ\nKKf7WNB9jw/0/QPAM8z8Uvx0pPoIAMy8A8DbUC6PfUklEgQS2/pV/0hNMuzIzFvh3u988k0AZxHR\nFwD+DuAkqGSHHSPQNwBfWbpg5o1QrschiM5vczWAVcw8J348BeoikPX+5Ur0v5rgRUTlUH6oqTl6\n70yxW09TAVwW378MwEvG+UuAr2Yxb4vfpr0G4NT4SP1+AE6NnysUngSwgJkfNM5Foo9E1FlHPxBR\nW6jxigUA3oKaRAioSYVm/y6N75uTDKcCGBWPgKkG0A/ArOz3wB1mvpmZezPzQVD/pzeZ+XuIQN8A\ngIiq4negIKJ9AJwG4DNE5LcZb9sqIjo0fupkAPORi/7lcOBiOFR0yFIAN+Z7IMVnm5+FumrGAKwE\n8AOo0fJ/x/vyOoB9jfIToKIfPgFwtHH+sni/lwC4JN/9Mtr1TQDNUNFUcwF8HP+e9o9CHwEcEe/T\nPACfArglfr4aavb4Eqholzbx8xUAno/340MAfY26bor3eyGA0/LdN1s/T4Q1kBuJvsX7oX+Xn2nN\niMpvM96uo6AM4nkA/gkVgZP1/snkLEEQhBKiIBZGFwRBEHKDiL4gCEIJIaIvCIJQQojoC4IglBAi\n+oIgCCWEiL4gCEIJIaIvCIJQQojoC4IglBD/H+OsDt9NPuhbAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x10908d410>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Average of 100 rewards till that episode vs episodes\n",
    "plt.plot(get_consecutive_100_episode_average_list(result['episode_rewards']))\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best 100-episode average reward was 0.73\n",
      "The average reward was 0.521824231748\n"
     ]
    }
   ],
   "source": [
    "print \"Best 100-episode average reward was {}\".format( \\\n",
    "                max(get_consecutive_100_episode_average_list(result['episode_rewards'])))\n",
    "\n",
    "print \"The average reward was {}\".format(sum(result['episode_rewards'])/len(result['episode_rewards']))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***References :***   \n",
    "[1] https://gym.openai.com/envs/FrozenLake-v0  \n",
    "[2] https://www.udacity.com/course/machine-learning-engineer-nanodegree--nd009  \n",
    "[3] https://medium.com/emergent-future/simple-reinforcement-learning-with-tensorflow-part-0-q-learning-with-tables-and-neural-networks-d195264329d0#.iljgpzh7m  \n",
    "[4] http://karpathy.github.io/2016/05/31/rl/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}