conditional-char-rnn/conditional-char-rnn.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "9a73330c-27c1-4957-8e95-c3b42bc14a71"
    }
   },
   "source": [
    "![](https://i.imgur.com/eBRPvWB.png)\n",
    "\n",
    "# Practical PyTorch: Generating Names with a Conditional Character-Level RNN\n",
    "\n",
    "[In the last tutorial](https://github.com/spro/practical-pytorch/blob/master/char-rnn-classification/char-rnn-classification.ipynb) we used a RNN to classify names into their language of origin. This time we'll turn around and generate names from languages. This model will improve upon the RNN we used to [generate Shakespeare one character at a time](https://github.com/spro/practical-pytorch/blob/master/char-rnn-generation/char-rnn-generation.ipynb) by adding another input (representing the language) so we can specify what kind of name to generate.\n",
    "\n",
    "```\n",
    "> python generate.py Russian\n",
    "Rovakov\n",
    "Uantov\n",
    "Shavakov\n",
    "\n",
    "> python generate.py German\n",
    "Gerren\n",
    "Ereng\n",
    "Rosher\n",
    "\n",
    "> python generate.py Spanish\n",
    "Salla\n",
    "Parer\n",
    "Allan\n",
    "\n",
    "> python generate.py Chinese\n",
    "Chan\n",
    "Hang\n",
    "Iun\n",
    "```\n",
    "\n",
    "Being able to \"prime\" the generator with a specific category brings us a step closer to the [Sequence to Sequence model](https://github.com/spro/practical-pytorch/blob/master/seq2seq-translation/seq2seq-translation.ipynb) used for machine translation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Recommended Reading\n",
    "\n",
    "I assume you have at least installed PyTorch, know Python, and understand Tensors:\n",
    "\n",
    "* http://pytorch.org/ For installation instructions\n",
    "* [Deep Learning with PyTorch: A 60-minute Blitz](https://github.com/pytorch/tutorials/blob/master/Deep%20Learning%20with%20PyTorch.ipynb) to get started with PyTorch in general\n",
    "* [jcjohnson's PyTorch examples](https://github.com/jcjohnson/pytorch-examples) for an in depth overview\n",
    "* [Introduction to PyTorch for former Torchies](https://github.com/pytorch/tutorials/blob/master/Introduction%20to%20PyTorch%20for%20former%20Torchies.ipynb) if you are former Lua Torch user\n",
    "\n",
    "It would also be useful to know about RNNs and how they work:\n",
    "\n",
    "* [The Unreasonable Effectiveness of Recurrent Neural Networks](http://karpathy.github.io/2015/05/21/rnn-effectiveness/) shows a bunch of real life examples\n",
    "* [Understanding LSTM Networks](http://colah.github.io/posts/2015-08-Understanding-LSTMs/) is about LSTMs specifically but also informative about RNNs in general\n",
    "\n",
    "I also suggest the previous tutorials:\n",
    "\n",
    "* [Classifying Names with a Character-Level RNN](https://github.com/spro/practical-pytorch/blob/master/char-rnn-classification/char-rnn-classification.ipynb) for using an RNN to classify text into categories\n",
    "* [Generating Shakespeare with a Character-Level RNN](https://github.com/spro/practical-pytorch/blob/master/char-rnn-generation/char-rnn-generation.ipynb) for using an RNN to generate one character at a time"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "cc294dae-dd8f-4288-8d3c-bb9fd3ad19bc"
    }
   },
   "source": [
    "# Preparing the Data\n",
    "\n",
    "See [Classifying Names with a Character-Level RNN](https://github.com/spro/practical-pytorch/blob/master/char-rnn-classification/char-rnn-classification.ipynb) for more detail - we're using the exact same dataset. In short, there are a bunch of plain text files `data/names/[Language].txt` with a name per line. We split lines into an array, convert Unicode to ASCII, and end up with a dictionary `{language: [names ...]}`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false,
    "nbpresent": {
     "id": "6a9d80df-1d38-4c41-849c-95e38da98cc7"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "O'Neal\n"
     ]
    }
   ],
   "source": [
    "import glob\n",
    "import unicodedata\n",
    "import string\n",
    "\n",
    "all_letters = string.ascii_letters + \" .,;'-\"\n",
    "n_letters = len(all_letters) + 1 # Plus EOS marker\n",
    "EOS = n_letters - 1\n",
    "\n",
    "# Turn a Unicode string to plain ASCII, thanks to http://stackoverflow.com/a/518232/2809427\n",
    "def unicode_to_ascii(s):\n",
    "    return ''.join(\n",
    "        c for c in unicodedata.normalize('NFD', s)\n",
    "        if unicodedata.category(c) != 'Mn'\n",
    "        and c in all_letters\n",
    "    )\n",
    "\n",
    "print(unicode_to_ascii(\"O'Néàl\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "# categories: 18 ['Arabic', 'Chinese', 'Czech', 'Dutch', 'English', 'French', 'German', 'Greek', 'Irish', 'Italian', 'Japanese', 'Korean', 'Polish', 'Portuguese', 'Russian', 'Scottish', 'Spanish', 'Vietnamese']\n"
     ]
    }
   ],
   "source": [
    "# Read a file and split into lines\n",
    "def read_lines(filename):\n",
    "    lines = open(filename).read().strip().split('\\n')\n",
    "    return [unicode_to_ascii(line) for line in lines]\n",
    "\n",
    "# Build the category_lines dictionary, a list of lines per category\n",
    "category_lines = {}\n",
    "all_categories = []\n",
    "for filename in glob.glob('../data/names/*.txt'):\n",
    "    category = filename.split('/')[-1].split('.')[0]\n",
    "    all_categories.append(category)\n",
    "    lines = read_lines(filename)\n",
    "    category_lines[category] = lines\n",
    "\n",
    "n_categories = len(all_categories)\n",
    "\n",
    "print('# categories:', n_categories, all_categories)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "4ff5f52a-2523-47f0-beba-f6c29d412e5f"
    }
   },
   "source": [
    "# Creating the Network\n",
    "\n",
    "This network extends [the last tutorial's RNN](#Creating-the-Network) with an extra argument for the category tensor, which is concatenated along with the others. The category tensor is a one-hot vector just like the letter input.\n",
    "\n",
    "We will interpret the output as the probability of the next letter. When sampling, the most likely output letter is used as the next input letter.\n",
    "\n",
    "I added a second linear layer `o2o` (after combining hidden and output) to give it more muscle to work with. There's also a dropout layer, which [randomly zeros parts of its input](https://arxiv.org/abs/1207.0580) with a given probability (here 0.1) and is usually used to fuzz inputs to prevent overfitting. Here we're using it towards the end of the network to purposely add some chaos and increase sampling variety.\n",
    "\n",
    "![](https://i.imgur.com/jzVrf7f.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true,
    "nbpresent": {
     "id": "597a765d-634b-41a8-a0c6-be5c019da150"
    }
   },
   "outputs": [],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "from torch.autograd import Variable\n",
    "\n",
    "class RNN(nn.Module):\n",
    "    def __init__(self, input_size, hidden_size, output_size):\n",
    "        super(RNN, self).__init__()\n",
    "        self.input_size = input_size\n",
    "        self.hidden_size = hidden_size\n",
    "        self.output_size = output_size\n",
    "        \n",
    "        self.i2h = nn.Linear(n_categories + input_size + hidden_size, hidden_size)\n",
    "        self.i2o = nn.Linear(n_categories + input_size + hidden_size, output_size)\n",
    "        self.o2o = nn.Linear(hidden_size + output_size, output_size)\n",
    "        self.softmax = nn.LogSoftmax()\n",
    "    \n",
    "    def forward(self, category, input, hidden):\n",
    "        input_combined = torch.cat((category, input, hidden), 1)\n",
    "        hidden = self.i2h(input_combined)\n",
    "        output = self.i2o(input_combined)\n",
    "        output_combined = torch.cat((hidden, output), 1)\n",
    "        output = self.o2o(output_combined)\n",
    "        return output, hidden\n",
    "\n",
    "    def init_hidden(self):\n",
    "        return Variable(torch.zeros(1, self.hidden_size))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "8ff6da45-57cd-46ca-b14a-3f560ce4d345"
    }
   },
   "source": [
    "# Preparing for Training\n",
    "\n",
    "First of all, helper functions to get random pairs of (category, line):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import random\n",
    "\n",
    "# Get a random category and random line from that category\n",
    "def random_training_pair():\n",
    "    category = random.choice(all_categories)\n",
    "    line = random.choice(category_lines[category])\n",
    "    return category, line"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For each timestep (that is, for each letter in a training word) the inputs of the network will be `(category, current letter, hidden state)` and the outputs will be `(next letter, next hidden state)`. So for each training set, we'll need the category, a set of input letters, and a set of output/target letters.\n",
    "\n",
    "Since we are predicting the next letter from the current letter for each timestep, the letter pairs are groups of consecutive letters from the line - e.g. for `\"ABCD<EOS>\"` we would create (\"A\", \"B\"), (\"B\", \"C\"), (\"C\", \"D\"), (\"D\", \"EOS\").\n",
    "\n",
    "![](https://i.imgur.com/JH58tXY.png)\n",
    "\n",
    "The category tensor is a [one-hot tensor](https://en.wikipedia.org/wiki/One-hot) of size `<1 x n_categories>`. When training we feed it to the network at every timestep - this is a design choice, it could have been included as part of initial hidden state or some other strategy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false,
    "nbpresent": {
     "id": "cf311809-10bf-40f7-87e1-1952342f7f35"
    }
   },
   "outputs": [],
   "source": [
    "# One-hot vector for category\n",
    "def make_category_input(category):\n",
    "    li = all_categories.index(category)\n",
    "    tensor = torch.zeros(1, n_categories)\n",
    "    tensor[0][li] = 1\n",
    "    return Variable(tensor)\n",
    "\n",
    "# One-hot matrix of first to last letters (not including EOS) for input\n",
    "def make_chars_input(chars):\n",
    "    tensor = torch.zeros(len(chars), n_letters)\n",
    "    for ci in range(len(chars)):\n",
    "        char = chars[ci]\n",
    "        tensor[ci][all_letters.find(char)] = 1\n",
    "    tensor = tensor.view(-1, 1, n_letters)\n",
    "    return Variable(tensor)\n",
    "\n",
    "# LongTensor of second letter to end (EOS) for target\n",
    "def make_target(line):\n",
    "    letter_indexes = [all_letters.find(line[li]) for li in range(1, len(line))]\n",
    "    letter_indexes.append(n_letters - 1) # EOS\n",
    "    tensor = torch.LongTensor(letter_indexes)\n",
    "    return Variable(tensor)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For convenience during training we'll make a `random_training_set` function that fetches a random (category, line) pair and turns them into the required (category, input, target) tensors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Make category, input, and target tensors from a random category, line pair\n",
    "def random_training_set():\n",
    "    category, line = random_training_pair()\n",
    "    category_input = make_category_input(category)\n",
    "    line_input = make_chars_input(line)\n",
    "    line_target = make_target(line)\n",
    "    return category_input, line_input, line_target"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "53fb987f-4f42-4bf8-81ae-280ebdd19aee"
    }
   },
   "source": [
    "# Training the Network\n",
    "\n",
    "In contrast to classification, where only the last output is used, we are making a prediction at every step, so we are calculating loss at every step.\n",
    "\n",
    "The magic of autograd allows you to simply sum these losses at each step and call backward at the end. But don't ask me why initializing loss with 0 works."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false,
    "nbpresent": {
     "id": "df50f546-6d02-4383-beab-90378f16576b"
    }
   },
   "outputs": [],
   "source": [
    "def train(category_tensor, input_line_tensor, target_line_tensor):\n",
    "    hidden = rnn.init_hidden()\n",
    "    optimizer.zero_grad()\n",
    "    loss = 0\n",
    "    \n",
    "    for i in range(input_line_tensor.size()[0]):\n",
    "        output, hidden = rnn(category_tensor, input_line_tensor[i], hidden)\n",
    "        loss += criterion(output, target_line_tensor[i])\n",
    "\n",
    "    loss.backward()\n",
    "    optimizer.step()\n",
    "    \n",
    "    return output, loss.data[0] / input_line_tensor.size()[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To keep track of how long training takes I am adding a `time_since(t)` function which returns a human readable string:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import time\n",
    "import math\n",
    "\n",
    "def time_since(t):\n",
    "    now = time.time()\n",
    "    s = now - t\n",
    "    m = math.floor(s / 60)\n",
    "    s -= m * 60\n",
    "    return '%dm %ds' % (m, s)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Training is business as usual - call train a bunch of times and wait a few minutes, printing the current time and loss every `print_every` epochs, and keeping store of an average loss per `plot_every` epochs in `all_losses` for plotting later."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false,
    "nbpresent": {
     "id": "81fde336-785e-461b-a751-718a5f6bff88"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0m 28s (5000 5%) 1.8674\n",
      "0m 53s (10000 10%) 2.4155\n",
      "1m 20s (15000 15%) 3.4203\n",
      "1m 45s (20000 20%) 1.3962\n",
      "2m 12s (25000 25%) 1.7427\n",
      "2m 38s (30000 30%) 2.9514\n",
      "3m 4s (35000 35%) 2.8836\n",
      "3m 31s (40000 40%) 1.6728\n",
      "3m 57s (45000 45%) 2.5014\n",
      "4m 22s (50000 50%) 1.9687\n",
      "4m 48s (55000 55%) 1.5595\n",
      "5m 16s (60000 60%) 2.3830\n",
      "5m 43s (65000 65%) 1.5155\n",
      "6m 10s (70000 70%) 1.7967\n",
      "6m 37s (75000 75%) 1.8564\n",
      "7m 3s (80000 80%) 1.9873\n",
      "7m 30s (85000 85%) 1.9569\n",
      "7m 56s (90000 90%) 1.7553\n",
      "8m 22s (95000 95%) 2.3103\n",
      "8m 48s (100000 100%) 1.7575\n"
     ]
    }
   ],
   "source": [
    "n_epochs = 100000\n",
    "print_every = 5000\n",
    "plot_every = 500\n",
    "all_losses = []\n",
    "loss_avg = 0 # Zero every plot_every epochs to keep a running average\n",
    "learning_rate = 0.0005\n",
    "\n",
    "rnn = RNN(n_letters, 128, n_letters)\n",
    "optimizer = torch.optim.Adam(rnn.parameters(), lr=learning_rate)\n",
    "criterion = nn.CrossEntropyLoss()\n",
    "\n",
    "start = time.time()\n",
    "\n",
    "for epoch in range(1, n_epochs + 1):\n",
    "    output, loss = train(*random_training_set())\n",
    "    loss_avg += loss\n",
    "    \n",
    "    if epoch % print_every == 0:\n",
    "        print('%s (%d %d%%) %.4f' % (time_since(start), epoch, epoch / n_epochs * 100, loss))\n",
    "\n",
    "    if epoch % plot_every == 0:\n",
    "        all_losses.append(loss_avg / plot_every)\n",
    "        loss_avg = 0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Plotting the Network\n",
    "\n",
    "Plotting the historical loss from all_losses shows the network learning:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x1102b8f60>]"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAg0AAAFkCAYAAACjCwibAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAAPYQAAD2EBqD+naQAAIABJREFUeJzt3XecVOX1x/HvoYs00YCoiBo1YAnKqhDsNZZYMLYVI5JY\nAc0PNZY0C2rUREUFYkw0wUaMsXdjUOyKgA17xQZEwQXpu3t+f5y9zuyy5c6yuzO7fN6v176WuXPL\nM3OHvWfOc57nmrsLAACgLq3y3QAAANA8EDQAAIBUCBoAAEAqBA0AACAVggYAAJAKQQMAAEiFoAEA\nAKRC0AAAAFIhaAAAAKkQNAAAgFRyChrM7GQze9XMSip+njOzfWtZf4iZPWZmc7PW32fVmw0AAJpa\nrpmGTyWdLWmApCJJkyXda2b9alh/F0mPSdqvYpsnJN1vZv3r11wAAJAvtqo3rDKzryWd6e5/T7n+\nG5L+6e4XrdKBAQBAk2pT3w3NrJWkIyR1lPR8ym1MUmdJ8+p7XAAAkB85Bw1mtpUiSOggaaGkIe7+\ndsrNfyVpTUn/quMYa0v6saSPJS3NtY0AAKzGOkjaSNKj7v51Q+64PpmGtyX1l9RV0mGSbjKzXeoK\nHMzsaEm/k3SQu39VxzF+LOnWerQNAACEoZJua8gd5hw0uHuppA8rHs4wsx0k/VLSKTVtY2ZHSbpe\n0mHu/kSKw3wsSbfccov69aupxhLNyejRo3XVVVfluxloIJzPloXz2bK89dZbOuaYY6SKa2lDqndN\nQ5ZWktrX9KSZFUv6m6Qj3f2RlPtcKkn9+vXTgAEDVr2FyLuuXbtyLlsQzmfLwvlssRq8ez+noMHM\nLpH0sKRZioLGoZJ2lbRPxfN/kLSeuw+reHy0pH9IOk3SVDPrWbGrJe6+oCFeAAAAaBq5ztPQQ9JE\nRV3D44q5GvZx98kVz68rqXfW+idIai1pvKQvsn7GrkKbAQBAHuSUaXD34+t4fniVx7vXp1EAAKDw\ncO8JNIni4uJ8NwENiPPZsnA+kRZBA5oEf5RaFs5ny8L5RFoEDQAAIBWCBgAAkApBAwAASIWgAQAA\npELQAAAAUiFoAAAAqRA0AACAVAgaAABAKgQNAAAgFYIGAACQCkEDAABIhaABAACkQtAAAABSIWgA\nAACpEDQAAIBUCBoAAEAqBA0AACAVggYAAJAKQQMAAEiFoAEAAKRC0AAAAFIhaAAAAKkQNAAAgFQI\nGgAAQCoEDQAAIBWCBgAAkApBAwAASIWgAQAApELQAAAAUiFoAAAAqRA0AACAVHIKGszsZDN71cxK\nKn6eM7N969hmNzObZmZLzexdMxu2ak0GAAD5kGum4VNJZ0saIKlI0mRJ95pZv+pWNrONJD0g6b+S\n+ku6WtLfzGzverYXAADkSZtcVnb3B6ss+q2ZnSJpkKS3qtnkFEkfuvtZFY/fMbOdJI2W9J9cGwsA\nAPKn3jUNZtbKzI6S1FHS8zWsNkjS41WWPSrpR/U9LgAAyI+cMg2SZGZbKYKEDpIWShri7m/XsPq6\nkuZUWTZHUhcza+/uy3I9PgAAyI/6ZBreVtQn7CDpz5JuMrO+DdoqAABQcHLONLh7qaQPKx7OMLMd\nJP1SUb9Q1WxJPass6ylpQZosw+jRo9W1a9dKy4qLi1VcXJxrswEAaHEmTZqkSZMmVVpWUlLSaMcz\nd1+1HZj9V9In7v7zap67VNJ+7t4/a9ltkrq5+/617HOApGnTpk3TgAEDVql9AACsTqZPn66ioiJJ\nKnL36Q2575wyDWZ2iaSHJc2S1FnSUEm7Stqn4vk/SFrP3ZO5GK6TNNLMLpN0o6Q9JR0mqcaAAQAA\nFKZcuyd6SJooqZekEkmvSdrH3SdXPL+upN7Jyu7+sZkdIOkqSadJ+kzSL9y96ogKAABQ4HKdp+H4\nOp4fXs2ypxQTQQEAgGaMe08AAIBUCBoAAEAqBA0AACAVggYAAJAKQQMAAEiFoAEAAKRC0AAAAFIh\naAAAAKkQNAAAgFQIGgAAQCoEDQAAIBWCBgAAkApBAwAASIWgAQAApFLQQUNpab5bAAAAEgUdNKxY\nke8WAACAREEHDcuX57sFAAAgQdAAAABSKeigge4JAAAKR0EHDcuW5bsFAAAgUdBBA5kGAAAKR0EH\nDdQ0AABQOAgaAABAKgUdNNA9AQBA4SjooIFCSAAACkdBBw1kGgAAKBwFHTRQ0wAAQOEo6KCBTAMA\nAIWjoIMGMg0AABQOggYAAJAKQQMAAEiFoAEAAKRS0EEDhZAAABSOgg4ayDQAAFA4cgoazOxcM3vJ\nzBaY2Rwzu9vMNk+x3VAze8XMFpnZF2Z2g5l1r2s7ggYAAApHrpmGnSVdK2mgpL0ktZX0mJmtUdMG\nZrajpImS/ippC0mHSdpB0vV1HYygAQCAwtEml5Xdff/sx2Z2nKS5kookPVPDZoMkfeTu4ysef2Jm\nf5F0Vl3HI2gAAKBwrGpNQzdJLmleLes8L6m3me0nSWbWU9Lhkh6sa+cUQgIAUDjqHTSYmUkaK+kZ\nd3+zpvXc/TlJx0i63cyWS/pS0nxJo+o6BpkGAAAKR07dE1VMUNQo7FjbSma2haSrJZ0v6TFJvST9\nSdJfJB1f27avvTZaBx3UtdKy4uJiFRcX17vRAAC0FJMmTdKkSZMqLSspKWm045m7576R2ThJB0ra\n2d1n1bHuTZI6uPsRWct2lPS0pF7uPqeabQZImlZUNE0vvzwg5/YBALC6mj59uoqKiiSpyN2nN+S+\nc+6eqAgYDpa0e10BQ4WOkkqrLCtX1EJYbRvSPQEAQOHIdZ6GCZKGSjpa0iIz61nx0yFrnUvMbGLW\nZvdL+qmZnWxmG1dkGa6W9KK7z67teBRCAgBQOHKtaThZkSF4ssry4ZJuqvh3L0m9kyfcfaKZdZI0\nUlHL8I2k/0o6p66DkWkAAKBw5DpPQ52ZCXcfXs2y8ZLGV7N6rQgaAAAoHNx7AgAApELQAAAAUino\noIFCSAAACkdBBw1kGgAAKBwFHTSsWCHVY+4pAADQCAo6aJDINgAAUCgKPmhYtizfLQAAABJBAwAA\nSImgAQAApELQAAAAUin4oGHp0ny3AAAASM0gaCDTAABAYSBoAAAAqRA0AACAVAgaAABAKgUfNFAI\nCQBAYSj4oIFMAwAAhYGgAQAApELQAAAAUinooKFNG4IGAAAKRUEHDe3aUQgJAEChKPiggUwDAACF\ngaABAACkQtAAAABSKeigoW1bggYAAApFQQcNFEICAFA4Cj5oINMAAEBhIGgAAACpEDQAAIBUCj5o\nWLIk360AAABSgQcNnTpJCxbkuxUAAEBqBkHDN9/kuxUAAEBqBkFDSUm+WwEAAKQcgwYzO9fMXjKz\nBWY2x8zuNrPNU2zXzswuNrOPzWypmX1oZsfVtV3nzgQNAAAUijY5rr+zpGslvVyx7R8kPWZm/dy9\ntpLFOyR9T9JwSR9I6qUUAUtS01BWJrVunWNLAQBAg8opaHD3/bMfV2QL5koqkvRMdduY2b6KYGMT\nd08qFGalOV7nzvF74UKpW7dcWgoAABraqtY0dJPkkubVss6BiszE2Wb2mZm9Y2Z/NLMOde28U6f4\nTTEkAAD5l2v3xHfMzCSNlfSMu79Zy6qbKDINSyUdImkdSX+W1F3SL2o7RpJpoK4BAID8q3fQIGmC\npC0k7VjHeq0klUs62t2/lSQzO13SHWY2wt1rnPPxmmtGS+qqESOktdeOZcXFxSouLl6FZgMA0DJM\nmjRJkyZNqrSspBG/aZu7576R2ThFt8PO7l5rfYKZ/UPSYHffPGtZX0kzJW3u7h9Us80ASdP+859p\n2nvvAbr3Xumgg3JuJgAAq53p06erqKhIkorcfXpD7jvnmoaKgOFgSbvXFTBUeFbSembWMWvZDxTZ\nh89q25CaBgAACkeu8zRMkDRU0tGSFplZz4qfDlnrXGJmE7M2u03S15L+bmb9zGwXSZdLuqG2rgkp\n7j3Rvj01DQAAFIJcMw0nS+oi6UlJX2T9HJG1Ti9JvZMH7r5I0t6KkRZTJd0s6V5Jv0xzwG7dyDQA\nAFAIcp2noc4gw92HV7PsXUk/zuVYia5dyTQAAFAICvreExKZBgAACkXBBw1kGgAAKAwFHzSQaQAA\noDAUfNBApgEAgMJA0AAAAFIp+KCB7gkAAApDwQcNZBoAACgMBR80dOsmLVsmLV2a75YAALB6K/ig\noWvX+E22AQCA/Cr4oKFbt/hNXQMAAPlV8EEDmQYAAApDwQcNZBoAACgMBR80kGkAAKAwFHzQ0Llz\n/CbTAABAfhV80NC6tdSlC5kGAADyreCDBim6KMg0AACQX80iaOjWjUwDAAD51iyCBjINAADkX7MI\nGsg0AACQf80iaOCmVQAA5F+zCBq4PTYAAPnXLIIGMg0AAORfswkayDQAAJBfzSJo6NZNWrBAKi/P\nd0sAAFh9NYugoUcPyV2aOzffLQEAYPXVLIKGPn3i96xZ+W0HAACrs2YRNGy4Yfz+5JP8tgMAgNVZ\nswgauneX1lyTTAMAAPnULIIGs8g2kGkAACB/mkXQIEVdA5kGAADyp9kEDWQaAADIr2YTNJBpAAAg\nv5pV0DBvnvTtt/luCQAAq6ecggYzO9fMXjKzBWY2x8zuNrPNc9h+RzNbYWbTc21oMuySbAMAAPmR\na6ZhZ0nXShooaS9JbSU9ZmZr1LWhmXWVNFHS47k2UspM8ERdAwAA+dEml5Xdff/sx2Z2nKS5kook\nPVPH5tdJulVSuaSDczmuJK23ntS6NZkGAADyZVVrGrpJcknzalvJzIZL2ljSBfU9UJs20vrrk2kA\nACBfcso0ZDMzkzRW0jPu/mYt620m6RJJO7l7eWxWPxtuSKYBAIB8qXfQIGmCpC0k7VjTCmbWStEl\ncZ67f5Asru8B+/Qh0wAAQL7UK2gws3GS9pe0s7t/WcuqnSVtJ2kbMxtfsaxV7MKWS9rH3Z+saePR\no0era9eu3z1+6y1p/vxiScX1aTYAAC3KpEmTNGnSpErLSkpKGu145u65bRABw8GSdnX3D+tY1yT1\nq7J4pKTdJf1U0sfuvqSa7QZImjZt2jQNGDDgu+V/+Ys0cqS0dGnUOAAAgMqmT5+uoqIiSSpy95yn\nOKhNTpdeM5ug+Jp/kKRFZtaz4qkSd19asc4lktZ392EeEcmbVfYxV9JSd38r18b26SOVlUlffJGZ\ntwEAADSNXEdPnCypi6QnJX2R9XNE1jq9JPVuiMZVlQQK1DUAAND0cgoa3L2Vu7eu5uemrHWGu/se\ntezjAncfUNPztUkmeProo/psDQAAVkWzufeEJK25ZmQb3qxxgCcAAGgszSpokKQtt5Rmzsx3KwAA\nWP0QNAAAgFSaZdDw0UfSokX5bgkAAKuXZhk0SDHREwAAaDrNLmjoVzFV1Btv5LcdAACsbppd0NCp\nk7TRRtQ1AADQ1Jpd0CBJW21F0AAAQFNrlkEDIygAAGh6zTZomDVLWrgw3y0BAGD10WyDBomZIQEA\naErNMmjo21cyo4sCAICm1CyDho4dpU02YdglAABNqVkGDZK0ww7SAw9IpaX5bgkAAKuHZhs0nHGG\n9N570u2357slAACsHppt0FBUJP3kJ9KYMVJZWb5bAwBAy9dsgwZJ+v3vpXfeke64I98tAQCg5WvW\nQcP220v77RfZhvLyfLcGAICWrVkHDZL0m9/EfA2PPJLvlgAA0LI1+6Bh8ODIOFx1Vb5bAgBAy9bs\ngwYzafRo6fHHpddfz3drAABouZp90CBJhx0mbbCBNHZsvlsCAEDL1SKChrZtpVGjpFtvlebOzXdr\nAABomVpE0CBJJ54Yv2++Ob/tAACgpWoxQcNaa0kHHCBNmpTvlgAA0DK1mKBBkoqLpWnTYnppAADQ\nsFpU0HDAAVKnTmQbAABoDC0qaFhjDemQQyJocM93awAAaFlaVNAgRRfF229Lr76a75YAANCytLig\nYe+9pe7dpX/+M98tAQCgZWlxQUPbtnETq8mT890SAABalhYXNEjSjjtKM2ZIixfnuyUAALQcLTJo\nGDxYKi2VXn453y0BAKDlyCloMLNzzewlM1tgZnPM7G4z27yObYaY2WNmNtfMSszsOTPbZ9WaXbut\ntoqhl88915hHAQBg9ZJrpmFnSddKGihpL0ltJT1mZmvUss0ukh6TtJ+kAZKekHS/mfXPvbnptG4t\nDRpE0AAAQENqk8vK7r5/9mMzO07SXElFkp6pYZvRVRb9xswOlnSgpEYbGDl4sDR+fMzXYNZYRwEA\nYPWxqjUN3SS5pHlpNzAzk9Q5l23qY/Bg6euvmVIaAICGUu+goeLiP1bSM+7+Zg6b/krSmpL+Vd9j\npzFwYGQY6KIAAKBhrEqmYYKkLSQdlXYDMzta0u8kHe7uX63CsevUrZu05ZYEDQAANJScahoSZjZO\n0v6Sdnb3L1Nuc5Sk6yUd5u5PpNlm9OjR6tq1a6VlxcXFKi4uTtXOwYOlZ5+t/rnFi6WZM6Xtt0+1\nKwAACs6kSZM0qcpdGktKShrteOY53tmpImA4WNKu7v5hym2KJf1N0pHu/kCK9QdImjZt2jQNGDAg\np/Zlu/VW6ZhjpM8+k9Zfv/JzZ54pXXGF9O9/Sz/9ab0PAQBAQZk+fbqKiookqcjdpzfkvnOdp2GC\npKGSjpa0yMx6Vvx0yFrnEjObmPX4aEkTJZ0haWrWNl0a5iXUbP/9pTZtpHvuqby8tFS65Rapa1fp\nZz9jEigAANLItabhZEldJD0p6YusnyOy1uklqXfW4xMktZY0vso2Y+vV4hystZa0xx7SXXdVXv7o\no9KcOdJDD0n9+0sHHijNndvYrQEAoHnLdZ6GOoMMdx9e5fHuuTaqIQ0ZIo0aFcMv1147lk2cGLNG\n/uhH0T2xwQbSf/8bt9UGAADVa5H3nsh28MFSebl0//3xeP586b77pGHDYkjm+utL3/se8zkAAFCX\nFh809OoVoyjuvjse/+tf0ooV0tChmXU224ygAQCAurT4oEGKLopHH5VGjJDOOkvad98IJhKbbSa9\n+27m8eTJ0kknNX07AQAoZKtF0HDooVJZWYyiGDlSuvHGys9XzTT885/S9ddLixY1bTsBAChk9Zrc\nqbnZeGPpo4+kddeNIZhVbbZZ1DokxZKvvBLL335biqGuAABgtcg0SDFCorqAQYqgQYpsQ2mp9Prr\n8XjmzKZpGwAAzcFqkWmoS3bQ0KmTtHRpPH4zl9twAQDQwhE0KAKFXr0q1zUMHkzQAABAttWme6Iu\nyQiKV16JGojBg2vvnliyRDr7bKkR7wsCAEBBIWiokIygmDFD2nZbaYstonhy8eLq13/4Yenyy2Ok\nBQAAqwOChgpJ0PDKK5mgwV16553q13/44fh9551N10YAAPKJoKHC5ptLCxfG0MskaJAyXRSPPy69\n9Vb82z2Chh49pCeekObNy0+bAQBoSgQNFZIRFFIEDZ07S717RzHk3LlxD4vhwyNgmDlT+vzz6J4o\nLc3c1yKtb75p2LYDANAUCBoqfP/78btHj8wU01tsEUHDlVfGMMwXX5SefjqyDB07SkceGQWTVW+9\nXZtXX40bZL32WsO/BgAAGhNBQ4U11ojMwrbbxt0vpQgapk6Vxo+XfvUracstI7vwyCPS7rtLHTrE\nFNWPPip9+2264zz3XGQn7rij8V4LAACNgaAhyy9/KZ14YubxlltKX3wRt9Y+44wIHB58UJoyJW56\nJUXQsGyZ9NBD6Y6RTFF9zz0N23YAABobQUOWM86IICCRFEOOGBFdCsXFMR11WZm0337x3MYbx/0p\nbr013TFmzJB69pTeeEN6//2GbT8AAI2JoKEWRUXSr38tnXNOPG7XTrrwwsgyJDUQkvTzn0cG4ssv\na99fcl+LU0+Nrg2yDQCA5oSgoRbt2kkXXxx3vkwMH56ZoyExdGis+/e/176/d9+Ngsodd5T22Ue6\n++6GbzMAAI2FoKEBdO0qHXGEdMMNUf9Qk6SeoX9/6ZBDpOefl2bPbpo2AgCwqggaGsgJJ0gffhiT\nPdXklVekPn2ktdaSDjwwRmnkOscDAAD5QtDQQAYPlvr1k377W+mii6RrrpFWrKi8zowZ0jbbxL/X\nWSf+/fzzTd9WAADqg6ChgZhF0eRnn0njxsXwzexJn9wz97VIbLNNTPYEAEBzQNDQgI45Rvr006hT\nGDRIuvnmzHNffCF99VUm0yDFv994Y+WMBAAAhYigoZEcc0zMHPm//8XjpAgyO2jo319avrzmO2kC\nAFBICBoayZFHRpfF7bfH4yeflLp1kzbcMLNO//7xOwkoavPtt9KECbWPzgAAoDERNDSSddaJWSNv\nvjmmmL7yyphZMrmvhRRDNTfeuHJdg3v1+xs/Xho5Upo2rfbj3nxz3IETAICGRtDQiH72M+mll2IO\nhwMOiNkkq+rfP5NpmDo1hmP+/vcxCVSirEy67rr494wZNR/v3/+Wjj1Wuvba+rf566+5dTcAoHoE\nDY3oJz+JbMImm8S9KVq3XnmdbbaJoME9hmm2aiVdemnlkRWPPCJ9/LHUpYs0fXr1x5o7VzrllNi+\ntrkiajJnjnT66XFvjaOPzn17AEDLR9DQiNZYI+ZhePJJqXPn6tfZZpsYVfHGG3G77HPOiWzCGmtI\nP/6x9NFHUctQVBSzSFYXNLhLJ50UXR8XXSS9/HJu2YIvv4ybc91wgzRwYAQd2ZmOXMydK915Z+Vl\nCxZIixfXvl15ubRoUf2OCQBoGgQNjaxfP6l795qfT4ohTz89LpzHHRe35H70UalTJ2nPPeNeFyNG\nRODw2msrD9G84464+dWf/xwFmOXl0lNPpW/j2WdHhuK996Srr46A4bnncn6p3+3r8MOlhQszy4YM\nkU47rfJ6VYOSCROkzTeP0SSri/oGZgCQLwQNedanT3RhPP543Ja7R49Y3qNHdEssXBijLo46KiaG\nWrZMevvtzPYLF0qjR0cW4qc/jcLKPn2kyZPTHf/ZZ6N48g9/iGNuvXXcBvy//839tXz+eXTDuMfd\nPKWox3jhhci2ZK/XvXvlwOTpp2Mui4ceyv24NbnppnhPaiouzafFi6X11os2AkBzQdCQZ2aZuRtO\nOqnyc5tuGhf1hx6SOnbMrJfdRTFmjDR/vnTVVZn97blnJmhYtCiGfb78cnyL//hjadKk6IqYMkUa\nNUrabru4vbcUGYc99qhf0HDttXHL7zZtMvUY770XF8gPPsjMWfHEE9KSJZVrL5ICz2RCLPeo0cie\nICtXDz0Us3KmDaCa0tSpcd7+9KfCDGoAoDpt8t0ASDvtJM2bJ+2228rPbb555t+dO8fj6dOlYcOk\nN9+MYOH886WNNsqst8ce0o03RnHjiSdK990Xy1u1yszzYJa5WL3wQjyX2HNP6eSToy6iW7d0r2Hh\nwhjhcdJJ0mOPZUaEZI/2ePHFKA5Nuk6mTs1s+/77ERTdf3+8Fy+8EPt7/fUYhVIf774bvy+9NF5T\nIXn22TgHr78e78euu+a7RQBQt5wyDWZ2rpm9ZGYLzGyOmd1tZpun2G43M5tmZkvN7F0zG1b/Jrc8\nF1wQF9TsORxqMmBABA3l5fFNfKONpDPPrLzO7rvH70MPjYvwnXdGQea4cRFAzJ0b3/TffDMu7gMH\nVt5+zz1j/1Om1N2esrKY0fK88yKr8ctfRp1GkmmYMSMmtOrRIwIBKS6SrVplgobXXosA5tJLY3+3\n3SadcUZkV154IQopc+UeQcPAgdH18/LLue+jMT37rLTXXlHzsipDZLGy88/PBIwAGpi7p/6R9JCk\nn0nqJ2lrSQ9I+ljSGrVss5GkbyVdLukHkkZKWiFp71q2GSDJp02b5qjs8svd11zTffx4d8l98uTq\n1+vbN54fM6Z+x9l4Y/dTT635+bIy97/8xb179ziO5P5//xfPXXGF+xpruJeWuu+9t/vBB7sfdJD7\nnnu6f/llrHvEEfH788/dx41zb9vWfdky9333jW3N3O+4I9a5//7q21Be7n7hhe6HHhr/zvb557Ht\nXXe5f//77kOGuE+d6n799e4ff1y/96ShlJW5d+vmfsEFcR5bt3afNSu/bWop5s2L837aafluCZA/\n06ZNc0kuaYDncI1P87NqG0vrSCqXtFMt61wm6bUqyyZJeqiWbQgaavD443HW2rVzP/74mtebMCEu\n+mVl9TvO8ce7b7FF9c/NmuW+447RjuHDo01z5qzcxrffdl9nHffzz3e/5BL3zp3dJ02K5158MX7f\nc4/7L37hvs02se2tt8byX/wiAoE+fTLBSLbycvdf/zoTsPznP5Wff+KJWP7WWxHcJOtJ7vvtV7/3\npKHMnBntePxx9wUL4n359a9z38+UKe4zZlT/XHl5/c99czZ1ary3m22W75YA+dOYQcOqFkJ2q2jY\nvFrWGSTp8SrLHpX0o1U89mopubX22mtLf/xjzeudckpmsqj62Gef6L74+OPKy5ctiyGUs2bFiIgb\nb4zujGTUh5QZRvrgg5k7ew4aFLUL118fBZ7bby/17BldFDNmZF7XoYdKZ50VozmSos7Hq356FLNm\nXnJJFBL27y9dcUXl5997L177JptEkectt0QXzcSJMYT1xRfr977U17ffZuahePbZmOhr4MCoUxk6\nNNqXS0Hku+/GNOVnnFH9c337xqia1c3778fv996L4lsADay+0YYkU3RPTKljvXcknV1l2X6SyiS1\nr2EbMg21OOWUlb9ZN7QFC9w7dHD/059WPnb79u51nZr113ffeuv41vfJJ7G/Vq3i8c9/Huv85Cfu\nu+8eWZNrrql+P7fdFtt8+WVm2TXXxLLLL4/HN90Uj994I7POmWdGt0RVpaWRQUmyDbffHu2cPbv2\n1/Pgg3Wvc9ll7qefXv1zQ4ZEe0pK3IcNcx8wIPPc5MnR/hdeqH3/iWXL3IuKogunY0f35cszzz3z\nTHQZtWkTWZ7S0nT7dHdfssT9hBPc//739NsUmjFj3Lt0ie6ua6/Nd2uA/CjI7glJf5b0oaRedaxX\n76Bhl1128QMPPLDSz2233dbgbzCqd8gh7oMGZR4nXQd/+Uvd2+6/f6y71lqZeoMf/jCWTZwYjy+4\nIC58kvuX7J+vAAAcmklEQVRTT1W/n9mz4/lbb43Hd9wR25x5ZmadZcvc11svE4y4Rw1FTd0QSRfJ\nqadmjn/zzTW/ls8+i4CnpoAgsdlm8XqrdgssXBiBluT+s5+5b7pp5XqR0lL3Hj3czzij9v0nzj03\ngoJrr419vvRSLJ89O+pBdtnF/aGH4rmnn063z8WL3ffZJ7ZZYw33Dz9Mt12hGTYsPrO77x6fwWyl\npe7bbRefw8MOc3/ggbw0EWhQt91220rXyV122aWwggZJ4yR9ImnDFOtOkXRllWXHSZpfyzZkGgrA\nLbfEJ2TWLPdvvolvrkcdtXLRYXXOPTe23WOPzLITT4xlH30Ujx9+2L+rMygpqXlfW2/t/uMfR/1E\n27buRx+98oX50ksjY5FkA/r2df/lL6vfX2mpe79+/l3B3JZbRg1FTS65JNbdaqua15k1K/NaZs6s\n/Nydd8by88/PrPPPf1Ze5+STo36jrvd2ypQIdC65JIKlDh3cr7wynrv++ghu/ve/eH969qwcXNVk\nyZI4Tx07ut97r/uGG8b7neY85+L++90vuqhh91nVjju6H3NMZKHWWCNeW+Kdd+K9P+igCNy23rpx\n2wLkS0FlGioChk8lbZJy/UslvVpl2W2iELLgffNNXIjHjnU/++y4qHz2Wbptb789Pl3Z386ffNJ9\n6NDMxeh//4t1Nt209n2dfnqst+GGmYtlVV9/HQHFVVe5r1gR/x4/vuZ9vvyy+403RltGjaq+K8M9\nnt9ss8gEJKM93OMb/PrrZ7pNJk6M51u3XjkTc+yxEZi4x+uX3D/9tPI6//2vf1cg6u7+7bcrt2X+\nfPfevSOTkHQ77LprjB5xj2/Wu+ySWf/4490337zm9yB5fccdF8HHlCmx7P77oy0NndTbZZfIkPzv\nfw2732w9e0Zw9vrr8RoefTTz3N13+3ddXUmX1ty5jdeWbJddFscEcvHww/G3LVcFEzRImiBpvqSd\nJfXM+umQtc4lkiZmPd5I0kLFKIofSBohabmkvWo5DkFDgfjJT+Jbefv27r//ffrtkm91taX93WNo\n52GH1b7OvHkRcNTVP3/IIZF+fv99r3ZERU3+/e/KF/Irr4xhoosXuz/7bDx3++3xDT/p70+GjCZ1\nFcOGxQiQoqIIEhIrVrivvXZmdMSiRREgVLVihfv3vhfdFr/5Tab7IVFeHlmerl2jRiTxm9/EdgsW\nxDlKsg7u7vfd59+NIKlJMnS36gXtsMPce/XKBGjl5dHdU1sgVpv58yOgStu9le1vf3P/3e/qXm/B\ngtj/LbdEe9dfv/LIm4svznSXffpprPuvf+XWlvooLY3z1r595boboDbffhv/Z6obPVaXQgoayitq\nEar+HJu1zt8lTa6y3S6SpklaIuk9ST+r4zgEDQXiH/+IT8m660bffC7uust96dLa15kyxf3NN+vf\nvmzJvA5jx/p3BZhpzJ2budgsXBhzKEgRhAwfHt0GZWURkBQXxzfldu1iqGTfvnER6t07MiKnnea+\nySaVX1/aIseTTop127Z13377mI8jmVMiubhX/fafdPFcfHH8/uCDzHOLF0eK/rLLqj/e889HcFJd\nN07yTf322+NxEjwdcEDdr6M6ybnZYovKXVZVff55DJtMrFgRwUu7dpH5qs2MGZXf6+HD3fv3zzx/\n9NHRfZHYbLMo7G1sr7wS7eraNQpgswtX3SMwy6VgFS3Tt9/GF6TE009n/vbm+vkomKChqX4IGgrH\nvHlRjf6Pf+S7JXVbsiTauu66kW7PZZ6CLbeMdP64cVEXkEy6JLmfd16s85vfRF3HFVfEhT0pqExS\n3Q88kOmW+eKL2OaMM6I9adoyc2ZkE954I2o81l8/uhxuuSWyHNVNWFRSEu3t3DkK/Ko6+GD3wYNX\nXl5e7v6jH0VmpOpFLLHTTu677Rb/Li6O17XBBnW/juoMHx7v8V//Gu3NHg3jHu9l797+Xc3H44/H\n8nvuySyr6zOYBCZffRWPkxqPpKtnm22iriZx4okR9DW2a66JoOepp+IzlXyeEjvtlK72BPXz8cdN\nk1FaVT//eeXC86uuynz2H3sst30RNCCv6soWFJLhw+NTnWuR28iRkSHYbDP3ww+PZX/7W6T+k8LN\nJGuw1lqRvi8rizqLtdeOi0FJSdR8SHEBW7EiaiVOOKF+r+Xee2NfZvG6ago8ttkm1qsuhZ8Us779\nduXlSYbikUdqPn4y3HXy5MhIDBoUj3PtYy0vj8DpzDNj26pdL+4RHG29dRSNDhwYWZ3y8shsbLed\n+8471z0p16WXxrf5pGbmtdeivUnXVocOkYVKJEFfEuBlW748sjZps1W1OfzwTIbjV7+K2qDkm+Oy\nZRGAbrBB/QtPly/PPQtYl7KyljM52OjR8ZnLrhN66in3996rfbtXX226L0tJd5pZdOW5R0HvwIFR\n8zVsWG77I2gAUkpmo0yKA9NKvqVKkYpPZP/hXL7cvVOnyhfb3/8+Hmd/Q9hoo+iHHDky/lilnX+h\nOqecEhmQ2tKTo0ZFG6r777J0aRQHjhiRWVZe7r7DDpFpqO1CtXRpBE3du8eFLumieOKJ3F7D9OmZ\n4MM9AoSddqq8Tu/eUWzrHhf5pJupVauogRg/vu4iyuOPj8xJorQ0ztell8YFouo3tmRK8+wun88/\nj6nMkzlFWreOz0Yuvvkm8xlKAqZzzonHScFr0iWXBDZSvE/1cc45UfDakBf5k0+OIbir6qOPom23\n3LLq+6qvwYMrf/7Ky6Ow+Ygjat/uqKPic79iReO3Mfl8SvFlwT2yYKNGRWFv587R3ZgWQQOQUmlp\nFFfmes+NOXPif8MOO9R+IT3kkMguJBfxDz+M7bKngR46NOoRpEiRN7YXX4wREDW1+4IL4o9f0l/6\n4IOeulD0nHNi3ZNOij+e7dtX/raexkUXxR+9pKgyGWmSjESZP9+/qylJ7LtvLOvYMTI4s2dnAoia\n7Lab+5FHVl62xx5xzpKsTdXRP/36RbBRXh7rrLNOzPkxblwEGMXFEazceWdkmk44ITIUtRk9Otr6\n1luZi8GDD8ZzX38dj5N5R5JM0Jprxnmqyfjx8fqqnuPy8shmSTGxV02+/TbeuzRZwy++iOxHq1a1\nZ5WWL68cYFe1eHHUcJjF+U/qc8rK6jciIPH111EwXN09ZJYsiYt9ci+X5csjwyRl/iYk9Tq9etX8\nf6asLD4LUgR2DeXdd6t/z/72t3i/e/aMz09JSabwOvkMJfVFaRA0ADkoKam5n742p59e97fozz5b\neTTCvfdWHrr35z/H/6y6JoNqKnPmRJ/65ZdH3cTGG8c3/TTp8E8+ifkpktc8YEDlSbTcI5g44IBI\noVZN9b/7btQyDBmSWZZ8w0/+CCYFX6++mlknyU5kH2uvvWovotxgg6g7yXbuufFN/5JLKnddJEaM\niO6mDTeM4+2/f+VzuWJFZqRMknnIzipVtXx5ZGeSbNeNN1ZOObtHYW0ykddZZ8XjI4+snCXJlgz7\nlWKocLa3347lrVpVziZde23ldS+4oPKFsza//W0U0EorzyeS7Y9/jHWqS/NnD+V98snIJO2xR2Qe\nkllgn3xy5e2efDIumtlZk3Hj4tv2okUR+A4Y4DUW5b7wQjz3hz/E45dfjsc9e8bcI+6ZQmkpRlpV\nJ/n8Se433FDze5CroUPj/19VxxwT53/YMPdtt81k215/PZ7fYQf3Aw9MfxyCBqAZWbQohpoWUkX8\n8OHxzalTpwgCskdZ5Lqfqhe3q66KC2P37vGNcuTIKNo86KBY3rNnZg6IxMYbZ4aSTZgQ3+arzr9x\nzz2ZbIR7FFGaVd9FsXhx/DWrOgV2kmHYeefojqlqypToMx4xIgLG6gKp5cvjQvPkkzGktUOHmoPS\n5Hhnnhm/t9228ggO98h8JMHPvvvGxSCZbbW6eVCSbqG2bVcefnfFFZH9GTEigpXlyzPTkn//+/HN\nu6QkAqMePWLd2vryFy+Oz8mpp0aNSfbw4WzLlkUfvBTnL3HXXRFUrb22f1ck7B5ZreQ19O4d66y1\nVuVam3vuycycevfdsezLLzPL+vSJ+p3u3ePutknxcbZktFcyX0nSrfXb30aRdGlpfC633bbyEOqq\nLr00sj+bbtqwI2yKinyluqBk9NUZZ0R7zCJzmd01khRmV1d/Ux2CBgCr5NVX44/OEUesWtHc2LHx\nRzz5Y/bppxGIjBwZfwhHjow+7K22ir7k666rPCtj4uij49uTe/xRTia/qs1HH8VfrOpulf7GG/Fc\n1Wmzk2nIpdpn/UzrmWe81vqDIUPim3BpabwH0sq3mL/wwsx8Eb16RXZk3rw4P9ddt/I+TzwxLiqn\nnRYBWHYf+x57RIFo8s34gQfiwrrFFnGBvvDC+Nbdrl3MndKnT9Qq1JRluv76uGi9/35kQXr0qL5W\n4uabMxfyJItUXh71PNttFxmN7Im13GPZySdHzcf8+dE1tMkmkUUYNSpef1I0OmhQ7O/cc+PzNXVq\nvM7u3aN2p7zcfc8946Ke3eWSdKe1bh3HGDYsLtTJN/dp0yJ4GDMmRhsNH55p+4svZgL9PfeMrNOx\nx8braQjl5ZmaqOyuwQ8+iGX33Zf5jK+9duXhwd98E9mfiy9eeb8TJsR5zkbQAGCVzZ696lNDJ99i\nk0K+IUPiwlfXHApVjRsXF7UlS6Kr5Kij6t4mKWCr2gXhHt/QpOpvKrbxxvFc9sRX9bVoUeVZP5cv\njwvVs89Gt0abNu5XXx3PPfBAHPff/668j2TGzSR1ngwH3H33qFvIvgguXhzdKr/+dWb9pAj3m2/i\neOPGxXvTt2/UY0gxB8fZZ0dWpHv3uFhnt2nUqKgJ+OyzCPS6dYvz2KlTZELcM7eXf/nl6OL64Q+j\n62DZssie7LtvdHt07RqBzEsvxfrJcNm6fPRRZDN69Yrg4f/+Ly7aSRsffDD2nd3Nlx0wzZwZ5+KP\nf8wsO+SQzBTx//pXvCcjRsR5a9s2goik2HnkyMxstEkh9Jgx8Z63bx8ZtGuuie2q1oL84Q8RsGX/\nf3rjjfgcjh9ffddLMrJKikxGomoXVp8+sU7V+VOGDYv3qWoQt/328T5kZw8JGgAUhKSQb9KkTEFj\nLgVaiWnT/LvMQNeuUXOQxkEHxbfAbKWlcXGoaUhmMsdEbcNLc/HDH0bxpHvsU4o/+tttFxeYpPuk\nvDyG9lXtpkouHqeeGr+TFP1tt8V+eveOmoQvv4yagmSdJDA45phYP7nQJUOCk5R9cXE8Xrgw6jza\ntKlcNDhmTLznrVrFxbF79/hGf/75ERQk7Vm+PLqbzj8/3vNu3eL1bbppJjh47rlMkPKrX0UXyaqO\nNigri8xTp07R9qpTrmc79NDoekr07Zu5n8yQIdG25AZ5AwfGxbVTp3htyXv7+eeRFUqOl9xr5o03\n4nVJlSccmzMnU+icFDV+9VV0JyRZjlatVp6ILRk5s956mWHd7pHNyL7r7XHHxXpVZ9NNan+yg7Ky\nssxxs4d2EzQAKBgbbBCp2w4dolCxPtmLFSvij91pp3mNXQ7VufjiuJBlX4iT+SSS+3ZUdfXV8XxS\nUb+qfvGLTJ3CCSdE7cDYsXEhyb4Y1KS8PC6ua68d72H2a3nzzSiWS4Z8duxYufDyooti2fjxUdiX\n3a0za1ZkK7Lnlpg6tfpixoULoyvkT3+q/WZxhx4aXRutWsVF76WXogti++3jdaxYEQHI+efH8pNO\nqvv1p5EEpHXNT/DHP2b6/pcvj4v+hAkRwCTf6pMg6IwzvFIB5eefx+PDD4/fU6ZkiizXWy9e3+LF\nsc8//zlzzNNPjy6OPn0yAdwFF0T3wezZcSE/9tgIHrKzTOPHR9A1alRm1tiknmH06JVfe9WC6/Ly\nyKJkjxBKRlbstVfsOzn3BA0ACkZy2/Pttqu+XiGt3XbLTNld3fC56iTf1pJ7OCRZhqq3wc42f36k\ngBvKddfFBWHBgigaPOusWP711+nrRZLbkNfUXz57dgRDJ51U+V4ln34aIynatIntc7kfTH389a9x\nnOxM0NKllV/nIYdErYVU/X1V6mPZssjE1HWL9qeeiuPOmJEZSTJ5cqYbrWvXTDr/rrti2RVXZLbf\nZJNYloysmDkzArnsYGWbbTKjeD77LJ4///wYjdSuXQRr66wT3R2J0tLMcN3ks33aafFZTepBknvq\nSJW7M5YtqzmIvuKKylOqJ6/p3XcjY5S0gaABQMEYMya+Ka/qbInJ7dO7dEmfrUjGryfD4JJ5DmrK\nMjSGpGslud9HfY6dFOxVHb6aVllZ1FA09qyNy5fHBay24yT3RWmIrolcJTUm112XmXL8iy/iwtu5\ns/vee2fWnT8/MjFJd457pivg+eczy6ZOrTxq54QToktq+fLIIHTvHhftuXPjAv7DH0YmpuqIpJKS\nysWt++wT07q/9ZZ/181w7LHR3ZP2859kFpJRIxdeGBmr8vL4f9m+fbz+xgwaWgkAcnDOOdL770sb\nbrhq+xk8OH5vtZVklm6bLl2kLbeUXnhBWrZM+t3vpJ/8RNphh1VrSy623lpq3166/HKpd29p++1z\n38e228bvH/6wfm1o1Ur63vfid2Nq2zbe39qOs/fe8fvQQ6U2bRq3PVV17Bjn48UXpbffjs/HuutK\n7dpJl10mjR6dWbdbN2nyZGmjjTLLRo2K9QYNyizbbjtpvfUqP545U+rXT7rpJumii6SuXeP9P/xw\n6bXX4vcmm1RuW5cu0sCB0n/+E4/feUfq21fafHOpUyfpiSekO+6Qhg9P//n//vejbVOmxOPXX4/X\nbyadeqp0yy1Sz56p3756aeJTDKC5a9Mm/iCuquQP9dZb577dCy9I114rzZolPfjgqrclF23bStts\nExeq445L/wc/26BBcWFLAqfmbNNNpfPOk4YOzc/xBw6Unn46Khj69cucj1NOqXvboqL4qc0uu8Tv\nLbeU7rqrcqB36qnSnXdKZ59d/bZ77y1dc4307bfxWf3BDyIA23ZbaezYCHyHDau7nQkzabfdKgcN\nSdDWtat02GHp91VfZBoA5MU668Qf9iOOyG27gQPjm9+YMdLJJ8eFoqltt138/ulP67f9hhtK//tf\n/bIUhcZMOv98abPN8nP8gQOlt96SXnopvsk3tL59pQULpHvvXTkzNHCgVFKSyRxVtdde0vz50j//\nGUFN0r6iImnRImnffaX118+tPbvuKk2bJn31lfTee5Gpa0pkGgDkzYQJuW8zaJBUXh4Xq/POa/g2\npXHoofEHe1UyBQ2RrUFcuN2lN9+UfvazxjlGx441P9euXc3PDRwode4sjRsXj3/wg/idZDd+/vPc\n27LrrlJZmXTDDfE710zdqiLTAKBZ6dcvagnGjIl+5XzYYw/p0Uel1q3zc3xk9O2bCcAaI9OwKtq2\nje6EV1+NzFr37rH8oIOkSy6J37nafPOoW0gC7qbONBA0AGhWWreWPv44+pOBVq0y3TyFFjRI0UUh\nVW5bly7SuedGUJErs8g2zJoVRZ2dOzdIM1MjaADQ7DT2qAE0L4MGRYFu1REMhSApVEy6JhrCrrvG\n76bumpCoaQAANHOnnZYZkVJo+vaN2obddmu4fRI0AABQTz16xHwShcgshgg3pC22kI4+WjrkkIbd\nbxoEDQAANCNm0q235ufY9AwCAIBUCBoAAEAqBA0AACAVggYAAJAKQQMAAEiFoAEAAKRC0AAAAFIh\naAAAAKkQNAAAgFQIGgAAQCoEDQAAIBWCBjSJSZMm5bsJaECcz5aF84m0cg4azGxnM7vPzD43s3Iz\nOyjFNkPN7BUzW2RmX5jZDWbWvX5NRnPEH6WWhfPZsnA+kVZ9Mg1rSnpF0ghJXtfKZrajpImS/ipp\nC0mHSdpB0vX1ODYAAMiTnG+N7e6PSHpEkszMUmwySNJH7j6+4vEnZvYXSWflemwAAJA/TVHT8Lyk\n3ma2nySZWU9Jh0t6sAmODQAAGkjOmYZcuftzZnaMpNvNrEPFMe+TNKqWzTpI0ltvvdXYzUMTKSkp\n0fTp0/PdDDQQzmfLwvlsWbKunR0aet/mXmdZQs0bm5VLOsTd76tlnS0k/UfSFZIek9RL0p8kTXX3\n42vY5mhJt9a7YQAAYKi739aQO2yKoOEmSR3c/YisZTtKelpSL3efU802a0v6saSPJS2tdwMBAFj9\ndJC0kaRH3f3rhtxxo3dPSOooaXmVZeWKkRfVFlJWvMgGjY4AAFiNPNcYO63PPA1rmll/M9umYtEm\nFY97Vzz/BzObmLXJ/ZJ+amYnm9nGFVmGqyW96O6zV/kVAACAJpFz94SZ7SrpCa08R8NEd/+5mf1d\nUh933yNrm5GSTpa0saRvJP1X0jnu/uWqNB4AADSdVappAAAAqw/uPQEAAFIhaAAAAKkUXNBgZiPN\n7CMzW2JmL5jZ9vluE+pmZudV3MAs++fNKutcWHHDssVm9h8z2zRf7UVlaW5EV9f5M7P2ZjbezL4y\ns4Vm9m8z69F0rwLZ6jqnZvb3av7PPlRlHc5pATCzc83sJTNbYGZzzOxuM9u8mvUa/f9oQQUNZnak\nYhKo8yRtK+lVSY+a2Tp5bRjSekNST0nrVvzslDxhZmcrZgE9UXHDskWKc9suD+3Eymq9EV3K8zdW\n0gGSfippF0nrSbqzcZuNWqS5ueDDqvx/trjK85zTwrCzpGslDZS0l6S2kh4zszWSFZrs/6i7F8yP\npBckXZ312CR9JumsfLeNnzrP3XmSptfy/BeSRmc97iJpiaQj8t12flY6V+WSDsrl/FU8XiZpSNY6\nP6jY1w75fk2r+08N5/Tvku6qZRvOaYH+SFqn4jzslLWsSf6PFkymwczaSipSDMeUJHm8qscl/Shf\n7UJONqtIhX5gZrdkzd2xseJbTPa5XSDpRXFuC17K87edYrK47HXekTRLnONCtltFuvttM5tgZt2z\nnisS57RQdVNkj+ZJTft/tGCCBkXk1FpS1Wml5yjeDBS2FyQdp5j+O5mT4ykzW1Nx/lyc2+Yqzfnr\nKWl5xR+qmtZBYXlY0rGS9pB0lqRdJT1kZslMveuKc1pwKs7PWEnPuHtSN9Zk/0ebYhpprAbc/dGs\nh2+Y2UuSPpF0hKS389MqADVx939lPZxpZq9L+kDSbooJ/FCYJkjaQtKO+Th4IWUavpJUpoiGsvWU\nxHTTzYy7l0h6V9KmivNn4tw2V2nO32xJ7cysSy3roIC5+0eKv8NJxT3ntMCY2ThJ+0vazSvPqNxk\n/0cLJmhw9xWSpknaM1lWkYbZU4104w00HjPrpPjj80XFH6PZqnxuuygqgTm3BS7l+ZsmqbTKOj+Q\ntKGk55ussag3M9tA0tqSkosR57SAVAQMB0va3d1nZT/XlP9HC6174kpJ/zCzaZJekjRacZfMf+Sz\nUaibmf1RcXOyTyStL+kCSSsk/bNilbGSfmtm7ytueT5GMTLm3iZvLFZSUXuyqTJ3nt3EzPpLmufu\nn6qO8+fuC8zsBklXmtl8SQslXSPpWXd/qUlfDCTVfk4rfs5TDLebXbHeZYrs4KMS57SQmNkExXDY\ngyQtMrMko1Di7ksr/t00/0fzPXSkmqEkIype8BJF9LNdvtvET6rzNqniA7pEUY17m6SNq6xzvmJY\n0GLFH6ZN891ufr47N7sqhl6VVfm5Me35k9ReMZb8q4o/SHdI6pHv17a6/tR2TiV1kPSIImBYKulD\nSX+W9D3OaeH91HAeyyQdW2W9Rv8/yg2rAABAKgVT0wAAAAobQQMAAEiFoAEAAKRC0AAAAFIhaAAA\nAKkQNAAAgFQIGgAAQCoEDQAAIBWCBgAAkApBAwAASIWgAQAApPL/J59eIFK2fFMAAAAASUVORK5C\nYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x106e5bdd8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import matplotlib.ticker as ticker\n",
    "%matplotlib inline\n",
    "\n",
    "plt.figure()\n",
    "plt.plot(all_losses)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Sampling the Network\n",
    "\n",
    "To sample we give the network a letter and ask what the next one is, feed that in as the next letter, and repeat until the EOS token.\n",
    "\n",
    "* Create tensors for input category, starting letter, and empty hidden state\n",
    "* Create a string `output_str` with the starting letter\n",
    "* Up to a maximum output length,\n",
    "    * Feed the current letter to the network\n",
    "    * Get the next letter from highest output, and next hidden state\n",
    "    * If the letter is EOS, stop here\n",
    "    * If a regular letter, add to `output_str` and continue\n",
    "* Return the final name\n",
    "\n",
    "*Note*: Rather than supplying a starting letter every time we generate, we could have trained with a \"start of string\" token and had the network choose its own starting letter."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "max_length = 20\n",
    "\n",
    "# Generate given a category and starting letter\n",
    "def generate_one(category, start_char='A', temperature=0.5):\n",
    "    category_input = make_category_input(category)\n",
    "    chars_input = make_chars_input(start_char)\n",
    "    hidden = rnn.init_hidden()\n",
    "\n",
    "    output_str = start_char\n",
    "    \n",
    "    for i in range(max_length):\n",
    "        output, hidden = rnn(category_input, chars_input[0], hidden)\n",
    "        \n",
    "        # Sample as a multinomial distribution\n",
    "        output_dist = output.data.view(-1).div(temperature).exp()\n",
    "        top_i = torch.multinomial(output_dist, 1)[0]\n",
    "        \n",
    "        # Stop at EOS, or add to output_str\n",
    "        if top_i == EOS:\n",
    "            break\n",
    "        else:    \n",
    "            char = all_letters[top_i]\n",
    "            output_str += char\n",
    "            chars_input = make_chars_input(char)\n",
    "\n",
    "    return output_str\n",
    "\n",
    "# Get multiple samples from one category and multiple starting letters\n",
    "def generate(category, start_chars='ABC'):\n",
    "    for start_char in start_chars:\n",
    "        print(generate_one(category, start_char))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false,
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Riberkov\n",
      "Urtherdez\n",
      "Shimanev\n"
     ]
    }
   ],
   "source": [
    "generate('Russian', 'RUS')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Gomen\n",
      "Ester\n",
      "Ront\n"
     ]
    }
   ],
   "source": [
    "generate('German', 'GER')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sandar\n",
      "Per\n",
      "Alvareza\n"
     ]
    }
   ],
   "source": [
    "generate('Spanish', 'SPA')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cha\n",
      "Hang\n",
      "Ini\n"
     ]
    }
   ],
   "source": [
    "generate('Chinese', 'CHI')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The final versions of the scripts [in the Practical PyTorch repo](https://github.com/spro/practical-pytorch/tree/master/conditional-char-rnn) split the above code into a few files:\n",
    "\n",
    "* `data.py` (loads files)\n",
    "* `model.py` (defines the RNN)\n",
    "* `train.py` (runs training)\n",
    "* `generate.py` (runs `generate()` with command line arguments)\n",
    "\n",
    "Run `train.py` to train and save the network.\n",
    "\n",
    "Then run `generate.py` with a language to view generated names: \n",
    "\n",
    "```\n",
    "$ python generate.py Russian\n",
    "Alaskinimhovev\n",
    "Beranivikh\n",
    "Chamon\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exercises\n",
    "\n",
    "* Adjust the `temperature` argument to see how generation is affected \n",
    "* Try with a different dataset of category -> line, for example:\n",
    "    * Fictional series -> Character name\n",
    "    * Part of speech -> Word\n",
    "    * Country -> City\n",
    "* Use a \"start of sentence\" token so that sampling can be done without choosing a start letter\n",
    "* Get better results with a bigger network"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Next**: [Translation with a Sequence to Sequence Network and Attention](https://github.com/spro/practical-pytorch/blob/master/seq2seq-translation/seq2seq-translation.ipynb)"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [conda root]",
   "language": "python",
   "name": "conda-root-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  },
  "nbpresent": {
   "slides": {
    "10393c05-7962-4245-9228-8b7db4eb79a1": {
     "id": "10393c05-7962-4245-9228-8b7db4eb79a1",
     "prev": "22628fc4-8309-4579-ba36-e5b01a841473",
     "regions": {
      "335fd672-4ee6-4b7c-a65f-3ecbf38305e1": {
       "attrs": {
        "height": 0.8,
        "width": 0.8,
        "x": 0.1,
        "y": 0.1
       },
       "content": {
        "cell": "cc294dae-dd8f-4288-8d3c-bb9fd3ad19bc",
        "part": "whole"
       },
       "id": "335fd672-4ee6-4b7c-a65f-3ecbf38305e1"
      }
     }
    },
    "22628fc4-8309-4579-ba36-e5b01a841473": {
     "id": "22628fc4-8309-4579-ba36-e5b01a841473",
     "prev": null,
     "regions": {
      "6cfa5157-02f6-48e3-8ce4-89641febbe59": {
       "attrs": {
        "height": 0.8,
        "width": 0.8,
        "x": 0.1,
        "y": 0.1
       },
       "content": {
        "cell": "9a73330c-27c1-4957-8e95-c3b42bc14a71",
        "part": "whole"
       },
       "id": "6cfa5157-02f6-48e3-8ce4-89641febbe59"
      }
     }
    },
    "2f34f0df-3ccc-4416-9d5d-cb4b075f539f": {
     "id": "2f34f0df-3ccc-4416-9d5d-cb4b075f539f",
     "prev": "3eb7f63f-04de-4f51-a240-38d5074bed6f",
     "regions": {
      "e25707b9-630e-4ece-9f66-bfcbc8342d76": {
       "attrs": {
        "height": 0.8,
        "width": 0.8,
        "x": 0.1,
        "y": 0.1
       },
       "content": {
        "cell": "df50f546-6d02-4383-beab-90378f16576b",
        "part": "whole"
       },
       "id": "e25707b9-630e-4ece-9f66-bfcbc8342d76"
      }
     }
    },
    "3eb7f63f-04de-4f51-a240-38d5074bed6f": {
     "id": "3eb7f63f-04de-4f51-a240-38d5074bed6f",
     "prev": "cc4bd43a-59ec-4127-b1d8-ebd30162207a",
     "regions": {
      "76282c28-a6ba-4a08-be6c-4f27f5b81ddf": {
       "attrs": {
        "height": 0.8,
        "width": 0.8,
        "x": 0.1,
        "y": 0.1
       },
       "content": {
        "cell": "53fb987f-4f42-4bf8-81ae-280ebdd19aee",
        "part": "whole"
       },
       "id": "76282c28-a6ba-4a08-be6c-4f27f5b81ddf"
      }
     }
    },
    "686bcaec-0623-4943-b227-f4e1c5975c4a": {
     "id": "686bcaec-0623-4943-b227-f4e1c5975c4a",
     "prev": "e4c6fc30-f833-4368-99fe-5297b99f1f14",
     "regions": {
      "659c021e-7f79-4612-aa7d-8f1c48f91f8b": {
       "attrs": {
        "height": 0.8,
        "width": 0.8,
        "x": 0.1,
        "y": 0.1
       },
       "content": {
        "cell": "4ff5f52a-2523-47f0-beba-f6c29d412e5f",
        "part": "whole"
       },
       "id": "659c021e-7f79-4612-aa7d-8f1c48f91f8b"
      }
     }
    },
    "964ac1b6-c781-47e8-89f0-1f593d473cd0": {
     "id": "964ac1b6-c781-47e8-89f0-1f593d473cd0",
     "prev": "cf8a3b4c-bbb5-4ef3-a8c6-43a1ad7eebc8",
     "regions": {
      "d22afc7e-4bbe-401e-9bd2-d12c4a103cf5": {
       "attrs": {
        "height": 0.8,
        "width": 0.8,
        "x": 0.1,
        "y": 0.1
       },
       "content": {
        "cell": "8ff6da45-57cd-46ca-b14a-3f560ce4d345",
        "part": "whole"
       },
       "id": "d22afc7e-4bbe-401e-9bd2-d12c4a103cf5"
      }
     }
    },
    "cc4bd43a-59ec-4127-b1d8-ebd30162207a": {
     "id": "cc4bd43a-59ec-4127-b1d8-ebd30162207a",
     "prev": "964ac1b6-c781-47e8-89f0-1f593d473cd0",
     "regions": {
      "1e6711af-7711-4579-ac7a-f893b0d86931": {
       "attrs": {
        "height": 0.8,
        "width": 0.8,
        "x": 0.1,
        "y": 0.1
       },
       "content": {
        "cell": "cf311809-10bf-40f7-87e1-1952342f7f35",
        "part": "whole"
       },
       "id": "1e6711af-7711-4579-ac7a-f893b0d86931"
      }
     }
    },
    "cf8a3b4c-bbb5-4ef3-a8c6-43a1ad7eebc8": {
     "id": "cf8a3b4c-bbb5-4ef3-a8c6-43a1ad7eebc8",
     "prev": "686bcaec-0623-4943-b227-f4e1c5975c4a",
     "regions": {
      "3b983e72-35fb-4d19-83b4-789a3394f61f": {
       "attrs": {
        "height": 0.8,
        "width": 0.8,
        "x": 0.1,
        "y": 0.1
       },
       "content": {
        "cell": "597a765d-634b-41a8-a0c6-be5c019da150",
        "part": "whole"
       },
       "id": "3b983e72-35fb-4d19-83b4-789a3394f61f"
      }
     }
    },
    "e4c6fc30-f833-4368-99fe-5297b99f1f14": {
     "id": "e4c6fc30-f833-4368-99fe-5297b99f1f14",
     "prev": "10393c05-7962-4245-9228-8b7db4eb79a1",
     "regions": {
      "98a6b3b6-d2db-4d8a-bb16-a4307ede4803": {
       "attrs": {
        "height": 0.8,
        "width": 0.8,
        "x": 0.1,
        "y": 0.1
       },
       "content": {
        "cell": "6a9d80df-1d38-4c41-849c-95e38da98cc7",
        "part": "whole"
       },
       "id": "98a6b3b6-d2db-4d8a-bb16-a4307ede4803"
      }
     }
    },
    "f1a487d8-4b0b-47df-988f-1161d66174b2": {
     "id": "f1a487d8-4b0b-47df-988f-1161d66174b2",
     "prev": "2f34f0df-3ccc-4416-9d5d-cb4b075f539f",
     "regions": {
      "2c817a32-203d-404b-8bf5-ba17f7d27034": {
       "attrs": {
        "height": 0.8,
        "width": 0.8,
        "x": 0.1,
        "y": 0.1
       },
       "content": {
        "cell": "81fde336-785e-461b-a751-718a5f6bff88",
        "part": "whole"
       },
       "id": "2c817a32-203d-404b-8bf5-ba17f7d27034"
      }
     }
    }
   },
   "themes": {}
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}