abstractive-text-summ.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Abstractive Text Summarization\n",
    "===\n",
    "This notebook is an in-progress implementation/experiment of the [Abstractive Text Summarization using Sequence-to-sequence RNNs and\n",
    "Beyond](https://arxiv.org/abs/1602.06023) paper.\n",
    "\n",
    "Current Features\n",
    "---\n",
    "* model architecture supports LSTM & GRU (biLSTM-to-uniLSTM or biGRU-to-uniGRU)\n",
    "* implements batch data processing \n",
    "* implements attention mechanism ([Bahdanau et al.](https://arxiv.org/abs/1409.0473) & [Luong et al.(global dot)](https://arxiv.org/abs/1508.04025))\n",
    "* implements [scheduled sampling (teacher forcing)](https://arxiv.org/abs/1506.03099)\n",
    "* implements [tied embeddings](https://arxiv.org/pdf/1608.05859.pdf)\n",
    "* initializes encoder-decoder with pretrained vectors (glove.6B.200d)\n",
    "* implements custom training callbacks (tensorboard visualization for PyTorch, save best model & log checkpoint)\n",
    "* implements attention plots\n",
    "\n",
    "\n",
    "To-Do\n",
    "---\n",
    "* Implement additional linguistic features embeddings  \n",
    "* Implement generator-pointer switch and replace unknown words by selecting source token with the highest attention score.\n",
    "* Implement large vocabulary trick \n",
    "* Implement sentence level attention \n",
    "* Implement beam search during inference\n",
    "* implement rouge evaluation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Requirements\n",
    "---\n",
    "\n",
    "1. Create conda environment \n",
    "\n",
    "`conda env create -f environment.yml`  --gpu\n",
    "\n",
    "`conda env create -f environment-cpu.yml`  --cpu\n",
    "\n",
    "2. Install dependencies (PyTorch, Fastai, TorchText, Tensorboard etc) via:\n",
    "\n",
    "`pip install -r requirements.txt`\n",
    "\n",
    "3. Download `spacy` english module\n",
    "\n",
    "`python -m spacy download en`\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Dataset\n",
    "--\n",
    "\n",
    "The dataset used is a subset of the gigaword dataset and can be found [here](https://drive.google.com/file/d/0B6N7tANPyVeBNmlSX19Ld2xDU1E/view?usp=sharing).\n",
    "\n",
    "It contains 3,803,955 parallel source & target examples for training and 189,649 examples for validation.\n",
    "\n",
    "After downloading, we create article-title pairs, save in tabular datset format (.csv) and extract a sample subset (80,000 for training & 20,000 for validation). This data preparation can be found [here](/data-preparation.ipynb).\n",
    "\n",
    "An example article-title pair looks like this:\n",
    "\n",
    "`article: the algerian cabinet chaired by president abdelaziz bouteflika on sunday adopted the #### finance bill predicated on an oil price of ## dollars a barrel and a growth rate of #.# percent , it was announced here .`\n",
    "\n",
    "`title: algeria adopts #### finance bill with oil put at ## dollars a barrel`\n",
    "\n",
    "\n",
    "Training on the complete dataset (3M) would take a really long time. So in order to train and experiment faster we use our sample subset of 80,000 in this tutorial. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "%reload_ext autoreload\n",
    "%autoreload 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "USE_GPU=True\n",
      "current_device=0\n"
     ]
    }
   ],
   "source": [
    "#declare the directory path to dataset  \n",
    "DATA_PATH = 'data/'\n",
    "SAMPLE_DATA_PATH = f'{DATA_PATH}sample_data/'\n",
    "PROCESSED_DATA_PATH = f'{DATA_PATH}processed_data/'\n",
    "\n",
    "#Enable GPU training \n",
    "import torch\n",
    "USE_GPU = torch.cuda.is_available()\n",
    "print('USE_GPU={}'.format(USE_GPU))\n",
    "if USE_GPU:\n",
    "    print('current_device={}'.format(torch.cuda.current_device()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "## 1. Process dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "In order to train, we performed common processing steps on the dataset such as:\n",
    "\n",
    "* Loading dataset\n",
    "* Preprocessing dataset (tokenizing, appending begining-of-sentence and end-of-sentence tokens, truncating, etc)\n",
    "* Building a vocabulary\n",
    "* Creating dataset iterators\n",
    "* Batching, padding, and numericalizing. \n",
    "\n",
    "To process the dataset, we use the [torchtext](https://github.com/pytorch/text) library."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "#import data, vocab from torchtext \n",
    "from torchtext import data, vocab"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "### 1.1. Load & define preprocessing pipeline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "To pre-process our data, we declare a `Field` class, and pass additional pre-processing arguments ( ex. tokenize with the `spacy` tokenizer, `lower` and append an `end-of-sentence` token to every example )."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "tokenizer = data.get_tokenizer('spacy')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "hidden": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "TEXT = data.Field(tokenize=tokenizer, lower=True, eos_token='_eos_')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "Next, we load our training & validation tabular dataset using `data.TabularDataset.splits` which applies the defined preprocessing pipeline and returns their respective `Dataset` objects"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "hidden": true,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 55.3 s, sys: 387 ms, total: 55.7 s\n",
      "Wall time: 55.5 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "trn_data_fields = [(\"source\", TEXT),\n",
    "                   (\"target\", TEXT)]\n",
    "\n",
    "trn, vld = data.TabularDataset.splits(path=f'{SAMPLE_DATA_PATH}',\n",
    "                                     train='train_ds.csv', validation='valid_ds.csv',\n",
    "                                     format='csv', skip_header=True, fields=trn_data_fields)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "hidden": true,
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['jason', 'blake', 'of', 'the', 'islanders', 'will', 'miss', 'the', 'rest', 'of', 'the', 'season', 'so', 'he', 'can', 'be', 'with', 'his', 'wife', ',', 'who', 'has', 'thyroid', 'cancer', 'and', 'is', 'to', 'give', 'birth', 'april', '#', '.'] ['blake', 'missing', 'rest', 'of', 'season']\n"
     ]
    }
   ],
   "source": [
    "# a sample of the preprocessed data\n",
    "print(trn[0].source, trn[0].target)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "### 1.2. Build vocabulary"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "Building a vocabulary simply means mapping each unique token in the corpus to an integer value, and storing as a dictionary ex `{'the': 2, 'brown':3, 'fox':4}`. \n",
    "\n",
    "In addition to building a vocabulary, we also use torchtext to load an embedding matrix for each token using the `glove.6B.200d` pretrained vector.\n",
    "\n",
    "We pass to `TEXT.build_vocab` our training dataset object `trn` and also the name of the pretrained vector we would like to use `glove.6B.200d`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 1.94 s, sys: 288 ms, total: 2.22 s\n",
      "Wall time: 2.22 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "pre_trained_vector_type = 'glove.6B.200d' \n",
    "TEXT.build_vocab(trn, vectors=pre_trained_vector_type )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('#', 152957),\n",
       " ('the', 130459),\n",
       " ('.', 105054),\n",
       " (',', 85497),\n",
       " ('to', 83508),\n",
       " ('in', 78169),\n",
       " ('of', 77424),\n",
       " ('a', 71025),\n",
       " ('on', 43536),\n",
       " ('and', 42555)]"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#10 most frequent words in the vocab\n",
    "TEXT.vocab.freqs.most_common(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "### 1.3. Create dataset iterator, batch, pad, and numericalize. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "Next, create a training & validation iterator object, numericalize (turn text to tensors), batch examples of similar lengths together, randomly shuffle the data and pad tensors using `data.BucketIterator.splits`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "batch_size = 64"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "train_iter, val_iter = data.BucketIterator.splits(\n",
    "                        (trn, vld), batch_sizes=(batch_size, int(batch_size*1.6)),\n",
    "                        device=(0 if USE_GPU else -1), \n",
    "                        sort_key=lambda x: len(x.source),\n",
    "                        shuffle=True, sort_within_batch=False, repeat=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "Next, we stick each article-title batch pair tensor into a tuple (article, title). To do this we create a custom helper class `BatchTuple`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "class BatchTuple():\n",
    "    def __init__(self, dataset, x_var, y_var):\n",
    "        self.dataset, self.x_var, self.y_var = dataset, x_var, y_var\n",
    "        \n",
    "    def __iter__(self):\n",
    "        for batch in self.dataset:\n",
    "            x = getattr(batch, self.x_var) \n",
    "            y = getattr(batch, self.y_var)                 \n",
    "            yield (x, y)\n",
    "            \n",
    "    def __len__(self):\n",
    "        return len(self.dataset)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "#returns tuple of article-title pair tensors\n",
    "train_iter_tuple = BatchTuple(train_iter, \"source\", \"target\")\n",
    "val_iter_tuple = BatchTuple(val_iter, \"source\", \"target\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(tensor([[   166,      8,   4750,  ...,    338,      4,      4],\n",
       "         [    91,     22,    107,  ...,    413,   1428,    188],\n",
       "         [   539,     24,     13,  ...,   4739,     16,      9],\n",
       "         ...,\n",
       "         [     1,      1,      1,  ...,      1,      1,      1],\n",
       "         [     1,      1,      1,  ...,      1,      1,      1],\n",
       "         [     1,      1,      1,  ...,      1,      1,      1]], device='cuda:0'),\n",
       " tensor([[   291,     18,    929,  ...,   1758,     40,   3545],\n",
       "         [  3429,  11609,     12,  ...,    512,      5,     12],\n",
       "         [    35,     13,    928,  ...,    338,    179,   2319],\n",
       "         ...,\n",
       "         [     1,      1,      1,  ...,      1,      2,      1],\n",
       "         [     1,      1,      1,  ...,      1,      1,      1],\n",
       "         [     1,      1,      1,  ...,      1,      1,      1]], device='cuda:0'))"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#an example of a batched and padded article-title tensor pair\n",
    "next(iter(train_iter_tuple))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "### 1.4. Create ModelData "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "In order to use our batched dataset with the FastAI library, we create a `ModelData` object. \n",
    "\n",
    "`ModelData` simply sticks the training, validation (and test) dataset into a single object"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "#import text module from fastai \n",
    "from fastai.text import *"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "model_data = ModelData(SAMPLE_DATA_PATH, trn_dl=train_iter_tuple, val_dl=val_iter_tuple)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "### 1.5. Processed dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "Finally, we are done with processing the dataset: pre-processing, numericalizing, batching and padding.\n",
    "\n",
    "Lets take a look at the final processed data "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1250, 197, 52221)"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#number of batches in training & validation set and number of tokens in vocabulary\n",
    "len(model_data.trn_dl), len(model_data.val_dl), len(TEXT.vocab)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(torch.Size([54, 64]), torch.Size([25, 64]))"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#shape of one batch in training set (sequence_length x batch_size)\n",
    "t, z = next(model_data.trn_dl.__iter__())\n",
    "t.size(), z.size()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "source:\n",
      "candidates from sunni and shiite islamist parties have secured # # of the # # seats in bahrain 's legislative assembly , following the first parliamentary elections in the small gulf state since # # # # , according to official results released early friday . _eos_ <pad> <pad> <pad> <pad> <pad> <pad> <pad> \n",
      "\n",
      "corresponding tensor:\n",
      "[1234   21 2677   12 1300 2753  855   58 5266    3    3    9    4    3    3 2975    8 3347   13 1718 1470\n",
      "    6  321    4   51 1006  247    8    4  603  694   73  270    3    3    3    3    6  174    7   84  587\n",
      "  423  214   39    5    2    1    1    1    1    1    1    1] \n",
      "\n",
      "target:\n",
      "islamists win almost half bahraini parliament seats _eos_ <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> \n",
      "\n",
      "corresponding tensor:\n",
      "[6764  217  989  431 8465  322 2975    2    1    1    1    1    1    1    1    1    1    1    1    1    1\n",
      "    1    1    1    1] \n",
      "\n"
     ]
    }
   ],
   "source": [
    "#lets look at an example pair\n",
    "sample_source = t.transpose(1,0)[0].data.cpu().numpy()\n",
    "sample_target = z.transpose(1,0)[0].data.cpu().numpy()\n",
    "\n",
    "print(\"source:\\n%s \\n\\ncorresponding tensor:\\n%s \\n\" %(' '.join([TEXT.vocab.itos[o] for o in sample_source]), sample_source))\n",
    "print(\"target:\\n%s \\n\\ncorresponding tensor:\\n%s \\n\" %(' '.join([TEXT.vocab.itos[o] for o in sample_target]), sample_target))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Model "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.1. Define model architecture\n",
    "The sequence model, consists of:\n",
    "* single layer encoder-decoder RNNs (biGRU-to-uniGRU)\n",
    "* feed-forward attention network (bahdanau)\n",
    "* option for luong global dot attention\n",
    "* option for teacher forcing\n",
    "* option for tied embeddings\n",
    "* option for multi-layer model\n",
    "* option for regularization (dropout)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch.nn.functional as F\n",
    "import torch.nn as nn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Seq2SeqRNN(nn.Module):\n",
    "    \n",
    "    def __init__(self, rnn_type, input_size, embz_size, hidden_size, batch_size,output_size,max_tgt_len,\n",
    "                 attention_type, tied_weight_type, pre_trained_vector, pre_trained_vector_type, padding_idx,\n",
    "                 num_layers=1, encoder_drop=(0.0,0.0), decoder_drop=(0.0,0.0), \n",
    "                 bidirectional=True, bias=False, teacher_forcing=True):\n",
    "        \n",
    "        super().__init__()\n",
    "        \n",
    "        rnn_type, attention_type, tied_weight_type = rnn_type.upper(), attention_type.title(), tied_weight_type.lower()\n",
    "        \n",
    "        if rnn_type in ['LSTM', 'GRU']: self.rnn_type = rnn_type\n",
    "        else: raise ValueError(\"\"\"An invalid option for '--rnn_type' was supplied,\n",
    "                                    options are ['LSTM', 'GRU']\"\"\")\n",
    "            \n",
    "        if attention_type in ['Luong', 'Bahdanau']: self.attention_type = attention_type\n",
    "        else: raise ValueError(\"\"\"An invalid option for '--attention_type' was supplied,\n",
    "                                    options are ['Luong', 'Bahdanau']\"\"\")\n",
    "            \n",
    "        if tied_weight_type in ['three_way', 'two_way']: self.tied_weight_type = tied_weight_type\n",
    "        else: raise ValueError(\"\"\"An invalid option for '--tied_weight_type' was supplied,\n",
    "                                    options are ['three_way', 'two_way']\"\"\")\n",
    "    \n",
    "                    \n",
    "        #initialize model parameters            \n",
    "        self.output_size, self.embz_size, self.hidden_size = output_size, embz_size, hidden_size//2\n",
    "        self.num_layers, self.input_size, self.max_tgt_len, self.pre_trained_vector = num_layers, input_size, max_tgt_len, pre_trained_vector\n",
    "        self.bidirectional,self.teacher_forcing, self.pre_trained_vector_type = bidirectional, teacher_forcing, pre_trained_vector_type\n",
    "        self.encoder_drop, self.decoder_drop, self.padding_idx = encoder_drop, decoder_drop, padding_idx\n",
    "        \n",
    "        \n",
    "        if self.teacher_forcing: self.force_prob = 1.0\n",
    "        \n",
    "        #set bidirectional\n",
    "        if self.bidirectional: self.num_directions = 2\n",
    "        else: self.num_directions = 1\n",
    "            \n",
    "        \n",
    "        #encoder\n",
    "        self.encoder_dropout = nn.Dropout(self.encoder_drop[0])\n",
    "        self.encoder_embedding_layer = nn.Embedding(self.input_size, self.embz_size, padding_idx=self.padding_idx)\n",
    "        if self.pre_trained_vector: self.encoder_embedding_layer.weight.data.copy_(self.pre_trained_vector.weight.data)\n",
    "            \n",
    "        self.encoder_rnn = getattr(nn, self.rnn_type)(\n",
    "                           input_size=self.embz_size,\n",
    "                           hidden_size=self.hidden_size,\n",
    "                           num_layers=self.num_layers,\n",
    "                           dropout=self.encoder_drop[1], \n",
    "                           bidirectional=self.bidirectional)\n",
    "        self.encoder_vector_layer = nn.Linear(self.hidden_size*self.num_directions,self.embz_size, bias=bias)\n",
    "        \n",
    "       #decoder\n",
    "        self.decoder_dropout = nn.Dropout(self.decoder_drop[0])\n",
    "        self.decoder_embedding_layer = nn.Embedding(self.input_size, self.embz_size, padding_idx=self.padding_idx)\n",
    "        self.decoder_rnn = getattr(nn, self.rnn_type)(\n",
    "                           input_size=self.embz_size,\n",
    "                           hidden_size=self.hidden_size*self.num_directions,\n",
    "                           num_layers=self.num_layers,\n",
    "                           dropout=self.decoder_drop[1]) \n",
    "        self.decoder_output_layer = nn.Linear(self.hidden_size*self.num_directions, self.embz_size, bias=bias)\n",
    "        self.output_layer = nn.Linear(self.embz_size, self.output_size, bias=bias)\n",
    "        \n",
    "        #set tied weights: three way tied weights vs two way tied weights\n",
    "        if self.tied_weight_type == 'three_way':\n",
    "            self.decoder_embedding_layer.weight  = self.encoder_embedding_layer.weight\n",
    "            self.output_layer.weight = self.decoder_embedding_layer.weight  \n",
    "        else:\n",
    "            if self.pre_trained_vector: self.decoder_embedding_layer.weight.data.copy_(self.pre_trained_vector.weight.data)\n",
    "            self.output_layer.weight = self.decoder_embedding_layer.weight  \n",
    "            \n",
    "        #set attention\n",
    "        self.encoder_output_layer = nn.Linear(self.hidden_size*self.num_directions, self.embz_size, bias=bias)\n",
    "        self.att_vector_layer = nn.Linear(self.embz_size+self.embz_size, self.embz_size,bias=bias)\n",
    "        if self.attention_type == 'Bahdanau':\n",
    "            self.decoder_hidden_layer = nn.Linear(self.hidden_size*self.num_directions, self.embz_size, bias=bias)\n",
    "            self.att_score = nn.Linear(self.embz_size,1,bias=bias)\n",
    "\n",
    "            \n",
    "    \n",
    "    def init_hidden(self, batch_size):\n",
    "        if self.rnn_type == 'LSTM':\n",
    "            return (V(torch.zeros(self.num_layers*self.num_directions, batch_size, self.hidden_size)),\n",
    "                    V(torch.zeros(self.num_layers*self.num_directions, batch_size, self.hidden_size)))\n",
    "        else:\n",
    "            return V(torch.zeros(self.num_layers*self.num_directions, batch_size, self.hidden_size))\n",
    "   \n",
    "\n",
    "    def _cat_directions(self, hidden):\n",
    "        def _cat(h):\n",
    "            return torch.cat([h[0:h.size(0):2], h[1:h.size(0):2]], 2)\n",
    "            \n",
    "        if isinstance(hidden, tuple):\n",
    "            # LSTM hidden contains a tuple (hidden state, cell state)\n",
    "            hidden = tuple([_cat(h) for h in hidden])\n",
    "        else:\n",
    "            # GRU hidden\n",
    "            hidden = _cat(hidden)\n",
    "        return hidden    \n",
    "    \n",
    "    \n",
    "    def bahdanau_attention(self, encoder_output, decoder_hidden, decoder_input):\n",
    "        encoder_output = self.encoder_output_layer(encoder_output) \n",
    "        encoder_output = encoder_output.transpose(0,1)\n",
    "        decoder_hidden = decoder_hidden.transpose(0,1)\n",
    "        att_score = F.tanh(encoder_output + decoder_hidden)\n",
    "        att_score = self.att_score(att_score)\n",
    "        att_weight = F.softmax(att_score, dim=1)\n",
    "        context_vector = torch.bmm(att_weight.transpose(-1, 1), encoder_output).squeeze(1)\n",
    "        att_vector = torch.cat((context_vector, decoder_input), dim=1)\n",
    "        att_vector = self.att_vector_layer(att_vector)\n",
    "        return att_weight.squeeze(-1), att_vector\n",
    "    \n",
    "    \n",
    "    def luong_attention(self, encoder_output, decoder_output):\n",
    "        encoder_output = self.encoder_output_layer(encoder_output) \n",
    "        encoder_output = encoder_output.transpose(0,1)\n",
    "        decoder_output = decoder_output.transpose(0,1)\n",
    "        att_score = torch.bmm(encoder_output, decoder_output.transpose(-1,1))\n",
    "        att_weight = F.softmax(att_score, dim=1)\n",
    "        context_vector = torch.bmm(att_weight.transpose(-1, 1), encoder_output).squeeze(1)\n",
    "        att_vector = torch.cat((context_vector, decoder_output.squeeze(1)), dim=1)\n",
    "        att_vector = self.att_vector_layer(att_vector)\n",
    "        att_vector = F.tanh(att_vector)\n",
    "        return att_weight.squeeze(-1), att_vector\n",
    "        \n",
    "    def decoder_forward(self, batch_size, encoder_output, decoder_hidden, y=None):\n",
    "        decoder_input = V(torch.zeros(batch_size).long())  \n",
    "        output_seq_stack, att_stack = [], []\n",
    "        \n",
    "        for i in range(self.max_tgt_len):\n",
    "            decoder_input = self.decoder_dropout(self.decoder_embedding_layer(decoder_input))\n",
    "            if self.attention_type == 'Bahdanau':\n",
    "                if isinstance(decoder_hidden, tuple):\n",
    "                    prev_hidden = self.decoder_hidden_layer(decoder_hidden[0][-1]).unsqueeze(0)\n",
    "                else:\n",
    "                    prev_hidden = self.decoder_hidden_layer(decoder_hidden[-1]).unsqueeze(0) \n",
    "                att, decoder_input = self.bahdanau_attention(encoder_output, prev_hidden, decoder_input)\n",
    "                decoder_output, decoder_hidden = self.decoder_rnn(decoder_input.unsqueeze(0), decoder_hidden)\n",
    "                decoder_output = self.decoder_output_layer(decoder_output.squeeze(0)) \n",
    "            else:\n",
    "                decoder_output, decoder_hidden = self.decoder_rnn(decoder_input.unsqueeze(0), decoder_hidden)\n",
    "                decoder_output = self.decoder_output_layer(decoder_output) \n",
    "                att, decoder_output = self.luong_attention(encoder_output, decoder_output)\n",
    "            att_stack.append(att)\n",
    "            output = self.output_layer(decoder_output)\n",
    "            output_seq_stack.append(output)\n",
    "            decoder_input = V(output.data.max(1)[1])\n",
    "            if (decoder_input==1).all(): break \n",
    "            if self.teacher_forcing:    \n",
    "                samp_prob = round(random.random(),1)\n",
    "                if (y is not None) and (samp_prob < self.force_prob):\n",
    "                    if i >= len(y): break\n",
    "                    decoder_input = y[i] \n",
    "                \n",
    "        return torch.stack(output_seq_stack), torch.stack(att_stack)\n",
    "        \n",
    "                \n",
    "    def forward(self, seq, y=None):\n",
    "        batch_size = seq[0].size(0)\n",
    "        encoder_hidden = self.init_hidden(batch_size)\n",
    "        encoder_input = self.encoder_dropout(self.encoder_embedding_layer(seq))\n",
    "        encoder_output, encoder_hidden = self.encoder_rnn(encoder_input, encoder_hidden) \n",
    "        if self.bidirectional:\n",
    "            encoder_hidden = self._cat_directions(encoder_hidden)\n",
    "        output = self.decoder_forward(batch_size, encoder_output, encoder_hidden, y=y)\n",
    "        if isinstance(encoder_hidden, tuple):\n",
    "            encoder_vector = self.encoder_vector_layer(encoder_hidden[0][-1])\n",
    "        else:\n",
    "            encoder_vector = self.encoder_vector_layer(encoder_hidden[-1])\n",
    "        output = output + (encoder_vector,)  \n",
    "        return output\n",
    "    \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "24"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#set maximum target summary size \n",
    "its = [next(model_data.trn_dl.__iter__())[1] for i in range(10)]\n",
    "max_tgt_len = int(np.percentile([its[o].size()[0] for o in range(len(its))], 99))\n",
    "max_tgt_len"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "def save_pickle(path, filename, file):\n",
    "    \"\"\"Function to save file as pickle\"\"\"\n",
    "    with open(f'{path}/{filename}', 'wb') as f:\n",
    "        pickle.dump(file, f)\n",
    "\n",
    "        \n",
    "def norm_pre_trained_embeddings(vecs, itos, em_sz, padding_idx):\n",
    "    \"\"\"Function to load and normalize pretrained vectors\"\"\"\n",
    "    emb = nn.Embedding(len(itos), em_sz, padding_idx=padding_idx)\n",
    "    wgts = emb.weight.data\n",
    "    for i,w in enumerate(itos):\n",
    "        try: \n",
    "            wgts[i] = torch.from_numpy(vecs[w]-vec_mean)\n",
    "            wgts[i] = torch.from_numpy(vecs[w]/vec_std)\n",
    "        except: pass \n",
    "    emb.weight.requires_grad = False    \n",
    "    return emb\n",
    "\n",
    "\n",
    "def embedding_param(path, data_field, pre_trained_vector_type, embz_size=128, save_vocab=False, itos='itos', stoi='stoi'):\n",
    "    \"\"\"Returns embedding parameters\"\"\"\n",
    "    pre_trained=None\n",
    "    padding_idx = data_field.vocab.stoi['<pad>']\n",
    "    index_to_string, string_to_index = data_field.vocab.itos, data_field.vocab.stoi\n",
    "    if save_vocab:\n",
    "        vocab_path = os.path.join(path, \"vocab\")\n",
    "        os.makedirs(vocab_path, exist_ok=True)\n",
    "        save_pickle(vocab_path, f'{itos}.pk', index_to_string) \n",
    "        save_pickle(vocab_path, f'{stoi}.pk', string_to_index) \n",
    "    if pre_trained_vector_type:\n",
    "        vec_mean, vec_std = data_field.vocab.vectors.numpy().mean(), data_field.vocab.vectors.numpy().std()\n",
    "        print('pre_trained_vector_mean = %s, pre_trained_vector_std = %s'%(vec_mean, vec_std))\n",
    "        vector_weight_matrix = data_field.vocab.vectors\n",
    "        embz_size = vector_weight_matrix.size(1)\n",
    "        pre_trained = norm_pre_trained_embeddings(vector_weight_matrix, index_to_string, embz_size, padding_idx)\n",
    "        print('Normalizing.... \\npre_trained_vector_mean = %s, pre_trained_vector_std = %s' %(pre_trained.weight.data.numpy().mean(), pre_trained.weight.data.numpy().std()))\n",
    "    return pre_trained, embz_size, padding_idx\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "rev = 1\n"
     ]
    }
   ],
   "source": [
    "rev=0\n",
    "rev += 1\n",
    "print(\"rev = %s\" %rev)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "pre_trained_vector_mean = 0.0019917963, pre_trained_vector_std = 0.43600857\n",
      "Normalizing.... \n",
      "pre_trained_vector_mean = -0.00042225976, pre_trained_vector_std = 1.0003649\n"
     ]
    }
   ],
   "source": [
    "pre_trained_vector,  embz_size, padding_idx = embedding_param(SAMPLE_DATA_PATH, TEXT, pre_trained_vector_type, save_vocab=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 131,
   "metadata": {},
   "outputs": [],
   "source": [
    "input_size = len(TEXT.vocab)\n",
    "hidden_size = 400\n",
    "output_size =  len(TEXT.vocab)\n",
    "rnn_type = 'gru'\n",
    "tied_weight_type ='three_way'\n",
    "max_tgt_len = max_tgt_len"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.2 Training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "from fastai.nlp import *\n",
    "from fastai.model import Stepper"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Seq2SeqStepper(Stepper):\n",
    "    def step(self, xs, y, epoch):\n",
    "        output = self.m(*xs, y)\n",
    "        xtra = []\n",
    "        if isinstance(output,tuple): output,*xtra = output\n",
    "        self.opt.zero_grad()\n",
    "        loss = raw_loss = self.crit(output, y)\n",
    "        if self.reg_fn: loss = self.reg_fn(output, xtra, raw_loss)\n",
    "        loss.backward()\n",
    "        if self.clip:   # Gradient clipping\n",
    "            nn.utils.clip_grad_norm(trainable_params_(self.m), self.clip)\n",
    "        self.opt.step()\n",
    "        return raw_loss.data\n",
    "    \n",
    "def seq2seq_loss(input, target):\n",
    "    sl,bs = target.size()\n",
    "    sl_in,bs_in,nc = input.size()\n",
    "    if sl>sl_in: input = F.pad(input, (0,0,0,0,0,sl-sl_in))\n",
    "    input = input[:sl]\n",
    "    return F.cross_entropy(input.view(-1,nc), target.view(-1))#, ignore_index=1)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "#custom callbacks\n",
    "from tensorboardX import SummaryWriter\n",
    "from fastai.sgdr import Callback, DecayScheduler\n",
    "from fastai.learner import Learner\n",
    "\n",
    "class TensorboardLogger(Callback):\n",
    "    def __init__(self, path, log_name, metrics_names=[]):\n",
    "        super().__init__()\n",
    "        self.metrics_names = [\"validation_loss\"]\n",
    "        self.metrics_names += metrics_names\n",
    "        log_path = os.path.join(path, \"logs\")\n",
    "        self.log_dir = os.path.join(log_path, log_name)\n",
    "        if os.path.exists(self.log_dir): shutil.rmtree(self.log_dir)\n",
    "        os.makedirs(self.log_dir)\n",
    "        \n",
    "    def on_train_begin(self):\n",
    "        self.iteration = 0\n",
    "        self.epoch = 0\n",
    "        self.writer = SummaryWriter(log_dir=self.log_dir)\n",
    "    def on_batch_begin(self): pass\n",
    "    def on_phase_begin(self): pass\n",
    "    def on_epoch_end(self, metrics):\n",
    "        self.epoch += 1\n",
    "        for val, name in zip(metrics, self.metrics_names):\n",
    "            self.writer.add_scalar(name, val, self.iteration) \n",
    "                        \n",
    "    def on_phase_end(self): pass\n",
    "    def on_batch_end(self, loss):\n",
    "        self.iteration += 1\n",
    "        self.writer.add_scalar(\"training_loss\", loss, self.iteration)\n",
    "    def on_train_end(self):\n",
    "        self.writer.close()\n",
    "        \n",
    "        \n",
    "class BestModelCheckPoint(Callback):\n",
    "    def __init__(self, learner, path, model_name, lr):\n",
    "        super().__init__()\n",
    "        self.learner = learner\n",
    "        self.model_name = model_name\n",
    "        self.learning_rate = lr\n",
    "        self.model_log = {}\n",
    "        self.model_path = self.learner.models_path\n",
    "        os.makedirs(self.model_path, exist_ok=True)\n",
    "\n",
    "    def on_train_begin(self): \n",
    "        self.first_epoch = True\n",
    "        self.epoch = 0\n",
    "        self.best_loss = 0.\n",
    "\n",
    "    def on_batch_begin(self): pass\n",
    "    def on_phase_begin(self): pass\n",
    "    def on_epoch_end(self, metrics): \n",
    "        self.epoch += 1\n",
    "        self.val_loss = metrics[0]\n",
    "        if self.first_epoch:\n",
    "            self.best_loss = self.val_loss\n",
    "            self.first_epoch = False\n",
    "        elif self.val_loss < self.best_loss:\n",
    "            self.best_loss = self.val_loss\n",
    "            self.learner.save(self.model_name)\n",
    "            self.model_log['training_loss'] = [str(self.train_losses)]\n",
    "            self.model_log['validation_loss'] = [str(self.val_loss)]\n",
    "            self.model_log['epoch_num'] = [str(self.epoch)]\n",
    "            self.model_log['learning_rate'] = [str(self.learning_rate)]\n",
    "            self.model_log['model_info'] = [w for s in [str(self.learner.model)] for w in s.split('\\n')]\n",
    "            self.model_log['model_info'].append(\"(attention_type): %s\" %self.learner.model.attention_type)\n",
    "            self.model_log['model_info'].append(\"(weight_tie): %s\" %self.learner.model.tied_weight_type)\n",
    "            self.model_log['model_info'].append(\"(pre_trained_vector_type): %s\" %self.learner.model.pre_trained_vector_type)\n",
    "            self.model_log['model_info'].append(\"(teacher_forcing): %s\" %self.learner.model.teacher_forcing)\n",
    "            if self.learner.model.teacher_forcing: self.model_log['model_info'].append(\"(teacher_forcing_prob): %s\" %self.learner.model.force_prob)\n",
    "            with open(f'{self.model_path}/{self.model_name}_model_log.json', 'w') as d: json.dump(self.model_log, d)\n",
    "        else: pass        \n",
    "    def on_phase_end(self): pass\n",
    "    def on_batch_end(self, loss):\n",
    "        self.train_losses = loss\n",
    "    def on_train_end(self): \n",
    "            self.learner.save(f'{self.model_name}_train_end')\n",
    "            self.model_log['training_loss'] = [str(self.train_losses)]\n",
    "            self.model_log['validation_loss'] = [str(self.val_loss)]\n",
    "            self.model_log['epoch_num'] = [str(self.epoch)]\n",
    "            self.model_log['learning_rate'] = [str(self.learning_rate)]\n",
    "            self.model_log['model_info'] = [w for s in [str(self.learner.model)] for w in s.split('\\n')]\n",
    "            self.model_log['model_info'].append(\"(attention_type): %s\" %self.learner.model.attention_type)\n",
    "            self.model_log['model_info'].append(\"(weight_tie): %s\" %self.learner.model.tied_weight_type)\n",
    "            self.model_log['model_info'].append(\"(pre_trained_vector_type): %s\" %self.learner.model.pre_trained_vector_type)\n",
    "            self.model_log['model_info'].append(\"(teacher_forcing): %s\" %self.learner.model.teacher_forcing)\n",
    "            if self.learner.model.teacher_forcing: self.model_log['model_info'].append(\"(teacher_forcing_prob): %s\" %self.learner.model.force_prob)\n",
    "            with open(f'{self.model_path}/{self.model_name}_train_end_model_log.json', 'w') as d: json.dump(self.model_log, d)\n",
    "\n",
    "class TeacherForcingSched(Callback):\n",
    "    def __init__(self, learner, scheduler):\n",
    "        super().__init__()\n",
    "        self.learner = learner\n",
    "        self.scheduler = scheduler\n",
    "        \n",
    "    def on_train_begin(self): \n",
    "        self.learner.model.force_prob = round(self.scheduler.next_val(),1)\n",
    "        \n",
    "    def on_batch_begin(self): pass\n",
    "    def on_phase_begin(self): pass\n",
    "    def on_epoch_end(self, metrics): \n",
    "        self.learner.model.force_prob = round(self.scheduler.next_val(),1)\n",
    "        \n",
    "    def on_phase_end(self): pass\n",
    "    def on_batch_end(self, loss):pass\n",
    "    def on_train_end(self): pass\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 135,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "====================================================================================================\n",
      "Model log:\n",
      "Seq2SeqRNN(\n",
      "  (pre_trained_vector): Embedding(52221, 200, padding_idx=1)\n",
      "  (encoder_dropout): Dropout(p=0.0)\n",
      "  (encoder_embedding_layer): Embedding(52221, 200, padding_idx=1)\n",
      "  (encoder_rnn): GRU(200, 200, bidirectional=True)\n",
      "  (encoder_vector_layer): Linear(in_features=400, out_features=200, bias=False)\n",
      "  (decoder_dropout): Dropout(p=0.0)\n",
      "  (decoder_embedding_layer): Embedding(52221, 200, padding_idx=1)\n",
      "  (decoder_rnn): GRU(200, 400)\n",
      "  (decoder_output_layer): Linear(in_features=400, out_features=200, bias=False)\n",
      "  (output_layer): Linear(in_features=200, out_features=52221, bias=False)\n",
      "  (encoder_output_layer): Linear(in_features=400, out_features=200, bias=False)\n",
      "  (att_vector_layer): Linear(in_features=400, out_features=200, bias=False)\n",
      ") \n",
      "\n",
      "- attention_type = Luong \n",
      "\n",
      "- weight_tie = three_way \n",
      "\n",
      "- teacher_forcing = True \n",
      " \n",
      "- pre_trained_embedding = glove.6B.200d \n",
      "\n",
      "====================================================================================================\n",
      "\n"
     ]
    }
   ],
   "source": [
    "attention_type='luong'\n",
    "model_luong = Seq2SeqRNN(rnn_type, input_size, embz_size, hidden_size, batch_size, output_size, max_tgt_len,\n",
    "               attention_type, tied_weight_type, pre_trained_vector, pre_trained_vector_type, padding_idx)\n",
    "\n",
    "print('='*100)\n",
    "print('Model log:')\n",
    "print(model_luong, '\\n')\n",
    "print('- attention_type = {} \\n'.format(model_luong.attention_type))\n",
    "print('- weight_tie = {} \\n'.format(model_luong.tied_weight_type))\n",
    "print('- teacher_forcing = {} \\n '.format(model_luong.teacher_forcing)) \n",
    "print('- pre_trained_embedding = {} \\n'.format(model_luong.pre_trained_vector_type)) \n",
    "print('='*100 + '\\n')\n",
    "\n",
    "if USE_GPU:\n",
    "    model_luong.cuda()\n",
    "opt_fn = partial(optim.Adam, betas=(0.8, 0.99))\n",
    "learn_luong = RNN_Learner(model_data, SingleModel(model_luong), opt_fn=opt_fn)\n",
    "learn.crit = seq2seq_loss\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "044a1ebec7c244b4b8d1129631ecbcc0",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Epoch', max=1), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " 58%|█████▊    | 1457/2500 [05:42<04:05,  4.25it/s, loss=27]  "
     ]
    }
   ],
   "source": [
    "learn_luong.lr_find()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYwAAAEOCAYAAACaQSCZAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3Xl8VNX5+PHPM9l3EhJCWCKLgiCyaMQF9wWXr7u1Slu/Lm2prVqttr/a7Wur/bb229pFbau4VNuqtda1SlW0IloVCAgCgoCsIUASCNkzWeb5/XFvcIyT5AYyK8/79ZrX3Dl3e3LhNc+ce849R1QVY4wxpi++aAdgjDEmPljCMMYY44klDGOMMZ5YwjDGGOOJJQxjjDGeWMIwxhjjiSUMY4wxnljCMMYY44klDGOMMZ5YwjDGGONJcrQDGEiFhYU6atSoaIdhjDFxY8mSJTWqWuRl27AlDBEZCfwZGAoEgDmq+jsReRIY7242CNijqlND7L8JaAA6gQ5VLevrnKNGjaK8vHyA/gJjjEl8IrLZ67bhrGF0ALeo6lIRyQGWiMg8Vb2sawMRuQuo6+UYp6hqTRhjNMYY41HYEoaqbge2u8sNIrIaGA58CCAiAnweODVcMRhjjBk4EWn0FpFRwDRgYVDxCcBOVV3Xw24KvCoiS0RkdngjNMYY05ewN3qLSDbwNHCTqtYHrZoFPNHLrjNUtVJEhgDzRGSNqi4IcfzZwGyA0tLSAYzcGGNMsLDWMEQkBSdZPKaqzwSVJwMXA0/2tK+qVrrvVcCzwPQetpujqmWqWlZU5Kmh3xhjzD4IW8Jw2ygeAlar6q+7rT4dWKOqFT3sm+U2lCMiWcBMYGW4YjXGGNO3cNYwZgBXAKeKyDL3dY677nK63Y4SkWEiMtf9WAy8LSLLgUXAS6r6chhjNcaYuLSjrpVdjf6InCucvaTeBqSHdVeFKKsEznGXNwBTwhWbMcYkimN+/joAm+78r7Cfy4YGMcYY44klDGOMMZ5YwjDGmDi1dmdDRM9nCcMYY+LUzX9fBsAxYwoicj5LGMYYE6dy01MAqGvpiMj5LGEYY0ycKspJA6C2qS0i57OEYYwxcaogKxWALx0TmWGRLGEYY0yc2t3URkqScP2ph0TkfAk1454xxhxInl9WGdHzWQ3DGGOMJ5YwjDHGeGK3pIwxJk7lpCXzubIRETuf1TCMMSZO+TsDpCZH7mvcEoYxxsQhVaW9M0BqkiUMY4wxvegIKKpYwjDGGNO7to4AQGLckhKRkSLyhoisFpFVInKjW/5jEdkWYha+7vufJSIfich6Ebk1XHEaY0w8au90EkZKBGsY4ewl1QHcoqpL3fm5l4jIPHfdb1T1Vz3tKCJJwO+BM4AKYLGIvKCqH4YxXmOMiRsJVcNQ1e2qutRdbgBWA8M97j4dWK+qG1S1DfgbcEF4IjXGmPjjT6SEEUxERgHTgIVu0fUi8oGIPCwi+SF2GQ5sDfpcgfdkY4wxCa/rllRCNXqLSDbwNHCTqtYDfwTGAlOB7cBdoXYLUaY9HH+2iJSLSHl1dfUARW2MMbGtrTPBahgikoKTLB5T1WcAVHWnqnaqagB4AOf2U3cVwMigzyOAkKNsqeocVS1T1bKioqKB/QOMMSZG7W3DSIQahogI8BCwWlV/HVReErTZRcDKELsvBg4RkdEikgpcDrwQrliNMSbebKxpAiAlgjWMcPaSmgFcAawQkWVu2feBWSIyFecW0ybgawAiMgx4UFXPUdUOEbkeeAVIAh5W1VVhjNUYY+LKjX9zvlb97Z0RO2fYEoaqvk3otoi5PWxfCZwT9HluT9saY4xx+CTU12yYzhWxMxljjBkwXdOynjZhSMTOaQnDGGPiUG1TO4OzUpEI1jBsPgxjjIlDL63YHvFzWg3DGGOMJ5YwjDHGeGIJwxhj4kwg4Ax8ccOpB0f0vJYwjDEmznQNC5KRmhTR81rCMMaYONPS5jysF8lhQcAShjHGxJ3fv7EegGVb90T0vJYwjDEmzmze3QzA1tqWiJ7XEoYxxsSZ48YOBuC28yZG9LyWMIwxJs64naQYW5Qd0fNawjDGmDjj73AavdMiOLQ5WMIwxpi4M3+NM7uo9ZIyxhjTq0WbdgPg80Vu4EGwhGGMMcajsI1WKyIjgT8DQ4EAMEdVfycivwTOA9qAj4GrVfUznYlFZBPQAHQCHapaFq5YjTEmnowpymJCSW7EzxvOGkYHcIuqTgCOAa4TkYnAPGCSqk4G1gLf6+UYp6jqVEsWxhjjeH7ZNjZUN9HY2hHxc4ctYajqdlVd6i43AKuB4ar6qqp2/aXvASPCFYMxxiSa55dVArCqsi7i545IG4aIjAKmAQu7rboG+FcPuynwqogsEZHZvRx7toiUi0h5dXX1QIRrjDExa0R+BgA1jW0RP3fYE4aIZANPAzepan1Q+Q9wbls91sOuM1T1COBsnNtZJ4baSFXnqGqZqpYVFRUNcPTGGBNbBmelAXBZ2ciInzusCUNEUnCSxWOq+kxQ+ZXAucAXVVVD7auqle57FfAsMD2csRpjTDxYsc3pI/Sziw+P+LnDljDEmZn8IWC1qv46qPws4LvA+ara3MO+WSKS07UMzARWhitWY4yJB/6OTl5bXQVAUoSfwYDw1jBmAFcAp4rIMvd1DnAvkAPMc8vuAxCRYSIy1923GHhbRJYDi4CXVPXlMMZqjDExr7UtENXzh+05DFV9GwiVAueGKOu6BXWOu7wBmBKu2IwxJh51jSEVLfaktzHGxAl/h1PD+NG5kR3WvIslDGOMiROt7U4Nozg3LSrnt4RhjDFxoKG1nfve3ACATyLf4A1hbMMwxhgzcG762zJeX+P0kKpu8EclBqthGGNMHFizo2Hv8ueOjM6ISpYwjDEmDowpytq7nJUWnZtDljCMMSYOFOemRzsESxjGGBMP2juj+9AeWMIwxpi40DWs+TUzRkctBksYxhgTR/7nvOg8tAeWMIwxJuZ1Deod3PAdDZYwjDEmxnUEnIRx0dThUY3DEoYxxsS4rjGk0lKi+5Xd59nduSl87vI4ETnfnRjJGGNMBNS3tAOQnpIU1Ti8pKsFQLqIDAdeB64GHglnUMYYYz5x9+vrAFi0cXdU4/CSMMSdGe9i4B5VvQjos5leREaKyBsislpEVonIjW55gYjME5F17nt+D/tf6W6zzp3S1RhjDkhdT3YfNiwvqnF4ShgicizwReAlt8zLc+kdwC2qOgE4BrhORCYCtwKvq+ohODWWW0OcsAC4DTgaZy7v23pKLMYYk+hG5GcAcNlRI6Mah5eEcRPwPeBZVV0lImOAN/raSVW3q+pSd7kBWA0MBy4AHnU3exS4MMTuZwLzVHW3qtYC84CzPMRqjDEJp6vROz3Kjd591hRU9U3gTQC38btGVb/Zn5OIyChgGrAQKFbV7e6xt4vIkBC7DAe2Bn2ucMuMMeaAsammiUZ/x96Jk9KSY7zRW0QeF5FcEckCPgQ+EpHveD2BiGQDTwM3qWq9191ClGkPx58tIuUiUl5dXe01LGOMiXmn3jWfc+95m9b2AClJQpIvOhMndfFSv5noftFfCMwFSoErvBzc7X77NPCYqj7jFu8UkRJ3fQlQFWLXCiD4Zt0IoDLUOVR1jqqWqWpZUVGRl7CMMSbmLd60G/d5PdbubCA9yrUL8JYwUtwv/guB51W1nR5+7QcTEQEeAlar6q+DVr0AdPV6uhJ4PsTurwAzRSTfbeye6ZYZY8wB4a11NXuX3/m4hrQoP4MB3hLG/cAmIAtYICIHAV5uLc3AqYmcKiLL3Nc5wJ3AGSKyDjjD/YyIlInIgwCquhu4A1jsvm53y4wx5oAwYlDG3uXW9gBpydEfmMNLo/fdwN1BRZtF5BQP+71N6LYIgNNCbF8OfCXo88PAw32dxxhjEpHfnf+iMDuNmkZ/1HtIgbdG7zwR+XVXw7KI3IVT2zDGGBMmfrdn1FdOcOa/iHYPKfB2S+phoAH4vPuqB/4UzqCMMeZA19WV9uTxTmeeWKhheHlie6yqXhL0+ScisixcARljjHHaLXwC44tzKMpJi/rAg+AtYbSIyPFumwQiMgNoCW9YxhhzYGtt7yQtOQkR4bbzJpKZGh8J4+vAoyKSh9OIvRu4KpxBGWPMga61o3PvbahzJw+LcjQOL72klgFTRCTX/ez1aW1jjDH7qLU9EBO3oYL1mDBE5OYeygHo9jCeMcaYAbB1dzM1jX4WbdxNagw8exGstxpGTsSiMMYYA8DFf3yH6gZ/tMMIqceEoao/iWQgxhhjiNlkAd6ewzDGGBMhk0d8MqveTacfEsVIPssShjHGxKjTJxRHO4RPsYRhjDExpNHfsXc5Nz0lipF8Vp/dakUkDbgEGBW8vareHr6wjDHmwNTs7+TiI4Yza3oppYMzox3Op3h5cO95oA5YAsRua4wxxsS5JZt3s6O+ldz0FI4aVRDtcD7DS8IYoapnhT0SY4w5gHUGlEv++C4AKUnRnYq1J17aMN4RkcPDHokxxhzAfjZ39d7litrYHK7PSw3jeOAqEdmIc0tKAFXVyb3tJCIPA+cCVao6yS17EhjvbjII2KOqU0PsuwlnSPVOoENVy7z9OcYYE5/+tWL73uVbZo6LYiQ985Iwzt7HYz8C3Av8uatAVS/rWnYnYqrrZf9TVLWml/XGGJMwppYOonLFDkry0jl4SGwOtOFl8MHNIjIFOMEtektVl3vYb4GIjAq1TpwBqT4PnOo9VGOMSVyqzvvjXz0muoH0wssUrTcCjwFD3NdfReSG/TzvCcBOVV3Xw3oFXhWRJSIyu4/4ZndNH1tdXb2fYRljTHQ0tXUydeQgRhfG7gzYXm5JfRk4WlWbAETkF8C7wD37cd5ZwBO9rJ+hqpUiMgSYJyJrVHVBqA1VdQ4wB6CsrEz3IyZjjImaZn8HWWmxNZx5d156SQlO43OXTrdsn4hIMnAx8GRP26hqpfteBTwLTN/X8xljTKzrDCi7m9rITPXyGz56vET3J2ChiDzrfr4QeGg/znk6sEZVK0KtFJEswKeqDe7yTMCeKjfGJKzvP7OCDTVNTBqe1/fGUdRnDcOdKOlqnKlZa4GrVfW3fe0nIk/g3LoaLyIVIvJld9XldLsdJSLDRGSu+7EYeFtElgOLgJdU9WWvf5AxxsSTzoDyZPlWANbubIhyNL3rbca9XFWtF5ECYJP76lpXoKq7ezuwqs7qofyqEGWVwDnu8gZgiofYjTEm7i1Y+0lnnZK89ChG0rfebkk9jvPg3RKcXktdxP08JoxxGWNMwlNVfvv6J51Ff3PZZ55jjim9zbh3rvs+OnLhGGPMgaO60c/yrXsAeP66GQzKTI1yRL3z8hzG617KjDHG9E9j6ydzX5QWxNZQ5qH01oaRDmQChSKSzyddaXOBYRGIzRhjEtrq7U4j97mTS8jPiu3aBfTehvE14Cac5LCETxJGPfD7MMdljDEJ77rHlwLwpWMOinIk3vTWhvE74HcicoOq7s9T3cYYY0IYWZDB1t0tTBkxKNqheOJl8MF7RGQSMBFIDyr/c897GWOM6YsqzJxYTEZqbA8J0sXLnN63ASfjJIy5OMOdv03QsOXGGGP65z/ra6iobaGupT3aoXjmZSypzwGnATtU9Wqch+rSwhqVMcYkuI01TQCcMn5IlCPxzkvCaFHVANAhIrlAFfbQnjHG7JfWdmdM1zsumBTlSLzzMvhguYgMAh7A6S3ViDPGkzHGmH3U5HcSRqwPaR7MS6P3N9zF+0TkZSBXVT8Ib1jGGJPYGv3tpKf4SE7ycqMnNvT24N4Rva1T1aXhCckYYxJfo7+D7LSUaIfRL73VMO5y39OBMmA5zsN7k4GFwPHhDc0YYxJXdYOfnPTYnjCpu94e3DsFQET+BsxW1RXu50nAtyMTnjHGJJ6tu5t5bXVVtMPoNy83zw7tShYAqroS6HMMXhF5WESqRGRlUNmPRWSbiCxzX+f0sO9ZIvKRiKwXkVu9/CHGGBMvtu1piXYI+8RLwlgtIg+KyMkicpKIPACs9rDfI8BZIcp/o6pT3dfc7itFJAlnrKqzcR4WnCUiEz2czxhj4kKT3xml9r4vHRnlSPrHS8K4GlgF3IgzGOGHblmvVHUBzrSu/TUdWK+qG1S1DfgbcME+HMcYY2LSH+d/DMC44uwoR9I/XrrVtgK/cV8D4XoR+W+gHLhFVWu7rR8ObA36XAEcPUDnNsaYqKtq8AMwbFBGlCPpnx5rGCLyd/d9hYh80P21j+f7IzAWpw1kO5/0xPrUqUOUaYiyrjhni0i5iJRXV1f3tJkxxsSM8UNzmFCSS3pK/Dy0B73XMG50388dqJOp6s6uZbct5MUQm1UAI4M+jwAqeznmHGAOQFlZWY+JxRhjYkWTv4OsOBmhNlhv3Wq3u++bB+pkIlLSdVzgImBliM0WA4eIyGhgG3A58IWBisEYY6Ktqa2TQRnx9dAe9P6kdwOhbwUJoKqa29uBReQJnGHRC0WkArgNOFlEprrH3YQzqx8iMgx4UFXPUdUOEbkeeAVIAh5W1VX9/cOMMSZWNfs7GD4ove8NY0xvNYyc/Tmwqs4KUfxQD9tWAucEfZ6LM/eGMcYkHOeWVHw95Q3eRqsFQESG8OkZ97aEJSJjjElwTW2dZKXFX8Lo8zkMETlfRNYBG4E3cW4l/SvMcRljTEK6+k+L4mqWvWBeHty7AzgGWKuqo3Fm3/tPWKMyxpgE9cZHTvf/Efnx9QwGeEsY7aq6C/CJiE9V38DDWFLGGGM+KzXZx38dXsIVxx4U7VD6zctNtD0ikg0sAB4TkSqgI7xhGWNM4rnkj+/Q1hFgXHEOacnx9xyGlxrGBUAz8C3gZeBj4LxwBmWMMYlGVVmy2RkJqSQOu9SCtxrGbOApVa0AHg1zPMYYk5CqG53xo75wdCkXTRse5Wj2jZcaRi7wioi8JSLXiUhxuIMyxphE8+JyZ5CLMYVZpMTRPN7B+oxaVX+iqocB1wHDgDdF5LWwR2aMMQmkpb0TgIuPGBHlSPZdf9JcFbAD2AUMCU84xhiTmO5705kDIz8z/saQ6uLlwb2vi8h84HWgEPiqqk4Od2DGGJNIMlKSKMhKRSTUDA7xwUuj90HATaq6LNzBGGNMogqocuZh8d0E7GXGvVsjEYgxxiSqbXtaqGlsi3YY+y0+m+qNMSaObKppAuDo0YOjHMn+sYRhjDFhVtvs1C4mlPQ6jVDMs4RhjDFhVtvkJIx47iEFYUwYIvKwiFSJyMqgsl+KyBoR+UBEnhWRQT3su0lEVojIMhEpD1eMxhgTCQvW1QAwODstypHsn3DWMB4BzupWNg+Y5HbLXQt8r5f9T1HVqapaFqb4jDEmIlZtqyPJJyT54rdLLYQxYajqAmB3t7JXVbVrpNv3gPh95NEYY/pQVd/KDU+8T2VdK186ujTa4ey3aLZhXEPPM/cp8KqILBGR2RGMyRhjBsxvXlvHP5dXAnDs2MIoR7P/ojKprIj8AGdOjcd62GSGqla684jPE5E1bo0l1LFm44yoS2lp/GdwY0z8e3XVDj6oqOOJRVsAWP+/Z5McpwMOBot4whCRK4FzgdNUVUNto6qV7nuViDwLTMeZwCnUtnOAOQBlZWUhj2eMMeGmqvz53c389rW11DZ/Mmf3/VccmRDJAiKcMETkLOC7wEmq2tzDNlmAT1Ub3OWZwO0RDNMYY/rt7tfX85vX1gIwfXQB3zh5LIMyU5k6MmRn0LgUtoQhIk8AJwOFIlIB3IbTKyoN5zYTwHuqeq2IDAMeVNVzgGLgWXd9MvC4qr4crjiNMWZ/vfbhzr3J4tVvnci44pwoRxQeYUsYqjorRPFDPWxbCZzjLm8ApoQrLmOMGSiN/g7+96UPeWLRVgC+d/ahCZssIEqN3sYYE++217Vw7M//DUBJXjr3zJpG2aiCKEcVXpYwjDGmn1SV//ePDwC46rhR3HbexLie58KrxGi6N8aYCHp9dRVvrauhJC+dW88+9IBIFmA1DGNMHPjhcyto8ncyaXgeRx6Uz8SSXKob/QwflOFpf39HJ7ub2nhy8VbaOwPkZaSQmZpMarKPzNQkOgPK0Nx0SvIyKM5LIy05ae++tU1t7Gpqoygnje11LbyxpppfvLwGgBdvOJ70lKSeTptwLGEYY2Lax9WN/PW9LWSkJPHs+9s+tW5MURZjCrMpykljQkkO/3V4yacG+Fu6pZafz13Nks21BPrxlFZhdiq5GSn42wNs29PymfVJPuH2Cw6L+8EE+8sShjEmZu2oa+UHz64A4PGvHs2wQRks2VzLoo27efb9bZQWZLJ4027qWpwH5W57YRWThuUxeUQe5Ztq+WhnA2nJPr520lhG5meSnuLjvCnDaG3vZGNNEylJPnY1tpGTnkxDaweVdS3sqGtle10r9a3t+NsDnD5hCIcNy6O2uY2mtk7OOmwoQ3LTKDzAkgWA9PCwdVwqKyvT8nIbDd2YRPG1v5TzyqqdnDy+iEeunh5yG1VFFT7a2cDLK3ewcOMuVlXW09DawfTRBdw7axpDctMjHHn8EJElXkcFtxqGMSYm7Wr08+81VZw4rudkASAiiDiz2XXNaKeqtLYHyEg9cNoXIsF6SRljYs6OulbO/t1bdASUa08c0+/9RcSSRRhYwsD5z5lIt+aMiXfffmo5e5rbefTq6Rx3cPwPC54oDvhbUoGAcuZvF5CSJEwfXcBRowqYOnIQE0pyD6jucsbEiqfKt/L2+hq+PXMcJ44rinY4JsgBnzA6Asr3zj6UhRt3s3DDLuau2AFASpJw6NBcpozM49RDh3DK+CEHzMM5xkTL5l1N/Oj5lRw7ZjDXnjQ22uGYbqyXVDfb61pYvnUPyyvqWL51Dysq6mjwdzChJJfrTzmYsyYNjft5eY2JVdc8sphFG3cz7+YTKcnz9lCe2T/WS2o/lORlUJKXwVmTSgBo7wzwwrJKfj9/Pdc9vpQxRVl86eiDOH1CMaWDM6McrTGJo8nfwfyPqvjaSWMtWcQoa/TuQ0qSj0uOHMG8b53EvV+YRnpyEre/+CEn/eoNvvJoOW+vq6G9MxDtMI2Je8sr9hBQZ/IhE5vCWsMQkYdxpmOtUtVJblkB8CQwCtgEfF5Va0PseyXwQ/fjT1X10XDG2pckn3Du5GGcO3kYW3Y189SSrTy2cAuvrd5Jdloyx44dzHFjBzOtNJ/Jw/Pw2W0rEyfaOwN8WFlPRW0LZ08aGrX/u+9v2QPAESPzo3J+07dw35J6BLgX+HNQ2a3A66p6p4jc6n7+bvBOblK5DSgDFFgiIi+ESizRUDo4k1tmjue6Uw7mjTVVLFhXw4K11cz7cCcAWalJHDQ4i3HF2RxSnMPRowuYOCyXzFS7A2hiz69e/Yj739wAwDFjCnjiq8dEpYNH5Z4WCrJSyctMifi5jTdh/QZT1QUiMqpb8QU4U7cCPArMp1vCAM4E5qnqbgARmQecBTwRplD3SXpKEmcfXsLZh5egquys9/PuhhqWb63j4+pG3t2wi+eWVQIgAhNLciktyGRccQ4jCzIpLchkRH4GQ3LSEmaSeBNfAgHlufe3UZidRkqS8N6G3fx7TRWnTSiOeCx7mtvJt2QR06Lxk7dYVbcDqOp2ERkSYpvhwNagzxVuWcwSEYbmpXPRtBFcNG3E3vLapjbKN9eyYlsd72+pZVVlPS+v2kFw5zSfQFFOGmOLspk6chBji7IpHZxJdloyBVmpDM5KtYRiwqJ8cy076/387vKpnHnYUE765Rvc+8Z6Tj008t3Idze1kZ+ZGtFzmv6J1Xskof6nhuz/KyKzgdkApaWl4Yxpn+RnpXLGxGLOmPjJLzZ/RyeVe1rZsruZbbUt7KhrYXtdKyu21TFnwQY6uo3DLAL5makUZqdSlOOMklmUnUZ+Vioj8jNI9jnJJDM1iSSfkJuRwqFDc+zBQ9OnJxdvJS3Zx2kTiklPSeL6Uw/hR8+t5N2Pd0X8Ceva5jZGFljPw1gWjYSxU0RK3NpFCVAVYpsKPrltBTAC59bVZ6jqHGAOOM9hDGyo4ZGWnMTowixGF2Z9Zl17Z4Ctu5upqG2hyd/BrqY2qhv81DT6974v3VJLdYOf1vbee2flpCeTlpzEmKIsJ9FkpVKYnUZhTholeekMH5TBkJx0cjOS7aHEA1AgoLyyagfnTxlGdprzVXDpkSP47by13PnyGp6/bkZE/1/UNrcxeURexM5n+i8aCeMF4ErgTvf9+RDbvAL8TES6ukvMBL4XmfCiKyXJx5iibMYUZfe6XddonFt2N+8ta27roDOgVDX42VjTRHWDn+a2DtZXNbK6sp6aRj/1rR2fOVayT1CcBFOQlcqI/EzSkn2kJvvITk0mMy2J7LRkMlKTyM9MJT8zlbQUH4MyUijMTiM3I4X0FB+CkJpst87ixc6GVhr9HUwZOWhvWXpKElccexC/fW0dH1TUfWpdOKkqtc3t5GfZLalYFu5utU/g1BQKRaQCp+fTncDfReTLwBbgUnfbMuBaVf2Kqu4WkTuAxe6hbu9qADeOrtE4xw/N6dd+/o5OdjW2UVHbwo76VqrqW9nV1IZPoL6lg11NfrbubqG9M0BbR4Cmtg6a/J00tXXgZVCAYXnpDM/PIDc9hcHZqQzJSae0IJNBmSmMKcqmtCDTkkqM2LLL+bFR2u020DXHj+aP8z/m7+VbI5Ywmts6aesIWBtGjAt3L6lZPaw6LcS25cBXgj4/DDwcptAOWGnJSQwblMEwj3Mhd1FV/B0Bp5bS0kFrRyd7mtvY1dhGXUs7re2ddASULbuaqaxrobKulVWV9VQ3+ukMapPxCRRmO7fEfD4hxecjOUkYW5TNhJJcRhdmMXZIFkXZaXabLMy21jpTj3ZPGLnpKZw/ZRjPLN3Gd84cz6AIfInXNrcBWC+pGBerjd4mxogI6SlJjMjPhH48V9XeGaCitoU9zW2s3dngNPLXt7Kj3k8goLR1Bmhp7+SZpRU0tXXu3S/JJ4wvzmHisFwCqmSnJTN+aA6jBmcxqjCLktx0ezhyP22obiTZJwzP/+yPhy+fMJqnllRw6X3v8uI3jyctObwdKPY0O1OsWg0jtlnCMGGVkuRzG/ezmFbac6bpDCjNuE6VAAARl0lEQVQ76ltZX9XIxupGPtrZyKaaJt5aV02yz8cedz7lLqnJPkYNzmRMYTbD8zMoyUtn/NAcjhpVYL3DPFq7s4GxRdmkhOiyfejQXC6cOoznllXyjb8u5YH/Lgtrgt7d5NYwrA0jplnCMDEhyScMH5TB8EEZnBRiDoSAm1A21TSxcVcTm3c1s6G6kXVVDby5tpqWdieZJPuE8UNzmFiSS1ZaMocUZ3NEaT7jinNslOFu1lU1Mml4z72Sfnv5NCpqW3h9TRWPvruJq2eMDlssdksqPljCMHHB55O9bS/dnw9QVfY0t7Nkcy1Lt9TyQUUdr6za8akeYdlpyUwZmccRpfkcNaqAY8cODvnL+kDR0RlgW20L504u6XW7X106hZN/NZ/739zA544cQU56eL7Qu/6tcjMsYcQySxgm7okI+VmpnD6xmNPdByRVlYDC1t3NLN3iJJL3t+zhD/M/pjOwnpz0ZE47dAhnH17CSeOKDrjbWNvrWukIKCPze39QblRhFo995Wi++OBCHnhrIzefMS4s8TT5nYTR9TyIiU32r2MSkoiQJM4X3qjCLC4+whmupcnfwbsf7+KVVTuYt3onzy2rJCs1iaNGF3D+lGGcNWnoATFI5NZap0utlyerZxxcyLTSQdz9+jqOGzuYY8YMHvB4mvwd+AQyDrDEHW8O3Dq5OSBlpSVz+sRifnnpFBb/4HT+8uXpXDBtOBuqm7j578s58o7X+OYT7/Pm2moCgbgYOGCfbKxpApyE6sUdF0wC4NtPLae57bMPf+6vRn8HWak24kCsS/yfUsb0ICXJxwmHFHHCIUUEAsriTbt5fnklc1ds54XllZQWZHL59JFceuRIinLSoh3ugNpQ3UR6io+S3HRP208anse9X5jG9Y+/zw+fXcmvL5s6oPE0+TvIsttRMc9qGMbgNKofPWYwP7vocBZ+/zTunjWNkrx0/u/ljzjuzte5/vGlvPNxDerlcfc4sHlXE6MGZ/Wrq+y5k4fxhaNLefGD7exxezUNlCZ/J1lpdjsq1lnCMKabtOQkzp8yjCe/diyv3XwiVxwzirfW1fCFBxZy2q/f5MG3NlDbNLBfmJG2q6mNwuz+15q+dPRBtHUG+MXLawY0eTb6O6zBOw5YwjCmFwcPyeF/zpvIwu+fxl2XTiE/M5WfvrSa4+78N3f+aw01jf5oh7hP6lra92lmu4nDcvnGyWN5YtFWnly8te8dPLJbUvHBEoYxHqSnJHHJkSN4+uvH8a8bT+CMicXcv+Bjjv/Fv/nVKx/R6B/4huBwUVV2Nbbt80Ny3545nuPGDuanL62mck/LgMTUaAkjLljCMKafJpTkcvesabx280nMnDiUe99Yz8m/fIPnl22LizaOHfWt1LW0M664fyMdd/H5hDsvnkxnQPn+sysG5G+2W1LxwRKGMftobFE2d8+axnPXzWBEfiY3/m0ZF/7hHZZt3RPt0Hq1cls9AIcN2/fJikoHZ3LzGeOY/1E1L36wfb9jcm5JWaN3rLOEYcx+mjpyEE9//TjuuHASVfWtXPyH//DzuavD8rzCQFi5rQ6fwMSS3P06zuXTRzKuOJsbnnifHz23cr9qGk4vKathxDpLGMYMgCSfcMUxB/HKt07k82UjuX/BBs6952121rdGO7TPqKhtoTg3nYzU/ftFn5Oewj++fhxDc9P5y3ubueXvy/nLe5v7nTha2ztp6wyQG6ZxqszAiXjCEJHxIrIs6FUvIjd12+ZkEakL2uZ/Ih2nMfsiNz2FOy+ZzGNfOZqdda3MmvMe1Q2x1ZOqqqGVIQP0IGJuegrzv3My00oH8cz72/jRcyt5a11Nv46xy+2iPNiGNo95EU8YqvqRqk5V1anAkUAz8GyITd/q2k5Vb49slMbsnxkHF/LoNdOprGvhiw++t3e+h1hQ3eCnKMfbE95epKckcc+saVx3yliG5KTx3w8vYtStL3HqXfP503827h1YsCe73K7Jg/fhuRATWdG+JXUa8LGqbo5yHMYMuLJRBdwz6wg21TTztb+Ux8zYVNUNfobkDuyX84j8TL5z5qH849rjOGW8M5/JhuomfvLPDznyp/P2JoVQquqddQNV6zHhE+2EcTnwRA/rjhWR5SLyLxE5LJJBGTNQzphYzE8vmsTiTbVMu2Mev39jPRW1zVHrftveGWBXUxtFYfo1Xzo4kz9dPZ3Xbj6Jf99yEqlJPlrbAxz509cYdetLXPHQQhZv2v2ped53NjjtPMUex7Uy0RO1bgkikgqcD3wvxOqlwEGq2igi5wDPAYf0cJzZwGyA0tLSMEVrzL679MgRLFjrdD/95Ssf8ctXPgKgMDuNjFQf3z3rUM6dPCwisXQ9mT7QNYzuDh6SDcDa/z2bHz63gr++twWAt9bV7G3juOWMcdxw2iHsrPcjAoXZ1oYR6yRav3RE5ALgOlWd6WHbTUCZqvbamlZWVqbl5eUDFKExA+vj6kb+tWI7L36wnYKsVHY3tbGnuZ09LW1cccxBFOemc8IhRYwfum8P1Hnx/pZaLvrDOzz432V7J5uKFH9HJ6sq67nyoUU0uO0aE0ty8Xd0UtfSQfkPT49oPMYhIktUtczLttHs+DyLHm5HichQYKeqqohMx7l1tiuSwRkz0MYWZXP9qYdw/amfVJa317Vw09+W8cBbG92S1XzuyBHccOrBrKqs55ml22jrDOBv72RkQSZtHQG+PXM8pYMz6Qwo9S3t5GWkfGrU2c6AElDln8srGTYog6KcNMYUZiEibHOH8hienxHJPx1wBnU8ojSfD348k8q6Vm54fClLtzgPOZ4+YUjE4zH9F5UahohkAluBMapa55ZdC6Cq94nI9cDXgQ6gBbhZVd/p67hWwzDx6qUPtrOuqoGlW/awYG21p32SfUJHQCnKSSM1yYfPBx2dSmNrx95f8F1GDc7k8BGD+OfySgA+vP3MmJhZ8P0ttfx7TRVXzxhNgXWrjYr+1DCidksqHCxhmETwn/U1zPtwJyPyM7jsqJE0+TtJ8gm5Gcl8WFnPt59aTmZqMrXNbVTUtpCZmkRzW+fe/UsLMhlXnE1OegprdjSwenv9p47/X4eX8PsvHhHpP8vEKEsYxhxAVJUd9a0MyUnHJ3xmmtPOgPLTlz4kIyWJ/3fWoVGK0sSqeGnDMMYMABGhJK/nNokkn3DbedYz3ey/aD+HYYwxJk5YwjDGGOOJJQxjjDGeWMIwxhjjiSUMY4wxnljCMMYY44klDGOMMZ5YwjDGGONJQj3pLSLVQPBkTHlAXbfNupcVAv2bU3LfhIolXPv3tW1v63ta5+Vahiqz6+ttnV1f79va9d2//btve5CqFnnaU1UT9gXM6asMKI9WLOHav69te1vf0zov19Kur11fu77xf317eyX6Lal/eiyLhP09b3/272vb3tb3tM7rtbTra9d3f/e36xve/ff5XAl1S2pfiEi5ehx4y/SfXd/wsusbXnZ9Py3RaxhezIl2AAnOrm942fUNL7u+QQ74GoYxxhhvrIZhjDHGE0sYxhhjPLGEYYwxxhNLGD0QkZNF5C0RuU9ETo52PIlIRLJEZImInBvtWBKNiExw/+/+Q0S+Hu14EpGIXCgiD4jI8yIyM9rxREJCJgwReVhEqkRkZbfys0TkIxFZLyK39nEYBRqBdKAiXLHGowG6vgDfBf4enijj10BcX1VdrarXAp8HrFtoNwN0jZ9T1a8CVwGXhTHcmJGQvaRE5EScL/s/q+oktywJWAucgZMAFgOzgCTg590OcQ1Qo6oBESkGfq2qX4xU/LFugK7vZJxhF9JxrvWLkYk+9g3E9VXVKhE5H7gVuFdVH49U/PFgoK6xu99dwGOqujRC4UdNcrQDCAdVXSAio7oVTwfWq+oGABH5G3CBqv4c6O2WSC2QFo4449VAXF8ROQXIAiYCLSIyV1UDYQ08TgzU/19VfQF4QUReAixhBBmg/8MC3An860BIFpCgCaMHw4GtQZ8rgKN72lhELgbOBAYB94Y3tITQr+urqj8AEJGrcGtzYY0u/vX3/+/JwMU4P3bmhjWyxNGvawzcAJwO5InIwap6XziDiwUHUsKQEGU93o9T1WeAZ8IXTsLp1/Xdu4HqIwMfSkLq7//f+cD8cAWToPp7je8G7g5fOLEnIRu9e1ABjAz6PAKojFIsiciub3jZ9Q0/u8Z9OJASxmLgEBEZLSKpwOXAC1GOKZHY9Q0vu77hZ9e4DwmZMETkCeBdYLyIVIjIl1W1A7geeAVYDfxdVVdFM854Zdc3vOz6hp9d432TkN1qjTHGDLyErGEYY4wZeJYwjDHGeGIJwxhjjCeWMIwxxnhiCcMYY4wnljCMMcZ4YgnDRI2INEbgHOd7HGp9IM95sogctw/7TRORB93lq0QkJsYwE5FR3YcBD7FNkYi8HKmYTHRYwjBxzx2WOiRVfUFV7wzDOXsbh+1koN8JA/g+cM8+BRRlqloNbBeRGdGOxYSPJQwTE0TkOyKyWEQ+EJGfBJU/587Kt0pEZgeVN4rI7SKyEDhWRDaJyE9EZKmIrBCRQ93t9v5SF5FHRORuEXlHRDaIyOfccp+I/ME9x4siMrdrXbcY54vIz0TkTeBGETlPRBaKyPsi8pqIFLtDZl8LfEtElonICe6v76fdv29xqC9VEckBJqvq8hDrDhKR191r87qIlLrlY0XkPfeYt4eqsYkzq+FLIrJcRFaKyGVu+VHudVguIotEJMetSbzlXsOloWpJIpIkIr8M+rf6WtDq5wCbNyaRqaq97BWVF9Dovs8E5uCMFuoDXgROdNcVuO8ZwEpgsPtZgc8HHWsTcIO7/A3gQXf5KpwJhAAeAZ5yzzERZ+4DgM/hDAHuA4bizIHyuRDxzgf+EPQ5n09GS/gKcJe7/GPg20HbPQ4c7y6XAqtDHPsU4Omgz8Fx/xO40l2+BnjOXX4RmOUuX9t1Pbsd9xLggaDPeUAqsAE4yi3LxRm5OhNId8sOAcrd5VHASnd5NvBDdzkNKAdGu5+HAyui/f/KXuF7HUjDm5vYNdN9ve9+zsb5wloAfFNELnLLR7rlu4BO4Olux+kajn4JzlwQoTynztwbH4ozmyLA8cBTbvkOEXmjl1ifDFoeATwpIiU4X8Ibe9jndGCiM98OALkikqOqDUHblADVPex/bNDf8xfg/4LKL3SXHwd+FWLfFcCvROQXwIuq+paIHA5sV9XFAKpaD05tBLhXRKbiXN9xIY43E5gcVAPLw/k32QhUAcN6+BtMArCEYWKBAD9X1fs/VehMAnQ6cKyqNovIfJwpXQFaVbWz23H87nsnPf/f9gctS7d3L5qClu/Bmb73BTfWH/ewjw/nb2jp5bgtfPK39cXzAHCqulZEjgTOAX4uIq/i3DoKdYxvATuBKW7MrSG2EZya3Csh1qXj/B0mQVkbhokFrwDXiEg2gIgMF5EhOL9ea91kcShwTJjO/zZwiduWUYzTaO1FHrDNXb4yqLwByAn6/CrOKKgAuL/gu1sNHNzDed7BGWobnDaCt93l93BuORG0/lNEZBjQrKp/xamBHAGsAYaJyFHuNjluI34eTs0jAFyBM5d1d68AXxeRFHffcW7NBJwaSa+9qUx8s4Rhok5VX8W5pfKuiKwA/oHzhfsykCwiHwB34HxBhsPTOJPnrATuBxYCdR72+zHwlIi8BdQElf8TuKir0Rv4JlDmNhJ/iNPe8CmqugZnqs+c7uvc/a92r8MVwI1u+U3AzSKyCOeWVqiYDwcWicgy4AfAT1W1DbgMuEdElgPzcGoHfwCuFJH3cL78m0Ic70HgQ2Cp29X2fj6pzZ0CvBRiH5MgbHhzYwARyVbVRhEZDCwCZqjqjgjH8C2gQVUf9Lh9JtCiqioil+M0gF8Q1iB7j2cBcIGq1kYrBhNe1oZhjONFERmE03h9R6STheuPwKX92P5InEZqAfbg9KCKChEpwmnPsWSRwKyGYYwxxhNrwzDGGOOJJQxjjDGeWMIwxhjjiSUMY4wxnljCMMYY44klDGOMMZ78f26Ykt7+5Q+YAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn_luong.sched.plot()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "import warnings\n",
    "warnings.filterwarnings('ignore')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "4ea4ef069a17472f964ba0ebb4dd4ba2",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Epoch', max=15), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch      trn_loss   val_loss                                                           \n",
      "    0      5.152559   7.031186  \n",
      "    1      5.244082   4.662893                                                           \n",
      "    2      4.782076   5.026441                                                           \n",
      "    3      4.491039   5.464396                                                           \n",
      "    4      5.084599   5.759669                                                           \n",
      "    5      5.026393   4.667596                                                           \n",
      "    6      4.828929   4.951861                                                           \n",
      "    7      5.256153   5.085142                                                           \n",
      "    8      4.692481   5.311208                                                           \n",
      "    9      4.464477   4.832439                                                           \n",
      "    10     5.100333   4.809128                                                           \n",
      "    11     4.403005   4.937141                                                           \n",
      "    12     4.066544   4.845156                                                           \n",
      "    13     4.075488   4.992695                                                           \n",
      "    14     3.725788   4.827368                                                           \n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[4.827368184949951]"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Luong Attention model\n",
    "lr=1e-3\n",
    "model_name = f'{model_luong.rnn_type}_{model_luong.attention_type}_rev_{rev}'.lower()\n",
    "cycle_len=15\n",
    "best_model = BestModelCheckPoint(learn_luong, model_data.path, model_name, lr)\n",
    "tb_logger = TensorboardLogger(model_data.path, model_name)\n",
    "sched = DecayScheduler(DecayType.LINEAR, cycle_len, 0.5, 0.1)\n",
    "teach_forcer = TeacherForcingSched(learn_luong, sched)\n",
    "learn_luong.fit(lr, 1, cycle_len=cycle_len, use_clr=(20,10), stepper=Seq2SeqStepper, \\\n",
    "          callbacks=[tb_logger, teach_forcer, best_model])\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 136,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "====================================================================================================\n",
      "Model log:\n",
      "Seq2SeqRNN(\n",
      "  (pre_trained_vector): Embedding(52221, 200, padding_idx=1)\n",
      "  (encoder_dropout): Dropout(p=0.0)\n",
      "  (encoder_embedding_layer): Embedding(52221, 200, padding_idx=1)\n",
      "  (encoder_rnn): GRU(200, 200, bidirectional=True)\n",
      "  (encoder_vector_layer): Linear(in_features=400, out_features=200, bias=False)\n",
      "  (decoder_dropout): Dropout(p=0.0)\n",
      "  (decoder_embedding_layer): Embedding(52221, 200, padding_idx=1)\n",
      "  (decoder_rnn): GRU(200, 400)\n",
      "  (decoder_output_layer): Linear(in_features=400, out_features=200, bias=False)\n",
      "  (output_layer): Linear(in_features=200, out_features=52221, bias=False)\n",
      "  (encoder_output_layer): Linear(in_features=400, out_features=200, bias=False)\n",
      "  (att_vector_layer): Linear(in_features=400, out_features=200, bias=False)\n",
      "  (decoder_hidden_layer): Linear(in_features=400, out_features=200, bias=False)\n",
      "  (att_score): Linear(in_features=200, out_features=1, bias=False)\n",
      ") \n",
      "\n",
      "- attention_type = Bahdanau \n",
      "\n",
      "- weight_tie = three_way \n",
      "\n",
      "- teacher_forcing = True \n",
      " \n",
      "- pre_trained_embedding = glove.6B.200d \n",
      "\n",
      "====================================================================================================\n",
      "\n"
     ]
    }
   ],
   "source": [
    "attention_type='bahdanau'\n",
    "model_bahdanau = Seq2SeqRNN(rnn_type, input_size, embz_size, hidden_size, batch_size, output_size, max_tgt_len,\n",
    "               attention_type, tied_weight_type, pre_trained_vector, pre_trained_vector_type, padding_idx)\n",
    "\n",
    "print('='*100)\n",
    "print('Model log:')\n",
    "print(model_bahdanau, '\\n')\n",
    "print('- attention_type = {} \\n'.format(model_bahdanau.attention_type))\n",
    "print('- weight_tie = {} \\n'.format(model_bahdanau.tied_weight_type))\n",
    "print('- teacher_forcing = {} \\n '.format(model_bahdanau.teacher_forcing)) \n",
    "print('- pre_trained_embedding = {} \\n'.format(model_bahdanau.pre_trained_vector_type)) \n",
    "print('='*100 + '\\n')\n",
    "\n",
    "if USE_GPU:\n",
    "    model_bahdanau.cuda()\n",
    "opt_fn = partial(optim.Adam, betas=(0.8, 0.99))\n",
    "learn_luong = RNN_Learner(model_data, SingleModel(model_bahdanau), opt_fn=opt_fn)\n",
    "learn.crit = seq2seq_loss\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "47aa15f01b35458cbf4199c8f39dd09f",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Epoch', max=15), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch      trn_loss   val_loss                                                           \n",
      "    0      5.829539   5.696929  \n",
      "    1      5.353158   7.027325                                                           \n",
      "    2      5.215214   3.87996                                                            \n",
      "    3      4.620513   6.084428                                                           \n",
      "    4      5.17706    5.872052                                                           \n",
      "    5      5.002981   5.75278                                                            \n",
      "    6      4.681607   5.661423                                                           \n",
      "    7      5.045871   4.982815                                                           \n",
      "    8      4.63047    4.597715                                                           \n",
      "    9      4.465461   4.168441                                                           \n",
      "    10     4.381314   4.716841                                                           \n",
      "    11     4.004436   4.689692                                                           \n",
      "    12     3.622511   4.577933                                                           \n",
      "    13     3.460605   4.552373                                                           \n",
      "    14     2.996284   4.521513                                                           \n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[4.521513411001551]"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Bahdanau Attention Model\n",
    "lr=1e-3\n",
    "model_name = f'{model.rnn_type}_{model.attention_type}_rev_{rev}'.lower()\n",
    "cycle_len=15\n",
    "best_model = BestModelCheckPoint(learn, model_data.path, model_name, lr)\n",
    "tb_logger = TensorboardLogger(model_data.path, model_name)\n",
    "sched = DecayScheduler(DecayType.LINEAR, cycle_len, 0.5, 0.1)\n",
    "teach_forcer = TeacherForcingSched(learn, sched)\n",
    "learn.fit(lr, 1, cycle_len=cycle_len, use_clr=(20,10), stepper=Seq2SeqStepper, \\\n",
    "          callbacks=[tb_logger, teach_forcer, best_model])\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.ticker as ticker"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# function for plotting the attention weights\n",
    "def plot_attention(attention, sentence, predicted_sentence):\n",
    "    fig = plt.figure(figsize=(10,10))\n",
    "    ax = fig.add_subplot(1, 1, 1)\n",
    "    ax.matshow(attention, cmap='viridis')\n",
    "    fontdict = {'fontsize': 14}\n",
    "    ax.set_xticklabels([''] + sentence, fontdict=fontdict, rotation=90)\n",
    "    ax.set_yticklabels([''] + predicted_sentence, fontdict=fontdict)\n",
    "    ax.xaxis.set_major_locator(ticker.MultipleLocator(1))\n",
    "    ax.yaxis.set_major_locator(ticker.MultipleLocator(1))\n",
    "    plt.show()\n",
    "    \n",
    "    \n",
    "def generate(x, y, m):\n",
    "    probs = m.model(V(x))\n",
    "    preds, attention, encoder_embedding = to_np(probs[0].max(2)[1]), to_np(probs[1].squeeze(1)), to_np(probs[2])\n",
    "    sentence = ' '.join([index_to_string[o] for o in x[:,0].data.cpu().numpy() if o != 1])\n",
    "    result = ' '.join([index_to_string[o] for o in preds[:,0] if o!=1])\n",
    "    orig = ' '.join([index_to_string[o] for o in y[:,0].data.cpu().numpy() if o != 1])\n",
    "    print('Input: {}'.format(sentence), '\\n')\n",
    "    print('Original summary: {}'.format(orig), '\\n')\n",
    "    print('Predicted summary: {}'.format(result))\n",
    "    attention_plot = attention[:len(result.split(' ')), :len(sentence.split(' '))]\n",
    "    plot_attention(attention_plot, sentence.split(' '), result.split(' '))\n",
    "    return preds, attention, encoder_embedding\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 111,
   "metadata": {},
   "outputs": [],
   "source": [
    "attention_type='luong'\n",
    "model_luong = Seq2SeqRNN(rnn_type, input_size, embz_size, hidden_size, batch_size, output_size, max_tgt_len,\n",
    "               attention_type, tied_weight_type, pre_trained_vector, pre_trained_vector_type, padding_idx)\n",
    "if USE_GPU:\n",
    "    model_luong.cuda()\n",
    "learn_luong = RNN_Learner(model_data, SingleModel(model_luong))\n",
    "learn_luong.load('gru_luong_rev_1_train_end')\n",
    "\n",
    "attention_type='bahdanau'\n",
    "model_bahdanau = Seq2SeqRNN(rnn_type, input_size, embz_size, hidden_size, batch_size, output_size, max_tgt_len,\n",
    "               attention_type, tied_weight_type, pre_trained_vector, pre_trained_vector_type, padding_idx)\n",
    "if USE_GPU:\n",
    "    model_bahdanau.cuda()\n",
    "learn_bahdanau = RNN_Learner(model_data, SingleModel(model_bahdanau))\n",
    "learn_bahdanau.load('gru_bahdanau_rev_1_train_end')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 128,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0\n",
      "Input: nigerian soldiers clashed with a group of protesting youths < unk > barrels , majority of the residents of the region are poor . _eos_ \n",
      "\n",
      "Original summary: nigerian soldiers clash with protesters in oil city _eos_ \n",
      "\n",
      "Predicted summary: nigerian soldiers clash with < unk > _eos_\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAoAAAAETCAYAAACm17dTAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3XeYJFW5x/Hvb5ZlyYogCLjuogTJKAsISL6wBMWIXi+o4NVVBJTgRUWiCiqsGECUJQgoiiACKkoGERARlIxESZLjsrtsnPf+cc44TdOTuqs6TP8+z7PPdtdUv3WmZrrm7VPnvEcRgZmZmZl1j55WN8DMzMzMmssJoJmZmVmXcQJoZmZm1mWcAJqZmZl1GSeAZmZmZl3GCaCZmZlZl3ECaGZmZtZlnACamZmZdRkngGZmZmZdxgmgmZmZWZdZqNUNMLPhkXTaAF8KYDZwP/CriHi8ea0yM7NOJK8FbNYZJP0O2BzoBe7Im9cGBNwMrAUsAWweEbe0pJFmZtYRfAvYrHNcB/wReHNEbBERWwBvBv4AXApMAC4Cvtu6JpqZWSdwD6BZh5D0BLBNRNxdtX1N4IqIWEHSO4DLI2KZljTSzMw6gnsAzTrHEsAKNba/KX8NYDoe22tmZkNwAmjWOc4HTpW0q6SJkiZI2hU4FfhN3mcj4N6WtdDMzDqCbwGbdQhJiwHHAXvS38s3HzgN+FJEzJS0PoAngZiVS9KiwGbAfRHxcKvbYzZSTgDNOoykxYG3kWb/3h8RM1vcJLNRT9LpwI0RcaKkhemfeT8X+EBE/LGV7TMbKd8CNuswETEzIm6LiFud/Jk1zWTghvx4F2BJ0vjbI/I/s47iHkCzDiFpEeCLwLbAclR9gIuIdVvRLrNuIGk2sEpEPCbpFOCliDhQ0kTg9ohYsqUNNBshzxY06xwnAh8AzgWuJ60AYmbN8SSwdi7HNBmYkrcvAcxrWavM6uQE0KxzvB/YNSIub3VDzLrQacCvgMeBBcAVefvGwD9b1SizejkBNOscs4BHW90Is24UEV+XdCfwFuDciJibvzQf+E7rWmZWH48BNOsQkr5AmnW4V0T0tro9NvpJugA4BfhDt//OSdoCuD4i5ldtXwjYNCKuaU3LzOrjBLBBksYDm1N7UP5xLWmUjUqSfkf6XXsJuIuqcUcRsUsr2mWjl6SzSEMPXgJOB34aEfe1tFEtImkBsEJEPF21fRng6YgY05qWmdXHCWADJO1GGhcyH3iGVw/Kj4h4a0saZqOSpJ8O9vWI2LNZbbHuIWkpYDdSAfJJwLWkXsFzI+KVVratmST1AstHxDNV21cDboqIpVrTMrP6OAFsgKQHSIOCD42IBQXG3RKYHRF/zc/3AD4N3AkcGBEzijqWmdlwSVqLdC36HKkA8tnA9yPi7pY2rESSfpsf7gxcDsyp+PIYYG3g7ojYodltM2uEJ4E0ZnnglCKTv+z75MKiklYHTiKt9/pu4Fhgr+EGknQVwywXEhHbjLShZtYdJK0IvA94D+mux6+B8cBtkr4aEVNb2b4SPZf/F/ACUNnrOZfUI3pysxtl1igngI35A6kEwIMFx30bcHt+/CHgsoj4vKSNgfMYQQII3FHxeAzpVs6TwF/zto2AFYCfN9RiK4Wk24AtI+IFSbczSDLvQtBWNEljSUnfp4DtgH8AxwC/7LsTIekjwDRgVCaAfUMrJD0ETPXqOzZaOAFszGXAd/Jtkdt57aD839QZN0jJGqRVH87Pj58ElhlRoIh9+x5L+h5wBvDFqLj3L+n7pE+31n7Oo/+W03m4+LM11xOka8MvgK9ExG019rmM1DM2qkXEka1ug1mRPAawAXlQ8ECi3llhki4nFRu9jHTrd42IeCCPDTw9IlauM+5zwCYRcW/V9tWAGyLiDfXENbPRSdLHSZM9Zre6La0m6Q3AUQy8FKMngVhHcQ9gAyKiZ+i96rIf6RP3+4CjIuKBvH1X0hJg9RKwDnBv1fZ1GohpTSLpSuCDEfFi1falgAs8htNKsDVwAfCqBFDS4sDxEfGplrSqNU4F3kG63f047o23DucewDYjqQd4O/BI9WxfSYsACyKirnUnJU0F/pdUtf6GvPldwEGk+l4H1t1wK13ucX5TjTpkywH/joixrWmZ1UPSosBmwH0R8XCr21PLILXvlgWejIiu6USQNB3Yrq86g1mn65o3b1nybYEdSMsDLVz5tYj4eh0hA7gFWBO4vypeo7dhDgKeBr4IHJ23PQF8G/hug7GtJJLeWfF0XUnPVzwfQ1qY/t8NxHfZoSaQdDpwY0ScKGlh4EbSyi5zJX0gIv7Y0gZWyNc15X9LS6pc/WIMqSTKU61oWws9Dfi9YKOGewAbIOldwEWkQfpvJP0RXiE/f6jeWZl5tueUiPhLUW2tcYylACJielnHsGLknr++N2qtyTqvAPtGxGl1xv8HcEREXJjLDt1Gf9mh6yJiJLPObQCSngB2joi/S/owadbsRqQZth+IiI1b2sAKVb9ztQRweEQc1aQmtZykjwIfAT7pD0U2GjgBbICkP5PKInwRmA6sB8wEfgmcGhFn1Rl3R+BrwD7ArVHwD0nSJFKpmd9HxMw8nmdO9RqX1h4kTSAlfg+SEobKlQjmkpahqrsWZb61tX5EPCjpYNK6pu/pKzsUEW9uoPmWSZoNrBIRj0k6BXgpIg6UNBG4PSKWbGkDK+ReYQFXkkpRVfY6zwUejojHW9G2VskfzCeSekAf5rVVH1yGyTqKbwE3Zl3gfyMi8liZcfmP6JdJkzjqSgCBc4BFgJuB+ZIqK8/XPdtM0vLAb4ENSZ/gVyUlFceRBnl/sc72WokqxoeVNemosLJDNqgngbVzT+BkYErevgRVyUSrRcSfACStTBqP7J6CVPjabNRwAtiYuRWPnwImAHeTxoms2EDcfRpp1CC+R/8f9Ucqtp8LHF/SMa0gueDuixFxaX5+GCmJuBPYIyKeqDP034BDJV0GbE5/YjKR9PtixTiNtHTk48AC4Iq8fWPgn61qVLU85vSWiOglXSuWkWqXCY2Ivzezba3kOoCvJmk86XpRqyTOcS1plI2IbwE3QNIlwJkRcZakk4ANSInU7sASEbFJSxtYRdJTwLYRcYekl4H1co/lysAdEbF4i5tog5B0F7BfRFya/0hfDxxGmoT0ZET8T51x1yb1WE8Ajuv7QyfpBGDpiNitkG/AkPQh0oSxcyPisbztk6TE/sKWNi6rnG1eMRawVgZYd63TTpUrMbyHNITmpIh4UdLbgBci4vnBXz16SNqN9IFmPmlISmUiERHx1pY0zEbECWAD8li6JSPiKklvBM4klXW4F9gzIm4fNMDgsZcHPk660BwaEc9K2gx4PCL+VWfM6cCkiLi3KgHcCPhjRHTd7b5ciuNt+ekDEfHKYPu3kqSZwJoR8bCkbwCrRsR/S1ofuCQili/4eA2VHbLOlMecPpKHtkwYbN92LV9TBkmrAJeTbtm/HlgtXz+nAq+PiE+3tIFNJOkBUm/2oY2MP7bW8i3gBkTETRWPnwF2LCKupA1It4f+RSoTcSzwLGktztWAunp6gGuAPYCD8/OQNAb4Mv23o7qCpHGkeoifJZXvETBH0jTgy2268sFsoG+iwLakT+AAL1VsL0ybnoOOIumDw923gaUjC9WX1OV1gPcGftRNid4gvg9cSlqLvbIY+2+Bn7akRa2zPHCKk7/WknQicFhEPFvP650AtqepwA8i4vDcU9fnEmDPBuIeBPxJ0obAOFLtv7WA15F6LrvJj4HtSfXu+srtbAJ8i5RMteMKB38GvivpWmAS8OG8fTXg0XqDeomrUg134kDlRJy2EBHzJH0eOLHVbWkTmwLviogFVWMiH6GxMd+d6A+ksasPtrohXW53Ur7gBLAZJN0GbBkRL+SyAAPeQ2+gLMAGpBU7qj1B+uRVl4i4S9I6pE+wc0gzjc8lfcKvdwJBp9qVtKzaZRXbHpT0NHAe7ZkA7kNKXD8MfK6iDMeOpA8H9fISVyUpcbnIZrkE2Ib+3uZuV2u1nbeQeuG7yWXAdyStBdzOa0vitEVvdheoPTtrmJwAjtx5pOQJyisL8AqwdI3tbydVo69bRDwJHN5IjFFiJrVXz/g36fy3nTxp4L01tu/XYOht8RJXVtsVwNGS1iWVpZpZ+cUu+0N/KXAA/R/OIxfUP5K0IEA3OSn/f3CNr7Vdb7bV5kkgdRpszd4CYk8D3kTqpXqWVG8wgAuBKyNi/wZiLwasT+1bfV1zMZf0NdJ53aNv4keeEHIacGdEfLOV7RtIGbMQJd0PvC8i7iywqVZDLvK+D/BWYPuIeFTSp4F/RUTbjcPNs4AH0lWzgCWtCFyVn76VtAjAKqQSYFvkceBmTVM5mbOu1zsBrI/SIJA5pFmZ9w+1/whjL0UaY7EusDipFtvywHXAThExc5CXDxb3v0irlNSa7dttF/PfAVuSyhjcljevQ+oV/1PlvhGxS3NbV1tZsxC9xFVz5NIZPwFOAT4HrJV/fp8lDUeY3NIG2pDyh8SPAe8kfYD+O3BWO1cPsNHLCWALlb1mr6RtqLjQRMTlDca7k1T09+BuW8apmqRhz9qLiEYm3hRG0u9JY/T6ZiH2lfHZAvhpRLxt0ACvjlU9fnVlvMRVqSTdCnwrIs6uKsO0HnBp0WV8zMokaWdSBYk1SdeSu4DvRMQfWtqwLtJoAugxgI05CDhWUilr9kbElaS1OIsyEdil25M/aJ+kboSKnIXoZa2ab1X6Z5xXmgG07Uzrbv5DL+kTw903Is4ssy3tJA9bOJG03OkZefPmwPmS9ooITxpqjp8D0+t9sRPAxhS2Zq+kA4ATI2J2fjygBpbZuQ5YHXigztdb6xUyC9HLWrXE46SSPdU19bagTd+T/kPPj6qeL0x6D/aNjewh9ZjPIS0E0C2+DBwQESdUbDtV0s3AV/Cs8YZI2hKY3TcxT9IepJJldwIH9g3ViYi9GjqObwHXLy/hNKCIOGOwr1fF+hdplY7n8uNBwg5/mZ28ZFificA3geOoPXW/a9b1LLGET2kknQ3MjIj/zV3/6wLPkSYHPRgRtUoHDSfug8CGEfFc1fbXk4YeeFmnAkg6iFTH89PAxaTJPBNJdbyOiIjqZKPlJN1Hqkl6QtX2fYF9I2K11rSs+XJP6BHAfkDfjPmNSdfTb0TE71vUtKbLnR1rVY9/z+OU74yIca1p2WvlguY/Jw19assPWtUk/YN0TbhQ0uqkceqnAu8Grms08fvPcZwAjm5DrOVZqdsmgVSXwhlLmh29Gaku4iHNb9XgypqFWLn2a9X25YFHI2Lh+lttlSQdBexPunMAqedoakQc2rpWDayT/tCXTdLdwKeqx3xL2gQ4PSJWb03Lmi9/MDguIn5ctf3zpPXK2+qDgaQXgA3qHSvXbHnZ1vXzGOGDgU0j4j2SNgbOi4g3F3Ec3wJuUBlr9hZs5WYdKM8mHWg1ibaYSdtnoFugkv4PGHT901aJiMfzur+VsxCnUecsxKplynaWVHkbeQzpZ9kOv8OjQi7BdBhp1ZU1ST+/u9p85vUjpCUoqysdbM9rb2WPdhOpqoOYzSINw+gmU4Hj8x2m60mdDO8m/S3ct5UNG8BvgA+S2t0JKmspbgucnx8/Se0qHnVxD2ADaqzZ+/acsR9BKtEx7DV7JR023H0j4usjbWvZJB1LujVyFTVWk+iUSRe5pt5NEVGrEHdL5dm+10fE/KrtC5E+IV4zwnh945hq9RDPAx4ijTfpmltbZclrbs8mzdi7q9XtGa5couZ40vi/1/yhj4hpLWxeU0m6Oj/cLSL+nbetBPyM9Ld061a1rRUkfQA4EFgjb7obODYiLmxdq2rLd3z2J5X4uonXFjSvd1x9KSRdTvo7ehnp1u8aEfFAHht4ekQU0rHjBLABkq4Cron+NXv7yjpsApwdEcPuScpj0ipNABYj/RJAmuU5C3io3vFpkj4CvBgRl+bnhwFTSANL94gGloOT9BSwd0R09OxSSXsC34yIlVrdlmqSFgAr1LhVuwzwdL238POY0w2jzgXFbXhywe0PR8QtrW7LSHTSH/oy5Q+HF5AWAOhbRWgl4B7g/UXXg7XiFDmuvhkkrQ38gpQHHNd3x0rSCcDSEbFbIcdxAli/qvv0lQngROCfEbHIoAEGjrsn8AlSYd5H8ra3AD8l3e6ra4aVpLtI4zMurei6PwzYAXhyJD2WNWI/A2zSKRdBSb+t3gSsQFoT98g27WXtBZavHusnaTVSr2XblhKx/0wa+xiwu5PtzpQXANiOlASKVBLn8qJLgJnVorQS1IKImDfkzsOJ59/b+uVer50i4uaqBHAHYFpE1DUuJH9aeX9E3Fq1fX3gwpH0LFa9fiZp5ZKHJX0DWDUi/jvHvaSRQrR5cPu8iDii3hjNVKMQdC/wDGmpvUtb0KQBVSSrO5NWAqksNzQGWBu4OyJ2qDN+WWWHSiPpPQPdmpZ0cEQc3ew2DSX38q9MmnD0GK+9DdV2M8/N+uQOj7fmce4vM3gVhbb9MCppCVKvX10rajWTpLfSX3/z7qInsXgSSGMuBA6XtGt+Hrn37zvAeQ3EXR5YtMb2RYBlG4g7G1gyP96W/lpNL1VsHzZJP6x42gPsJmk70pT16hIzXxhxa0vUKWMSs77yLAJeAConfMwFrgVObiB+9aDtsaTe0FeAp0llLtrNWZJ2iojrKjcqrfF8INB2CSAdUnx7JH/oSUWs7wC+FBG3DbJfR2pSfdZOsS/wcn68TysbUg9Je5PqF66Unz9GKmh+YksbVoPScrCnAh+iv+akJJ0H/G9EvDzgi0dyHPcA1k/lrdl7IanMx2dIS7cBbAicRFo0/v11xr2AlFheCxwKTMwzSycDPxxpGYM8BnJY2nWAdNmfsIqUBzJPbcYn1zy7/afAyRFx/lD7N5uk3YEfAFv3JR6SDgEOACZHxN8Ge32z5Yk62wN/jap6i+0m36o+OyLmaIhap8A4YCdgfERsUH7rhiZpWVJVhlsiYs5Q+w8Rq5T6rM1S5LlopqLbnUupfJU0C/javHlz0vXi6Ij4dqPHKFK+Q7UpaYz+9XnzZqS1xK+LOmu+vuY4TgAbp+LX7H0jadbdDsCCvLkHuIQ0LrDeem9vBn5MKlnwg76xhJK+D/S0Wy9dmQb6hEXquS3sE1YZmpW0SnoHcE5ErFpG/EZJ+gLpov5u4H9IF/Pt2y356yNpNqlSwEOtbkuRJI0Hbo6I5VrcjiVJ7+kPk94bq+YhOT8hjXE+opXta6ZOPRdltVvSI8CXI+KXVdt3IyWAbVX6S9JzpGFgf67avgVwfkQUUgqmZ+hdbCgRcWVETI2IYxpN/nK8ZyJiJ9JA4w+R3gxrRMRO9SZ/Oe5jEfHeiFivciJJROzXaPIn6bT85q3evrikdlwW6AekntutSb2ii5Jui68LfL+F7RqQpCUlnUuqyXYBaQjCfZLOqXXuC9BD6tFuSxHxQ9Kt77+RShD9V7smf9mtpMLdo0pEPNrq5C/7Dun23jt59TCJ3wMfKOOASqtMtKNSz4WkXkkLBvg3U9Kt+QNau7R7OfrvplW6kfa8xi1K/9CfSs/TX0S+Ye4BbIAGrt0XpPF29wMXRx1FejvNICVKliV9cmur8abN+oRVpLJuC+jVBaGhf0b03qQl5naur8XFGmQM1n7An0lrcgPtOR5L0o7At4HDSW2tngTyfCvaNRhJCwNfI81efgtVa1FHG60elMd0fSAi/lY1Ka/vVmJDH5JyQvPviDgvPz+NVK3hAWCXiLinwW+hME04F58nLYt3Pq9eFu/9pCRuPPBZUq/b8a1ut6TbgF9XV3fIw2o+GBHr1RO3LJIuA6YDH4+IWXnb4qT1ppeKiO2KOE5b/VHuQLuSLoqL8+p6fTNJM0rHA09L2nKo23RVEyoGVW9vXRkXc0lvICUMApaWVFmkeAxp5upT9bS3ZE35hFWwXXht0nq1pCmkC3G940KqJycEeUY0aUJFuxhohYEFpMR40/w8aM+JKxfl/3/DqydWiFdX/m8n3wA+CnwL+B7wf6QVMf6bNI64nSxN7ff0kvQPpWnEF4BPwX8+KO5KGnrwIeC7pLWd20XZ52Iy8NWIOLVi22mSbiQlw++TdA/pPTvsBJDy2n0EcE7+uV1Hf0HzLUk/x3azP2m98H/n5DWA9Ui5xeSiDuIEsDHfBXYnFVF+DP4zzu400uLTFwHnkC6c7xsi1jrDPGYjXbZlXMyfzW0KUk2sakHq8Wg31wHfkFT9CetI+nvX2k0pSWtEdMRQkCio+n0LteVEqCF8BPhcRFwsaSqpDNUDSuvibkeamNYu/kb6kNQ3hKPvWvlZinlPr0RaHQfgvcC5EXFOLu/z5wFf1Rpln4ttqf3h8E9AX2fGZYz8g1gp7Y6I3yito7s/KVHvq+G4UUT8o964ZYmIOyStSsov+mpO/pw6l/0c7ED+V+c/0hJw69bYvj5pxQ6AdwFPtbqtFe3dIT9+GXhbfrwXqXu8nphbAluRJlJ8ID/v+7cJsGKrv+8B2r0OqRbbC6SL1tX58WPAWq1u3wBtvow0SWWxim2L522Xtbp9/jf6/pHXuc2PnwA2yI9XBqa3un1Vbd00X9dOJo0fO57Uiz0DeGcB8Z+q+P5vIS0JB2lc54xWf/9NPhcPk0r/VG//EvBwfrwe8EQ7tdv/Xv3PPYCNWZ7aPS/jSINOIV00Fqv3AAUXrVye/l66GcDr8+OLSeM2Riwi/gQgaWXgkcjv4nYXEbc35RNWsQ4A/kgJtwUk7UyqkdU3u/guUo2sPzTU4hJJ+iipJ2I5qia0RcQuLWnUMEhakTQEY+HK7THCtZyb5BHSsJZHSGOaJ5PGL27Cqwfpt1xEXC9pU1IS8gDpd+PvpBWKqpfarMelwMmS/kFK+v6Yt69F+nDdNppwLo4knYttSBMpAtiIVOroM3mf7Ugfrkfa7k1Id6cKbbekccBu9F/j7gR+GW1aHiePGd6bVBJuckQ8KunTpFJwVxRxDCeAjbkcOCmPweobgL4BqdTKZfn5OtRxcSipaGWhF3Ol5eRuiYheYBlgGUk1942Iv9fZ5sLlmXs/Bw6OiEYKKDdVWUlrvqicCJxFKj8EqUbW+ZL2ijqXHiyTpGNJkz+uIo2/bfsPHjnx+wWwBam9fWP/+rTjGMDzSX+EbyDNnP+lpM+QrkvHtrJhteQkYajahfXaGziKlLx/OPon7bwT+OWAr2qRMs9FRJyWhwHsS7plK9Ia0ZtHxA15n6kjiZlrZU4BLoiIQtstaU1SR8dSQF8i+RngSEk7RMTdRR6vUbk8zU+AU0jvv77x+mOAg4BCEsCWd0F28j9Sz8PFpNuf8/K/BaRPhsvlfbYm1SYbSdyDSd3gh+cf/rakQazTga800N5vAV/Ljz+c2/sv0moSR9URr7fi++zN33tvjX8LWv2zqtH2F0irHRQV70rg9fnxJ4BxBbd3LPAr8m37gmPfB+xTY/u+wL2t/lkN0OanSH+EW96WEbT5HNKF++35/b0Z8EHSShrbtbp9w/weNib1RL+njteW+h6pOM6KpGE476z81+pz1+zz0AnnokZ7ZwITSoh7Gals1lIV25YCfktaBrXl33tVe28F/js/frnvbxXpjk9hQ8pcBqYAklYHVid/CoqIexuM15SilZLeRRpzcW8MsK7qEK+fQL7tmx8PKCIerrOZpZB0KulnNaJPqYPEmwOsHGlllZolcQo4xgukMUiFFn7ObV8rIu6v2r4KcGdEjCvyeEWQ9AzpttD9Q+7cJpTWDt85Im5SWm5tUkTcm2+/HxoR72pxE2uS9CbSdaL6VntExI9HEKfU94hS4fKf0987XimigJI1SivkfJy0SsWhkZbL2wx4PCKGdaenSdeKZp6LtwKH1XMuasS8AvhRRPym0fZVxZ0FbBgRd1ZtXwe4ISIWL/J4jcrtXSMiHq5RDueOiKi1VOyI+RZwASLVfyqyBlRTilZG6qq/oYHXVyZ100i3464G/hYRRZQaKNMjwCGSNgdu4rU12UY6e+2fwNFKy+MJ+Ej+I/8aEXFmHe2FVD7kg6TljIr0CGm8TnUytT1psHc7mka6FX5Ei9sxEouSZs1Dmrm9HHAvabzluq1q1GCUltw7Bf6zDnVlj0GQhrsMV9nvkWnAo6Rbe4UPC5C0AakH91+kcX/Hkn6e2wGrkUrCDEczrhXNPhdTqe9cVDsZmCrpLdSulVnvUKLZ9I95r/S6/LV28zjpPFZff7cgjY0shHsARyjX6/tqRMwcqnZf1F+vr7CilTWK/A6okU9dko4izfzdkHRL+XpSMng1cGO7JYQqeF3PPOD6B6TB4UuRxlTWenNFRCw1ktgVxzicVMbgTxSTtPbF/Sxptt0ZpJ9bX42sjwP7RsS0euLm2K8jzZR/TZmM3FtwV0S8UEfcH5H+yNwF3EYazvAf9b73qo4xljQ+9saIaPiPRK6RdlikkioXkCZifY10q/190YZL7kl6mPR78fWImD/U/kPEKvU9Imkm8I5G78AMEv8q4JqIOLyqV2YT0trJw7oz06RrRUecixpxewf5ct09l5LOIP1t+gz9nR6bkMoY3RgRe9YTtyySDgL2BD5NGmb2HlLJtqnAERHxo0KO4wRwZPIv/gci4sX8eEARUVfdr5y0nUNKniqLVm5FGvd0wQhiDfaGqlTUbYFFSWObtsr/NgJm13sh60T5nL+phNs6hSatVbE/QKrrtUbedDdwbERcWG/MHHdJUvmQyRFxXcX29UkrCKwUEc8O9PpB4pby3qs6xq7A2aS1oU8vIN5uwNiIOD1PoLoYWBaYA3wiIs5t9BhFK3HYQeHvEUk3AAdFSbOpcy/d+jnRqUx6JgL/jIgR1+Is8VrRcecixy1lKJGk15M+yLyX/oLSY0jjAveIiJfqiVum3KmyP/2VRuYAUyPi0Ip93ky65T7cv/OvPoYTwPaUu9j3J/1BFmnK+nHRhkUrK+VxIVsB25AmwIwnjbGo+w9ynh22EbVLZ9R1i0QDr09cuYzfryLi8QH2Gyz2BFIv6F68uuTAiUVd6JXKAxERMwqIdQHpNt8f6r2QDBH/LFKdtM9WbJsKrBbtXa7l96QB9PdHxFYFx16CNJ77u1fjAAAgAElEQVRuFdI42hEnwc0g6QTgnhjBcl7DjFvIe0RpJaI+6wNHA4eQZnpW9wo3tNReHsO5U0TcXJX07ABMi4i31BGzsGtFp5+LitgDXe8jIn7WYLtXof9D7l0RUdjt1DJIWoz0e9FDau+Mqq//JxGvK74TwMZo4FpkERFDrf4xUMw1STNn78nPtyfNFrsTOKbe26n5E8WjEfGTqu2fIxVsHmht4+HE/hEp4ZtAGqvYV1z5L9FAnSVJbwd+Ryo8K9Knt4VIF7Q5Ddwi+R2p1EkvaRYmwNr5GDeTxrUsQSprcMsIY29G6t15CvhL3rwJ6XdkckT8ZaDXDiP2fqRZmCvlTY+Tqu1/P+p8M+cE7f3AS8DpwGlR4OQKSZNJZTKWj4h5knpIBbf3qXfYgaTfDvLlut97FfGXI42h2hG4BFil3t6HqriF//zKpLR85AWkJKVWIvH1Wq8bRtxC3iO5B63yvPVNeKje1vAdDknTgDeRlg57ljRuM0i9SFdGxP51xCzsWtHp5yLHLeV6n2N31HtvOCqT77pe36Hfd1vQELXI6h1XIOkvwA8i4uzcxXsPKaFaF/hZRHy1zriPALtGxF+rtm9EWtao7tnF+eLzDHACqQzOzUW8qSRdDLxIWuf2SdIn29eRBp8fEhGXDfLyweJ+hTSl/n+jfym4xUiDkG8lLUV0JvDGiNh2hLH/Qvpj+bm+HrWc9PwEWDsiNh3s9YPEPYZUJ+tYXv3H4kvAyRFxUD1xc+ylSEVS9wQmAdeSegXPjQYLY+fv/RHgC5GWZNqOlBCuEBHzBn/1gDF/WrVpLOnnOR74TUR8qsE2708qw7CxpEuBP0fENxqMWdrPryyS9iWNV3sWeJqqSSARUdfklaLeI5K2rHg6kZS0V39A7iGtZnIGDcjvkYtIv2eLk65Hy5OG6ewUdRTrL/Ja0ennIsct63rfce+94Wg0AWx5vZtO/kdJtchIb4DV8uP9gavy463JS8zVGXc2NWrfkabxz26wzauQBqz+nHTheYH0Se4AGqg7RVr7du38+CVg9fx4S+C2BuI+QZpmX719TfLyRcA7gOfqiP1KXzurtr8deKWBNj9f6/eNVNNxxO0c5DhrkdaKfiWf85NqnasRxvwOqcArpMT6R0W1t+o43yUNkm40zi3A3vnxJyigHmKzfn4Fn8+ngf1LiFv4e4SU7CxXY/syNFiLlPQB46+kcl/bkBKHg4D/arfz0KnnIscu63rfce+9YX5f/6kRWM+/jlgEvo31kP5QFG0M6ZYLpNvLfctxPUBjZWAeId32rLYF6ZZc3SLi/og4JSJ2j4jxpLphz5L+8NcqaTNcIq1HCqmHsa/7/jFS0lmvJYAVamx/U/4apMLb9ZRKeol0C6PayqTkvhG3DbCtkPey0moV7yPNOpsP/JrUq3abpC81EPpMYAdJ40lrRjfUAzGIk4DPNxJA0rqkcUJn503nASvmGY6NKvXnV4IxpGK5RSvjPVK9skqfJWiw1EeknuqV08O4MiKmRsQxEXF5I3Ep71rRiecCyrveQ+e990rnOoCNKasW2R3AXnkQ+rZA3y3fleivI1aPk4Dv5XE9V+Zt25JWCKlrLeA++bbFJFIv5VakmcCLkMbTDTpjcwh3kG4zPEgaW/hlpeKpn+G1detG4nzg1Dzd/m/0r2V5DKneHvl5PWUUzq6IXVlW5ds0tmTUmaTlqL5YtX0voO7B0UrlTt4HfIpUx+sfpPPwy8iDjiV9hPT7XlcNwoi4U9LtpKXQHouIG+tt7xBWLyDGJ4GLI+I5gEglny4A9qD/9lE9Svn5DUapntpjUf/knp+ShgbUNdZvEIW9R9RfjiuAbykV0e0zhvQ+LuKD+hmk687/FRCrT6HXig4/F1De9b7p770maWiYlRPAxrwe+J88pqnIWmRfJg28/hJwRvQvgr0L6U1Rl4j4rqRlgR/SP7tqLmm84TH1xs1eBMaRkoerSeOG/hx1jgWpcBRpjAmkGW2/JyWUzwIfaSDu50gDgH9O//tgPnAa6bxDKoXymde+dEgHkT7JnlYRex5pHMtX6mwvpPP7P3lSRV8tq41Jyz2dVXHxH+nv3hO5vb8gLTVY65PyZaTb+o34GWls5dcajFP5h+4/m0g9ujuSznu9cceQ6gvuU/WlnwO/kLRvRMx97SuH1c6FgN0H+vnV2+YhPATcKWnvqK8kyGLAp3Obi7zGFfkeWSf/L1LPbeXPZy7wd4opnr44sFu+3tcqUlzPuSj6WtHJ5wLKu96Xde0ckNJayatGRJl5VvUqLyN7cb6PbHXQ4LXIIiK2aSD2GNK6hS9UbJsIzIoGS4lIWpw01k3UmFpeZ8wdKCbhG86x3gC8EAX88uZz8TbSubi/yPbnSSWVsWcN8ZKh4g23J3VEv3uSPk6a7FFqRfz8c9sXOCkinmwwVvW56JuEdCVpFnNdRYslrUBK+r9dmejlHu6DgTMj4pEG2jmQhq4Xgxx/D9Ltuh0iYuM6Xl/aNS7HL+w9kicGfTEiaq6q0aiSr/dFXys69lzUOFbD1/tWvA8l7QMsExFHFhFvgGOMJ9UBrK8yiBNAMzMzs+7StYMfzczMzLqVE0AzMzOzLuMEsECSpnRS3DJjd1rcMmN3WtwyY3da3DJjO275sTstbpmxOy1umbE7LW5ZsZ0AFqusH35pv1Qlxu60uGXG7rS4ZcbutLhlxnbc8mN3WtwyY3da3DJjd1rcUmI7ATQzMzPrMp4FPIRl3zAmJo4fO6x9n3luAW9cZnhrbN939/DXtJ7b+woL9yw67P2ZP/wZ4XOZw8KMG9a+I/ldmcccxg4zLhp+KaN5MZuxWmTY+1NWm0egLeKOsFrUvJjDWA0z9gguIW1xLtokdjvEVc/w+wDmxmwWHsF7L3qHX3u6Hc5FO8QtM3anxR1x7BFc40Z0fRuBEcct6dr5Mi88GxFvHGo/F4IewsTxY7nxkvGFx935nZMLj9mn98WXyok7d97QO9VBY8v7NYx5dZWDG0bgehdWGCpuOR/ItFCJ53h+See4E/UM7wPgiPXWVeZrSD2LLlZKXIDeWQ2VshtYWee4TCX9/Eby4Xm005jO+70o69p5efz64eHs51vAZmZmZl3GCaCZmZlZl3ECaGZmZtZlnACamZmZdZmGE0BJV0s6oYjGDONYR0i6oxnHMjMzMxutipga+EGgnOmhrzUVOL5JxzIzMzMblRpOACPi+SIaMhhJPaSahTOAGWUfz8zMzGw0G/IWcL7Fe6KkoyU9K+lpSVNzUvaaW8CSlpf0W0mvSHpY0p6S7pB0RMU+r5M0Lcd6WdKfJE2q+PoekmZI2inf8p0LrFF9C1jShpIuze2aLulaSZtUtT8kTZF0rqSZkh6UtHtDZ83MzMysgw13DOBuwHxgU2AfYD/gowPsewYwAdgGeB+we34OgCQBFwErAe8B3gFcA1wpaYWKOIsAhwCfBdYEahU2XBL4GbA5sBFwC/AHSctW7XcYcCGwHvAr4DRJEzAzMzPrQsNNAO+KiMMi4t6IOAe4Cti2eidJqwOTgc9GxF8i4hZgD6Cy3PzWwPrAhyPixoi4PyIOBR4EPl6x3xhg34i4Lh/35erjRcSVEfGziLg7Iv4J7AvMBnao2vVnEfHziLgfOJSUzG4+0DebewxvknTTM8+VVMHdzMzMrEWGmwDeVvX8cWC5Gvu9HegFburbEBGP5v37bEBKCJ/Jt3lnSJoBrA28rWK/+aQevQFJWk7SSZLulfQS8HJu11sGan9EzAeeGaD9fftMi4hJETFpuGv7mpmZmXWK4U4CqZ7lG9ROHoezMGEP8BS1e+CmVzyeExFDdb+dASwP7A88BMwBrgAWrtpvuO03MzMzG/WKXiH+blJitQHwVwBJbwZWrNjn76SkrTciHmzweO8GvhARF+VjLQ+sMPhLzMzMzLpbob1gEXEPcAnwE0nvkrQ+8FNgFqnXDeBy4DrgQkk7SlpZ0iaSjpQ04Li8AdwL7C5pTUkbAmeTZgybmZmZ2QDKuA26B/AYcDXwW+As4GnS5AwiIoCdgCuBk4F7gHOA1Xn1WMHh+BSwBHAzKfk7jXQr2MzMzMwGoJSPlXiAVJLlceBjEXFeqQcrwaT1FokbLxlfeNyd3zm58Jh9el98qZy4c8tZ8EVjix6J0C/mzS8pcG9Jcct5P2qhEs/x/JLOcSfqKWnSWG851Qh6Flts6J3q1DtrVjmByzrHZSrp54eGM+y+O2hM5/1elHXtvDx+fXNETBpqv8L/KkjahlSf73bSTNujgGeBi4s+lpmZmZmNXBndAmOBbwJvJY39+yuwRUTMLOFYZmZmZjZChSeAEXEJaSKImZmZmbWh8gYGjRL3PrgM2310z8Ljjv3lU4XH7DPruHVLibvYlXeWErescW8AMbekSeFltbmsMT0qsexlp41DKnnccyfpfeWVVjdh5Eoaf1vmGLKyhgx33HWI8sadlvXziwXlrQbW6vHTLoZsZmZm1mWcAJqZmZl1GSeAZmZmZl3GCaCZmZlZl3ECaGZmZtZlnACamZmZdZmmJoCSrpZ0whD73CHpiIrnD0n6UumNMzMzM+sSnVAHcEPAq4iYmZmZFaTtE8CIeKaR10vqARQR5VVzNDMzM+sgI74FLGkLSTdImiHpJUl/lbR2/toHJd0uaY6kRyV9TRq4pLik5SRdKOkVSQ9L+lSNfV51C1jS6yRNk/S0pJcl/UnSpIqv75HbtpOkO4C5wBqS1pF0haTp+XW3Stp6pN+/mZmZWacbUQ+gpIWAC4FTgd2AscA7gQWSNgDOBb4JnEW6dXsSMB04foCQpwMTgP8CZgHfAyYOcnwBFwEvAe8Bngc+CVwpafWIeCLvughwCPBZ4BngCeB64FZgI2A+sA4weyTfv5mZmdloMNJbwEsBrwd+FxEP5G3/BJB0FvCniDg8b79X0qrAl6mRAEpaDdgReHdEXJe3fRJ4cJDjbw2sD7wxIvoWsTxU0nuBjwPH5G1jgH0j4uaK400ApkbEP/Om+wc6iKQpwBSAceNeN0hzzMzMzDrPiG4BR8TzpF67SyRdJOkASePzl9cArqt6ybXASpKWqhFuDaAXuLEi/sPA44M0YQNgMeCZfJt3hqQZwNrA2yr2mw/cUvXa44BTJF2Zb02/fZDvc1pETIqISQuPXXyQ5piZmZl1nhGPAYyIPYGNgWuAXUg9fZMBATHQy2psG3Bs4CB6gKdIvYCV/94OHFqx35zqSR8RcQSwJnABsClwW60xh2ZmZmajXV2zgCPiVtJ4uu9I+iNpHN5dwLurdn038FhEvFwjzN2khG5D0vg8JL0FWHGQQ/8dWB7ojYjBbhUP1O77gPuAH0r6MfBp4LSRxjEzMzPrZCPqAZS0sqRvS9pU0oQ8i3ZdUvL3XWBLSUdIWk3SbsCB9I/Le5WIuAe4GDhJ0iaS1ifdXn6l1v7Z5aTbzBdK2jG3ZxNJR0rafJB2LyrpR5K2kjRR0sak5PSukXz/ZmZmZqPBSHsAZwGrkWb7Lku6HXsW8J2ImCdpV+BI4OD8tW8Dg638sQdwMnAl8Gx+7XID7RwRIWkn0kzjk/O+T5GSwjMHOc4CYGngDOBNwHPA7wGvMGJmZmZdRxEDDdszgKWWXCk2fOfehccd+/WnCo/ZZ+Zxby4l7mJX3llKXEr8Hex9ZbAO5QaU1eaBy2Y2FnahsaXEBYj580qLXYoyr3k9Y8qJ21tSHfuSft+AznuPjCnpZwfE/PmlxS5Fib8XPYstVkrcsn5+saC8NSR6Z5azyNnl8eubI2LSUPs1dS1gMzMzM2s9J4BmZmZmXcYJoJmZmVmXcQJoZmZm1mXqqgPYTTRvAWOfeLHwuDN+UM5EDYDHJpczgHeVZ1cpJe5Cz0wvJS6AHh1sYZn6xdy5pcQtTU95g7o7bvB1iQPcSzsX0VtK3I5U0uSSMgf7d5wyJ0p12MTTniWXKC1276xZ5QQe5il2D6CZmZlZl3ECaGZmZtZlnACamZmZdRkngGZmZmZdxgmgmZmZWZdpywRQUkj6cEGxJuZ4Qy6LYmZmZtYN2jIBNDMzM7PyOAE0MzMz6zItSwCVHCjpPklzJD0m6VsD7PttSfdIekXSQ5KOkbRIxdfHS7pQ0vOSZkn6p6T/rgozQdJl+et3Sdqu1G/QzMzMrE21ciWQo4G9gAOAa4A3Au8YYN+ZwKeAfwNrAj8B5gCH5q+fCCwCbA1MB1avEeMo4P+AzwOHAGdLmhARM4r4ZszMzMw6RUsSQElLAPsD+0XEaXnz/cBfau0fEd+oePqQpKOBL9GfAE4AzouIW/Pzf9UI872I+F0+/sHAJ4D1gWsb+V7MzMzMOk2regDXBMYBVwxn5zwjeD9gFWAJYEz+1+cHwE8k7ZBjnh8RN1eFua3icd8CscsNcLwpwBSARRZaajhNNDMzM+sYrRoDOOzV2CW9CzgbuAR4L+k28SHA2L59IuJUYGXgp8BqwPWSjqgKNa9i/76lkmt+/xExLSImRcSkhccsOtymmpmZmXWEViWAd5HG8G07jH03A/4dEd+IiL9FxH2kW76vEhGP5cTtI8Bh5B48MzMzM3u1ltwCjoiXJf0A+JakOaRJIMsAG0TEj6t2vxdYSdJupDGCk4GPVe6QY/0x77sUsAMpyTQzMzOzKq2cBfxV4AXSRI43A08BZ1bvFBG/k3Qs8H1gUeBSUg/fiRW79QDHA+OBl0njAA8ss/FmZmZmnaplCWBE9ALfzv+qv6aq518lJYyVflzx9X0HOc5D1BhzWH0MMzMzs27hlUDMzMzMuowTQDMzM7Mu4wTQzMzMrMs4ATQzMzPrMq2cBdwResctxKxVly087hL3vFB4zD5vWLb49gLMW2rs0DvVYczMcaXEBehZqpyVXHqnTy8lrsaMGXqneuIuXM7PDqD3ldnlBF6woJy4Ku9zb1nnOUo6F+opby5czJ9fTmB5/l4ni7lzW92EkVnQW17sEq9Fw+EeQDMzM7Mu4wTQzMzMrMs4ATQzMzPrMk4AzczMzLqME0AzMzOzLuME0MzMzKzLdHQCKGmipJA0aYj9rpZ0QrPaZWZmZtbOOjoBBB4FVgBuAZC0VU4IyymEZ2ZmZjYKdHQh6IhYADzZ6naYmZmZdZK26wGUtKOklyUtlJ+vmnv1flyxz1GSLqu8BSxpInBV3uWZvP30itA9ko6W9KykpyVNlVpchtvMzMysBdoxAfozsAjQN65vK+BZYOuKfbYCrq563aPAh/LjtUi3hr9Y8fXdgPnApsA+wH7ARwtrtZmZmVmHaLsEMCJmAH+nP+HbCjgBmCBpBUmLARtSlQDm28HP56dPR8STEfFSxS53RcRhEXFvRJxD6i3ctlYbJE2RdJOkm+bNnVnUt2ZmZmbWFtouAcyuJiV+AFsCfwRuzNs2A+bl5yNxW9Xzx4Hlau0YEdMiYlJETBq78OIjPIyZmZlZe2vnBHAzSWsCSwI3521bk5LA6yNi3ghjVu8ftO/3b2ZmZlaadk2A/gyMAw4Crs23d6+mPwG8eoDXzc3/jym3eWZmZmadqy0TwIpxgLvTP7P3L8B4YGMGTgAfJvXs7SzpjZKWKLmpZmZmZh2nLRPA7CpST97VABExG7gBmMMA4/8i4t/A4cBRwFOkySNmZmZmVqFtC0FHxFeAr1Rt26rq+UOAqrZ9A/jGYK/L2/YopKFmZmZmHaadewDNzMzMrAROAM3MzMy6jBNAMzMzsy7TtmMA20WMEfMXLz5P1oxXCo/ZZ5k7ylm9ZKHHnislLgsWlBMXYEw5n3E0ppxKQz1vWLqUuL1vWLKUuAA9jz9TStzel14uJS49GnqfekMv/fpS4sacp0qJq0UXLSUuQMyYUU7gkpZw71lkXClxAXpnzSonsEr6XS7pHJcZWwuVk86opL8h7WD0fmdmZmZmVpMTQDMzM7Mu4wTQzMzMrMs4ATQzMzPrMk4AzczMzLqME0AzMzOzLuME0MzMzKzLOAE0MzMz6zJdnQBKWkpSOZVbzczMzNpU1yWAksZImizpF8CTwHqtbpOZmZlZM3VNAihpLUnHAI8AvwJmAjsA17S0YWZmZmZNNqrXApa0DLAb8AlgXeBiYD/gtxExZ5DXTQGmACy8mO8Qm5mZ2egyqhNAYF/gcOB6YNWIeHg4L4qIacA0gCXeMD7Ka56ZmZlZ8432W8DTgEOAZYE7Jf1M0vaSxrS4XWZmZmYtM6oTwIh4PCKOiojVgf8CZgBnA49J+q6kd7S2hWZmZmbNN6oTwEoRcUNE7AWsQLo1vBpwo6TNW9syMzMzs+Ya7WMAXyNP/vg18GtJywELWtwkMzMzs6bqugSwUkQ83eo2mJmZmTVb19wCNjMzM7PECaCZmZlZl3ECaGZmZtZlnACamZmZdZmungQyHD3zeln0yQFXjavfvHnFx8w0p5yJzTFzVjlx588vJS6AFirnVzwWlDR5XCon7AIvaNNHJZ1jgAVvWrqUuHrhxVLi9iy+WClxAXpLul6UpsTfi06jnvLOhcaU1O+06CKdFbcNuAfQzMzMrMs4ATQzMzPrMk4AzczMzLqME0AzMzOzLuME0MzMzKzLdEUCKOkhSV9qdTvMzMzM2kFXJIBmZmZm1s8JoJmZmVmXafsEUNLVkk6o2na6pN9XfP1ESUdLelbS05KmShrwe5O0u6TpknYpu/1mZmZm7abtE8Bh2g2YD2wK7APsB3y01o6SvgAcD7wnIn7btBaamZmZtYnRkgDeFRGHRcS9EXEOcBWwbfVOkr4OfA3YJiKuaXYjzczMzNrBaFkL+Laq548Dy1Vt+yKwJLBhRNw3WDBJU4ApAOPGva6oNpqZmZm1hU7oAewFqlemHlv1fF7V8+C139u1efvHhjpgREyLiEkRMWnhsYuPpK1mZmZmba8TEsBngBWqtq1XR5ybge2BAyQd2nCrzMzMzDpUJySAVwI7StpF0uqSjgPG1xMoIv5GSgIPlHRIkY00MzMz6xSdMAbwNGDd/D/AicD5wLL1BIuIGyVtD1wqiYj4ZjHNNDMzM+sMbZ8ARsQ8YO/8r9bXt6qxbY+q5xOrnt8IvL6oNpqZmZl1kk64BWxmZmZmBXICaGZmZtZlnACamZmZdRkngGZmZmZdpu0ngbSaFgQLvTyn8Lgxr7p2dXF6Zs8tJ3D0lhO3t6S4AAtX1wwvhuaW89aJRRYuJ+7YMaXEBZBK+hzZU13/vSBjSjwXc+eXE7inpHNc4rnoOBHlxVZJv8tlKes9DURJ51nzS3rvlUhlXeMWDG839wCamZmZdRkngGZmZmZdxgmgmZmZWZdxAmhmZmbWZZwAmpmZmXUZJ4BmZmZmXcYJoJmZmVmXcQJoZmZm1mVGdQIoaUVJLnZtZmZmVmFUJ4DAZ4DHJB0nab1WN8bMzMysHYz2BPA7wL7AqsBNkm6VdICk5VvcLjMzM7OWGdUJYETMjohzI+K9wIrAKcDHSL2CF0n6iKTXLL4qaYqkmyTdNHf+rGY328zMzKxUozoBrBQRz0TE8RGxIbADsD7wK2DTGvtOi4hJETFp4YUWa3ZTzczMzErVNQmgpCUkfULSZcAlwAPAp4EbW9syMzMzs+Ya1TNkJY0BtgM+DrwfeAL4GTAlIv7VyraZmZmZtcqoTgCBg4EDgXOByRFxbYvbY2ZmZtZyoz0B/BlwbETMbnVDzMzMzNrFqE4AI+KhVrfBzMzMrN10zSQQMzMzM0ucAJqZmZl1GSeAZmZmZl3GCaCZmZlZlxnVk0CKEGPE/CXHFR53bM+YwmP20Zx5pcTtnTe/lLgxv5y4ACor9oIFpYTVgt5S4jK3vHNMWee4N0oJq7HlXfZ6x40tJW7PGH9Wt+ZRJ/6+LVTS+1oqJy6AWnueO/CnbGZmZmaNcAJoZmZm1mWcAJqZmZl1GSeAZmZmZl3GCaCZmZlZl3ECaGZmZtZlnACamZmZdRkngGZmZmZdpm0SQCUHSXpA0iuSbpe0e9U+60i6PH/9eUmnS3pd1devkDRd0suSbpW0dfO/GzMzM7P21U4rgXwT+DCwN3APsAlwsqQXIuIiSYsBFwN/AzYC3gCcDJwGfCjH+AVwa/76fGAdYHYzvwkzMzOzdtcWCaCkxYEDgO0j4s95878kbURKCC8CdgOWAD4eES/n100BrpK0SkTcD0wApkbEP3OM++tszxRgCsC4ca8bYm8zMzOzztIWCSCwJrAIcLGkygVAxwIP5cdrALf1JX/Z9UBvfv39wHHAKZI+CVwBnFeRDA5bREwDpgEsteRK5SxIamZmZtYi7TIGsK8d7wXWr/i3FrB9/pqAgZKxAIiII0jJ4AXApsBtkj5VTpPNzMzMOlO7JIB3AXOACRFxf9W/hyv2WU/SkhWv25T0PdzdtyEi7ouIH0bEzsCpwKeb9D2YmZmZdYS2uAUcES9LmgpMlSTgGtJ4v3cBvfmW7FnAkcCZkg4DlgZOAn4TEfdLWhSYCpxLum28PPBu4K/N/n7MzMzM2llbJIDZocBTwJeAHwPTgVuAYwAiYpakycD3gRtJs3svBL6YX7+AlBSeAbwJeA74fY5nZmZmZlnbJIAREcDx+d9A+9wObDvA1+YC/1NO68zMzMxGj3YZA2hmZmZmTdK0BFDSWyTNGOTfW0o89k8GOe5PyjqumZmZWTtq5i3gx0mlXQb7elkOI00QqWV6icc1MzMzaztNSwAjYj51rsxRwLGfBp5uxbHNzMzM2o3S3AsbiKRngIeH3DFZFni2hGaUFbfM2J0Wt8zYnRa3zNidFrfM2I5bfuxOi1tm7E6LW2bsTos70tgTIuKNQ+3kBLBAkm6KiEmdErfM2J0Wt8zYnRa3zNidFrfM2I5bfuxOi1tm7E6LW2bsTotbVmzPAjYzMzPrMk4AzczMzLqME8BiTeuwuGXG7rS4ZcbutLhlxu60uGXGdtzyY3da3DJjd1rcMmN3WtxSYnsMoJmZmVmXcQ+gmZmZWdwuPnwAAAAmSURBVJdxAmhmZmbWZZwAmpmZmXUZJ4BmZmZmXcYJoJmZmVmX+X9O2PIGTR8L+wAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 720x720 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Luong (Global Dot) Attention\n",
    "x,y = next(iter(model_data.trn_dl))\n",
    "for i in range(1):\n",
    "    print(i)\n",
    "    preds, attention, encoder_embedding = generate(x.transpose(1,0)[i].unsqueeze(1), y.transpose(1,0)[i].unsqueeze(1), learn_luong)\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 129,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0\n",
      "Input: nigerian soldiers clashed with a group of protesting youths < unk > barrels , majority of the residents of the region are poor . _eos_ \n",
      "\n",
      "Original summary: nigerian soldiers clash with protesters in oil city _eos_ \n",
      "\n",
      "Predicted summary: nigerian clash clash protesting protesting clash _eos_\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAo4AAAD9CAYAAAAyJ/amAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzs3Xmcm1XZ//HPt6ULUEAEqYhYNkFZBAVZBQQebMUVRdQHUEAtoiCr+ICA9UFAoSI8KLKWRVGEH0JdgbK5AMqirGUrUBDK3gpt6UI71++Pc+KkITOTSXJPks73/XrNa5I7d677JDNJrpz7nOsoIjAzMzMz68uQVjfAzMzMzDqDE0czMzMzq4kTRzMzMzOriRNHMzMzM6uJE0czMzMzq4kTRzMzMzOriRNHMzMzM6uJE0czMzMzq4kTRzMzMzOriRNHMzMzM6vJMq1ugJkVT9KkHm4KYD4wDfhVRMwYuFaZmVmnkdeqNlv6SfotsD3QBdyfN28MCLgL2AgYBWwfEXe3pJFmZtb2fKrabHC4Bfgj8PaI2CEidgDeDvwBuA4YA/we+GHrmmhmZu3OPY5mg4CkZ4GdI+LBiu0bAjdExOqS3gtcHxGrtKSRZmbW9tzjaDY4jAJWr7L9rfk2gFfxuGczM+uFE0ezweEq4AJJn5G0lqQxkj4DXAD8Ou+zJfBIy1poZmZtz6eqzQYBScsBpwH70d2ruAiYBBwZEXMlbQbgyTFmxZG0LLAd8GhEPNnq9pj1lxNHs0FE0vLAuqTZ1NMiYm6Lm2S2VJN0EXB7RJwlaTjdVQwWArtHxB9b2T6z/vKparNBJCLmRsS9EXGPk0azATEW+Fu+/HFgBdLY4gn5x6yjuMfRbBCQNBI4BNgFWI2KL40R8Z5WtMtsaSdpPrBeRDwt6XzglYg4QtJawH0RsUJLG2jWT55BaTY4nAXsDlwB3EpaMcbMivccsHEuiTUWGJ+3jwJeb1mrzOrkxNFscPgk8JmIuL7VDTEbZCYBvwJmAIuBG/L2rYCHWtUos3o5cTQbHF4D/tXqRpgNNhHxv5IeAN4BXBERC/NNi4AftK5lZvXxGEezQUDSN0gzOQ+MiK5Wt8eWbpKuBs4H/jDY/98k7QDcGhGLKrYvA2wbEX9uTcvM6uPEsUUkrQlsT/WJCqe1pFG21JL0W9L/2yvAVCrGVkXEx1vRLls6SbqUNDziFeAi4MKIeLSljWoRSYuB1SPihYrtqwAvRMTQ1rTMrD5OHFtA0l6kcS+LgBdZcqJCRMQ6LWmYLbUkXdjb7RGx30C1xQYHSSsCe5GKzm8B/JXUC3lFRMxrZdsGkqQuYHREvFixfX3gzohYsTUtM6uPE8cWkPQYabD0cRGxuIlxdwTmR8Tf8/V9gS8DDwBHRMScZh3LzKxWkjYivRd9lVT4+jLg9Ih4sKUNK5Ck3+SLHwGuBxaU3TwU2Bh4MCLGDXTbzBrhyTGtMRo4v5lJY3Y6uaCspA2Ac0hrEX8AOBU4sD/BJN1EjWVbImLn/sQ2s8FB0tuATwAfJZ1l+X/AmsC9ko6OiImtbF+BXs6/BcwCyntZF5J6YM8b6EaZNcqJY2v8gVSK4fEmx10XuC9f/jQwJSK+Jmkr4Er6mTgC95ddHko67fQc8Pe8bUtgdeDndbfYCiPpXmDHiJgl6T56+RLgAuDWTJKGkZLF/YFdgX8CpwC/LJ35kLQncC6wVCaOpeEfkqYDE71Sky0tnDi2xhTgB/n0zX28caLCr+uMG6QED9IKIVfly88Bq/Q7WMTBpcuSfgRcDBwSZeMbJJ1O+kZt7edKuk+PXYmLftvAeZb0vvAL4H8i4t4q+0wh9cQt1SLiu61ug1kzeYxjC+TB0j2JemfZSbqeVGR2CukU9bsj4rE89vGiiFi7nrg59svANhHxSMX29YG/RcSb641tZksXSfuQJsHMb3VbWk3Sm4ET6Xm5T0+OsY7iHscWiIghfe9Vl0NJ3/A/AZwYEY/l7Z8hLTPXCAGbAI9UbN+kwbg2ACTdCHwqIv5dsX1F4GqPUbUm2wm4GlgicZS0PHBmROzfkla1xgXAe0mn5Wfgnn/rcO5xXEpIGgK8C3iqcva0pJHA4oioe11USROBL5FWOvhb3rw1cBSpRtsR9ca24uVe7rdWqSW3GvBMRAxrTcusvyQtC2wHPBoRT7a6PdX0UrtwVeC5iBg0nRaSXgV2LVW7MOt0g+bF227y6YtxpGWohpffFhH/W0fIAO4GNgSmVcRrxumio4AXgEOAk/K2Z4HvAz9sQnwrgKT3lV19j6SZZdeHAmOBZ+qM7fJPA0DSRcDtEXGWpOHA7aRVgBZK2j0i/tjSBpbJ72vKPytLKl8tZSipNM3zrWhbC70A+LVgSw33OLaApK2B35MmLryF9MG9er4+vd4Zrnnm7PiIuK1Zbe3hOCsCRMSrRR7HGpd7Gksv8mqTmOYBB0fEpDpi/xOYEBGTc/mne+ku/3RLRPR3Fr9VIelZ4CMR8Q9Je5BmIW9JmrG8e0Rs1dIGlqn4f6smgO9ExIkD1KSWk/RZYE/gi/4yZUsDJ44tIOkvpPIUhwCvApsCc4FfAhdExKV1xv0w8G3gIOCeKOCPK2kLUtmf30XE3DxmaUHlOqzWHiSNISWMj5OSjfLVKxaSljyrq55oPgW3WUQ8LukY0rq7Hy2Vf4qItzfYfAMkzQfWi4inJZ0PvBIRR0haC7gvIlZoaQPL5F5oATeSSoKV93AvBJ6MiBmtaFur5C/0a5F6XJ/kjVU0XArLOopPVbfGe4AvRUTksUAj8ofvt0iTW+pKHIHLgZHAXcAiSeUrFTQ0e0/SaOA3wPtJvQbvJCUjp5EGwB9Sb2wrTtkYuCImZDW1/JP16Dlg49zzOBYYn7ePoiIJabWI+BOApLVJ463dM5EKnpstNZw4tsbCssvPA2OAB0njYN7WQNyDGmlUH35Ed0LwVNn2K4AzCzyuNUEutvzviLguXz+elIA8AOwbEc/WEfYO4DhJU4Dt6U5o1iL9r1hzTCItUToDWAzckLdvBTzUqkZVyuNp746ILtL7xCpS9RKvEfGPgWxbK7mO45IkrUl6v6hWmui0ljTK+sWnqltA0rXAJRFxqaRzgM1JydfewKiI2KalDaxC0vPALhFxv6TZwKa5l3Rt4P6IWL7FTbReSJoKHBoR1+UP+FuB40kTtJ6LiP+uI+bGpB7yMcBppQ9IST8GVo6IvZr2AAY5SZ8mTaS7IiKeztu+SPoyMLmljcvKZ+6XjXWsljnWXau2U+XKFh8lDfM5JyL+LWldYFZEzOz93ksPSXuRvggtIg2bKU9AIiLWaUnDrF+cOLZAHie4QkTcJOktwCWk8hqPAPtFxH29Bug99mhgH9Ib1HER8ZKk7YAZEfFEA3FfBbaIiEcqEsctgT9GxKA7NZnLoqybrz4WEfN627+VJM0FNoyIJyWdALwzIj4naTPg2ogY3cRjNVz+yTpPHk/7VB6CM6a3fdu1jFARJK0HXE8aWvAmYP383jkReFNEfLmlDRxAkh4j9Z4fV+/Yams9n6pugYi4s+zyi8CHmxFX0uak01hPkMp1nAq8RFordn2g371KZf4M7Asck6+HpKHAt+g+dTYoSBpBqmd5AKmUkoAFks4FvtWmq2XMB0qTKHYhfesHeKVse1O06ePvKJI+Veu+DSxR2lSlZDCvU/114CeDKUHsxenAdcCBQHkB/t8AF7akRa0zGjjfSWNrSToLOD4iXqrn/k4cly4TgTMi4ju5V7DkWmC/BmMfBfxJ0vuBEaTajRsBK5F6SweTnwIfItUsLJU+2gY4mZSEteOqGH8Bfijpr8AWwB55+/rAv+oJ6KXUClXrhIryCUptISJel/Q14KxWt6VNbAtsHRGLK8Z8PkVjY9o70R9IY3Mfb3VDBrm9SfmCE8d2JuleYMeImJXLM/Q4RqCB8gybk1Z3qfQs6Zte3SJiqqRNSN+aF5Bmb19B6lWoZ2JFJ/sMafm+KWXbHpf0AnAl7Zk4HkRKePcAvlpWEuXDpC8W9fBSagUpcFnSgXItsDPdPduDXbWVmd5B6vEfTKYAP5C0EXAfbyxN1Ba954NA9VlrNXLiOHCuJCVcUFx5hnnAylW2v4u0ekFDIuI54DuNxlkKzKX6aivPkP4GbSdPqPhYle2HNhB2F7yUmlV3A3CSpPeQyoPNLb9xkCUI1wGH0/2lPvIiCt8lLQQxmJyTfx9T5ba26z236jw5ZoD1tqZ0E2KfC7yV1CP2EqleZACTgRsj4rAG4y8HbEb105KD5oNA0rdJz+2+pQkxeaLMJOCBiPheK9vXk2bP7JQ0DfhERDzQ5KZahVzc/yBgHeBDEfEvSV8GnoiIthtjnGdV92RQzaqW9Dbgpnx1HdLiD+uRSrHtkMe5mw2Y8gmudd3fiePAUhrksoA0w3VaX/v3M/aKpDEk7wGWJ9XSGw3cAuwWEXN7uXtfsf+LtLJNtdnTg+2D4LfAjqSSEvfmzZuQevD/VL5vRHx8YFtXXREzO72U2sDIJUzOBs4HvgpslP92B5CGTIxtaQOtT/mL5eeB95G+dP8DuLSdKzHY0suJYwcqek1pSTtT9gYVEdc3IeYDpILPxwy2JcMqSap5JmRENDopqSkk/Y40DrE0s7NUTmkH4MKIWLfXAN1xKsfnro2XUiuUpHuAkyPisopSWJsC1zWzlJJZ0SR9hFSNY0PSe8lU4AcR8YeWNmwQaTRx9BjH1jgKOFVSIWtKR8SNpLVim2kt4OODPWmE9kkG+6lZMzu9fNrAeyfds/fLzQHadub6YE4QJH2h1n0j4pIi29JO8vCKs0jL6l6cN28PXCXpwIjwZKqB8XPg1Xrv7MSxNZq2prSkw4GzImJ+vtyjBpdzugXYAHisgRjWWg3P7PTyaS0xg1Q2qbIm4g606evRCQI/qbg+nPT6K439HELqoV9AWgBisPgWcHhE/Lhs2wWS7gL+B8/Cb4ikHYH5pQmLkvYllY17ADiiNKQoIg5s6Dg+VT3w8lJhPYqIi3u7vSLWE6QVXV7Ol3sJ27/lnPLSdCVrAd8DTqN6GYVBs/ZsgeWUCiPpMmBuRHwpn6Z4D/AyaeLU4xFRrYxTXzEfB94fES9XbH8TaYiElw9rAklHkeqwfhm4hjTBaS1SHbYJEVGZpLScpEdJNWV/XLH9YODgiFi/NS0beLnndQJwKFCqQLAV6b30hIj4XYuaNuByJ8lGleP78xjsByJiRGtaVl0uZv9z0hCttvySVk7SP0nvCZMlbUAag38B8AHglkYTxv8cx4mj9aSP9WbLDbbJMZUliYaRZptvR6preezAt6p3RczsLF+buGL7aOBfETG8sVZbiaQTgcNIZyog9VRNjIjjWteqnnVaglAkSQ8C+1eOaZe0DXBRRGzQmpYNvPyF4rSI+GnF9q8Bh7bjFwpJs4DN6x0POJDy0sCb5THQxwDbRsRHJW0FXBkRb2/GcXyqukWKWlO6ydYeiIPk2bk9rT7SFrOSy/V0ulbSN4Fe1+htlYiYkdelLp/ZeS51zOysWA7vI5LKT3UPJf0t2+V/uOPlMljHk1bp2ZD0t5va5jPZnyItdVpZOeJDvPGU+9JuLSrqWGavkYaKDCYTgTPz2axbSR0THyB9Fh7cyob14tfAp0htb3fltTB3Aa7Kl5+jekWUurjHsQWqrCn9rvwNYQKpTErNa0pLOr7WfSPif/vb1qJJOpV0Cucmqqw+0kkTUXJNxDsjoloR9pbKs6dvjYhFFduXIX0r/XM/YpXGaVXrjX4dmE4aTzNoTsEVJa8HP580A3Jqq9tTq1wq6EzS+MY3JAgRcW4LmzegJN2cL+4VEc/kbWsAPyN9Bu/Uqra1gqTdgSOAd+dNDwKnRsTk1rWqZ/kM02GkUmt38sZi9o3MHWgqSdeTPkenkE5RvzsiHstjHy+KiKZ0BjlxbAFJNwF/ju41pUvlNbYBLouImnut8ni7cmOA5Uj/PJBmzL4GTG9k7J2kPYF/R8R1+frxwHjSoNt9o85lByU9D3w9Ijp+tq6k/YDvRcQarW5LJUmLgdWrnFZeBXihnqEGeUzt+yOirvVOrTa50PoeEXF3q9vSH52WIBQlf6G8mrTwQ2nFqTWAh4FPNruerzVXs+cOFEnSxsAvSHnAaaWzY5J+DKwcEXs15ThOHAdexTiE8sRxLeChiBjZa4Ce4+4HfIFUkPmpvO0dwIWkU5J1z1iTNJU0BuW6stMMxwPjgOf600taEfdFYJtOevOU9JvKTcDqpHWbv9umPbtdwOjKsYyS1if1krZtWZfBLk+m+zywt5P0zpQXftiVlDyKVJro+maXYjOrRmnVsMUR8XqfO9cSz/+3Ay/3su0WEXdVJI7jgHMjoq5xL/mb0Scj4p6K7ZsBk/vTk1kl9lzSajdPSjoBeGdEfC7HvrbeIsR50P/rETGh3rYNtCoFwLuAF0nLOl7Xgib1qCzJ/Qhp5Zjy0k9DgY2BByNiXB2xiyz/VBhJH+3pNLqkYyLipIFuU2/yWYW1SZOwnuaNp8rabha/WUnuKFknj+OfTe8VKdr6C6ykUaRexrpXYRsIktahu37qg82e2OPJMa0xGfiOpM/k65F7G38AXNlA3NHAslW2jwRWbSAupHFWK+TLu9Bdb+uVsu01kfR/ZVeHAHtJ2pVUOqCyzM836mptgTpp3CWp5A6kXo5ZQPlEmIXAX4Hz6oxdOZh9GKnndR7wAqncSDu6VNJuEXFL+UalNciPANoqcaRDiq73J0EgFS+/HzgyIu7tZb+ONID1dTvBwcDsfPmgVjakXpK+TqpBuUa+/jSpmP1ZLW1YBaVlhy8APk13zVBJuhL4UkTM7vHO/TmOexwHngpaU1rSZFKpla+QlgcEeD9wDvBERHyygTZfTUpK/wocB6yVZ+qOBf6vPyUl8hjPmrTzwPGiv9U1Ux7gPbHob8q5WsCFwHkRcVVf+7eCpL2BM4CdSkmLpGOBw4GxEXFHb/cfSHny0oeAv0dFvcx2k0+pXxYRC9RHrVpgBLAbsGZEbF5862ojaVVSpYu7I2JBX/v3Eqew+roDpVnPxUAqos25rM3RpFnVf82btye9X5wUEd9vxnGaIZ8N25Y0/+DWvHk70lr3t0Qd9XqrHseJY+uoyWtKS3oLaRbjOGBx3jwEuJY07rHftfrKYr8d+CmpfMQZpfGSkk4HhrRjz2BRevpWR+otbtq3uiIMRLIr6b3A5RHxzmbHbhZJ3yB9GHwA+G/Sh8CH2ilpLJE0n1R5YXqr29JMktYE7oqI1dqgLSuQXtN7kF4b78zDh84mjeGe0Mr2DaROfC6KbLOkp4BvRcQvK7bvRUoc26YEm6SXScPV/lKxfQfgqohoSkmeIX3vYkWJiBsjYmJEnNJo0pjjvRgRu5EGYH+a9CJ6d0Ts1kjSmGM/HREfi4hNyyfZRMShjSSNkiblF33l9uUltevyU2eQeot3IvXCLks6ff8e4PQWtqtHklaQdAWprt7VpOESj0q6vNrz36AhpB70thUR/0c6RX8HqRzUf7Vj0pjdQyrWvlSJiH+1Q9KY/YB0GvJ9LDmc43fA7s0+mNKKJO2qsOdCUpekxT38zJV0T/5S1zZtJtUXrvbecDvt9z63LN3Dk8rNpHvxgIa5x7EF1HPtxSCNJZwGXBP9LMzciXopE7Mq6Zti243DHahvdc1UxCkMLVkIHLpnl3+dtIzhR+pvcXP1Ms7sUOAvpHXjgfYbcybpw8D3ge+Q2lk5OWZmK9rVG0nDgW+TZoO/g4p10qPNVprKY9Z2j4g7KiYslk571v3lKidCz0TElfn6JFL1i8eAj0fEw014CE1T8HPxNdLyi1ex5PKLnyQlf2sCB5B6+M5skzbfC/y/ymoZefjPpyJi03pjN5ukKcCrwD4R8VretjxpPfQVI2LXZhyn7T6UB4nPkN5Ml2fJeotzSbNz1wRekLRjX6cSKyaa9KrBnsGmfhBIejMp0RCwsqTywtRDSbOAn6+3vQUbkG91TfZx3pjs3ixpPOlNvJ6xL5WTNoI8u5w0yaSd9LQqxWJSQr1tvh6036Se3+ffv2bJCSdiyZUi2skJwGeBk4EfAd8kraDyOdIY6XazMtVf0yvQPeynXt8A9of/fLn8DGl4xKeBH5LWHm8nRT4XY4GjI+KCsm2TJN1OSqI/Ielh0uu15sSRYts8Abg8/+1uobuY/Y6kv2U7OYy0nv0zOeENYFNSbjG2WQdx4tgaPwT2JhXOfhr+M4ZwEmlB9d8Dl5PecD/RR6xNajxmo13Lzf4geCm3KUg1zSoFqYelHd0CnCCp8lvdd+nuzWs3TU92I6JjhrpEk1ZMaJG2nSDWiz2Br0bENZImksqBPaa0bvOupAl77eQO0per0lCT0vvlATT+ml6DtJoSwMeAKyLi8lxm6S893qt1inwudqH6l8o/AaVOkCn0/8tbYW2OiF8rrfV8GCnJL9Xh3DIi/tlI7GaLiPslvZOUX5Rqhv6cOpaW7etA/hngH9JSg++psn0z0govAFsDz7e6rRVtHpcvzwbWzZcPJHXj9zfejsAHSZNLds/XSz/bAG9r9WPupe2bkOrpzSK94d2cLz8NbNTq9vXQ5imkyTvLlW1bPm+b0ur2+Wfp+iGvw5wvPwtsni+vDbza6vZVae+2+X3tPNIYuTNJPedzgPc1GPv5ssd/N2npQUjjVue0+rEP8HPxJKkEU+X2I4En8+VNgWfbpc3+eeOPexxbYzTVe3lGkAbiQnqzWa7eAxRQqHQ03T2Dc4A35cvXkMam9EtE/AlA0trAU5Ff/Z0gIu4bkG91zXU48EeafApD0kdI9c1KM7Wnkuqb/aHhFhdI0mdJvR+rUTFJMCI+3pJG9UHS20jDRIaXb49+rDM+gJ4iDb95ijRmeyxpfOY2LDl5oS1ExK2StiUlMI+R/jf+QVrVqnJZ1/66DjhP0j9JyeIf8/aNSF/I20rBz8V3Sc/FzqTJJQFsSSo59ZW8z66kL+T9bfM2pDNhzW4zkkYAe9H9PvcA8MtowzJFeUz010ml+cZGxL8kfZlUku+GZhzDiWNrXA+ck8eXlQblb04qdzMlX9+EOt5UCixU2rQPAqUlC++OiC5gFWAVSVX3jYh/NNDmpsuzIX8OHBMR9RbOHnBFJLv5zegs4FJSGShI9c2uknRgNLDEZZEknUqaFHMTaYxxW39pyQnjL4AdSG0tjW0saccxjleRPrz/RqpC8EtJXyG9L53ayob1JCcYfdWfrMfXgRNJSf8e0T2Z6X3AL3u8VwsV9VxExKQ8XOFg0qllkdYw3z4i/pb3mdifmLnW6Xjg6ohoepslbUjqIFkRKCWhXwG+K2lcRDzY7GPWK5cIOhs4n/T6K81FGAocBTQlcWx5l+dg/CH1clxDOk37ev5ZTPomulreZydSXbn+xD2G1F3/nfxPswtpYO+rwP802OaTgW/ny3vkNj9BWn3kxH7G6ip7nF35sXdV+Vnc6r9VD+2fRVohoxmxbgTelC9/ARhRQHuHAb8iDy9oYtxHgYOqbD8YeKTVf6de2v086QO85W2psb2Xk97w35Vf39sBnyKtvLJrq9tX42PYitTr/dE67lv4a6TsWG8jDRl6X/lPq58/Pxd9tnUuMKag2FNI5ctWLNu2IvAb0nK7LX/8Ze26B/hcvjy79DlFOrvUtKFvLsfTQpI2ADYgf+uKiEcajDdghUolbU0aV/JI9LDuby/3HUM+PZ0v9yginmygmYWQdAHp79Wvb8Y9xFoArB1pFZ6qpYmaQdIs0jirphX8zm3fKCKmVWxfD3ggIkY061jNJOlF0imsaX3u3AaU1rb/SETcqbSs3xYR8UgeJnBcRGzd4iZWJemtpPeIyuEAERE/7Uecwl8jSkXrf053b3y5iAbLBymtqLQPaVWT4yIty7gdMCMiaj6ztJQ9F+sAx9f7XFTEvAH4SUT8upG29RD7NeD9EfFAxfZNgL9FxPLNPma9clvfHRFPVilLdH9EVFuSuN98qrqFItXvamYNrwErVBrptMLf6rxveTJ4LumU4c3AHRHRaOmEgfAUcKyk7YE7eWNdvf7MCHwIOElpGUYBe+bk4A0i4pI62wuplMunSMtmNctTpPFIlQnYh0iD4NvVuaRT9hNa3I5aLUuqQgBpFvxqwCOk8aTvaVWjeqO0rOP58J810st7KII0LKdWA/EaORf4F+kUZFOHL0janNRj/ARpXOOppL/nrsD6pNI8tVranouJ1P9clDsPmCjpHVSvddrIkKf5dI/pL7dSvq2dzCA9j5XvvzuQxn42hXscB0iut3h0RMztq/Zi1FlvsdmFSqsUeO5Rvd/0JJ1Imkn9ftJp71tJSeTNwO3tmEiqiWvP5kHoZ5AGza9IGi9a7UUZEbFivxq65HG+Qyon8ScaT3ZLMQ8gzV68mPR3K9U32wc4OCLObaC9K5EqD7yhXEnunZgaEbPqjP0T0gfUVOBe0rCL/6j39VdxjGGk8b+3R0RDHy65xt3xkUrbXE2anPZt0pCAT0QbLu0o6UnS/8X/RsSivvbvI1bhrxFJc4H3NnrWp4fYNwF/jojvVPQCbUNa27vmM0F+LnqM29XLzQ31kkq6mPT59BW6O0u2IZWUuj0i9qs3drNJOgrYD/gyaTjcR0ll8yYCEyLiJ005jhPHgZFfMLtHxL/z5R5FRF1123Kidzkp6SovVPpB0piuq/sZr7cXY7lmnL5YljR264P5Z0tgfiPJUqfJz/dbCzpV3bRktyLu7qS6bO/Omx4ETo2IyfXEK4u7AqmMy9iIuKVs+2akFSfWiIiXerp/H7ELef1VHOMzwGWktcsvajDWXsCwiLgoTyy7BlgVWAB8ISKuaLS9zVbE0Igct5DXiKS/AUdFATPUc4/gZjlBKk+W1gIeioi66qj6uVgibmFDniS9ifQl6GN0FxMfShr3uG9EvFJv7CLkzpjD6K7csgCYGBHHle3zdtLQgFo/45c8hhPHpUs+FXAY6YNcpLIBp0WbFSqtlMe9fBDYmTQxaE3S+JG6P8TzbLstqV7CpO7Tvup5De3yJSN/FREzetivp7hjSL2uB7Jk2YezmvnhoFSqiYiY02Ccq0mnI/9Q7xtQH/EvJdW5O6Bs20Rg/WjTkjmKmR2gAAAgAElEQVQlkn5HmlgwLSI+2MS4o0jjBdcjjROuK3kumqQfAw9HP5aNqzFu014jSqtXlWwGnAQcS5o5W9kLXfeyjnmM6m4RcVdFsjQOODci3lFnXD8XS8bu6f0+IuJn9cYti78e3V+Qp0ZE0079Npuk5Uj/E0NIbZ1Tcft/Evi64jtxbA31XEcuIqKv1WJ6irkhaSbyw/n6h0gz7x4ATmnktG/+FvOviDi7YvtXScW6e1p/u6+4PyElimNIYzFLBbVviwZqZEl6F/BbUsFhkb4pLkN6E1zQ4Gnf35LKznSRZrYCbJyPcxdp7M4oUomJu/sRdztSb9LzwG158zak/5GxEXFbT/etMf6hpJmta+RNM0grNJwedbwR5MTuk8ArwEXApGjihBNJY0nlSkZHxOuShpCKrB/UyCB4Sb/p5ea6X39l8VcjjRH7MHAtsF4jPR45ZlP/dkVTWqL0alJiUy35+N9q96shbtNeI7nHrvy5K00EqdzW6KnOc4G3kpane4k0LjVIPVY3RsRhdcb1c9Edt7D3+xy/o15/fSlP2uu6fwc+5o6nPurI1TtmQtJtwBkRcVnuin6YlIi9B/hZRBzdQJufAj4TEX+v2L4laQmtRsamvAj8mFSO6K5mvBAlXQP8m7QG83Okb9ErkQblHxsRU3q5e1+x/4dU3uBL0b3k4HKkAdr3kJa9ugR4S0Ts0o+4t5E+ZL9a6sHLydLZwMYRsW1v9+8j9imkWmensuSHzJHAeRFxVJ1xVyQVxt0P2AL4K6kX8oposBh6fuxPAd+ItOzXrqREcvWIeL33e/ca98KKTcNIf881gV9HxP71xs7xDyOVxNhK0nXAXyLihAbiFfK3K5Kkg0lj8V4CXqBickxE1DWpp5mvEUk7ll1di5TsV365HkJaAedi6pRfI78n/Y8tT3o/Gk0aTrRb1LlIg5+LJeIW+X7fca+/vjSaOLa87tBg/KGgOnKkF876+fJhwE358k7kpQwbiD2fKrULSSUV5jcQdz3SQN6fk96sZpG+OR5OAzXDSOsyb5wvvwJskC/vCNzb4HPxLKnkQeX2DclLZQHvBV7uZ9x5pXZWbH8XMK/BNs+s9j9HqsnZr3b2coyNSOuYz8vP+TnVnqd+xvwBqbAvpGT8J81oaw/H+iFpAHmjce4Gvp4vf4EGa1oOxN+ugOfyBeCwAuIW8hohJUmrVdm+Cg3UkyV9Kfk7qezazqRk4yjgv/xcNPW5KPL9vuNefzU8pv/UeKznZ4mltmzADCF9uDTbUNKpIUinwUvLvj1G4+V4niKdnq20A+n0YV0iYlpEnB8Re0fEmqS6by+REoZqpYVqJdJ6uZB6NEunGJ4mJauNGAWsXmX7W/NtkIqu97fc1SukUy2V1iZ9KWjUvT1sa/h9QGl1k0+QZvEtAv4fqQfvXklHNhD6EmCcpDVJa5rX3dtRg3OArzUSQNJ7SOOgLsubrgTelmeNNqKwv11BhpIKJDdbUa+RytV4SkbRQMmVSD3ja6eLcWNETIyIUyLi+npjlvFz0a3I93vovNdfoVzHsTWKqiN3P3BgHpi/C1A6Nb0G3XXg6nUO8KM8dunGvG0X0ooy/V6ruiSfWtmC1Cv6QdLM6pGksYK9zn7tw/2k0yGPk8ZOfkupYO5XeGPdwf66Crgglz64g+71Vk8h1UskX+9vOYvLyuKWl7f5Po0vTXYJaemzQyq2HwjUNXBcqeTMJ4D9SXXY/kl6Dn4ZeTC2pD1J/+911Y+MiAck3Udacu/piLi9njg12qAJMb4IXBMRLwNEKr91NbAv3ae5+qvpf7u+KNXDezrqn/R0IWkIQ11jGXvR1NeIukujBXCyUgHlkqGk13GjX/IvJr3vfLPBOJX8XHQr8v1+wF9/A6Ch4WBOHFvjTcB/5zFbzawj9y3SgPQjgYuje3H3j5NeTHWLiB9KWhX4P7pnrC0kjak8pYHQ/wZGkJKOm0njov4SdY51KXMiaQwNpNmBvyMloi8BezYY+6ukgdE/p/s1tAiYRHruIZWl+Uo/4x5F+uY8qSzu66RxOv/TQHshPcf/nSeclGqRbUVaVuzSsg+N/vz/PZvb+wvSkpbVvpVPIQ0/aMTPSONGv91gHGCJD8j/bCL1IH+Y9NzXG3coqT7kQRU3/Rz4haSDI2LhG+/ZZxuXAfbu6W9Xb3v7MB14QNLXo76yLMsBX85tbuZ7XLNfI5vk3yL1FJf/fRYC/6DxovnLA3vl9/tqxan9XGQNPBdFvt8X8d7ZI6W1vN8ZEUXmZ5UrAvXvzvl8tw0g9V5HLiJi5wZiDyWtqTmrbNtawGvRhJIukpYnjeUTVab51xFvHM1JFGs51puBWdGkf/r8XKxLei6mNesx5Ik25XFf6+MutcSstfe25v8/SfuQJsEUunpC/rsdDJwTEc81IV7lc1GaoHUjaWZ4XQWrJa1O+rLw/fIEMfeqHwNcEhFP1dnGnjT0ftHL8fclnVYcFxFb1XH/wt7jcvymvkbyhKlDIqLqKiwNxvZz0R270Oei4lhNeb8f6NeipIOAVSLiu43G6uUYa5LqONZVacWJo5mZmZnVZFAO7DQzMzOz/nPiaGZmZmY1ceLYBiSN77TYnRa3yNiOW3zsTotbZOxOi1tk7E6LW2TsTotbZOxOi1tk7CLiOnFsD4X9MxYYu9PiFhnbcYuP3Wlxi4zdaXGLjN1pcYuM3Wlxi4zdaXGLjO3E0czMzMxaw7OqCzJ86LKx7NDa1lVf2DWP4UOWrWnfrmWH9asdry+cy7Dhy/e9Yz/1J+6QObVXa1kY8xmukTXtG139q038OgsYxoia9tWQ2r9TFdXmdmgv/YgL/ftfjkW1V73pz3PRH0XFLTJ2O8TVMkP7FXth13yGD6nxf66r9s+kfv0v9+OzbiELGF7rczy0v89F7a+R/tRp7tdz3I+P/YUxj+Gqsb39/b9Y/BrDhy5X2879eO/s13Os2ksaLlw8j+FDa4y7qH+Vbvr1OdKP/+XXYz7Daow7O2a+FBFv6Ws/FwAvyLJDV2Tbt36+6XHnbVhtpbvmiCEN1QTt0chbHiokbtfs2YXEBRiybI1vZv3UNX9BIXGHLF9MezWyxg+iOix+qdHFjAZYkV+y+/Hh1Q6GvmnlwmJHQa8RFtdVsq5PWmGFQuIC0FVMmwt7Lt5c3P8F8woqF7tMMWlQ18xG1z3oWSx8ve+d6jBl4S+erGU/n6o2MzMzs5o4cTQzMzOzmjhxNDMzM7OaOHE0MzMzs5q0NHGUdLOkHw/QsSZIun8gjmVmZma2NGr1rOpPAcVMD3qjicCZA3QsMzMzs6VOSxPHiJhZ9DEkDSHVq5wDzCn6eGZmZmZLq0JPVedT0WdJOknSS5JekDQxJ3NvOFUtabSk30iaJ+lJSftJul/ShLJ9VpJ0bo41W9KfJG1Rdvu+kuZI2i2fml4IvLvyVLWk90u6LrfrVUl/lbRNRftD0nhJV0iaK+lxSXsX+JSZmZmZta2BGOO4F7AI2BY4CDgU+GwP+14MjAF2Bj4B7J2vAyBJwO+BNYCPAu8F/gzcKKm8MvZI4FjgAGBDoFpRyxWAnwHbA1sCdwN/kLRqxX7HA5OBTYFfAZMkjcHMzMxskBmIxHFqRBwfEY9ExOXATcAulTtJ2gAYCxwQEbdFxN3AvkD5khg7AZsBe0TE7RExLSKOAx4H9inbbyhwcETcko/7hiVGIuLGiPhZRDwYEQ8BBwPzgXEVu/4sIn4eEdOA40hJ8PbVHmjunbxT0p0Lu+bV8NSYmZmZdY6BGON4b8X1GcBqVfZ7F9AF3FnaEBH/kjSjbJ/NSYnki1pyia6RwLpl1xeRehB7JGk14ARSMjqalGwuC7yjp/ZHxCJJL/bQfiLiXOBcgJWGj/Yi4GZmZrZUGYjEsXLWdFC9p7OWxVqHAM9Tvcfv1bLLCyKir8U4LyYljIcB04EFwA3A8Ir9am2/mZmZ2VKt1eV4yj1ISsg2B/4OIOntwNvK9vkHKdnriojHGzzeB4BvRMTv87FGA6v3fhczMzOzwattes4i4mHgWuBsSVtL2gy4EHiN1MsHcD1wCzBZ0oclrS1pG0nflVR13GEvHgH2lrShpPcDl5FmYJuZmZlZFW2TOGb7Ak8DNwO/AS4FXiBNWiEiAtgNuBE4D3gYuBzYgDR2sj/2B0YBd5GSxkmkU9ZmZmZmVoVSLtaecmmcGcDnI+LKVrenP1YaPjq2fevnmx533obFnU2PIbUMM+2/kbc8VEjcrtlvmCzfNEOWW67vnerQNX9BIXGHLF9MezVyZCFxARa/9FJhsQtR5HulinntFWXom1cuLHYU9BphcV/D3uujFVYoJC4AXcW0ubDnosD/C+bNLybuMsWM2OuaOauQuACxsJgF96Ys/MVdEbFFX/u10xhHJO1Mqq94H2nm8onAS8A1rWyXmZmZmbVZ4ggMA74HrEMa2/h3YIeImNvSVpmZmZlZeyWOEXEtaYKMmZmZmbWZtkoclyoSLDO06WGf/HBxf7I3PVjMOKsRL1bWVG+OIQ9NLyQuwOvv36CQuCOemllI3BhW0P/Fy8WN00HFzM1TQc9FLCyu6MKQESMKids1v5hxYV2vzikkLkAUNP6O6CokrBYtKiQuFPhcFGRIUeNTgVhc0N+vA98voqu1c1PabVa1mZmZmbUpJ45mZmZmVhMnjmZmZmZWEyeOZmZmZlYTJ45mZmZmVhMnjmZmZmZWk6UqcZQUkvZoUqy1crw+l98xMzMzGwyWqsTRzMzMzIrjxNHMzMzMatJxiaOSIyQ9KmmBpKclndzDvt+X9LCkeZKmSzpF0siy29eUNFnSTEmvSXpI0ucqwoyRNCXfPlXSroU+QDMzM7M21YlLDp4EHAgcDvwZeAvw3h72nQvsDzwDbAicDSwAjsu3nwWMBHYCXgWqrTN3IvBN4GvAscBlksZERHFrbpmZmZm1oY5KHCWNAg4DDo2ISXnzNOC2avtHxAllV6dLOgk4ku7EcQxwZUTck68/USXMjyLit/n4xwBfADYD/trIYzEzMzPrNB2VOJJ6DUcAN9Syc55hfSiwHjAKGJp/Ss4AzpY0Lse8KiLuqghzb9nlGfn3aj0cbzwwHmDk0BVqaaKZmZlZx+i0MY6qeUdpa+Ay4FrgY6TT2ccCw0r7RMQFwNrAhcD6wK2SJlSEer1s/8gXqz5vEXFuRGwREVsMH7pcrU01MzMz6widljhOJY1R3KWGfbcDnomIEyLijoh4lHRqegkR8XRO+PYEjif3GJqZmZnZkjrqVHVEzJZ0BnCypAWkyTGrAJtHxE8rdn8EWEPSXqQxkGOBz5fvkGP9Me+7IjCOlJyamZmZWYWOShyzo4FZpAkubweeBy6p3CkifivpVOB0YFngOlKP4llluw0BzgTWBGaTxjkeUWTjzczMzDpVxyWOEdEFfD//VN6miutHkxLNcj8tu/3gXo4znSpjKiuPYWZmZjZYdNoYRzMzMzNrESeOZmZmZlYTJ45mZmZmVhMnjmZmZmZWk46bHNMxurqIufOaHnaDs15oesySrhWXLSTukKdfLCTu4tdeKyQuwIubjiwk7hr3zy4k7jp/LGbp9Mc/PKqQuABEVyFhtUxBb2uLFxcTF+heW8CK+r+goOdYI0YUEhcg5s0vJnBXMf/LXQsWFBK3SBpZ3N+vMEW9RmrkHkczMzMzq4kTRzMzMzOriRNHMzMzM6uJE0czMzMzq4kTRzMzMzOryVKVOEoKSXs0KdZaOd4WzYhnZmZm1umWqsTRzMzMzIrjxNHMzMzMatJxiaOSIyQ9KmmBpKclndzDvt+X9LCkeZKmSzpF0siy29eUNFnSTEmvSXpI0ucqwoyRNCXfPlXSroU+QDMzM7M21Ykrx5wEHAgcDvwZeAvw3h72nQvsDzwDbAicDSwAjsu3nwWMBHYCXgU2qBLjROCbwNeAY4HLJI2JiGKW6jAzMzNrUx2VOEoaBRwGHBoRk/LmacBt1faPiBPKrk6XdBJwJN2J4xjgyoi4J19/okqYH0XEb/PxjwG+AGwG/LWRx2JmZmbWaToqcST1Go4Abqhl5zzD+lBgPWAUMDT/lJwBnC1pXI55VUTcVRHm3rLLM/Lv1Xo43nhgPMDIIQWu8WtmZmbWAp02xlE17yhtDVwGXAt8jHQ6+1hgWGmfiLgAWBu4EFgfuFXShIpQr5ftH/li1ectIs6NiC0iYovhQ5attalmZmZmHaHTEseppDGKu9Sw73bAMxFxQkTcERGPkk5NLyEins4J357A8eQeQzMzMzNbUkedqo6I2ZLOAE6WtIA0OWYVYPOI+GnF7o8Aa0jaizQGcizw+fIdcqw/5n1XBMaRklMzMzMzq9BRiWN2NDCLNMHl7cDzwCWVO0XEbyWdCpwOLAtcR+pRPKtstyHAmcCawGzSOMcjimy8mZmZWafquMQxIrqA7+efyttUcf1oUqJZ7qdltx/cy3GmU2VMZeUxzMzMzAaLThvjaGZmZmYt4sTRzMzMzGrixNHMzMzMauLE0czMzMxq0nGTYzpHQHQ1PWrX9KebHrNEI0cUEnfx3NcKict/6rE335seX1RI3FiwsJC4P17j74XEHbveFwqJC8DMfxcSdshqqxYSd/GM5wqJCzB0jdULibvo8emFxGVIgXMEVVR/RvPfjwGGjH5LIXEB4qli3u8L+GhKCvvbAV2LCwkb8+YVE7eruM+nIj/7auEeRzMzMzOriRNHMzMzM6uJE0czMzMzq4kTRzMzMzOriRNHMzMzM6uJE0czMzMzq8mgSBwlTZB0f6vbYWZmZtbJWpY4SrpI0u+aHHMtSSFpi4qbJgI7NvNYZmZmZoNN0wuASxoeEcVUOa5TRMwB5rS6HWZmZmadrM8eR0k3Szpb0hmSZuWfU6VUIl7S9HwqeJKkfwOX5u2bSLpe0jxJM3MP40r5tgnAF4GP5B7CkPTBfNsaki4rO9bvJb2zrD1rSpqcY74m6SFJn8s3P5F/35Fj3lw6Xvmp6lJvp6RDJD2Tj3OhpOXK9lle0iWS5kh6XtLR+T4X1flcm5mZmXW0Wk9V75X33QY4ABgPHFp2++HAQ8AWwDE5AbuG1Mu3JbA7sC0wKe8/EbgcuB5YPf/cmu93EzCfdGp5G+BZ4PqypO4sYDlgJ2Cj3I7S2mVb5t/jcsxP9fKYtgc2Bv4L+Gxu4yFlt/8wt2F3YGdg03wfMzMzs0Gp1lPVzwLfiIgAHpK0PilZPC3f/qeIOKW0s6SvAKOAfSJidt42HrhJ0noRMU3SPGBBRDxXdr+9AQH75WMh6QDgBeCjpGRzDHBlRNyT71bqZQR4Mf9+uTxuD14FDoyIRcCDkq4AdgFOljQK2B/4QkRMye34EtDrwqH5MY4HGDlkVB+HNzMzM+sstfY4/q2UyGW3AWtIWjFfv7Ni/3cD95aSxuxW0irzG/ZynM2BtYHZ+RTxHOAVYGVg3bzPGcCxkm6T9D1Jm9f4GCpNzUljyQxgtXx5XWAYcHvpxoiYC/Q6Mzsizo2ILSJii+FDRtbZLDMzM7P21KzJMXMrrguIajv2sh1SIns38Lkqt80EiIgLJF0L7EY6zXyrpJMjYkK/WgyvV2lXKZFWDW01MzMzG1Rq7XHcSpLKrm8NzIiIV3vYfyqwqaQVyrZtm4/3YL6+EBhacb9/AOsBL0XEtIqfmaWdIuLp3Lu3J3A8+fRwjkmVuP01jZRYlsZMksdYbtxgXDMzM7OOVWvi+DbgdEkbSNoD+Cbwo172v5TUC3lJnl29A3AO8OuImJb3mQ5snGOuKmlYvt/zwGRJO0paW9IOkn5YmlmdZ3ePk7SOpM1IE2Gm5pgvAPOAsZJGl2Zx91cu3zMJ+IGkXSRtCJxPer7cC2lmZmaDUq2J46WkXry/A+cBF9BL4hgRrwFjgRVJ4wQnk8ZF7l+223mk3sc7SZNatsv32wF4HLiCNFP7YtIYx1llbT6TlCxOISWaX8zHXQR8A/gyaczi5BofXzVHAn8BfkOa6X1vbuv8BmKamZmZdaxaxzguioiDgIMqb4iItardISLuI81SrioiXgQ+VGX788B+vdzv4N4aGhHnk3oHy7dNACaUXd+3yv0q95kD7JN/kDSCVPrnD70d38zMzGxp1fSVY5YWkt5Lmh1+O7AC8K38+1etbJeZmZlZqzhx7N3hwAbAItJs7x0iotdajmZmZmZLqz4Tx4j44AC0o+1ExD9JK+GYmZmZGbVPjjEzMzOzQc6nqosUza/cE4sXNz3mf2LPfa2YwF3Ftbkoo+6eUUjcxQsX9r1THU6buU4hcee+fdlC4gKMulN971SH11d/UyFxl3l1dt871WnhGisXEnfI49MLiduRVEw/yfy1VikkLsCwpwoaGaViXnudqMjP1KWVexzNzMzMrCZOHM3MzMysJk4czczMzKwmThzNzMzMrCZOHM3MzMysJoMicZQ0QdL9rW6HmZmZWSdrWeIo6SJJv2tyzLUkhaTKwt0TgR2beSwzMzOzwabpdRwlDY+IYorV1Ski5gBzWt0OMzMzs07WZ4+jpJslnS3pDEmz8s+pUqqmKml6PhU8SdK/gUvz9k0kXS9pnqSZuYdxpXzbBOCLwEdyD2FI+mC+bQ1Jl5Ud6/eS3lnWnjUlTc4xX5P0kKTP5ZufyL/vyDFvLh2v/FR1qbdT0iGSnsnHuVDScmX7LC/pEklzJD0v6eh8n4vqfK7NzMzMOlqtp6r3yvtuAxwAjAcOLbv9cOAh0trOx+QE7BpSL9+WwO7AtsCkvP9E4HLgemD1/HNrvt9NwHzSqeVtgGeB68uSurOA5YCdgI1yO/6db9sy/x6XY36ql8e0PbAx8F/AZ3MbDym7/Ye5DbsDOwOb5vuYmZmZDUq1nqp+FvhGRATwkKT1Scniafn2P0XEKaWdJX0FGAXsExGz87bxwE2S1ouIaZLmAQsi4rmy++0NCNgvHwtJBwAvAB8lJZtjgCsj4p58t1IvI8CL+ffL5XF78CpwYEQsAh6UdAWwC3CypFHA/sAXImJKbseXgF7Xf8qPcTzAyCGj+ji8mZmZWWeptcfxb6VELrsNWEPSivn6nRX7vxu4t5Q0ZrcCXcCGvRxnc2BtYHY+RTwHeAVYGVg373MGcKyk2yR9T9LmNT6GSlNz0lgyA1gtX14XGAbcXroxIuYCvc7MjohzI2KLiNhi+JCRdTbLzMzMrD01a3LM3IrrAqLajr1sh5TI3g18rsptMwEi4gJJ1wK7kU4z3yrp5IiY0K8Ww+tV2lVKpFW2zczMzMyovcdxK0kqu741MCMiXu1h/6nAppJWKNu2bT7eg/n6QmBoxf3+AawHvBQR0yp+ZpZ2ioinc+/ensDx5NPDOSZV4vbXNFJiWRozSR5juXGDcc3MzMw6Vq2J49uA0yVtIGkP4JvAj3rZ/1JSL+QleXb1DsA5wK8jYlreZzqwcY65qqRh+X7PA5Ml7ShpbUk7SPphaWZ1nt09TtI6kjYjTYSZmmO+AMwDxkoaXZrF3V+5fM8k4AeSdpG0IXA+6flyL6SZmZkNSrUmjpeSevH+DpwHXEAviWNEvAaMBVYkjROcTBoXuX/ZbueReh/vJE1q2S7fbwfgceAK0kzti0ljHGeVtflMUrI4hZRofjEfdxHwDeDLpDGLk2t8fNUcCfwF+A1ppve9ua3zG4hpZmZm1rFqHeO4KCIOAg6qvCEi1qp2h4i4jzRLuaqIeBH4UJXtzwP79XK/g3traEScT+odLN82AZhQdn3fKver3GcOsE/+QdIIUumfP/R2fDMzM7OlVdNXjllaSHovaXb47cAKwLfy71+1sl1mZmZmreLEsXeHAxsAi0izvXeIiF5rOZqZmZktrfpMHCPigwPQjrYTEf8krYRjZmZmZtQ+OcbMzMzMBjmfqi5KVxALK2uMN0F0NT+mvUHXzFl971SHeH1R3zvV4aJpWxUSd8QKxX23HDW00XKr1S1arpi3tWErrdj3TnUasqCY/4vCdBVXlUxD1PdO9QUuJOzsNYcXEhfgzQW1GRYXErWwvx3+6Gsn7nE0MzMzs5o4cTQzMzOzmjhxNDMzM7OaOHE0MzMzs5o4cTQzMzOzmixViaOkkLRHk2KtleO5lqOZmZkZS1niaGZmZmbFceJoZmZmZjXpuMRRyRGSHpW0QNLTkk7uYd/vS3pY0jxJ0yWdImlk2e1rSposaaak1yQ9JOlzFWHGSJqSb58qaddCH6CZmZlZm+rElWNOAg4EDgf+DLwFeG8P+84F9geeATYEzgYWAMfl288CRgI7Aa8CG1SJcSLwTeBrwLHAZZLGRMScZjwYMzMzs07RUYmjpFHAYcChETEpb54G3FZt/4g4oezqdEknAUfSnTiOAa6MiHvy9SeqhPlRRPw2H/8Y4AvAZsBfG3ksZmZmZp2moxJHUq/hCOCGWnbOM6wPBdYDRgFD80/JGcDZksblmFdFxF0VYe4tuzwj/16th+ONB8YDjNTytTTRzMzMrGN02hjHmldQl7Q1cBlwLfAx0unsY4FhpX0i4gJgbeBCYH3gVkkTKkK9XrZ/5ItVn7eIODcitoiILYZ3D6U0MzMzWyp0WuI4lTRGcZca9t0OeCYiToiIOyLiUdKp6SVExNM54dsTOJ7cY2hmZmZmS+qoU9URMVvSGcDJkhaQJsesAmweET+t2P0RYA1Je5HGQI4FPl++Q471x7zvisA4UnJqZmZmZhU6KnHMjgZmkSa4vB14HrikcqeI+K2kU4HTgWWB60g9imeV7TYEOBNYE5hNGud4RJGNNzMzM+tUHZc4RkQX8P38U3mbKq4fTUo0y/207PaDeznOdKqMqaw8hpmZmdlg0WljHM3MzMysRZw4mpmZmVlNnDiamZmZWU2cOJqZmZlZTZw4mpmZmVlNOm5WdUfp6mp+TBWY63ctLi52h4lFiwoKXMD/BLD49pULifv6CoWEBUAqpkDBMnOL+dvFMkP73qlOQ3n3iegAAAOLSURBVOcsKCTu4oKe41hc3HuFhhTTZg0t5r1z/qoFFtoo7Lko6H+5yM+ngv6XC2tzQe/1AAwp6O9X48vaPY5mZmZmVhMnjmZmZmZWEyeOZmZmZlYTJ45mZmZmVhMnjmZmZmZWEyeOZmZmZlYTJ45mZmZmVhMnjmZmZmZWk45PHJUcJekxSfMk3Sdp74p9NpF0fb59pqSLJK1UcfsNkl6VNFvSPZJ2GvhHY2ZmZta+loaVY74H7AF8HXgY2AY4T9KsiPi9pOWAa4A7gC2BNwPnAZOAT+cYvwDuybcvAjYB5g/kgzAzMzNrdx2dOEpaHjgc+FBE/CVvfkLSlqRE8vfAXsAoYJ+ImJ3vNx64SdJ6ETENGANMjIiHcoxpdbZnPDAeYKSWr/NRmZmZmbWnjk4cgQ2BkcA1kqJs+zBger78buDeUtKY3Qp05ftPA04Dzpf0ReAG4MqyJLJmEXEucC7ASkNXjT52NzMzM+sonT7GsdT+jwGblf1sBHwo3yagpyQuACJiAimJvBrYFrhX0v7FNNnMzMysM3V64jgVWACMiYhpFT9Plu2zqaQVyu63LemxP1jaEBGPRsT/RcRHgAuALw/QYzAzMzPrCB19qjoiZkuaCEyUJODPpPGMWwNd+dTxpcB3gUskHQ+sDJwD/DoipklaFpgIXEE6vT0a+ADw94F+PGZmZmbtrKMTx+w44HngSOCnwKvA3cApABHxmqSxwOnA7aTZ0pOBQ/L9F5OSyYuBtwIvA7/L8czMzMz+f3t3bNNAEARQdLYHSE0TdEAXFEMTJBY1kVEBCYEhQkQIaQnOIbYXdAM76D3polv9vXBk39rslR8ce+89Im7316E1DxFxdeDee0Rc5zwdAMD/Uf0dRwAAfsn0g2NrbdNaeztybRL33h7Zd5u1LwDAjCp8Vf0Uy0/sHLuf5SaWgzNfeU3cFwBgOtMPjr33j/jhP7mssPcuInZ/sTcAwGzacraEtbXWniPi8eTCxVlEvCQ9Sla7WjezrZvfrtbNbFfrZrardTPb1bqZ7WrdzPZ3uhe99/NTiwyOE2it3ffeLyu1q3Uz27r57WrdzHa1bma7WjezXa2b2a7WzWxndKc/HAMAwBwMjgAADDE4zuGuYLtaN7Otm9+u1s1sV+tmtqt1M9vVupntat3M9upd7zgCADDEJ44AAAwxOAIAMMTgCADAEIMjAABDDI4AAAz5BGG2Kzowz3tiAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 720x720 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Bahdanau Attention\n",
    "for i in range(1):\n",
    "    print(i)\n",
    "    preds, attention, encoder_embedding = generate(x.transpose(1,0)[i].unsqueeze(1), y.transpose(1,0)[i].unsqueeze(1), learn_bahdanau)\n",
    "    "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}