4_3_tensorflow_eager_execution_tutorial.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "##### Copyright 2018 The TensorFlow Authors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "#@title Licensed under the Apache License, Version 2.0 (the \"License\");\n",
    "# you may not use this file except in compliance with the License.\n",
    "# You may obtain a copy of the License at\n",
    "#\n",
    "# https://www.apache.org/licenses/LICENSE-2.0\n",
    "#\n",
    "# Unless required by applicable law or agreed to in writing, software\n",
    "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
    "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
    "# See the License for the specific language governing permissions and\n",
    "# limitations under the License."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup and basic usage\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To start eager execution, add `tf.enable_eager_execution()` to the beginning of\n",
    "the program or console session. Do not add this operation to other modules that\n",
    "the program calls."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "from __future__ import absolute_import, division, print_function, unicode_literals\n",
    "\n",
    "import tensorflow as tf\n",
    "\n",
    "tf.enable_eager_execution()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now you can run TensorFlow operations and the results will return immediately::"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf.executing_eagerly()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "привет, [[4.]]\n"
     ]
    }
   ],
   "source": [
    "x = [[2.]]\n",
    "m = tf.matmul(x, x)\n",
    "print(\"привет, {}\".format(m))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Enabling eager execution changes how TensorFlow operations behave—now they\n",
    "immediately evaluate and return their values to Python. `tf.Tensor` objects\n",
    "reference concrete values instead of symbolic handles to nodes in a computational\n",
    "graph. Since there isn't a computational graph to build and run later in a\n",
    "session, it's easy to inspect results using `print()` or a debugger. Evaluating,\n",
    "printing, and checking tensor values does not break the flow for computing\n",
    "gradients.\n",
    "\n",
    "Eager execution works nicely with [NumPy](http://www.numpy.org/). NumPy\n",
    "operations accept `tf.Tensor` arguments. TensorFlow\n",
    "[math operations](https://www.tensorflow.org/api_guides/python/math_ops) convert\n",
    "Python objects and NumPy arrays to `tf.Tensor` objects. The\n",
    "`tf.Tensor.numpy` method returns the object's value as a NumPy `ndarray`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tf.Tensor(\n[[1 2]\n [3 4]], shape=(2, 2), dtype=int32)\n"
     ]
    }
   ],
   "source": [
    "a = tf.constant([[1, 2],\n",
    "                 [3, 4]])\n",
    "print(a)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tf.Tensor(\n[[2 3]\n [4 5]], shape=(2, 2), dtype=int32)\n"
     ]
    }
   ],
   "source": [
    "# Broadcasting support\n",
    "b = tf.add(a, 1)\n",
    "print(b)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tf.Tensor(\n[[ 2  6]\n [12 20]], shape=(2, 2), dtype=int32)\n"
     ]
    }
   ],
   "source": [
    "# Operator overloading is supported\n",
    "print(a * b)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[ 2  6]\n [12 20]]\n"
     ]
    }
   ],
   "source": [
    "# Use NumPy values\n",
    "import numpy as np\n",
    "\n",
    "c = np.multiply(a, b)\n",
    "print(c)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[1 2]\n [3 4]]\n"
     ]
    }
   ],
   "source": [
    "# Obtain numpy value from a tensor:\n",
    "print(a.numpy())\n",
    "# => [[1 2]\n",
    "#     [3 4]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `tf.contrib.eager` module contains symbols available to both eager and graph execution\n",
    "environments and is useful for writing code to [work with graphs](#work_with_graphs):"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dynamic control flow\n",
    "\n",
    "A major benefit of eager execution is that all the functionality of the host\n",
    "language is available while your model is executing. So, for example,\n",
    "it is easy to write [fizzbuzz](https://en.wikipedia.org/wiki/Fizz_buzz):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "def fizzbuzz(max_num):\n",
    "  counter = tf.constant(0)\n",
    "  max_num = tf.convert_to_tensor(max_num)\n",
    "  for num in range(1, max_num.numpy()+1):\n",
    "    num = tf.constant(num)\n",
    "    if int(num % 3) == 0 and int(num % 5) == 0:\n",
    "      print('FizzBuzz')\n",
    "    elif int(num % 3) == 0:\n",
    "      print('Fizz')\n",
    "    elif int(num % 5) == 0:\n",
    "      print('Buzz')\n",
    "    else:\n",
    "      print(num.numpy())\n",
    "    counter += 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1\n2\nFizz\n4\nBuzz\nFizz\n7\n8\nFizz\nBuzz\n11\nFizz\n13\n14\nFizzBuzz\n"
     ]
    }
   ],
   "source": [
    "fizzbuzz(15)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This has conditionals that depend on tensor values and it prints these values\n",
    "at runtime."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Build a model\n",
    "\n",
    "Many machine learning models are represented by composing layers. When\n",
    "using TensorFlow with eager execution you can either write your own layers or\n",
    "use a layer provided in the `tf.keras.layers` package.\n",
    "\n",
    "While you can use any Python object to represent a layer,\n",
    "TensorFlow has `tf.keras.layers.Layer` as a convenient base class. Inherit from\n",
    "it to implement your own layer:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "class MySimpleLayer(tf.keras.layers.Layer):\n",
    "  def __init__(self, output_units):\n",
    "    super(MySimpleLayer, self).__init__()\n",
    "    self.output_units = output_units\n",
    "\n",
    "  def build(self, input_shape):\n",
    "    # The build method gets called the first time your layer is used.\n",
    "    # Creating variables on build() allows you to make their shape depend\n",
    "    # on the input shape and hence removes the need for the user to specify\n",
    "    # full shapes. It is possible to create variables during __init__() if\n",
    "    # you already know their full shapes.\n",
    "    self.kernel = self.add_variable(\n",
    "      \"kernel\", [input_shape[-1], self.output_units])\n",
    "\n",
    "  def call(self, input):\n",
    "    # Override call() instead of __call__ so we can perform some bookkeeping.\n",
    "    return tf.matmul(input, self.kernel)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use `tf.keras.layers.Dense` layer instead  of `MySimpleLayer` above as it has\n",
    "a superset of its functionality (it can also add a bias).\n",
    "\n",
    "When composing layers into models you can use `tf.keras.Sequential` to represent\n",
    "models which are a linear stack of layers. It is easy to use for basic models:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From D:\\Tools\\Python\\Anaconda\\lib\\site-packages\\tensorflow\\python\\ops\\resource_variable_ops.py:642: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.\nInstructions for updating:\nColocations handled automatically by placer.\n"
     ]
    }
   ],
   "source": [
    "model = tf.keras.Sequential([\n",
    "  tf.keras.layers.Dense(10, input_shape=(784,)),  # must declare input shape\n",
    "  tf.keras.layers.Dense(10)\n",
    "])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Alternatively, organize models in classes by inheriting from `tf.keras.Model`.\n",
    "This is a container for layers that is a layer itself, allowing `tf.keras.Model`\n",
    "objects to contain other `tf.keras.Model` objects."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "class MNISTModel(tf.keras.Model):\n",
    "  def __init__(self):\n",
    "    super(MNISTModel, self).__init__()\n",
    "    self.dense1 = tf.keras.layers.Dense(units=10)\n",
    "    self.dense2 = tf.keras.layers.Dense(units=10)\n",
    "\n",
    "  def call(self, input):\n",
    "    \"\"\"Run the model.\"\"\"\n",
    "    result = self.dense1(input)\n",
    "    result = self.dense2(result)\n",
    "    result = self.dense2(result)  # reuse variables from dense2 layer\n",
    "    return result\n",
    "\n",
    "model = MNISTModel()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It's not required to set an input shape for the `tf.keras.Model` class since\n",
    "the parameters are set the first time input is passed to the layer.\n",
    "\n",
    "`tf.keras.layers` classes create and contain their own model variables that\n",
    "are tied to the lifetime of their layer objects. To share layer variables, share\n",
    "their objects."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Eager training"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Computing gradients\n",
    "\n",
    "[Automatic differentiation](https://en.wikipedia.org/wiki/Automatic_differentiation)\n",
    "is useful for implementing machine learning algorithms such as\n",
    "[backpropagation](https://en.wikipedia.org/wiki/Backpropagation) for training\n",
    "neural networks. During eager execution, use `tf.GradientTape` to trace\n",
    "operations for computing gradients later.\n",
    "\n",
    "`tf.GradientTape` is an opt-in feature to provide maximal performance when\n",
    "not tracing. Since different operations can occur during each call, all\n",
    "forward-pass operations get recorded to a \"tape\". To compute the gradient, play\n",
    "the tape backwards and then discard. A particular `tf.GradientTape` can only\n",
    "compute one gradient; subsequent calls throw a runtime error."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tf.Tensor([[2.]], shape=(1, 1), dtype=float32)\n"
     ]
    }
   ],
   "source": [
    "w = tf.Variable([[1.0]])\n",
    "with tf.GradientTape() as tape:\n",
    "  loss = w * w\n",
    "\n",
    "grad = tape.gradient(loss, w)\n",
    "print(grad)  # => tf.Tensor([[ 2.]], shape=(1, 1), dtype=float32)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Train a model\n",
    "\n",
    "The following example creates a multi-layer model that classifies the standard\n",
    "MNIST handwritten digits. It demonstrates the optimizer and layer APIs to build\n",
    "trainable graphs in an eager execution environment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Fetch and format the mnist data\n",
    "(mnist_images, mnist_labels), _ = tf.keras.datasets.mnist.load_data()\n",
    "dataset = tf.data.Dataset.from_tensor_slices(\n",
    "  (tf.cast(mnist_images[...,tf.newaxis]/255, tf.float32),\n",
    "   tf.cast(mnist_labels,tf.int64)))\n",
    "dataset = dataset.shuffle(1000).batch(32)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Build the model\n",
    "mnist_model = tf.keras.Sequential([\n",
    "  tf.keras.layers.Conv2D(16,[3,3], activation='relu'),\n",
    "  tf.keras.layers.Conv2D(16,[3,3], activation='relu'),\n",
    "  tf.keras.layers.GlobalAveragePooling2D(),\n",
    "  tf.keras.layers.Dense(10)\n",
    "])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "Even without training, call the model and inspect the output in eager execution:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Logits:  [[ 0.01911251 -0.01804457  0.00284918 -0.02342504 -0.03042551  0.00880419\n  -0.00653339  0.07787658 -0.00415065 -0.01776585]]\n"
     ]
    }
   ],
   "source": [
    "for images,labels in dataset.take(1):\n",
    "  print(\"Logits: \", mnist_model(images[0:1]).numpy())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "While keras models have a builtin training loop (using the `fit` method), sometimes you need more customization. Here's an example, of a training loop implemented with eager:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [],
   "source": [
    "optimizer = tf.train.AdamOptimizer()\n",
    "\n",
    "loss_history = []"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      ".WARNING:tensorflow:From D:\\Tools\\Python\\Anaconda\\lib\\site-packages\\tensorflow\\python\\ops\\losses\\losses_impl.py:209: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\nInstructions for updating:\nUse tf.cast instead.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "."
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "."
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "."
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "."
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "."
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "."
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "."
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "."
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "."
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "."
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "."
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "."
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "."
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "."
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "."
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "."
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "."
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "."
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "."
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "."
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "."
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "."
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "."
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "."
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "."
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "."
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "."
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "."
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "."
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "."
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "."
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "."
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "."
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "."
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "."
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "."
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "."
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "."
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "."
     ]
    }
   ],
   "source": [
    "for (batch, (images, labels)) in enumerate(dataset.take(400)):\n",
    "  if batch % 10 == 0:\n",
    "    print('.', end='')\n",
    "  with tf.GradientTape() as tape:\n",
    "    logits = mnist_model(images, training=True)\n",
    "    loss_value = tf.losses.sparse_softmax_cross_entropy(labels, logits)\n",
    "\n",
    "  loss_history.append(loss_value.numpy())\n",
    "  grads = tape.gradient(loss_value, mnist_model.trainable_variables)\n",
    "  optimizer.apply_gradients(zip(grads, mnist_model.trainable_variables),\n",
    "                            global_step=tf.train.get_or_create_global_step())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEKCAYAAAD9xUlFAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJztnXd4HOW1/79nu3qxZNmWi9wwLhhcKAbTDKGHFpKQECAXcgkhJHBDeuNyUyj5wb03uSSEEggECCGGhFBDTO+4Y3DvcpXV2/b398fMO/vO7MzuSNbuSvL5PI8e7c6+M3O0mPfM6SSEAMMwDMMAgKfQAjAMwzCDB1YKDMMwjAErBYZhGMaAlQLDMAxjwEqBYRiGMWClwDAMwxiwUmAYhmEMWCkwDMMwBqwUGIZhGANfoQXoKzU1NaKhoaHQYjAMwwwpli1bdkAIUZtt3ZBTCg0NDVi6dGmhxWAYhhlSENF2N+vYfcQwDMMYsFJgGIZhDFgpMAzDMAasFBiGYRgDVgoMwzCMASsFhmEYxoCVAsMwDGPASmEY8+zq3WjriRZaDIZhhhCsFIYpbT1RXP/YCvxtxa5Ci8IwzBCClcIwpSea0H7HEgWWhGGYoQQrhWFKWFcG4ViywJIwDDOUYKUwTInEk/pvthQYhnEPK4VhiqEUMlgKzV0R7G7rzZdIDMMMAYZcl1TGHRHdfZTJUpj3838BALbddm5eZGIYZvDDlsIwJezCUmAYhrHCSmGYIi2FcB9iCtF4Et94fAXW7+3MlVgMwwxyWCkMId7c2ISb/rLK1Vo3MQUry3e04h+rduMnf1vTL/kYhhn6sFIYQlz+wAdYvLwRXZF41rVhI6bgXim0dmvVz8VBr+tz2ntjOOG2V7Bse6vrc/LF5qYu9ESzf1cMw6RgpTAE2dcRzrpGKoNwH4rXdrT0AABKg6n8g2gWpbLtQDd2tfViw77B4XJq6Y7iW0+sRGc4htPufB1ffvDDQovEMEMKVgo6G/Z1YuXOtkKL4Yq+KIW+WArbdaUQTwgAwJK1+3DYj1/A2j0djue06NZFb3Rw1EP8/o3NeGrFLvzxnW0AgA+2thRWIIYZYhxSSiGRFGlPzne9vAFLt7Xg/P97Cxfe/Tbae2MFkk5j0/4uXP/Y8rRUUvX9/o5I1uuEXaSkSpJJTQnsaNaUQqveRO+p5VrfpExWQLOuFPoS0M4lQZ/m+uoIs9uIYfpDzpQCEY0joleJaC0RfUxEN9isuYCIVhPRSiJaSkQLcyUPANzx0jqcdufraOuJ4n//tRG723rx6yUbcck97xrtIOQTZjyRhBAil+LY8t2/rsKzq/dgdWO76fitz68zXu/rCGNnSw8eentr2vlr93QgkRSK+8jeUlDdQtGE9npnq1kpNHVpyqc85HeUt1lfE7ZYCo2tPa4smoGmOKAphfaewip3hhmq5LJ4LQ7gJiHEciIqA7CMiF4WQnyirFkC4BkhhCCi2QD+AuDwXAgTSyTx5NJGtHRH8fnfv4f1+zptn4Cf+HAnrlo4ERf831uYVV+B2y6eje5oHDWlQdvrxhNJ+Lyabr3lHx/jqHGVuOCo+oyyPP/RHswdX4VRFaG0zzr1J1y52XZH4mhs7cVDurICgL0dYVzzyDKs3dOBs48Yjbpy7Tq723px9v++icuOHY/SkPaf1s5SmPHTF3Hl8Q2m7ybk9xoWSEt3zCRDJmujxbAUzMpn4e2vAsh/YZyhFBSLrycaR3GA6zQZxg05sxSEEHuEEMv1150A1gKot6zpEqnH8RIAOXs0f2NDE1q6o/AQsF5XBkvW7TOtufTocdjV1ovT73wdm5u68feVu3H6Xa9j/s//hcc/2GGsSyYFNuzrxM+e/QRTfvQCfv/6ZhzoiuDBt7fhhj+vxNYD3Y5ybG7qwnWPLscPnlptHFu5sw1Lt2m+717d7dPYqrWf+MJ97+HM/3kDAPCNRVMwqbYE+zsixoa9U48DAEBTp3bs0fd3GKmo1phCLJFETzSB37222TgWjSfRHYmjN5aAz0No7YlCCIEDXVGTTHYYaxRLoRAWlsRDBMCsFOR3yTBMdvISUyCiBgBzALxv89lFRLQOwHMArsqVDA01JbjmpEm4YkGDcSwcS6KuPIgJI4oBANcvmoLD6kqxtyOMLx47HkGfB7v03kC/fG4tmjojePjdbbjknndwxn+/gQfe0tw3j76/A+9ubjau+4e3tqI3msDnf/8uVu5sgxACTZ0R3Pr8Wjzy7nYAKZfN2j0duPDut3HJPe/ie39dbWxg8rfqRpo2qgx1ZSHs6wgj4NP+021vTimFFmWgjjzfGkPpsImZRBNJHNCVzNS6MiSSAh3huLGx9kadg9Ut3dp5j7y3HZfd/x4AoFVx3XS7SJ8dSGL699oRTslQCDcWwwxVcm5TE1EpgMUAbhRCpKWxCCGeBvA0EZ0E4GcATre5xjUArgGA8ePH90uOybWl+OE507F+b6fJFbNwSi3+41NT8caGAxhbVYyHrzoW/1q7D5cePQ6rG9uwZlcHTp1Wi9c2NOHoX2i9gsqUlM1rTpqEe9/Ygnvf2ILSoA+nTR+Jp1fswqLpI/H+1hZcePfbqCr244ixlXhjQ5NxXiwusLOlB8+s2m0ce2LpTuN1Y2tqs5eUBH2oKw9i2Y5W+DzaE/F2xVKQdQYA8P4WTUlF4lpshGyeoCUPvbMNJ0yuAQBMqyvF2j0dpvtnshRalHt+pCsw1XrZeqAbs+orEIkn8NtXN+OrJ09CccCHFTtaMXtsJbz63zFQyFiJqhRkNtVAEokn8OaGAzh9Rt2AX5thCklOLQUi8kNTCI8KIZ7KtFYI8QaAyURUY/PZvUKI+UKI+bW1tQcl07RRZfjDl+fjni/NxaVHj8PPLpyJsVXF+OKxmrIZVRHCl46bAJ/XY2SyXDR3LD4zdywAzYXz0S1n4rVvn4LbP3MErj15MgDgo13tOLqhClcsaEBXJI7bX0gFhlt7YiaFMK66CLvbe3Hyr17F717bjKAv9Z/hzs8eiVOn1WJnS7rLoyzoQ115CPs6IobbZkez5qp6dvVu3PKPVLimU39CFyJllQD2WTm/f30LvvH4CgDAvAlVAMwWyvbmbuzvtH/alnIAQFckDiGEUe8AALe+sBbxRBJPfLgT/7tkI+57Yys+3NaCi377Du57c4vtNQ8GqRTUQHMsMfD9n/71yX585eGl2N7s7CpkmKFIziwF0h5NHwCwVghxl8OaKQA264HmuQACAJrt1g4kiw7Xnu7OmjU647qzZ43Csu2tmDmmHKdPH4kTp9bgnCO0cxpqStBQUwIAOKyuFBv2dWHB5BGYO74S00eXp+X2T6otwa8uORLLtregvTeG37++BXomKDxEuHrhRKzZ1Y7PzBuL7c3deH1DU9pGXBL0YWR5CNF40tj8tunuo+sf0zZ1r4dwRH2FqeYiEk8aCs4p5VYeX6BbDCt3pM5/+N3tePjd7bZBY9VSSApt4puqFN7e1IwbnliJlz/R4jeJZBKb93cBgPF7IEm5j1LKL5EcOEthX0cY7b0x4/vqGST1GQwzUOTSUjgBwOUAFukppyuJ6BwiupaIrtXXfAbAGiJaCeBuAJ8XhYxSWrh64US88/1FmFxbiuKADxccVQ+/N/0rO2ZiNQBgwaQaEBGuP3WK6fP/umAmXrrxJMybUIVrTpqM0RVFiCsbVVnIh5+cNwNPfHUBAOC8I8cgKYB/rNpjuk6p7j6ShPyetHkIiaTA3PFVpmNq/yO7mIKECGgYUYzRFSGs2JnetkJurkIIPLNqN9p7Ymmupa5IHPs6wigP+bD4a9rf89zqPYYSe21DE+7XYzEyU2ggidhYBbEBVArH3boEZ/z3G8bfnQsrhGEKSc4sBSHEWwAyOoyFELcDuD1XMhwsRIQxlUVZ133xmAnwEmHGmHIAwDlHjMKXjhuPkoAP72xuxvlHjjEpk3rlmtUlATz0b8eYrndYXRkmjCjGi2vMSqEs5DPSTwFg4ZQaLFm3P60VxdwJlfjD26n3u9p64fMQqkoCGYvzqosD8Hk9OKyuDK8r7i7JlqYuTK0rw4qdbfjm4yvwnTOnpa1pbO1Be28MFcV+VBYH0j5X3VLFwYH/52fXliM+gBu3fGTp1XsqxXIQr2CYQsLJ2wPAjDHluOWCWcZ7IsLPLzzCcf246mLj9Q2nTTWUicqYiiJ8tMtcwFYS9GGUohQ+NaMO/1q7H/s6wigOeA1XxhyLpXDh3W+jtiyID390uikAa0UqnIk1JbZKYXVjO6bWlWHTPs3tI1tIhPweo0juM797FwAwc0y5qYeSHXYx5lU72zBtVBlC/v5ZEbZKYQAtBUk+LYX3tjRjT3svLpozNuf3YphDqs3FYGG8ohRGlKY/TQPAyPJgWjdUv9eD2rKU+6i+UrvOrrZeVClP5WMqQhhVHjJtyrKGIZOlMLZKs2Aqi+0rmO9+bRPae2LYfEBTCsv1zqh21lRZyIeSLErB6o9v6ozggrvfxvcXr3Y4I51IPGGyBOwthYFXClL2XFzbyqX3vof/eMJdy3SGOVhYKRSAgJJt5FQpPbLM/njI78Ut58/EP//jJIypTFUyy3qEn543A0SE+66Yj7/qPn1JMinQ0etcNzC2SlMylUXpSuHcI0ajsaUXX7jvPcNSkBlO1TZuovKQH8VZnvatTfTa9DqLFX1oTDjtxy/iM797x3hv9+QeTw7803yYYwrMMIWVQoGpcbIUytJbYEiuPL4Bh9WVGU/of13WiPbeGL560iRctXAiAOCIsRU4fFQ5/vLVBYZ1caA7kjHQXK9bChWKpSDrIc6YWYfvnX04PtnTgbc3HzCdVxZKtwjKi/zwKP6hBZNGpK1r742ZitvadNlkVbLkntc3466XNzjKvUqJU0RtNum1ezqxxuKKy8bLn+zD8x/tcfxcWgqsFJjhBiuFAiE3W0dLQc8yylTbFfJ7Ma66CO9sbkY8KWz7+xwzsRq3XazFN3a19mas7q3XLY8KxVKQAfLyIj+mjyoDoFWCq5aMavlIrE30zj9qDEaUmBXgC2v2YubNLxkuH5neuvVAt2nC3D9W7cYzK3eZzm3pjtq207BzHz3+wQ6c95u30o5n4t8fXorrHl3u+LlUCjc/8zF+9dI6x3UMM9RgpVAgnrx2Aa5YMMG0AavU6sqiLEOHUgBY/LXjjdclDhPTpAWwamcblu9IpZpWWWIH5bosFUX27iBZlwEA31Yyj+zqAKxWQWnQZ2zY150y2RS3+Lu+4asV2YuXNxpP4Xvaw9jTHjaUwJamLsz92cv403vb0+6baX5ETzSOtzcdcPzcDqc5EZ16wH5Pexh3v7rZaD/OMEMdVgoFYs74KvzXBbOM9hNWqvSn6vIibXO1c9EAQE1J6ondKbArU2AffGcbkgK4eG69fs3Uxnzi1BqjvkFVVNIdU1HkN2U+HdNQjXu+NA8/u3CWbVpmuUXZlYZ8COoxhisWNGD22ErjM5nppPZuAoA5//UyHnl3G1q6o4jEk4YlsU2vIl6ybn/afTO5c763+CNcdv/7pjYc2di4336WhDU2s6pxaAxoYphscErqIGViTQlm1Zfjh+dMx8wxFY49glS/vVMxWFnIj/KQD9ube1AW8uHIsZV4avkuU3bSI1cfa7xWlYK0AsqLfKZ7ja8uNiwHaz0FAJRblFh5yIf7rpiHv63YjbryIEoUWVfsaMO3n1wFr0VBdkXi+MnfPzbe72kPY0RpEKSXv9iVOWYaH/qqrkQ+2tVuSgu2w+shJJIC6/Z2mhSYxJra++G2lrRUYIYZirClMEgJ+b149hsn4vjJNago8mfN+QeAkgwzA2Rm0biqYoT82n92p2vaubRkjODc2aNxdEOVSUF8bv64tPVFFgVVGvRjysgyfPvMaSAiFCmZSbvaevHXZY2mhoB2yOptoXdYV3XCjX9egXAsgWgiabq2ikzxtdZ/2CGtos1N9q04rAH75q6o7bqBZCDbdTCME6wUhgEy0FvsEFMAUnGFsVVFRmGYUwxCDRxfuWACABjn3P3FuXjy2uNN6y84qh7bbjsXy358Ok6ZpjUstG5gpRbLoT8FZXe8tB7hWEKpEUhZBX9buRuv6NXdTq42iZtMJOmGkvUdVqw1JM3duVcKnOnE5ANWCsMA6TbKZCnIuMLYqmIj5dP6NG/Hf54/E1t+eY4rOUaUBnFEfQUAmIrpgHSrRG7sU0eWuro2oM2vfn1DE3oicv60eZP0eQixhDApIKvbbcKIYmxSGvHd9JdVWHTna6YGgkAqlnLAwQKw6rSW7iiSSZHTgLNdui3DDDSsFIYBxVme/IFUtfLYqiLjiTNg09xPZWRZEERkchVl45unTcXvL5+HE6dq3VYb9AFGVqXQG9OetA+rKzPWfd7GDfXijSfiuW8uxKNf0WIerd1RdOt9h6zxA7/Xo1sKKfeX1ZU0Y3Q59ndGsH5vJ674wwdYvLwRW5q6jepsiWwieMDBUrDS0h3Fcbcuwel3ve5qfX/IR/U0w3CgeRhQUxbE7vYwMvUfTFkKRUYWj119gWTVzWfA7+37ABy/14MzZ44y3j/x1QX4qLE97YldpnqO15VGwOfB7ZfMxvlHjcFl96cG9B0+SusL1aMrgrbemOE2sioFr4cQiSdNQe6Q32ty9Rw+qhwvrNmLO15cZ5px8feVu7BiZxt+felR+Hh3hzGXuqnLvVLY3xnBfhdKpDMcQ8CXmtfhFnYfMfmAlcIw4LeXzcUTH+7E5NoSxzXHT67B5+ePw3GTRxgB27Nnjcac8VWYPjq9IZ9T/URfqSsPoW5GenX2KdNGYvmONswao7mbpII6YUoNtt12Lhq+/5xpfZHfi4DXg7aeGGSSktWvH08mEY0nTDEFq2KbphfgWdtpr2psx6rGdhzTUGXKeJJuoWzWUouLmEJPNI4Ptrbgyw9+iJMPq8Ufrzom6zkqmTKrGGagYKUwDBhbVYybzkhvY61SUezH7ZfMBgCUj/Jj8y/PGfBRmH3h+lOn4AvHpEarXn7cBNPnZ86sM1VoExEqiv1o740hoG/0rZa6hmg8iVhCoCyYUmjWv1G60dQhSESp9NanV6Qqp8dUhLC7PYzWnihGOFSeA8DoihD2tGefA/3Y+zvw8+fWAoBtF1o71BhFLrq9MowVVgqHKIVUCIBWXyF7Mm36xdnwWeIbv798fto5FUV+tPdGDWVh7bIaiScRTSRNgWbrUKRRFZrV0tQZQX1lEf7xjYVYcOsSI2i9XJk4V19VhN3tYRzoyqwURrlUCup3fkxDddb1gDm4zO4jJh+wUmAKjlUhOFFZ5MfzH+11/Lw3mkAiKUxBbavyqy4OIOD1IJpIYkxlCNUl2nu79hhaw8FWNHVGUBzw4sQ7XrW975iKIqxASpnYuZteWbfPlPXkJvMLMGdYsVJg8gErBWbI4DTnQSJnRWQa0OPxEOqrirD1QDeq9VYifp8HsIkPy2aFXZF4xtoGaX1IemKJtGyrqx5aaryeVV/ueoOPmSwFdh8xuYdTUpkhQ7aq7tfWa376mcokO7tOqnLSXbXeN8opNVcG2yPxhGOPKgCmnlBAqlmeE8UBn2ulEGVLgckzrBSYIUNbhlkQAPDulmZUFvuxYPII45iqE+aM13oYzdCzrWRmkt9nv+FLpdAbTSCDTkizFLrClqwoZTP3kJZJFXX51M9Kgck37D5ihgx7XQRz50+oNgWX5db7tVMm4xuLpgBIZSDJ61mD0RKpFMKxRIYKEM3NJBvoAamJdJKeWCog7vd64Pd6EHOZXhpl9xGTZ9hSYIYMPz53Bg6ry9wWQ7Yal0j3UcOIYiNr6ZTDRmJSTQmuO1VTEtncR72xZEb3UVHAa5pNYbUU1JkMAZ8HAR/1y30UZ0uByQNsKTBDhoVTa/DP/zgZa/d0YN3eDtth9tb+TzK1X7UGKor9eOXbpxjvnSwFqWB6YwkkMsx5Dng9qC4JGH2SrEV16sjRgLQU3CoFTkll8gwrBWbIMX10uWmjVbF2ipVttp02fsC53UfQ50XQ50Eklsg40S3g8xiZTEC6pdBjsRQ0pdD3mILbOATDHAzsPmKGJE55/mmWgr6nZlIKTj2eQn4PigJe9MYSGVtMBH0ejFAm4MmYQnckjngiaVIKMqbgtuOpk/toydp9aYN+GGYgYKXADEmKHdqEyzbiH/zwNHzwo9OMmELAIcMIcFYYQZ8XIZ/XGN6jonZfVS0Fv5fw1sYmJJMCM29+CTc8sdLo6irXBrzZYwr7O8NobO2xzT7a2x7G1X9cim88tgLtPTFWDsyAwkqBGZKoo0dfuvEkY5qcnFM9sjyEkWUhI/so4HUuaHMKNAd80lJIpmULqZZK0OfBadNH4tKjx+G7Zx6OV9c34Xl9ROlzq/cY8x+A7NlHz6zajZ0tPTjmF0uw8PZXTcpDuo/ksU37u3D948vxo6fXOP5tDNNXOKbADEnUTbmqxG+kg1rnVCeFjCk4WwrOMQUPQn4veqPZLYVTpo3EKdNGIhxL4M6X1+PJpY3G5229UdNavy89prC9uRtLt7XipidXmTKZ1PtK95FMhIolkth6oBsNI8AwAwZbCsyQpFjZlP0ej9FB1BpTkMVr/gyzI5zcRwGfByG/B5F4ekxBVUqqpRHye3HcpBGmLqgfbm1R1pIRU1Crrc/7zVu46Uktm6q1J+UOsut9JIftxJMCLd1RxDNkRtmRTAr84KmPXI0lzQU/e/YT3PnP9QW5N5MdVgrMkERtouf3eYzNPz37SCPTlDlHpeD1oEhaClaloCgla0O/4yebH93X7e1MXVOPKQDmVtidYftsKnNMwew+6gzH0BNNYEtTN2bd/BK2Hei2/wMttPXG8PgHO/DGRnftuweapdtbsWJHW/aFTEFgpcAMeVTXULqlkD0l1frRc99ciJs+dRh8ulIIxxOIWN1HGbqcztQHB0nkUCMph5TFTd2BnaWQUg7a7/2dEXRF4li8vDH9AjbIdN5wrDB1D4lkss/WDZM/OKbADHn8ntSubp1TbbiP+jBadOaYCmNjD7mwFKxYJ9l1KFaASSnEBRBARsJKiwzDfeSwofZa5ks4IbOhIjF36weaeEIYMSBm8MFKgRnyqLMLrKmqcuvJZClkIuT3IhxLpimF4oAXf77mOKzame4GUQvZykI+k2uIkIpvdEXjqMjSDrw3moCHNLdT3OI+SlvrcpPv1rOhwgVSComk4Clygxh2HzHDivQ2F7JOwfmfuk13bYOQ34NwLJG2EVeXBHDcpBH46smTbc+bVV+OybUlqCo2mwICMGIKJ9z2Cjbt77Q5O0VPNIEiv9dU8OZUDS2VwlsbD+Cz97zj2CupJ1po9xFbCoMZthSYYYXV1y83fF8/x48W+dMrmn/zhTk4cWpNxvOe+fpCEAEX3v02dqSSjyCE2WpZubMdU0aWOV6nKxJDUcALIdSYgv1mHo4lcPKvXsX25h4AWhaTHHmqYlgK8QK5j1gpDGpyZikQ0TgiepWI1hLRx0R0g82ay4hotf7zDhEdmSt5mEMDq0UgLQWfp3//1IsC6RXNDSNKUFmcORjg8RCIyGadMCmFaDyJZIYNsqM3jpBuKcTieiqqg6XQGY4bCgFI/e1WUpZC4dxHrBQGL7l0H8UB3CSEmA7gOABfJ6IZljVbAZwshJgN4GcA7s2hPMww5qTDam2PnzFjFAAg6M/gPspw3ZDfi6Qwp4z6+hC0lkpqyshUy29VKYRjCcd0VEAbMVrk96KqJIADXdrMUCdLQXZplTj1a+rWA9KZmvzlkngyaYopfH/xary6bn9BZGHSyZn7SAixB8Ae/XUnEa0FUA/gE2XNO8op7wEYmyt5mOHHCVNG4O1NzQCA+6+Yb+sOue0zR+A7Z07LOLc5U0yhTh+1uWFfyvffF1dUhz4tblpdGTbt74IQ5j5MLd1RY7a0He29MYT8XtSVB9HYqqW2OsUUmjrNg6admu71RAaXpfDU8l0oL/Lj1MNHFkQexkxeYgpE1ABgDoD3Myy7GsALDudfA+AaABg/fvwAS8cMVR656thUcZrPYxtM9ns9aeMy+8KxE6sBAPs6UhtuXxwfcsOfWFNikknS3B3JqhTqK4tQX1mED/TKaKeUVGlJSLJZCoUMNKt/QzyZZHfSICLn2UdEVApgMYAbhRAdDmtOhaYUvmf3uRDiXiHEfCHE/NpaezcBc+jh8RC8/Qwg23HyYbX4w5fnm46Nqy5GfWWR6ZiTT9+Oq06YCCDlPhIwK4Wmzihae6J2pwLQLI1QwIsxlUXoCMfRGY5lbOOt4qgUXFoK8UQSP3v2E+zryD4GtS/Ek8JoaZ5MCiQFWCkMInKqFIjID00hPCqEeMphzWwA9wO4QAjRnEt5GMYOOYjnvNmjsejwurTP502o6ve1P3f0OGy77VyUhTSjXAhhKqRr7o5g/V7ntNTOSBxFfg/G6Ippd1vYdY6/U+xBBpplTGH93k40fP85rG5sQ1ckbpy3vaUHD7y1Fa+vH9h2GKqlkNB9dyKTD28I8bl73sXPn/0k+8JBTC6zjwjAAwDWCiHuclgzHsBTAC4XQmzIlSwM4wanOcyTa7Wn/Bmjy/GdM6dh+mjnFFLna2u/rZZCc1cUqxoz9wEq8ntRXyWVQq/rWc3OloK5eO3V9VqQ99nVezDr5pfwjcdWAEhVSFtbfBwsakqq8XuYKIUPtrXg/re2FlqMgyKXlsIJAC4HsIiIVuo/5xDRtUR0rb7mpwBGAPit/vnSHMrDMLZcdcJEFPm9OMmh9mDySC0eEE8m8fVTpzgqD7eoSqG1O4qPdrWbspOshPxejNUthR0tPaaxnGr1tBWnzdyakirlkW0vXvx4L4BUMZzbdhhvbmzCs6t3Z12nVjTHDeXg6hZMHnAMNBPRahfnNwkhTrP7QAjxFrSqfkeEEF8B8BUX92GYnDGrvgJrf3aW4+eTarQNe2dLr+OabJD+v4K1eK0zEkdnJI7PzB2LTfu7bM8N+b2oLQuiJODF1gPdGK0EzkeWBdHSbR+TcBrkk7IUtM9lhXWbJeDd28fU1csf+AAAcN7sMY5rhNCshISu2OTv4eI+Gg5kyj7yAjgnw+cE4JmBFYdhBh8yc8htbyE7ZJO8z80fZ9ucr7rEuQfdiIKZAAAgAElEQVRSUcALIsKk2lJsOdCNmtKUdTCuutjUmhsAbr34CPzgqY+cU1KlpRBP6DEOTUlZlYthKQxgPYPVXWTEFjjQPGjI5D76qhBie4afbQCuy5OcDFMwigJenDt7NP7vi3P6fY1RFSFsu+1cnDt7tG0AuDSYQSnoNRYTa0qwpanLqFN4+rrj0zqyAsCCSdo8BzWmIIQwnsZlSqoQwEm/ehVyP27Th/sE9dTesKEUBq6eId1tNLxiCsMBR6Wgu39AROcRke06uYZhhjt3f3FuRrdIXxhbVYwxFSEsnJKKYZSGnI12qRQm1ZZgV1svuiJx+DyEOeOrbAvpZL2GqhQ+/X9v4coHPwSQKl4DNJeYTImVv2Whn3QfuU2BdYM1wCyVA+uEwYObQPOlADYS0R1END3XAjHMcCfk9+KdH5yG06enKnhLg84V1yG9Rcfk2lIIoVVXy1Ybsk5DjX3bDfFZs6sDb2xoQmNrD7oicZMLS1ZKS0tB3i8n7iORUgoyviDfM4ODrEpBCPElaNXImwE8SETvEtE1RNT3vDyGYQwCvpQiKAk6WwryyV26ilbtbDM2fkMpmK6rZxPZbOavrW9CTzRhaund2Ko10evSLQjDUjCyj5yVwoV3v43zfvOm4+dWEkrmlJqFxO6jwYOrNhdCiA4iWgygCMCNAC4C8B0i+rUQ4je5FJBhhitqW47igBdEgJcorThNtgOfWFOCkN+DjnDcSEVV3UePXH0M4klhxATsAs2t3VHELUNudrWas6pCurIKR7PHFFbaDBnKhHrfhBBI6IFmzj4aPGS1FIjo00T0NIBXAPgBHCOEOBvAkQC+nWP5GGbYorpwgj4vAsqoTpWxVcUANKvg8FGatSCVQcp9RDhxai1OnTbSNO5TC0ynlEOT3h9JHSfaaFUKAYulkIOYgnxtDTi3dkdx+E9eMPo8MfnHTUzhswD+WwgxWwjxKyHEfgAQQvQAuCqn0jHMMCaoWAoBnwcBr8f4rTKpNtVMb1qd5rWVG79doNmr94R6dvVuLLrzdSxe1mh8JjupXnvKZNz8aa2TvdWiCPnSYwp3/nM9bv77Gsd+SW6f9M2N8ITRR0rqihU7WxGOJfHb1za5uh4z8LiJKVwBYAMRna9bDaOUz5bkVDqGGcao7qOA1wO/T7MUrN1ey0OpdFU5SU0Gmj0ODQEDXg826sVw+5WW2vJ1dXEA/3bCRIywqYiW9++Naht4JJbAb17ZhD++ux1/X7nL9n5ONRH//HivURcBWCyFRCrQLAcNefVhSBx4Lhxu3EdXA/gAwMUALgHwHhGxhcAwB0nAm3LhBP26peAl/Pfnj8IMm/oDAKgs1hSETBd1mu2gKhb16V5aCsV6tlNNafq4Trkhy/PUDV9mKFmxa8O9bm8HrnlkGX78tzXGMTWmELcJNMu/py+daJmBxY376LsA5gghviyEuBLAPDi0uGYYxj3plgLB7/PgUzPq8PwNJwJIuYskMsAsM4Xkk7VVNagjPtV5DVIplAS0HJOasnRLQW7U0n3UrigCpxkMdv2RZItudUSoagEklZRUeVjGSNhSKBxuso8aAah19J0AduZGHIY5dAhYYgp+rwcepeDgze+eiiqLe0emkvZksRS6FZeNqhTkRl+sB5Pl9abVlWG9Pl1ObsjS7bNXmafglImkKgshhN40kIz3EtUCiCsttKUSMywFh0FChyJ728MHNSiqr7ixFHYBeJ+I/pOIboY2NnMTEX2LiL6VW/EYZviiBpRlgFnNPhpXXYxSS/2CVUk4DRlSH7SlUlAViLyujCmMKA0YRWuyNXevvtFLBQSkNv/mrgi++kiqqbE6ClUqFXk7VZaksI8pWC2DoWgpJHMg88uf7MNxty7Ba+vzN8PajVLYDOBvSE0h/Du02ctl+g/DMP1AtRSCPq+uGDK35a4qNvdIcjN5riOsPfGPUBrpyZiCVDI+rwfLfvwpHDep2nAfhaPpVoHc/O99Ywte+nhf6rjiPpLnyxbj6lZpjikkjfdJYVYObgcJDSaSOai1WLGjFQDw8W7boZU5Iav7SAhxCwDoFcxCCGHf35dhmD4RNCkFzUrItsW7tRRUOnRLobokaMyaljEFGaMIRxMoCfpQUeTHe1tasPD2V2wb98nN3zpTQnUfyY3dcBup1kHSvC7poBSGoqWQi6rsQnwNbrKPZhHRCgBrAHxMRMuIaGbuRWOY4Y010LxwSg0WOgz6kZRZ3Ek+m95HVqT7SG25LYvXpFKQMQifHrhubO01FIikNOgzCtmsuqjXYil0hmPY267FIkyWgtrmQqQXr1m7qGZi/d5O3PHiukFTDZ0LMeSo2IOc69Qn3ASa7wXwLSHEqwBARKcAuA/A8TmUi2GGPWpMweMh/MenDst6jvUJPZulUF0SMJSCjB8U+b1GfUO1JXCd6Xp15UEjy8hjkaMznApmJ5ICi+56Hc36fAZ1s1QtgHgiPftIPm278c9fdv97ONAVxb+fOCnNgioEObFu9EtSVhty4HATUyiRCgEAhBCvAShxXs4wjBusRWp9Yc74SgCpJ3snigNeY7MaodckqHEJuZnK9FFrNpPc+70eQkWR33ATWXVHZziV7RRPJg2FAJh97abeR0qdguE+SvQ9phArUKZSIilwoCtlTeUipiCvmE9Lwc2/yi1E9BMiatB/fgxgaE+mZphBQH+VwsZfnI2/XqsZ6ll0gukeUhnMrK8wjknrwclSqCjSzikP+RD0eR1jClZLQcXRUkimGuJZh+24eeqWCrFQhW7Prt6Nk+541SgkzIVukq6xPOoEV0rhKgC1AJ7Sf2oA/FsuhWKYQwGnGoNs+L0eY/P2GcVr5mvde/k8/OKiWQgq7bkPdGlP70coSqFSdx9N1vsr+SzZT7Jj6rETRyDk9xjZR1bloVoKzV3msZ5O2UcJm95HqdhC5h12+k9eNOonBrJhX19o6oygJ5ow6jlyYinol7S663JJxpgCEXkB/FAI8c08ycMwhwzWp+3+YGzOlkudMVNrUfaXpVozvIDXgwuOGoM/vbcdF8+tN9YFfB489u/HGpXT6Zu9ZgGcNn0kXlm335itkMl9tLnJnKAoMmQfWXsfuck+iieSpsD2QI4L7Qtp1k0Os48GTaBZCJEgonn5EoZhmL6RzdoI6sHskN+DOeOrsOmX56StOX5yKuPJGqO44fSpiMSSuGhOPd7Z3GxYClaF1qG4jzbvd85aV7NcbWMKLrKPYhZ3UaYhQLkkFRTX3ucmppB/15ib7KMVRPQMgCcBdMuDQoinciYVwzCuyJZ9FPRLpeA87lPFqmTGVRXj7CNG69fwGIFm6307elOWQpPVfaTsa+bW2UnHWEIiQ5zAOjPaqZ13rklaZLfzeO1p78X1j63AfVfMN9J/+4IwLIXBlX1UDaAZwCIAn9Z/zsulUAzDuCObUpBpr7LXUdbrWWMKynmmQLPlvA6lv5Js1icxtbZwyj5yqFeww9qm2ymmEEskcfVDH2JVH6fDObHtQLdJAUkxEgmztaNy/5tbsWx7K55a3pj2WV/IZ6DZjaVwvxDibfUAEZ2QI3kYhukDRvGaw+cy+6ii2N1TqtVSUCe0Bf0eYwO2+s9V91FX2Nxe27nNRSr7SB62FrPZ4VYpbG/uxpJ1+7G1uRuv3HSK4/XckEgKnPL/XsNph4/EA18+Wj9m/i7sZJbfZn9dS0b20SBLSbWbwcxzmRlmEJDVUtCVQmWRP+O61PXMW4KqFEI+L6LxJJJJkebeUTuxdkfM7hzhYCkkbcZxyg02U/ZRzLX7SP9uBsAtL+V5Y2OTccyqDOz2fbmZ9zfcIE8bFNlHRLQAWtVyraUbajkAd7YowzAZqSkNYFx1cb/Pz1a8JvsrVRa7UwpploLiPpJxiUg8iVjSWSl0WtxHAlqG0IJbX8HMManhQXFFuaSK18yWgx1uLQVyqROi8ST++clenHZ4nenvVbHTUQlLgNnOGvDYNAXsC8kCWAqZ3EcBAKX6GrUbage0CWwMwxwkS3/8qYM6323xWoVrS8HZfSRba4djCVNqKWBur90VsUxnE9pMgJbuKN7ceMA4bJt95GL3tAaanVJS3e6jF//ubazZ1YHfXTbXCKpbkZaCWg+S1sTPRinIAHH/3Uf6dfp1dv9wVApCiNcBvE5EDwkhtudRJoZhXJLNUpCfu3UfWS0FNWtJFsKF4wnbQLDMTrK6j5JC2D75x5PqPAXov7Onl1otBadpcPKW2RrmrdmltaXOVARnuL0o/VhaV1iFgXIf5dNUcBNoDhLRvQAa1PVCiEW5EophGHd4s3RJlRuo20BzmqUQSLcUIrGkbWuJspAf4VgEXeF095Hdk3LCdp5C6vN4IgmfN13pubUUZEaT2/04Y8GczWdyfW8sgXgiCZtO40aRX387uRaiAawbpfAkgHsA3A+gMAnBDMPYkq14TfblKQ+5+V/dxlJQeidJqyEcT9huoGVBH5o6I2lP8vGEsJ3NkEhCyT6SSiG1LuqgFKzXcipek+4ctxtrJqUgFYz69UiZP3vPuzhxag1+eM70tPOku6n/DVSdrZBc4eZfSlwI8bucS8IwTJ+x1hVYkX15igMulYI6ItTrSXsPaE/qdpt8mYPiiSaSthu3ailY3UjyPq+s240Tp9aaYiJpxWsOloK0ZtxWBWdqU2FMk1P8R6oSeXPjAdN7OadaWnDtvTG098Zcx3ZS19F+52LUpxNuUlL/QUTXEdFoIqqWPzmXjGGYrHiz+JplALgo4K4jq2opSHeRRAato/Gk7VN1qaIU1FkRsUQyzXoAzDGFVPFaat17W5px/WMrcNsLa03nubYU+riRZiqYk9ciG0tBor61ViI/8NZWHHnLP/skj3qPfDaCdfP4cKX++zvKMQFg0sCLwzBMX5CblNMQFpnDX+R3ZymoMQWrdaEqBbsNtCyYegouC/mMmQqxuJOloGYf6ceUnfXlT7Rh9Wm9jtJiCvZKIW4EgG0/TsPuafyP72zDGTPrbBWMNa6iyp4QAp4BGI2Tmmg6iCwFIcREm5+sCoGIxhHRq0S0log+JqIbbNYcTkTvElGEiL7d3z+CYQ5VivxenDmzDvdfOd/28+tOnQKvhzBDqQ/IhNo625qzL5VCJJFE3ObJX7UUyhU3SSwhbIPBap1Cqmgttfm9sGYPAKBhhLmOIy3Q7FC8lqmozA6romvvjeHmZz7Gc6v3KO4j5fqWC9u18zjYojN5xXzOrM76+EBExQC+BWC8EOIaIpoKYJoQ4tksp8YB3CSEWE5EZQCWEdHLQohPlDUtAL4J4MJ+ys8whzREhN9fbq8QAODUaSOx2aYzqhOyorks5MP3zjrc9JkaU7C1FBSloL6OJpKmp3mfhwzXUeppXhavCYT82jO2dH1Zm8FlsxwkbjZS9QncailIxae6uVRZrOuTppiC9rufIzPSrpNHneAqpvAggChSM5kbAfw820lCiD1CiOX6604AawHUW9bsF0J8CCBmcwmGYfKMjCkc3VCNs2aNMn0mq6Pf2ngAK3akN5krC5ndRyrdSpVzSVD7LJZI2k5e83s88CsWi9UyiFqsDqeU1Ez1AxJ1s7UqOrWNRap4Tf3c+Vry3IMtL5BB8ly05XbCjVKYLIS4A/rGLYToRR8L7IioAcAcAO/3UT6GYfKIdHfYPeH6dUvhkfe2Y1dbb9rnZUHFfRQyZ9monVOLA174vYRIPGmKKQihPZF7vYSL5qSeH62BZaul4FS8lm16G2DpxWTZeNXiNLtLWS0Fu2sddMvrQZp9FCWiIujiEdFkAJHMp6QgolIAiwHcKITo6I+QRHQNES0loqVNTU3ZT2AYpl8YgWubzSzbTGk1SJ3JUgj6PAj5vOiNJixpnNrG6vMQfnzeDKy55UwU+b1pSiG991EWSyGDzE5tvdX3JkvBpqI5JX+6K+pgdUIq+2hwKYWbAbwIYBwRPQpgCYDvurk4EfmhKYRHD2YojxDiXiHEfCHE/Nra2v5ehmGYLMiNzS7V1Ukp3HeFFtOYPTY1+9lqKXSalIIXoYAXEUu7jIRuKXiI4Pd6UBr0we+lNMsgvaK5/9lHqlJ4dvVu/HrJxpQ8ilKwiylYN+qEjYI52Pwj+fUMqpiCEOJlABcD+DKAxwHMF0K8lu080r69BwCsFULcdXBiMgyTD+TmY9dSyUkpnDi1BltvPQfTR6cynMotRVpq64ug32P0SbIO3ZGWgnrPbJZCbzSB7/11NfZ3hs1/i2EpZK8/AIAN+7pw18sbjPdGYZ0QtoH1tECzSH99sIHm1FS3QZB9RESjhBB7AUAI0QzguUxrbDgBwOUAPiKilfqxHwIYr1/zHiIaBWAptHbcSSK6EcCM/rqZGIY5ODL5wgM2LScALThNRBndR53hdPdROGa2FKT7yKNcx++1UQoWy2Dj/i5s3N+F3lgCv/7CHOO4O0shw2emmEK6OygtJdU2puB8fTdkmuqWKzKlpD4PYG6W8x3XCCHeQpaAtK5Qxma5B8MweUI+Odu6jxyUglQG6hO+dR7xM6t2G6+DPi9C/mRaC+6EEEgIs6WgKQXzhmjXYgMwK4tYIpk2FtSOTE/gagsO2zqFNEshXSkc7AO+aq3ki0xK4UgiyvTETtBmKzAMM0zIlF/v8Wi9fOSaI+or8L2zDjesCtVSmDKy1Hh91sxRePHjlEMh6Eu5j6yB3niapUBp7iKrpSBRlcXlD7yP97a0aH9Thr830xN41phChuwjo33HQW7m8vx8dkvNNE+Bp6sxzCGG3IQ8Ds5wdXOqKw9h4dQa4726YU6uTSmFaaPKzErB70Eo4UVnOG6yCoTQKpzTLAVrnYKTpaAclwrBKrOVTE/gdkrB7nOJKaaQBJ5bvQevrz+4bEk3M6sHGnddshiGOSQ4ol7LIDpz5qgsKzOjDucZUWp2JWnuo/SYQiKpuY/U1hC2gWYXloIZ5w3VTifIDCy7QHNrTxSvrdd6MlmtAKv76OuPLcf7W1twMFhbi+cDVgoMwxhMrSvD5l+e40opuJmSBgAjSoKm95r7yIuIpduqTElV+y/1JaZgXSdR99OVO9twx4vrlL/B2QKQG/Fj7+/AtX9aZlzryw9+qMdDnJVCpo6rfcHaRTYfsFJgGMaEdfqaE243PmvQOeT3IuTzOGYfqUFun6d/MQUVVcoL734bv31tszFnwk4pSOViN11Oolk16cck+zvCGAhSCmpALucKNw3xJgNoFEJEiOgUALMBPCyESG9+wjDMIYPd5vzI1cegYUSJ6VhNmvvIg0TSm559pPvuvZY6BWsWkWNMwUFZSHdQRzjVYq25K4riap+t+yiaSKII3szjORMi7eldvdbO1h7Hc/tCIbKP3FgKiwEkiGgKtGK0iQAey6lUDMMMeuwshROn1mJctbnV9YhSO/eRln2kXiMp3UdK5ZybOgWJc0xBY6XSxK+pS+vUY7fZyutk2ohjyfRBQ+r7nS3pvaH6g1GAN0hSUiVJIUSciC4C8D9CiN8Q0YpcC8YwzODGbqaCym++MAfjq4tRaaluDvq9EJAD79V+QbJ4LbXW7yXE4uYN0akBnmNMQf/d2hM1jh3o1JSCXQBXKp1MMZN4QmQMNA+4pTDIYgoxIvoCtAlscoZC3waNMgwz7HDahCWfPnIMjhxXmZbe6vOQkZ3UE00Y7qJU8ZrFUrBszr0OQ3Wc3Ufa745exX3UHUVTZwR/W7HL5u/S5yhk+Pu0tt/OSmFPW+aYwh0vrsPbmw5kXAMUJqbgRin8G4AFAH4hhNhKRBMB/Cm3YjEMM9hx05raDg+llEJ3JG7MTkiK9OK1gI37KOygFJzcR3KzbleUwoHOCK7+44f4zSubHK+TKQ1UHbyTuk/qtZPikvz2tc247P7skwQGZfaREOITIcQ3hRCPE1EVgDIhxG15kI1hmEFMNktBZe1/nYUvHTcegNYPKOTXtp7eWMJon5FMCiSSSZviNav7yMFScFAKcmNt740h5PcY86N3tdr7/aNxc52CHbFEMr1LqrLeyWrpKwNVGd0XsioFInqNiMqJqBrAKgAPEhF3PWWYQ5xsgV2VooDXcAt5iBDypYrbAvprrU7BPNfY76O0+/RG7ZVCZziOW/7xcdpxubm39cRQWRRAbWkQTV0Rx5I2I9CcTSlkmKfgpKCA7LEY01oj+8j1KQeNG/dRhd619GIADwoh5gE4PbdiMQwz2Mnkc7dDbpoeMlc8B6T7KAlbS0HdYIUQGV0zD769zUZO7fz23hgqivyoKQ1ib3vY8enbjVKwS0n9+8pU079MloLT/Ac7BqWlAMBHRKMBfA6pQDPDMIc4fbEUAHVWAxnuIyA1p0GmpHozxBSiiWTWoKs1fTMpNNeUVApzJlRidWObqZ23ilRCmdxH8WS6+2jp9lbjtdM0OCFEn1xLgzKmAOC/ALwEYLMQ4kMimgRgY5ZzGIYZ5kxWOqG6QZ3VYLIUdKVgV7xmbXMRjmbfUO1iHXFdKZQX+XHqtJGIJeyb3KnnZ9qItfOdZYg4pM0mRR8tBZF/SyFrnYIQ4kkATyrvtwD4TC6FYhhmcPOnq4/FrPry7AsV1Glk6hQ3v1exFES6UlCVhXQdFQe86HGILdg9pceTSXT0xlBZ78e8CVXwecjREpBdWbMFmjNt1JEMQW8nK8Jpvfbb9SkHjZtA81giepqI9hPRPiJaTEQ8GIdhDkFeuOFE3PGZ2Vg4tQaVxYHsJygsnKK12Z41psJQBIDFfZQwKwXZHE+6kKRSKAk6P8/aPYnHkwJtuvvI7/UgaBktqt7TVUpqBksDcI4pJPvoPpLxkHxWNLtxHz0I4BkAYwDUA/iHfoxhmEOM6aPL8bmjx/Xr3HNnj8aqn56BI8dVGrUJQGqiWyKJtMlr8jNDKejWQWkGpbBk7b60Y40tveiJJlBVrNXdWudNq/eMuixe64+fPylEn9xH8haDrfdRrRDiQSFEXP95CEBtjuViGGYYUiE3ZRtL4UBXBLFE+uQ1IOXnV91HTnxv8Udpx259YS2CPg8+feQY0z0lPpOlkL21RFwZ0dkX+uo+ihvzFPp8q37jRikcIKIvEZFX//kSgOZcC8YwzPDFpyoF/fV1jy5HS3cU46pSDfX8PrOlIAvXSgKapaBaHJn4ZHcHFkwegQl6B1e/Zd60OjXOVUM8m+I1NySTKfeWz0WL8sGafXQVtHTUvQD2ALgEWusLhmGYfmFyH1me2uX0N21dSik8/sEObDnQDQAoCWqWgton6ZwjnAcDRePm+gfrPRWd4LJ4Lb1OwQ0JxX0kLaLvL16Nvyzdab++AHUKbrKPdgA4Xz1GRDcC+J9cCcUwzPBGdR9Zn9pVpSDX7WkP4wdPpdxCMtCsbvTWlt0qkUTSPObTck/1MxkIzhRTiPfXUlACzT4PQQiBp1fsQiSexOfmm2M1yaRIxRQGmaVgx7cGVAqGYQ4pVEVAFi+KjDsAqSf6lu6oaY0MNKvxh6DXeTuLxpNpw3tUVBmiLt1H/Xl4TyZTloKXCC3dUUTiSdt+Tmp32DwaCq7mKdjhzpHHMAxjg1/ZlOVmcsm8sfj6qVNM64r0Irf2npj5uB5oVt3yQb9z8Flb685SkA34Ms1T6EsGkUpCCER0BeDxEHbrLbbtlMK6PZ2m8/JFf5VCHvUWwzDDDTWmcPzkGlw4px4nTq1Nmw8tK5/bes2WglQW1pYYmfBkshSU16mYgvO1+qsUkiJlifg8hN3tWqdWu8FBH25rAQBMqysbHDEFIuqE/eZPAIpyJhHDMMMevxIg9nkJp0wbabtOWgTNXWalIJWFqaNqlkwk9WNX2UeZLIUs8xKcSCaF0QJDsxQ0pWCXpvr+1hZMGFGMURUhtPVE0z7PFY6qVQhRJoQot/kpE0L018JgGIYxPbVbrQMVaREcUJSCz0PGOVIpeMic5nqpTYFdJkvBYxNTyFSHEO6v+0iJKfgUpWBnKexo7sG0ujJ4PTTo6hQYhmFyhppWakUqhebuSGq98sgvN3Ovh0yWwuULJqRdy0vuAs1Gm4sMO/HqxjbHzzKhZh95iLC7XY8pKJbCzpYeJJICrT1RVBUH4CHgo13t+PvK9NGhuYCVAsMwBSWjpWDjPlKViNebshi8Huc0V7lG4ibQnMlSeG9Li+NnmdDaXGgKQAiRch/plkJ3JI7T7nodf/5wB9p6Y6gs8Ruy3fDnlf26Z19hpcAwTEHJVNkrlcKBrpSloCoR+fRvtRTsrmmd/ayirpabdi6CuwmlojmeFIr7SLtnTzSBaDyJtzYeQDSeRGVRwKSw8gHHBhiGKSjeDAHikO7mabbEFCRyw/QSpbXcTruPcijdfZQ6V/ZX6utkOSsBnyetI6rqPgrHEuiMxI3XQKpI7Z3NWiehqmJ/RksqF7ClwDBMQclkKfi8HgQsIznVTVI+/Xs8lDbG04rbmIKc03CwVcRSoVUW+/HZeWONa0pLpCMchxBATWlAsR5So0O1cwNpxX25hpUCwzAFJduTcJGlI6rPwX3ks6S5WiFyVhoe5TqyPXd/C8akG0sW040qD+HMmVpfJrt5ChNrSrSuq4lkmiKqZEuBYZhDjUzZR0AqA0miups8Smqqetzv8WDlTz+FJ645LnWeizYXpUGfYSn0pzX2Y1851vh71LiFvLeakiqZVKONNQ3Hk2n31LKPWCkwDHMI0VdLwUtkbOKplNT0grjK4gBKQ6mwqbn62XxPufGWBn1GTCGhxBQunlOPK2zSXFXGVxfj+Ck1hiUT9GvyCJFSOnYzmsdUarXA4VgizVKoKvYPH6VAROOI6FUiWktEHxPRDTZriIh+TUSbiGg1Ec3NlTwMwwxOss0VCFktBZvCN2ugWbqPvDZBacCciQSkNu3igNfWfTR9dDmmjizNKKe8hrx30KfJLZAaMZoUwqiDkFSVaA0Aw7FEWnC7othvyoLKx1jOXFoKcQA3CSGmAzgOwNeJaIZlzdkApuo/1wD4XQ7lYbAQTdkAABKASURBVBhmEJLNUrBOWVPdTUZFs4dMcQRpNajBZTWMQLC3FIqDPvREtYwg9amdKPv0M3lFWVkdUiwFKYed+6g8JJVCKqYwY3Q56sqDCPq8phhE7CAzotyQs5RUIcQeaEN5IIToJKK10GY8f6IsuwDAw0JTf+8RUSURjdbPZRjmEMAuKKySFlMwPf2njvkcspJSa1OvrR4Z+bbY7025jyxaIFvdggxk+6X7SI9bCEUOO0tBWkLhWMLIPrrh9Kk4uqEagLkvUjSRTIuHDDR5iSkQUQOAOQDet3xUD0AdOdSoH7Oefw0RLSWipU1NTbkSk2GYApDNfWR19ahKRHUf2QWsVUvBpBSs99A/Kwl6EUtoG7fZUiDbmQZPXrsAP7tgpr5GyqcHmn0pZSavn0zC9OTv85BhUUTiqXsWB7yoLgkYxyX9bcTXF3KuFIioFMBiADcKITqsH9uckvbVCyHuFULMF0LMr62tzYWYDMMUCG+W7KOdLT2W9ekbvdV9ZLfWKb4AqDEFzXnSE009tQPaRmXdmLbddi6Obqg25DfcR1ZLQQjDdZWwWAo+LxmWQiSWMLKPVAVnUgr9bMTXF3KqFIjID00hPCqEeMpmSSMAtZ3hWAC7cykTwzCDi2yWwlZ9LnNNaSBtvdlSyNzaQlUKVvfR5FotiFxbFgQA9EYTUDtnEzkHeeWGL91HUjnJzV4IxVIQwhQX8Hs8xrpeJftIVXCqZWGtccgFucw+IgAPAFgrhLjLYdkzAK7Qs5COA9DO8QSGObTIFmg+/8gxAIAJI0rS1nuV2IHddZzcR1Zuv2Q2/vDl+ZhVXw4A6InG0y0FfS8/uqEKf712Qeoz/bopS0HbVoO+9IB4MmkuXtMsBW3d1X9ciqbOSNrfOJwshRMAXA5gERGt1H/OIaJriehafc3zALYA2ATgPgDX5VAehmEGIdkshTs/dyTW3HJmyirwkJE9JDdkr8epM6r9a7IoiNKgD4sOr0ORX3MfLbrzdexs7TWtF7oD6cixlZivB4GBlOKRl5QVzQEl0KwWr0UTSSVA7jFSVwFg3V5tBKf6nUSVQLPdMJ6BJpfZR28hyyxnPevo67mSgWGYwU82S8Hv9cDv9RgbpRqDkF4Wa52CxNF95HAvNf1VPrUD0n2Uem2+h/7bcB9lsBT0NhfFAR+6InH4vYSxVUWYO74Sy3e0YWdrT5qsat+nIe0+YhiGcUO2NhcSuVH6PITTp2vjO88/SnMtETkEmh3cRw01xbb3sFZPS9RAs9UNZX2fCjTLmIIwFEdSaEN85H2kwvvjVccAABp160T9TqLDyH3EMAyTlUyts03rFPfR1LoybLvtXBw5ttI4ZqdcnCyFRYfXYfHXjk9bb62JMCAy6hSsrieP4T4yB5rV1hZq8Vo0njTuI9eWhfwoC/rQ2JJuKZhjCsMgJZVhGMYOI6/fZRdQubE6ZR/ZBpptCtok8yZUpa0POhSGBb0eR/eRvIc10KwmK3mUmEI8KQw3ldqvaXRlCM3d0bS/cdhkHzEMw2RCbntuG76ploIkVacA0+Q14xyT+yj7PUaWhUzv//fSo3D9qVNw0dx6IyXVqfBN7u9SDmlZCAhDDjlMR7qP1L9lVEVRSm7luEyTBdh9xDDMMOb/ffZI1FcW2W7mdqgxBYlHURT2gWblfBfKp6LYj223nWu8P3JsJb595jT4M1gK8rYyIyplKehKQalTMJSCX8YUFKVQntr81fjI4/9+HL571jQAqVnOuYSVAsMwBeHiuWPx9vcXpfnonUhZCmr2kf6UTmRyxVg/B9LdR5mQp42rTgWkp40qAwDMHFNhK5c854vHjgcAzB2vuaeESCmnsP6kL91HPiWNVlZTq9cEtNban52n1fhGErlXCjyjmWGYIUFKKaSOyc3W6yHbTd9U6NaHuQQv3nASNuzrNJ1/xsxRWHLTyUb1syGDpXjtpMNqse22c7FmV3uaHNJSkFPZVKtHbXRnDZrLz/LR+4iVAsMwQ4JMloLThq9aIX0ZazltVJlhGahYFQKQskCcspKEEIr7SHvSl8pALbizm9QmkQFwjikwDMPoZMo+cuMaytUAMyOm4JCVpK0xWwrqTGiJ2VKwVwqcfcQwDKNjO03Nk9lSUOmLpdAXvBb3kfV+apsLqRTkWjXQrCoFq6xEhIDXkxdLgd1HDMMMCeyyj7w2T9yO5+fIVDAa4lmur47glOIZSsGweuzdR3Z9nO6/cj7GVhWlHR9oWCkwDDMkMBSAzZAdN+6jvmQf9QV5Wevl7TKfZExhUq3W8XV+Q6qALmDqlZR+n5MOy88sGVYKDMMMCWSA2Wfjq5d6YnJtCU6fXmd/fq6CCjppc5/1PV6d0RzW21TMm1CFf33rZEzWlQNgVgpu03RzASsFhmGGBF4j/TTd9y6fxJfcdIrj+S777vUZo5uFZR83itiQHmgO+DyYMtKcyeTUYiPfDA4pGIZhskA26afGXAIXT9Zu22n0laRj+wvtt6l4TXcfBWxiBnbHCsHgkIJhGMYlagsIIgKRy0BzjmIK0lRIUzqUWmDtfWQXSA6wpcAwDNN3rJuvl+yrma3YWRMDoSek+yhdJ6QHxGVKqZ0CGCxKgWMKDMMMCWSDOetG7vGQK/eRXfB2xU/OQCRxcK0jnBrlqZ9Li6YnGgdg39F1sLiPWCkwDDOkSKscdpilYMVuTUWxH4D/oORJxRTM1w/5tU1+xphyAEBtaRD7OrQRn2wpMAzDHCTq0BqVmrIAakoDWc/P1YO4k/uoLOTH4q8twGF1Wg+liTUl2K/PfbYNNLNSYBiG6TvWJ/Jnvr4QxUGHMZoKuco+Eg5jOgFg3oRq4/XEmhK8v7UFQGp+s8pgSUllpcAwzJDAwVBAVUl2KwHIXfaR7KZ6ybyxGddNrNEK1eori4zJayoBb3bFlg9YKTAMc0iQK0thdEWRaVqbEzWl2mS1o8ZV2n7O7iOGYZg8kiul4JbTp9fh3Nmj8dPzZth+zkqBYRimH/R3b89Z8ZpLKor9uPuLcx0/HyxKYXBIwTAMkwWn7CO3DJIyAEcGS53C4JCCYRgmxxTafZQNu4K2QsBKgWGYIYFwzD9yx2BXCoVsl63CSoFhmCFFfzfPQscUhgqsFBiGGRIcbEwhV5PXhhusFBiGOSTI9eS14QIrBYZhDglyNXltuMFfE8MwQwLZTmJiTXG/zh/sgWbJvAlVBb0/F68xDDMkuPTocZg5phyzx9q3icjGUHAfbb31nEKLkDtLgYj+QET7iWiNw+dVRPQ0Ea0mog+IaFauZGEYZuhDRP1WCMDQCDRrw3gKK2cu3UcPATgrw+c/BLBSCDEbwBUA/jeHsjAMc4jDKanuyJlSEEK8AaAlw5IZAJboa9cBaCCiulzJwzDMoc1QcB8NBgoZaF4F4GIAIKJjAEwAkLkhOcMwTD9hneCOQiqF2wBUEdFKAN8AsAJA3G4hEV1DREuJaGlTU1M+ZWQYZpjA7iN3FCz7SAjRAeDfAIC0yMpW/cdu7b0A7gWA+fPnH2RdI8MwhyLsPnJHwSwFIqokIjlH7ysA3tAVBcMwzIAzFLKPBgM5sxSI6HEApwCoIaJGADcD8AOAEOIeANMBPExECQCfALg6V7IwDMMw7siZUhBCfCHL5+8CmJqr+zMMwzB9h9tcMAzDMAasFBiGYRgD7n3EMMyw5tlvLMTyHa2FFmPIwEqBYZhhzaz6Csyqryi0GEMGdh8xDMMwBqwUGIZhGANWCgzDMIwBKwWGYRjGgJUCwzAMY8BKgWEYhjFgpcAwDMMYsFJgGIZhDEiIoTWegIiaAGzv5+k1AA4MoDgDyWCVjeXqGyxX32C5+k5/ZZsghKjNtmjIKYWDgYiWCiHmF1oOOwarbCxX32C5+gbL1XdyLRu7jxiGYRgDVgoMwzCMwaGmFO4ttAAZGKyysVx9g+XqGyxX38mpbIdUTIFhGIbJzKFmKTAMwzAZOGSUAhGdRUTriWgTEX2/wLJsI6KPiGglES3Vj1UT0ctEtFH/XZUHOf5ARPuJaI1yzFYO0vi1/v2tJqK5eZbrP4lol/6drSSic5TPfqDLtZ6IzsyhXOOI6FUiWktEHxPRDfrxgn5nGeQaDN9ZiIg+IKJVumy36McnEtH7+nf2BBEF9ONB/f0m/fOGPMv1EBFtVb6zo/Tjefv3r9/PS0QriOhZ/X3+vi8hxLD/AeAFsBnAJAABAKsAzCigPNsA1FiO3QHg+/rr7wO4PQ9ynARgLoA12eQAcA6AFwAQgOMAvJ9nuf4TwLdt1s7Q/3sGAUzU/zt7cyTXaABz9ddlADbo9y/od5ZBrsHwnRGAUv21H8D7+nfxFwCX6sfvAfA1/fV1AO7RX18K4Ik8y/UQgEts1uft379+v28BeAzAs/r7vH1fh4qlcAyATUKILUKIKIA/A7igwDJZuQDAH/XXfwRwYa5vKIR4A0CLSzkuAPCw0HgPQCURjc6jXE5cAODPQoiIEGIrgE3Q/nvnQq49Qojl+utOAGsB1KPA31kGuZzI53cmhBBd+lu//iMALALwV/249TuT3+VfAZxGRJRHuZzI279/IhoL4FwA9+vvCXn8vg4VpVAPYKfyvhGZ/6fJNQLAP4loGRFdox+rE0LsAbT/yQGMLJBsTnIMhu/wet10/4PiXiuIXLqZPgfaE+ag+c4scgGD4DvTXSErAewH8DI0y6RNCBG3ub8hm/55O4AR+ZBLCCG/s1/o39l/E1HQKpeNzAPN/wD4LoCk/n4E8vh9HSpKwU5zFjLt6gQhxFwAZwP4OhGdVEBZ3FLo7/B3ACYDOArAHgB36sfzLhcRlQJYDOBGIURHpqU2x3Imm41cg+I7E0IkhBBHARgLzSKZnuH+eZPNKhcRzQLwAwCHAzgaQDWA7+VTLiI6D8B+IcQy9XCGew+4XIeKUmgEME55PxbA7gLJAiHEbv33fgBPQ/sfZZ80R/Xf+wsknpMcBf0OhRD79P+JkwDuQ8rdkVe5iMgPbeN9VAjxlH644N+ZnVyD5TuTCCHaALwGzSdfSUQ+m/sbsumfV8C9K/Fg5TpLd8UJIUQEwIPI/3d2AoDziWgbNDf3ImiWQ96+r0NFKXwIYKoewQ9AC8g8UwhBiKiEiMrkawBnAFijy3OlvuxKAH8vhHwZ5HgGwBV6FsZxANqlyyQfWPy3F0H7zqRcl+pZGBMBTAXwQY5kIAAPAFgrhLhL+aig35mTXIPkO6slokr9dRGA06HFPF4FcIm+zPqdye/yEgCvCD2Kmge51inKnaD57dXvLOf/LYUQPxBCjBVCNEDbp14RQlyGfH5fAxkxH8w/0LIHNkDzZ/6ogHJMgpb5sQrAx1IWaH7AJQA26r+r8yDL49DcCjFoTxxXO8kBzUy9W//+PgIwP89yPaLfd7X+P8JoZf2PdLnWAzg7h3IthGaarwawUv85p9DfWQa5BsN3NhvACl2GNQB+qvx/8AG0IPeTAIL68ZD+fpP++aQ8y/WK/p2tAfAnpDKU8vbvX5HxFKSyj/L2fXFFM8MwDGNwqLiPGIZhGBewUmAYhmEMWCkwDMMwBqwUGIZhGANWCgzDMIwBKwXmkIaIEno3zFVEtJyIjs+yvpKIrnNx3deIyPUcXSJ6nIgaiOhGIrrU7XkMM9CwUmAOdXqFEEcJIY6E1uLg1izrK6F1phxoJgohtgE4GcCbObg+w7iClQLDpCgH0ApofYSIaIluPXxERLKr7m0AJuvWxa/0td/V16wiotuU632WtJ79G4joRLsbEtGjRPQJgGl6c7YzADxHRF/J2V/JMBnwZV/CMMOaIn0zDkGbS7BIPx4GcJEQooOIagC8R0TPQJuXMEtojdRARGdDa4dwrBCih4iqlWv7hBDHkDbc5mZorRRMCCEuI6LPQetfsxjAr4QQn83Nn8ow2WGlwBzq9Cob/AIAD+vdMgnAL/UOtkloLYrrbM4/HcCDQogeABBCqM3IZMO8ZQAaMsgwB8C/ABwBrUUFwxQMVgoMoyOEeFe3Cmqh9Q6qBTBPCBHTu1aGbE4jOLcqjui/E7D5f023IH4JbfrZefr9uonodCHEqQfztzBMf+GYAsPoENHh0Ea3NkNrQbxfVwinApigL+uENvJS8k8AVxFRsX4N1X2UESHE8wDmQRs7egS0BolzWCEwhYQtBeZQR8YUAO2p/0ohRIKIHgXwDyJaCs2lsw4AhBDNRPQ2Ea0B8IIQ4jukDXdfSkRRAM8D+GEf7j8HwCq9pbtfZB7awzA5h7ukMgzDMAbsPmIYhmEMWCkwDMMwBqwUGIZhGANWCgzDMIwBKwWGYRjGgJUCwzAMY8BKgWEYhjFgpcAwDMMY/H9oW/HHSkzvIwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "plt.plot(loss_history)\n",
    "plt.xlabel('Batch #')\n",
    "plt.ylabel('Loss [entropy]')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Variables and optimizers\n",
    "\n",
    "`tf.Variable` objects store mutable `tf.Tensor` values accessed during\n",
    "training to make automatic differentiation easier. The parameters of a model can\n",
    "be encapsulated in classes as variables.\n",
    "\n",
    "Better encapsulate model parameters by using `tf.Variable` with\n",
    "`tf.GradientTape`. For example, the automatic differentiation example above\n",
    "can be rewritten:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Initial loss: 70.138\nLoss at step 000: 67.347\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loss at step 020: 30.115\nLoss at step 040: 13.786\nLoss at step 060: 6.617\nLoss at step 080: 3.467\nLoss at step 100: 2.082"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\nLoss at step 120: 1.472\nLoss at step 140: 1.203\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loss at step 160: 1.084\nLoss at step 180: 1.032\nLoss at step 200: 1.009\nLoss at step 220: 0.998\nLoss at step 240: 0.994"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\nLoss at step 260: 0.992\nLoss at step 280: 0.991\nFinal loss: 0.991\nW = 2.997786283493042, B = 2.0234856605529785\n"
     ]
    }
   ],
   "source": [
    "class Model(tf.keras.Model):\n",
    "  def __init__(self):\n",
    "    super(Model, self).__init__()\n",
    "    self.W = tf.Variable(5., name='weight')\n",
    "    self.B = tf.Variable(10., name='bias')\n",
    "  def call(self, inputs):\n",
    "    return inputs * self.W + self.B\n",
    "\n",
    "# A toy dataset of points around 3 * x + 2\n",
    "NUM_EXAMPLES = 2000\n",
    "training_inputs = tf.random_normal([NUM_EXAMPLES])\n",
    "noise = tf.random_normal([NUM_EXAMPLES])\n",
    "training_outputs = training_inputs * 3 + 2 + noise\n",
    "\n",
    "# The loss function to be optimized\n",
    "def loss(model, inputs, targets):\n",
    "  error = model(inputs) - targets\n",
    "  return tf.reduce_mean(tf.square(error))\n",
    "\n",
    "def grad(model, inputs, targets):\n",
    "  with tf.GradientTape() as tape:\n",
    "    loss_value = loss(model, inputs, targets)\n",
    "  return tape.gradient(loss_value, [model.W, model.B])\n",
    "\n",
    "# Define:\n",
    "# 1. A model.\n",
    "# 2. Derivatives of a loss function with respect to model parameters.\n",
    "# 3. A strategy for updating the variables based on the derivatives.\n",
    "model = Model()\n",
    "optimizer = tf.train.GradientDescentOptimizer(learning_rate=0.01)\n",
    "\n",
    "print(\"Initial loss: {:.3f}\".format(loss(model, training_inputs, training_outputs)))\n",
    "\n",
    "# Training loop\n",
    "for i in range(300):\n",
    "  grads = grad(model, training_inputs, training_outputs)\n",
    "  optimizer.apply_gradients(zip(grads, [model.W, model.B]),\n",
    "                            global_step=tf.train.get_or_create_global_step())\n",
    "  if i % 20 == 0:\n",
    "    print(\"Loss at step {:03d}: {:.3f}\".format(i, loss(model, training_inputs, training_outputs)))\n",
    "\n",
    "print(\"Final loss: {:.3f}\".format(loss(model, training_inputs, training_outputs)))\n",
    "print(\"W = {}, B = {}\".format(model.W.numpy(), model.B.numpy()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Use objects for state during eager execution\n",
    "\n",
    "With graph execution, program state (such as the variables) is stored in global\n",
    "collections and their lifetime is managed by the `tf.Session` object. In\n",
    "contrast, during eager execution the lifetime of state objects is determined by\n",
    "the lifetime of their corresponding Python object.\n",
    "\n",
    "### Variables are objects\n",
    "\n",
    "During eager execution, variables persist until the last reference to the object\n",
    "is removed, and is then deleted."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [],
   "source": [
    "if tf.test.is_gpu_available():\n",
    "  with tf.device(\"gpu:0\"):\n",
    "    v = tf.Variable(tf.random_normal([1000, 1000]))\n",
    "    v = None  # v no longer takes up GPU memory"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf.test.is_gpu_available()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Object-based saving\n",
    "\n",
    "`tf.train.Checkpoint` can save and restore `tf.Variable`s to and from\n",
    "checkpoints:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = tf.Variable(10.)\n",
    "checkpoint = tf.train.Checkpoint(x=x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'./ckpt/-1'"
      ]
     },
     "execution_count": 69,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x.assign(2.)   # Assign a new value to the variables and save.\n",
    "checkpoint_path = './ckpt/'\n",
    "checkpoint.save('./ckpt/')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<tf.Variable 'Variable:0' shape=() dtype=float32, numpy=2.0>\n"
     ]
    }
   ],
   "source": [
    "x.assign(11.)  # Change the variable after saving.\n",
    "\n",
    "# Restore values from the checkpoint\n",
    "checkpoint.restore(tf.train.latest_checkpoint(checkpoint_path))\n",
    "\n",
    "print(x)  # => 2.0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To save and load models, `tf.train.Checkpoint` stores the internal state of objects,\n",
    "without requiring hidden variables. To record the state of a `model`,\n",
    "an `optimizer`, and a global step, pass them to a `tf.train.Checkpoint`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.training.checkpointable.util.CheckpointLoadStatus at 0x1a805b436a0>"
      ]
     },
     "execution_count": 71,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import os\n",
    "import tempfile\n",
    "\n",
    "model = tf.keras.Sequential([\n",
    "  tf.keras.layers.Conv2D(16,[3,3], activation='relu'),\n",
    "  tf.keras.layers.GlobalAveragePooling2D(),\n",
    "  tf.keras.layers.Dense(10)\n",
    "])\n",
    "optimizer = tf.train.AdamOptimizer(learning_rate=0.001)\n",
    "checkpoint_dir = tempfile.mkdtemp()\n",
    "checkpoint_prefix = os.path.join(checkpoint_dir, \"ckpt\")\n",
    "root = tf.train.Checkpoint(optimizer=optimizer,\n",
    "                           model=model,\n",
    "                           optimizer_step=tf.train.get_or_create_global_step())\n",
    "\n",
    "root.save(checkpoint_prefix)\n",
    "root.restore(tf.train.latest_checkpoint(checkpoint_dir))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Object-oriented metrics\n",
    "\n",
    "`tfe.metrics` are stored as objects. Update a metric by passing the new data to\n",
    "the callable, and retrieve the result using the `tfe.metrics.result` method,\n",
    "for example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: id=62827, shape=(), dtype=float64, numpy=5.5>"
      ]
     },
     "execution_count": 73,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "m = tfe.metrics.Mean(\"loss\")\n",
    "m(0)\n",
    "m(5)\n",
    "m.result()  # => 2.5\n",
    "m([8, 9])\n",
    "m.result()  # => 5.5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Summaries and TensorBoard\n",
    "\n",
    "[TensorBoard](../guide/summaries_and_tensorboard.md) is a visualization tool for\n",
    "understanding, debugging and optimizing the model training process. It uses\n",
    "summary events that are written while executing the program.\n",
    "\n",
    "`tf.contrib.summary` is compatible with both eager and graph execution\n",
    "environments. Summary operations, such as `tf.contrib.summary.scalar`, are\n",
    "inserted during model construction. For example, to record summaries once every\n",
    "100 global steps:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [],
   "source": [
    "global_step = tf.train.get_or_create_global_step()\n",
    "\n",
    "logdir = \"./tb/\"\n",
    "writer = tf.contrib.summary.create_file_writer(logdir)\n",
    "writer.set_as_default()\n",
    "\n",
    "for _ in range(10):\n",
    "  global_step.assign_add(1)\n",
    "  # Must include a record_summaries method\n",
    "  with tf.contrib.summary.record_summaries_every_n_global_steps(100):\n",
    "    # your model code goes here\n",
    "    tf.contrib.summary.scalar('global_step', global_step)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Advanced automatic differentiation topics\n",
    "\n",
    "### Dynamic models\n",
    "\n",
    "`tf.GradientTape` can also be used in dynamic models. This example for a\n",
    "[backtracking line search](https://wikipedia.org/wiki/Backtracking_line_search)\n",
    "algorithm looks like normal NumPy code, except there are gradients and is\n",
    "differentiable, despite the complex control flow:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [],
   "source": [
    "def line_search_step(fn, init_x, rate=1.0):\n",
    "  with tf.GradientTape() as tape:\n",
    "    # Variables are automatically recorded, but manually watch a tensor\n",
    "    tape.watch(init_x)\n",
    "    value = fn(init_x)\n",
    "  grad = tape.gradient(value, init_x)\n",
    "  grad_norm = tf.reduce_sum(grad * grad)\n",
    "  init_value = value\n",
    "  while value > init_value - rate * grad_norm:\n",
    "    x = init_x - rate * grad\n",
    "    value = fn(x)\n",
    "    rate /= 2.0\n",
    "  return x, value"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Additional functions to compute gradients\n",
    "\n",
    "`tf.GradientTape` is a powerful interface for computing gradients, but there\n",
    "is another [Autograd](https://github.com/HIPS/autograd)-style API available for\n",
    "automatic differentiation. These functions are useful if writing math code with\n",
    "only tensors and gradient functions, and without `tf.variables`:\n",
    "\n",
    "* `tfe.gradients_function` —Returns a function that computes the derivatives\n",
    "  of its input function parameter with respect to its arguments. The input\n",
    "  function parameter must return a scalar value. When the returned function is\n",
    "  invoked, it returns a list of `tf.Tensor` objects: one element for each\n",
    "  argument of the input function. Since anything of interest must be passed as a\n",
    "  function parameter, this becomes unwieldy if there's a dependency on many\n",
    "  trainable parameters.\n",
    "* `tfe.value_and_gradients_function` —Similar to\n",
    "  `tfe.gradients_function`, but when the returned function is invoked, it\n",
    "  returns the value from the input function in addition to the list of\n",
    "  derivatives of the input function with respect to its arguments.\n",
    "\n",
    "In the following example, `tfe.gradients_function` takes the `square`\n",
    "function as an argument and returns a function that computes the partial\n",
    "derivatives of `square` with respect to its inputs. To calculate the derivative\n",
    "of `square` at `3`, `grad(3.0)` returns `6`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [],
   "source": [
    "def square(x):\n",
    "  return tf.multiply(x, x)\n",
    "\n",
    "grad = tfe.gradients_function(square)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "9.0"
      ]
     },
     "execution_count": 78,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "square(3.).numpy()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "6.0"
      ]
     },
     "execution_count": 79,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grad(3.)[0].numpy()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2.0"
      ]
     },
     "execution_count": 80,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# The second-order derivative of square:\n",
    "gradgrad = tfe.gradients_function(lambda x: grad(x)[0])\n",
    "gradgrad(3.)[0].numpy()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[None]"
      ]
     },
     "execution_count": 81,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# The third-order derivative is None:\n",
    "gradgradgrad = tfe.gradients_function(lambda x: gradgrad(x)[0])\n",
    "gradgradgrad(3.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {},
   "outputs": [],
   "source": [
    "# With flow control:\n",
    "def abs(x):\n",
    "  return x if x > 0. else -x\n",
    "\n",
    "grad = tfe.gradients_function(abs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1.0"
      ]
     },
     "execution_count": 83,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grad(3.)[0].numpy()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-1.0"
      ]
     },
     "execution_count": 84,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grad(-3.)[0].numpy()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Custom gradients\n",
    "\n",
    "Custom gradients are an easy way to override gradients in eager and graph\n",
    "execution. Within the forward function, define the gradient with respect to the\n",
    "inputs, outputs, or intermediate results. For example, here's an easy way to clip\n",
    "the norm of the gradients in the backward pass:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {},
   "outputs": [],
   "source": [
    "@tf.custom_gradient\n",
    "def clip_gradient_by_norm(x, norm):\n",
    "  y = tf.identity(x)\n",
    "  def grad_fn(dresult):\n",
    "    return [tf.clip_by_norm(dresult, norm), None]\n",
    "  return y, grad_fn"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Custom gradients are commonly used to provide a numerically stable gradient for a\n",
    "sequence of operations:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {},
   "outputs": [],
   "source": [
    "def log1pexp(x):\n",
    "  return tf.log(1 + tf.exp(x))\n",
    "grad_log1pexp = tfe.gradients_function(log1pexp)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.5"
      ]
     },
     "execution_count": 87,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# The gradient computation works fine at x = 0.\n",
    "grad_log1pexp(0.)[0].numpy() "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "nan"
      ]
     },
     "execution_count": 88,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# However, x = 100 fails because of numerical instability.\n",
    "grad_log1pexp(100.)[0].numpy()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here, the `log1pexp` function can be analytically simplified with a custom\n",
    "gradient. The implementation below reuses the value for `tf.exp(x)` that is\n",
    "computed during the forward pass—making it more efficient by eliminating\n",
    "redundant calculations:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {},
   "outputs": [],
   "source": [
    "@tf.custom_gradient\n",
    "def log1pexp(x):\n",
    "  e = tf.exp(x)\n",
    "  def grad(dy):\n",
    "    return dy * (1 - 1 / (1 + e))\n",
    "  return tf.log(1 + e), grad\n",
    "\n",
    "grad_log1pexp = tfe.gradients_function(log1pexp)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.5"
      ]
     },
     "execution_count": 91,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# As before, the gradient computation works fine at x = 0.\n",
    "grad_log1pexp(0.)[0].numpy()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1.0"
      ]
     },
     "execution_count": 92,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# And the gradient computation also works at x = 100.\n",
    "grad_log1pexp(100.)[0].numpy()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Performance\n",
    "\n",
    "Computation is automatically offloaded to GPUs during eager execution. If you\n",
    "want control over where a computation runs you can enclose it in a\n",
    "`tf.device('/gpu:0')` block (or the CPU equivalent):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Time to multiply a (1000, 1000) matrix by itself 200 times:\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU: 7.271521091461182 secs\nGPU: not found\n"
     ]
    }
   ],
   "source": [
    "import time\n",
    "\n",
    "def measure(x, steps):\n",
    "  # TensorFlow initializes a GPU the first time it's used, exclude from timing.\n",
    "  tf.matmul(x, x)\n",
    "  start = time.time()\n",
    "  for i in range(steps):\n",
    "    x = tf.matmul(x, x)\n",
    "  # tf.matmul can return before completing the matrix multiplication\n",
    "  # (e.g., can return after enqueing the operation on a CUDA stream).\n",
    "  # The x.numpy() call below will ensure that all enqueued operations\n",
    "  # have completed (and will also copy the result to host memory,\n",
    "  # so we're including a little more than just the matmul operation\n",
    "  # time).\n",
    "  _ = x.numpy()\n",
    "  end = time.time()\n",
    "  return end - start\n",
    "\n",
    "shape = (1000, 1000)\n",
    "steps = 200\n",
    "print(\"Time to multiply a {} matrix by itself {} times:\".format(shape, steps))\n",
    "\n",
    "# Run on CPU:\n",
    "with tf.device(\"/cpu:0\"):\n",
    "  print(\"CPU: {} secs\".format(measure(tf.random_normal(shape), steps)))\n",
    "\n",
    "# Run on GPU, if available:\n",
    "if tfe.num_gpus() > 0:\n",
    "  with tf.device(\"/gpu:0\"):\n",
    "    print(\"GPU: {} secs\".format(measure(tf.random_normal(shape), steps)))\n",
    "else:\n",
    "  print(\"GPU: not found\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A `tf.Tensor` object can be copied to a different device to execute its\n",
    "operations:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {},
   "outputs": [],
   "source": [
    "if tf.test.is_gpu_available():\n",
    "  x = tf.random_normal([10, 10])\n",
    "\n",
    "  x_gpu0 = x.gpu()\n",
    "  x_cpu = x.cpu()\n",
    "\n",
    "  _ = tf.matmul(x_cpu, x_cpu)    # Runs on CPU\n",
    "  _ = tf.matmul(x_gpu0, x_gpu0)  # Runs on GPU:0\n",
    "\n",
    "  if tfe.num_gpus() > 1:\n",
    "    x_gpu1 = x.gpu(1)\n",
    "    _ = tf.matmul(x_gpu1, x_gpu1)  # Runs on GPU:1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Benchmarks\n",
    "\n",
    "For compute-heavy models, such as ResNet50 training on a GPU, eager execution performance is comparable to graph execution. But this gap grows larger for models with less computation and there is work to be done for optimizing hot code paths for models with lots of small operations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}