{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "# Predictions with Pyro + QPyTorch (High-Level Interface)\n",
    "\n",
    "## Overview\n",
    "\n",
    "In this example, we will give an overview of the high-level Pyro-QPyTorch integration - designed for predictive models.\n",
    "This will introduce you to the key GPyTorch objects that play with Pyro. Here are the key benefits of the integration:\n",
    "\n",
    "**Pyro provides:**\n",
    "\n",
    "- The engines for performing approximate inference or sampling\n",
    "- The ability to define additional latent variables\n",
    "\n",
    "**GPyTorch provides:**\n",
    "\n",
    "- A library of kernels/means/likelihoods\n",
    "- Mechanisms for efficient QEP computations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import math\n",
    "import torch\n",
    "import pyro\n",
    "import tqdm\n",
    "import qpytorch\n",
    "from matplotlib import pyplot as plt\n",
    "\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this example, we will be doing simple variational regression to learn a monotonic function. This example is doing the exact same thing as [QPyTorch's native approximate inference](../04_Variational_and_Approximate_QEPs/SVQEP_Regression_CUDA.ipynb), except we're now using Pyro's variational inference engine.\n",
    "\n",
    "In general - if this was your dataset, you'd be better off using QPyTorch's native exact or approximate QEPs.\n",
    "(We're just using a simple example to introduce you to the QPyTorch/Pyro integration)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0x12cf9f430>"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAT0AAADaCAYAAADUrZUTAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjcsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvTLEjVAAAAAlwSFlzAAAPYQAAD2EBqD+naQAAIKBJREFUeJztnQlwFFX+x39JzIGIAQRySEi4lnXlCIQNC7VUoKC4sm5YCoWwQGAFXf7srpgVMbscIpbhKsCFLFis4dDlUAxgrYgcGw41gBVcLgEJBgRJIICJJJxJ3r9+L/Y4M5lJupPJTB/fT1UzdM/rnu7p6W/ee7/LTwghCAAALIK/r08AAAC8CUQPAGApIHoAAEsB0QMAWAqIHgDAUkD0AACWAqIHALAUED0AgKWA6AEALMVDvj4BPVJZWUlXrlyhJk2akJ+fn69PBwDgBAeS3bp1iyIjI8nfX2PfTfiQ/fv3i9/85jciIiKCQ+HE1q1ba90nOztbdO/eXQQFBYn27duLNWvWVGuzYsUKER0dLYKDg0V8fLw4fPiwpvO6dOmSPB8s+A7wGyBdfwf8rGrFpz29srIy6tatG/3hD3+gESNG1No+Pz+fEhMT6Y9//CP9+9//pr1799KkSZMoIiKCBg8eLNts3ryZUlNTadWqVdSrVy9atmyZfO/s2bPUqlUrVefFPTzm0qVL9Oijj9bzKgEAnuaHH36gqKgo27OqBT9WPtIBPIzcunUrDR8+3G2bGTNm0EcffUQnT560bRs9ejQVFxfTzp075ToL3S9/+UtasWKFbajKX86f//xneuWVV1R/oaGhoVRSUgLRA8ALVFQQHTxIVFBAFBFB1LcvUUBAwzyjhjJk5OTk0MCBAx22cS+OtzP379+n3NxchzY83ud1pY0r7t27J79E+wUA4B2ysohiYoj69ycaM6bqldd5e0NgKNErLCyksLAwh228ziJ1584dun79OlVUVLhsw/u6Iz09Xf7VUBbuGQIAGh4WtpEjiS5fdtz+3XdV2xtC+Awleg1FWlqa7CYrC8/lAQAafkj7wgtsTK3+nrJt2rSqdpZ1WQkPD6erV686bON1HtM3atSIAgIC5OKqDe/rjuDgYLlogadCy8vLZc8SAHfw7/Ghhx6C65MLeA7PuYfnLHzc/+B2/fqRNUWvd+/etGPHDodtu3fvltuZoKAgiouLk1ZdxSDChgxe/9Of/uSx8+C5w4KCArp9+7bHjgnMy8MPPyw9DPj3CX6CjRaebGcI0SstLaW8vDwHl5T//e9/1Lx5c2rTpo0cdn733Xe0fv16+T67qrBV9uWXX5ZuLv/973/pvffekxZdBXZXSUlJoZ49e1J8fLx0WWHXmIkTJ3rknFlE+Tz5Lzg7RvIPGQ7MwN1ogP9AFhUVyd9Mx44dtTvSmpiICM+2U43wIexo7MrhMCUlRb7PrwkJCdX2iY2Nlc7J7dq1c+mcvHz5ctGmTRvZhp2TDx06pOm8SkpK5HnwqzN37twRX331lSgrK9N8vcCa8G+FfzP82wE/UV4uROvWQvj58UC2+sLbo6Kq2ml5RmtDN356eqImH6C7d+/Kv9pt27alkJAQn50jMA74zdRuvWXslUiJ/tyyhchV3IJl/PQAAOZixIgqYXv8ccftrVu7FzxLGTIAAOZjxAiipCRtERn1AT09H8GeLvv2EW3cWPVqRM+XmJgYaShSy759+6TRh8MGvc3atWupadOmXv9coA4WOHZLSU6uem0owWMgehYIu2GhqWl59dVX63TcL774gp577jnV7fv06SNdfXguxoyiDowBhrc+mrh1Nh8pYTcNMY/BQqPAWWhmz54ts84oPPLII7b/s12LHa7ZobY2WrZsqek82L2nJidxYO0kAt4CPT0LhN2w0CgL97K4d6esnzlzRqbn+fjjj6VjN0emfPrpp3T+/HlKSkqSccssipy5Zs+ePTX2hPi4//rXv+h3v/uddMhlv7QPP/zQ7fBWGXJ+8skn9MQTT8jPGTJkiINIc9TLX/7yF9nusccek5l22A+zpmw8yrHZ15PPg8/nxo0bDu/Xdn39+vWjixcv0osvvmjrETN8nOTkZHr88cflsbt06UIbeY4C+HQ0owWInk7DbrwNp92aP38+nT59mrp27Sodx4cNGyajWb788kspRk899RR9++23NR5n7ty59Mwzz9Dx48fl/r///e/p5s2bbttzVMvixYvpnXfeoQMHDsjjv/TSS7b3FyxYIHMnrlmzhj777DPpqrBt27Yaz+Hw4cP07LPPyigcdnbv378/vf766w5taru+rKwsat26Nb322mtShBUhZvcT/uOgpDjj4f24cePoyJEjqr5nK5DlgyQCmtDs2WcB1Dgn18XRdMMG106Yzgu3ayjYmTs0NLSag/i2bdtq3ffJJ5+Ujt8KnJ166dKltnU+zsyZM23rpaWlctvHH3/s8Fnff/+97Vx4PS8vz7ZPRkaGCAsLs63z/xctWmRbLy8vl47nSUlJbs8zOTlZDBs2zGHbqFGjHK67LtfnjsTERPHXv/7V7fv1+c0Y1eGYSLvDsRbq45yMnp4Vwm5UwGF7zj0h7nHxsJOHljwE5F5gbT097iUqNG7cWDqOXrt2zW17HiK2b9/ets4xqkp7djzlZBEcTqjA4X/c06oJPk9OJmuPEp9d3+vj+c558+bJYS2HS/J+PDyvbT+rcFDHoxkFGDK8CE/kstMld/NdzevxtBG/z+28DQuUPSwInMyBh54dOnSQWWxGjhwpY0lrIjAw0GGd58I4XllLe28ECdX1+hYtWkRvvvmmnMtk4ePvbdq0abXuZxUKfJREQAvo6XkRtly9+WbV/52LrCnrbBfQg4WL588mTJggjQD8cLPR48KFC149Bza6sKGBXWPse1pHjx6tcT/uvfG8nj2HDh3SfH1sbXZOHcb7sQFk7Nixsr5Lu3bt6Ouvv67HVZqLCB2PZhQgehYIu6kLbHnlyXw2BBw7dozGjBlTY4+toeDaJpzZevv27dLN5oUXXqDvv/++xsw2bO3lmincizt37pzMzKPUUNFyfWydZuMKZ/rhrNzKftxD/Pzzz+Vw+Pnnn6+Wv9HK9P1xNOPu9vB2Tkzui9GMAkTPB7CwcaciO5tow4aq1/x8/Qges2TJEmrWrJl0KGarJtci6dGjh9fPg11U2EVk/Pjxcl6O59D4XGpK9vCrX/2KVq9eLYeh3BvbtWsXzZw5U/P1seWWe38856j4JPJxuB23Z7cW7iHW5j5jJQIMMJpBlhUXIMuKfuHeGA9f2S2GDQpGwIpZVrKyqnxS7Y0a3MNjwfPEH/f6ZFmBIQPoGnYQ5p5aQkKCrFrHQ1UWEB6OAv0ywstJBLQA0QO6hjMNc3QFW1vZqtu5c2cZOcG9PWCMJAJ6A6IHdA2X42SLKdAPFTqNqVULRA8AUK+5OrbWsvFCT4a4moD1to4gyz6w2m8lS+8xtSqB6GlEiSBA+UegFuW34hx9YiQqfJQhqCHA8FYjHPvJsZpKfCjHjqIEJHDXw2PB498K/2b4t2NUDvqoMHdDANGrA0oizJoC6QFQYMEzevLUAgPE1KoFolcHuGfH2UBatWpFDx488PxdAaaBh7RG7uEZKaZWLRC9esA/ZjP8oAEwcoYgrcCQAQAwRUytoUQvIyNDZrTguERO/lhT6m0O8nZV0SsxMdHWhlMGOb/P6cABAObPEKT74S1X50pNTaVVq1ZJwePkjJzBgtMI8ZyZM5wOyD5hIxdq4UwaTz/9tEM7Fjmuq6DABW8AAOaNqTWM6HGKn8mTJ9PEiRPlOosfF13JzMyUxWqc4RTd9mzatEm6jTiLHoucWosZB7LzYp/BAQArUaEhtEyvMbWGGN5yjy03N5cGDhz40wn5+8v1nJwcVcd4++23afTo0dXSnXO5Qe4pdurUiaZMmVKtBKA9nKSS09QoC8d7AmAVsnRcrtF0osfZaDkdN6cEt4fXCwsLa92f5/64DN+kSZOqDW3Xr18vy/txCcH9+/fT0KFDq6X+VkhLS5N5uZTlEntZAmABskwSWmao4W194F4e1zewr5bFcM9Pgd/nCl2c/ZZ7fwMGDKh2HB4KY84PWI2KWkLL2CrLoWU8h2ekOTtd9/RatGgh/dycawzwem3zcWVlZXI+j4s61wYXb+HPysvLq/c5A2AWDhqgXKPpRI+rTXENUx6G2qcD53XnOqXOvP/++9L4wFWpauPy5ctyTo+jKAAA5gstM5SfHrurcBGXdevWyepSbHTgXpxizeWCMDzn5mpoywVZHnvssWpFnKdPny5L/nFRFxZQLtnHtU3ZFQYAYL7QMkPN6Y0aNYqKiopo9uzZ0ngRGxsry/Upxg2uHM8WXXvYh+/TTz+VtROc4eHy8ePHpYgWFxdTZGQkDRo0SBaRwbwdAOYMLdMCqqF5uNISAEa03jL2wqeEluk10qI+z6jPh7cAAN8xwiShZYYa3gIAfMsIE4SWaQGiBwAgo4eWaQHDWwCApYDoAQAsBUQPAGApIHoAAEsBQwYAJkZLnjyrANEDwMSOx5xFxT6pQOvWVbUuzOh/pxYMbwEwIVbMk6cWiB4AFsuTx3CePDc5dU0PRA8Ak2HVPHlqgegBYDKsmidPLRA9AEyGVfPkqQWiB4BJ8+Qp6aGc8fMj4oJ/ZsuTpxaIHgAmg/3w2C2FcRY+vx/Xly2zrr8eRA8AE2LFPHlqgXMyACaNsrBanjy1QPQAMHGUhZXy5KkFw1sADAKiLDwDRA8AA4AoC88B0QPAACDKwnNA9AAwAIiy8BwQPQAMAKIsTCZ6GRkZFBMTQyEhIdSrVy86cuSI27Zr164lPz8/h4X3s0cIQbNnz6aIiAhq1KgRDRw4kM6dO+eFKwGgYUCUhYlEb/PmzZSamkpz5syho0ePUrdu3Wjw4MF07do1t/twRfOCggLbcvHiRYf3Fy5cSP/4xz9o1apVdPjwYWrcuLE85t27d71wRQB4HkRZeBDhY+Lj48XUqVNt6xUVFSIyMlKkp6e7bL9mzRoRGhrq9niVlZUiPDxcLFq0yLatuLhYBAcHi40bN6o6p5KSEs46Jl8B0BMffCBE69acHOqnJSqqaruVKKnHM+rTnt79+/cpNzdXDj8V/P395XpOTo7b/UpLSyk6OpqioqIoKSmJTp06ZXsvPz+fCgsLHY4ZGhoqh83ujnnv3j364YcfHBYAfOGWsm8f0caNVa+uknyyA/KFC0TZ2UQbNlS95udbO6xMK5pFLyUlhQ4cOECe4Pr161RRUUFhYWEO23mdhcsVnTp1oszMTNq+fTu9++67VFlZSX369KHLP7qoK/tpOWZ6eroURmVhMQXA247HMTFE/fsTjRlT9crrrtK6K1EWyclVr1YPK2tw0SspKZG9qI4dO9Ibb7xB33HSfS/Su3dvGj9+PMXGxlJCQgJlZWVRy5Yt6a233qrzMdPS0uR1KcslTisLgJdApIXORW/btm1S6KZMmSKNEGx1HTp0KG3ZsoUePHig6VgtWrSggIAAunr1qsN2Xg8PD1d1jMDAQOrevTvl5eXJdWU/LccMDg6WxhH7BQBvgEgL71OnOT3uWbHF9dixY9I62qFDBxo3bhxFRkbSiy++qNo9JCgoiOLi4mjv3r22bTxc5XXu0amBh8cnTpyQ7ilM27ZtpbjZH5Pn6Pg81R4TAG+BSAvvUy9DBruL7N69Wy7cYxs2bJgUoF/84he0dOlSVcdg8Vy9ejWtW7eOTp8+LXuQZWVlNHHiRPk+D2V5+Knw2muv0a5du+ibb76RLi5jx46VLiuTJk2S77Pf3rRp0+j111+nDz/8UJ4PH4MFefjw4fW5XAA8DiItfIBWc+/9+/fFli1bRGJioggMDBRxcXFi5cqVDqbjrKws0bRpU9XHXL58uWjTpo0ICgqSLiyHDh2yvZeQkCBSUlJs69OmTbO1DQsLE8OGDRNHjx6t5rYya9Ys+T67qgwYMECcPXtW9fnAZQV4i+xsR/cTdwu3A555Rv34H63zcDwETU5OpsmTJ0uDgjPFxcVyno3dR4wID4fZistGDczvgYae02MrLdsDXT2JnN6d8+XxowQrrWeeUc1JRHnY+vTTT1cL/bKnadOmhhU8ALyZ5ViJtBg5skrg7IUP9Sx0MqfHBouaBA8AoM33DvUsvIvm4a0VwPAWeMr3zvnpUnpvrorzqK19AahezyhEzwUQPeCJeTr7OhYODx3m6Xz6jPo8ywoAZgO+d/oGogeAh4Hvnb6B6AHgYZDlWN9A9ADwMMhyrG8gegB4GGQ51jcQPQAaAPje6RfNERkAAPXCl5QE3zu9AdEDQCNanIiVLMdAP0D0ANAYafHCC46Ox5wQgONnUafCGGBODwCVhXmQ1t0cQPQAUJEcAGndzQOGt8DyuEsOwDnueDsnB2je3H0sLcP7cj0pnuvDHJ6+QU8PWBq1PTi1Rf/UhqAB3wHRA5ZGbXKAoiLPhqAB3wHRA5ZGbc+sZcsqK62SD88Z3s414tl9BegbiB6wtEVWbc/s8cer3FIYZ+FDWndjAdEDlk7XriU5AELLzAEyJ7sAmZOtla5dacu4KszjnNodad19D9LF6+gLBcZM1+4q0oJ7eMuWIdKCrF4CEgAzpWtXfOqQHMA6QPSA6ahrunYkB7AGujBkZGRkUExMjKyn26tXLzpy5IjbtqtXr6a+fftSs2bN5DJw4MBq7SdMmEB+fn4Oy5AhQ7xwJUAPIF070LXobd68mVJTU2nOnDl09OhR6tatGw0ePJiuXbvmsv2+ffsoOTmZsrOzKScnh6KiomjQoEH0nZPLPItcQUGBbdnIPgvAEiBdO6gR4WPi4+PF1KlTbesVFRUiMjJSpKenq9q/vLxcNGnSRKxbt862LSUlRSQlJdX5nEpKStiGJ1+BMfngAyH8/KqWqlm8qkXZxu8D41KfZ9SnPb379+9Tbm6uHKIq+Pv7y3Xuxanh9u3b9ODBA2rOEeFOPcJWrVpRp06daMqUKXTjxg23x7h37560BtkvwNjApw7o0pBx/fp1qqiooLCwMIftvH7mzBlVx5gxYwZFRkY6CCcPbUeMGEFt27al8+fP09/+9jcaOnSoFNIAFylu09PTae7cuR64IqAnYJEFprPezp8/nzZt2iR7dWwEURg9erTt/126dKGuXbtS+/btZbsBAwZUO05aWpqcV1Tgnh7PFQLjA4sscManw9sWLVrIntfVq1cdtvN6eHh4jfsuXrxYit6uXbukqNVEu3bt5Gfl5eW5fD84OFg6ONovAABz4lPRCwoKori4ONq7d69tW2VlpVzv3bu32/0WLlxI8+bNo507d1LPnj1r/ZzLly/LOb0I5P0BwPL43GWFh5Xse7du3To6ffq0NDqUlZXRxIkT5fvjx4+Xw0+FBQsW0KxZsygzM1P69hUWFsqltLRUvs+v06dPp0OHDtGFCxekgCYlJVGHDh2kKwwAwNr4fE5v1KhRVFRURLNnz5biFRsbK3twinHj22+/lRZdhZUrV0qr70glQvxH2M/v1VdflcPl48ePSxEtLi6WRg724+OeIQ9jgTlA0D+oK8iy4gIkHNA3KMMI6vOM+nx4C4AWUIYR1BeIHjAMKMMIPAFED5gyZRQA7oDoAdOnjAJAV9ZbANRaZJEyCngCiB7QhXuJGouskjKKs4i5Ks6tpIFHGUZQExjeAp9XJFNrkWWhRBlGUF8geqDBUCNmWi2ySBkF6guck10A52TvVSRbs4bILiuYW7Kzfyrio3bIDMzLD6iGBrxNbaKj1r1k3z51n4ciPsBTwJABNKPG6OBptxEkyAGeAnN6QBNqjQ5qRYqHrCyYPNx1BW/nfK6wyAJPAdEDqtFidFBbkYxFDxZZ4E0geqBBwsC0uJfAIgu8CUQPNFgYmBYx4/9fuFBlpd2woeo1P9+xDQCeAIYMoJq6hIFpqUiGIj7AG0D0dEZD+J+pPWZt7eoaBgYxA7qiQcqPW7h6en344AMhWrdmOflp4XXe3tDH1NLOz69qsW+rbKvPuQLgjWcUoufhL7SuKGJiLyQ1iUl5uRDZ2UJs2FD1yut1PabWz3YlkFFREDxgjGcUYWg6CENTG7LFE/s8VFTjHKz2mFwKuH179Z9tf84IAwO+AmFoFnIFuXmzygnYeU5NcQ5WrKJqj/nPf6r/bPvYV8zTAaMClxUDuYKwsKl1DlZ7zPPnPXuOAOgdiJ6BXEGKitT3ytQek4e2njxHAPQORM8LcM+Ls4ls3Fj1quSGU1AbstWypfpemdpj/t//IfYVWAuIXgOKmdrMwWpDtpwjG2rqlak9ZlAQYl+BxRA6YMWKFSI6OloEBweL+Ph4cfjw4Rrbv/fee6JTp06yfefOncVHH33k8H5lZaWYNWuWCA8PFyEhIWLAgAHi66+/9qg5XI1fm6ddQdgthd93dUzluNze3n1FrXsJ3FCAkTC0n96mTZtEUFCQyMzMFKdOnRKTJ08WTZs2FVevXnXZ/rPPPhMBAQFi4cKF4quvvhIzZ84UgYGB4sSJE7Y28+fPF6GhoWLbtm3i2LFj4re//a1o27atuHPnjke+UDVipgiUK3FyJ1Bq/O/q4hysxqdPSzsAfI2hRY97dlOnTrWtV1RUiMjISJGenu6y/TPPPCMSExMdtvXq1Us8//zztl4e9/AWLVpke7+4uFj2Cjdu3FjvL1StmO3Z476N/cLiohX0yoDVKamH6Pl0Tu/+/fuUm5tLA+2KJPj7+8v1nJwcl/vwdvv2zODBg23t8/PzqbCw0KENOxr36tXL7THv3bsnnR3tF3c0dBp0NSAjCQAGTThw/fp1qqiooLCwMIftvH7mzBmX+7CguWrP25X3lW3u2jiTnp5Oc+fOVXXOekmDDudgAOoGrLdElJaWJkPOlOUSd9XcgDToABgbn4peixYtKCAggK5eveqwndfDw8Nd7sPba2qvvGo5ZnBwsIyxtV/cgTToABgbn4peUFAQxcXF0d69e23bKisr5Xrv3r1d7sPb7dszu3fvtrVv27atFDf7NjxHd/jwYbfH1ALSoANgcIQOXFbYsrp27VrpgvLcc89Jl5XCwkL5/rhx48Qrr7zi4LLy0EMPicWLF4vTp0+LOXPmuHRZ4WNs375dHD9+XCQlJXnUZUWrBRWuIAB4FkO7rDDLly8Xbdq0kf567MJy6NAh23sJCQkiJSWlmnPyz372M9n+ySefdOucHBYWJgWVnZPPnj3r8S8UYgaAb0A+PQ/DxoymTZtKg4Y38ukBALTBU1ZRUVFUXFwsXdK0gBoZLrh165Z85S8VAKDvZ1Wr6CFzsgvYmHLlyhVq0qQJ+blLU+L0F8csvUKzXQ+DazLffRJCSMGLjIyUAQ1aQE/PBfwltua8TBqozdXFaJjtehhckzFQe5+09vAU4JwMALAUED0AgKWA6NUTjuaYM2eOfDUDZrseBtdkDIK99NuDIQMAYCnQ0wMAWAqIHgDAUkD0AACWAqIHALAUED0nMjIyKCYmhkJCQmSK+SNHjtT4Bb7//vv085//XLbv0qUL7dixo5rn+OzZsykiIoIaNWok09ifO3eO9HpNq1evpr59+1KzZs3kwufr3H7ChAkyUsV+GTJkCOn1mtauXVvtfHk/I9+nfv36VbsmXhITE3Vxnw4cOEBPPfWUjJjgz922bVut++zbt4969OghrbcdOnSQ962+z6dLfJQkQZfosTKbt69pzJgxIiMjQ3z55ZcyddeECRPk+V++fNnWhrPeDBkyRBQUFNiWmzdveuV66nJNa9asEY8++qjD+Sqpy4x6n27cuOFwPSdPnpS/Rb5WPdynHTt2iL///e8iKytLZizaunVrje2/+eYb8fDDD4vU1FT5LHHmJb6enTt31vk7cgdET+eV2bx9Tc6Ul5eLJk2aiHXr1jk8TJyj0FdovSYWAhY0d5jhPi1dulTep9LSUt3cJwU1ovfyyy/LNHH2jBo1SgwePNhj35EChrc6q8zm62ty5vbt2/TgwQNq3rx5taFIq1atqFOnTjRlyhS6ceMGeYO6XlNpaSlFR0fLgPakpCQ6deqU7T0z3Ke3336bRo8eTY0bN9bFfdJKbc+SJ74j236az86k1FSZzV0VtYaozObra3JmxowZcl7G/sfG80Lr16+XKfkXLFhA+/fvp6FDh8rP0uM18QOfmZlJ27dvp3fffVdm0enTpw9d/rGWp9HvE89rnTx5kiZNmuSw3Zf3SSvuniXOvHLnzh2P/JYVkGUFuGX+/Pm0adMm2Vuwn/jnHoUCG2+6du1K7du3l+0GDBigu2+Ua6PY10dhwXviiSforbfeonnz5pHR4V4e34f4+HiH7Ua7T94CPT2dVWbz9TUpLF68WIrerl275MNSE+3atZOflZeXR3q+JoXAwEDq3r277XyNfJ/KysrkH6Znn3221s/x5n3SirtniVNMsTXdE/ddAaJn4MpsDXFNzMKFC2UPaOfOndSzZ89aP4eHiTxXxO4eer0me3iYdOLECdv5GvU+KS5T9+7do7Fjx+rqPmmltmfJE/fdhiazh8nRY2U2b18Tny+7BWzZssXB1eHWrVvyfX596aWXRE5OjsjPzxd79uwRPXr0EB07dhR3797V5TXNnTtXfPLJJ+L8+fMiNzdXjB49WoSEhEi3B6PeJ4Vf//rX0srpjK/v061bt6TbEy8sM0uWLJH/v3jxonyfr4WvydllZfr06fJZYrcpVy4rNX1HaoHo6bwym7evKTo6Wv5InRcWdOb27dti0KBBomXLllLguT37S2n94XnzmqZNm2Zry/dh2LBh4ujRo4a+T8yZM2fkvdm1a1e1Y/n6PmVnZ7v8HSnXwK98Tc77xMbGyutv166dg8+hmu9ILUgtBQCwFJjTAwBYCogeAMBSQPQAAJYCogcAsBQQPQCApYDoAQAsBUQPAGApIHoAAEsB0QMAWAqIHgDAUkD0AACWAqIHTEtRUZFMGfXGG2/Ytn3++ecyTZFzGiNgHZBwAJgaLsk5fPhwKXacNj42NlbWyFiyZImvTw34CIgeMD1Tp06lPXv2yISonDz0iy++kLVVgTWB6AHTw4VlOnfuTJcuXZIVtbheBLAumNMDpuf8+fN05coVmV78woULvj4d4GPQ0wOmhuulcpUwnsvjOb1ly5bJIS7XggXWBKIHTM306dNpy5YtdOzYMXrkkUcoISFBFvL+z3/+4+tTAz4Cw1tgWri+K/fs3nnnHVlK0N/fX/7/4MGDtHLlSl+fHvAR6OkBACwFenoAAEsB0QMAWAqIHgDAUkD0AACWAqIHALAUED0AgKWA6AEALAVEDwBgKSB6AABLAdEDAFgKiB4AgKzE/wNK939CsgPtQQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 300x200 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "train_x = torch.linspace(0., 1., 21)\n",
    "train_y = torch.pow(train_x, 2).mul_(3.7)\n",
    "train_y = train_y.div_(train_y.max())\n",
    "train_y += torch.randn_like(train_y).mul_(0.02)\n",
    "\n",
    "fig, ax = plt.subplots(1, 1, figsize=(3, 2))\n",
    "ax.plot(train_x.numpy(), train_y.numpy(), 'bo')\n",
    "ax.set_xlabel('x')\n",
    "ax.set_ylabel('y')\n",
    "ax.legend(['Training data'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The PyroQEP model\n",
    "\n",
    "In order to use Pyro with QPyTorch, your model must inherit from `qpytorch.models.PyroQEP` (rather than `qpytorch.models.ApproximateQEP`). The `PyroQEP` extends the `ApproximateQEP` class and differs in a few key ways:\n",
    "\n",
    "- It adds the `model` and  `guide` functions which are used by Pyro's inference engine.\n",
    "- It's constructor requires two additional arguments beyond the variational strategy:\n",
    "    - `likelihood` - the model's likelihood\n",
    "    - `num_data` - the total amount of training data (required for minibatch SVI training)\n",
    "    - `name_prefix` - a unique identifier for the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "POWER = 2.0\n",
    "class PVQEPRegressionModel(qpytorch.models.PyroQEP):\n",
    "    def __init__(self, train_x, train_y, likelihood):\n",
    "        self.power = torch.tensor(POWER)\n",
    "        # Define all the variational stuff\n",
    "        variational_distribution = qpytorch.variational.CholeskyVariationalDistribution(\n",
    "            num_inducing_points=train_y.numel(), power=self.power\n",
    "        )\n",
    "        variational_strategy = qpytorch.variational.VariationalStrategy(\n",
    "            self, train_x, variational_distribution\n",
    "        )\n",
    "        \n",
    "        # Standard initializtation\n",
    "        super(PVQEPRegressionModel, self).__init__(\n",
    "            variational_strategy,\n",
    "            likelihood,\n",
    "            num_data=train_y.numel(),\n",
    "            name_prefix=\"simple_regression_model\"\n",
    "        )\n",
    "        self.likelihood = likelihood\n",
    "        \n",
    "        # Mean, covar\n",
    "        self.mean_module = qpytorch.means.ConstantMean()\n",
    "        self.covar_module = qpytorch.kernels.ScaleKernel(\n",
    "            qpytorch.kernels.MaternKernel(nu=1.5)\n",
    "        )\n",
    "\n",
    "    def forward(self, x):\n",
    "        mean = self.mean_module(x)  # Returns an n_data vec\n",
    "        covar = self.covar_module(x)\n",
    "        return qpytorch.distributions.MultivariateQExponential(mean, covar, power=self.power)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = PVQEPRegressionModel(train_x, train_y, qpytorch.likelihoods.QExponentialLikelihood(power=torch.tensor(POWER)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Performing inference with Pyro\n",
    "\n",
    "Unlike all the other examples in this library, `PyroQEP` models use Pyro's inference and optimization classes (rather than the classes provided by PyTorch).\n",
    "\n",
    "If you are unfamiliar with Pyro's inference tools, we recommend checking out the [Pyro SVI tutorial](http://pyro.ai/examples/svi_part_i.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ca4e30151d8b45a6abfe8981275f0224",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/200 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 1min 18s, sys: 8.25 s, total: 1min 27s\n",
      "Wall time: 11.6 s\n"
     ]
    }
   ],
   "source": [
    "# this is for running the notebook in our testing framework\n",
    "import os\n",
    "smoke_test = ('CI' in os.environ)\n",
    "num_iter = 2 if smoke_test else 200\n",
    "num_particles = 1 if smoke_test else 256\n",
    "\n",
    "\n",
    "def train(lr=0.01):\n",
    "    optimizer = pyro.optim.Adam({\"lr\": 0.1})\n",
    "    elbo = pyro.infer.Trace_ELBO(num_particles=num_particles, vectorize_particles=True, retain_graph=True)\n",
    "    svi = pyro.infer.SVI(model.model, model.guide, optimizer, elbo)\n",
    "    model.train()\n",
    "\n",
    "    iterator = tqdm.notebook.tqdm(range(num_iter))\n",
    "    for i in iterator:\n",
    "        model.zero_grad()\n",
    "        loss = svi.step(train_x, train_y)\n",
    "        iterator.set_postfix(loss=loss)\n",
    "        \n",
    "%time train()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this example, we are only performing inference over the QEP latent function (and its associated hyperparameters). In later examples, we will see that this basic loop also performs inference over any additional latent variables that we define."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Making predictions\n",
    "\n",
    "For some problems, we simply want to use Pyro to perform inference over latent variables. However, we can also use the models' (approximate) predictive posterior distribution. Making predictions with a PyroQEP model is exactly the same as for standard QPyTorch models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0x12eca5de0>"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAEmCAYAAACOMEBlAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjcsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvTLEjVAAAAAlwSFlzAAAPYQAAD2EBqD+naQAAT4pJREFUeJztnQd4W/XV/7+WZEnee484e5C9B4GGhgYSKIEGUqBhFtoC/VN427JTRiFAKU1fGuCFllHakLBCaQJhBAIJ2XuQOMPx3tuWbe3/c44iIzteWpYsnc/zKPa9ule6V4q/93fP75zvCbFarVYIgiAIQYHC1wcgCIIg9B8i+oIgCEGEiL4gCEIQIaIvCIIQRIjoC4IgBBEi+oIgCEGEiL4gCEIQIaIvCIIQRKgQZFgsFpSWliIqKgohISG+PhxBEAS3oRrbpqYmpKenQ6HoeSwfdKJPgp+VleXrwxAEQfA4RUVFyMzM7HGboBN9GuHbP5zo6GhfH44gCILbNDY28mDWrm89EXSibw/pkOCL6AuCEEj0JWQtE7mCIAhBhIi+IAhCECGiLwiCEEQEXUy/r+lPJpMJZrPZ14cieAilUgmVSiVpukLQI6LfCYPBgLKyMrS0tAT9f45AIzw8HGlpaVCr1b4+FEHwGSL6nQq3zpw5w6NCKnIgcZACrsC4c6OLeVVVFX+/w4cP77WARRACFRF9B0gYSPgp35VGhULgEBYWhtDQUBQUFPD3rNVqfX1IguATZLjT1Ycio8CARL5Xwd+p1Rnw2tYzKKr1XnhZRF8QBMEPaGwz4t87C3CgqA4NrUavvY+EdwRBEHyMTm/C6p2FOFzcAIXCu0aQMtL3EpTtuXkz8Pbbtp8DMfszJycHK1euHDCvKwgDkTajGe/sKcL+wjoMTozw+vuJ6HuBDz4gYQPmzQOuu872k5ZpvTegDKOeHo8++qhLr7t7927cfvvt8DVvvPEGYmNjfX0YguBxDCYL3ttbjB15NciOD4dG5X1JlvCOhyFhX7KE0gQ7ri8psa1/7z3gqqs8+55UV2Bn7dq1WL58OXJzc9vXRUZGdkhfpKIzKlTqjaSkJM8eqCAI7ZjMFvznQAm2nqxGZmw4wtUq/vv0NjLS9yAUwrn77nMFn7Cv+81vPB/qSU1NbX/ExMTw6N6+fPz4cbZb/eSTTzBlyhRoNBps3boVp0+fxhVXXIGUlBS+KEybNg1ffPFFj2EYet2///3vuPLKKzmllfLdP/roox6PrbKyEpdffjmnTA4ePBj//ve/z9nm+eefx7hx4xAREcHpsnfccQeam5v5uc2bN+Pmm29GQ0PDOXcub731FqZOncrnR+d63XXX8fsJgr9jsVix4XAZvjpeidQYLSK1/Tf+FtH3IFu2AMXF3T9Pwl9UZNuuv7n//vvx9NNP49ixYxg/fjyL6sKFC7Fp0ybs378fl1xyCYtzYWFhj6/z2GOP4ZprrsGhQ4d4/+uvvx61tbXdbn/TTTdx74KvvvoK7733Hl588cVzhJlSKf/3f/8XR48exZtvvokvv/wSv//97/m52bNn84WHbLDpjoYev/3tb/k5o9GIJ554AgcPHsSHH36I/Px8fj9B8GesVis+/64cnx2tQGKkBjFhof36/j4V/W+++YaFhqpfaQRHf7i9QSO/yZMn84h12LBhHO/1FxyiLB7ZzpM8/vjjuPjiizF06FDEx8djwoQJ+MUvfoGxY8fyiJ3Ek57rbeROonrttdfyZ//UU0/xxWPXrl1dbnvixAm+w3j11Vcxc+ZMvtP4xz/+gdbW1g7b/eY3v8G8efP4zuKiiy7CH//4R7zzzjv8HFVFd757sYerbrnlFlx66aUYMmQIvz5dOOj97HcJguCPgr/5RBXWHy5HbHgo4iL63xLEp6Kv0+lYfFatWtWn7amEftGiRSwQBw4cYLH4+c9/jk8//RT+QFqaZ7fzJBQGcYSEkUbMo0eP5klSElK6C+htpE93CXYoHEMj8O5CKvR6NHdAYm9n1KhR50zKUljphz/8ITIyMjhUs2zZMtTU1PTqf7R3714eNGRnZ/N+F154Ia/v7RwEwVdsz6vBf/aXIEKt5FG+L/DpRC6N0ujRV15++WWOC//5z3/mZRIsik//5S9/wYIFC+Br5s4FqD0lTdp2Fdenpjb0PG3X35BAO0KC//nnn+O5557jUTvF3JcsWcIWBT1BVgaO0AicrCtchUIyl112GX71q1/hySef5LsQ+k5vvfVWPpbu7DBowEDfOT1onoAmnUnsabm3cxAEX7C3oA4f7C2GSqlASrTvbEAGVEx/+/btmD9/fod19EdO67tDr9dz/0jHh7dQKoG//tX2e+euZfZlmhel7XzNt99+y6EampSlSVQKm5AAexIa1ZNFNY3I7VBWUX19ffsyPUcXDbqQU4hmxIgR3LzeEQrxdLa5pglquhugeYq5c+fye8kkruCvHClpwDu7C2G2AhmxYV1uYzEDpw6G4dT2OOzepvRabc+AEv3y8nLONnGElknIO8eJ7axYsYJjwvYHZYd4E0rHpLTMjIyO62mE7410TVehOP4HH3zAYTKaCKXMF3dG7F0xcuRIniCmuYOdO3eywFM4ju4q7NBdBk3IvvDCC8jLy+OMHLqjc4Ri/RSOoknn6upqDvtQSIcuBvb9aC6C5iUEwd84UdGEt3cVotVoRlZc14J/aGsknlg2GC/9PhubXhyCW66J9Fptz4ASfVd44IEHON3P/qBMEm9Dwk6D5q++Alavtv08c8Z/BN+eJhkXF8fZMRQXpzsmmiD3NK+//jpP1FO8/aqrruJir+Tk5PbnaU6HjuWZZ57hSWUK1dCF2hE6xl/+8pdYunQph3GeffZZ/kmT+O+++y7GjBnDI34KVQmCP5FfrcPqnQVoaDEiJyGiS6t2Evw3Hk9DQ7Wqy9oeTwt/iLU/qgH6AH0Y69atw+LFi7vd5oILLmBhcswdJ1GhCV0S9L5AdwU04qftaRLSkba2Np4spnkDsd4NPOT7FfqT4roWvPFtPkrrWzE0ORKKLgSfQjo0wrcJfki384A0aOwpLNyTrg3okf6sWbP4Ft8Rmoyk9YIgCP5CRWMb/rWjACX1rRiS1LXgE3lHwtBQTckRIf1W2+NT0ac4LcWU6UHQKJt+t6fcUWjmhhtuaN+ebvEpfkuFOzSRR4U+lM99zz33+OwcBEEQHKlu1uOf2/NxplqHIUkRUPbgmtlYq+r32h6fiv6ePXswadIkfhD33nsv/07eMQRVXzrmXFPYZcOGDTy6p1gwZXyQLYA/pGsKgiDU6Qx4a3sBTlU2Y1hSJFS9NGSKjjf1e22PT/P0f/CDH/RoMNRVtS3tQ7YBgiAI/kRDqxH/2lmAY2WNGJYcyfn4vTFkbCtiEo29xvQ9WdszoGL6giAI/kgzN0Ep4CYoFNIJ7YPgEwolcOUdVWeXrIDSDGV0i1dre0T0BUEQ3KDFYMLbOwuxv7Cem6BoVM4p9Pjzm3HT8jLEJBmQdPl+pC3bhtDERq/V9oifviAIghtdr9buLsKe/FrOw9eGujYkHzenCaWxp5Bb0QRYQ3Df0w149KZor1Tvi+gLgiC4gN5kxrt7irjrVVZ8OMLUSrecN0nwKaQzMTsGSy/xjuATEt4RnIL8ehwL6GhinYrj3METryEI/YnRbMG6fSX49pSt61WExvXx87bTNThUbCsuvXh0itfN2ET0A0iM7Z2lyJOGPG3IQ58Mz7wJ+ff01fOGeiHQ8Tkarjn7GoLgD20OPzpQwqPztNgwt7pe7c6vxZ6COv79opHJGJUaBW8j4Z0AgszNyJaCnEU//vhj3HnnnWyFTEVujpD1MF0YPAFZIfvDawhCf7Y5/OJYJVKjtYjWut716mBxPY/yifOHJWJcZoz0yBWcg7qJkUXyoEGD2J+ebKjJfdIekiG/ejI/I/dLgsznqPUhNTUh4aWeuY72ymRnTAVz9HxCQgJXQneuq+gcmqELzn333cdupvbuZtQti16Xmt8QZPRGI357a8POr1FXV8eV2LQd+elTz4WTJ092qN+gY6LmOdRTgRrA0AXPsUG8IHgaq9WKjUeozWE5ktxsc0i5/Jtzbama03PiMWVQHPoLCe/04YumlCxfPNz1wiMLY3tDEfIsIi97qmZev3492xlTJTN1nNqyZQv769vF074PVTyTwL722mvc2IR64ZIpXk+QWL/99tvcupA6Z/3f//0fvy5dBN5//33eho6DBPqv9uYDnaCLAVVr0wWLeiXQ50D9eOmY7ZC9MrlqkhUztd2kym1771xB8DRWqxVfHKvAx0fKEBuudqvNIVXrfv5dBf8+ITMGM4f0752uhHd6gTywxyz3TTvG7x5fgHC1yqX/oCTyNBL+9a9/jaqqKu6cRZYV9rDOv/71L/bPp3V2u1cKDdEImmLvP/rRj9jNlEJDZIlMkM99T60pqScueSHRhcXe7Ib613YO45C1cueWiXZoRE9iTxchslQmyG6ZLhrUQ/nqq6/mdXQBoOOhvr7EXXfdxXMYguBprFYrvj5Rhf8eLOVwTm9tDsk5k4zUyFeHbBao6paKsIiCGh3fLdBwbnRaFC4ckdSl3bI3EdEPIGgET6NqEkQSdGqM8uijj3Jsn7pjOcbxqXHKqVOneKTf2X749OnTbNFKo/EZM2a0P0f9bqnXbnd3IGSWp1Qq23vVuoK9r67j+1JoiUJS9JwdCvvYBZ9IS0uTzlmCV9h+ugYf7i9BmFqFpKieBZ+88de9mHTWOdMG2SxQ1W3C2CqsP1QGs9XKNg3zR6X0u+ATIvq9EBaq5BG3r97bGShm/tJLL7G4U+yexLO7HrnkcEoNy2kU3RlqUOLS8Tp0xPI2XfXq9ZPWEEIAsTu/Fu/vK2ZbBZq47Ql7M5TOkK/O6v8LR+ZNpTDDikEJ4bjkvFQoenDf9CYi+r1AYuJKiMUXkLDTxGlfoGY0a9eu5VBLd00XaPRMbQ6peQ1h73fbXYctupugO4yvv/76nF7GhP1Oo3O/W0doYpbeh97XHt6hXrg0D0AdsgShvzhQVI93dhexp316N20OHUM6NMK30VHMVQnNSL5mN8ywID1Gi0Xj0nq0W/Y2MpEbpFx//fVITEzkjB2ayKVeBhTL/3//7/+huLiYt7n77ru5DSHF0ql/wR133HFOjn3nXrY33ngjbrnlFt7H/poU5ycoq4guohSGonkGutvoqncvHdNtt93Gk8cUhvrZz36GjIwMXi8I/cHR0gas2VXIRViZvQh+T81QVDEtSFm6E8pwA/RlMRirHNpnMzZvIaIfpFBMnLJeqME4TdTSCPvWW2/lmL595P8///M/WLZsGQs5dSej+P+VV17Z4+tSeGnJkiV8gRg1ahSLt06n4+dIuB977DHcf//93NCeJl+7giaUKfR02WWX8ftS2IbqDjqHdATBW43MV+8s5Ay67PjwPsXdu2qGooxsQ/LSnVBF6WGojkTlu9PRWt/znEB/4Dc9cvsL6ZEbvEiPXKE38qqa8ca2fNTqDBiS2HUj8644dTAML/4uq31ZEWZAynXboU5shrEuHBWrZ8HcrMUdfyrCsAmt3b4OyfGJymbcNW8YxmbEoK8EbI9cQRAEb1FY08J9bWuanRN8x2Yo5IkfojYi+epdLPimJi0q186AuVmD2CQjb+drRPQFQQh6Sutb8daOfJQ1tDkt+I7NUEJCTUi+ejc0aQ0wt6hRsXY6TA22OYHFv6pqz9f3JQMjLUUQBMFLVDa2seAX1bVyX1tXUynHzGrE+P85iXprM8xtKpvg10TxCJ8En5ql+AMi+oIgBC3VzXr8c0cB8qp0XDDlaiql+awRW721BaHKEMxOyYHy1hZExzd2qMj1B0T0BUEISupbDBzDP1neZGtkrlC47LxJ1gr5NS1QKULw4wnpyIyji0cT/BGJ6XdBkCU0BQ3yvQp2GtuM+NeOQnxX2oghSZFQuZg7T/+nPj9WgVNVzVCGhOCy8WnIjAuHPyOi74A9D5wcHIXAw/69Sr5/cKPTmzgP/1BxPYYkRUCtcl3wv8ytxPFyW5vDS8elYlBCR7sTf0TCOw6QWRi5P1ZWVrYXMPnCEEnwgj12Swt/r/T90vcsBCetBjPW7C7EvoI6DE6MgEaldMktk/5PbTlZjSMljby8YEwqhiZFun18VB+gVSmgcfFC1BdE9DtBTUgIu/ALgQMJvv37FYKzkfk7e4qw60wtchIioO3B0LAnt0zKwtmRV4v9RTZLkvmjkzHSA20O61oMqG0xsDePJy4gfluRu2rVKvzpT39CeXk5JkyYgBdeeAHTp0/vdnvyeKdSf2qaQd4xVPK/YsUKaLVaj1aukSmYY9MOYWBDIR0Z4QcvBpMF7+4twjcnqpAV13Mj845umY53+japnPfgfuSZbV3afjAiCROyuu4N4QyNrUaUN7ZhwXmpuHxCutNZRM5U5Pp0pE8uj9SOj5phkH86CTp1cyJHRXJ/7Mzq1avZt4U6OZEDIzXtsDcEf/755z16bCQQIhKCEBiNzD88UIItJ6qRERvWo+D35JZJy1FTzrQL/pyhCR4R/KY2IxeF/XB0Mk8Ee9uB06cTuSTUZMh18803s20uiT/F0UnUu2Lbtm2YM2cONwchR0fq7nTttddi165d/X7sgiD4P5Q//9HBUnx1vBJpMVpE9dLIvDu3TCJyfCHi53/Hvw8LS8bUnHiPTCqX1Ldi7ohEXDExw+UsogEh+tSHlbzZHX3XFQoFL1Nf1K6g0T3tYxf5vLw8dl+k/qndQY266dbH8SEIQuBD+fMbDpXhi+8qkBylQXQfGpk3duGWSYSPLkH8JYdt2+wajNS2DI9MKhfVtWD20ET8ZHKmy1lEzuKz8E51dTXHzcli1xFaJu/2rqARPu13/vnn8+w5Ndv45S9/iQcffLDb96F4P9n5CoIQXIL/yZEyfHq0jHvaUjPzvhAdbzpnXdjwciRedpDTMpv2Z6Puq9GIWWjrOeEqeqMZ+TU6TMuJxzVTs3qcVA7qPH1qyPHUU0/hxRdfxL59+/DBBx9gw4YNeOKJJ7rdhxp70+SG/VFUVNSvxywIQv9iL5j65Eg54sLViIvom+B3dssktIMrkXTFPoQorGg+nIHaz85DbJItfdOdSeW8ah0mZcfi2hnZCFP3bwqxz0b6lHlDE6UVFRUd1tNyd2l1jzzyCDf1+PnPf97eno8adNx+++146KGHODzUGY1Gww9BEAIfLpg6XskNyKO1oUiI1LjklvnG42nQZlch6cq9CFFaoTuehppPxnGs3x23TOrEdbqqGeMyY3DdjEGI7GFSOeBG+tQvlbojbdq0qX0d9VelZeqW1BVUYNNZ2O0ZNlJiLwjBjb1g6qMDpYhUK5EU5dpgb/z5zVj88EkkL9kDRagFLSeTUf3fiYhNtOCm5WUuu2Wazgr+6LRo/GzGIMT0YY7BG/g0ZZPSNakV39SpUzk3n1I2aeRO2TzEDTfcwC32KC5PXH755ZzxM2nSJE7xPHXqFI/+ab2kVwpCcLM9rwbr9hdzfDw5um91O9156x+xlCAk1Iqk0CiMGJGO2GdK3HLLpCwi8uchY7efzRzkVMgpoER/6dKl3CB7+fLlXJw1ceJEbNy4sX1ylwqwHEf2Dz/8MOfk08+SkhIkJSWx4D/55JM+PAtBEHwNVdm+t7eYnTJTY1wX/LKGVvznQCmMZiv3x718fDJUSluPZ3cmlWmEPyghHMtm5rh8BxIwFbn9jTOVa4Ig+D/7Cuvw9s5CWKxWtxwuKxrb8MG+EhjMFmTGhbFFcqibefN0TKcrm5EWq8VNswcjK947DpzSI1cQhKCAnDLX7CqEyWLlaltXqWxqw7r9NsFPj9V6RPBpPE3NWZKjNTzC95bgO4sYrgmCMCA5WtrAFsl6o4VDJ9054vbklklUNemxbl8J9CYLV+1eMSHDI4J/pkaHuPBQXD9jEHIS/cdyWURfEIQBR255Ewt+i8HEjpndCX5vbpk1zXoe4beZLEiJ1uCKieluV8aS4BfUtCBCreK0zOEp7jtwBm1xliAIwqnKZvx7ZwE7U/Ym+JRv31DdcWzbUK3i9du/UeGD/SVoNZrZpuHKiRk9+uv3FWqwrglV4LoZ2RiT7n/zhiL6giAMGM5U67ivbU2zgZug9BTS6cktUxWnw46GPLQYzEiK1ODKSRnQeMAKoaSuFcoQYOm0bIzPdN+B0xuI6AuCMCDIq2rGm9vyedKV2hz21NWuJ7dMVawOKdfuhCJcjyillgXfE943lO5J2TpXT83ClEFx8FdE9AVBGBCC/8/tBSz41FVK0Usb0+7cMpXRLUj56U6ootpgqI7EGMtwj3jfULonTQT/ZEompg9233LZm4joC4IwIASfsmz6IvjduWUqo1qReu0OqGJaYayJQOWaGUhMcL9hCR0X+eIvnpSB2UMT/L6vtoi+IAh+y2kHwaeQTl8Evyu3TGVkG1JI8GNbYawLR8WaGYgKV7rllklQ9k9jmxE/npjOrRP9XfAJEX1BEPxW8N9yQfAd3TIJZWQrUn66A6FxLTDWh6Hi7RkwN2vdcssk6nQGbma+cGwafjgqZUAIPiF5+oIg+GVa5r925KOqyeC04NuhPPyfPlKArVVnoIxpgamBBH8mt0xcfI/rbplEfYsB1c16XDI2DZeMTYXCy31tPYmIviAIASf4BIVdvlPmQxljRJgiFGNjhyD5kRq33DL5dVuNqGzSY/7oZCwanzagBJ8Q0RcEIfAEv9WI9/cVo7HNhGitCj+ZnIHoMIPbx9fcZkJZQxt+MDIJP56YAeUAE3xCRF8QBL8R/Le256O62T3Bbzgr+E1tJm5UQoJPIR130elNKK5vxdzhibhqcqbb/jy+QkRfEASfc6qyibN0atwUfIq1k7UCCX4sC34mIrXuy1yLwYSiuhbMHpqIJVMy3fbn8SUi+oIg+I3gD+2l0rY3wX9/Xwma9SZ2t6TRuCd60LYZzWygNmNIPK6ZmuWR6l1fIqIvCMKAF3xKnaSQjk5vRny4GldNzkCEBwRfbzSz38/kQXH46bRsj1Tv+hoRfUEQfMLJiia8tcN9wa/V2QSfzNMSImyCH672jODnVeswMTsW18/I9shFxB8IjLMQBGFAcaKiqd0t0x3Bp4pYiuGz4EeqcdUkzwh+29kR/qTsWG5k7omJYH9BRF8QhH7lSEkD3t5VyFk27gg+FUdRT1vyw7fbI3si/NLmENKhEX4gCT4hoi8IQr+xt6AO7+wu5E5VQ3rww+8NbnHo2ADFQ/bIbQ6CTyN8T0wE+xuBd0aCIPgd1EJwe14N3t9bDIsVGBTffU/bvgj+B/uK+cLhScFvNZqRX63D1Jw4bnMYiIJPBOZZCYLgV4K/+UQV/rO/BCqlAtlxYS69DnXD2n/Aih31JTDBgpQoaoCS7pGOV60GM/JrdJiWE89tDgNl0rYrBm6FgSAIfo/FYsVnR8s59k6j8YxY1wSf+t0+dW8cvi7PhwlmtJXEInfVLOTujPGI4BfUBofgEyL6giB4BZPZgv8eKsVHB8vY/yYlWuuy4K95IxQRl+yFUmtCW3EcKt+ZgYayMG5wTs+7U2mbf1bwr58Z+ILvF6K/atUq5OTkQKvVYsaMGdi1a1eP29fX1+POO+9EWloaNBoNRowYgY8//rjfjlcQhN4xmCw80brxSDnnzidGalwO6fx3vRXJV++BQm1G65lEVL4zHVYDibNtTuDDl5J4O1cEv7CmBTMH20b4nkj1HAj49CzXrl2Le++9Fy+//DIL/sqVK7FgwQLk5uYiOTn5nO0NBgMuvvhifu69995DRkYGCgoKEBvrn13nBSEYoQyY9/YWY+vJaqTFaBEd5nrK45Y9bYj44SmEKKxoyU1B1X8nAWbHGH4I6qtCuRH6sAmtzgl+rc1a4drpgwKi0nZAiP7zzz+P2267DTfffDMvk/hv2LABr732Gu6///5ztqf1tbW12LZtG0JDbf+R6C5BEAT/gJwo1+4uxM4ztciMDXfL7OxwcQMONFciRAE0H8lAzcfjAavCqUbo3R1jUV0LZg5JCBhrhQER3qFR+969ezF//vzvD0ah4OXt27d3uc9HH32EWbNmcXgnJSUFY8eOxVNPPQWzuft7O71ej8bGxg4PQRA8DxVbka3CjrxaZMe7J/h7CmrxZW4l/964dxBqNkzoVvC7a4Tek+DPGpKAa6cHn+D7VPSrq6tZrEm8HaHl8vLyLvfJy8vjsA7tR3H8Rx55BH/+85/xxz/+sdv3WbFiBWJiYtofWVlZHj8XQQh2yP/mn9vzsa+gDoMTI1yOj1N657bT1fj2VA0vT82Og+XAiJ72QGySsU8Nzu2CP3toIn46PXvAu2UO2IlcZ7BYLBzPf+WVVzBlyhQsXboUDz30EIeFuuOBBx5AQ0ND+6OoqKhfj1kQAp3Kpja88e0ZDscMTYp0WUxJ8L8+UYXd+XW8PGdoAuYMT8SVd1Tbt+i8B//blwbnzQ6Cv3TawLdHHpAx/cTERCiVSlRUVHRYT8upqald7kMZOxTLp/3sjB49mu8MKFykVqvP2YcyfOghCILnKalv5faGeVUtGJYc6XI3Kcrn/+J4BY6VNfHyvJFJGJ9pS9CgBuY3LS/DuheT0FD9/aRwbJKJBb+3BufUOpFaHJ4/LBFXB4Af/oAVfRJoGq1v2rQJixcvbh/J0/Jdd93V5T5z5szB6tWreTuK/xMnTpzgi0FXgi8Ignetkck4jQR1WHIEVGf/Jp3FZLFwaufpKh3ImeFHY1IwKjW6wzYk7GNnNXOWDk3aUgy/Lw3OyWefbBsuGp2MxRMzBnTHK0/h00+A0jVfffVVvPnmmzh27Bh+9atfQafTtWfz3HDDDRyesUPPU/bO3XffzWJPmT40kUsTu4Ig9K9x2uvf5qOyUY9hSZEuC76RCrgOlrHgK0NCsGhc2jmCb4cEntIyJ89r4p+9CT65cJJ18yVjU9lyWQTfD1I2KSZfVVWF5cuXc4hm4sSJ2LhxY/vkbmFhYfuInqBJ2E8//RT33HMPxo8fz3n6dAG47777fHgWghA8UBjmm5NV+OhAKUfUqZ+ts8ZpVEhFI/baGuC48hTqTC0IVYbgsvHpnPXjCSoa26AzmPHjCWm4eEwqFArXzN0CkRArzZ4EEZSySVk8NKkbHd31iEIQhK5H5R8fLsPn31WwXUGqC7YKZJlAsfkmnQXJ1+yCJrURFr0KUyOHYO4FVo/NM5AFxOJJGbhwRJLLbp6BqmvBUXcsCILbpmTUkvDbU9XcsCQuQu2S4JNXjjKqFanX70Jogg5mnRoVa6djXVU0YpaX9Top2xM0fi2qa4UyBFg6LRszh8QHheA7i8xqCILQI/UtBry5PR9bTlYhPSbMJcGnkA6N8EOTGpG6bBsLvqlRi/J/z4KxKsYtDx274JM1MsXtr5s5CLOGJojgd4OM9AVB6Jbyhjas3lmA3PIm5CRGuJzuSDH8tvAGpF61FwqNCYbqSDZOMzeFueWhQ1isVuRV6RAXHoprZ2TjvHT37ZYDGRF9QRC65HRVM97eWchFTUPdyMHn16ppRMrVRxCisqCtKB5V70+FRR/qlocOYbaQ4DcjOVqLn83MxrDkKJePMVgQ0RcE4RwOFtXjnT1FqG8xYnhyFJRuZL/sK6zDcWs1QlSA7ngqqtdP7OSU6byHDkGTtaerdciKC+N+toMSIuSb7AMi+oIgdIiNbz1VjQ/3l8BksWKoCymZjq+15WQ19hfV87L+aDaqN5zXjXEaeejYCq76mklEdyJk+0CCn+5iR65gRERfEIT2kTNVxn76XQXCQhXIjHM9Z56qbD8/WoETlbZsHLJAUKnj8eZ6uoBQamaISx46hN5k5hj+6LRoFvykKLFZcQYRfUEQuPHJun0lXHhFna4SXOx0ReiNZqw/VIbi+lZQVOhiu63CIJ1bHjpEq9GM/GodJmTGcrcrVzKJgh0RfUEIcup0Bo7fU+ydwiTRWtc7XTW3mfDhwRK2P1ArFVg0Pq1Dla2rHjpEU5uRC6+on+1Pp2chyo3jDGZE9AUhiKFR89o9hcir1GFQYgTC3HCgrGnW48MDpWxjHK5WssFZV6EXu4eOsxemqmY9LhiRhJ9Mzgx6p0x3ENEXhCCEJllpZP/BvhIWVLJFVrmRkllS14r/HiqF3mThfHkSfHd643b20aELyaVj07BwXKpbxymI6AtCUE7Ykn/Op0fLOTOHBN+VDB27cdrpmiacsJbAAis3Qr98QrpbdwyOF6biOtsdwU+mZOLC4UlinOYBnL5k3njjjfjmm2888d6CIPQzNGImD/yPDpayaRrF210RfPLReWLZYLz1tgnHLGdY8I0FSRjeOtIjgs9VttU2W4XrZ2bjByNE8H0m+uTiRs3Lhw8fzl72JSUlHjsYQRC8a6nwjy15nDtPE7aJLmbosHHaE6lQjD+B+Iu/48YnTfuyUbp2Kt56IoOfdwdK9zxV2czHd9OcHEwZJMZpPhX9Dz/8kIWeGpqsXbsWOTk5uPTSS7lhudFo9OjBCYLgGY6VNeKVb07ju7JGLmiK1Lg2ncfGaa/GIfknexA9LZ/X1X09ErWfj20vunLHOM1gsgn+oIRw3Hr+4G4bqgiu49KMSFJSEne9OnjwIHbu3Ilhw4Zh2bJlSE9P5wYnJ0+edOOQBEHwZNMTcsd8/dszqGzSY0RylFsdpI4cVCB84S6EDa2CxahA1YeT0bhjmEOx1ffGaa7YN1OV7Zi0aNx6/hBkeaihitARt6bBy8rK8Pnnn/ODmpUvXLgQhw8fxpgxY/CXv/zFnZcWBMETBVf7i7F2dxEvD0mMcGsilDJ0ttSdhjqxGaYmDSpWz0JLblqX2zprnEY5+AW1Os7Bv/n8wVJl60WcvsejEM5HH32E119/HZ999hm3LfzNb36D6667rr1jy7p163DLLbfwqF8QhP6H0jDf3VuEfQV1SInWIjbcvcrV70obsel4BSxUcVsejar3p8Hc3H3nLGeM02p1Bu5n+4MRybhycobk4Pub6KelpcFiseDaa6/Frl27uK9tZ+bNm4fY2FhPHaMgCE5QUKPj0T3FxskD351sGsqi2XaqBnsL63iZmqDveYMEv7tJYOeM02hyWWcwcUN0amAuOfh+KPoUtrn66quh1XZ/lSfBP3PmjLvHJgiCCwVX5KFDo+fhbhZc0aQq5fJT6iQxPSeeWxBm/aKW2x66Y5xmb21ImT9LKAc/SHrZ+gPSGF0QAiR+TwL95fFKlmFX8+/tNLYaucK2utnAXvrzRyd3yKSxNzjvaJxm7JNxGk0un6nWIUqrwpKpWZgyKM7l4xRsSGN0QQgiyKbg/b3FOFRcj6QoLeKddJ60V9baDdDCsmqx4UgZO1qSh87l49ORGtPxzt5V4zS6e8irbkZmXBg3Lx+RIp2u+hvx3hGEAQqFSA4VN3DDk9KGNuQkON/DtvOIPWJMCRIuLeW2homRarZU6M5101njNJ3ehMLaFvbBv3Z69jkXEqF/ENEXhAEINRKhJiWcUWMFRiRHOp2OyZW1HJsnrIidewIxs0/xUsvJZIw+LxXRWufcMLujrsWAykY9Zg1NYB8dd+ybBfcQ0ReEAUZVkx7v7yvGgcJ6Ho270vCEK2tfTOLfQ0LNSFh0EBEjy3m5YcdQ1H89Auu3mDFx9pk+ed33dDdS3tiGFoMZC8amcpaOO8Vhgvv4xae/atUqtnOgjKAZM2ZwKmhfWLNmDU9WLV682OvHKAj+wJGSBrz89SnsL6hDdkK4yx2uKBZPIR1VXAtSl21jwbeaQ1C9YQLqvx7F0uBqZa1jumd+jY7vRK6ZmoUrJqSL4PsBPh/pk38PWTq8/PLLLPgrV67EggULkJubi+Tk5G73y8/Px29/+1vMnTu3X49XEHwBTYB+dbyC+9dSU/DhKVGcVeMqNPkaNqwciZcdhEJj4rx7slTQl8Sfs52r9s2U6klNVEjwx2bEuHysQoCN9J9//nncdtttuPnmm9m+gcQ/PDwcr732Wrf7mM1mXH/99XjssccwZMiQfj1eQehvKOf+n9vzse5AKbQqJYYkRrol+DQCL9UUI/kne1nw24rjUPbm+ecIvrOVtY7poycrmzEkKQK3zR0igu9n+HSkbzAYsHfvXjzwwAPt6xQKBVs3b9++vdv9Hn/8cb4LuPXWW7Fly5Ye30Ov1/PDMZ9VEAYKx8sbOR2zsKaFwznhavf+ZMnU7JOjZShqtU3QNu7JQd1XowGLwq3KWsf8/tKGVkweFIelU7Okcbkf4lPRr66u5lF7SkpKh/W0fPz48S732bp1K/7xj3/gwIEDfXqPFStW8B2BIAwkKDyyObcSG49W8MjZ3XAOQROqGw6VcSMVlSIEIxTZ+GTTmC627HtlbecJ5vpWI+aNTMbiSeKh46/4PKbvDE1NTWzh/OqrryIxMbFP+9BdBM0ZOI70s7KyvHiUguAeZD7234Ol2H2mlvvMkjumO9W1lEFztLQRm3OrYLZaERsWikXj0zjzJ0VZ1kVlralPlbWdLRXoYrF4Yjp+ODrF7QuUEKCiT8JNlswVFRUd1tNyamrqOdufPn2aJ3Avv/zy9nVk/kaoVCqe/B06dGiHfTQaDT8EYWB459Rj/aFStjEmKwVqaejuHcNXuVXcPIWgC8iPzkuBRqV0q7LWjvmspUJMWCh+MiUDk7PjxEPHz/Gp6KvVakyZMgWbNm1qT7skEaflu+6665ztR40axX79jjz88MN8B/DXv/5VRvDCgKWh1YhPDpfh21PVPEoemRLldhNwiq9vOFzGzVPolagwauqgc0XZ2cpaOxR2opRMujj9dHo2d+QS/B+fh3co9ELN1qdOnYrp06dzyqZOp+NsHuKGG25ARkYGx+Ypj3/s2LEd9rdbOHdeLwgDZXR/rKwJHx0sQV6VjnvX0qjZXUiMPz1SjjaTha2VybaYxNlT1LcYeI5gQlYsrp6SJU1PBhA+F/2lS5eiqqoKy5cvR3l5Ofvzb9y4sX1yt7CwkDN6BCHQoEyaz7+r4AlbA+Xeu2mFbL+I7MqvxY68Wl5OidZg4bg0j9ke0OuXNbSxGdvFY1Jw2fh0aXoywBBrZUHwARQHJ6M0SslMitS4XFnraKuQeygUe2qLUQtb/H5sRjT71Ks8NGii+D3dQdA8w48npGPWkAS3Q1CCZxBrZUHw48rab05U4rPvKtDUZuJCK3e9aMg47aP3VQi74DBU0W3csLxt+2gkXayGalTfMnD6Gr8flBDOFbbDksUSeaDi8/COIAQLZQ2tPLonO2SK21M4x91uUQe2ROA/XzYhZuEphCgAY204qv4zGcbKaLyxHbhpeVmfUy97dshsw6TsOFw9JdPtuxLBt4joC4KXobDIzrwafHy4DFXNegxK6L5vbeeGJj2lT9brjNhUmI/Y8+t5uflwJmq/OA9Wg/3P2ooPX0rilExXnDIpfl9a3wa92YIF56Xi0nFpEr8PAET0BcGL1DTrsf5QGXadqYU2VMGpmN2N7rtqQRiTaMSVd5xbKHWyogmfHa2EKtUCi16Fmk/HouVYRqdXDGl3ynQ2JdNksfC8A00AL5majRmD4yX/PkAQ0RcEL0BFUXsK6jhtsqS+FVnx4YjsodCqY0OT72moVvF6e5iGHDa/PlHFFbaEvjQW1R9Ngqmh+3RMZ50yKauI4vdkmEbx+yGSfx9QiOgLgocpqm3BJ0fKcLCoHmqVkvvA9mRL4NjQhEbnHQlpD9OkjK3Gp8fKUddi5GeGhiXjy39P6cIszXWnTHL0JA+daTnxWDIlUwzTAhARfUHwEDRC/uZkFb48XsnFS1QM1RdXTHtDk54wDyrGO3uLYYEVERolFoxJRUZMOPbGm9FQTReGri4qfXfKpPh9cV0rh3Uor3/B2O+tGoTAQkRfEDxUVfvxkTKOtceFq3uM3TsTflGE65G48CDChlbBctY7Z/7oFISpbYJM8X5bWIicMUNccsqkkFF+tQ6x4WosnpSNaTninxPIiOgLghvU6Qz49Gg5duTVwGS2upR33134RZtThcRFB6GM1MNqUmBsTCp+OD68w8WE4vwU73fVKZP8eWjOYVRqFDcsp8wiIbAR0RcED0zUpsVoeaTsSuolPUdZOjRpy6N1hQWxF+QiZkYeP2+oioR+6zhc9NdqdHXz4IpTJt2d0HHrTRZcNCoZl01I73GiWQgc5FsWBA9P1DqTekmQONvDNKHJ9UhcdAjq5CZ+rmlfNne2uvHByh5F3BmnTKoKpuwcCkMtmZLF6ZhipxA8iPeOILgwUdvQYkRWfNg5E7UdUy/PjbF3VyFLE6jrv2lGvrESIQorzC2hqNk4Hpr6BKcamvTFwrmMwjlp0ZydQ6mkwsBHvHcEwYNQKISakHxypLx9onZEyrkWCn1NvexcIVvR2MZumzVmA1sppKljkB2SjcTbjRgy9oxL1bTdZecYLVbMH5PCnbPc7bcrDEzkWxeEXkI5m45VYH9RPcy9TNT2nnrZsUKWRvdUqUtzA1Yr2Jph3sgk7ocLtHnse9GbqNiqBUmRavx4YkaXjVSE4EFEXxC66VP7dW4l+9I3thm5uUlvnvR9rXyl7crPju6pGIqgOweyQfb06JvM0uhO4rz0GM7OyYgN8+jrCwMPEX1BcKBZb8K2U9VsdUCVqclRmj7n3Pep8lVpRqmmCFt3V3GUn0b3lD0zLNmzrQYt1Ky8toV/slna2LT23H4huBHRF4SzIZC9BXX48lgliupaEBumxsjUKCicCIOck3rZCXV6LZIvP4TTrTpepovJhSOTunXcdBW90YwzNTqkRGtxxURqVh4r4RyhHRF9IaixWKw4UtqATccqeZJWE6rEsCTX2hY6pl46VsiGqMyIOT8X0dPO8ERtuNo2uvdGI3Fy9ST75vEZsRzOSY3Revw9hIGNiL4QlFA2S161jidpDxc38LrshHC3/WY6V8hqMmqRcOkhhCbYRvdU+Uqxe62HR/dULFZQ24JQpQKLxqXjR+eliPe90CUi+kLQUd7Qhi+PV2BPfh03+KbJTer76ilI+IdPa8BnexqQ31bD6yLOju69YVNMufel9a0YnBTBvWvHpEVLOEfoFhF9IagycnacrsHWU9XsmUOhj8y4MI8KJN1BHC9v4vdoMZh53ei0KFww3POje+rIRfMP9J50QaHOVtSGURB6QkRfCHjIY4baFVJOPKVIJkTaJmk9natO2T6bcytR2mDLsY8LD+VQjjdMzCjLiLJz6KJFvjmTsmSyVugbIvpCQEKjX2r3tz2vBgcK6zkEkhipcTojp6+ZPztO1+JgST0XWakUIexnQ43Ee2qe4gqUgkmhnDajBXOGJeKy8WnSqFxwChF9IeCycU5UNnGu/ZGSRugMJqREadkF09Mj+65COZRvf8HwRET1UsjlqvcPTdYmR6k5M2fG4ASPX1SEwEdEXwgIqBHId6WN2HqyGrkVjextT3nqno7ZdxfKiQ0PxQ+8FMqhi0tFo54rgynnnnLvJRVTcBXnk5G9wKpVq5CTkwOtVosZM2Zg165d3W776quvYu7cuYiLi+PH/Pnze9xeCGzajGaO1b+w6SRe+SYPx8oakRylZf+a6LBQjws+hXKoWvft3YUs+BTKmT00AdfPyPaK4NP7nahoYh/9a6Zl4ZbzB4vgCwN7pL927Vrce++9ePnll1nwV65ciQULFiA3NxfJycnnbL9582Zce+21mD17Nl8knnnmGfzoRz/C0aNHkZGR4ZNzEPofmsjcX1jHI3ua0KRiKhrVezpDxlehHHq/6mYDanR69s1ZPDGD6wgEYcD76ZPQT5s2DX/729942WKxICsrC7/+9a9x//3397q/2WzmET/tf8MNN3jUd1rwv3h9YW0LjpQ0sDMl5duTn0xqtNbpFoX+GsqxNzmh8wwLVeCiUSm4aHSyFFoJgeGnbzAYsHfvXjzwwAPt6xQKBYdstm/f3qfXaGlpgdFoRHx8fJfP6/V6fjh+OMLAor7FwPF6Evoz1c080qZ89KFJES7ZJfQVyvjZfqoGuZW2LlZKhGD64HhMzomFSqHwTuy+SY+GFgNGpEbh8vHpZ22WBcFz+FT0q6ureaSekpLSYT0tHz9+vE+vcd999yE9PZ0vFF2xYsUKPPbYYx45XqH/oNHuycombkl4uKSBLYLVSiWSojTIilN6teK0xWDieYJDxY2wnu14pTuWxm0LazQqaLtpe+jue9Lonhq0XDU5Exd4wapBEPwipu8OTz/9NNasWcNxforvdwXdRdCcgeNIn8JHgv9BI92yhjYcLW3ErjM1KK1v47z0+Ag1hiV17EPrrQvNvsI6fhjNNrFvPZOI+q9HwVARw8sNTVY2VOuu7aErISt7g/Ipg+LYAllaGAoBK/qJiYlQKpWoqKjosJ6WU1NTe9z3ueeeY9H/4osvMH78+G6302g0/BD8F53exFk3ZG1MTpdNehMiNSpkxYWx66W3oQ5WlNNPo3vy4uF1VdGo2TQKbQX29oe9tz10lsaznjlpsWFskDY9J96r4SpB8Lnoq9VqTJkyBZs2bcLixYvbJ3Jp+a677up2v2effRZPPvkkPv30U0ydOrUfj1jwZDiDKmZPlDdxK0KaLKWRPFXNUpeq/mjnR3cWueVNXLXb2GZrgBIbFoohqjSse2Zsl574XbU9dMURs7CuBQqE4IKRSbjkvFSpqhWCJ7xDoZcbb7yRxXv69OmcsqnT6XDzzTfz85SRQ6mYFJsnKEVz+fLlWL16Nef2l5eX8/rIyEh+CP4JCWyNzsBCT0JLD/LBMVutNqFN9O6kbOdjoZ6x205Xc1qk3QWTKlzHpEfj4NeU/dD7Raev7RE7fwbVTXoMSY7gUM64jBhxxBSCS/SXLl2KqqoqFnIS8IkTJ2Ljxo3tk7uFhYWc0WPnpZde4qyfJUuWdHidP/zhD3j00Uf7/fiFnl0gS+pakVfdzHH6ghodj6gpNE/ZN9nx4V5NteyKsoZWfHuqhuPoBL0/NQqfmBXLXvR9bnvoxHb2blY0URuuUWHR+HROw6QQliAEXZ5+fyN5+t6F/GHya3Q4XdmMw6UNqGxs4xRLak5C+e0x2lAofOAXQ7FzW8qnrZkJhZImZsZiak7cOVkyFjPwxLLB3bY9pJh+bJIJD//zTK8xffrzoibodLE7Ly0aC8eneaVjlhDcNA6UPH1h4EPZJ9Sej6piyS7gWNn3YRsKmSREeD/Fsrcwzp782vbCKiJLE4f5k2IQHRHqVNvDs6/K/y7+VVWvgk9eOXSxoXmKn07LwqyhCW535hIEdxHRF5z2uqFK2NKGVhb6U5XN3JBEZzCzNEZ7OWxDo3CaQKV4OoVXqBl5Z/GlsBJdgCgbiGLohNUcguYjmWjcNQQFtZE4nGhkYe8u7bJz20M7NMInwe8pXZP8copqW6FSki9PIhaclyp+OYLfIOEdoceRcmOriQWeYuF5VToU1LTwCFZvtHBsntoMRmlViFCrvB62ObQ18hwRjnEQb3LapLkDyrNvOpuNo4QCtTtz0LQnB+bmMMez4397y7fvy0XG8WJDI3tK+xyREsVpmNK6UOgPJLwjuASlElI2C4k8TcDSKL6isQ1NbUaYLUCoMgSRWhVbFmtVin4N2ZDg28ItHaG4+z+fTcBcUxnKFZXcXIQIC1VyzH7j4+NRX0piH+JSvj2t7y0t026ORu0YM+LC8JPRmTxXIKEcwR+R8E4QQiNSsjWg2DsJVU2zHiV1bShrbEVzm4knXq1nhTNKo2JjMXtmiy+g0TaN8G18L97KqFZET89D5Pgi5JvNgNmWFUSe8zTCzj8SgfrSnpwp3cu3J+jzKmloQbQ2FIvGp+EHI5IREy59agX/RUQ/wCdZ61uNLOqcH87ibgvV6PRmFneyOaCRKo1KybGSRDM9JszroRpnwia0nWNIJzSxkcU+YkwpQpS2MI2+PBozh8fi/Knq9mPvax69s/n2dssGakpO70SVtBePSRX7BGFAIKI/gCGxpglUCr9QDNvxZ3mjHmX1rew7T+JuslhZoGjETuJOOeJkXuaLEXxvsfmuRDkk1ITwUWWInFAIbUZ9+3Ot+Qlo3DkUbfmJiHygHAqFzRHTW/n2dCElfyD6XIelROJHY1IwNj3GJ2moguAKIvp+HIKhLBAakdvFnISGfreFZWzhGcqmoVEnGXbRPhRmp8qLUFUIwkNVCFerOGXQl+GZvsbmuzIyozz/AnUNMu88BIXG1J6J03IyhcXeUB7brXjT3QNdTHrLt6ft+nKBpc+7sknPmTiXT0zn5ufihCkMNET0PQyJg8VqM/EiEbY/7MJMIk2TjW0mc/vvJO7NZ0WdHmRARs/R6JzcHg0mMwxm20idghk0qKRwDKVFalQKHrXTTyo48kU+vLux+c4TqyOmNeBkVTOOUHFXk60XgkIDGOvC0Xwwi1MvLTptr+LtqXx7utDS6J7qDhacl4J5o1LY+VMQBiIi+k6wI6+GM1ooNdD2IFG2cBNu/mmxrbMLvy1ebvtJwk8iTttS4RLJj13EWcjpy1AquOcq5XdTk45QhQJhoSFQq9RQK/s3W8YbdI7Nd8QKdVo9FBOK8PetJTDDloWjDAnB0OQIhNem4j+vDD/7qfVdvN3Jt6eLL6VghqoUbHv8w9EpGJzonW5ZgtBfiOg7waGiemw7XcN56YoQGlXTqDukw++OP0m4aVROYk2jcBZ0he33gS7grtDVhKlCY0TEeSUcq1cn2+LxZG4cFx6KsRkxGJ0azXMQRNzycpfEm56jtMy+ThyTlQR589B3Ny4zBvNGJWNEcpTE7YWAQETfSSgdL8dLvVEDHXvMPURlhjanChGjyhA2ohyKUNuo3mJUoCU3DRfP0WLalHMvjM6Kt7P59mSKVlzfyndno9KiMG9kMkanRXu9eYsg9Cci+kK/QOEvS0ol0pZUQpVZBYXG1qyEMFRGoflgNnRH0xEdHYKpd53hOyVXxdtZaL6FRvYUfhuWHMFiT5bH0tBECERE9AWvQWJKrpanqpqRX61jUVUPtT1natTyqF53LB2GMlsrQmLxr8rc6kbl7IWIYvY0wU6xegrjTMiM7Xe7Z0HoT0T0/RRnipd8+X6d98sY1YyCWh1OVtLPFp7AthOtVWF4chRQloRN7w5CQ/X3GTCxScZeY/OegibcyTSO0mGz4sNY7Cdnn2uxLAiBiIi+H+Js8ZKv3s++X1OzFWHDKhA+shxhFVXtVbL8OmGhGJ4cyQ8qBuM4/XArZs/J79eLml3sKxv1bBiXERuGKyZmsEcO1TIIQrAgLptO8MrXp3G0rNGrE7kdi5dCnHaF7I/3I/H8dmsIvv7aCm1ONdSpDQhxiIgYqiMxPCUSc6aokRip9nmmEoWZqJFJq8HEhVVzhydh+uB4RGnFI0cIDMRlc4DS1+KlnlwhnQnT9PX9zpvZhNpWA7f7Iw99+6RnzGx0mIylGH1LbiqMNZFoTTLhx3O7n5DtD8jimKwoKMRE7pdzh2dgUnasiL0Q1Mh97YApXurdFdLZME1P76eMbkFYTjVUOdV45ZtKGKzfZ9sQ5mYNWgsS2POmrSAR5iZHr3q47V7pDlTVTBW0lGk5JCkS5w9LxPjMGInZC4KIvn/hjiuks542HV/HCmV0KzTp9dBm13DIJjSupX07A3n5KEM4Dk5dsZpPJWHdqsHd+Nk4fz6egKqgG1qNbNtA2TfjMqIxZ1giWyxL6qUgfI+M9P0IV10hnQ0LUcUpNUcp1zYi6SfF0KTVQxlhaytoh0zN9KWxPIq/dJEFUybbKomJU3V0dxDiUfdKd8TebhtNHkQzhyRg9tAEbj4uzpeCcC4i+n6Eq66QPYVpyJJYndIAS3o93ttWCZ1Ch8azrQSJ8GFnX9kcAkNVFPTF8baQTVECrAYlv9/Uh89AofCOe6WrUJy+qknPzWDiItS4eEwKZgxOQGZcmM8njgXBnxHR9yNcdYW0hVGsUIQbEJrQjNDEJmhSG6BOa0BoQlN7Zk2Zw2CevG2o7aG5OgbfvJ4FQ0U0YFb16f085V7pCuQ+SncpNEmbHKXFFRPTMW1wPP8uCELviOj7Gb25Qo6d3YS6FiPqdAbUnm15WKYoQebdh6DUdh1OoepXqnqdNCkEY0YqkRKlgcahEClT2YZ1L0ahoRp9NjJzx73SlRAOnTOFcCjERAVVNKqflBUnrQkFwUkkT78PUPvVLVuAf31ZgTo0YvZsq0vVqn0tQCKRa241I/c7JaoajTBrW2GN0KG+xYj6VgPbNne9H2CqD4exNgLGihjoy2JZ7M06DYvxw/8845FUT0/s19f8+somasxuQuxZ102yOB6REuU3TWEEwR8YcHn6q1atwp/+9CeUl5djwoQJeOGFFzB9+vRut3/33XfxyCOPID8/H8OHD8czzzyDhQsXeuXYPvgAuPtuoLiYllL4scmJatXO6ZOLf1WJodMabF2w9MYOzVPs3bHIx72DsFNo3CE8TvbMFMemEE18uJobelTlxuK9FYMAs9KlcIurRmaeNkCjCx7NOVAWTgisSIsJwyVjU9kTJzlaQjiCMOBH+mvXrsUNN9yAl19+GTNmzMDKlStZ1HNzc5GcnHzO9tu2bcMFF1yAFStW4LLLLsPq1atZ9Pft24exY8d69IpIgr9kiW0E3RHbihsfKcWoWY3Qn+2EZf95+lgodnweDoXWCIXWBEWYAcqoNqii2qCM1CNE0bePPEKjRFyY+nuBp58RakRpVF1OVnZ1oelPTxt3MJktqGrWo6HFiEitCqNSozlWPyo1SvLrBcGDuuZz0SehnzZtGv72t7/xssViQVZWFn7961/j/vvvP2f7pUuXQqfTYf369e3rZs6ciYkTJ/KFw1MfDoV0cnLsI3wgeuYpaLNqzwq5oV3Q+yrgjlgtIYgOU7K4UZqh/WeUw+/kB+OKj3t/G7W5A4ex9LZRPWXj0MTytJw4TMiK5ZoAycIRhAAL7xgMBuzduxcPPPBA+zqFQoH58+dj+/btXe5D6++9994O6xYsWIAPP/ywy+31ej0/HD+cvkAxfLvgE5T2GDakqsttSZy11K82VAmrXoWS45GwtIbCog+FpS2Ufzc1a7lq1dyk5Rj7Hc8We6Va1Rt+897IwKFJWRJ8urhRARWN6s9LjxbzM0HwMj4V/erqapjNZqSkUKz8e2j5+PHjXe5Dcf+utqf1XUFhoMcee8zpYysr67jcfGAQWk+ntIu4XdCvuasG0+bp2rfb91UUDrx/bmWsL6tV/QGalKVMo/pWI9TKEDY+o9x66kxFVb4yqheE/iHglYfuIhzvDGikT+Gj3kjrpNtUmdoVcQm2Vn/uVtUGIuTGWaczcgEViXpylAYzhybwiJ6alkgGjiAEmegnJiZCqVSioqKiw3paTk1N7XIfWu/M9hqNhh/OMncukJkJlJR0NZHbfdWpP1Sr+hILeeC0GNkagX6nCei5wxM53ZIaqNibnAuC4Bt8muysVqsxZcoUbNq0qX0dTeTS8qxZs7rch9Y7bk98/vnn3W7vKkol8Ne/2n4/N1Gm92pVx+36st9AhtMsW43Iq27GyYpmzmCanB2LW84fjPsXjsZ1MwZhfGasCL4g+AE+D+9Q6OXGG2/E1KlTOTefUjYpO+fmm2/m5ymdMyMjg2PzxN13340LL7wQf/7zn7Fo0SKsWbMGe/bswSuvvOLxY7vqKuC99xzz9P2vWtVXULYNuVpS6IbSLSM0KoxIjsKk7DiMSotCYqTzd1eCIASB6FMKZlVVFZYvX86TsZR6uXHjxvbJ2sLCQs7osTN79mzOzX/44Yfx4IMPcnEWZe70JUffVeG/4grnK3JJ2MnVcqCkT/YFvcnMdgg0qqebn+iwUPapp8lY8q1Pj9HKhKwg+Dk+z9P353xWX7RL9CfovwY1D6fRfIvBxL70CRFqFvkRqVEYkhiB2PDvm5sLguAbBkyevuCfYRsayZPQG8xWRKiVnF45NiMaQ5OiMCghXCpkBWEAI6If5FCGDfv+tJqgM9jSSClsMyY9pj21kqpjpSGJIAQGIvpBhsViE/nGNiNaDLa+tzSaT4jSYE5KArITwjE4MZJ9fgRBCDxE9INR5DUqJEVpMDIlikU+My4cSZEaGc0LQhAgoh9gGM0WtmYmobeLfKRd5FNtlgdZcWGcUikhG0EIPkT0A0DgKcOGBN4KCxQhCrZkJsdKGsln8Ug+DIkRIvKCIIjoDyjDMppoJZFvNZhhOdtMheLxiVFqnnBNjw3jXrHkcUOdpsTETBCEzshI38+g6lZq+t1GDVmMZv6dKilUyhCOxdMIPocEPiYMydEaFviYMBF4QRD6hoi+jyZXyZ/GUdhNFitXuVLflLBQJbRqpS00Ext2dgRPAq9FdFjXXbMEQRD6goi+lypZDWYLh2SMZiv/JJGnn/w8SNgVXOREqZFU/JQWo+Xq1jh+hPLvapU0/xYEwbOI6LswSqeYul3U+WEmcf/eV59EnRqFqJUKhKpsE6s5ieFIjdYiIVLDwk4xdxL8cLVSRu6CIPQbIvrOfFhKBcxWKyqb2ngUTo+kaBLxUCREaBBF/W211Os2lIXe3u+WwjUSkhEEwR8Q0XeCS8elYtbQBFsTc62KJ1al+5MgCAMJEX0nSIsJ44cgCMJARWYKBUEQgggRfUEQhCBCRF8QBCGIENEXBEEIIkT0BUEQgggRfUEQhCBCRF8QBCGIUAWjL469e7wgCEIgYNczu771RNCJflNTE//Mysry9aEIgiB4XN9iYmJ63CbE2pdLQwBhsVhQWlqKqKgop/xw6EpKF4qioiJER0cjEAn0c5TzG/jId9g1JOMk+Onp6VAoeo7aB91Inz6QzMxMl/cnMQxEQQymc5TzG/jId3guvY3w7chEriAIQhAhoi8IghBEiOj3EY1Ggz/84Q/8M1AJ9HOU8xv4yHfoPkE3kSsIghDMyEhfEAQhiBDRFwRBCCJE9AVBEIIIEX1BEIQgQkTfgVWrViEnJwdarRYzZszArl27evzw3n33XYwaNYq3HzduHD7++GME0jm++uqrmDt3LuLi4vgxf/78Xj+TgfYd2lmzZg1XaC9evBiBdH719fW48847kZaWxpkvI0aM8Pv/p86e48qVKzFy5EiEhYVxRfk999yDtrY2+CPffPMNLr/8cq6cpf9vH374Ya/7bN68GZMnT+bvb9iwYXjjjTfcOwjK3hGs1jVr1ljVarX1tddesx49etR62223WWNjY60VFRVdfjzffvutValUWp999lnrd999Z3344YetoaGh1sOHDwfMOV533XXWVatWWffv3289duyY9aabbrLGxMRYi4uLrYFwfnbOnDljzcjIsM6dO9d6xRVXWP0VZ89Pr9dbp06dal24cKF169atfJ6bN2+2HjhwwBoo5/jvf//bqtFo+Ced36effmpNS0uz3nPPPVZ/5OOPP7Y+9NBD1g8++ICyJq3r1q3rcfu8vDxreHi49d5772WdeeGFF1h3Nm7c6PIxiOifZfr06dY777yz/YMxm83W9PR064oVK7r84K655hrrokWLOqybMWOG9Re/+IU1UM6xMyaTyRoVFWV98803rYFyfnROs2fPtv7973+33njjjX4t+s6e30svvWQdMmSI1WAwWAcKzp4jbXvRRRd1WEcCOWfOHKu/gz6I/u9//3vreeed12Hd0qVLrQsWLHD5fSW8A8BgMGDv3r0cvnD06KHl7du3d3mHROsdtycWLFjQ7fYD8Rw709LSAqPRiPj4eATK+T3++ONITk7GrbfeCn/GlfP76KOPMGvWLA7vpKSkYOzYsXjqqadgNpsRKOc4e/Zs3sceAsrLy+Pw1cKFCxEIbPeCzgSd4VpXVFdX8x8C/WE4QsvHjx/vcp/y8vIut6f1gXKOnbnvvvs4Ftn5P+FAPb+tW7fiH//4Bw4cOAB/x5XzIwH88ssvcf3117MQnjp1CnfccQdfuKnyOhDO8brrruP9zj//fHaaNJlM+OUvf4kHH3wQgUB5NzpDbqOtra08j+EsMtIX+sTTTz/Nk53r1q3jCbaBDtnQLlu2jCerExMTEag24nQX88orr2DKlClYunQpHnroIbz88su+PjSPQZOcdPfy4osvYt++ffjggw+wYcMGPPHEE74+NL9FRvoA/9ErlUpUVFR0+HBoOTU1tcsPjtY7s/1APEc7zz33HIv+F198gfHjxyMQzu/06dPIz8/nTApHkSRUKhVyc3MxdOhQDOTvjzJ2QkNDeT87o0eP5tEjhVLUajX8CVfO8ZFHHuGL989//nNepiw6nU6H22+/nS9wvXnL+zvd6QxZS7syyicG9ifiIeg/P42ENm3a1EEAaJliol1B6x23Jz7//PNutx+I50g8++yzPGrauHEjpk6dCn/F2fOjVNvDhw9zaMf++PGPf4x58+bx7/7WWc2V72/OnDkc0rFfzIgTJ07wxcDfBN/Vc6R5ps7Cbr/IBYKt2Cxv6IzLU8ABBqWKUerXG2+8walRt99+O6eKlZeX8/PLli2z3n///R1SNlUqlfW5557jdMY//OEPAyJl05lzfPrppzl97r333rOWlZW1P5qamqyBcH6d8ffsHWfPr7CwkLOt7rrrLmtubq51/fr11uTkZOsf//hHa6CcI/3d0Tm+/fbbnN742WefWYcOHcrZdf5IU1MTp0DTg+T3+eef598LCgr4eTo3OsfOKZu/+93vWGcohVpSNj0I5cBmZ2ez0FHq2I4dO9qfu/DCC1kUHHnnnXesI0aM4O0prWrDhg3WQDrHQYMG8X/Mzg/6QwuU73Agib4r57dt2zZOJSYhpfTNJ598ktNUA+UcjUaj9dFHH2Wh12q11qysLOsdd9xhraurs/ojX331VZd/U/Zzop90jp33mThxIn8e9B2+/vrrbh2DWCsLgiAEERLTFwRBCCJE9AVBEIIIEX1BEIQgQkRfEAQhiBDRFwRBCCJE9AVBEIIIEX1BEIQgQkRfEAQhiBDRFwRBCCJE9AVBEIIIEX1B8DBVVVVsiUs+73a2bdvGLpKdHRMFob8R7x1B8ALUqWrx4sUs9iNHjsTEiRNxxRVX4Pnnn5fPW/ApIvqC4CWoNy01nqE+BOTdv3v3bmg0Gvm8BZ8ioi8IXoJ6mFIz8qKiIm7eTV2dBMHXSExfELwEtWQsLS3l7k/UmlEQ/AEZ6QuCF6AetNOnT+dYPsX0V65cySEealQuCL5ERF8QvMDvfvc7vPfeezh48CAiIyNx4YUXIiYmBuvXr5fPW/ApEt4RBA+zefNmHtm/9dZbiI6O5sbd9PuWLVvw0ksvyect+BQZ6QuCIAQRMtIXBEEIIkT0BUEQgggRfUEQhCBCRF8QBCGIENEXBEEIIkT0BUEQgggRfUEQhCBCRF8QBCGIENEXBEEIIkT0BUEQgggRfUEQhCBCRF8QBAHBw/8HkezCeu9UyXEAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 400x300 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig, ax = plt.subplots(1, 1, figsize=(4, 3))\n",
    "train_data, = ax.plot(train_x.cpu().numpy(), train_y.cpu().numpy(), 'bo')\n",
    "\n",
    "model.eval()\n",
    "with torch.no_grad():\n",
    "    output = model.likelihood(model(train_x))\n",
    "    \n",
    "mean = output.mean\n",
    "lower, upper = output.confidence_region()\n",
    "line, = ax.plot(train_x.cpu().numpy(), mean.detach().cpu().numpy())\n",
    "ax.fill_between(train_x.cpu().numpy(), lower.detach().cpu().numpy(),\n",
    "                upper.detach().cpu().numpy(), color=line.get_color(), alpha=0.5)\n",
    "ax.set_xlabel('x')\n",
    "ax.set_ylabel('y')\n",
    "ax.legend([train_data, line], ['Train data', 'Prediction'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Next steps\n",
    "\n",
    "This was a pretty boring example, and it wasn't really all that different from QPyTorch's native SVGP implementation! The real power of the Pyro integration comes when we have additional latent variables to infer over. We will see an example of this in the [next example](./Clustered_Multitask_QEP_Regression.ipynb), which learns a clustering over multiple time series using multitask QEPs and Pyro."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.18"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}