pytorch_exercise_solution.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "2fd8d146",
   "metadata": {},
   "source": [
    "# Exercise: MNIST\n",
    "by Tobias Jülg\n",
    "\n",
    "Disclaimer: This exercise is partly based on pytorch's [Quickstart Tutorial](https://pytorch.org/tutorials/beginner/basics/quickstart_tutorial.html).\n",
    "## Install PyTorch and PyTorch Lightning\n",
    "Have a look at the [pytorch's installation page](https://pytorch.org/get-started/locally/). If the CPU version is sufficient for you then running the following cell should be sufficient. Note that pytorch only works with python3.9 so far."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "5f873f06",
   "metadata": {},
   "outputs": [],
   "source": [
    "# uncomment to install\n",
    "#!pip install numpy\n",
    "#!pip install torch\n",
    "#!pip install torchvision\n",
    "#!pip install pytorch_lightning"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0e8edb7e",
   "metadata": {},
   "source": [
    "## Exercise 1: Plain PyTorch\n",
    "### Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "0eeab0f1",
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "from torch import nn\n",
    "from torch.utils.data import DataLoader\n",
    "from torchvision import datasets\n",
    "from torchvision.transforms import ToTensor, Lambda"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e2e209b8",
   "metadata": {},
   "source": [
    "### Data\n",
    "In this exercise we use the MNIST dataset which consists of 28x28 pictures of handwritten digits. The task is to classify the pictures into the numbers  0 to 9. PyTorch already has a Dataset implemented for MNIST so we dont have to perform this yourselfs. Conveniently, the dataset can also download the dataset for us. The following code does exactly that.\n",
    "\n",
    "1. Look up what **transforms** are in the [pytorch docu](https://pytorch.org/vision/stable/transforms.html) and especially check out what the `ToTensor()` transform does. Do we have to normalize our data before we put it into the network? A: No as ToTensor already normilizes image input. We can also see this when outputting `training_data[0]`.\n",
    "2. Create two dataloaders for our train and validation datasets (TODO2)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "2d61a360",
   "metadata": {},
   "outputs": [],
   "source": [
    "training_data = datasets.MNIST(\n",
    "    root=\"data\",\n",
    "    train=True,\n",
    "    download=True,\n",
    "    transform=ToTensor()\n",
    ")\n",
    "\n",
    "val_data = datasets.MNIST(\n",
    "    root=\"data\",\n",
    "    train=False,\n",
    "    download=True,\n",
    "    transform=ToTensor()\n",
    ")\n",
    "\n",
    "train_dataloader = DataLoader(training_data, batch_size=64)\n",
    "val_dataloader = DataLoader(val_data, batch_size=64)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "36590e97",
   "metadata": {},
   "source": [
    "### Network\n",
    "3. Checkout the shapes of our training data. How must the input of your network look like to take such data? A: We must flatten the data to 28*28."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "35feb507",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(torch.Size([1, 28, 28]), 5)"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "training_data[0][0].shape, training_data[0][1]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bce10484",
   "metadata": {},
   "source": [
    "4. Complete the network's code using the following topology using the [`nn.Sequential` module](https://pytorch.org/docs/stable/generated/torch.nn.Sequential.html) (TODO4):\n",
    "\n",
    "$$W_3 ReLU(W_2 ReLU(W_1 x + b_1) + b_2) + b_3$$\n",
    "\n",
    "where $W_1\\in\\mathbb{R}^{512\\times28\\cdot28}$, $b_1\\in\\mathbb{R}^{512}$,\n",
    "$W_2\\in\\mathbb{R}^{512\\times512}$, $b_2\\in\\mathbb{R}^{512}$,\n",
    "$W_3\\in\\mathbb{R}^{10\\times512}$, $b_3\\in\\mathbb{R}^{10}$\n",
    "\n",
    "What does the [`nn.Flatten()` layer](https://pytorch.org/docs/stable/generated/torch.nn.Flatten.html) do? A: it flattnes the multidimensional $N\\times C\\times H\\times W$ data to single dimensional $N\\times N\\cdot H\\cdot W$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "4cccfa38",
   "metadata": {},
   "outputs": [],
   "source": [
    "class ClassicNN(nn.Module):\n",
    "    def __init__(self):\n",
    "        super(ClassicNN, self).__init__()\n",
    "        self.flatten = nn.Flatten()\n",
    "        self.linear_relu_stack = nn.Sequential(\n",
    "            nn.Linear(28*28, 512),\n",
    "            nn.ReLU(),\n",
    "            nn.Linear(512, 512),\n",
    "            nn.ReLU(),\n",
    "            nn.Linear(512, 10),\n",
    "        )\n",
    "\n",
    "    def forward(self, x):\n",
    "        x = self.flatten(x)\n",
    "        logits = self.linear_relu_stack(x)\n",
    "        return logits"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "729004cb",
   "metadata": {},
   "source": [
    "We now want to create a second network which uses convolutions instead of fully connected layers. Convolutions can be seen as learnable filters. They are much better in detecting local patterns as the weight kernel \"slides\" over the image and uses the same weights over and over again. This means that they will also result in less trainable parameters. Thus, they are perfect feature extractors for images. [This article](https://towardsdatascience.com/a-comprehensive-guide-to-convolutional-neural-networks-the-eli5-way-3bd2b1164a53) will give you a good overview of how convolutions work if you are new to the topic."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "267c29f6",
   "metadata": {},
   "source": [
    "5. Look up how [`nn.Conv2d` layers](https://pytorch.org/docs/stable/generated/torch.nn.Conv2d.html) work. As the handwritten digits are just in grayscale they only have one color channel as you should know from question 3. So start with one channel dimension 1 then go up to 16 in the first convolution, then to 32 in the second. Use a kernel size of 2, stride of 2 and padding of 1 for the convolutions. Use the `nn.ReLU` activation function in between the layers. Finally use a `nn.Flatten` layer and fully connected (linear) layer to get the output down to our 10 output neurons.\n",
    "\n",
    "The following formula might come in handy to calculate the input size of the fully connected layer:\n",
    "\n",
    "$$H_{out} = \\lfloor\\frac{H_{in} + 2*padding - (kernel-1)-1}{stride}+1\\rfloor$$\n",
    "\n",
    "where $H_{in}$ is the channel dimension of the layer before and $H_{out}$ is the output dimension of the current layer. What is the output dimension of the second convolution? A: $32\\cdot 7 \\cdot 7 = 1568$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "8c1b1f88",
   "metadata": {},
   "outputs": [],
   "source": [
    "class CNN(nn.Module):\n",
    "    def __init__(self):\n",
    "        super(CNN, self).__init__()\n",
    "        self.cnn_relu_stack = nn.Sequential(\n",
    "            nn.Conv2d(1, 16, kernel_size=3, stride=2, padding=1),\n",
    "            nn.ReLU(),\n",
    "            nn.Conv2d(16, 32, kernel_size=3, stride=2, padding=1),\n",
    "            nn.ReLU(),\n",
    "            nn.Flatten(),\n",
    "            nn.Linear(32*7*7, 10),\n",
    "        )\n",
    "\n",
    "    def forward(self, x):\n",
    "        logits = self.cnn_relu_stack(x)\n",
    "        return logits"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "221a7263",
   "metadata": {
    "cell_style": "center"
   },
   "source": [
    "4. Which loss function should we use this kind of problem. A: Classification problem with more than 2 classes: Cross entropy\n",
    "5. Given that we want to use `nn.CrossEntropyLoss()` as loss. Do we need to add a softmax layer in the end? Look up the documentation of the loss function for your answer. A: No as `nn.CrossEntropyLoss()` already includes the softmax layer\n",
    "5. The class label comes in the form of a single number {0, ..., 9}. Look up what a one-hot vector is. Given that we want to use `nn.CrossEntropyLoss()` as loss, do we need to convert our ground truth labels to one-hot vectors? A: No as `nn.CrossEntropyLoss()` also handles this and does not need one-hot vectors.\n",
    "6. Complete the loss function in the code snipped below (TODO7)\n",
    "7. Compltete the train_loop by adding the forward pass, the loss function calculation and the optimizer code (TODO8)\n",
    "8. Why is the val_loop function inefficient? Add according code to make it more efficient. A: `torch.no_grad()` is missing. It is inefficient as we do not need to track gradients in the validation loop.\n",
    "9. Test you code, you might also want to change the model to your CNN model from above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "2a3934ed",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1\n",
      "-------------------------------\n",
      "loss: 2.302592  [    0/60000]\n",
      "loss: 2.303575  [ 6400/60000]\n",
      "loss: 2.296248  [12800/60000]\n",
      "loss: 2.286207  [19200/60000]\n",
      "loss: 2.289458  [25600/60000]\n",
      "loss: 2.280091  [32000/60000]\n",
      "loss: 2.269974  [38400/60000]\n",
      "loss: 2.285682  [44800/60000]\n",
      "loss: 2.271315  [51200/60000]\n",
      "loss: 2.257305  [57600/60000]\n",
      "Val Error: \n",
      " Accuracy: 49.0%, Avg loss: 2.258471 \n",
      "\n",
      "Epoch 2\n",
      "-------------------------------\n",
      "loss: 2.258856  [    0/60000]\n",
      "loss: 2.254012  [ 6400/60000]\n",
      "loss: 2.253575  [12800/60000]\n",
      "loss: 2.221390  [19200/60000]\n",
      "loss: 2.229884  [25600/60000]\n",
      "loss: 2.213601  [32000/60000]\n",
      "loss: 2.180507  [38400/60000]\n",
      "loss: 2.215419  [44800/60000]\n",
      "loss: 2.166340  [51200/60000]\n",
      "loss: 2.127961  [57600/60000]\n",
      "Val Error: \n",
      " Accuracy: 66.5%, Avg loss: 2.124746 \n",
      "\n",
      "Epoch 3\n",
      "-------------------------------\n",
      "loss: 2.125901  [    0/60000]\n",
      "loss: 2.088705  [ 6400/60000]\n",
      "loss: 2.092088  [12800/60000]\n",
      "loss: 1.965946  [19200/60000]\n",
      "loss: 1.963956  [25600/60000]\n",
      "loss: 1.900382  [32000/60000]\n",
      "loss: 1.756889  [38400/60000]\n",
      "loss: 1.834056  [44800/60000]\n",
      "loss: 1.632993  [51200/60000]\n",
      "loss: 1.495110  [57600/60000]\n",
      "Val Error: \n",
      " Accuracy: 75.7%, Avg loss: 1.451783 \n",
      "\n",
      "Epoch 4\n",
      "-------------------------------\n",
      "loss: 1.482021  [    0/60000]\n",
      "loss: 1.287086  [ 6400/60000]\n",
      "loss: 1.277903  [12800/60000]\n",
      "loss: 1.025349  [19200/60000]\n",
      "loss: 0.977398  [25600/60000]\n",
      "loss: 0.910022  [32000/60000]\n",
      "loss: 0.740618  [38400/60000]\n",
      "loss: 0.936585  [44800/60000]\n",
      "loss: 0.811318  [51200/60000]\n",
      "loss: 0.703712  [57600/60000]\n",
      "Val Error: \n",
      " Accuracy: 84.2%, Avg loss: 0.669671 \n",
      "\n",
      "Epoch 5\n",
      "-------------------------------\n",
      "loss: 0.739334  [    0/60000]\n",
      "loss: 0.582266  [ 6400/60000]\n",
      "loss: 0.575581  [12800/60000]\n",
      "loss: 0.554699  [19200/60000]\n",
      "loss: 0.543625  [25600/60000]\n",
      "loss: 0.533234  [32000/60000]\n",
      "loss: 0.421850  [38400/60000]\n",
      "loss: 0.637241  [44800/60000]\n",
      "loss: 0.592707  [51200/60000]\n",
      "loss: 0.513634  [57600/60000]\n",
      "Val Error: \n",
      " Accuracy: 87.2%, Avg loss: 0.481413 \n",
      "\n",
      "Epoch 6\n",
      "-------------------------------\n",
      "loss: 0.540968  [    0/60000]\n",
      "loss: 0.433253  [ 6400/60000]\n",
      "loss: 0.390006  [12800/60000]\n",
      "loss: 0.460294  [19200/60000]\n",
      "loss: 0.422298  [25600/60000]\n",
      "loss: 0.444533  [32000/60000]\n",
      "loss: 0.333097  [38400/60000]\n",
      "loss: 0.549886  [44800/60000]\n",
      "loss: 0.514053  [51200/60000]\n",
      "loss: 0.468741  [57600/60000]\n",
      "Val Error: \n",
      " Accuracy: 88.4%, Avg loss: 0.415751 \n",
      "\n",
      "Epoch 7\n",
      "-------------------------------\n",
      "loss: 0.465008  [    0/60000]\n",
      "loss: 0.381666  [ 6400/60000]\n",
      "loss: 0.323221  [12800/60000]\n",
      "loss: 0.429200  [19200/60000]\n",
      "loss: 0.363718  [25600/60000]\n",
      "loss: 0.412248  [32000/60000]\n",
      "loss: 0.293090  [38400/60000]\n",
      "loss: 0.510213  [44800/60000]\n",
      "loss: 0.472104  [51200/60000]\n",
      "loss: 0.459132  [57600/60000]\n",
      "Val Error: \n",
      " Accuracy: 89.1%, Avg loss: 0.383158 \n",
      "\n",
      "Epoch 8\n",
      "-------------------------------\n",
      "loss: 0.424861  [    0/60000]\n",
      "loss: 0.355427  [ 6400/60000]\n",
      "loss: 0.291361  [12800/60000]\n",
      "loss: 0.414204  [19200/60000]\n",
      "loss: 0.329303  [25600/60000]\n",
      "loss: 0.395164  [32000/60000]\n",
      "loss: 0.270914  [38400/60000]\n",
      "loss: 0.486711  [44800/60000]\n",
      "loss: 0.444741  [51200/60000]\n",
      "loss: 0.459182  [57600/60000]\n",
      "Val Error: \n",
      " Accuracy: 89.7%, Avg loss: 0.363444 \n",
      "\n",
      "Epoch 9\n",
      "-------------------------------\n",
      "loss: 0.398382  [    0/60000]\n",
      "loss: 0.339580  [ 6400/60000]\n",
      "loss: 0.272729  [12800/60000]\n",
      "loss: 0.405422  [19200/60000]\n",
      "loss: 0.306587  [25600/60000]\n",
      "loss: 0.383476  [32000/60000]\n",
      "loss: 0.257265  [38400/60000]\n",
      "loss: 0.471014  [44800/60000]\n",
      "loss: 0.425186  [51200/60000]\n",
      "loss: 0.461669  [57600/60000]\n",
      "Val Error: \n",
      " Accuracy: 90.1%, Avg loss: 0.349974 \n",
      "\n",
      "Epoch 10\n",
      "-------------------------------\n",
      "loss: 0.378121  [    0/60000]\n",
      "loss: 0.329074  [ 6400/60000]\n",
      "loss: 0.260102  [12800/60000]\n",
      "loss: 0.399772  [19200/60000]\n",
      "loss: 0.290095  [25600/60000]\n",
      "loss: 0.374283  [32000/60000]\n",
      "loss: 0.248285  [38400/60000]\n",
      "loss: 0.459840  [44800/60000]\n",
      "loss: 0.410378  [51200/60000]\n",
      "loss: 0.464433  [57600/60000]\n",
      "Val Error: \n",
      " Accuracy: 90.4%, Avg loss: 0.339991 \n",
      "\n",
      "Done!\n"
     ]
    }
   ],
   "source": [
    "model = CNN()\n",
    "learning_rate = 1e-3\n",
    "batch_size = 64\n",
    "epochs = 5\n",
    "loss_fn = nn.CrossEntropyLoss()\n",
    "\n",
    "\n",
    "optimizer = torch.optim.SGD(model.parameters(), lr=learning_rate)\n",
    "def train_loop(dataloader, model, loss_fn, optimizer):\n",
    "    size = len(dataloader.dataset)\n",
    "    for batch, (X, y) in enumerate(dataloader):\n",
    "        # Compute prediction and loss\n",
    "        pred = model(X)\n",
    "        loss = loss_fn(pred, y)\n",
    "\n",
    "        # Backpropagation\n",
    "        optimizer.zero_grad()\n",
    "        loss.backward()\n",
    "        optimizer.step()\n",
    "\n",
    "        if batch % 100 == 0:\n",
    "            loss, current = loss.item(), batch * len(X)\n",
    "            print(f\"loss: {loss:>7f}  [{current:>5d}/{size:>5d}]\")\n",
    "\n",
    "\n",
    "def val_loop(dataloader, model, loss_fn):\n",
    "    size = len(dataloader.dataset)\n",
    "    num_batches = len(dataloader)\n",
    "    val_loss, correct = 0, 0\n",
    "\n",
    "    with torch.no_grad():\n",
    "        for X, y in dataloader:\n",
    "            pred = model(X)\n",
    "            val_loss += loss_fn(pred, y).item()\n",
    "            correct += (pred.argmax(1) == y).type(torch.float).sum().item()\n",
    "\n",
    "    val_loss /= num_batches\n",
    "    correct /= size\n",
    "    print(f\"Val Error: \\n Accuracy: {(100*correct):>0.1f}%, Avg loss: {val_loss:>8f} \\n\")\n",
    "    \n",
    "    \n",
    "loss_fn = nn.CrossEntropyLoss()\n",
    "optimizer = torch.optim.SGD(model.parameters(), lr=learning_rate)\n",
    "\n",
    "epochs = 10\n",
    "for t in range(epochs):\n",
    "    print(f\"Epoch {t+1}\\n-------------------------------\")\n",
    "    train_loop(train_dataloader, model, loss_fn, optimizer)\n",
    "    val_loop(val_dataloader, model, loss_fn)\n",
    "print(\"Done!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "45495ee8",
   "metadata": {},
   "source": [
    "## Exercise 2: PyTorch Lightning\n",
    "### Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "46541f75",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pytorch_lightning as pl\n",
    "from pytorch_lightning.callbacks import EarlyStopping, ModelCheckpoint"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5423aa1f",
   "metadata": {},
   "source": [
    "1. Complete the missing validation step: After the forward pass and the loss function: Log the loss and the accuracy. You might need to use the `validation_epoch_end` method to calculate the validation accuracy after the epoch."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "ba341d97",
   "metadata": {},
   "outputs": [],
   "source": [
    "class PLModule(pl.LightningModule):\n",
    "    def __init__(self, model, data, hparms):\n",
    "        super().__init__()\n",
    "        self.model = model\n",
    "        self.crit = nn.CrossEntropyLoss()\n",
    "        self.hparams.update(hparams)\n",
    "        self.data = data\n",
    "        \n",
    "    def training_step(self, batch, batch_idx):\n",
    "        x, y = batch[0], batch[1]\n",
    "        y_hat = self.model(x)\n",
    "        loss = self.crit(y_hat, y)\n",
    "        self.log(\"loss\", loss)\n",
    "        return loss\n",
    "\n",
    "    def validation_step(self, batch, batch_idx):\n",
    "        x, y = batch[0], batch[1]\n",
    "        y_hat = self.model(x)\n",
    "        loss = self.crit(y_hat, y)\n",
    "        self.log(\"val_loss\", loss)\n",
    "        return (y_hat.argmax(1) == y).type(torch.float).sum().item(), len(x)\n",
    "    \n",
    "    def validation_epoch_end(self, validation_step_outputs):\n",
    "        count = sum([i[0] for i in validation_step_outputs])\n",
    "        size = sum([i[1] for i in validation_step_outputs])\n",
    "        self.log(\"val_acc\", count/size)\n",
    "        \n",
    "    def train_dataloader(self):\n",
    "        return DataLoader(self.data[0], batch_size=self.hparams[\"batch_size\"])\n",
    "\n",
    "    def val_dataloader(self):\n",
    "        return DataLoader(self.data[1], batch_size=self.hparams[\"batch_size\"])\n",
    "\n",
    "\n",
    "    def configure_optimizers(self):\n",
    "        return torch.optim.SGD(self.model.parameters(),\n",
    "                                self.hparams[\"learning_rate\"])\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0aea36e9",
   "metadata": {},
   "source": [
    "2. Create the `hparams` dictionary containing our `learning_rate` (0.001), the `batch_size` (64) and the amount of `epochs` that you want to train. This dictionary is passed to the pytorch lightning module."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "32362b19",
   "metadata": {},
   "outputs": [],
   "source": [
    "hparams = {\"learning_rate\": 1e-3, \"batch_size\": 64, \"epochs\": 10}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "25b375c5",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "GPU available: True, used: True\n",
      "TPU available: False, using: 0 TPU cores\n",
      "IPU available: False, using: 0 IPUs\n",
      "LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]\n",
      "\n",
      "  | Name  | Type             | Params\n",
      "-------------------------------------------\n",
      "0 | model | ClassicNN        | 669 K \n",
      "1 | crit  | CrossEntropyLoss | 0     \n",
      "-------------------------------------------\n",
      "669 K     Trainable params\n",
      "0         Non-trainable params\n",
      "669 K     Total params\n",
      "2.679     Total estimated model params size (MB)\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation sanity check: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/tobi/cloud/Uni/sem-3/idp/test_gpu/venv/lib/python3.9/site-packages/pytorch_lightning/trainer/data_loading.py:132: UserWarning: The dataloader, val_dataloader 0, does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` (try 24 which is the number of cpus on this machine) in the `DataLoader` init to improve performance.\n",
      "  rank_zero_warn(\n",
      "/home/tobi/cloud/Uni/sem-3/idp/test_gpu/venv/lib/python3.9/site-packages/pytorch_lightning/trainer/data_loading.py:132: UserWarning: The dataloader, train_dataloader, does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` (try 24 which is the number of cpus on this machine) in the `DataLoader` init to improve performance.\n",
      "  rank_zero_warn(\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "2885fc9cfd744bdf8f51f9e41509ef6d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Training: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "model = ClassicNN()\n",
    "pl_module = PLModule(model, (training_data, val_data), hparams)\n",
    "trainer = pl.Trainer(\n",
    "    #callbacks=callbacks,\n",
    "    max_epochs=hparams[\"epochs\"],\n",
    "    deterministic=True,\n",
    "    # TODO: uncomment out if you have a GPU\n",
    "    #gpus=[0],\n",
    "    #profiler=\"simple\",\n",
    ")\n",
    "trainer.fit(pl_module)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "0b8f4cd3",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "      <iframe id=\"tensorboard-frame-90fc59c89fabeb22\" width=\"100%\" height=\"800\" frameborder=\"0\">\n",
       "      </iframe>\n",
       "      <script>\n",
       "        (function() {\n",
       "          const frame = document.getElementById(\"tensorboard-frame-90fc59c89fabeb22\");\n",
       "          const url = new URL(\"/\", window.location);\n",
       "          const port = 6006;\n",
       "          if (port) {\n",
       "            url.port = port;\n",
       "          }\n",
       "          frame.src = url;\n",
       "        })();\n",
       "      </script>\n",
       "    "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Visualize the logged data in tensorboard:\n",
    "%load_ext tensorboard\n",
    "%tensorboard --logdir lightning_logs"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "446ecb49",
   "metadata": {},
   "source": [
    "Possible further steps to go down the rabbit hole:\n",
    "- Train more epochs\n",
    "- Play around with the two differnt models. Extend the models with your own ideas.\n",
    "- Add early stopping and model checkpoints\n",
    "- If you have a GPU, try out GPU training\n",
    "- Research how we can use [TorchMetrics](https://torchmetrics.readthedocs.io/en/stable/pages/lightning.html) to simpify metric logging.\n",
    "- Replace SGD with the Adam optimizer and add weight decay to combat overfitting\n",
    "- profile the training to find out how much faster GPU training is. What takes the most time in your training?\n",
    "- Load the data with `num_workers=6` to fully utilze your CPU\n",
    "- Visualize the wrongly classified images. Why do you think they are wrongly classified?\n",
    "- Use a different dataset e.g. FashionMNIST. Do you achive a better accuracy? Why could there be a difference?\n",
    "- Look up transfer learning. How does pytorch support transfer learning? Use a pretrained VGG net\n",
    "- Implement a ResNet variant"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2395360e",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}