Last active
May 16, 2018 04:16
-
-
Save xu-song/671914119955c5963204d2ae9e9f2b62 to your computer and use it in GitHub Desktop.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| { | |
| "cells": [ | |
| { | |
| "cell_type": "code", | |
| "execution_count": null, | |
| "metadata": { | |
| "collapsed": false | |
| }, | |
| "outputs": [], | |
| "source": [ | |
| "%matplotlib inline" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": {}, | |
| "source": [ | |
| "\nNeural Transfer with PyTorch\n============================\n**Author**: `Alexis Jacq <https://alexis-jacq.github.io>`_\n\nIntroduction\n------------\n\nWelcome! This tutorial explains how to impletment the\n`Neural-Style <https://arxiv.org/abs/1508.06576>`__ algorithm developed\nby Leon A. Gatys, Alexander S. Ecker and Matthias Bethge.\n\nNeural what?\n~~~~~~~~~~~~\n\nThe Neural-Style, or Neural-Transfer, is an algorithm that takes as\ninput a content-image (e.g. a tortle), a style-image (e.g. artistic\nwaves) and return the content of the content-image as if it was\n'painted' using the artistic style of the style-image:\n\n.. figure:: /_static/img/neural-style/neuralstyle.png\n :alt: content1\n\nHow does it work?\n~~~~~~~~~~~~~~~~~\n\nThe principle is simple: we define two distances, one for the content\n($D_C$) and one for the style ($D_S$). $D_C$ measures\nhow different the content is between two images, while $D_S$\nmeasures how different the style is between two images. Then, we take a\nthird image, the input, (e.g. a with noise), and we transform it in\norder to both minimize its content-distance with the content-image and\nits style-distance with the style-image.\n\nOK. How does it work?\n^^^^^^^^^^^^^^^^^^^^^\n\nWell, going further requires some mathematics. Let $C_{nn}$ be a\npre-trained deep convolutional neural network and $X$ be any\nimage. $C_{nn}(X)$ is the network fed by $X$ (containing\nfeature maps at all layers). Let $F_{XL} \\in C_{nn}(X)$ be the\nfeature maps at depth layer $L$, all vectorized and concatenated\nin one single vector. We simply define the content of $X$ at layer\n$L$ by $F_{XL}$. Then, if $Y$ is another image of same\nthe size than $X$, we define the distance of content at layer\n$L$ as follow:\n\n\\begin{align}D_C^L(X,Y) = \\|F_{XL} - F_{YL}\\|^2 = \\sum_i (F_{XL}(i) - F_{YL}(i))^2\\end{align}\n\nWhere $F_{XL}(i)$ is the $i^{th}$ element of $F_{XL}$.\nThe style is a bit less trivial to define. Let $F_{XL}^k$ with\n$k \\leq K$ be the vectorized $k^{th}$ of the $K$\nfeature maps at layer $L$. The style $G_{XL}$ of $X$\nat layer $L$ is defined by the Gram produce of all vectorized\nfeature maps $F_{XL}^k$ with $k \\leq K$. In other words,\n$G_{XL}$ is a $K$\\ x\\ $K$ matrix and the element\n$G_{XL}(k,l)$ at the $k^{th}$ line and $l^{th}$ column\nof $G_{XL}$ is the vectorial produce between $F_{XL}^k$ and\n$F_{XL}^l$ :\n\n\\begin{align}G_{XL}(k,l) = \\langle F_{XL}^k, F_{XL}^l\\rangle = \\sum_i F_{XL}^k(i) . F_{XL}^l(i)\\end{align}\n\nWhere $F_{XL}^k(i)$ is the $i^{th}$ element of\n$F_{XL}^k$. We can see $G_{XL}(k,l)$ as a measure of the\ncorrelation between feature maps $k$ and $l$. In that way,\n$G_{XL}$ represents the correlation matrix of feature maps of\n$X$ at layer $L$. Note that the size of $G_{XL}$ only\ndepends on the number of feature maps, not on the size of $X$.\nThen, if $Y$ is another image *of any size*, we define the\ndistance of style at layer $L$ as follow:\n\n\\begin{align}D_S^L(X,Y) = \\|G_{XL} - G_{YL}\\|^2 = \\sum_{k,l} (G_{XL}(k,l) - G_{YL}(k,l))^2\\end{align}\n\nIn order to minimize in one shot $D_C(X,C)$ between a variable\nimage $X$ and target content-image $C$ and $D_S(X,S)$\nbetween $X$ and target style-image $S$, both computed at\nseveral layers , we compute and sum the gradients (derivative with\nrespect to $X$) of each distance at each wanted layer:\n\n\\begin{align}\\nabla_{\textit{total}}(X,S,C) = \\sum_{L_C} w_{CL_C}.\\nabla_{\textit{content}}^{L_C}(X,C) + \\sum_{L_S} w_{SL_S}.\\nabla_{\textit{style}}^{L_S}(X,S)\\end{align}\n\nWhere $L_C$ and $L_S$ are respectivement the wanted layers\n(arbitrary stated) of content and style and $w_{CL_C}$ and\n$w_{SL_S}$ the weights (arbitrary stated) associated with the\nstyle or the content at each wanted layer. Then, we run a gradient\ndescent over $X$:\n\n\\begin{align}X \\leftarrow X - \\alpha \\nabla_{\textit{total}}(X,S,C)\\end{align}\n\nOk. That's enough with maths. If you want to go deeper (how to compute\nthe gradients) **we encourage you to read the original paper** by Leon\nA. Gatys and AL, where everything is much better and much clearer\nexplained.\n\nFor our implementation in PyTorch, we already have everything\nwe need: indeed, with PyTorch, all the gradients are automatically and\ndynamically computed for you (while you use functions from the library).\nThis is why the implementation of this algorithm becomes very\ncomfortable with PyTorch.\n\nPyTorch implementation\n----------------------\n\nIf you are not sure to understand all the mathematics above, you will\nprobably get it by implementing it. If you are discovering PyTorch, we\nrecommend you to first read this :doc:`Introduction to\nPyTorch </beginner/deep_learning_60min_blitz>`.\n\nPackages\n~~~~~~~~\n\nWe will have recourse to the following packages:\n\n- ``torch``, ``torch.nn``, ``numpy`` (indispensables packages for\n neural networks with PyTorch)\n- ``torch.optim`` (efficient gradient descents)\n- ``PIL``, ``PIL.Image``, ``matplotlib.pyplot`` (load and display\n images)\n- ``torchvision.transforms`` (treat PIL images and transform into torch\n tensors)\n- ``torchvision.models`` (train or load pre-trained models)\n- ``copy`` (to deep copy the models; system package)\n\n" | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": null, | |
| "metadata": { | |
| "collapsed": false | |
| }, | |
| "outputs": [], | |
| "source": [ | |
| "from __future__ import print_function\n\nimport torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nimport torch.optim as optim\n\nfrom PIL import Image\nimport matplotlib.pyplot as plt\n\nimport torchvision.transforms as transforms\nimport torchvision.models as models\n\nimport copy" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": {}, | |
| "source": [ | |
| "Cuda\n~~~~\n\nIf you have a GPU on your computer, it is preferable to run the\nalgorithm on it, especially if you want to try larger networks (like\nVGG). For this, we have ``torch.cuda.is_available()`` that returns\n``True`` if you computer has an available GPU. Then, we can set the\n``torch.device`` that will be used in this script. Then, we will use\nthe method ``.to(device)`` that moves a tensor or a module to the desired\ndevice. When we want to move back this tensor or module to the\nCPU (e.g. to use numpy), we can use the ``.cpu()`` method.\n\n" | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": null, | |
| "metadata": { | |
| "collapsed": false | |
| }, | |
| "outputs": [], | |
| "source": [ | |
| "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": {}, | |
| "source": [ | |
| "Load images\n~~~~~~~~~~~\n\nIn order to simplify the implementation, let's start by importing a\nstyle and a content image of the same dimentions. We then scale them to\nthe desired output image size (128 or 512 in the example, depending on gpu\navailablity) and transform them into torch tensors, ready to feed\na neural network:\n\n.. Note::\n Here are links to download the images required to run the tutorial:\n `picasso.jpg <http://pytorch.org/tutorials/_static/img/neural-style/picasso.jpg>`__ and\n `dancing.jpg <http://pytorch.org/tutorials/_static/img/neural-style/dancing.jpg>`__.\n Download these two images and add them to a directory\n with name ``images``\n\n" | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": null, | |
| "metadata": { | |
| "collapsed": false | |
| }, | |
| "outputs": [], | |
| "source": [ | |
| "# desired size of the output image\nimsize = 512 if torch.cuda.is_available() else 128 # use small size if no gpu\n\nloader = transforms.Compose([\n transforms.Resize(imsize), # scale imported image\n transforms.ToTensor()]) # transform it into a torch tensor\n\n\ndef image_loader(image_name):\n image = Image.open(image_name)\n # fake batch dimension required to fit network's input dimensions\n image = loader(image).unsqueeze(0)\n return image.to(device, torch.float)\n\n\nstyle_img = image_loader(\"images/picasso.jpg\")\ncontent_img = image_loader(\"images/dancing.jpg\")\n\nassert style_img.size() == content_img.size(), \\\n \"we need to import style and content images of the same size\"" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": {}, | |
| "source": [ | |
| "Imported PIL images has values between 0 and 255. Transformed into torch\ntensors, their values are between 0 and 1. This is an important detail:\nneural networks from torch library are trained with 0-1 tensor image. If\nyou try to feed the networks with 0-255 tensor images the activated\nfeature maps will have no sense. This is not the case with pre-trained\nnetworks from the Caffe library: they are trained with 0-255 tensor\nimages.\n\nDisplay images\n~~~~~~~~~~~~~~\n\nWe will use ``plt.imshow`` to display images. So we need to first\nreconvert them into PIL images:\n\n\n" | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": null, | |
| "metadata": { | |
| "collapsed": false | |
| }, | |
| "outputs": [], | |
| "source": [ | |
| "unloader = transforms.ToPILImage() # reconvert into PIL image\n\nplt.ion()\n\ndef imshow(tensor, title=None):\n image = tensor.cpu().clone() # we clone the tensor to not do changes on it\n image = image.squeeze(0) # remove the fake batch dimension\n image = unloader(image)\n plt.imshow(image)\n if title is not None:\n plt.title(title)\n plt.pause(0.001) # pause a bit so that plots are updated\n\n\nplt.figure()\nimshow(style_img, title='Style Image')\n\nplt.figure()\nimshow(content_img, title='Content Image')" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": {}, | |
| "source": [ | |
| "Content loss\n~~~~~~~~~~~~\n\nThe content loss is a function that takes as input the feature maps\n$F_{XL}$ at a layer $L$ in a network fed by $X$ and\nreturn the weigthed content distance $w_{CL}.D_C^L(X,C)$ between\nthis image and the content image. Hence, the weight $w_{CL}$ and\nthe target content $F_{CL}$ are parameters of the function. We\nimplement this function as a torch module with a constructor that takes\nthese parameters as input. The distance $\\|F_{XL} - F_{YL}\\|^2$ is\nthe Mean Square Error between the two sets of feature maps, that can be\ncomputed using a criterion ``nn.MSELoss`` stated as a third parameter.\n\nWe will add our content losses at each desired layer as additive modules\nof the neural network. That way, each time we will feed the network with\nan input image $X$, all the content losses will be computed at the\ndesired layers and, thanks to autograd, all the gradients will be\ncomputed. For that, we just need to make the ``forward`` method of our\nmodule returning the input: the module becomes a ''transparent layer''\nof the neural network. The computed loss is saved as a parameter of the\nmodule.\n\nFinally, we define a fake ``backward`` method, that just call the\nbackward method of ``nn.MSELoss`` in order to reconstruct the gradient.\nThis method returns the computed loss: this will be useful when running\nthe gradient descent in order to display the evolution of style and\ncontent losses.\n\n\n" | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": null, | |
| "metadata": { | |
| "collapsed": false | |
| }, | |
| "outputs": [], | |
| "source": [ | |
| "class ContentLoss(nn.Module):\n\n def __init__(self, target,):\n super(ContentLoss, self).__init__()\n # we 'detach' the target content from the tree used\n # to dynamically compute the gradient: this is a stated value,\n # not a variable. Otherwise the forward method of the criterion\n # will throw an error.\n self.target = target.detach()\n\n def forward(self, input):\n self.loss = F.mse_loss(input, self.target)\n return input" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": {}, | |
| "source": [ | |
| ".. Note::\n **Important detail**: this module, although it is named ``ContentLoss``,\n is not a true PyTorch Loss function. If you want to define your content\n loss as a PyTorch Loss, you have to create a PyTorch autograd Function\n and to recompute/implement the gradient by the hand in the ``backward``\n method.\n\nStyle loss\n~~~~~~~~~~\n\nFor the style loss, we need first to define a module that compute the\ngram produce $G_{XL}$ given the feature maps $F_{XL}$ of the\nneural network fed by $X$, at layer $L$. Let\n$\\hat{F}_{XL}$ be the re-shaped version of $F_{XL}$ into a\n$K$\\ x\\ $N$ matrix, where $K$ is the number of feature\nmaps at layer $L$ and $N$ the lenght of any vectorized\nfeature map $F_{XL}^k$. The $k^{th}$ line of\n$\\hat{F}_{XL}$ is $F_{XL}^k$. We let you check that\n$\\hat{F}_{XL} \\cdot \\hat{F}_{XL}^T = G_{XL}$. Given that, it\nbecomes easy to implement our module:\n\n\n" | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": null, | |
| "metadata": { | |
| "collapsed": false | |
| }, | |
| "outputs": [], | |
| "source": [ | |
| "def gram_matrix(input):\n a, b, c, d = input.size() # a=batch size(=1)\n # b=number of feature maps\n # (c,d)=dimensions of a f. map (N=c*d)\n\n features = input.view(a * b, c * d) # resise F_XL into \\hat F_XL\n\n G = torch.mm(features, features.t()) # compute the gram product\n\n # we 'normalize' the values of the gram matrix\n # by dividing by the number of element in each feature maps.\n return G.div(a * b * c * d)" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": {}, | |
| "source": [ | |
| "The longer is the feature maps dimension $N$, the bigger are the\nvalues of the Gram matrix. Therefore, if we don't normalize by $N$,\nthe loss computed at the first layers (before pooling layers) will have\nmuch more importance during the gradient descent. We dont want that,\nsince the most interesting style features are in the deepest layers!\n\nThen, the style loss module is implemented exactly the same way than the\ncontent loss module, but it compares the difference in Gram matrices of target\nand input\n\n\n" | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": null, | |
| "metadata": { | |
| "collapsed": false | |
| }, | |
| "outputs": [], | |
| "source": [ | |
| "class StyleLoss(nn.Module):\n\n def __init__(self, target_feature):\n super(StyleLoss, self).__init__()\n self.target = gram_matrix(target_feature).detach()\n\n def forward(self, input):\n G = gram_matrix(input)\n self.loss = F.mse_loss(G, self.target)\n return input" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": {}, | |
| "source": [ | |
| "Load the neural network\n~~~~~~~~~~~~~~~~~~~~~~~\n\nNow, we have to import a pre-trained neural network. As in the paper, we\nare going to use a pretrained VGG network with 19 layers (VGG19).\n\nPyTorch's implementation of VGG is a module divided in two child\n``Sequential`` modules: ``features`` (containing convolution and pooling\nlayers) and ``classifier`` (containing fully connected layers). We are\njust interested by ``features``:\nSome layers have different behavior in training and in evaluation. Since we\nare using it as a feature extractor. We will use ``.eval()`` to set the\nnetwork in evaluation mode.\n\n\n" | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": null, | |
| "metadata": { | |
| "collapsed": false | |
| }, | |
| "outputs": [], | |
| "source": [ | |
| "cnn = models.vgg19(pretrained=True).features.to(device).eval()" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": {}, | |
| "source": [ | |
| "Additionally, VGG networks are trained on images with each channel normalized\nby mean=[0.485, 0.456, 0.406] and std=[0.229, 0.224, 0.225]. We will use them\nto normalize the image before sending into the network.\n\n\n" | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": null, | |
| "metadata": { | |
| "collapsed": false | |
| }, | |
| "outputs": [], | |
| "source": [ | |
| "cnn_normalization_mean = torch.tensor([0.485, 0.456, 0.406]).to(device)\ncnn_normalization_std = torch.tensor([0.229, 0.224, 0.225]).to(device)\n\n# create a module to normalize input image so we can easily put it in a\n# nn.Sequential\nclass Normalization(nn.Module):\n def __init__(self, mean, std):\n super(Normalization, self).__init__()\n # .view the mean and std to make them [C x 1 x 1] so that they can\n # directly work with image Tensor of shape [B x C x H x W].\n # B is batch size. C is number of channels. H is height and W is width.\n self.mean = torch.tensor(mean).view(-1, 1, 1)\n self.std = torch.tensor(std).view(-1, 1, 1)\n\n def forward(self, img):\n # normalize img\n return (img - self.mean) / self.std" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": {}, | |
| "source": [ | |
| "A ``Sequential`` module contains an ordered list of child modules. For\ninstance, ``vgg19.features`` contains a sequence (Conv2d, ReLU,\nMaxPool2d, Conv2d, ReLU...) aligned in the right order of depth. As we\nsaid in *Content loss* section, we wand to add our style and content\nloss modules as additive 'transparent' layers in our network, at desired\ndepths. For that, we construct a new ``Sequential`` module, in which we\nare going to add modules from ``vgg19`` and our loss modules in the\nright order:\n\n\n" | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": null, | |
| "metadata": { | |
| "collapsed": false | |
| }, | |
| "outputs": [], | |
| "source": [ | |
| "# desired depth layers to compute style/content losses :\ncontent_layers_default = ['conv_4']\nstyle_layers_default = ['conv_1', 'conv_2', 'conv_3', 'conv_4', 'conv_5']\n\ndef get_style_model_and_losses(cnn, normalization_mean, normalization_std,\n style_img, content_img,\n content_layers=content_layers_default,\n style_layers=style_layers_default):\n cnn = copy.deepcopy(cnn)\n\n # normalization module\n normalization = Normalization(normalization_mean, normalization_std).to(device)\n\n # just in order to have an iterable access to or list of content/syle\n # losses\n content_losses = []\n style_losses = []\n\n # assuming that cnn is a nn.Sequential, so we make a new nn.Sequential\n # to put in modules that are supposed to be activated sequentially\n model = nn.Sequential(normalization)\n\n i = 0 # increment every time we see a conv\n for layer in cnn.children():\n if isinstance(layer, nn.Conv2d):\n i += 1\n name = 'conv_{}'.format(i)\n elif isinstance(layer, nn.ReLU):\n name = 'relu_{}'.format(i)\n # The in-place version doesn't play very nicely with the ContentLoss\n # and StyleLoss we insert below. So we replace with out-of-place\n # ones here.\n layer = nn.ReLU(inplace=False)\n elif isinstance(layer, nn.MaxPool2d):\n name = 'pool_{}'.format(i)\n elif isinstance(layer, nn.BatchNorm2d):\n name = 'bn_{}'.format(i)\n else:\n raise RuntimeError('Unrecognized layer: {}'.format(layer.__class__.__name__))\n\n model.add_module(name, layer)\n\n if name in content_layers:\n # add content loss:\n target = model(content_img).detach()\n content_loss = ContentLoss(target)\n model.add_module(\"content_loss_{}\".format(i), content_loss)\n content_losses.append(content_loss)\n\n if name in style_layers:\n # add style loss:\n target_feature = model(style_img).detach()\n style_loss = StyleLoss(target_feature)\n model.add_module(\"style_loss_{}\".format(i), style_loss)\n style_losses.append(style_loss)\n\n # now we trim off the layers after the last content and style losses\n for i in range(len(model) - 1, -1, -1):\n if isinstance(model[i], ContentLoss) or isinstance(model[i], StyleLoss):\n break\n\n model = model[:(i + 1)]\n\n return model, style_losses, content_losses" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": {}, | |
| "source": [ | |
| ".. Note::\n In the paper they recommend to change max pooling layers into\n average pooling. With AlexNet, that is a small network compared to VGG19\n used in the paper, we are not going to see any difference of quality in\n the result. However, you can use these lines instead if you want to do\n this substitution:\n\n ::\n\n # avgpool = nn.AvgPool2d(kernel_size=layer.kernel_size,\n # stride=layer.stride, padding = layer.padding)\n # model.add_module(name,avgpool)\n\n" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": {}, | |
| "source": [ | |
| "Input image\n~~~~~~~~~~~\n\nAgain, in order to simplify the code, we take an image of the same\ndimensions than content and style images. This image can be a white\nnoise, or it can also be a copy of the content-image.\n\n\n" | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": null, | |
| "metadata": { | |
| "collapsed": false | |
| }, | |
| "outputs": [], | |
| "source": [ | |
| "input_img = content_img.clone()\n# if you want to use a white noise instead uncomment the below line:\n# input_img = torch.randn(content_img.data.size(), device=device)\n\n# add the original input image to the figure:\nplt.figure()\nimshow(input_img, title='Input Image')" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": {}, | |
| "source": [ | |
| "Gradient descent\n~~~~~~~~~~~~~~~~\n\nAs Leon Gatys, the author of the algorithm, suggested\n`here <https://discuss.pytorch.org/t/pytorch-tutorial-for-neural-transfert-of-artistic-style/336/20?u=alexis-jacq>`__,\nwe will use L-BFGS algorithm to run our gradient descent. Unlike\ntraining a network, we want to train the input image in order to\nminimise the content/style losses. We would like to simply create a\nPyTorch L-BFGS optimizer ``optim.LBFGS``, passing our image as the\nTensor to optimize. We use ``.requires_grad_()`` to make sure that this\nimage requires gradient.\n\n\n" | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": null, | |
| "metadata": { | |
| "collapsed": false | |
| }, | |
| "outputs": [], | |
| "source": [ | |
| "def get_input_optimizer(input_img):\n # this line to show that input is a parameter that requires a gradient\n optimizer = optim.LBFGS([input_img.requires_grad_()])\n return optimizer" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": {}, | |
| "source": [ | |
| "**Last step**: the loop of gradient descent. At each step, we must feed\nthe network with the updated input in order to compute the new losses,\nwe must run the ``backward`` methods of each loss to dynamically compute\ntheir gradients and perform the step of gradient descent. The optimizer\nrequires as argument a \"closure\": a function that reevaluates the model\nand returns the loss.\n\nHowever, there's a small catch. The optimized image may take its values\nbetween $-\\infty$ and $+\\infty$ instead of staying between 0\nand 1. In other words, the image might be well optimized and have absurd\nvalues. In fact, we must perform an optimization under constraints in\norder to keep having right vaues into our input image. There is a simple\nsolution: at each step, to correct the image to maintain its values into\nthe 0-1 interval.\n\n\n" | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": null, | |
| "metadata": { | |
| "collapsed": false | |
| }, | |
| "outputs": [], | |
| "source": [ | |
| "def run_style_transfer(cnn, normalization_mean, normalization_std,\n content_img, style_img, input_img, num_steps=300,\n style_weight=1000000, content_weight=1):\n \"\"\"Run the style transfer.\"\"\"\n print('Building the style transfer model..')\n model, style_losses, content_losses = get_style_model_and_losses(cnn,\n normalization_mean, normalization_std, style_img, content_img)\n optimizer = get_input_optimizer(input_img)\n\n print('Optimizing..')\n run = [0]\n while run[0] <= num_steps:\n\n def closure():\n # correct the values of updated input image\n input_img.data.clamp_(0, 1)\n\n optimizer.zero_grad()\n model(input_img)\n style_score = 0\n content_score = 0\n\n for sl in style_losses:\n style_score += sl.loss\n for cl in content_losses:\n content_score += cl.loss\n\n style_score *= style_weight\n content_score *= content_weight\n\n loss = style_score + content_score\n loss.backward()\n\n run[0] += 1\n if run[0] % 50 == 0:\n print(\"run {}:\".format(run))\n print('Style Loss : {:4f} Content Loss: {:4f}'.format(\n style_score.item(), content_score.item()))\n print()\n\n return style_score + content_score\n\n optimizer.step(closure)\n\n # a last correction...\n input_img.data.clamp_(0, 1)\n\n return input_img" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": {}, | |
| "source": [ | |
| "Finally, run the algorithm\n\n" | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": null, | |
| "metadata": { | |
| "collapsed": false | |
| }, | |
| "outputs": [], | |
| "source": [ | |
| "output = run_style_transfer(cnn, cnn_normalization_mean, cnn_normalization_std,\n content_img, style_img, input_img)\n\nplt.figure()\nimshow(output, title='Output Image')\n\n# sphinx_gallery_thumbnail_number = 4\nplt.ioff()\nplt.show()" | |
| ] | |
| } | |
| ], | |
| "metadata": { | |
| "kernelspec": { | |
| "display_name": "Python 3", | |
| "language": "python", | |
| "name": "python3" | |
| }, | |
| "language_info": { | |
| "codemirror_mode": { | |
| "name": "ipython", | |
| "version": 3 | |
| }, | |
| "file_extension": ".py", | |
| "mimetype": "text/x-python", | |
| "name": "python", | |
| "nbconvert_exporter": "python", | |
| "pygments_lexer": "ipython3", | |
| "version": "3.6.5" | |
| } | |
| }, | |
| "nbformat": 4, | |
| "nbformat_minor": 0 | |
| } |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
todo