Skip to content

Instantly share code, notes, and snippets.

@sadnow

sadnow/dev-rev-animationkit_rife-realesrgan_upscaling-interpolation.ipynb Secret

Created December 28, 2021 18:00
Show Gist options
  • Save sadnow/0da9e55b7ae5e3ae2203560cd22d1cc8 to your computer and use it in GitHub Desktop.
Save sadnow/0da9e55b7ae5e3ae2203560cd22d1cc8 to your computer and use it in GitHub Desktop.
Download ZIP
dev-rev AnimationKit_Rife-RealESRGAN_Upscaling-Interpolation
Display the source blob
Display the rendered blob
Raw
dev-rev-animationkit_rife-realesrgan_upscaling-interpolation.ipynb
{
"nbformat": 4,
"nbformat_minor": 0,
"metadata": {
"colab": {
"name": "dev-rev AnimationKit_Rife-RealESRGAN_Upscaling-Interpolation",
"provenance": [],
"collapsed_sections": [],
"private_outputs": true,
"machine_shape": "hm",
"include_colab_link": true
},
"kernelspec": {
"name": "python3",
"display_name": "Python 3"
},
"language_info": {
"name": "python"
},
"accelerator": "GPU"
},
"cells": [
{
"cell_type": "markdown",
"metadata": {
"id": "view-in-github",
"colab_type": "text"
},
"source": [
"<a href=\"https://colab.research.google.com/gist/sadnow/0da9e55b7ae5e3ae2203560cd22d1cc8/dev-rev-animationkit_rife-realesrgan_upscaling-interpolation.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "CppIQlPhhwhs"
},
"source": [
"# AnimationKit AI - Synthetic Video Upscaling & Interpolation Tool\n",
"\n",
"**ALPHA 2.6: Warning - Testing branch; This rev has only been tested with default settings.**\n",
"\n",
"\n",
"#Changelog:\n",
"\n",
"- Partial implementation of method-b for imported_mp4s\n",
"- Succesful implementation and testing of RIFE animevideo\n",
"\n",
"Not completely implemented/tested:\n",
"\n",
"- audio support\n",
"- method-a (previous method) untested\n",
"- other models untested\n",
"- indiviudal frame support (untested)\n",
"\n",
"\n",
"---\n",
"\n",
"Main (stable) branch and any potential updates can be found here: https://github.com/sadnow/AnimationKit-AI_Upscaling-Interpolation_RIFE-RealESRGAN\n",
"\n",
"---\n",
"#Credits\n",
"\n",
"Motion smoothing utilizes [Practical-RIFE](https://github.com/hzwer/Practical-RIFE) 4.0 model.\n",
"\n",
"Upscaling uses Real-ESRGAN (https://github.com/xinntao/Real-ESRGAN). A demo notebook for static images can be found here: https://colab.research.google.com/drive/1k2Zod6kSHEvraybHl50Lys0LerhyTMCo?usp=sharing. The demo was based on the following paper: [''Real-ESRGAN: Training Real-World Blind Super-Resolution with Pure Synthetic Data''](https://arxiv.org/abs/2107.10833). A demo can be found at https://raw.githubusercontent.com/xinntao/Real-ESRGAN"
]
},
{
"cell_type": "code",
"metadata": {
"id": "VA1PHoJrRiK9",
"cellView": "form"
},
"source": [
"# @title AnimationKit AI License(s): { form-width: \"300px\" }\n",
"# @markdown Practical-RIFE uses the MIT License.\n",
"\n",
"# @markdown Real-ESRGAN uses the BSD 3-Clause License.\n",
"\n",
"# @markdown *To view the included licenses, click \"Show code.\"* \n",
"# -------------------------------------------------\n",
"# Below is the MIT license for RIFE & Practical-RIFE\n",
"# ------------------------------------------------\n",
"\n",
"# MIT License\n",
"\n",
"# Copyright (c) 2021 hzwer\n",
"\n",
"# Permission is hereby granted, free of charge, to any person obtaining a copy\n",
"# of this software and associated documentation files (the \"Software\"), to deal\n",
"# in the Software without restriction, including without limitation the rights\n",
"# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell\n",
"# copies of the Software, and to permit persons to whom the Software is\n",
"# furnished to do so, subject to the following conditions:\n",
"\n",
"# The above copyright notice and this permission notice shall be included in all\n",
"# copies or substantial portions of the Software.\n",
"\n",
"# THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\n",
"# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\n",
"# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\n",
"# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\n",
"# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\n",
"# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\n",
"# SOFTWARE.\n",
"\n",
"# -------------------------------------------------\n",
"# Below is the BSD3-Clause License for Real-ESRGAN.\n",
"# ------------------------------------------------\n",
"\n",
"# BSD 3-Clause License\n",
"\n",
"# Copyright (c) 2021, Xintao Wang\n",
"# All rights reserved.\n",
"\n",
"# Redistribution and use in source and binary forms, with or without\n",
"# modification, are permitted provided that the following conditions are met:\n",
"\n",
"# 1. Redistributions of source code must retain the above copyright notice, this\n",
"# list of conditions and the following disclaimer.\n",
"\n",
"# 2. Redistributions in binary form must reproduce the above copyright notice,\n",
"# this list of conditions and the following disclaimer in the documentation\n",
"# and/or other materials provided with the distribution.\n",
"\n",
"# 3. Neither the name of the copyright holder nor the names of its\n",
"# contributors may be used to endorse or promote products derived from\n",
"# this software without specific prior written permission.\n",
"\n",
"# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\"\n",
"# AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE\n",
"# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE\n",
"# DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE\n",
"# FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL\n",
"# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR\n",
"# SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER\n",
"# CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,\n",
"# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE\n",
"# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.\n",
"\n",
"\n"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "yDn13MTGIDVt",
"cellView": "form"
},
"source": [
"#@markdown ##Check GPU\n",
"#@markdown ***This Notebook supports \"Run All. Once everything is set to your liking, you can do `Ctrl+F9`!***\n",
"import torch\n",
"# Check the GPU status\n",
"device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')\n",
"print('Using device:', device)\n",
"\n",
"enable_error_checking = False #saves ram\n",
"if enable_error_checking:\n",
" !nvidia-smi\n",
"else:\n",
" !nvidia-smi\n",
" !nvidia-smi -i 0 -e 0\n",
"#@markdown *Special thanks to sportracer48 and his Discord channel. If you want to make AI animations, PyTTI is king.* https://www.patreon.com/sportsracer48"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "xNsqYx5tEBo5"
},
"source": [
"# Mount Google Drive, Install Libraries, Load Functions\n"
]
},
{
"cell_type": "code",
"metadata": {
"id": "4hULXQndE4xk",
"cellView": "form"
},
"source": [
"#@title { form-width: \"300px\" }\n",
"#@markdown *Note: This project has been tested around Google Drive and may not yet support non-Drive configurations.*\n",
"#https://unix.stackexchange.com/questions/70963/difference-between-2-2-dev-null-dev-null-and-dev-null-21\n",
"import os, sys, re, gc #>/dev/null\n",
"gc.collect()\n",
"from multiprocessing import Process\n",
"\n",
"def Dir_check(a):\n",
" import os.path\n",
" from os import path\n",
" if not path.exists(a):\n",
" print(\"ERROR: \"+a+\" does not exist!\")\n",
"\n",
"def Dir_make(a):\n",
" import os.path\n",
" from os import path\n",
" if not path.exists(a):\n",
" print(\"Creating \"+a+\" ...\")\n",
" !mkdir -p $a\n",
"\n",
"def File_check(a):\n",
" import os.path\n",
" from os import path\n",
" if not path.isfile(a):\n",
" return False\n",
" else:\n",
" return True\n",
"\n",
"# def need_file(a):\n",
"# import os.path\n",
"# from os import path\n",
"# if not path.isfile(a):\n",
"# print(\"Creating\"+a+\"...\")\n",
"# !mkdir -p $a\n",
"\n",
"\n",
"\n",
"# def File_check(a):\n",
"# import os.path\n",
"# from os import path\n",
"# if not path.isfile(a):\n",
"# print(\"ERROR: \"+a+\" does not exist!\")\n",
"\n",
"\n",
"class AnKit:\n",
" #def __init__(self):\n",
" # pass\n",
" import os.path\n",
" from os import path\n",
" #video tools\n",
" def video_sortFrames(sourceframes, destframes): #takes frames from input folder, moves to init_frame_storage\n",
" Dir_check(sourceframes)\n",
" Dir_make(destframes)\n",
" %cd $sourceframes\n",
" print(\"Copying frames to \"+destframes+\" for processing...\")\n",
" !find -maxdepth 1 -name '*.png' -print0 | xargs -0 cp -t $destframes\n",
" %cd $destframes\n",
" #!find -maxdepth 1 -name '*.png' -print0 | xargs -0 'mv \"$0\" \"${0##*_}\"' \n",
" #!find . -type f -name \"*_*.png\" -execdir bash -c 'mv \"$0\" \"${0##*_}\"' {} \\; #removes anything not numbers\n",
" !find . -type f -name \"*.png\" -execdir bash -c 'mv \"$0\" \"${0##*_}\"' {} \\; #removes anything not numbers\n",
" #print(\"Padding filenames in \"+destframes+\".\")\n",
" !rename 's/\\d+/sprintf(\"%05d\",$&)/e' * #adds padding to numbers\n",
" print(\"Finished copying frames to \"+destframes+\".\")\n",
" def video_splitFrames(sourcefile, destframes):\n",
" File_check(sourcefile)\n",
" Dir_make(destframes)\n",
" !ffmpeg -y -r 1 -i $sourcefile -r 1 $destframes/frame%05d.png\n",
" def frames2video(sourceframes,fps,outputmp4):\n",
" Dir_check(sourceframes)\n",
" %cd $sourceframes\n",
" !ffmpeg -framerate $fps -pattern_type glob -i '*.png' -y $outputmp4\n",
" \n",
" #!ffmpeg -framerate $fps -pattern_type glob -i '*.png' -vcodec hevc_nvenc -y $outputmp4\n",
" #this vcodec doesnt work with a100; nixing for now\n",
"\n",
" def video_runUpscale(esrgan,mname,scale,input,output):\n",
" Dir_check(input)\n",
" Dir_make(output)\n",
" %cd $esrgan\n",
" !python inference_realesrgan.py --model_name $mname --input $input --output $output\n",
" def RIFE_video(fps,exp,input,scale,output):\n",
" %cd /content/Practical-RIFE\n",
" !python3 /content/Practical-RIFE/inference_video.py --fps=$fps --exp=$exp --video=$input --scale $scale --output=$output\n",
" def RIFE_frames(fps,exp,input,scale,output): #not currently working right\n",
" %cd /content/Practical-RIFE\n",
" input = input + '/'\n",
" !python3 /content/Practical-RIFE/inference_video.py --fps=$fps --exp=$exp --img=$input --scale $scale --output=$output\n",
" def detect_fps(input): #needs portable\n",
" import re\n",
" fps_ffprobe = !ffprobe -v error -select_streams v -of default=noprint_wrappers=1:nokey=1 -show_entries stream=avg_frame_rate $input\n",
" fps_unfinished = [str(i) for i in fps_ffprobe] # Converting integers into strings\n",
" fps_unfinishedTwo = str(\"\".join(fps_unfinished)) # Join the string values into one string\n",
" numbers = re.findall('[0-9]+', fps_unfinishedTwo)\n",
" newNum = numbers[0:1]\n",
" strings = [str(integer) for integer in newNum]\n",
" a_string = \"\".join(strings)\n",
" fps = int(a_string)\n",
" #print(\"Detected FPS is\",fps)\n",
" return fps\n",
" def detect_duration(input): #needs portable\n",
" import re\n",
" duration_ffprobe = !ffprobe -v error -select_streams v:0 -show_entries stream=duration -of default=noprint_wrappers=1:nokey=1 $input\n",
" duration_unfinished = [str(i) for i in duration_ffprobe] # Converting integers into strings\n",
" duration_unfinishedTwo = str(\"\".join(duration_unfinished)) # Join the string values into one string\n",
" numbers = re.findall('[0-9]+', duration_unfinishedTwo)\n",
" newNum = numbers[0:1]\n",
" strings = [str(integer) for integer in newNum]\n",
" a_string = \"\".join(strings)\n",
" duration = int(a_string)\n",
" #print(\"Detected duration INTEGER (in seconds) is\",duration)\n",
" return duration\n",
" def exp_calc(measured_duration,_target_length_seconds): #needs portable\n",
" import numpy as np\n",
" a = measured_fps * measured_duration\n",
" b = _target_fps * _target_length_seconds\n",
" c = b / a\n",
" l = np.log(c) / np.log(2)\n",
" print(\"Un-rounded --exp is\",l)\n",
" x = round(l)\n",
" if x < 1:\n",
" x = 1\n",
" print(\"Rounding up to an --exp of \",x)\n",
" return x\n",
" #installation for Colab (untested on other platforms)\n",
" #\n",
" #\n",
" def install_ESRGAN():\n",
" print(\"Installing libraries for Real-ESRGAN upscaling.\")\n",
" #!git clone https://github.com/xinntao/Real-ESRGAN.git &> /dev/null #using own fork for longevity\n",
" !git clone https://github.com/sadnow/Real-ESRGAN_AnimationKit Real-ESRGAN &> /dev/null #using own fork for longevity\n",
" %cd Real-ESRGAN\n",
" !pip -q install basicsr\n",
" !pip -q install facexlib\n",
" !pip -q install gfpgan\n",
" !pip -q install -r requirements.txt &> /dev/null\n",
" !python setup.py develop &> /dev/null\n",
" !wget https://github.com/xinntao/Real-ESRGAN/releases/download/v0.1.0/RealESRGAN_x4plus.pth -P experiments/pretrained_models &> /dev/null\n",
" !wget https://github.com/xinntao/Real-ESRGAN/releases/download/v0.2.2.4/RealESRGAN_x4plus_anime_6B.pth -P experiments/pretrained_models &> /dev/null\n",
" !wget https://github.com/xinntao/Real-ESRGAN/releases/download/v0.2.1/RealESRGAN_x2plus.pth -P experiments/pretrained_models &> /dev/null\n",
" print(\"Finished Installing libraries for Real-ESRGAN upscaling.\")\n",
" #\n",
" def install_RIFE():\n",
" %cd /content/\n",
" print(\"Installing libraries for RIFE motion smoothing.\")\n",
" #!git clone https://github.com/hzwer/Practical-RIFE Practical-RIFE &> /dev/null #using my own archive for longevity\n",
" !git clone https://github.com/sadnow/Practical-RIFE_AnimationKit Practical-RIFE &> /dev/null #using my own archive for longevity\n",
" #!gdown --id 1O5KfS3KzZCY3imeCr2LCsntLhutKuAqj &> /dev/null #DEPRECATED 3.8 MODEL\n",
" !gdown --id 1mUK9iON6Es14oK46-cCflRoPTeGiI_A9 &> /dev/null #NEWER 4.0 MODEL\n",
" #!7z e RIFE_trained_model_v3.8.zip &> /dev/null\n",
" !7z e RIFE_trained_model_v4.0.zip &> /dev/null\n",
" !mkdir /content/Practical-RIFE/train_log &> /dev/null\n",
" !mv *.py /content/Practical-RIFE/train_log/ &> /dev/null\n",
" !mv *.pkl /content/Practical-RIFE/train_log/ &> /dev/null\n",
" %cd /content/Practical-RIFE/\n",
" #!gdown --id 1i3xlKb7ax7Y70khcTcuePi6E7crO_dFc &> /dev/null #useless - mp4 demo \n",
" !pip3 install -r requirements.txt &> /dev/null\n",
" print(\"Finsihed Installing libraries for RIFE motion smoothing.\")\n",
" # \n",
" %cd /content/\n",
" def depcrecated_installOldRIFE():\n",
" print(\"Installing libraries for RIFE motion smoothing.\")\n",
" !git clone https://github.com/hzwer/arXiv2020-RIFE RIFE\n",
" !gdown --id 1wsQIhHZ3Eg4_AfCXItFKqqyDMB4NS0Yd\n",
" !7z e RIFE_trained_model_HDv2.zip\n",
" !mkdir /content/RIFE/train_log\n",
" !mv *.pkl /content/RIFE/train_log/\n",
" %cd /content/RIFE/\n",
" !gdown --id 1i3xlKb7ax7Y70khcTcuePi6E7crO_dFc\n",
" !pip3 install -r requirements.txt\n",
" print(\"Done.\")\n",
" print(\"Finsihed Installing libraries for RIFE motion smoothing.\")\n",
" %cd /content/\n",
" \n",
" def install_3dinpainting(): #untested as a function\n",
" !pip3 install torch==1.4.0+cu100 torchvision==0.5.0+cu100 -f https://download.pytorch.org/whl/torch_stable.html\n",
" !pip3 install opencv-python==4.2.0.32\n",
" !pip3 install vispy==0.6.4\n",
" !pip3 install moviepy==1.0.2\n",
" !pip3 install transforms3d==0.3.1\n",
" !pip3 install networkx==2.3\n",
" !sudo apt install sed\n",
" %cd /content/\n",
" !git clone https://github.com/sadnow/3d-photo-inpainting.git\n",
" %cd 3d-photo-inpainting\n",
" !sh download.sh\n",
" !sed -i 's/offscreen_rendering: True/offscreen_rendering: False/g' argument.yml\n",
"\n",
" def run_3dinpainting(): #untested as a function\n",
" !mkdir image\n",
" !mkdir video\n",
" %cd image\n",
" from google.colab import files\n",
" uploaded = files.upload()\n",
" for fn in uploaded.keys():\n",
" print('User uploaded file \"{name}\" with length {length} bytes'.format(\n",
" name=fn, length=len(uploaded[fn])))\n",
" %cd ..\n",
" !python main.py --config argument.yml\n",
"###\n",
"\n",
"\n",
"###############################################################\n",
"\n",
"#\n",
"#If RIFE (P2) stops for no reason, go to Runtime>Change Runtime type and set the notebook to \"high memory\"\n",
"\n",
"#Params\n",
"mount_google_drive = True #@param {type:\"boolean\"}\n",
"\n",
"#Mount Google Drive\n",
"if mount_google_drive:\n",
" from google.colab import drive\n",
" drive.mount('/content/drive')\n",
"\n",
"#install dependencies\n",
"AnKit.install_ESRGAN()\n",
"AnKit.install_RIFE()\n",
"!pip install lpips datetime -q #for overwrite protection\n",
"####################################################"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "MhMORNgduDyt",
"cellView": "form"
},
"source": [
"#@title { form-width: \"300px\" }\n",
"#@markdown # AnimationKit AI\n",
"#@markdown >*This cell combines Real-ESRGAN upscaling, RIFE smoothing/interpolation, and h265 compression.*\n",
"#@markdown ---\n",
"#@markdown ##Input settings\n",
"_import_mp4_file = False #@param {type:\"boolean\"}\n",
"_imported_mp4_path='/content/drive/MyDrive/VQLIPSE/videos/filespace-dinner.mp4' #@param {type:\"string\"}\n",
"#@markdown >If you want to process from static images, uncheck `_import_mp4_file`\n",
"_import_frame_folder='/content/drive/MyDrive/VQLIPSE/images_out/rickmorty/' #@param {type:\"string\"}\n",
"#@markdown Note: *`_import_frame_folder` is for importing individual frames from your projects. Will be ignored if `_import_mp4_file` is checked.*\n",
"\n",
"\n",
"#@markdown ---\n",
"#@markdown ##Output settings\n",
"_output_folder = '/content/drive/MyDrive/AI/AnimationKit/videos/' #@param {type:\"string\"}\n",
"_output_filename = 'zelda3_AnKit.mp4' #@param{type:\"string\"}\n",
"_target_fps=60#@param {type:\"integer\"}\n",
"_target_length_seconds=30#@param {type:\"integer\"}\n",
"_keep_audio = False #@param{type:\"boolean\"}\n",
"#@markdown ---\n",
"_upscale_model_name='RealESRGANv2-animevideo-xsx2' #@param ['RealESRGAN_x4plus','RealESRGAN_x4plus_anime_6B','RealESRGAN_x2plus','RealESRGANv2-animevideo-xsx2','RealESRGANv2-animevideo-xsx4'] {type:\"string\"}\n",
"_upscale_method = 'new_method' #@param ['new_method','old_method']\n",
"_constant_quality=32#@param {type:\"slider\", min:20, max:50, step:1}\n",
"_face_enhance_gfpgan=True #@param{type:\"boolean\"}\n",
"_skip_static_frames = False #deprecated\n",
"# keep_mp4_audio = False #@param {type: \"boolean\"}\n",
"#@markdown Default `_constant_quality` is `30`. Higher values = lower filesize, lower quality\n",
"\n",
"\n",
"#---------------------------------------------------------------#\n",
"#@markdown ---\n",
"#@markdown ##Performance\n",
"#input_path='/content/drive/MyDrive/pytti_test/videos/zelda4.mp4' #@param {type:\"string\"}\n",
"_target_scale_RIFE='1.0'#@param ['0.25','0.5','1.0','2.0']{type:\"string\"}\n",
"_half_precision_realesrgan=False #@param{type:\"boolean\"}\n",
"#length_multiplier=3#@param {type:\"integer\"}\n",
"#@markdown *These options can speed up processing at the costs of quality.*\n",
"#@markdown If you're seeing weird warping in your outputs, try increasing or decreasing `_target_scale_RIFE`.\n",
"#@markdown \n",
"#---------------------------------------------------------------#\n",
"\n",
"\n",
"#compress_path='/content/drive/MyDrive/pytti_test/videos/zelda4_16X_180fps.mp4' #@param {type:\"string\"}\n",
"#outputStr = '_tblend2.mp4' #@param {type:\"string\"}\n",
"enable_tblend = False #deprecated\n",
"#deflicker_on = False #@param {type:\"boolean\"}\n",
"#deflicker_avg_frames=76#@param {type:\"slider\", min:2, max:129, step:1}\n",
"#mpdecimate_on = False #@param {type:\"boolean\"}\n",
"#minterpolate_on = False #@param {type:\"boolean\"}\n",
"#minterpolate_fps=60#@param {type:\"integer\"}\n",
"#tblend_on = False #@param {type:\"boolean\"}\n",
"#tmix_on = True #@param {type:\"boolean\"}\n",
"tblend_framestep_value = \"2\" #deprecated\n",
"##@markdown Set `deflicker_avg_frames` filter size in frames. FFmpeg's default is 5. I haven't tested too much but I would try 100.\n",
"\n",
"#---------------------------------------------------------------#\n",
"#@markdown ---\n",
"#@markdown ##Master Overrides\n",
"#@markdown If you want to skip certain processes, this is the master toggle. All related settings above will be ignored, so no need to change them.\n",
"#skip_cleanup = True #@param{type:\"boolean\"}\n",
"_skip_upscaling = False #@param{type:\"boolean\"}\n",
"_skip_RIFEsmoothing = False #@param{type:\"boolean\"}\n",
"##@markdown `skip_cleanup` will be automatically enabled if you pick either of these options.\n",
"#---------------------------------------------------------------#\n",
"skip_cleanup = False\n",
"if _skip_upscaling:\n",
" skip_cleanup = True\n",
"if _skip_RIFEsmoothing:\n",
" skip_cleanup = True\n",
"\n",
"###################################################################\n",
"#---------------------------------------------------------------#\n",
"#CHAINING METHODS (swapped to user-choice)\n",
"# if _upscale_model_name == 'RealESRGANv2-animevideo-xsx2' or 'RealESRGANv2-animevideo-xsx4': #anime VIDEO models \n",
"# new_method = True\n",
"# old_method = False\n",
"# else: \n",
"# old_method = True\n",
"# new_method = False\n",
"\n",
"if _upscale_method == 'new_method':\n",
" new_method = True\n",
" old_method = False\n",
"else:\n",
" new_method - False\n",
" old_method = True\n",
"#---------------------------------------------------------------#\n",
"# check if the message ends with .mp4\n",
"contains_extension = (_output_filename.endswith('.mp4'))\n",
"if not contains_extension:\n",
" _output_filename = _output_filename + '.mp4'\n",
"output_fullpathname = _output_folder + _output_filename\n",
"\n",
"contains_slash = (_output_folder.endswith('/'))\n",
"if not contains_slash:\n",
" _output_folder = _output_folder + '/'\n",
"#---------------------------------------------------------------#\n",
"\n",
"if enable_tblend: #DEPRECATED\n",
" visual_effects = '-vf tblend=all_mode=average,framestep=' + tblend_framestep_value\n",
"else:\n",
" visual_effects = ''\n",
"\n",
"\n",
"#---------------------------------------------------------------#\n",
"\n",
"added_params = \"\"\n",
"if _half_precision_realesrgan:\n",
" added_params = added_params + \" --half\"\n",
"if _face_enhance_gfpgan: \n",
" added_params = added_params + \" --face_enhance\"\n",
"\n",
"if _skip_static_frames: #DEPCRECATED\n",
" _target_scale_RIFE = _target_scale_RIFE + ' --skip'\n",
"\n",
"%cd /content/\n",
"#cleanup\n",
"if not skip_cleanup:\n",
" print(\"\\n Cleaning up from last run...\\n\")\n",
" !rm -rf \"/content/Real-ESRGAN/results\"\n",
" !rm -rf '/content/frames_storage/init_frame_storage'\n",
" !rm -rf '/content/frames_storage/upscaled_frame_storage'\n",
"\n",
"###############################################################\n",
"#P1 REAL-ESRGAN UPSCALING\n",
"\n",
"print(\"\\n ------------------------------------\\n\")\n",
"print(\"\\n Beginning Real-ESRGAN Upscaling phase...\\n\")\n",
"\n",
"from pathlib import Path\n",
"Dir_make('/content/frames_storage/init_frame_storage')\n",
"if _import_mp4_file:\n",
" if old_method:\n",
" #---splitframes------------------------------------------------\n",
" print(\"\\n Starting splitFrames... \\n\")\n",
" File_check(_imported_mp4_path)\n",
" !ffmpeg -y -r 1 -i $_imported_mp4_path -r 1 '/content/frames_storage/init_frame_storage'/frame%05d.png\n",
" print(\"\\n Completed splitFrames... \\n\")\n",
" #---end------------------------------------------------\n",
" if new_method:\n",
" print(\"Skipping splitframes (new_method)... \\n\")\n",
"\n",
"\n",
"if new_method and _import_mp4_file: print(\"Skipping sortframes... \\n\")\n",
"else:\n",
" print(\"\\n Beginning sortFrames... \")\n",
" #if not _import_mp4_file:\n",
" Dir_check(_import_frame_folder)\n",
" %cd $_import_frame_folder\n",
" #---sortframes------------------------------------------------\n",
" print(\"\\n Copying frames to \"+'/content/frames_storage/init_frame_storage'+\" for processing...\\n\")\n",
" !find -maxdepth 1 -name '*.png' -print0 | xargs -0 cp -t '/content/frames_storage/init_frame_storage'\n",
" %cd '/content/frames_storage/init_frame_storage'\n",
" !find . -type f -name \"*.png\" -execdir bash -c 'mv \"$0\" \"${0##*_}\"' {} \\; #removes anything not numbered (junk files) from new dir\n",
" !rename 's/\\d+/sprintf(\"%05d\",$&)/e' * #adds padding to numbers\n",
" print(\"\\n Finished copying frames to \"+'/content/frames_storage/init_frame_storage'+\".\\n\")\n",
" print(\"Completed sortFrames... \\n\")\n",
" #---end-------------------------------------------------\n",
"\n",
"%cd /content/\n",
"\n",
"#---runUpscale------------------------------------------------\n",
"if _skip_upscaling:\n",
" print(\"NOTICE: _skip_upscaling is checked. Skipping Real-ESRGAN upscaling...\\n\")\n",
"else:\n",
" Dir_check('/content/frames_storage/init_frame_storage')\n",
" Dir_make('/content/frames_storage/upscaled_frame_storage')\n",
" %cd '/content/Real-ESRGAN/'\n",
" if new_method:\n",
" #print(\"\\n WARNING: RIFE is temporarily disabled with the anime-video modles. \\n\")\n",
" #_skip_RIFEsmoothing = True #temporary testing fix\n",
" !pip install ffmpeg-python\n",
" !wget https://github.com/xinntao/Real-ESRGAN/releases/download/v0.2.3.0/RealESRGANv2-animevideo-xsx2.pth -P experiments/pretrained_models\n",
" if _keep_audio and _import_mp4_file:\n",
" added_params = added_params + ' --audio'\n",
" ###############################################################\n",
" # method b new video inference\n",
" if _import_mp4_file: inference_filepath = _imported_mp4_path\n",
" else: \n",
" #inference_filepath = _import_frame_folder\n",
" run_frames2video = True\n",
" #--------------------------------------\n",
" #--------------------------------------\n",
" #getting the filename of upscaled_out.mp4\n",
" #if _import_mp4_file:\n",
" if run_frames2video: #frames2video\n",
" %cd '/content/frames_storage/init_frame_storage'\n",
" #!ffmpeg -framerate $_target_fps -pattern_type glob -i '*.png' -y '/content/frames2video.mp4' #old method\n",
" print(\"RUNNING THE EXPERIMENTAL\")\n",
" #!ffmpeg -r 23.98 -pattern_type glob -i '*.png' -c:v libx264 -r 23.98 -pix_fmt yuv420p '/content/frames_storage/init_frame_storage/frames2video.mp4' #documented example realsrgan-video.md\n",
" !ffmpeg -r 24 -pattern_type glob -i '*.png' -c:v libx264 -r 23.98 -pix_fmt yuv420p '/content/frames_storage/init_frame_storage/frames2video.mp4' #documented example realsrgan-video.md\n",
" #todo: add source_fps option\n",
" _imported_mp4_path = '/content/frames_storage/init_frame_storage/frames2video.mp4' #bad practic\n",
" %cd '/content/Real-ESRGAN'\n",
" !python inference_realesrgan_video.py --input $_imported_mp4_path --output '/content/frames_storage/upscaled_frame_storage' --model_name $_upscale_model_name --suffix upscaled --video$added_params\n",
" #---this is for recognizing the outputted files' name\n",
" upscaled_mp4_path = (os.path.splitext(_imported_mp4_path)[0]) #remove mp4 extension\n",
" upscaled_mp4_path = upscaled_mp4_path + '_upscaled.mp4'\n",
" upscaled_mp4_path = os.path.basename(os.path.normpath(upscaled_mp4_path)) #isolate filename\n",
" upscaled_mp4_path = '/content/frames_storage/upscaled_frame_storage/' + upscaled_mp4_path\n",
" #--------\n",
"\n",
" #print(upscaled_mp4_path) #tmp\n",
" #--------------------------------------\n",
" #-------------------------------------- \n",
" \n",
" if old_method: #old method\n",
" !python inference_realesrgan.py --model_name $_upscale_model_name --input '/content/frames_storage/init_frame_storage' --output '/content/frames_storage/upscaled_frame_storage'$added_params\n",
" ######################################################################\n",
" #---frames2video---------------------------------------------\n",
" print(\"\\n NOTICE: Running frames2video \\n\")\n",
" Dir_check('/content/frames_storage/upscaled_frame_storage')\n",
" %cd '/content/frames_storage/upscaled_frame_storage'\n",
" print(\"\\n Creating wip_file from frames... \\n\")\n",
" !ffmpeg -framerate $_target_fps -pattern_type glob -i '*.png' -y '/content/frames2video.mp4' \n",
" print(\"\\n NOTICE: Finished running frames2video \\n\")\n",
" #---end------------------------------------------------\n",
"\n",
"\n",
"\n",
"print(\"End of upscaling phase.\\n\")\n",
"#---end------------------------------------------------\n",
"\n",
"#END OF UPSCALING PHASE P1\n",
"#END OF UPSCALING PHASE\n",
"\n",
"###############################################################################\n",
"###############################################################################\n",
"\n",
"#P2 RIFE MOTION SMOOTHING\n",
"#P2 RIFE MOTION SMOOTHING\n",
"gc.collect()\n",
"\n",
"print(\"\\n ------------------------------------\\n\")\n",
"print(\"\\n Beginning RIFE motion smoothing phase... \\n\")\n",
"\n",
"%cd /content/Practical-RIFE/\n",
"\n",
"#----------------------------\n",
"if _import_mp4_file:\n",
" measured_fps = AnKit.detect_fps(_imported_mp4_path)\n",
" print(\"\\n NOTICE: Detected average FPS of \",_imported_mp4_path,\" is \",measured_fps)\n",
" measured_duration = AnKit.detect_duration(_imported_mp4_path)\n",
"else: \n",
" measured_fps = AnKit.detect_fps(upscaled_mp4_path)\n",
" print(\"\\n NOTICE: Detected average FPS of \",upscaled_mp4_path,\" is \",measured_fps)\n",
" measured_duration = AnKit.detect_duration(upscaled_mp4_path)\n",
"\n",
"print(\"\\n NOTICE: Detected duration INTEGER (in seconds) is \",measured_duration)\n",
"\n",
"if measured_duration < 1: #failsafe\n",
" print(\"\\n NOTICE: Your input appears to be less than one second... \\n\")\n",
" measured_duration = 1\n",
" # if new_method:\n",
" # measured_fps = 24\n",
"\n",
"exp_value = AnKit.exp_calc(measured_duration,_target_length_seconds)\n",
"\n",
"\n",
"#print(\"\\n NOTICE: Target duration currently rounds to the closest --exp RIFE can handle. \\n\")\n",
"\n",
"if (exp_value < 0.5):\n",
" _skip_RIFEsmoothing = True\n",
" print(\"\\n NOTICE: Your _target_fps doesn't necessitate RIFE motion smoothing. Skipping RIFE...\\n\")\n",
"\n",
"if _skip_RIFEsmoothing:\n",
" print(\"\\n NOTICE: Skipping RIFE motion smoothing...\\n\")\n",
"else:\n",
" #---RUN RIFE------------------------------------\n",
" print(\"\\n NOTICE: Running RIFE... \\n\")\n",
" %cd /content/Practical-RIFE\n",
" if old_method:\n",
" !python3 /content/Practical-RIFE/inference_video.py --fps=$_target_fps --exp=$exp_value --video='/content/frames2video.mp4' --scale=$_target_scale_RIFE --output='/content/RIFE_frames2video.mp4'\n",
" if new_method: \n",
" !python3 /content/Practical-RIFE/inference_video.py --fps=$_target_fps --exp=$exp_value --video=$upscaled_mp4_path --scale=$_target_scale_RIFE --output='/content/RIFE_frames2video.mp4'\n",
"\n",
" #---END--------------------------------------\n",
"#!ffmpeg -y -i $input $visual_effects -c:v hevc_nvenc -rc vbr -cq $_constant_quality -qmin $_constant_quality -qmax $_constant_quality -b:v 0 $output_fullpathname\n",
"#END OF MOTION SMOOTHING PHASE\n",
"print(\"\\n End of RIFE interpolation phase.\\n\")\n",
"\n",
"#########################################################################\n",
"#P3 h265 compression\n",
"print(\"\\n ------------------------------------\\n\")\n",
"print(\"\\n NOTICE: Beginning h265 compression phase...\\n\")\n",
"%cd /content/\n",
"Dir_make(_output_folder)\n",
"\n",
"#---File overwrite protection---\n",
"overwrite_protection = File_check(output_fullpathname)\n",
"if overwrite_protection: #add datetime to end of file\n",
" from datetime import datetime\n",
" current_time = datetime.now().strftime('%y%m%d-%H%M%S_%f')\n",
" output_fullpathname = output_fullpathname + '_' + current_time + '.mp4'\n",
" print(\"NOTICE: Overwrite protection has renamed your output file as \",output_fullpathname,\" \\n\")\n",
"#---end--------------------\n",
"\n",
"if _skip_RIFEsmoothing: \n",
" !ffmpeg -i '/content/frames2video.mp4' $visual_effects -c:v hevc_nvenc -rc vbr -cq $_constant_quality -qmin $_constant_quality -qmax $_constant_quality -b:v 0 $output_fullpathname\n",
"else:\n",
" !ffmpeg -i '/content/RIFE_frames2video.mp4' $visual_effects -c:v hevc_nvenc -rc vbr -cq $_constant_quality -qmin $_constant_quality -qmax $_constant_quality -b:v 0 $output_fullpathname\n",
"\n",
"print(\"\\n End of h265 compression phase.\\n\")\n",
"file_verified = File_check(output_fullpathname)\n",
"if file_verified:\n",
" print(\"\\n NOTICE: Finished! Your final file is saved as \",output_fullpathname)\n",
"else:\n",
" print (\"\\n ERROR: Cannot verify that your file completed processing! Check /content/ for mp4 files to see what part of the process went wrong.\")\n"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "sg_wojOzdtZu"
},
"source": [
"This Notebook has been designed so you can \"Run All\". [**Ctrl+F9**]"
]
}
]
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment