REDS dataset downloading script

download_REDS.py

# reference: https://stackoverflow.com/questions/25010369/wget-curl-large-file-from-google-drive

import os
import requests
import argparse

def download_file_from_google_drive(id, destination):
    def get_confirm_token(response):
        for key, value in response.cookies.items():
            if key.startswith('download_warning'):
                return value

        return None

    def save_response_content(response, destination):
        CHUNK_SIZE = 32768

        with open(destination, "wb") as f:
            for chunk in response.iter_content(CHUNK_SIZE):
                if chunk: # filter out keep-alive new chunks
                    f.write(chunk)

    URL = "https://docs.google.com/uc?export=download"

    session = requests.Session()

    response = session.get(URL, params = {'id': id}, stream=True)
    token = get_confirm_token(response)

    if token:
        params = {'id' : id, 'confirm' : token}
        response = session.get(URL, params=params, stream=True)

    save_response_content(response, destination)

def download_file_from_server(server, subset, destination):
    def get_confirm_token(response):
        for key, value in response.cookies.items():
            if key.startswith('download_warning'):
                return value

        return None

    def save_response_content(response, destination):
        CHUNK_SIZE = 32768

        with open(destination, "wb") as f:
            for chunk in response.iter_content(CHUNK_SIZE):
                if chunk: # filter out keep-alive new chunks
                    f.write(chunk)

    session = requests.Session()
    if server == 'google':
        URL = "https://docs.google.com/uc?export=download"
        params = {'id': ids[subset]}
    elif server == 'snu':
        URL = 'http://data.cv.snu.ac.kr:8008/webdav/dataset/REDS/' + subset + '.zip'
        params = {}

    response = session.get(URL, params=params, stream=True)
    token = get_confirm_token(response)
    if token:
        params['confirm'] = token
        response = session.get(URL, params=params, stream=True)

    save_response_content(response, destination)

parser = argparse.ArgumentParser(description='Download REDS dataset from google drive to current folder', allow_abbrev=False)

parser.add_argument('--server', type=str, default='snu', choices=('google', 'snu'), help='download server choice.')
parser.add_argument('--all', action='store_true', help='download full REDS dataset')
parser.add_argument('--train_sharp', action='store_true', help='download train_sharp.zip')
parser.add_argument('--train_blur', action='store_true', help='download train_blur.zip')
parser.add_argument('--train_blur_comp', action='store_true', help='download train_blur_comp.zip')
parser.add_argument('--train_sharp_bicubic', action='store_true', help='download train_sharp_bicubic.zip')
parser.add_argument('--train_blur_bicubic', action='store_true', help='download train_blur_bicubic.zip')
parser.add_argument('--val_sharp', action='store_true', help='download val_sharp.zip')
parser.add_argument('--val_blur', action='store_true', help='download val_blur.zip')
parser.add_argument('--val_blur_comp', action='store_true', help='download val_blur_comp.zip')
parser.add_argument('--val_sharp_bicubic', action='store_true', help='download val_sharp_bicubic.zip')
parser.add_argument('--val_blur_bicubic', action='store_true', help='download val_blur_bicubic.zip')
parser.add_argument('--test_blur', action='store_true', help='download test_blur.zip')
parser.add_argument('--test_blur_comp', action='store_true', help='download test_blur_comp.zip')
parser.add_argument('--test_sharp_bicubic', action='store_true', help='download test_sharp_bicubic.zip')
parser.add_argument('--test_blur_bicubic', action='store_true', help='download test_blur_bicubic.zip')

args = parser.parse_args()

ids = { 'train_sharp': '1YLksKtMhd2mWyVSkvhDaDLWSc1qYNCz-',
        'train_blur': '1Be2cgzuuXibcqAuJekDgvHq4MLYkCgR8',
        'train_blur_comp': '1hi6348BB9QQFqVx2PY7pKn32HQM89CJ1',
        'train_sharp_bicubic': '1a4PrjqT-hShvY9IyJm3sPF0ZaXyrCozR',
        'train_blur_bicubic': '10u8gthv2Q95RMCb1LeCN8N4ozB8TVjMt',
        'val_sharp': '1MGeObVQ1-Z29f-myDP7-8c3u0_xECKXq',
        'val_blur': '1N8z2yD0GDWmh6U4d4EADERtcUgDzGrHx',
        'val_blur_comp': '13d1uzqLdbsQzeZkWgdF5QVHqDSjfE4zZ',
        'val_sharp_bicubic': '1sChhtzN9Css10gX7Xsmc2JaC-2Pzco6a',
        'val_blur_bicubic': '1i3NAb7EmF4fCYadGaHK54-Zgx9lIC2Gp',
        'test_blur': '1dr0--ZBKqr4P1M8lek6JKD1Vd6bhhrZT',
        'test_blur_comp': '1OctyKR3ER_YWrZxKxQsZzLis3BvLSOFO',
        'test_sharp_bicubic': '1y0Jle6xB41TdRK_QMJ_E8W_iBMxwq_Rh',
        'test_blur_bicubic': '14YszfzUAeAfwP0ZA2FRzAiVxxZLg7-tY',
        }

# Download files in REDS directory
if os.path.basename(os.getcwd()) == 'REDS':
    root_dir = '.'
else:
    os.makedirs('REDS', exist_ok=True)
    root_dir = 'REDS'

for subset in ids:
    argdict = args.__dict__
    if args.all or argdict[subset]:
        filename = '{}/{}.zip'.format(root_dir, subset)
        servername = 'Google Drive' if args.server == 'google' else 'SNU CVLab'
        print('Downloading {}.zip from {}'.format(subset, servername))
        download_file_from_server(args.server, subset, filename)    # download the designated subset

Hi Seungjun, why is there no data for test_sharp?

Hi @zenjieli,
We have been hosting REDS-related challenges from NTIRE 2019. As we plan to continue the challenges, the ground-truth for the test set should be kept undisclosed. You can use the validation set to evaluate & compare the algorithms. In that sense, I'd recommend using the training set but not the validation set to train models.

I see. Thanks for the explanation!

Hi Seungjun, I have a cctv video which is very blur and grayscale video so, I want clean it to see clearly the running car in the video. Can you please help me.? please give your email, I want to personaly contact to you and if you have any issue to give email please contact my email. m.saqibuddin@yahoo.com . thanks.

@SeungjunNah Thank you for your great contributions. The methods of generating samples are impressive. Could you please release the code to generate the samples ?

Hi @thesby,
As we are preparing extensions, we cannot release the source code.
Nevertheless, the method itself is simple.
You can use open-source frame interpolation methods to create intermediate frames and average them.
ex) sepconv-slomo

Hi Seungjun, could you please offer the scipt to download the REDS 120fps dataset? Or are there any faster method to download it?

Hi @linjing-ai,
The downloading speed should not be related to downloading methods.
You can download by clicking the links or via wget method.
ex) wget http://data.cv.snu.ac.kr:8008/webdav/dataset/REDS/orig/train_orig_part0.zip

Otherwise, you can create your own script by modifying the above code.
ex)

parser.add_argument('--train_orig_part0', action='store_true', help='train_orig_part0.zip')

...

ids['train_orig_part0'] = '1SbbGH3V5PbqfezDaWMMK4eeDCFUD54HS'
# The google drive file id can be found from the corresponding public download link:
# https://drive.google.com/file/d/1SbbGH3V5PbqfezDaWMMK4eeDCFUD54HS/view?usp=sharing

Hi Seungjun, thank you very much, I will try them.

hi @SeungjunNah ,

I am engaged in the joint deblurring and super resolution job now and i wonder if i could send you my result on REDS test dataset to get a performance result

Best

Jinshuang

i can not extract the zip file. I get the following error:
Archive: test_blur_bicubic.zip End-of-central-directory signature not found. Either this file is not a zipfile, or it constitutes one disk of a multi-part archive. In the latter case the central directory and zipfile comment will be found on the last disk(s) of this archive. unzip: cannot find zipfile directory in one of test_blur_bicubic.zip or test_blur_bicubic.zip.zip, and cannot find test_blur_bicubic.zip.ZIP, period

@gulzainali98
I checked the files in SNU_CVLab and google drive but I didn't have any problems extracting them.
Your download process might have been interrupted and led to a crash in the downloaded file.
It could happen due to unstable connections / when google drive has reached the traffic limit.
Please try downloading again later.
If retrying does not solve the issue, let me know by email.

Generally, I recommend sending an email to me rather than commenting here for data/file-related issues.

Hi SeungjunNah
I have two questions about the REDS dataset, could u please answer me, thanks
(1) I read your article about how to create REDS dataset. firstly, you get several sequent frames, the resolution is 1920x1080. Then you downsample it to 720x1280. Then you use AI frame interpolate to get middle frame about two sequent frames, for example, you get 1.5 about frame1 and frame2. Then, get 1.25 about frame1 and frame1.5 and so on. Finally you get several frames 1.1 1.2 1.3 1.4 1.5 1.6 1.7 1.8 1.9 for example, and average these so as to get blur image. And select milddle frame 1.5 as sharp image. Is it right?

(2) I use the same AI frame interpolate network as u methoned in article. But the quality of interpolate is bad. For instance, if the two frames diff large(camera shake serious or motion move fast), the interpolation quality is bad, some things become pretty blur. So how u sove this question, because i find your dataset, the blur image quality is well. So you drop the bad quality interpolation image? IS it right?

Thanks a lot, appreciate for you

@askies
Please note that the original videos are taken at 120fps.
While the sharp images are subsampled at 24 fps, the blurry image is created from the interpolated frames raised from 120fps to 1920 fps.
I assume you got suboptimal frame interpolation results because the interpolation was done from 24 fps frames.
You may want to check out the slides and other information on the dataset website.

@SeungjunNah
ok, so you mean that the original video is 120 fps, and then interpolated to 1920 fps, then average them to 24fps. Which means you average 80 images to one blur-sharp pair because 1920/24 = 80. Is it right? And why you simulate 24 fps, what is the reason. Thank you very much

@askies
65 frames including the original and the interpolated are averaged to generate a single blurry image.
This issue explains the details.

By the way, if not related to the download- or traffic-related issues, please do not comment here as it will alert everyone who has ever commented here.
For general questions, I would prefer to receive emails.

Is there a separation between the input blurry images and the ground-truth images? I downloaded train_blur.zip and this is all I see:

@SeungjunNah

@sayakpaul
You can refer to https://seungjunnah.github.io/Datasets/reds
Please try not to leave general questions here other than the code itself. I don't want to alarm everybody here in the thread.

Hello,

It seems direct downloading from google server results to a webpage instead of the file itself, which warns you that the file is so large that google cannot scan it for virus. If I download manually from the browser, a further confirmation is required to download the zip file itself. But this script does not further handle this.

As shown below, this script actually downloads this webpage.

Google Drive can't scan this file for viruses.

train_sharp.zip(32G) is too large for Google to scan for viruses. Would you still like to download this file?

@wangruohui The file size warning message is normal and you are good to proceed.
The download script failure is less likely to be related to that. Download script may fail due to the traffic limit set by Google Drive when there are many download requests. Please try again later and if you have further issues, please send me an email.

Hi @SeungjunNah is it possible to release test_sharp.zip, now that the challenge has been over for 5 years?