infojsonredact.py

import json
import sys

if len(sys.argv) < 2:
    print("infojsonredact - A simple script to redact private information from ytdl info.json files")
    print("Output will be saved in info.json.redacted files")
    print("Usage: infojsonredact.py file1.info.json [file2.info.json, file3.info.json...]")
    sys.exit(2)

redacted = ["url","manifest_url","fragment_base_url","fragments","http_headers","User-Agent","Accept-Charset","Accept","Accept-Encoding","Accept-Language","player_url","playlist","playlist_id","playlist_title","playlist_uploader","playlist_uploader_id","playlist_index","thumbnail","_filename","downloader_options","http_chunk_size","initialization_url","annotations", "playlist_count","version","_version","repository","release_git_head","filesize_approx","_format_sort_fields"]
allowed = ["id","uploader","uploader_id","uploader_url","channel_id","channel_url","upload_date","license","creator","title","alt_title","thumbnails","width","height","resolution","description","categories","tags","subtitles","automatic_captions","duration","age_limit","chapters","webpage_url","view_count","like_count","dislike_count","average_rating","formats","ext","format_note","acodec","abr","container","format_id","tbr","asr","fps","language","filesize","vcodec","path","protocol","format","is_live","start_time","end_time","series","season_number","episode_number","track","artist","album","release_date","release_year","extractor","webpage_url_basename","extractor_key","n_entries","display_id","vbr","stretched_ratio","fulltitle","quality","ar","bs","bg","ca","zh","zh-TW","hr","cs","da","nl","en","fi","fr","de","el","iw","hi","hu","id","it","ja","ko","no","pl","pt","pt-BR","pt-PT","ro","ru","sr-Cyrl","sr-Latn","sk","es","sv","th","tr","vi","subscriber_count","live_chat","video_id","en-US","en-UK","en-GB","fr-FR","de-DE","hi-Latn","es-MX","es-419","es-US","zh-CN","preference","segment_urls","af","sq","am","ar","hy","az","bn","eu","be","bs","bg","my","ca","ceb","zh-Hans","zh-Hant","co","hr","cs","da","nl","en","eo","et","fil","fi","fr","gl","ka","de","el","gu","ht","ha","haw","iw","hi","hmn","hu","is","ig","id","ga","it","ja","jv","kn","kk","km","rw","ko","ku","ky","lo","la","lv","lt","lb","mk","mg","ms","ml","mt","mi","mr","mn","ne","no","ny","or","ps","fa","pl","pt","pa","ro","ru","sm","gd","sr","sn","sd","si","sk","sl","so","st","es","su","sw","sv","tg","ta","tt","te","th","tr","tk","uk","ur","ug","uz","vi","cy","fy","xh","yi","yo","zu","rows","columns","audio_ext","video_ext","source_preference","audio_channels","playable_in_embed","om","qu","ts","_type","was_live","webpage_url_domain","ti","sa","nso","name","ln","lg","kri","gn","en-orig","dynamic_range","dv","channel","channel_follower_count","comment_count","duration_string","bho","ay","as","aspect_ratio","ak","epoch","ee","availability","live_status","has_drm","language_preference","und","channel_is_verified"]



def redactRecursively(data, allowed, redacted):
    if isinstance(data, (dict, list)):
        for k, v in (data.items() if isinstance(data, dict) else enumerate(data)):
            if k in redacted:
                data[k] = "[REDACTED]"
                continue
            elif not k in allowed:
                if not isinstance(k, int):
                    raise KeyError("Key", k, "not found in both the redacted and allowed keysets")
            redactRecursively(v, allowed, redacted)

for filename in sys.argv[1:]:
    print(filename)
    with open(filename,"r",encoding="utf-8") as f:
        infojson = json.load(f)
    redactRecursively(infojson, allowed, redacted)
    with open(filename + ".redacted","w",encoding="utf-8") as f:
       json.dump(infojson,f)
    

Strictly speaking, to those two goals you still don't need to compare against two lists. We can still have control where tags go whilst still only masking with one list, as long as we're confident what's outside the mask is the contents of the other list. The reason we can't do this however, is the uncertainty in the changing info.json format.

Which leads me a different question, this script was started in 2020, does it know about sensitive tags in use before 2020? Assuming the aim is to be able to redact private information from any info.json, that also includes info.json files created at any time. Can we know some youtube dl version in 2011 or at other time didn't write a tag that should be redacted?

Only when we've done an entire history of info.json files can we handle all info.json files.

• We could look in the git history for the part of the code that writes info.json files, which would be a long, long reading of the same piece of code with slight changes over time.
• We could also try to spin up old versions of the program to get it to emit info.json files that we can programmatically search for any strange key names. However, emitting those info.json files might not be straightforward as old versions of ytdl can't download videos from the services online today. We'll have trouble making any downloads, as those versions were made for APIs that no longer exist. Old versions should still work for direct downloads, but do/did info.json files for direct downloads emit the same keys that downloading a youtube video did? Perhaps old youtube pages on the wayback machine (which often saved the video as well) emulate the site to the degree old versions of youtube dl could download them.

as long as we're confident what's outside the mask is the contents of the other list

Yeah this is why we can't do that.

does it know about sensitive tags in use before 2020

I think I worked on files from 2017 when I first wrote this, but it's always possible there's something that slipped through the cracks at some point.

What you're proposing is pretty involved and would take a lot of effort to do. You can feel free to do that if you'd like, but this script here is just supposed to be quick and simple. I don't think it's worth it to go through all of that effort.

I think I worked on files from 2017 when I first wrote this, but it's always possible there's something that slipped through the cracks at some point.

Good to know.

What you're proposing is pretty involved and would take a lot of effort to do. You can feel free to do that if you'd like, but this script here is just supposed to be quick and simple. I don't think it's worth it to go through all of that effort.

It's no project of mine. I just wanted to make explicit what it would take for the script to be "complete". It's no life or death situation, even though the way I wrote that might've made it sound like that

Personally, the only change I would make (and did myself while running this tool) with regards to what's done when an unknown key is found is to print the contents of the unknown value before throwing the error, this made determining which category the key should be in a little easier.

I plan on adding that, as well as something to deal with language keys so they don't all need to be in the list one-by-one.