infojsonredact.py

import json
import sys

if len(sys.argv) < 2:
    print("infojsonredact - A simple script to redact private information from ytdl info.json files")
    print("Output will be saved in info.json.redacted files")
    print("Usage: infojsonredact.py file1.info.json [file2.info.json, file3.info.json...]")
    sys.exit(2)

redacted = ["url","manifest_url","fragment_base_url","fragments","http_headers","User-Agent","Accept-Charset","Accept","Accept-Encoding","Accept-Language","player_url","playlist","playlist_id","playlist_title","playlist_uploader","playlist_uploader_id","playlist_index","thumbnail","_filename","downloader_options","http_chunk_size","initialization_url","annotations", "playlist_count","version","_version","repository","release_git_head","filesize_approx","_format_sort_fields"]
allowed = ["id","uploader","uploader_id","uploader_url","channel_id","channel_url","upload_date","license","creator","title","alt_title","thumbnails","width","height","resolution","description","categories","tags","subtitles","automatic_captions","duration","age_limit","chapters","webpage_url","view_count","like_count","dislike_count","average_rating","formats","ext","format_note","acodec","abr","container","format_id","tbr","asr","fps","language","filesize","vcodec","path","protocol","format","is_live","start_time","end_time","series","season_number","episode_number","track","artist","album","release_date","release_year","extractor","webpage_url_basename","extractor_key","n_entries","display_id","vbr","stretched_ratio","fulltitle","quality","ar","bs","bg","ca","zh","zh-TW","hr","cs","da","nl","en","fi","fr","de","el","iw","hi","hu","id","it","ja","ko","no","pl","pt","pt-BR","pt-PT","ro","ru","sr-Cyrl","sr-Latn","sk","es","sv","th","tr","vi","subscriber_count","live_chat","video_id","en-US","en-UK","en-GB","fr-FR","de-DE","hi-Latn","es-MX","es-419","es-US","zh-CN","preference","segment_urls","af","sq","am","ar","hy","az","bn","eu","be","bs","bg","my","ca","ceb","zh-Hans","zh-Hant","co","hr","cs","da","nl","en","eo","et","fil","fi","fr","gl","ka","de","el","gu","ht","ha","haw","iw","hi","hmn","hu","is","ig","id","ga","it","ja","jv","kn","kk","km","rw","ko","ku","ky","lo","la","lv","lt","lb","mk","mg","ms","ml","mt","mi","mr","mn","ne","no","ny","or","ps","fa","pl","pt","pa","ro","ru","sm","gd","sr","sn","sd","si","sk","sl","so","st","es","su","sw","sv","tg","ta","tt","te","th","tr","tk","uk","ur","ug","uz","vi","cy","fy","xh","yi","yo","zu","rows","columns","audio_ext","video_ext","source_preference","audio_channels","playable_in_embed","om","qu","ts","_type","was_live","webpage_url_domain","ti","sa","nso","name","ln","lg","kri","gn","en-orig","dynamic_range","dv","channel","channel_follower_count","comment_count","duration_string","bho","ay","as","aspect_ratio","ak","epoch","ee","availability","live_status","has_drm","language_preference","und","channel_is_verified"]



def redactRecursively(data, allowed, redacted):
    if isinstance(data, (dict, list)):
        for k, v in (data.items() if isinstance(data, dict) else enumerate(data)):
            if k in redacted:
                data[k] = "[REDACTED]"
                continue
            elif not k in allowed:
                if not isinstance(k, int):
                    raise KeyError("Key", k, "not found in both the redacted and allowed keysets")
            redactRecursively(v, allowed, redacted)

for filename in sys.argv[1:]:
    print(filename)
    with open(filename,"r",encoding="utf-8") as f:
        infojson = json.load(f)
    redactRecursively(infojson, allowed, redacted)
    with open(filename + ".redacted","w",encoding="utf-8") as f:
       json.dump(infojson,f)
    

Thank you so much for making this!

Here's some additional tags I found:

• manifest_stream_number (not fully sure, but it just contained the value "0", I set it to redacted to be safe but the rest below I set as allowed)
• modified_date (seems to be when the video or description was changed?)
• "af-orig", "sq-orig", "am-orig", "ar-orig", "hy-orig", "az-orig", "bn-orig", "eu-orig", "be-orig", "bs-orig", "bg-orig", "my-orig", "ca-orig", "ceb-orig", "zh-Hans-orig", "zh-Hant-orig", "co-orig", "hr-orig", "cs-orig", "da-orig", "nl-orig", "en-orig", "eo-orig", "et-orig", "fil-orig", "fi-orig", "fr-orig", "gl-orig", "ka-orig", "de-orig", "el-orig", "gu-orig", "ht-orig", "ha-orig", "haw-orig", "iw-orig", "hi-orig", "hmn-orig", "hu-orig", "is-orig", "ig-orig", "id-orig", "ga-orig", "it-orig", "ja-orig", "jv-orig", "kn-orig", "kk-orig", "km-orig", "rw-orig", "ko-orig", "ku-orig", "ky-orig", "lo-orig", "la-orig", "lv-orig", "lt-orig", "lb-orig", "mk-orig", "mg-orig", "ms-orig", "ml-orig", "mt-orig", "mi-orig", "mr-orig", "mn-orig", "ne-orig", "no-orig", "ny-orig", "or-orig", "ps-orig", "fa-orig", "pl-orig", "pt-orig", "pa-orig", "ro-orig", "ru-orig", "sm-orig", "gd-orig", "sr-orig", "sn-orig", "sd-orig", "si-orig", "sk-orig", "sl-orig", "so-orig", "st-orig", "es-orig", "su-orig", "sw-orig", "sv-orig", "tg-orig", "ta-orig", "tt-orig", "te-orig", "th-orig", "tr-orig", "tk-orig", "uk-orig", "ur-orig", "ug-orig", "uz-orig", "vi-orig", "cy-orig", "fy-orig", "xh-orig", "yi-orig", "yo-orig", "zu-orig" (same as en-orig but for other languages)
• heatmap (I think it's the video popularity thing above the play bar, but im not fully sure)
• value (used within the heatmap object)
• location (specified by the uploader)

Yep there are new keys all the time, I eventually want to refactor the code a little so I don't have to specify literally every language each time a new one gets added

modified_date (seems to be when the video or description was changed?)
"af-orig", "sq-orig", "am-orig", "ar-orig", "hy-orig", "az-orig", "bn-orig", "eu-orig", "be-orig", "bs-orig", "bg-orig", "my-orig", "ca-orig", "ceb-orig", "zh-Hans-orig", "zh-Hant-orig", "co-orig", "hr-orig", "cs-orig", "da-orig", "nl-orig", "en-orig", "eo-orig", "et-orig", "fil-orig", "fi-orig", "fr-orig", "gl-orig", "ka-orig", "de-orig", "el-orig", "gu-orig", "ht-orig", "ha-orig", "haw-orig", "iw-orig", "hi-orig", "hmn-orig", "hu-orig", "is-orig", "ig-orig", "id-orig", "ga-orig", "it-orig", "ja-orig", "jv-orig", "kn-orig", "kk-orig", "km-orig", "rw-orig", "ko-orig", "ku-orig", "ky-orig", "lo-orig", "la-orig", "lv-orig", "lt-orig", "lb-orig", "mk-orig", "mg-orig", "ms-orig", "ml-orig", "mt-orig", "mi-orig", "mr-orig", "mn-orig", "ne-orig", "no-orig", "ny-orig", "or-orig", "ps-orig", "fa-orig", "pl-orig", "pt-orig", "pa-orig", "ro-orig", "ru-orig", "sm-orig", "gd-orig", "sr-orig", "sn-orig", "sd-orig", "si-orig", "sk-orig", "sl-orig", "so-orig", "st-orig", "es-orig", "su-orig", "sw-orig", "sv-orig", "tg-orig", "ta-orig", "tt-orig", "te-orig", "th-orig", "tr-orig", "tk-orig", "uk-orig", "ur-orig", "ug-orig", "uz-orig", "vi-orig", "cy-orig", "fy-orig", "xh-orig", "yi-orig", "yo-orig", "zu-orig" (same as en-orig but for other languages)
heatmap (I think it's the video popularity thing above the play bar, but im not fully sure)
value (used within the heatmap object)
location (specified by the uploader)

@FrostBird347 Why should these random keys be excluded? From your own descriptions on the side, they all seem like public information.

They should be included, they just listed the tags but not whether to exclude them

Oh right, I hadn't actually tested the script . The script purposefully exits if it encounters a tag neither in the "allowed" or "redacted" list. In my view it would suffice to just redact anything not on an allowed list. The script is sensitive to new/unknown tags, though I can see how that is useful too.

The goal is to redact sensitive tags while otherwise also leaving everything in place. Overreaching removal is bad for archival, we don't want to remove public information that might be useful to someone.

The goal is to redact sensitive tags while otherwise also leaving everything in place.

So, you only need one list for that, is what I'm saying. A programmer would just exclude k in redacted, looking to write less code as most do, but your program specifies two lists, so your goal must have unstated demands. What it is, is what I'm wondering.
I'm guessing that being so loud about unknown tags is good to keep on top of new, potentially sensitive additions to the info.json format, is it that?

No, you need two to achieve the goals:

1. Redacted list to make sure all keys that need to be redacted ALWAYS get redacted
2. Allowed list to make sure all keys that don't need to be redacted NEVER get redacted

If you get a new unknown tag, it must be compared to both lists, otherwise unknown tags might default to (either) wrong category

Strictly speaking, to those two goals you still don't need to compare against two lists. We can still have control where tags go whilst still only masking with one list, as long as we're confident what's outside the mask is the contents of the other list. The reason we can't do this however, is the uncertainty in the changing info.json format.

Which leads me a different question, this script was started in 2020, does it know about sensitive tags in use before 2020? Assuming the aim is to be able to redact private information from any info.json, that also includes info.json files created at any time. Can we know some youtube dl version in 2011 or at other time didn't write a tag that should be redacted?

Only when we've done an entire history of info.json files can we handle all info.json files.

• We could look in the git history for the part of the code that writes info.json files, which would be a long, long reading of the same piece of code with slight changes over time.
• We could also try to spin up old versions of the program to get it to emit info.json files that we can programmatically search for any strange key names. However, emitting those info.json files might not be straightforward as old versions of ytdl can't download videos from the services online today. We'll have trouble making any downloads, as those versions were made for APIs that no longer exist. Old versions should still work for direct downloads, but do/did info.json files for direct downloads emit the same keys that downloading a youtube video did? Perhaps old youtube pages on the wayback machine (which often saved the video as well) emulate the site to the degree old versions of youtube dl could download them.

as long as we're confident what's outside the mask is the contents of the other list

Yeah this is why we can't do that.

does it know about sensitive tags in use before 2020

I think I worked on files from 2017 when I first wrote this, but it's always possible there's something that slipped through the cracks at some point.

What you're proposing is pretty involved and would take a lot of effort to do. You can feel free to do that if you'd like, but this script here is just supposed to be quick and simple. I don't think it's worth it to go through all of that effort.

I think I worked on files from 2017 when I first wrote this, but it's always possible there's something that slipped through the cracks at some point.

Good to know.

What you're proposing is pretty involved and would take a lot of effort to do. You can feel free to do that if you'd like, but this script here is just supposed to be quick and simple. I don't think it's worth it to go through all of that effort.

It's no project of mine. I just wanted to make explicit what it would take for the script to be "complete". It's no life or death situation, even though the way I wrote that might've made it sound like that

Personally, the only change I would make (and did myself while running this tool) with regards to what's done when an unknown key is found is to print the contents of the unknown value before throwing the error, this made determining which category the key should be in a little easier.

I plan on adding that, as well as something to deal with language keys so they don't all need to be in the list one-by-one.