Skip to content

Instantly share code, notes, and snippets.

@hby

hby/explore.clj

Last active November 1, 2015 14:12
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save hby/374aeabc076ee7780a97 to your computer and use it in GitHub Desktop.
Save hby/374aeabc076ee7780a97 to your computer and use it in GitHub Desktop.
Download ZIP
For safely exploring potentially large structures in the repl
Raw
explore.clj
(declare explore-map)
(declare explore-vector)
(declare explore-seq)
(defn explore
"Safely explore possibly large map/vector structures in the repl.
morv - map or vector or seq
threshold - if count is under threshold, recursively explore
peekn - if over threshold, recursivley explore peekn entries
and summarize rest"
[morv threshold peekn]
(cond
(map? morv) (explore-map morv threshold peekn)
(vector? morv) (explore-vector morv threshold peekn)
(seq? morv) (explore-vector (into [] morv) threshold peekn)
:else (str morv)))
(declare big-map)
(defn explore-map
[m t p]
(if (< (count m) t)
(into {} (map (fn [[key val]] [key (explore val t p)]) m))
(big-map m t p)))
(defn big-map
[m t p]
(let [peeked (take p m)
rest (drop p m)
more (str (count rest) " entries")]
(assoc (into {}
(map (fn [[key val]] [key (explore val t p)]) peeked))
:more more)))
(declare big-vector)
(defn explore-vector
[v t p]
(if (< (count v) t)
(into [] (map (fn [e] (explore e t p)) v))
(big-vector v t p)))
(defn big-vector
[v t p]
(let [peeked (take p v)
rest (drop p v)
more (str (count rest) " entries")]
(conj (mapv (fn [e] (explore e t p)) peeked)
more)))
@hby
Copy link
Author

hby commented Oct 31, 2015

For example, I'm exploring my iTunes library.

(ns itunesquery.parse
  (:require [com.github.bdesham.clj-plist :as pl]
                [clojure.pprint :as pp])
  (:import [java.io File]))

;; It's kinda big
(def itl
  (pl/parse-plist
    (File. "path-to-itunes-lib/iTunes Music Library.xml")))

(pp/pprint (explore itl 20 2))

gives

{"Major Version" "1",
 "Show Content Ratings" "true",
 "Playlists"
 [{"Playlist Persistent ID" "7DFD01EA07FCC290",
   "All Items" "true",
   "Visible" "false",
   "Master" "true",
   "Playlist Items"
   [{"Track ID" "6085"} {"Track ID" "6083"} "20310 entries"],
   "Playlist ID" "82971",
   "Name" "####!####"}
  {"Playlist Persistent ID" "73F3C095D7F3E012",
   "Music" "true",
   "All Items" "true",
   "Smart Info" "[B@2ffe2aec",
   "Distinguished Kind" "4",
   "Smart Criteria" "[B@4603236e",
   "Playlist Items" [{"Track ID" "8083"} {"Track ID" "7031"} "15606 entries"],
   "Playlist ID" "103282",
   "Name" "Music"}
  "193 entries"],
 "Music Folder" "...",
 "Minor Version" "1",
 "Date" "2015-10-27T19:44:01.000-05:00",
 "Tracks"
 {:more "23175 entries",
  "30645" {:more "25 entries", "Library Folder Count" "1", "Disc Number" "1"},
  "43515" {:more "25 entries", "Library Folder Count" "1", "Total Time" "3539200"}},
 "Application Version" "12.3.1.23",
 "Library Persistent ID" "88ABD0BA83F503C5",
 "Features" "5"}

Hmm, the tracks map look like it only has 25 more keys.

(pp/pprint (explore itl 30 2))
{"Major Version" "1",
 "Show Content Ratings" "true",
 "Playlists"
 [{"Playlist Persistent ID" "7DFD01EA07FCC290",
   "All Items" "true",
   "Visible" "false",
   "Master" "true",
   "Playlist Items"
   [{"Track ID" "6085"} {"Track ID" "6083"} "20310 entries"],
   "Playlist ID" "82971",
   "Name" "####!####"}
  {"Playlist Persistent ID" "73F3C095D7F3E012",
   "Music" "true",
   "All Items" "true",
   "Smart Info" "[B@2ffe2aec",
   "Distinguished Kind" "4",
   "Smart Criteria" "[B@4603236e",
   "Playlist Items"
   [{"Track ID" "8083"} {"Track ID" "7031"} "15606 entries"],
   "Playlist ID" "103282",
   "Name" "Music"}
  "193 entries"],
 "Music Folder" "...",
 "Minor Version" "1",
 "Date" "2015-10-27T19:44:01.000-05:00",
 "Tracks"
 {:more "23175 entries",
  "30645"
  {"Library Folder Count" "1",
   "Disc Number" "1",
   "Total Time" "332066",
   "Disc Count" "1",
   "Equalizer" "Rock",
   "Persistent ID" "5B219C7E2973CDCE",
   "Artist" "Joe Louis Walker",
   "Album" "Live At Slim's Volume 1",
   "Play Date" "3428123742",
   "Location" "...",
   "Track ID" "30645",
   "Track Count" "10",
   "Track Number" "3",
   "Date Modified" "2011-07-03T17:03:17.000-05:00",
   "Bit Rate" "675",
   "Size" "28068234",
   "Date Added" "2011-07-03T17:02:51.000-05:00",
   "Play Count" "1",
   "Year" "2008",
   "File Type" "1295270176",
   "Track Type" "File",
   "Name" "Don't Play Games",
   "Sample Rate" "44100",
   "Genre" "Blues",
   "Play Date UTC" "2012-08-18T08:35:42.000-05:00",
   "File Folder Count" "5",
   "Kind" "Apple Lossless audio file"},
  "43515"
  {"Library Folder Count" "1",
   "Total Time" "3539200",
   "Persistent ID" "4208E480BB2E44CC",
   "Comments"
   "Long before being recognized as an outstanding interviewer and talk show host, Dick Cavett started his career as a performer by doing magic.  On this episode of The Spirit of Magic podcast, Mr. Cavett talks about his lifelong passion for magic, his apprec",
   "Artist" "Dodd Vickers",
   "Album" "The Magic Newswire",
   "Play Date" "3475728561",
   "Location" "...",
   "Composer" "Wizard & The Vegas Showgirl",
   "Track ID" "43515",
   "Date Modified" "2013-09-16T09:39:16.000-05:00",
   "Sort Album" "Magic Newswire",
   "Bit Rate" "128",
   "Size" "56898510",
   "Date Added" "2013-09-16T09:39:16.000-05:00",
   "Artwork Count" "1",
   "Release Date" "2010-06-22T11:27:40.000-05:00",
   "Play Count" "1",
   "Year" "2010",
   "Track Type" "File",
   "Name" "MNW #195 :: DICK CAVETT",
   "Sample Rate" "44100",
   "Genre" "Podcast",
   "Play Date UTC" "2014-02-20T08:09:21.000-06:00",
   "File Folder Count" "4",
   "Podcast" "true",
   "Kind" "MPEG audio file"}},
 "Application Version" "12.3.1.23",
 "Library Persistent ID" "88ABD0BA83F503C5",
 "Features" "5"}

Ok, Need to look at a few more tracks.

(pp/pprint (explore itl 30 10))
{"Major Version" "1",
 "Show Content Ratings" "true",
 "Playlists"
 [{"Playlist Persistent ID" "7DFD01EA07FCC290",
   "All Items" "true",
   "Visible" "false",
   "Master" "true",
   "Playlist Items"
   [{"Track ID" "6085"}
    {"Track ID" "6083"}
    {"Track ID" "6091"}
    {"Track ID" "6093"}
    {"Track ID" "6075"}
    {"Track ID" "6089"}
    {"Track ID" "6077"}
    {"Track ID" "6081"}
    {"Track ID" "6087"}
    {"Track ID" "6079"}
    "20302 entries"],
   "Playlist ID" "82971",
   "Name" "####!####"}
  {"Playlist Persistent ID" "73F3C095D7F3E012",
   "Music" "true",
   "All Items" "true",
   "Smart Info" "[B@2ffe2aec",
   "Distinguished Kind" "4",
   "Smart Criteria" "[B@4603236e",
   "Playlist Items"
   [{"Track ID" "8083"}
    {"Track ID" "7031"}
    {"Track ID" "5705"}
    {"Track ID" "5665"}
    {"Track ID" "5667"}
    {"Track ID" "5785"}
    {"Track ID" "5791"}
    {"Track ID" "5787"}
    {"Track ID" "5789"}
    {"Track ID" "6299"}
    "15598 entries"],
   "Playlist ID" "103282",
   "Name" "Music"}
  {"Playlist Persistent ID" "8BFA8F6E5F0B5084",
   "All Items" "true",
   "Smart Info" "[B@7cf4786d",
   "Distinguished Kind" "47",
   "Smart Criteria" "[B@41340bc2",
   "Playlist Items"
   [{"Track ID" "6697"} {"Track ID" "6695"} {"Track ID" "8293"}],
   "Playlist ID" "118893",
   "Name" "Music Videos"}
  {"Playlist Persistent ID" "33DCBC88DFFB8799",
   "All Items" "true",
   "Smart Info" "[B@1f43f042",
   "Distinguished Kind" "7",
   "Smart Criteria" "[B@4084b897",
   "Playlist ID" "118899",
   "Name" "Rentals"}
  {"Playlist Persistent ID" "73F3C095D7F3E013",
   "All Items" "true",
   "Movies" "true",
   "Smart Info" "[B@1f6c87fa",
   "Distinguished Kind" "2",
   "Smart Criteria" "[B@7e67f5ce",
   "Playlist Items" [{"Track ID" "51115"} {"Track ID" "51135"}],
   "Playlist ID" "118902",
   "Name" "Movies"}
  {"Playlist Persistent ID" "09AAD36AF0EC579A",
   "All Items" "true",
   "Smart Info" "[B@7ea6dba0",
   "Distinguished Kind" "48",
   "Smart Criteria" "[B@1b0394e3",
   "Playlist Items"
   [{"Track ID" "42665"}
    {"Track ID" "42667"}
    {"Track ID" "42877"}
    {"Track ID" "42841"}
    {"Track ID" "42867"}
    {"Track ID" "42819"}
    {"Track ID" "42869"}
    {"Track ID" "42837"}
    {"Track ID" "42871"}
    {"Track ID" "42833"}
    "33 entries"],
   "Playlist ID" "118907",
   "Name" "Home Videos"}
  {"Playlist Persistent ID" "73F3C095D7F3E014",
   "All Items" "true",
   "Smart Info" "[B@602b832e",
   "Distinguished Kind" "3",
   "Smart Criteria" "[B@21a3d360",
   "Playlist Items"
   [{"Track ID" "6577"}
    {"Track ID" "11231"}
    {"Track ID" "12605"}
    {"Track ID" "12607"}
    {"Track ID" "12609"}
    {"Track ID" "12611"}
    {"Track ID" "12613"}
    {"Track ID" "12615"}
    {"Track ID" "12617"}
    {"Track ID" "12619"}
    "299 entries"],
   "TV Shows" "true",
   "Playlist ID" "118953",
   "Name" "TV Shows"}
  {"Playlist Persistent ID" "383331EC97F026FB",
   "All Items" "true",
   "Visible" "false",
   "Distinguished Kind" "10",
   "Podcasts" "true",
   "Playlist Items"
   [{"Track ID" "44983"}
    {"Track ID" "44987"}
    {"Track ID" "46543"}
    {"Track ID" "48161"}
    {"Track ID" "45997"}
    {"Track ID" "45995"}
    {"Track ID" "46005"}
    {"Track ID" "46001"}
    {"Track ID" "45999"}
    {"Track ID" "46003"}
    "2192 entries"],
   "Playlist ID" "119265",
   "Name" "Podcasts"}
  {"Playlist Persistent ID" "2F1A0354405F5EB3",
   "All Items" "true",
   "Visible" "false",
   "Distinguished Kind" "31",
   "Playlist Items"
   [{"Track ID" "12381"}
    {"Track ID" "12389"}
    {"Track ID" "12391"}
    {"Track ID" "12387"}
    {"Track ID" "12393"}
    {"Track ID" "12397"}
    {"Track ID" "12395"}
    {"Track ID" "12401"}
    {"Track ID" "12399"}
    {"Track ID" "12403"}
    "2494 entries"],
   "iTunesU" "true",
   "Playlist ID" "121473",
   "Name" "iTunes U"}
  {"Playlist Persistent ID" "73F3C095D7F3E015",
   "All Items" "true",
   "Visible" "false",
   "Audiobooks" "true",
   "Smart Info" "[B@6990672f",
   "Distinguished Kind" "5",
   "Smart Criteria" "[B@6eb35272",
   "Playlist Items"
   [{"Track ID" "5717"}
    {"Track ID" "6175"}
    {"Track ID" "8449"}
    {"Track ID" "8451"}
    {"Track ID" "8453"}
    {"Track ID" "8811"}
    {"Track ID" "8813"}
    {"Track ID" "8815"}
    {"Track ID" "8817"}
    {"Track ID" "8819"}
    "372 entries"],
   "Playlist ID" "123980",
   "Name" "Audiobooks"}
  "185 entries"],
 "Music Folder" "...",
 "Minor Version" "1",
 "Date" "2015-10-27T19:44:01.000-05:00",
 "Tracks"
 {"30645"
  {"Library Folder Count" "1",
   "Disc Number" "1",
   "Total Time" "332066",
   "Disc Count" "1",
   "Equalizer" "Rock",
   "Persistent ID" "5B219C7E2973CDCE",
   "Artist" "Joe Louis Walker",
   "Album" "Live At Slim's Volume 1",
   "Play Date" "3428123742",
   "Location" "...",
   "Track ID" "30645",
   "Track Count" "10",
   "Track Number" "3",
   "Date Modified" "2011-07-03T17:03:17.000-05:00",
   "Bit Rate" "675",
   "Size" "28068234",
   "Date Added" "2011-07-03T17:02:51.000-05:00",
   "Play Count" "1",
   "Year" "2008",
   "File Type" "1295270176",
   "Track Type" "File",
   "Name" "Don't Play Games",
   "Sample Rate" "44100",
   "Genre" "Blues",
   "Play Date UTC" "2012-08-18T08:35:42.000-05:00",
   "File Folder Count" "5",
   "Kind" "Apple Lossless audio file"},
  "43515"
  {"Library Folder Count" "1",
   "Total Time" "3539200",
   "Persistent ID" "4208E480BB2E44CC",
   "Comments"
   "Long before being recognized as an outstanding interviewer and talk show host, Dick Cavett started his career as a performer by doing magic.  On this episode of The Spirit of Magic podcast, Mr. Cavett talks about his lifelong passion for magic, his apprec",
   "Artist" "Dodd Vickers",
   "Album" "The Magic Newswire",
   "Play Date" "3475728561",
   "Location" "...",
   "Composer" "Wizard & The Vegas Showgirl",
   "Track ID" "43515",
   "Date Modified" "2013-09-16T09:39:16.000-05:00",
   "Sort Album" "Magic Newswire",
   "Bit Rate" "128",
   "Size" "56898510",
   "Date Added" "2013-09-16T09:39:16.000-05:00",
   "Artwork Count" "1",
   "Release Date" "2010-06-22T11:27:40.000-05:00",
   "Play Count" "1",
   "Year" "2010",
   "Track Type" "File",
   "Name" "MNW #195 :: DICK CAVETT",
   "Sample Rate" "44100",
   "Genre" "Podcast",
   "Play Date UTC" "2014-02-20T08:09:21.000-06:00",
   "File Folder Count" "4",
   "Podcast" "true",
   "Kind" "MPEG audio file"},
  "9333"
  {"Library Folder Count" "1",
   "Disc Number" "1",
   "Total Time" "307400",
   "Disc Count" "1",
   "Persistent ID" "941D2F3691078CC6",
   "Artist" "Israel Kamakawiwo'ole",
   "Play Date" "3279050530",
   "Location" "...",
   "Track ID" "9333",
   "Track Count" "1",
   "Track Number" "1",
   "Date Modified" "2007-11-27T22:14:19.000-06:00",
   "Bit Rate" "248",
   "Size" "9619241",
   "Date Added" "2007-11-27T22:13:27.000-06:00",
   "Play Count" "1",
   "File Type" "1295270176",
   "Track Type" "File",
   "Name" "Somewhere Over The Rainbow - What a Wonderful World",
   "Sample Rate" "44100",
   "Play Date UTC" "2007-11-27T23:22:10.000-06:00",
   "File Folder Count" "5",
   "Kind" "AAC audio file"},
  "42151"
  {"Library Folder Count" "1",
   "Disc Number" "1",
   "Total Time" "1223333",
   "Disc Count" "1",
   "Equalizer" "Rock",
   "Persistent ID" "4CFE3CB56515881F",
   "Artist" "Sometime Sunday",
   "Album" "Stone",
   "Location" "...",
   "Track ID" "42151",
   "Track Count" "11",
   "Track Number" "11",
   "Date Modified" "2012-09-02T19:39:05.000-05:00",
   "Bit Rate" "695",
   "Size" "106435547",
   "Date Added" "2012-09-02T19:38:14.000-05:00",
   "Year" "1994",
   "File Type" "1295270176",
   "Track Type" "File",
   "Name" "Home",
   "Sample Rate" "44100",
   "Genre" "Gospel",
   "File Folder Count" "5",
   "Kind" "Apple Lossless audio file"},
  "41163"
  {"Total Time" "20000",
   "Persistent ID" "A533AA533DD1FEDF",
   "Artist" "Maxwell House",
   "Album" "Maxwell House",
   "Location" "...",
   "Composer" "Duke",
   "Track ID" "41163",
   "Has Video" "true",
   "Date Added" "2012-07-20T16:44:44.000-05:00",
   "iTunesU" "true",
   "Movie" "true",
   "Release Date" "2010-06-10T09:36:15.000-05:00",
   "Unplayed" "true",
   "Track Type" "URL",
   "Name" "Maxwell House: December Bride promo, 1950s (dmbb40505)",
   "Genre" "iTunes U"},
  "34531"
  {"Library Folder Count" "1",
   "Disc Number" "1",
   "Total Time" "205000",
   "Disc Count" "1",
   "Equalizer" "Jazz",
   "Persistent ID" "D367A4073B69D5CB",
   "Artist" "Sonny Rollins",
   "Album" "Sonny Rollins With The Modern Jazz Quartet",
   "Play Date" "3402904579",
   "Location" "...",
   "Composer" "Alan Jay Lerner/Frederick Loewe",
   "Track ID" "34531",
   "Track Count" "13",
   "Track Number" "2",
   "Date Modified" "2011-10-22T17:23:46.000-05:00",
   "Bit Rate" "476",
   "Size" "12244738",
   "Date Added" "2011-10-22T17:23:27.000-05:00",
   "Play Count" "1",
   "Year" "1953",
   "File Type" "1295270176",
   "Track Type" "File",
   "Name" "Almost Like Being In Love",
   "Sample Rate" "44100",
   "Genre" "Jazz",
   "Play Date UTC" "2011-10-31T11:16:19.000-05:00",
   "File Folder Count" "5",
   "Kind" "Apple Lossless audio file"},
  "40365"
  {"Library Folder Count" "1",
   "Disc Number" "1",
   "Total Time" "173346",
   "Disc Count" "1",
   "Equalizer" "Rock",
   "Persistent ID" "9BA8C41DA1EC33D5",
   "Comments" "Amazon.com Song ID: 203553394",
   "Artist" "Willie Nelson",
   "Album" "The Complete Atlantic Sessions",
   "Location" "...",
   "Composer" "Nelson",
   "Track ID" "40365",
   "Track Count" "61",
   "Track Number" "17",
   "Date Modified" "2012-07-12T21:46:16.000-05:00",
   "Sort Album" "Complete Atlantic Sessions",
   "Bit Rate" "256",
   "Size" "6767132",
   "Date Added" "2012-07-11T16:56:02.000-05:00",
   "Artwork Count" "1",
   "Year" "2006",
   "Album Artist" "Willie Nelson",
   "Track Type" "File",
   "Name" "Slow Down Old World (Alternate)",
   "Sample Rate" "44100",
   "Genre" "Country",
   "File Folder Count" "5",
   "Kind" "Matched AAC audio file"},
  "32069"
  {"Library Folder Count" "1",
   "Disc Number" "2",
   "Total Time" "363933",
   "Disc Count" "2",
   "Equalizer" "Jazz",
   "Persistent ID" "709FA4536BC9BE6E",
   "Artist" "Dean Martin",
   "Album" "Dino: The Essential Dean Martin",
   "Play Date" "3393180161",
   "Location" "...",
   "Composer" "Intro/Jonson/Lerner - Loewe/Porter",
   "Track ID" "32069",
   "Track Count" "7",
   "Track Number" "1",
   "Date Modified" "2014-01-25T15:00:35.000-06:00",
   "Bit Rate" "897",
   "Size" "40950806",
   "Date Added" "2011-07-10T20:58:38.000-05:00",
   "Artwork Count" "1",
   "Play Count" "1",
   "Year" "2005",
   "File Type" "1295270176",
   "Track Type" "File",
   "Name"
   "Introduction by Frank Sinatra / Drink To Me Only With Thine Eyes / Almost Like Being In Love / I Love Tahoe (Paris)",
   "Sample Rate" "44100",
   "Genre" "Singers",
   "Play Date UTC" "2011-07-10T22:02:41.000-05:00",
   "File Folder Count" "5",
   "Kind" "Apple Lossless audio file"},
  "26823"
  {"Library Folder Count" "1",
   "Disc Number" "1",
   "Total Time" "329600",
   "Disc Count" "1",
   "Equalizer" "Rock",
   "Persistent ID" "D1359AA1E1730EEF",
   "Artist" "Pink Floyd",
   "Album" "The Division Bell",
   "Location" "...",
   "Composer" "The Wright's/Gilmour",
   "Track ID" "26823",
   "Track Count" "11",
   "Track Number" "4",
   "Date Modified" "2011-03-17T21:04:47.000-05:00",
   "Sort Album" "Division Bell",
   "Bit Rate" "724",
   "Size" "29867738",
   "Date Added" "2011-03-17T21:04:29.000-05:00",
   "Year" "1994",
   "File Type" "1295270176",
   "Sort Composer" "Wright's/Gilmour",
   "Track Type" "File",
   "Name" "Marooned",
   "Sample Rate" "44100",
   "Genre" "Rock",
   "File Folder Count" "5",
   "Kind" "Apple Lossless audio file"},
  "31223"
  {"Library Folder Count" "1",
   "Disc Number" "1",
   "Total Time" "125533",
   "Disc Count" "1",
   "Equalizer" "Acoustic",
   "Persistent ID" "C005FE1D60AA65F5",
   "Artist" "Dwight Yoakam",
   "Album" "dwightyoakamacoustic.net",
   "Play Date" "3501442068",
   "Location" "...",
   "Track ID" "31223",
   "Track Count" "25",
   "Track Number" "4",
   "Date Modified" "2011-07-04T15:52:43.000-05:00",
   "Bit Rate" "829",
   "Size" "13032357",
   "Date Added" "2011-07-04T15:52:37.000-05:00",
   "Play Count" "3",
   "Year" "2000",
   "File Type" "1295270176",
   "Track Type" "File",
   "Name" "Please, Please Baby",
   "Sample Rate" "44100",
   "Genre" "Country",
   "Play Date UTC" "2014-12-14T22:47:48.000-06:00",
   "File Folder Count" "5",
   "Kind" "Apple Lossless audio file"},
  :more "23167 entries"},
 "Application Version" "12.3.1.23",
 "Library Persistent ID" "88ABD0BA83F503C5",
 "Features" "5"}

Now I have better idea of the structure and its variability.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment