Skip to content

Instantly share code, notes, and snippets.

@kyptin
Last active August 30, 2022 05:29
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 1 You must be signed in to fork a gist
  • Save kyptin/c3cb36c350885b06590c31bb97d07c23 to your computer and use it in GitHub Desktop.
Save kyptin/c3cb36c350885b06590c31bb97d07c23 to your computer and use it in GitHub Desktop.
Shannon's entropy in Clojure
(defn entropy
"Calculate the Shannon entropy, a.k.a. amount of information, in a
distribution. Uses the log-base-2 version, so the result can be thought of as
the number of bits of information. Accepts either a seq of numbers or a map of
freqencies like that returned by the `clojure.core/freqencies` function.
Ex:
(entropy [1]) ; => 0.0
(entropy [0 1 0 1]) ; => 1.0
(entropy [0 1 2 3]) ; => 2.0
(entropy {0 2, 1 2}) ; => 1.0
(entropy {0 50, 1 1}) ; => 0.1392329990550989"
[xs]
(assert (or (sequential? xs) (map? xs))
(str "input should be either sequential or a map but was: "
(pr-str xs)))
(if (sequential? xs)
(assert (every? number? xs)
(str "every value in input sequence should be a number; input = "
(pr-str xs)))
(assert (every? #(and (integer? %) (pos? %))
(vals xs))
(str "every value in input map should be a positive int; input = "
(pr-str xs))))
(let [freqs (vals (if (map? xs) xs (frequencies xs)))
n (reduce + 0 freqs)
log2 #(/ (Math/log %) (Math/log 2))
p #(double (/ % n))
summand (fn [freq]
(let [p (p freq)]
(* p (log2 p))))]
(reduce - 0 (map summand freqs))))
;; The MIT License (MIT)
;; Copyright (c) 2016 Jeff Terrell
;;
;; Permission is hereby granted, free of charge, to any person obtaining
;; a copy of this software and associated documentation files (the
;; "Software"), to deal in the Software without restriction, including
;; without limitation the rights to use, copy, modify, merge, publish,
;; distribute, sublicense, and/or sell copies of the Software, and to
;; permit persons to whom the Software is furnished to do so, subject to
;; the following conditions:
;;
;; The above copyright notice and this permission notice shall be
;; included in all copies or substantial portions of the Software.
;;
;; THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
;; EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
;; MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
;; NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
;; BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
;; ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
;; CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
;; SOFTWARE.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment