Skip to content

Instantly share code, notes, and snippets.

@jdfight
Last active January 4, 2023 03:16
Show Gist options
  • Save jdfight/eedc98831dad9582a269e06b9b789cd0 to your computer and use it in GitHub Desktop.
Save jdfight/eedc98831dad9582a269e06b9b789cd0 to your computer and use it in GitHub Desktop.
Clojure web scraper: Scrape Nasa Apod and download the image
;;Scrape Nasa Apod webpage and download the image.
;; This is a quick project to familiarize myself with Clojure and Enlive
(ns apod-scraper.core
(:require [net.cgrand.enlive-html :as html])
(:require [clojure.string :as str])
(:gen-class))
(def apod-base-url "https://apod.nasa.gov/apod/")
(def apod-url (apply str [apod-base-url "astropix.html"]))
(def apod-content
"Get website content from Nasa Astronomy image of the day.
Returns: list of html content as hash-maps"
(html/html-resource (java.net.URL. apod-url)))
;; Example hash-mapped Image Element from Nasa Site:
;; {:tag :img, :attrs {:src image/2301/KembleCascade_Lease_960.jpg,
;; :alt The featured image shows a line of bright stars...,
;; :style max-width:100%},
;; :content nil}
;; On the apod website, this is the first img tag on the page:
(def img-link (apply str [apod-base-url (:src (:attrs (first (html/select apod-content [:img]))))]))
;;Copy uri to file - From Stack Overflow:
;;https://stackoverflow.com/questions/15628682/in-clojure-how-do-you-download-an-image-from-the-web-and-save-it-to-your-file-s
(defn copy-uri-to-file [uri file]
"Downloads a file from a uri and saves to local file."
(with-open [in (clojure.java.io/input-stream uri)
out (clojure.java.io/output-stream file)]
(clojure.java.io/copy in out)))
(defn -main []
"Download and save the image, using the file name split from the img link.."
(copy-uri-to-file img-link (last (str/split img-link #"/")))
(println "Downloaded: " img-link))
(defproject apod-scraper "0.0.1"
:description "Scrape Nasa Apod webpage and download the image. This is a quick project to familiarize myself with Clojure and Enlive"
:dependencies [[org.clojure/clojure "1.11.1"]
[enlive "1.1.6"]]
:main apod-scraper.core)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment