quick way to parse huge xml files in Clojure

core.clj

(ns xmltest.core
  (:require [clojure.data.xml :as xml])
  (:import [java.io FileInputStream]
           [java.util.zip GZIPInputStream]))

(defn parse [filename]
  (xml/parse (FileInputStream. filename)))

(defn parse-gzipped [filename]
  (xml/parse (GZIPInputStream. (FileInputStream. filename))))

(defn get-title-values-from-file
  [tree]
  (map (fn [page]
         (->> (filter #(= (:tag %) :title) (:content page))
              (first)
              (:content)
              (apply str)))
       (:content tree)))

(comment
  (require '[xmltest.core :as c])
  ;; big.xml.gz is a gzipped file containing a billion <page> tags,
  ;; with a compressed size of 234M (original is 3.4GB).
  (->> (c/parse-gzipped "/home/fmw/clj/xmltest/big.xml.gz")
      (c/get-title-values-from-file)
      (take 100000))) ;; remove (take 100000) to get the full sequence

Thanks. clojure.data.xml seems promising.

I'm currently trying out with-open and clojure.java.io:

(ns example.core
  (:require [clojure.data.xml :as xml]
            [clojure.java.io :as io])) 

(with-open [rdr (io/reader filename)]
  (xml/parse rdr))

I've seen some errors like:

XMLStreamException ParseError at [row,col]:[89,245]
Message: Stream closed  com.sun.org.apache.xerces.internal.impl.XMLStreamReaderImpl.next (XMLStreamReaderImpl.java:592)

Strangely, your way, using (FileInputStream. filename) worked. But my way, with io/reader did not.

Does your code close the file when it is done? My understanding is that with-open handles that.

My version of this is here: https://gist.github.com/bluemont/8e8b717a732446fc9a8e