secret
Created

  • Download Gist
gistfile1.md
Markdown

Creating new Reducer sources

Issues

  • It is reasonably straightforward to create new reducer-transformers, because they must necessarily be separate from the existing lazy-seq transformers (eg take-while): if given a lazy-seq they must do one thing, and if given a reducer they must do something entirely different, so currently the best approach is to just have two separate functions.

  • However, because clojure is based on interfaces, when creating a new "source" from nothing (eg range), we have the opportunity to return an object which satisfies both the existing sequence interfaces and the new reducible protocols, able to integrate into either usage pattern. So ideally we would change clojure.core/range to return a new type which implements both ISeq and CollFold+CollReduce.

  • The ISeq interface itself, including explicit superinterfaces, is only seven methods, so it is no great hardship to implement those seven methods for every reducible source. However, in practice a sequence must also implement java.util.List to be indistinguishable from current lazy-seq objects, and interop has always been a high priority for clojure. List is just over two dozen methods, so implementing it for every source is not very pleasant.

Main Options

  • There are at least three options we've considered to deal with this difficulty:
    • Reject it entirely. Leave core/range exactly as is, unable to participate in the reducer protocols, and create a new reducers/range, which cannot act as a lazy sequence. This reduces the overall usefulness of the reducers, but is easy and (I think) simple.
    • Do it in a macro. One version of my reducers/range patch creates a defseq macro, which acts like deftype but also implements ISeq and List in terms of your Seqable implementation. This gets us all the functionality we asked for, but requires a large addition to core or to core_deftype, which needs to be scrutinized very carefully.
    • Do it in java. The main difficulty with doing this in clojure is lack of implementation inheritance: we need a named class that is very much like (perhaps even identical to?) LazySeq, but has a distinct name so that we can extend protocols to it. In java that would be easy, but it is progressing in the wrong direction: we don't want to move functionality from clojure to java if it's avoidable. In particular, this would need a new java class for each lazy-seq (even in "user" code) that wants to participate in the reducer library, greatly increasing the friction.

"Oddball" Options

  • I also have some "scratchpad" ideas which I haven't thought hard about but might be viable:

    • Add a single ReducibleLazySeq class, which has a field for storing a reducer as well as the existing lazy-seq code. Then we can extend CollFold/CollReduce to this type once, and anyone can create a new instance with something like

        (defn iterate [f init]
            (reducible-lazy-seq (reify CollReduce ...)
                                (cons init (iterate f (f init)))))
      

      Edit Note that after some further investigation I noticed LazySeq is final, so we can't inherit from it even in java. I'm not sure if making it non-final is acceptable in terms of backwards-compatibility, performance, or design, but it would simplify things. Perhaps we could use the existing LazySeq class and add a new, optional field to it to hold a reducer?

    • Don't require that ISeq instances also implement List. This would need only a few direct changes to clojure.lang, but would hinder the ability to use these sequences in an interop context. I don't personally think this is a good idea, but because ISeq does not explicitly extend List it is possible, and that seems to imply Rich considered it might be desirable at some point. If we do that, most of the difficulties disappear.

Please sign in to comment on this gist.

Something went wrong with that request. Please try again.