transducers-chat.txt

user> ; i guess that's the power of transducers, is that they let you transform the values going through an arbitrary reduction
user> ;; like if we were just going to take a list of chars and turn it into a list of chars, then we could just (map (cesar 3) list-of-chars)
user> ;; but what a transducer lets us do is take *any* reducing function (e.g. str, sum-string, or anything else we can thing of) and transform it according to some function we wrote
user> ;; my impression is that another thing thats cool about them is that they work on any "reducable" thing, not just lists of maps of w/e
user> ;; the most interesting example probably being core.async channels
user> ;; so i think we could use this (map (cesar 3)) thing to transform a core.async channel of characters into another one
user> ;; that would be a cool project actually, use core.async to write a network server where you can type characters into one end and they'll be encrypted on the other
user> ;; or like, have the user start it up with an ecryption algorithm and a parter to connect to
user> ;; and then it will take everything they type and encrypt it on the fly, and send it over the network to the partner, who will decrypt it
user> ;; using whatever algorithm they specified :)
user> ;; so this could be just cesar cyphers where you have to specify the same character, or later you could plug in more complicated algorithms
user> ;; or even do stuff like compress the stream by writing a compressing transducer
user> 
user> ;; anyway i think it could be pretty cool, and would also give you more practice with / a chance to learn if you havent, core.async, which i think is one of the big draws of clojure
user> ;; who knows? only one way to find out :)
user> ;; you could then even demonstrate man-in-the-middle attacking it once you've made it
user> ;; like have the data stored uncompressed, but when someone requests it over the network, deliver compressed
user> ;; importantly, using constant memory / diskspace, since the files were very big
user> ;; like an easy way to do this would be to just download the whole file onto disk, use the `tar' command line tool to compress it, and then send it over once its compressed
user> ;; but that takes a long time and a ton of diskspace
user> ;; we wanted to user to be able to start reading data immediately, and also not take up a ton of diskspace doing it
user> ;; the gzip algorithm (where the "gz" in .tar.gz) comes from can be done streaming (i.e. you don't need to have all the data up front, you can just get the initial chunk of it and compress that, send it out to the user, then get the next chunk and compress that, etc., etc.)
user> ;; which we were able to do
user> ;; this was in python so the hard part was more just getting all the standard libraries to play nice together, since they were all mostly built to work with actual files on disk, not the in memory / network file-like objects i was trying to get them to work with
user> ;; so it involved a fair amount of hacking around and i think i even ended up copying/pasting some parts of the stdlib into my code :P
user> 
user> ; id be really curious to see what doing this using clojure channels / core.async o rhwatever would look like
user> ; basically the theory is that clojure is a much "nicer"/ more  modern / better designed language, so it might be a lot easier than what i had to do for python!
user> ;; another thing to think about is, you could implement the gzip compression yourself to learn about it if you wanted
user> ;; iwas just using a gzip library since i had a job to do, but at RC it can be fun to do these things yourself to see how it works 
user> 
user> ;; maybe? i'm not sure since i don't program in clojure a lot :/
user> ;; i rembmer when they were announced and people were all excited but i dont really follow it enough to know 
user> ;; my impressoin though is that the common thing to do isnt so much writing your own transducers, but that the standard library (map, take, filter, etc.) are all written such that they can return transducers, so its easy to generate your own for whatever you want to do
user> ;; i cant think of a good example of a transducer you'd write yourself from scratch off the top of my head
user> ;; thats not to say they dont exist! just that i dont know about them
user> ;; and also it seems like more of a thing that libraries would have rather than that you'd write yourself in like end-user facing applications if that makes sense
user> ;; idk, definitely worth googling around more, or like especially finding examples of libraries outside the std lib that have transducers
user> ;; i think probalby reading and trying to understand / play around with anything like that you find would be useful
user> 
user> ;; i think network servers / clients like the encryption / compression deamons i was talking about are good examples
user> ;; good question? i think it comes from latin?? maybe
user> ;; idk
user> ;; "demon" and "daemon" i think have similar roots, but for some reason everyone uses the later in talking about computer programs
user> ;; probably just a weird cultural thing