I think response and collection are the two classes where the bulk of the stuff is happenening, the importer is just an example of the external api. Actually looking back at this I think it could definitly be improved but I guess its an example of how to do it.
I recently did something similar, this time it was streaming a file from s3, but yielding the file line by line. The AWS SDK just gives you the s3 object chunk by chunk as its read from the socket, so I had to buffer and then yield as many lines as possible from each chunk. I returned an enumerator to do this rather than mixing in enumerable, I can't really share this code though, which is a shame cos I thought it was a nice implementation at the time.