Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?

Problem

Yahoo Pipes is closing, downloading http://pipes.yahoo.com/pipes/pipe.info?_out=json&_id= via browser for hundreds of pipes infeasible. There is https://gist.github.com/psychemedia/2351614 but it only works for public pipes and you don't want to make all your pipes public.

Solution

  1. Login to Yahoo Pipes, open debug console (F12 in Chrome/FF), click the "json feed" button to get to a page with this url scheme: http://pipes.yahoo.com/pipes/person.info?_out=json&display=pipes&guid=
  2. In the debug console get the cURL command for the current request (Net/Network tab, copy as cURL)
  3. Execute the command which we will create in the following to download all descriptions of your pipes including their ID
  • Prepend for page in ``seq 1 10``; do to (2.), where you replace 10 by the number of pages you have in your pipes overview
  • After _out=json&display=pipes in (2.) insert &page=$page
  • Append -o pipes_$page.json; done to (2.)
  1. Now that you have files including all IDs of your pipes you will extract all IDs into a single file
  1. Next you will curl all your pipe definitions, start with the plain cURL command from (2.) again and then execute the command we create in the next three substeps
  • Prepend while read -r line; do curl "http://pipes.yahoo.com/pipes/pipe.info?_out=json&_id=$line"
  • Remove the old curl url (e.g., curl "http://pipes.yahoo.com/pipes/person.info?_out=json&display=pipes&guid=...")
  • Append -o pipe_$line.json; done < pipeids.txt
  1. Now you have all json definitions of all your pipes and you can continue converting them to python: https://github.com/ggaughan/pipe2py or to node.js: https://github.com/neyric/pipes2js
@alexott

This comment has been minimized.

Copy link

commented Sep 10, 2015

Thank you for detailed description! It saved me time figuring all these details myself (to store my 385 pipes).

The only one thing that I encounter is that jq output strings with " character, that should be stripped from the pipeids.txt lines.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.