-
-
Save mahemoff/9207040 to your computer and use it in GitHub Desktop.
Refactor processing, make script standalone with initial env, add command-line argument
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
#!/usr/bin/env ruby | |
require 'open-uri' | |
require 'json' | |
language = 'en' | |
article = ARGV[0] || | |
begin | |
print 'What do you need to know? : ' | |
URI::encode gets.chomp | |
end | |
request_url = "http://#{language}.wikipedia.org/w/api.php?action=parse&page=#{article}&format=json&prop=text§ion=0" | |
open(request_url) do |file| | |
puts JSON.parse(file.read())['parse']['text'].first[1] | |
.gsub(/<\/?[^>]+>/, '') # strip tags | |
.gsub(/[[:space:]]+/, ' ') # strip whitespace | |
.gsub(/&#[0-9]+;/,'') # strip encoded | |
.gsub(/\[[0-9]+\]/,'') # strip referencing | |
end |
@schmerg The original arose in the Ruby G+ community, but Node ports welcome. I agree DOM munging would work nicely and could be achieved in a Ruby context with Nokogiri.
@mahemoff -- how about port as a ruby gem?
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Strip tags with regexp?? The horror ("the center cannot hold" etc :)
If you did it with Node you could pull in Caja and detag it properly and safely... just saying...
https://www.npmjs.org/package/sanitizerhttps://www.npmjs.org/package/sanitizer