Skip to content

Instantly share code, notes, and snippets.

What would you like to do?
My first python script, using pandoc to convert a webpage to markdown
# Requires: pyandoc
# (Change path to pandoc binary in before installing package)
# TODO: Iterate over a list of webpages
# TODO: Clean up HTML by removing hard linebreaks
# TODO: Delete header and footer
import urllib2
import pandoc
# Open the desired webpage
url = ''
response = urllib2.urlopen(url)
webContent =
# Call on pandoc to convert webContent to markdown and write to file
doc = pandoc.Document()
doc.html = webContent
webConverted = doc.markdown
f = open('wendell-phillips.txt','w')

This comment has been minimized.

Copy link

commented Oct 30, 2012

here you go:

start = webContent.find('<title>')+7
end = webContent.find('</title>')
rawTitle = webContent[start:end]

Then, you probably want to clean that title up:

rawTitle = rawTitle.replace('Offprints  &raquo; Blog Archive   &raquo; ','')
if rawTitle.find('&#8220;'):
    rawTitle = rawTitle.replace('&#8220;','')
    rawTitle = rawTitle.replace('&#8221;','')
title = rawTitle.replace(' ','_')

The replace method is a built-in string method that let's you, well, replace parts of a string. But, strings are immutable, so you have to re-assign the new string to your variable each time you use it to "save" the change. This little bit then gets rid of un-necessary parts of your html title, and the codes for typographic quote marks, and replaces the remaining spaces with underscores to your new file will be named: blog_post_title.txt

Then, you'd use this to name the new file:

f = open(title+'.txt', 'w').write(webConverted)

This comment has been minimized.

Copy link

commented Oct 30, 2012

Also, in the interest of a little more brevity, you could make a few little changes to the code you have:

import urllib2, pandoc

url = ''
response = urllib2.urlopen(url).read()

# The first line creates a pandoc object. The second assigns the html response to that object
doc = pandoc.Document()
doc.html = response

f = open('wendell-phillips.txt','w').write(doc.markdown)

This way, the file is also automatically closed. That gives you six lines of code instead of eleven!


This comment has been minimized.

Copy link
Owner Author

commented Oct 30, 2012

Thanks for the tips!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.