Skip to content

Instantly share code, notes, and snippets.

What would you like to do?
Parse and dump a sitemap (using Python)
#! /usr/bin/env python
# -*- coding: utf-8 -*-
Inspired by Craig Addyman (
Enhanced by Viktor Petersson ( / @vpetersson
from bs4 import BeautifulSoup
import requests
def get_sitemap(url):
get_url = requests.get(url)
if get_url.status_code == 200:
return get_url.text
print 'Unable to fetch sitemap: %s.' % url
def process_sitemap(s):
soup = BeautifulSoup(s)
result = []
for loc in soup.findAll('loc'):
return result
def is_sub_sitemap(s):
if s.endswith('.xml') and 'sitemap' in s:
return True
return False
def parse_sitemap(s):
sitemap = process_sitemap(s)
result = []
while sitemap:
candidate = sitemap.pop()
if is_sub_sitemap(candidate):
sub_sitemap = get_sitemap(candidate)
for i in process_sitemap(sub_sitemap):
return result
def main():
sitemap = get_sitemap('')
print '\n'.join(parse_sitemap(sitemap))
if __name__ == '__main__':

This comment has been minimized.

Copy link

HQJaTu commented Jul 11, 2018

Above code doesn't account for <sitemap><loc> URL to have any arguments in it.

I created an improved version at to address that issue. The obvious solution is to first parse the url, and check the URL path-part.


This comment has been minimized.

Copy link

dvir-cdsoft commented Feb 5, 2019

sorry for the question, im new at python
to where the code dump the sitemap ?
do i need to write any writing to file ?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.