Python's XMLPullParser runs out of memory

1.py

import xml.etree.ElementTree as ET
import argparse

arg_parser = argparse.ArgumentParser()
arg_parser.add_argument('in_file')
args = arg_parser.parse_args()

def main(args):
    with open(args.in_file) as f:
        parser = ET.XMLPullParser(['start', 'end'])
        for line in f:
            parser.feed(line)
            for event, elem in parser.read_events():
                pass

main(args)

This seems to be intended behavior.

http://docs.python.jp/3/library/xml.etree.elementtree.html

If you don’t mind your application blocking on reading XML data but would still like to have incremental parsing capabilities, take a look at iterparse(). It can be useful when you’re reading a large XML document and don’t want to hold it wholly in memory.

Using lxml or cElementTree and calling element.clear() solved the problem.

import lxml.etree as ET
import argparse

arg_parser = argparse.ArgumentParser()
arg_parser.add_argument('in_file')
args = arg_parser.parse_args()

def main(args):
    with open(args.in_file, 'rb') as f:
        it = ET.iterparse(f, ['start', 'end'])
        for event, elem in it:
            elem.clear()

main(args)