Python script to break large XML files

XML_breaker.py

import os
import sys
from xml.sax import parse
from xml.sax.saxutils import XMLGenerator

class CycleFile(object):

    def __init__(self, filename):
        self.basename, self.ext = os.path.splitext(filename)
        self.index = 0
        self.open_next_file()

    def open_next_file(self):
        self.index += 1
        self.file = open(self.name(), 'w')

    def name(self):
        return '%s%s%s' % (self.basename, self.index, self.ext)

    def cycle(self):
        self.file.close()
        self.open_next_file()

    def write(self, str):
        self.file.write(str)

    def close(self):
        self.file.close()

class XMLBreaker(XMLGenerator):
    
    def __init__(self, break_into=None, break_after=1000, out=None, *args, **kwargs):
        XMLGenerator.__init__(self, out, encoding='utf-8', *args, **kwargs)
        self.out_file = out
        self.break_into = break_into
        self.break_after = break_after
        self.context = []
        self.count = 0

    def startElement(self, name, attrs):
        XMLGenerator.startElement(self, name, attrs)
        self.context.append((name, attrs))

    def endElement(self, name):
        XMLGenerator.endElement(self, name)
        self.context.pop()
        if name == self.break_into:
            self.count += 1
            if self.count == self.break_after:
                self.count = 0
                for element in reversed(self.context):
                    self.out_file.write("\n")
                    XMLGenerator.endElement(self, element[0])
                self.out_file.cycle()
                XMLGenerator.startDocument(self)
                for element in self.context:
                    XMLGenerator.startElement(self, *element)

filename, break_into, break_after = sys.argv[1:]
parse(filename, XMLBreaker(break_into, int(break_after), out=CycleFile(filename)))

To get this working for python3.6, change the following:

Line 15: self.file = open(self.name(), 'w') to self.file = open(self.name(), 'wb')
Line 52: self.out_file.write("\n") to self.out_file.write("\n".encode('utf-8'))

Great job, saved a lot of time! Thank you so much.

The last chunk I´m creating is always way bigger in file size than the other chunks. Does anybody know where this comes from?!

Thanks. You saved my time :)

Very helpful. Thank you.

To get this working for python3.6, change the following:

Line 15: self.file = open(self.name(), 'w') to self.file = open(self.name(), 'wb')
Line 52: self.out_file.write("\n") to self.out_file.write("\n".encode('utf-8'))

Thanks your tips !

@nicwolf - thanks so much for creating this. This saved days of sed, awk and grepping through some huge OSS exports.

I am ripping a 30GB xml file to shreds without producing a very big memory footprint (~700MB). Thanks!

Thanks a lot @nic Wolff as it helped me to mitigate a production issue. And am trying out few enhancements over your code and will share the snippet (if I succeed ;)

I am trying to split these xml files by tags like what should i do?

zahidinho,
I use the parent MO in my case. Say I want to split out a bunch of Network Elements and the parent MO is xn:MeContext.

Here I want 20 Network Elements at a time:
Windows:
C:\Users\tools\scripts> python .\XMLBreaker.py C:\Route to file.xml xn:MeContext 20

*nix:
$ python xmlbreaker.py ~/Route to file.xml xn:MeContext 20

zahidinho,
I use the parent MO in my case. Say I want to split out a bunch of Network Elements and the parent MO is xn:MeContext.

Here I want 20 Network Elements at a time:
Windows:
C:\Users\tools\scripts> python .\XMLBreaker.py C:\Route to file.xml xn:MeContext 20

*nix:
$ python xmlbreaker.py ~/Route to file.xml xn:MeContext 20

I have no idea about python :) So i need a completed project...

You're the man it is fantastic

I'm new to Python. When I try to run this in a Jupyter notebook, I get a ValueError on line 59.
ValueError: not enough values to unpack (expected 3, got 2)
Can someone help please?

@fafzal820 The script expects three command-line parameters: the path to the input file, the element to split on, and the number of elements to put in each file. If you pass it fewer than three arguments, you'll get that error.

Your script has been very useful once I figured out a little more python. One more question. If it comes across an XML that is missing a tag, it will crash. Fair enough. Where can I add try / except to explicitly ask this script to release the file that was being written when it came across a bad xml file with missing tag?
I'm trying to automate splitting. In case splitting fails, I want to automatically delete the splits that were created. Currently, I'm able to delete all splits except for the last one where the script crashed, at least until the python script is done running.

thanks !!!! , this script solved my problem.

Man, you're a life saver. I had a 12GB file that I couldn't even open and your script has helped me soooo much.

Thanks for that!
Worked flawlessly without any error.

Call like:

python XML_breaker.py books.xml book 1000

where "books.xml" is the name of your input file, "book" is the element you want to split on, and 1000 is how many you want in each file. It will create "books1.xml", "books2.xml", &c.

I am pretty new to the python code, can you please hlep me where in the given python code i can put my xml file name with it's location.

@nicwolff Great script! Thank you! I am trying to adapt the script but am at my limits. My file looks like this:

<?xml version="1.0" encoding="utf-8"?>
<tmx version="1.4">
<header creationtool="Olifant" creationtoolversion="3.0.8.0" datatype="unkown" segtype="paragraph" adminlang="EN" srclang="DE" creationdate="20170608T122356Z" creationid="DZ" changedate="20170608T132457Z" changeid="DZ">
</header>
<body>
<tu>bla</tu>
<tu>bla</tu>
</body>
</tmx>

Splitting by <tu> elements works just fine. However, in the generated files, only the first file contains the header. In all other files the header is gone. Any tip how to adjust the script to get the header into the other split files?
Any help is much appreciated!
Thanks!

I am in the same boat, can you please let me know if you have got any solution.

Your script has been very useful once I figured out a little more python. One more question. If it comes across an XML that is missing a tag, it will crash. Fair enough. Where can I add try / except to explicitly ask this script to release the file that was being written when it came across a bad xml file with missing tag?
I'm trying to automate splitting. In case splitting fails, I want to automatically delete the splits that were created. Currently, I'm able to delete all splits except for the last one where the script crashed, at least until the python script is done running.

Can you please let me know where you are giving you the input file name, the element to split on etc..,

Great job.

Thank you very much!

I think there is a bug in this script, in that it is not able to distinguish between elements named the same, that occur at different levels. If you feed it:

python38 my.xml name 25000

Where my.xml look like:

<root>
    <name>
        <cd>
            <name>
            </name>
        </cd>
       <cassette>
            <name>
             </name>
        </cassette>
    </name>
</root>

The script believes that there are three name elements, where you probably only want it to believe there is one. So it consequently splits the files in strange positions, and you'll probably end up with more records than you thought you had.

@positivity13
for those who are getting the following error ,

self.file.write(str)
TypeError: write() argument must be str, not bytes

change the write function into

    def write(self, str_text):
        self.file.write(str_text.decode("utf-8") if isinstance(str_text,bytes) else str_text)

Thank you so much. It works very well.

@positivity13 for those who are getting the following error ,

self.file.write(str)
TypeError: write() argument must be str, not bytes

change the write function into

    def write(self, str_text):
        self.file.write(str_text.decode("utf-8") if isinstance(str_text,bytes) else str_text)

Thanks!

Hello Nic,
A licence would be great. I would like to know whether or not I can reuse your code and what are the terms.
Cheers!
R

Thanks for sharing the script! One issue I got is that the last splitted file always failed to open, the error looked something like 'error on line {N} at column {C}: Premature end of data in tag debt line {N}'. Is there anywhere in the script that can be modified to resolve this issue?

Thanks for sharing this script! It works so well!