deterministic xpath generation of BeautifulSoup elements for web crawling

soup_xpath_gen.py

import html

BAD_CHARS = set(["\"", "'", "[", "]"])

# generates an xpath string for a given BeautifulSoup element
def soup_xpath_gen(element):
    xpath = ""
    while element.name != 'document':
        if element.name == 'html':
            xpath = "/html/" + xpath
            break  # second to last
        else:
            items = list(element.attrs.items())
            if not items:
                el_xpath = str(element.name)
            else:
                el_xpath = str(element.name) + "["
                one = False
                for i, (k, v) in enumerate(items):
                    if not any(char in BAD_CHARS for char in v):
                        if k == "title":
                            continue
                        if one:
                            el_xpath = el_xpath + " and "
                        one = True
                        if "/" in v:  # URL matching is wonky, so we ignore
                            el_xpath += "normalize-space(@%s)" % k
                        else:
                            if isinstance(v, list):
                                v = " ".join(v)
                            el_xpath += "normalize-space(@%s)=normalize-space(\'%s\')" % (k, html.escape(v))
                el_xpath += "]"
            xpath = el_xpath + "/" + xpath
        element = element.parent
    return xpath.rstrip("/")

This is based on attributes, so it should work even if the DOM is updated after pulling the page content but before accessing the element (useful for web crawling). Sorry if not clean!

How useful, thank you for sharing!

One suggestion I have is line 1;
import html
Otherwise, line 30 will not work with:
html.escape(v)

How useful, thank you for sharing!

One suggestion I have is line 1;
import html
Otherwise, line 30 will not work with:
html.escape(v)

Good catch, thanks! Fixed.

Thank you for this code.

Question. What to do when get this error:
AttributeError: 'WebElement' object has no attribute 'name'
?