XPath recipes

example.rst

Tutorials

• Scrapy Selectors on tutorialspoint.com

Code examples

The scrapy way of iterating through the response.

import requests

from scrapy.selector import Selector

url = 'https://f4e.europa.eu/careers/vacancies/Default.aspx'
response = requests.get(url)

items = Selector(text=response.text).xpath(
    "//div[contains(@class, 'careersPurple')]"
    "//div[contains(@class, 'careerList')]"
    "//div[contains(@class, 'careersItemPanel')]"
).extract()
for item in items:
    title = item.select("//div[contains(@class, 'careersTitle')]//text()") \
                .extract_first() \
                .strip()

You could also do it using lxml:

import requests

from lxml.html.soupparser import fromstring

url = 'https://f4e.europa.eu/careers/vacancies/Default.aspx'
response = requests.get(url)

tree = fromstring(response.content)

items = tree.xpath(
    "//div[contains(@class, 'careersPurple')]"
    "//div[contains(@class, 'careerList')]"
    "//div[contains(@class, 'careersItemPanel')]"
)

for item in items:
    title = item.xpath(
        "self::*//div[contains(@class, 'careersTitle')]//text()"
    )[0].strip()