In the context of web scraping, is a nice tool to have in your belt, as it allows you to write specifications of document locations more flexibly than CSS selectors. In case you’re looking for a tutorial, . XPath here is a XPath tutorial with nice examples In this post, we’ll show you some tips we found valuable when using XPath in the trenches, using for our examples. Scrapy Selector API Avoid using contains(.//text(), ‘search text’) in your XPath conditions. Use contains(., ‘search text’) instead. Here is why: the expression yields a collection of text elements — a . And when a node-set is converted to a string, which happens when it is passed as argument to a string function like or , results in the text for the element only. .//text() node-set contains() starts-with() first >>> from scrapy import Selector>>> sel = Selector(text='<a href="#">Click here to go to the Next Page</a>')>>> xp = lambda x: sel.xpath(x).extract() # let's type this only once>>> xp('//a//text()') # take a peek at the node-set[u'Click here to go to the ', u'Next Page']>>> xp('string(//a//text())') # convert it to a string[u'Click here to go to the '] A converted to a string, however, puts together the text of itself plus of all its descendants: node >>> xp('//a[1]') # selects the first a node[u'<a href="#">Click here to go to the Next Page</a>']>>> xp('string(//a[1])') # converts it to string[u'Click here to go to the Next Page'] So, in general: GOOD: >>> xp("//a[contains(., 'Next Page')]")[u'<a href="#">Click here to go to the Next Page</a>'] BAD: >>> xp("//a[contains(.//text(), 'Next Page')]")[] GOOD: >>> xp("substring-after(//a, 'Next ')")[u'Page'] BAD: >>> xp("substring-after(//a//text(), 'Next ')")[u''] You can read . more detailed explanations about string values of nodes and node-sets in the XPath spec Beware of the difference between //node[1] and (//node)[1] selects all the nodes occurring first under their respective parents. //node[1] selects all the nodes in the document, and then gets only the first of them. (//node)[1] >>> from scrapy import Selector>>> sel=Selector(text="""....: <ul class="list">....: <li>1</li>....: <li>2</li>....: <li>3</li>....: </ul>....: <ul class="list">....: <li>4</li>....: <li>5</li>....: <li>6</li>....: </ul>""")>>> xp = lambda x: sel.xpath(x).extract()>>> xp("//li[1]") # get all first LI elements under whatever it is its parent[u'<li>1</li>', u'<li>4</li>']>>> xp("(//li)[1]") # get the first LI element in the whole document[u'<li>1</li>']>>> xp("//ul/li[1]") # get all first LI elements under an UL parent[u'<li>1</li>', u'<li>4</li>']>>> xp("(//ul/li)[1]") # get the first LI element under an UL parent in the document[u'<li>1</li>'] Also, gets a collection of the local anchors that occur first under their respective parents. //a[starts-with(@href, '#')][1] gets the first local anchor in the document. (//a[starts-with(@href, '#')])[1] When selecting by class, be as specific as necessary If you want to select elements by a CSS class, the XPath way to do that is the rather verbose: *[contains(concat(' ', normalize-space( ), ' '), ' someclass ')] @class Let’s cook up some examples: >>> sel = Selector(text='SomeoneSome content')>>> xp = lambda x: sel.xpath(x).extract() doesn’t work because there are multiple classes in the attribute BAD: >>> xp("//*[ ='content']")[] @class gets more than we want BAD: >>> xp("//*[contains( ,'content')]")[u’Someone’] @class GOOD: >>> xp("//*[contains(concat(' ', normalize-space( ), ' '), ' content ')]")[u'Some content'] @class And many times, you can just use a CSS selector instead, and even combine the two of them if needed: ALSO GOOD: >>> sel.css(".content”).extract()[u'Some content']>>> sel.css('.content').xpath(' ').extract()[u'content text-wrap'] @class Read . more about what you can do with Scrapy’s Selectors here Learn to use all the different axes It is handy to know how to use the axes, you can to quickly review this. follow through the examples given in the tutorial In particular, you should note that and are not the same thing, this is a common source of confusion. The same goes for and , and also and . following following-sibling preceding preceding-sibling ancestor parent Useful trick to get text content Here is another XPath trick that you may use to get the interesting text contents: 1//*[not(self::script or self::style)]/text()[normalize-space(.)] This excludes the content from and tags and also skip whitespace-only text nodes. Source: script style http://stackoverflow.com/a/19350897/2572383 Do you have another XPath tip? Please, leave us a comment with your tips or questions. Originally published on the . Scrapinghub Blog This post was written by Elias Dorneles ( ), a Software Developer at Scrapinghub. @eliasdorneles Please heart “Recommend” and share these tips far and wide. Find out what web scraping and web data can do for you .

Twitter

Too Long; Didn't Read

In your car, at home, or at work — Bosch technology shapes many areas of life.

XPath Tips from the Web Scraping Trenches

Too Long; Didn't Read

Company Mentioned

ScrapingHub

About Author

TOPICS

THIS ARTICLE WAS FEATURED IN...

XPath Tips from the Web Scraping Trenches

Too Long; Didn't Read

Company Mentioned

ScrapingHub

About Author

TOPICS

THIS ARTICLE WAS FEATURED IN...

RELATED STORIES