S.N.	方法 & 描述
extract()	它返回一個(gè)unicode字符串以及所選數(shù)據(jù)
re()	它返回Unicode字符串列表，當(dāng)正則表達(dá)式被賦予作為參數(shù)時(shí)提取
xpath()	它返回選擇器列表，它代表由指定XPath表達(dá)式參數(shù)選擇的節(jié)點(diǎn)。
css()	它返回選擇器列表，它代表由指定CSS表達(dá)式作為參數(shù)所選擇的節(jié)點(diǎn)。

在Shell中使用選擇器

若要演示選擇器在內(nèi)置Scrapy Shell 中，必須要在您的系統(tǒng)中安裝 IPython。這里最重要的是，在運(yùn)行時(shí)網(wǎng)址應(yīng)包含Scrapy引號(hào)之內(nèi); 否則使用的 URL “&” 字符將不起作用。可以通過(guò)在該項(xiàng)目的頂級(jí)目錄中，使用下面的命令啟動(dòng)一個(gè) shell：

scrapy shell "http://www.yiibai.com/scrapy/scrapy_environment.html"

shell 執(zhí)行后結(jié)果如下圖所示：

D:first_scrapy>scrapy shell "http://www.yiibai.com/scrapy/scrapy_environment.html"
2016-10-03 11:45:08 [scrapy] INFO: Scrapy 1.1.2 started (bot: first_scrapy)
2016-10-03 11:45:08 [scrapy] INFO: Overridden settings: {'NEWSPIDER_MODULE': 'first_scrapy.spiders', 'ROBOTSTXT_OBEY': True, 'DUPEFILTER_CLASS': 'scrapy.dupefilters.BaseDupeFilter', 'SPIDER_MODULES': ['first_scrapy.spiders'], 'BOT_NAME': 'first_scrapy', 'LOGSTATS_INTERVAL': 0}
2016-10-03 11:45:08 [scrapy] INFO: Enabled extensions:
['scrapy.extensions.telnet.TelnetConsole',
 'scrapy.extensions.corestats.CoreStats']
2016-10-03 11:45:08 [scrapy] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware',
 'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
 'scrapy.downloadermiddlewares.retry.RetryMiddleware',
 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
 'scrapy.downloadermiddlewares.chunked.ChunkedTransferMiddleware',
 'scrapy.downloadermiddlewares.stats.DownloaderStats']
2016-10-03 11:45:08 [scrapy] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
 'scrapy.spidermiddlewares.referer.RefererMiddleware',
 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
 'scrapy.spidermiddlewares.depth.DepthMiddleware']
2016-10-03 11:45:08 [scrapy] INFO: Enabled item pipelines:
[]
2016-10-03 11:45:08 [scrapy] DEBUG: Telnet console listening on 127.0.0.1:6023
2016-10-03 11:45:08 [scrapy] INFO: Spider opened
2016-10-03 11:45:09 [scrapy] DEBUG: Crawled (200) <GET http://www.yiibai.com/robots.txt> (referer: None)
2016-10-03 11:45:09 [scrapy] DEBUG: Crawled (200) <GET http://www.yiibai.com/scrapy/scrapy_environment.html> (referer: None)
[s] Available Scrapy objects:
[s]   crawler    <scrapy.crawler.Crawler object at 0x00000000042E3E80>
[s]   item       {}
[s]   request    <GET http://www.yiibai.com/scrapy/scrapy_environment.html>
[s]   response   <200 http://www.yiibai.com/scrapy/scrapy_environment.html>
[s]   settings   <scrapy.settings.Settings object at 0x00000000042E3E10>
[s]   spider     <firstSpider 'first' at 0x47f9f28>
[s] Useful shortcuts:
[s]   shelp()           Shell help (print this help)
[s]   fetch(req_or_url) Fetch request (or URL) and update local objects
[s]   view(response)    View response in a browser
>>>

當(dāng) shell 加載后，可以分別通過(guò)使用 response.body 和 response.header 訪問(wèn)主體或頭信息。同樣，也可以通過(guò)使用 response.selector.xpath（）或 response.selector.css（）運(yùn)行查詢的響應(yīng)結(jié)果。

例如：

>>> response.xpath('//title')
[<Selector xpath='//title' data=u'<title>Scrapyu5b89u88c5 - Scrapyu6559u7a0b</title>'>]
>>> response.xpath('//title').extract()
[u'<title>Scrapyu5b89u88c5 - Scrapyu6559u7a0b</title>']
>>> response.xpath('//title/text()')
[<Selector xpath='//title/text()' data=u'Scrapyu5b89u88c5 - Scrapyu6559u7a0b'>]
>>> response.xpath('//title/text()').extract()
[u'Scrapyu5b89u88c5 - Scrapyu6559u7a0b']
>>> response.xpath('//title/text()').extract()
[u'Scrapyu5b89u88c5 - Scrapyu6559u7a0b']
>>> response.xpath('//title/text()').re('(w+):')
[]
>>>

提取數(shù)據(jù)

從一個(gè)普通的HTML網(wǎng)站提取數(shù)據(jù)，查看該網(wǎng)站得到的 XPath 的源代碼。檢測(cè)后，可以看到數(shù)據(jù)將在UL標(biāo)簽，并選擇 li 標(biāo)簽中的元素。

代碼的下面行顯示了不同類(lèi)型的數(shù)據(jù)的提?。?/p>

選擇 li 標(biāo)簽內(nèi)的數(shù)據(jù)：

response.xpath('//ul/li')

對(duì)于選擇描述：

response.xpath('//ul/li/text()').extract()

對(duì)于選擇網(wǎng)站標(biāo)題：

response.xpath('//ul/li/a/text()').extract()

對(duì)于選擇網(wǎng)站的鏈接：

response.xpath('//ul/li/a/@href').extract()

下面的代碼用于演示上述提取的用法：

import scrapy

class MyprojectSpider(scrapy.Spider):
    name = "project"
    allowed_domains = ["dmoz.org"]
    start_urls = [
        "http://www.dmoz.org/Computers/Programming/Languages/Python/Books/",
        "http://www.dmoz.org/Computers/Programming/Languages/Python/Resources/"
    ]

    def parse(self, response):
        for sel in response.xpath('//ul/li'):
            title = sel.xpath('a/text()').extract()
            link = sel.xpath('a/@href').extract()
            desc = sel.xpath('text()').extract()
            print title, link, desc

上一篇：Scrapy項(xiàng)目（Items）下一篇：Scrapy教程