如何提升mongodb find速率

linkbase中存有幾十萬(wàn)條url，數(shù)據(jù)如下所示：

{ "_id" : ObjectId("5a322cad6c85e0369ab54161"), "url" : "https://www.xxxx.com/", "status_code" : 0, "spider_name" : "test_spider", "crawl_time" : "2017-12-14 15:47:57" }
{ "_id" : ObjectId("5a322c9f6c85e03699b54227"), "url" : "https://www.xxxxaa.com/", "status_code" : 0, "spider_name" : "test_spider", "crawl_time" : "2017-12-14 15:47:43" }
{ "_id" : ObjectId("5a2f082c6c85e0c65bc553c9"), "url" : "https://aabb.com", "status_code" : 403, "spider_name" : "test_spider", "crawl_time" : "2017-12-12 06:35:24" }
{ "_id" : ObjectId("5a2f082b6c85e0c65bc553c8"), "url" : "https://cccc.com", "status_code" : 403, "spider_name" : "test_spider", "crawl_time" : "2017-12-12 06:35:23" }

我用數(shù)據(jù)庫(kù)查詢工具，db.linkbase.find({"spider_name":"test_spider"})大約耗時(shí)122秒
數(shù)據(jù)庫(kù)有相關(guān)索引

[
    {
        "v" : 1,
        "key" : {
            "_id" : 1
        },
        "name" : "_id_",
        "ns" : "linkbase.linkbase"
    },
    {
        "v" : 1,
        "key" : {
            "spider_name" : 1
        },
        "name" : "spider_name_1",
        "ns" : "linkbase.linkbase"
    }
]

程序調(diào)用一個(gè)函數(shù)，max_requests每次取10個(gè)：

    def get_mongodb_linkbase(self,max_requests):
        linkbase = []
        for i in range(max_requests):
            result = self.db_linkbase_collection.find_one_and_delete({'spider_name':'test_spider'})
            if result == None:
                continue
            else:
                linkbase.append(result)
        return linkbase

可是爬蟲(chóng)運(yùn)行的時(shí)候，感覺(jué)速度并不快，爬蟲(chóng)代碼采用了協(xié)程gevent，每個(gè)協(xié)程開(kāi)50個(gè)

    def handle_linkbase():
        while True:
            linkbase_info = mongo_insert.get_mongodb_linkbase(10)
            if len(linkbase_info) == 0:
                time.sleep(random.choice(range(60)))
            else:
                for url in linkbase_info:
                    test.header['User-Agent'] = random.choice(setting.user_agent)
                    test.save_url_to_task(url['url'],header=test.header)

    def gevent_fun_linkbase():
        jobs = [gevent.spawn(handle_linkbase) for i in range(50)]
        gevent.joinall(jobs)
        
     m2 = multiprocessing.Process(target=gevent_fun_linkbase)
     m2.start()
     m2.join()

并將協(xié)程放入多進(jìn)程。也就是三個(gè)進(jìn)程分別運(yùn)行三個(gè)協(xié)程，每個(gè)協(xié)程里開(kāi)50個(gè)程序。但是感覺(jué)速度并不快，甚至有卡頓現(xiàn)象。就是在tail -f log的時(shí)候，發(fā)現(xiàn)有幾秒中，無(wú)信息輸出，并且運(yùn)行速度慢。

目前不太清楚是不是mongodb查詢時(shí)候造成的卡頓。

感覺(jué)也不是代理造成的卡頓，requests我把重試關(guān)了，timeout為1.

請(qǐng)各位大牛幫忙分析。