
使用自己的find_all


2017-09-11 16:50:09 [DBService.DBService] INFO: task_list len is 1
2017-09-11 16:50:09 [DBService.DBService] INFO: execute self.db["drugfutureDetailCrawler"].find({}, projection={'url': True, '_id': False, 'cas_no': True, 'desc': True}, limit=5000, skip=0, sort=[])
2017-09-11 16:50:09 [DBService.DBService] INFO: task start
2017-09-11 16:50:38 [DBService.DBService] INFO: execute self.db["drugfutureDetailCrawler"].find({}, projection={'url': True, '_id': False, 'cas_no': True, 'desc': True}, limit=5000, skip=0, sort=[]) end
2017-09-11 16:50:38 [DBService.DBService] DEBUG: Thread-[0] fetch rate [5000 / 117566]
2017-09-11 16:50:38 [DBService.DBService] INFO: execute self.db["drugfutureDetailCrawler"].find({}, projection={'url': True, '_id': False, 'cas_no': True, 'desc': True}, limit=5000, skip=5000, sort=[])
2017-09-11 16:51:13 [DBService.DBService] INFO: execute self.db["drugfutureDetailCrawler"].find({}, projection={'url': True, '_id': False, 'cas_no': True, 'desc': True}, limit=5000, skip=5000, sort=[]) end
2017-09-11 16:51:13 [DBService.DBService] DEBUG: Thread-[0] fetch rate [10000 / 117566]
2017-09-11 16:51:13 [DBService.DBService] INFO: execute self.db["drugfutureDetailCrawler"].find({}, projection={'url': True, '_id': False, 'cas_no': True, 'desc': True}, limit=5000, skip=10000, sort=[])
2017-09-11 16:51:46 [DBService.DBService] INFO: execute self.db["drugfutureDetailCrawler"].find({}, projection={'url': True, '_id': False, 'cas_no': True, 'desc': True}, limit=5000, skip=10000, sort=[]) end
2017-09-11 16:51:46 [DBService.DBService] DEBUG: Thread-[0] fetch rate [15000 / 117566]
2017-09-11 16:51:46 [DBService.DBService] INFO: execute self.db["drugfutureDetailCrawler"].find({}, projection={'url': True, '_id': False, 'cas_no': True, 'desc': True}, limit=5000, skip=15000, sort=[])
2017-09-11 16:52:18 [DBService.DBService] INFO: execute self.db["drugfutureDetailCrawler"].find({}, projection={'url': True, '_id': False, 'cas_no': True, 'desc': True}, limit=5000, skip=15000, sort=[]) end
2017-09-11 16:52:18 [DBService.DBService] DEBUG: Thread-[0] fetch rate [20000 / 117566]
2017-09-11 16:52:18 [DBService.DBService] INFO: execute self.db["drugfutureDetailCrawler"].find({}, projection={'url': True, '_id': False, 'cas_no': True, 'desc': True}, limit=5000, skip=20000, sort=[])
2017-09-11 16:52:50 [DBService.DBService] INFO: execute self.db["drugfutureDetailCrawler"].find({}, projection={'url': True, '_id': False, 'cas_no': True, 'desc': True}, limit=5000, skip=20000, sort=[]) end
2017-09-11 16:52:50 [DBService.DBService] DEBUG: Thread-[0] fetch rate [25000 / 117566]
2017-09-11 16:52:50 [DBService.DBService] INFO: execute self.db["drugfutureDetailCrawler"].find({}, projection={'url': True, '_id': False, 'cas_no': True, 'desc': True}, limit=5000, skip=25000, sort=[])


使用pymongo 官方的接口


