腾讯招聘信息 爬取案例

前段时间看了了scrapy，有一个案例是爬取腾讯招聘信息的，当时看了腾讯网站，已经更新换代了，于是自己看了看，写下一个实际可行的demo

首先是观察腾讯招聘首页不在页面element内，但是依旧很容易就可以发现你要的信息

观察url，可以提取出 https://careers.tencent.com/tencentcareer/api/post/Query?pageIndex=1&pageSize=10 pageIndex是页码，pageSize是当页容量

接下来练练手scrapy框架

import scrapy import json class HrSpider(scrapy.Spider): name = 'hr' allowed_domains = ['tencent.com'] start_urls = ['https://careers.tencent.com/tencentcareer/api/post/Query?pageIndex=1&pageSize=10'] def __init__(self): self.url = 'https://careers.tencent.com/tencentcareer/api/post/Query?pageIndex={}&pageSize=10' self.count = 1 def parse(self, response): response = json.loads(response.text) # response.text可以获取response的文本字符串形式 li_list = response["Data"]["Posts"] # 获取当前页数据列表 count = response["Data"]["Count"] # 获取数据总计数 for li in li_list: item = {} item["title"] = li["RecruitPostName"] item["position"] = li["CountryName"] + li["LocationName"] item["cate"] = li["CategoryName"] item["public_info"] = li["Responsibility"] yield item # 构造下一页地址 if self.count*10 <= count: self.count += 1 next_url = self.url.format(self.count) yield scrapy.Request(next_url, callback=self.parse)

然后再在pipelines.py添加数据处理，写进mongodb

from itemadapter import ItemAdapter from pymongo import MongoClient # 创建数据库实例对象 client = MongoClient(host="192.168.152.145", port=27017) collection = client["tencent"]["hr"] class MyspiderPipeline: def process_item(self, item, spider): collection.insert(item) print(item) return item

最后就可以在数据库查看我们存储的信息了，很简单，有没有。