获取拉勾网招聘信息数据

使用 curl 模拟 ajax 请求获取数据

1、找到目标城市和目标职位，然后按“最新排序”。

链接为：https://www.lagou.com/jobs/list_iOS?px=new&city=%E5%8C%97%E4%BA%AC#order

2、右键点击页面，选择"检查"，进入浏览器调试界面，切换到调试器的 network -> xhr 标签下

3、cmd + R 刷新页面，此时会捕捉到此页面发出的xhr请求，找到 positionAjax.json 开头的请求，右键点击选择 Copy as cUrl(bash)。

4、本文是wins，安装curl

下载curl，本文使用的是：curl-7.33.0-win64-ssl-sspi.zip

下载地址为：http://download.csdn.net/detail/u012689336/9683328

下载好后解压到桌面就好了。

5、安装git

本文使用的是：Git-2.9.3-64-bit.exe

下载地址为：http://download.csdn.net/detail/u012689336/9683333

下载之后直接安装就可以了。

6、桌面右键打开，选择Git Bash Here

7、上一步中的curl指令复制到终端，即右键选择粘贴。

curl 'https://www.lagou.com/jobs/positionAjax.json?px=new&city=%E5%8C%97%E4%BA%AC&needAddtionalResult=false' -H 'Cookie: LGMOID=20161115105219-5AA07DE3A870C9C28DAD87306DE31CFF; user_trace_token=20161115105219-86a4c72f-aade-11e6-98ea-5254005c3644; LGUID=20161115105219-86a4cb14-aade-11e6-98ea-5254005c3644; HISTORY_POSITION=2330069%2C15k-25k%2CUdesk%EF%BC%8D%E4%BC%81%E4%B8%9A%E7%BA%A7%E6%99%BA%E8%83%BD%E5%AE%A2%E6%9C%8D%E5%B9%B3%E5%8F%B0%2CJava%7C; JSESSIONID=29F589B05C47BAFBAD36BCEC390793BC; _gat=1; PRE_UTM=; PRE_HOST=www.cnblogs.com; PRE_SITE=http%3A%2F%2Fwww.cnblogs.com%2Fios122%2Fp%2F5828673.html; PRE_LAND=https%3A%2F%2Fwww.lagou.com%2Fjobs%2Flist_iOS%3Fpx%3Dnew%26city%3D%25E5%258C%2597%25E4%25BA%25AC; LGSID=20161115133620-703f2d88-aaf5-11e6-a7f9-525400f775ce; LGRID=20161115133840-c33e5bf7-aaf5-11e6-a7fa-525400f775ce; Hm_lvt_4233e74dff0ae5bd0a3d81c6ccf756e6=1479178447,1479179434,1479188155,1479188294; Hm_lpvt_4233e74dff0ae5bd0a3d81c6ccf756e6=1479188294; _ga=GA1.2.1088550872.1479178314; SEARCH_ID=12f8eb7a0d384e5eba66ed04f73c4cdf' -H 'Origin: https://www.lagou.com' -H 'X-Anit-Forge-Code: 0' -H 'Accept-Encoding: gzip, deflate, br' -H 'Accept-Language: zh-CN,zh;q=0.8' -H 'User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36' -H 'Content-Type: application/x-www-form-urlencoded; charset=UTF-8' -H 'Accept: application/json, text/javascript, */*; q=0.01' -H 'Referer: https://www.lagou.com/jobs/list_iOS?px=new&city=%E5%8C%97%E4%BA%AC' -H 'X-Requested-With: XMLHttpRequest' -H 'Connection: keep-alive' -H 'X-Anit-Forge-Token: None' --data 'first=true&pn=1&kd=iOS' --compressed

这个 curl代码非常长，对于本次分析来说，最关键的是末尾的 pn=1&kd=iOS，分别代表着页面和职位，动态设置，即可获取更多职位的更多数据。

8、按下回车键，即可看到输出。可以看到，与网站的第一页获取的实际数据是完全对应的。

{"success":true,"requestId":null,"resubmitToken":null,"msg":null,"content":{"pageNo":1,"pageSize":15,"positionResult":{"resultSize":15,"locationInfo":{"city":"北京","district":null,"queryByGisCode":false,"businessZone":null,"locationCode":null},"queryAnalysisInfo":{"positionName":"ios","companyName":null,"industryName":null,"usefulCompany":false},"strategyProperty":{"name":"dm-csearch-newSimScorer","id":1},"totalCount":805,"result":[{"businessZones":null,"imState":"today","lastLogin":1479189411000,"publisherId":3003531,"explain":null,"plus":null,"pcShow":0,"appShow":0,"deliver":0,"gradeDescription":null,"promotionScoreExplain":null,"firstType":"开发/测试/运维类","secondType":"前端开发/移动开发","positionLables":null,"createTime":"2016-11-15 13:58:23","companyId":101,"positionName":"WX-iOS开发工程师","education":"本科","city":"北京","positionId":2578379,"financeStage":"上市公司","companyShortName":"迅雷","companyLogo":"image1/M00/00/01/Cgo8PFTUV_SADj9rAABrXBj73Q8580.jpg","salary":"15k-30k","industryField":"移动互联网","district":null,"companyLabelList":["年底双薪","节日礼物","技能培训","免费班车"],"positionAdvantage":"五险一金+班车+早晚餐+健康体检+运动日","jobNature":"全职","workYear":"3-5年","approve":1,"score":0,"companySize":"500-2000人","companyFullName":"深圳市迅雷网络技术有限公司","adWord":0,"formatCreateTime":"13:58发布"},{"businessZones":["立水桥","北苑"],"imState":"today","lastLogin":1479188246000,"publisherId":5071165,"explain":null,"plus":null,"pcShow":0,"appShow":0,"deliver":0,"gradeDescription":null,"promotionScoreExplain":null,"firstType":"技术","secondType":"前端开发/移动开发","positionLables":["平台","iOS","游戏"],"createTime":"2016-11-15 13:54:42","companyId":129516,"positionName":"iOS游戏平台接入SDK开发工程师","education":"不限","city":"北京","positionId":1906251,"financeStage":"成长型(A轮)","companyShortName":"天游互动","companyLogo":"i/image/M00/2B/8B/CgqKkVc1RVWATTcRAABOmw2czME554.jpg","salary":"5k-10k","industryField":"移动互联网,游戏","district":"朝阳区","companyLabelList":["移动游戏","专项奖金","午餐补助","通讯津贴"],"positionAdvantage":"弹性工作，可享期权，项目奖金提成","jobNature":"全职","workYear":"1-3年","approve":0,"score":0,"companySize":"50-150人","companyFullName":"北京天游互动科技有限公司","adWord":0,"formatCreateTime":"13:54发布"},{"businessZones":["知春路","中关村","双榆树"],"imState":"today","lastLogin":1479188052000,"publisherId":208,"explain":null,"plus":null,"pcShow":0,"appShow":0,"deliver":0,"gradeDescription":null,"promotionScoreExplain":null,"firstType":"技术","secondType":"前端开发/移动开发","positionLables":["iOS"],"createTime":"2016-11-15 13:34:44","companyId":139,"positionName":"iOS","education":"本科","city":"北京","positionId":1941892,"financeStage":"成长型(A轮)","companyShortName":"九玉","companyLogo":"image1/M00/00/02/Cgo8PFTUV_WAD7hfAABXYsAPI8k943.png","salary":"15k-25k","industryField":"移动互联网","district":"海淀区","companyLabelList":["技能培训","股票期权","带薪年假","岗位晋升"],"positionAdvantage":"股票期权,弹性工作制,免费工作餐,水果零食","jobNature":"全职","workYear":"不限","approve":1,"score":0,"companySize":"15-50人","companyFullName":"九玉（北京）科技有限公司","adWord":0,"formatCreateTime":"13:34发布"},{"businessZones":["望京","大山子","酒仙桥"],"imState":"today","lastLogin":1479187427000,"publisherId":1493277,"explain":null,"plus":null,"pcShow":0,"appShow":0,"deliver":0,"gradeDescription":null,"promotionScoreExplain":null,"firstType":"技术","secondType":"移动开发","positionLables":["iOS"],"createTime":"2016-11-15 13:25:43","companyId":59119,"positionName":"iOS高级开发工程师","education":"大专","city":"北京","positionId":1216114,"financeStage":"成长型(A轮)","companyShortName":"君融贷","companyLogo":"i/image/M00/21/36/Cgp3O1cQU_-ADtiUAAAaLx-Z_J0830.jpg","salary":"15k-25k","industryField":"金融","district":"朝阳区","companyLabelList":["节日礼物","带薪年假","岗位晋升","管理规范"],"positionAdvantage":"带薪年假 弹性工作 领导好","jobNature":"全职","workYear":"5-10年","approve":1,"score":0,"companySize":"50-150人","companyFullName":"君融贷（北京）信息技术服务有限公司","adWord":0,"formatCreateTime":"13:25发布"},{"businessZones":["新街口","德胜门","小西天"],"imState":"today","lastLogin":1479189406000,"publisherId":2647983,"explain":null,"plus":null,"pcShow":0,"appShow":0,"deliver":0,"gradeDescription":null,"promotionScoreExplain":null,"firstType":"开发/测试/运维类","secondType":"前端开发/移动开发","positionLables":["iOS"],"createTime":"2016-11-15 13:18:13","companyId":97172,"positionName":"iOS开发工程师","education":"本科","city":"北京","positionId":2578159,"financeStage":"初创型(天使轮)","companyShortName":"Humax Tech","companyLogo":"image2/M00/07/B8/CgpzWlYAG72AQz4dAAAWnjtTGgg550.jpg?cc=0.3403482724724128","salary":"10k-18k","industryField":"硬件","district":"西城区","companyLabelList":["年终分红","交通补助","午餐补助","绩效奖金"],"positionAdvantage":"前景好,晋升空间,弹性","jobNature":"全职","workYear":"3-5年","approve":1,"score":0,"companySize":"15-50人","companyFullName":"合优智景（北京）科技有限公司","adWord":0,"formatCreateTime":"13:18发布"},{"businessZones":null,"imState":"disabled","lastLogin":1479185306000,"publisherId":6082811,"explain":null,"plus":null,"pcShow":0,"appShow":0,"deliver":0,"gradeDescription":null,"promotionScoreExplain":null,"firstType":"技术","secondType":"移动开发","positionLables":["iOS","架构师"],"createTime":"2016-11-15 12:53:55","companyId":10723,"positionName":"iOS架构师","education":"本科","city":"北京","positionId":2535611,"financeStage":"成长型(B轮)","companyShortName":"酷划在线","companyLogo":"image1/M00/00/15/CgYXBlTUWEOAUTXzAABQJHYVxSk792.jpg","salary":"20k-35k","industryField":"移动互联网","district":null,"companyLabelList":["国际公司","岗位晋升","扁平管理","美女多"],"positionAdvantage":"平台大 牛人多 技术氛围好 14薪","jobNature":"全职","workYear":"5-10年","approve":1,"score":0,"companySize":"50-150人","companyFullName":"北京酷划在线网络技术有限公司","adWord":0,"formatCreateTime":"12:53发布"},{"businessZones":["五道口","学院路","北航"],"imState":"today","lastLogin":1479181759000,"publisherId":5561461,"explain":null,"plus":null,"pcShow":0,"appShow":0,"deliver":0,"gradeDescription":null,"promotionScoreExplain":null,"firstType":"技术","secondType":"移动开发","positionLables":["iOS"],"createTime":"2016-11-15 12:11:27","companyId":51665,"positionName":"iOS","education":"本科","city":"北京","positionId":2501368,"financeStage":"成长型(A轮)","companyShortName":"乐动力","companyLogo":"image1/M00/3A/9F/Cgo8PFWt-HGAZhiDAAAb-JIwkf4960.jpg","salary":"15k-25k","industryField":"移动互联网","district":"海淀区","companyLabelList":["工程师文化","股票期权","年度旅游","扁平管理"],"positionAdvantage":"千万级用户量，弹性工作制，13薪期权奖金","jobNature":"全职","workYear":"1-3年","approve":1,"score":0,"companySize":"15-50人","companyFullName":"北京朗动科技有限公司","adWord":0,"formatCreateTime":"12:11发布"},{"businessZones":["上地","西二旗","马连洼"],"imState":"today","lastLogin":1479180359000,"publisherId":3616515,"explain":null,"plus":null,"pcShow":0,"appShow":0,"deliver":0,"gradeDescription":null,"promotionScoreExplain":null,"firstType":"技术","secondType":"前端开发/移动开发","positionLables":["iOS"],"createTime":"2016-11-15 11:59:45","companyId":7292,"positionName":"iOS","education":"本科","city":"北京","positionId":1635518,"financeStage":"成长型(B轮)","companyShortName":"畅聊天下(派派交友）","companyLogo":"image1/M00/1A/42/CgYXBlUbifGASvN9AACTHYLzJKc557.jpg","salary":"15k-30k","industryField":"移动互联网","district":"海淀区","companyLabelList":["年底双薪","午餐补助","带薪年假","股票期权"],"positionAdvantage":"期权期权期权！！！","jobNature":"全职","workYear":"3-5年","approve":1,"score":0,"companySize":"150-500人","companyFullName":"北京畅聊天下科技有限公司","adWord":0,"formatCreateTime":"11:59发布"},{"businessZones":["五道口","学院路","北航"],"imState":"today","lastLogin":1479181767000,"publisherId":1544460,"explain":null,"plus":null,"pcShow":0,"appShow":0,"deliver":0,"gradeDescription":null,"promotionScoreExplain":null,"firstType":"技术","secondType":"前端开发/移动开发","positionLables":["iOS"],"createTime":"2016-11-15 11:49:32","companyId":37236,"positionName":"高级IOS开发工程师","education":"本科","city":"北京","positionId":2195782,"financeStage":"成熟型(C轮)","companyShortName":"斗鱼直播","companyLogo":"image1/M00/1E/66/Cgo8PFUuJ7SAcErwAACtNjsgHHk099.jpg","salary":"15k-25k","industryField":"移动互联网","district":"海淀区","companyLabelList":["技能培训","绩效奖金","年度旅游","管理规范"],"positionAdvantage":"优厚的薪酬和绩效激励，全年16-18薪","jobNature":"全职","workYear":"3-5年","approve":1,"score":0,"companySize":"500-2000人","companyFullName":"武汉斗鱼网络科技有限公司","adWord":0,"formatCreateTime":"11:49发布"},{"businessZones":["小营","对外经贸","芍药居"],"imState":"today","lastLogin":1479187254000,"publisherId":633272,"explain":null,"plus":null,"pcShow":0,"appShow":0,"deliver":0,"gradeDescription":null,"promotionScoreExplain":null,"firstType":"开发/测试/运维类","secondType":"前端开发/移动开发","positionLables":null,"createTime":"2016-11-15 11:46:01","companyId":6731,"positionName":"iOS高级开发","education":"本科","city":"北京","positionId":2577834,"financeStage":"成长型(不需要融资)","companyShortName":"有缘网","companyLogo":"i/image/M00/54/7F/CgqKkVfD9yqAJ5QyAAAypAzMizU130.jpg","salary":"15k-20k","industryField":"移动互联网,社交网络","district":"朝阳区","companyLabelList":["绩效奖金","年终分红","股票期权","年底双薪"],"positionAdvantage":"更专注的开发工作","jobNature":"全职","workYear":"3-5年","approve":1,"score":0,"companySize":"150-500人","companyFullName":"北京友缘在线网络科技有限责任公司","adWord":0,"formatCreateTime":"11:46发布"},{"businessZones":["苏州街","万泉河","颐和园"],"imState":"today","lastLogin":1479183814000,"publisherId":7931,"explain":null,"plus":null,"pcShow":0,"appShow":0,"deliver":0,"gradeDescription":null,"promotionScoreExplain":null,"firstType":"技术","secondType":"前端开发/移动开发","positionLables":["iOS"],"createTime":"2016-11-15 11:40:07","companyId":1249,"positionName":"IOS Lead","education":"本科","city":"北京","positionId":2347426,"financeStage":"成熟型(C轮)","companyShortName":"出门问问","companyLogo":"image1/M00/40/3A/CgYXBlXC_4OASRgsAAAislBu0uU808.jpg","salary":"30k-40k","industryField":"移动互联网,O2O","district":"海淀区","companyLabelList":["帅哥多","美女多","绩效奖金","股票期权"],"positionAdvantage":"谷歌技术氛围，行业技术大咖带队","jobNature":"全职","workYear":"5-10年","approve":1,"score":0,"companySize":"150-500人","companyFullName":"北京羽扇智信息科技有限公司","adWord":0,"formatCreateTime":"11:40发布"},{"businessZones":["大山子","酒仙桥","将台路"],"imState":"today","lastLogin":1479179230000,"publisherId":5569772,"explain":null,"plus":null,"pcShow":0,"appShow":0,"deliver":0,"gradeDescription":null,"promotionScoreExplain":null,"firstType":"技术","secondType":"移动开发","positionLables":["iOS"],"createTime":"2016-11-15 11:31:27","companyId":114110,"positionName":"iOS开发实习生","education":"硕士","city":"北京","positionId":2483585,"financeStage":"上市公司","companyShortName":"乐视","companyLogo":"i/image/M00/04/2B/Cgp3O1bGwzWAEAB4AAAbnMQCudk783.jpg","salary":"4k-5100 15675    0 15653  100    22  34630     48 --:--:-- --:--:-- --:--:-- 35819期权","交通补助","通讯津贴","带薪年假"],"positionAdvantage":"转正机会,有竞争力的待遇,期权,节日福利","jobNature":"实习","workYear":"应届毕业生","approve":1,"score":0,"companySize":"2000人以上","companyFullName":"乐视网信息技术（北京）股份有限公司","adWord":0,"formatCreateTime":"11:31发布"},{"businessZones":null,"imState":"today","lastLogin":1479189156000,"publisherId":127369,"explain":null,"plus":null,"pcShow":0,"appShow":0,"deliver":0,"gradeDescription":null,"promotionScoreExplain":null,"firstType":"技术","secondType":"移动开发","positionLables":["iOS"],"createTime":"2016-11-15 11:13:16","companyId":10007,"positionName":"高级iOS开发工程师","education":"本科","city":"北京","positionId":1831196,"financeStage":"成长型(A轮)","companyShortName":"野兽骑行","companyLogo":"image1/M00/11/C9/CgYXBlUBXjuARqvEAAB5Sv6Sr1Q235.jpg","salary":"20k-28k","industryField":"移动互联网,硬件","district":null,"companyLabelList":["年底双薪","午餐补助","定期体检","股票期权"],"positionAdvantage":"股票期权、年底双薪、年终奖、带薪休假","jobNature":"全职","workYear":"3-5年","approve":1,"score":0,"companySize":"50-150人","companyFullName":"北京野兽科技有限公司","adWord":0,"formatCreateTime":"11:13发布"},{"businessZones":null,"imState":"today","lastLogin":1479179248000,"publisherId":7123,"explain":null,"plus":null,"pcShow":0,"appShow":0,"deliver":0,"gradeDescription":null,"promotionScoreExplain":null,"firstType":"技术","secondType":"移动开发","positionLables":["iOS"],"createTime":"2016-11-15 11:07:44","companyId":546,"positionName":"iOS Leader","education":"本科","city":"北京","positionId":1927349,"financeStage":"成熟型(C轮)","companyShortName":"V电影","companyLogo":"image1/M00/00/03/Cgo8PFTUV_uAcfZsAAAxyPH5hXQ445.jpg","salary":"30k-50k","industryField":"移动互联网,文化娱乐","district":null,"companyLabelList":["专项奖金","节日礼物","岗位晋升","领导好"],"positionAdvantage":"提供早餐，加班晚餐，零食饮料、期权","jobNature":"全职","workYear":"5-10年","approve":1,"score":0,"companySize":"150-500人","companyFullName":"北京新片场传媒股份有限公司","adWord":0,"formatCreateTime":"11:07发布"},{"businessZones":["望京","大山子","花家地"],"imState":"today","lastLogin":1479188721000,"publisherId":1467174,"explain":null,"plus":null,"pcShow":0,"appShow":0,"deliver":0,"gradeDescription":null,"promotionScoreExplain":null,"firstType":"技术","secondType":"前端开发/移动开发","positionLables":["iOS","架构师"],"createTime":"2016-11-15 11:05:23","companyId":59251,"positionName":"ios架构师","education":"本科","city":"北京","positionId":2476028,"financeStage":"成长型(B轮)","companyShortName":"映客直播","companyLogo":"i/image/M00/08/49/Cgp3O1bPycGAMj0hAAAgshBCvv4840.jpg","salary":"40k-55k","industryField":"移动互联网,文化娱乐","district":"朝阳区","companyLabelList":["年底双薪","专项奖金","股票期权","带薪年假"],"positionAdvantage":"福利完善,年底奖金,环境优美,帅哥美女","jobNature":"全职","workYear":"3-5年","approve":1,"score":0,"companySize":"500-2000人","companyFullName":"北京蜜莱坞网络科技有限公司","adWord":0,"formatCreateTime":"11:05发布"}]}},"code":0}

如何将数据保存为文件?
将curl的结果，直接保存为文件，才方便进一步处理，方法就是使用重定向符 >，是保存到指定文件 1.json

$ curl 'https://www.lagou.com/jobs/positionAjax.json?px=new&city=%E5%8C%97%E4%BA%AC&needAddtionalResult=false' -H 'Cookie: LGMOID=20161115105219-5AA07DE3A870C9C28DAD87306DE31CFF; JSESSIONID=29F589B05C47BAFBAD36BCEC390793BC; user_trace_token=20161115105219-86a4c72f-aade-11e6-98ea-5254005c3644; PRE_UTM=; PRE_HOST=www.cnblogs.com; PRE_SITE=http%3A%2F%2Fwww.cnblogs.com%2Fios122%2Fp%2F5828673.html; PRE_LAND=https%3A%2F%2Fwww.lagou.com%2Fjobs%2Flist_iOS%3Fpx%3Dnew%26city%3D%25E5%258C%2597%25E4%25BA%25AC; LGUID=20161115105219-86a4cb14-aade-11e6-98ea-5254005c3644; Hm_lvt_4233e74dff0ae5bd0a3d81c6ccf756e6=1479178314,1479178447; Hm_lpvt_4233e74dff0ae5bd0a3d81c6ccf756e6=1479178447; _ga=GA1.2.1088550872.1479178314; LGSID=20161115105219-86a4c8d9-aade-11e6-98ea-5254005c3644; LGRID=20161115105432-d5bbfdde-aade-11e6-a55e-525400f775ce; SEARCH_ID=baf7de67edbc4524b3e39f79042601e9' -H 'Origin: https://www.lagou.com' -H 'X-Anit-Forge-Code: 0' -H 'Accept-Encoding: gzip, deflate, br' -H 'Accept-Language: zh-CN,zh;q=0.8' -H 'User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36' -H 'Content-Type: application/x-www-form-urlencoded; charset=UTF-8' -H 'Accept: application/json, text/javascript, */*; q=0.01' -H 'Cache-Control: max-age=0' -H 'X-Requested-With: XMLHttpRequest' -H 'Connection: keep-alive' -H 'X-Anit-Forge-Token: None' -H 'Referer: https://www.lagou.com/jobs/list_iOS?px=new&city=%E5%8C%97%E4%BA%AC' --data 'first=true&pn=1&kd=iOS' --compressed > 1.json

如何获取其他职位的数据?
需要一个for in 循环来遍历一组给定的职位，动态更改前面curl脚本中的末尾的kd属性的值，并写入职位对应的文件中，注意：末尾 --data后的单引号对，要改成双引导对，否则无法应用变量。完整代码如下,职位数组,可按需自行添加:

$ for ssd in "Java" "PHP" "C" "C++" "Android" "iOS"

do 

curl 'https://www.lagou.com/jobs/positionAjax.json?px=new&city=%E5%8C%97%E4%BA%AC&needAddtionalResult=false' -H 'Cookie: LGMOID=20161115105219-5AA07DE3A870C9C28DAD87306DE31CFF; JSESSIONID=29F589B05C47BAFBAD36BCEC390793BC; user_trace_token=20161115105219-86a4c72f-aade-11e6-98ea-5254005c3644; PRE_UTM=; PRE_HOST=www.cnblogs.com; PRE_SITE=http%3A%2F%2Fwww.cnblogs.com%2Fios122%2Fp%2F5828673.html; PRE_LAND=https%3A%2F%2Fwww.lagou.com%2Fjobs%2Flist_iOS%3Fpx%3Dnew%26city%3D%25E5%258C%2597%25E4%25BA%25AC; LGUID=20161115105219-86a4cb14-aade-11e6-98ea-5254005c3644; Hm_lvt_4233e74dff0ae5bd0a3d81c6ccf756e6=1479178314,1479178447; Hm_lpvt_4233e74dff0ae5bd0a3d81c6ccf756e6=1479178447; _ga=GA1.2.1088550872.1479178314; LGSID=20161115105219-86a4c8d9-aade-11e6-98ea-5254005c3644; LGRID=20161115105432-d5bbfdde-aade-11e6-a55e-525400f775ce; SEARCH_ID=baf7de67edbc4524b3e39f79042601e9' -H 'Origin: https://www.lagou.com' -H 'X-Anit-Forge-Code: 0' -H 'Accept-Encoding: gzip, deflate, br' -H 'Accept-Language: zh-CN,zh;q=0.8' -H 'User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36' -H 'Content-Type: application/x-www-form-urlencoded; charset=UTF-8' -H 'Accept: application/json, text/javascript, */*; q=0.01' -H 'Cache-Control: max-age=0' -H 'X-Requested-With: XMLHttpRequest' -H 'Connection: keep-alive' -H 'X-Anit-Forge-Token: None' -H 'Referer: https://www.lagou.com/jobs/list_iOS?px=new&city=%E5%8C%97%E4%BA%AC' --data "first=true&pn=1&kd=$ssd" --compressed > $ssd.json

done

如何批量获取?
curl 脚本，现在是每次只可以获取单页，要想获取多页，加个for循环就可以了。经过观察，拉勾有效数据大概最多在100页左右，所以写个1~100的循环，并以 $kd_$pn.json 的格式保存:

for (( pn=1; pn<=100; pn=pn+1 )); do

for ssd in "Java" "PHP" "C" "C++" "Android" "iOS"

do 

curl 'https://www.lagou.com/jobs/positionAjax.json?px=new&city=%E5%8C%97%E4%BA%AC&needAddtionalResult=false' -H 'Cookie: LGMOID=20161115105219-5AA07DE3A870C9C28DAD87306DE31CFF; JSESSIONID=29F589B05C47BAFBAD36BCEC390793BC; user_trace_token=20161115105219-86a4c72f-aade-11e6-98ea-5254005c3644; PRE_UTM=; PRE_HOST=www.cnblogs.com; PRE_SITE=http%3A%2F%2Fwww.cnblogs.com%2Fios122%2Fp%2F5828673.html; PRE_LAND=https%3A%2F%2Fwww.lagou.com%2Fjobs%2Flist_iOS%3Fpx%3Dnew%26city%3D%25E5%258C%2597%25E4%25BA%25AC; LGUID=20161115105219-86a4cb14-aade-11e6-98ea-5254005c3644; Hm_lvt_4233e74dff0ae5bd0a3d81c6ccf756e6=1479178314,1479178447; Hm_lpvt_4233e74dff0ae5bd0a3d81c6ccf756e6=1479178447; _ga=GA1.2.1088550872.1479178314; LGSID=20161115105219-86a4c8d9-aade-11e6-98ea-5254005c3644; LGRID=20161115105432-d5bbfdde-aade-11e6-a55e-525400f775ce; SEARCH_ID=baf7de67edbc4524b3e39f79042601e9' -H 'Origin: https://www.lagou.com' -H 'X-Anit-Forge-Code: 0' -H 'Accept-Encoding: gzip, deflate, br' -H 'Accept-Language: zh-CN,zh;q=0.8' -H 'User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36' -H 'Content-Type: application/x-www-form-urlencoded; charset=UTF-8' -H 'Accept: application/json, text/javascript, */*; q=0.01' -H 'Cache-Control: max-age=0' -H 'X-Requested-With: XMLHttpRequest' -H 'Connection: keep-alive' -H 'X-Anit-Forge-Token: None' -H 'Referer: https://www.lagou.com/jobs/list_iOS?px=new&city=%E5%8C%97%E4%BA%AC' --data "first=true&pn=1&kd=$ssd" --compressed > $ssd\_$pn.json

done

done