通过Jsoup发送Request Payload方式的Ajax post请求

绝地灬酷狼 2023-06-13 06:23 33阅读 0赞

最近在做一个旅游类的网站，在使用爬虫时，发现某程网站上某一个景点的用户评论这部分数据是通过Ajax异步请求动态加载的，首先通过Chrome跟踪异步请求，发现有两次请求，

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dub3NlZA_size_16_color_FFFFFF_t_70][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dub3NlZA_size_16_color_FFFFFF_t_70 1][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dub3NlZA_size_16_color_FFFFFF_t_70 2][]

前一次请求方法的OPTION,后一次请求才是POST而且返回目标数据，不过两次请求的url是一样的，直接在地址栏输入url返回的是没有用户评论信息的json格式的数据，

![20191114183345999.png][]

这说明某程服务器并没有接收到可用的参数，比如页数、页大小等等，因此返回空的json数据。而且这个ur有且只有一个参数，并且不随页数的改变而改变，url始终唯一，那某程服务器如何确定返回哪一页的数据呢？

琢磨很久发现后一次请求的头部比前一次的增加了一个称为Request Payload(请求负载)的数据，里边存放了页数、页大小等数据，以json格式随着post请求发送出去，服务器才返回响应的数据。

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dub3NlZA_size_16_color_FFFFFF_t_70 3][]

至于什么是Request Payload？它就是一种实现Ajax异步请求中存放传输数据的方式之一，另一种是Form Data即以表单提交的方式，携带的数据通过url参数的方式传输，但是这不是安全的，这就是为什么上面提到的不管我们查询哪一页的评论，异步请求的url都没有改变，其实Request Payload存放了页数、页大小等数据的原因。概念参考[此文章][Link 1]。

接下来，只要模拟POST请求，在头部添加Request Payload属性，应该能返回数据。如何模拟？我选择了Jsoup，当然也可以用okhttp。现在的项目是基于webmagic框架来实现，虽然它基于Jsoup但是不提供相应的接口，所有要独立于webmagic使用Jsoup爬取用户评论这部分数据。

首先，查看POST请求头的属性：content-type、cookie、referer、sec-fetch-mode、sec-fetch-site等，

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dub3NlZA_size_16_color_FFFFFF_t_70 4][]

为postConnection添加以上请求头，然后发起请求，实例代码：

String url = "https://sec-m.ctrip.com/restapi/soa2/12530/json/viewCommentList?_fxpcqlniredt=09031094411358401742";
            int pagenum = 10;
            int pagesize = 10;
    
            String jsonBody = "{\"pageid\":10650000804,\"viewid\":2888,\"tagid\":0,\"pagenum\":" + pagenum + ",\"pagesize\":10" + pagesize + ",\"contentType\":\"json\",\"head\":{\"appid\":\"100013776\",\"cid\":\"09031176211061113479\",\"ctok\":\"\",\"cver\":\"1.0\",\"lang\":\"01\",\"sid\":\"8888\",\"syscode\":\"09\",\"auth\":\"\",\"extension\":[{\"name\":\"protocal\",\"value\":\"https\"}]},\"ver\":\"7.10.3.0319180000\"}";
    
            String cookie = "浏览器里缓存的cookie";
    
            Connection postConnect = Jsoup.connect(url)
                    .header("authority", "sec-m.ctrip.com")
                    .method(Connection.Method.POST)
                    .header("path", "/restapi/soa2/12530/json/viewCommentList?_fxpcqlniredt=09031176211061113479")
                    .header("schema", "https")
                    .header("accept", "*/*")
                    .header("content-type", "application/json")
                    .cookie("cookie", cookie)
                    .header("cookieorigin", "https://piao.ctrip.com")
                    .header("dnt","1")
                    .referrer("https://piao.ctrip.com/ticket/dest/t2888.html")
                    .header("sec-fetch-mode","cors")
                    .header("sec-fetch-site","same-site")
                    .userAgent("Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.97 Safari/537.36")
                    .requestBody(jsonBody)
                    .ignoreContentType(true)
                    .validateTLSCertificates(false);
            Connection.Response postResponse = postConnect.execute();
            System.out.println(postResponse.body());

但是返回的数据和上面直接输入地址栏一样总是空的，苦思许久后想到OPTIONS请求肯定不是拿来摆设的，因此在POST之前，模拟了OPTIONS请求，其头属性有：access-control-request-headers，access-control-request-method，sec-fetch-mode，sec-fetch-site等，

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dub3NlZA_size_16_color_FFFFFF_t_70 5][]

OPTIONS请求实例代码：

String jsonBody = "{\"pageid\":10650000804,\"viewid\":2888,\"tagid\":0,\"pagenum\":"+pagenum+",\"pagesize\":10"+pagesize+",\"contentType\":\"json\",\"head\":\"appid\":\"100013776\",\"cid\":\"09031176211061113479\",\"ctok\":\"\",\"cver\":\"1.0\",\"lang\":\"01\",\"sid\":\"8888\",\"syscode\":\"09\",\"auth\":\"\",\"extension\":[{\"name\":\"protocal\",\"value\":\"https\"}]},\"ver\":\"7.10.3.0319180000\"}";
    
            String cookie="浏览器里缓存的cookie";
            Connection optionsConnect = Jsoup.connect(url)
                    .header("authority", "sec-m.ctrip.com")
                    .method(Connection.Method.OPTIONS)
                    .header("path", "/restapi/soa2/12530/json/viewCommentList?_fxpcqlniredt=09031176211061113479")
                    .header("schema", "https")
                    .header("accept", "*/*")
                    .header("access-control-request-headers", "content-type,cookieorigin")
                    .header("access-control-request-method", "POST")
                    .header("dnt","1")
                    .header("origin", "https://piao.ctrip.com/")
                    .referrer("https://piao.ctrip.com/ticket/dest/t2888.html")
                    .header("sec-fetch-mode","cors")
                    .header("sec-fetch-site","same-site")
                    .userAgent("Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.97 Safari/537.36")
                    .ignoreContentType(true)
                    .validateTLSCertificates(false);
            Connection.Response optionsResponse = optionsConnect.execute();
            System.out.println(optionsResponse.body());

先发起OPTION请求，接着发起POST请求，

Connection.Response optionsResponse = optionsConnect.execute();
            Connection.Response postResponse = postConnect.execute();

最终得到相应的Json格式数据，第10页的10条用户评论信息。因此，可以猜测目标服务器的策略是通过请求头的一些属性来识别两次请求是同一次用户点击查询事件， 对于独立的POST请求不予返回数据（这就是我跳的坑），至于是请求头的具体哪一些属性，通过排除法一次次测试应该能确定出来。

{
    	"ResponseStatus": {
    		"Timestamp": "/Date(1573735234501+0800)/",
    		"Ack": "Success",
    		"Errors": [],
    		"Build": null,
    		"Version": null,
    		"Extension": [{
    			"Id": "CLOGGING_TRACE_ID",
    			"Version": null,
    			"ContentType": null,
    			"Value": "5567249818146943341"
    		}, {
    			"Id": "RootMessageId",
    			"Version": null,
    			"ContentType": null,
    			"Value": "921812-0a1c72d3-437148-1836197"
    		}]
    	},
    	"head": {
    		"auth": "",
    		"errcode": 0,
    		"errmsg": ""
    	},
    	"data": {
    		"cmtquantity": 11921,
    		"cmtscore": 4.6,
    		"totalpage": 1194,
    		"recompct": "97%推荐",
    		"stscs": [{
    			"tagid": -21,
    			"count": 4758,
    			"desc": "带图",
    			"moodtype": 0,
    			"location": "tab"
    		}, {
    			"tagid": -12,
    			"count": 367,
    			"desc": "低分",
    			"moodtype": 0,
    			"location": "tab"
    		}, {
    			"tagid": -250,
    			"count": 0,
    			"desc": "最近游玩",
    			"moodtype": 0,
    			"location": "tab"
    		}, {
    			"tagid": -22,
    			"count": 4000,
    			"desc": "来自订单",
    			"moodtype": 0,
    			"location": "tag"
    		}, {
    			"tagid": -24,
    			"count": 15,
    			"desc": "视频",
    			"moodtype": 0,
    			"location": "tag"
    		}, {
    			"tagid": -25,
    			"count": 82,
    			"desc": "来自旅拍",
    			"moodtype": 0,
    			"location": "tag"
    		}],
    		"comments": [{
    			"id": 160683925,
    			"uid": "渊非渊",
    			"userImage": "https://dimg04.c-ctrip.com/images/Z80416000000yvnru88FD_C_180_180.jpg",
    			"title": "",
    			"content": "值得一有，坐船游漓江是一件乐事，从桂林到阳朔，一路的山水都能够领略，九马画山，黄瀑倒影，都值得一看，从书上走下来的世界！
    【景色】",
    			"date": "2019-08-29 22:40",
    			"score": "5",
    			"isselect": false,
    			"simgs": ["https://dimg06.c-ctrip.com/images/100e1700000139ylg34C8_C_160_160_Q60.jpg", "https://dimg02.c-ctrip.com/images/100w170000013d8su396E_C_160_160_Q60.jpg", "https://dimg07.c-ctrip.com/images/100n17000001369ra386D_C_160_160_Q60.jpg"],
    			"bimgs": ["https://dimg06.c-ctrip.com/images/100e1700000139ylg34C8_C_480_640_Q60.jpg", "https://dimg02.c-ctrip.com/images/100w170000013d8su396E_C_640_640_Q60.jpg", "https://dimg07.c-ctrip.com/images/100n17000001369ra386D_C_640_360_Q60.jpg"],
    			"videos": [],
    			"replyeditor": null,
    			"reply": null,
    			"usecount": 0,
    			"sourcetype": 0,
    			"playdate": "",
    			"keywordlist": null,
    			"cmttype": "others",
    			"commentOrderInfo": ""
    		}, {
    			"id": 109842886,
    			"uid": "365happy",
    			"userImage": "https://dimg04.c-ctrip.com/images/Z8010g0000007n8m7D0B4_C_180_180.jpg",
    			"title": "",
    			"content": "&quot;桂林山水甲天下，绝妙漓江泛秋图&quot;。漓江位于广西壮族自治区桂林市，属珠江水系的桂江上游河段，是驰名中外的风景名胜区。&quot;江山惹得游人醉，印入肝肠都是诗&quot;。从桂林至阳朔，漓江水路全程83公里。漓江风光最优美的路段，是启程码头到兴坪之间约40公里的河道上。漓江堪称百里画卷，青峰夹岸，绿水萦洄，峡谷峭壁，悬泉飞瀑，绿洲险滩，奇洞美石，景致万千，画卷随着游船顺流而下逐渐展开，尤以草坪，冠岩，杨堤，兴坪为胜，有浪石奇景，九马画山，黄布倒影等美景，到达阳朔碧莲峰时为佳胜顶点。1999年版人民币20元券背面的风光即取材于兴坪段漓江风光黄布倒影。&quot;江作青罗带，山如碧玉簪&quot;。令人有船在水中游，人在画中游之感。",
    			"date": "2017-06-11 23:08",
    			"score": "5",
    			"isselect": true,
    			"simgs": ["https://dimg02.c-ctrip.com/images/100r0h0000008tj6o1743_C_160_160_Q60.jpg", "https://dimg04.c-ctrip.com/images/100a0h0000008ti5j9968_C_160_160_Q60.jpg", "https://dimg06.c-ctrip.com/images/100u0h0000008thu457D3_C_160_160_Q60.jpg", "https://dimg07.c-ctrip.com/images/100t0h0000008tj0aFECE_C_160_160_Q60.jpg", "https://dimg06.c-ctrip.com/images/100e0h0000008timvBEE0_C_160_160_Q60.jpg", "https://dimg08.c-ctrip.com/images/10010h0000008tjfc831A_C_160_160_Q60.jpg", "https://dimg07.c-ctrip.com/images/100w0h0000008thrkF668_C_160_160_Q60.jpg", "https://dimg07.c-ctrip.com/images/100f0h0000008thb21A90_C_160_160_Q60.jpg", "https://dimg05.c-ctrip.com/images/10020h0000008titj71E8_C_160_160_Q60.jpg", "https://dimg05.c-ctrip.com/images/10010h0000008tjffD37F_C_160_160_Q60.jpg", "https://dimg06.c-ctrip.com/images/100g0h0000008th8b1955_C_160_160_Q60.jpg", "https://dimg08.c-ctrip.com/images/10030h0000008tioh7CF4_C_160_160_Q60.jpg", "https://dimg05.c-ctrip.com/images/10020h0000008titm61C5_C_160_160_Q60.jpg", "https://dimg05.c-ctrip.com/images/100h0h0000008u6nk5CAC_C_160_160_Q60.jpg", "https://dimg03.c-ctrip.com/images/10050h0000008u3j0F612_C_160_160_Q60.jpg"],
    			"bimgs": ["https://dimg02.c-ctrip.com/images/100r0h0000008tj6o1743_C_640_480_Q60.jpg", "https://dimg04.c-ctrip.com/images/100a0h0000008ti5j9968_C_640_480_Q60.jpg", "https://dimg06.c-ctrip.com/images/100u0h0000008thu457D3_C_640_480_Q60.jpg", "https://dimg07.c-ctrip.com/images/100t0h0000008tj0aFECE_C_640_480_Q60.jpg", "https://dimg06.c-ctrip.com/images/100e0h0000008timvBEE0_C_640_480_Q60.jpg", "https://dimg08.c-ctrip.com/images/10010h0000008tjfc831A_C_640_480_Q60.jpg", "https://dimg07.c-ctrip.com/images/100w0h0000008thrkF668_C_640_480_Q60.jpg", "https://dimg07.c-ctrip.com/images/100f0h0000008thb21A90_C_640_480_Q60.jpg", "https://dimg05.c-ctrip.com/images/10020h0000008titj71E8_C_640_480_Q60.jpg", "https://dimg05.c-ctrip.com/images/10010h0000008tjffD37F_C_640_480_Q60.jpg", "https://dimg06.c-ctrip.com/images/100g0h0000008th8b1955_C_480_640_Q60.jpg", "https://dimg08.c-ctrip.com/images/10030h0000008tioh7CF4_C_640_480_Q60.jpg", "https://dimg05.c-ctrip.com/images/10020h0000008titm61C5_C_640_480_Q60.jpg", "https://dimg05.c-ctrip.com/images/100h0h0000008u6nk5CAC_C_640_480_Q60.jpg", "https://dimg03.c-ctrip.com/images/10050h0000008u3j0F612_C_640_480_Q60.jpg"],
    			"videos": [],
    			"replyeditor": null,
    			"reply": null,
    			"usecount": 5,
    			"sourcetype": 0,
    			"playdate": "",
    			"keywordlist": null,
    			"cmttype": "others",
    			"commentOrderInfo": ""
    		}, 
    //省略剩下的8条评论

最后，Ajax异步请求的策略是因网站而异的，比如某程的这个模块的信息是连续发送两次请求来返回数据，对应的JS代码肯定也是对应的逻辑，当然以上是我个人的大胆猜测，小心求证，很多网站防爬虫的策略不是一时半会就能猜出来的，完。

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dub3NlZA_size_16_color_FFFFFF_t_70]: https://img-blog.csdnimg.cn/20191114190413264.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dub3NlZA==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dub3NlZA_size_16_color_FFFFFF_t_70 1]: https://img-blog.csdnimg.cn/20191114190749253.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dub3NlZA==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dub3NlZA_size_16_color_FFFFFF_t_70 2]: https://img-blog.csdnimg.cn/20191114190838351.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dub3NlZA==,size_16,color_FFFFFF,t_70
[20191114183345999.png]: https://img-blog.csdnimg.cn/20191114183345999.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dub3NlZA_size_16_color_FFFFFF_t_70 3]: https://img-blog.csdnimg.cn/20191114192622666.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dub3NlZA==,size_16,color_FFFFFF,t_70
[Link 1]: http://xiaobaoqiu.github.io/blog/2014/09/04/form-data-vs-request-payload/
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dub3NlZA_size_16_color_FFFFFF_t_70 4]: https://img-blog.csdnimg.cn/20191114192243497.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dub3NlZA==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dub3NlZA_size_16_color_FFFFFF_t_70 5]: https://img-blog.csdnimg.cn/20191114192115470.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dub3NlZA==,size_16,color_FFFFFF,t_70