鍍金池/ 問答/Java  Linux  HTML/ httpClient請(qǐng)求得到的頁面和Chrome監(jiān)控到的返回頁面相比,少了一些內(nèi)

httpClient請(qǐng)求得到的頁面和Chrome監(jiān)控到的返回頁面相比,少了一些內(nèi)容?

背景

用httpClient爬取頁面的時(shí)候發(fā)現(xiàn)爬到的頁面和chrome監(jiān)控到的少了幾行。

Chrome監(jiān)控到的:

clipboard.png

我抓到的:

clipboard.png

可以看到下面比上面少了一些內(nèi)容。

問題

  1. 我第一感覺是js生成的,請(qǐng)問chrome監(jiān)控到的頁面是經(jīng)過js腳本加工之后的嗎,不應(yīng)該請(qǐng)求返回的原始頁面嗎?
  2. 我在source中看了所有的js腳本,都沒有查找到相關(guān)的添加元素的腳本代碼。請(qǐng)問這個(gè)問題的可能原因是什么,我這樣找對(duì)嗎?

clipboard.png

補(bǔ)充1

以下是我模擬參數(shù)的代碼

/以下參數(shù)除去了抓包時(shí)看到的undefined的值
                nvps.add(new BasicNameValuePair("businessType", "PCIS"));
                nvps.add(new BasicNameValuePair("businessNo", ePosPaymentReturnVO.getBusinessNo()));
                nvps.add(new BasicNameValuePair("customerName", ePosPaymentReturnVO.getCustomerName()));
                nvps.add(new BasicNameValuePair("currencyNo", ePosPaymentReturnVO.getCurrencyNo()));
                nvps.add(new BasicNameValuePair("amount", ePosPaymentReturnVO.getAmount()));
                nvps.add(new BasicNameValuePair("regionCode", ePosPaymentReturnVO.getRegionCode()));
                nvps.add(new BasicNameValuePair("insuredName", ePosPaymentReturnVO.getInsuredName()));
                nvps.add(new BasicNameValuePair("tellerNo", ePosPaymentReturnVO.getTellerNo()));
                nvps.add(new BasicNameValuePair("branchNo", ePosPaymentReturnVO.getBranchNo()));
                nvps.add(new BasicNameValuePair("documentNo", ""));
                nvps.add(new BasicNameValuePair("callBackInfo", ""));
                nvps.add(new BasicNameValuePair("certPubKey", ePosPaymentReturnVO.getCertPubKey()));
                nvps.add(new BasicNameValuePair("signData", ePosPaymentReturnVO.getSignData()));
                nvps.add(new BasicNameValuePair("applicantName", ePosPaymentReturnVO.getApplicantName()));
                nvps.add(new BasicNameValuePair("applicantCertificateType", ePosPaymentReturnVO.getApplicantCertificateType()));
                nvps.add(new BasicNameValuePair("applicantCertificateNo", ePosPaymentReturnVO.getApplicantCertificateNo()));
                nvps.add(new BasicNameValuePair("ncpInsuredName", ePosPaymentReturnVO.getNcpInsuredName()));
                nvps.add(new BasicNameValuePair("insuredCertificateType", ePosPaymentReturnVO.getInsuredCertificateType()));
                nvps.add(new BasicNameValuePair("insuredCertificateNo", ePosPaymentReturnVO.getInsuredCertificateNo()));
                nvps.add(new BasicNameValuePair("networkFlag", ePosPaymentReturnVO.getNetworkFlag()));
                nvps.add(new BasicNameValuePair("applicantTelephone", ePosPaymentReturnVO.getApplicantTelephone()));
                nvps.add(new BasicNameValuePair("channelSource", ePosPaymentReturnVO.getChannelSource()));
                nvps.add(new BasicNameValuePair("businessTranChnl", ePosPaymentReturnVO.getBusinessTranChnl()));
                nvps.add(new BasicNameValuePair("businessTranCode", ePosPaymentReturnVO.getBusinessTranCode()));
                nvps.add(new BasicNameValuePair("prePayAmount", ""));
                nvps.add(new BasicNameValuePair("isSocialSecurityPay", ePosPaymentReturnVO.getIsSocialSecurityPay()));
                nvps.add(new BasicNameValuePair("customerCertType", ePosPaymentReturnVO.getCustomerCertType()));
                nvps.add(new BasicNameValuePair("customerCertId", ePosPaymentReturnVO.getCustomerCertId()));
                nvps.add(new BasicNameValuePair("userName", ePosPaymentReturnVO.getUserName()));

以下是在chrome中看到的請(qǐng)求參數(shù)

businessType:PCIS
businessNo:12010088000293602413
customerName:(unable to decode value)
currencyNo:RMB
amount:10429
regionCode:000000
insuredName:(unable to decode value)
tellerNo:YWSHP-00001
branchNo:2120971
documentNo:
callBackInfo:
certPubKey:LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUVqRENDQTNTZ0F3SUJBZ0lNTjRBVzdPVDM0
dXVkZDFHRk1BMEdDU3FHU0liM0RRRUJCUVVBTUNzeEN6QUoKQmdOVkJBWVRBa05PTVF3d0NnWURW
UVFLREFOVFJFSXhEakFNQmdOVkJBTU1CVk5FUWtOQk1CNFhEVEUwTURVeApNekF5TlRjd09Gb1hE
VEU1TURVeE1qQXlOVGN3T0Zvd2J6RUxNQWtHQTFVRUJoTUNRMDR4Q3pBSkJnTlZCQWdUCkFsTmFN
UTR3REFZRFZRUUhEQVZNVDBOQlRERU5NQXNHQTFVRUNnd0VVRUZKUXpFVE1CRUdBMVVFQ3d3S1JV
VmYKVTBsSFRrbE9SekVmTUIwR0ExVUVBd3dXWlhCamFYTXRZWFYwYnk1d1lXbGpMbU52YlM1amJq
Q0JuekFOQmdrcQpoa2lHOXcwQkFRRUZBQU9CalFBd2dZa0NnWUVBbWN6WmdnOUx4UTZLN1pvWVhS
b2ZXcEYxLzY3MGttekFWSlZQCjF3WXQzVVZVMXNXTDB0RGhZUFdmZHp3ZzNFbzNVL0toTldDNzFS
dm8xSXRsQWlqUWI1cDZ0Rk1vL0xXZmtxOU8KMHpBS00xM2l3MjNlZytkcjI4RHhLKzZIbXJBNkVa
dHkwKzFkVWhzWUZiMXN2S3dDdUFIWjJ5WWYvU2ZrcDBIVwpwSkZoa3lrQ0F3RUFBYU9DQWU0d2dn
SHFNQThHQTFVZEV3RUJBQVFGTUFNQkFRQXdEZ1lEVlIwUEFRRUFCQVFECkFnRENNQlFHQ1dDR1NB
R0crRUlCQVFFQkFBUUVBd0lBNGpBZ0JnTlZIU1VCQVFBRUZqQVVCZ2dyQmdFRkJRY0QKQVFZSUt3
WUJCUVVIQXdJd0lnWURWUjBqQVFFQUJCZ3dGb0FVN0tZVEM2QkIyTU9DZEtpN2RrTW9IenBjRkJj
dwpnWjhHQ0NzR0FRVUZCd0VCQVFFQUJJR1BNSUdNTUlHSkJnZ3JCZ0VGQlFjd0FvWjliR1JoY0Rv
dkx6RXdMakl1Ck1UQXhMakUxT2pRNE9TOURUajFUUkVKRFFTeERUajFUUkVKRFFTeFBWVDFqUVVO
bGNuUnBabWxqWVhSbGN5eHYKUFhOa1ltTmhMR005WTI0L1kwRkRaWEowYVdacFkyRjBaVDlpWVhO
bFAyOWlhbVZqZEVOc1lYTnpQV05sY25ScApabWxqWVhScGIyNUJkWFJvYjNKcGRIa3dnYVlHQTFV
ZEh3RUJBQVNCbXpDQm1EQ0JsYUNCa3FDQmo0YUJqR3hrCllYQTZMeTh4TUM0eUxqRXdNUzR4TlRv
ME9Ea3ZRMDQ5VTBSQ1EwRXNRMDQ5VTBSQ1EwRXNiM1U5UTFKTVJHbHoKZEhKcFluVjBaVkJ2YVc1
MGN5eHZQWE5rWW1OaExHTTlZMjQvWTJWeWRHbG1hV05oZEdWU1pYWnZZMkYwYVc5dQpUR2x6ZEQ5
aVlYTmxQMjlpYW1WamRHTnNZWE56UFdOU1RFUnBjM1J5YVdKMWRHbHZibEJ2YVc1ME1DQUdBMVVk
CkRnRUJBQVFXQkJTODBieXE5ejF0eEx2cWRLYmJldElOWnkxU2hUQU5CZ2txaGtpRzl3MEJBUVVG
QUFPQ0FRRUEKYjRBSVljRnExRXF0OHhJZ1pleDczdGx6aGFmNEgyTXlyS0Z0dG1ubFRFc3RZbkpi
UEdJQ0x6MitNcU1FV0JYcAp4amw3M0JNYUROR0xvMkIremZrVTJxVlk3eEh4b1dNcUR1Zm9EUVRR
Sm5SU2xHRGFKM3QyMDV2dmZ2NWEramFnCk9KeW9kZ0hCUXhrbEJ5alNqSityaFpRdEV1UGNJVlhI
U2t6MVYzK0hFL3piWE9QUVdkUVVUQWZjU25kMlpIOFYKbmxrTHFBTU0rbWI3ZlJTUDJCelFkVmIv
ZFVqSlZpckhuMGN5SzZ3MUwzVTY0dEhXWm5jdE9NKzdWUGx0V09jaQpwcFF2M21mOEwzUmxmRnVC
dXE3aUdBQlBlMFZyT2IwVXdEczFGSGNRSHN1L21kdW5kenU5YURESnc2UHVpQTVuCnlKbWU2R29D
eWQ2a0tTMzFxRFova3c9PQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0t
signData:J5sgq0DZJj4GK6cVdQe3B0Pt+KuM1mLMUSapFxHq5XrpHbsv1KHWfGikQn9mjez7W9p12cq9QOTS
+5UNkzO/dVoSv92GwN8i115vwIMkpA3Zp5PksOn66obtCOLD7CdZAeB57QtOYvlFqEI4RDZ8oqLk
eYROa0P4f5AKxa1BRu8=
callBackURL:undefined
applicantName:(unable to decode value)
applicantCertificateType:01
applicantCertificateNo:330183199009264314
ncpInsuredName:(unable to decode value)
insuredCertificateType:01
insuredCertificateNo:330183199009264314
prepaidAccountId:undefined
dataSource:33
payType:undefined
prepaidAccountType:undefined
networkFlag:02
applicantTelephone:15057179897
channelSource:G
businessTranChnl:EPO
businessTranCode:000052
prePayAmount:
isContainTax:undefined
vehicleNum:undefined
bargainNum:undefined
departmentName:undefined
operationByName:undefined
inputByName:undefined
undrContractName:undefined
payNo:undefined
payMode:undefined
paBankAccount:undefined
paBankAccountName:undefined
vehicleTaxCountStr:undefined
collectAmountStr:undefined
paymentEndDate:undefined
circPaymentNo:undefined
printFlag:undefined
departmentCode:undefined
isSocialSecurityPay:N
socialSecurityPayInfo:undefined
customerCertType:01
customerCertId:330183199009264314

可以看undefined的參數(shù)是沒有加的。這樣有問題嗎?

請(qǐng)各位大佬相助。

先謝過了。

回答
編輯回答
離觴

ctrl+u 看一下他的源代碼,就是不包含 js 生成的源代碼,

clipboard.png

這個(gè)就是你抓取所獲取的內(nèi)容,應(yīng)該是不包含這部分的:

clipboard.png

chrome監(jiān)控到的頁面是經(jīng)過js腳本加工之后的嗎?

當(dāng)然,沒有經(jīng)過 js 修飾過得,多難用;

我在source中看了所有的js腳本,都沒有查找到相關(guān)的添加元素的腳本代碼。請(qǐng)問這個(gè)問題的可能原因是什么,我這樣找對(duì)嗎?

如果你前面 ctrl+u 確實(shí)證明我說的是對(duì)的話,那這段多出來的代碼就是 js 生成的,你沒有找到很正常呀,那么多代碼你一行行看了,都明白什么意思?不太可能吧

2017年6月28日 04:37
編輯回答
涼心人

一直忘了,現(xiàn)在自答一下,之所以請(qǐng)求頁面不完整。是因?yàn)檎?qǐng)求參數(shù)沒有模擬成功,當(dāng)前post請(qǐng)求的參數(shù),是由上一步的請(qǐng)求得到的數(shù)據(jù)進(jìn)行模擬的,其中有一個(gè)netFlag的參數(shù),在上一步請(qǐng)求的返回參數(shù)中值是"Internet",但是在實(shí)際頁面請(qǐng)求時(shí),在頁面上被轉(zhuǎn)成了對(duì)應(yīng)的code值"02",我在模擬這個(gè)值得時(shí)候直接就是拿了上一步返回參數(shù)里的值,也就是"internet",而實(shí)際上應(yīng)該是要返回"02"。修改了之后就請(qǐng)求成功了。

2017年10月25日 02:00
編輯回答
慢半拍

對(duì)于httpclient而言,是在io層面,數(shù)據(jù)都只是文件,字符。而瀏覽器的作用就是解釋HTML,css,js代碼,他會(huì)執(zhí)行頁面嵌入的腳本,渲染頁面等等。

2017年12月2日 03:53