我正在开发一个小的Web内容 scraper 。代码的一部分是发送HTTP请求并从响应标头中获取cookie,因此可以在后续请求中设置。获得cookie的代码是这样的:
HttpClient client = HttpClientBuilder.create().build();
HttpGet request = new HttpGet(url);
request.setHeader("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8");
request.setHeader("Accept-Encoding","gzip,deflate,sdch");
if(cookie!=null)
{
request.setHeader("Cookie", cookie);
}
request.setHeader("Accept-Language","en-US,en;q=0.8,zh-CN;q=0.6");
request.setHeader("Cache-Control", "max-age=0");
request.setHeader("Connetion", "keep-alive");
request.setHeader("Host", "www.booking.com");
request.setHeader("User-Agent", "Mozilla/5.0 (Windows NT 6.1; WOW64)
AppleWebKit/537.36 (KHTML, like Gecko)
Chrome/32.0.1700.76 Safari/537.36");
try {
HttpResponse response = client.execute(request);
int statusCode = response.getStatusLine().getStatusCode();
System.out.println(statusCode);
//get all headers
Header[] headers = response.getAllHeaders();
for (Header header : headers) {
System.out.println("Key : " + header.getName()
+ " ,Value : " + header.getValue());
}
System.out.println("----------------------------------------------------------");
} catch (HttpException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
我用来测试的URL是http://www.booking.com/hotel/il/herods-hotels-pa.html#tab-reviews
印刷的结果就是这样:
200
Key : Server ,Value : nginx
Key : Date ,Value : Mon, 03 Feb 2014 05:15:41 GMT
Key : Content-Type ,Value : text/html; charset=UTF-8
Key : Connection ,Value : keep-alive
Key : Cache-Control ,Value : private
Key : Vary ,Value : User-Agent, Accept-Encoding
Key : Set-Cookie ,Value : bkng=11UmFuZG9tSVYkc2RlIyh9YdMHS7ByVcpJ6zdHwCKMHsY37i1DyVPCutMoSY%2F9OR7ixF74JFUj1%2BJ3pF8ntbVX55kLQJvNnfE6Qco2NDwnHPzomws7z40vIxLRgwBTWU9CTbAN3zZqJGksaPN3GqHpSWJ%2BMIKlI5hQN6ZcJnKsU3rR9KXmRVS4plyPQf4gqmsjR131%2BtuuBiULzmDsKzejJZg%2BFgWWUOWS71bCxUGvJbeBBo1HRmUVmigKDEyHylYplnhKkriMof25dYccWyLQoBjIyUL4QZWr58O5D7fKPHDYWSY9y7k%2Bxfk7irIsyKdu%2B0owjpGp2%2BncNdphtqPZqdpeCyky1ReSjWVQ4QuZemceNGmfZGwxm%2BQxu0%2BkBEsJA5zY%2BoqulR8MJIBKZpFqsuvbeDZ9r5UJzl5c%2Fqk7Vw5YU1I%2FQunbw7PHra7IaGp6%2BmHnH2%2BeyiMDhAjWL769ebuwG2DhrgfB6eI0AGZE%2F6T0uA4j7bxA%2FwUdhog6yOu%2FSeTkPl%2FTAiIetVyKLfT1949ggWKfk1kGzmjnowOlZzPbxr1L%2FAifBjInWZ6DreY1Mr2A3%2BfjFYaHJYnS8VpB%2BZappBpGXBUVfHe%2FQ7lbDwNd6TCCzigpsb17LtvFYsb3JiZ%2BQFF82ILNwWFKz6B1xxEEbCRVoq8N%2FcXXPStyGSwApHZz%2Bew6LNI7Hkd2rjB1w3HenUXprZWR3XiWIWYyhMAbkaFbiQV2LThkl2Dkl%2FA%3D; domain=.booking.com; path=/; expires=Sat, 02-Feb-2019 05:15:41 GMT; HTTPOnly
Key : X-Recruiting ,Value : Like HTTP headers? Come write ours: booking.com/jobs
但是,当我将此小程序上传到服务器并运行时,结果变成了:
200
Key : Server ,Value : nginx
Key : Date ,Value : Mon, 03 Feb 2014 05:14:14 GMT
Key : Content-Type ,Value : text/html; charset=UTF-8
Key : Connection ,Value : keep-alive
Key : Cache-Control ,Value : private
Key : Vary ,Value : User-Agent, Accept-Encoding
Key : X-Recruiting ,Value : Like HTTP headers? Come write ours: booking.com/jobs
set-cookie标题消失了,我随后对同一站点内其他内容页面的请求(我要求的第一页中应该由 javascript
加载),我猜是因为cookie丢失了400个错误。我不知道为什么,以及我所知道的PC和服务器之间的区别:
- 我的PC正在运行Windows 7,实际上具有Chrome浏览器,而服务器正在运行Linux,并且没有任何实际浏览器。
- IP地址不同。除此之外,我还想不到。
将不胜感激解决此问题的任何建议或建议。谢谢。
set-cookie
是禁止响应标头名称,您无法使用浏览器端JavaScript
developer.mozilla.org
浏览器阻止前端javascript代码访问set cookie标头,根据fetch Spec的要求,将set-cookie定义为禁止响应标头的名称,必须从暴露于前端代码的任何响应中过滤出来。