用python从Intranet下载文件

Question

我想从内部网下载一系列pdf文件。我能够在我的web浏览器中看到文件没有问题，但当试图通过python自动提取文件时，我遇到了问题。通过在我办公室设置的代理对话后，我可以很容易地从互联网下载文件，答案如下:

url = 'http://www.sample.com/fileiwanttodownload.pdf'
user = 'username'
pswd = 'password'
proxy_ip = '12.345.56.78:80'
proxy_url = 'http://' + user + ':' + pswd + '@' + proxy_ip
proxy_support = urllib2.ProxyHandler({"http":proxy_url})
opener = urllib2.build_opener(proxy_support,urllib2.HTTPHandler)
urllib2.install_opener(opener)
file_name = url.split('/')[-1]
u = urllib2.urlopen(url)
f = open(file_name, 'wb')
f.close()

但无论出于何种原因，如果url指向我的内部网上的某些内容，它将不起作用。返回以下错误:

Traceback (most recent call last):
  File "<ipython-input-13-a055d9eaf05e>", line 1, in <module>
    runfile('C:/softwaredev/python/pdfwrite.py', wdir='C:/softwaredev/python')
  File "C:Anacondalibsite-packagesspyderlibwidgetsexternalshellsitecustomize.py", line 585, in runfile
    execfile(filename, namespace)
  File "C:/softwaredev/python/pdfwrite.py", line 26, in <module>
    u = urllib2.urlopen(url)
  File "C:Anacondaliburllib2.py", line 127, in urlopen
    return _opener.open(url, data, timeout)
  File "C:Anacondaliburllib2.py", line 410, in open
    response = meth(req, response)
  File "C:Anacondaliburllib2.py", line 523, in http_response
    'http', request, response, code, msg, hdrs)
  File "C:Anacondaliburllib2.py", line 442, in error
    result = self._call_chain(*args)
  File "C:Anacondaliburllib2.py", line 382, in _call_chain
    result = func(*args)
  File "C:Anacondaliburllib2.py", line 629, in http_error_302
    return self.parent.open(new, timeout=req.timeout)
  File "C:Anacondaliburllib2.py", line 410, in open
    response = meth(req, response)
  File "C:Anacondaliburllib2.py", line 523, in http_response
    'http', request, response, code, msg, hdrs)
  File "C:Anacondaliburllib2.py", line 448, in error
    return self._call_chain(*args)
  File "C:Anacondaliburllib2.py", line 382, in _call_chain
    result = func(*args)
  File "C:Anacondaliburllib2.py", line 531, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
HTTPError: Service Unavailable

使用以下代码中的requests.py，我可以成功地从互联网上拉下文件，但是当试图从我的办公室内部网拉出pdf时，我只是得到一个以html格式发送给我的连接错误。运行以下代码:

import requests
url = 'www.intranet.sample.com/?layout=attachment&cfapp=26&attachmentid=57142'
proxies = {
  "http": "http://12.345.67.89:80",
  "https": "http://12.345.67.89:80"
}
local_filename = 'test.pdf'
r = requests.get(url, proxies=proxies, stream=True)
with open(local_filename, 'wb') as f:
    for chunk in r.iter_content(chunk_size=1024): 
        print chunk
        if chunk:
            f.write(chunk)
            f.flush()

返回的html:

Network Error (tcp_error) 
A communication error occurred: "No route to host"
The Web Server may be down, too busy, or experiencing other problems preventing it from responding to requests. You may wish to try again at a later time.
For assistance, contact your network support team.

是否有可能有一些网络安全设置阻止web浏览器环境外的自动请求?

Answer 1

在urllib2中安装打开器不会影响请求。您需要使用请求自己对代理的支持。将它们在proxies参数中传递给get就足够了，或者您可以设置HTTP_PROXY和HTTPS_PROXY环境变量。看到http://docs.python-requests.org/en/latest/user/advanced/代理

import requests
proxies = {
  "http": "http://10.10.1.10:3128",
  "https": "http://10.10.1.10:1080",
}
requests.get("http://example.org", proxies=proxies)

Answer 2

你有没有试过在内部网上不使用代理下载你的文件?

你可以在python2

中尝试这样做

from urllib2 import urlopen
url = 'http://intranet/myfile.pdf'
with open(local_filename, 'wb') as f:
    f.write(urlopen(url).read())