从 Google 存储下载数据时的 SocketTimeoutException

Question

我在从Google Storage下载数据时收到大量SocketTimeoutException。增加超时的最佳方法是什么？

用例

使用 Map/Reduce 从多个服务器并行下载 24 个文件（总共 10GB）。

源代码

  Storage storage = new Storage.Builder(
                new NetHttpTransport(),
                new JacksonFactory(),
                new GoogleCredential().setAccessToken(accessToken))
            .setApplicationName("FooBar")
            .build();
  Storage.Objects.Get getObject = storage.objects().get(bucket, fn);
  getObject.getMediaHttpDownloader().setDirectDownloadEnabled(true);
  getObject.executeMediaAndDownloadTo(outputStream);

堆栈跟踪

java.net.SocketTimeoutException: Read timed out
    at java.net.SocketInputStream.socketRead0(Native Method)
    at java.net.SocketInputStream.read(SocketInputStream.java:150)
    at java.net.SocketInputStream.read(SocketInputStream.java:121)
    at sun.security.ssl.InputRecord.readFully(InputRecord.java:442)
    at sun.security.ssl.InputRecord.read(InputRecord.java:480)
    at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:927)
    at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:884)
    at sun.security.ssl.AppInputStream.read(AppInputStream.java:102)
    at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
    at java.io.BufferedInputStream.read1(BufferedInputStream.java:275)
    at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
    at sun.net.www.MeteredStream.read(MeteredStream.java:134)
    at java.io.FilterInputStream.read(FilterInputStream.java:133)
    at sun.net.www.protocol.http.HttpURLConnection$HttpInputStream.read(HttpURLConnection.java:3052)
    at sun.net.www.protocol.http.HttpURLConnection$HttpInputStream.read(HttpURLConnection.java:3046)
    at com.google.api.client.util.ByteStreams.copy(ByteStreams.java:51)
    at com.google.api.client.util.IOUtils.copy(IOUtils.java:94)
    at com.google.api.client.util.IOUtils.copy(IOUtils.java:63)
    at com.google.api.client.googleapis.media.MediaHttpDownloader.executeCurrentRequest(MediaHttpDownloader.java:261)
    at com.google.api.client.googleapis.media.MediaHttpDownloader.download(MediaHttpDownloader.java:209)
    at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeMediaAndDownloadTo(AbstractGoogleClientRequest.java:553)
    at com.google.api.services.storage.Storage$Objects$Get.executeMediaAndDownloadTo(Storage.java:4494)

编辑

这只发生在我从日本下载数据时。从美国东部和美国西部一切都很好。

Answer 1

查找一个名为"CommandTimeout"的变量，或者类似性质的变量。通常，这决定了连接在标记错误之前可以持续多长时间。

"连接超时"通常意味着程序可以尝试连接到您的源多长时间，这并不是您真正需要更改的内容，因为连接发生得很快，下载速度很慢。

Answer 2

尝试以下操作...

1. 在 linux 框中设置代理。

2. 创建存储时 - 新的 NetHttpTransport（） 提供了您自己的方法来创建套接字并在传输中添加代理。

   public HttpClient myHttpClient() throws Exception {
       SchemeRegistry schemeRegistry = new SchemeRegistry();
       //SetRegisrty for both HTTP and HTTPS - Check google for this.
       schemeRegistry.register(new Scheme("http", PlainSocketFactory
               .getSocketFactory(), YOUR_PROXY_PORT));
       schemeRegistry.register(new Scheme("https", SSLSocketFactory
               .getSocketFactory(), 443));
       HttpParams params = new BasicHttpParams();
       HttpConnectionParams.setConnectionTimeout(params, 30 * 1000);  // SET the timeout
       HttpProtocolParams.setVersion(params, HttpVersion.HTTP_1_1);
       ClientConnectionManager connManager = new ThreadSafeClientConnManager(
               params, schemeRegistry);
       DefaultHttpClient httpClient = new DefaultHttpClient(connManager,
                   params);
       try {
           int proxyPort = YOUR_PROXY_PORT;
           String proxyHost = "YOUR_PROXT_HOST_NAME";
           if (proxyPort > 0 && proxyHost != null && proxyHost.length() > 0) {
               System.setProperty("https.proxyHost", proxyHost);
               System.setProperty("https.proxyPort", proxyPort + "");
               System.setProperty("http.proxyHost", proxyHost);
               System.setProperty("http.proxyPort", proxyPort + "");
               HttpHost proxy = new HttpHost(proxyHost, proxyPort);
               httpClient.getParams().setParameter(
                       ConnRoutePNames.DEFAULT_PROXY, proxy);
           }
       } catch (NullPointerException e) {
           System.out.println("Proxy error here");
       }
       return httpClient;
   }
   public static HttpTransport myNetHttpTransport()
           throws Exception {
       return new ApacheHttpTransport(myHttpClient());
   }
   

使用.setTransport(myNetHttpTransport())而不是NetHttpTransport()。

我们花了很多时间在这上面。 但到目前为止，这似乎正在起作用。如果对此有任何帮助，请告诉我...