如何在java中下载大型文件(大小>50MB)

Question

我正在从远程位置下载文件，较小文件的下载是完整的，大文件(>10 MB)的下载是不完整的。这是我用来从远程服务器下载文件的代码。

    File dstFile = null;
    // check the directory for existence.
    String dstFolder = LOCAL_FILE.substring(0,LOCAL_FILE.lastIndexOf(File.separator));
    if(!(dstFolder.endsWith(File.separator) || dstFolder.endsWith("/")))
        dstFolder += File.separator;
    // Creates the destination folder if doesn't not exists
    dstFile = new File(dstFolder);
    if (!dstFile.exists()) {
        dstFile.mkdirs();
    }
    try {
        URL url = new URL(URL_LOCATION);
        HttpURLConnection connection = (HttpURLConnection) url.openConnection(); 
        connection.addRequestProperty("User-Agent", "Mozilla/4.76"); 
        //URLConnection connection = url.openConnection();
        BufferedInputStream stream = new BufferedInputStream(connection.getInputStream());
        int available = stream.available();
        byte b[]= new byte[available];
        stream.read(b);
        File file = new File(LOCAL_FILE);
        OutputStream out  = new FileOutputStream(file);
        out.write(b);
    } catch (Exception e) {
        System.err.println(e);
        VeBLogger.getInstance().log( e.getMessage());
    }

Answer 1

您可以使用apache commons IO库。很容易。我已经在很多项目中使用了它。

File dstFile = null;
// check the directory for existence.
String dstFolder = LOCAL_FILE.substring(0,LOCAL_FILE.lastIndexOf(File.separator));
if(!(dstFolder.endsWith(File.separator) || dstFolder.endsWith("/")))
    dstFolder += File.separator;
// Creates the destination folder if doesn't not exists
dstFile = new File(dstFolder);
if (!dstFile.exists()) {
    dstFile.mkdirs();
}
try {
    URL url = new URL(URL_LOCATION);
    FileUtils.copyURLToFile(url, dstFile);
} catch (Exception e) {
    System.err.println(e);
    VeBLogger.getInstance().log( e.getMessage());
}

Answer 2

首先，我建议你使用:

FileInputStream in = new FileInputStream(file);  

代替:

BufferedInputStream in = new BufferedInputStream(new FileInputStream(file));

(避免增加内存使用)

try
{
    FileInputStream fileInputStream  = new FileInputStream(file);
    byte[] buf=new byte[8192];
    int bytesread = 0, bytesBuffered = 0;
    while( (bytesread = fileInputStream.read( buf )) > -1 ) {
        out.write( buf, 0, bytesread );
        bytesBuffered += bytesread;
        if (bytesBuffered > 1024 * 1024) { //flush after 1MB
            bytesBuffered = 0;
            out.flush();
        }
    }
}
finally {
    if (out != null) {
        out.flush();
    }
}

Answer 3

请阅读API中BufferedInputStream的方法。available()。

返回已下载的可用字节数。可以在不访问/等待网络的情况下从流中读出的字节数。

你应该创建一个固定大小的字节数组fx。2048字节，并使用read()方法，直到它返回-1。

Answer 4

public String DownloadFile(String path, String outputFileName) {    
        try {
            Connection.Response response = Jsoup.connect(path).cookie()
                .maxBodySize(0)
                .ignoreContentType(true)
                .execute();
            FileOutputStream out = new FileOutputStream(new File(outputFileName));
            out.write(response.bodyAsBytes());
            out.close();
            return outputFileName;
        } catch (Exception e) {
            e.printStackTrace();
            return "";
        }
    }