使用NodeJS下载有速度限制的文件的最佳方式是什么

Question

我正试图找出使用NodeJS的内置HTTPS模块下载有速度限制的文件的最佳方式。(底部有一个完整的python实现，可以实现我想要做的事情。(我已经编写了两个不同的函数，它们似乎都像预期的那样完成了任务。

download1功能，检查当前一秒内是否超过了速度限制，如果超过，则暂停下载并创建一个超时，该超时将在该秒结束时触发，以恢复下载。

然而，download2并没有创建超时，而是创建了一个每1000毫秒触发一次的间隔，并在下载暂停时恢复下载。

我想知道这两种方法中哪一种更好，或者我是否应该采用完全不同的方法。

以下是功能：

export const download1 = (url: string, fileName: string, speedLimitInKb: number) => {
return new Promise((resolve, _reject) => {
https.get(url, res => {
const stream = fs.createWriteStream(fileName);
let totalSize = 0;
let size = 0;
let speedLimit = kbToBytes(speedLimitInKb);
let startDate: number;
let lastSecond = Date.now();
res.pipe(stream);
res.once("resume", () => {
startDate = Date.now();
console.log(`Started at ${new Date(startDate)}`)
})
res.on("data", (chunk) => {
size += chunk.length;
const now = Date.now();
if (now - lastSecond > 1000) {
lastSecond = Date.now();
totalSize += size;
size = 0;
} else if (size >= speedLimit) {
res.pause();
setTimeout(() => res.resume(), 1000 - (now - lastSecond));
}
});
res.on("resume", () => {
lastSecond = Date.now();
totalSize += size;
size = 0;
})
res.on("end", () => {
const elapsed = (Date.now() - startDate) / 1000;
totalSize += size
stream.end();
console.log(`${bytesToMb(totalSize)} mb of data downloaded in ${elapsed} seconds with a speed of ${bytesToKb(totalSize) / elapsed}`)
resolve(undefined);
});
res.on("error", console.log);
})
})
};

export const download2 = (url: string, fileName: string, speedLimitInKb: number) => {
return new Promise((resolve, _reject) => {
https.get(url, res => {
const stream = fs.createWriteStream(fileName);
let totalSize = 0;
let size = 0;
let speedLimit = kbToBytes(speedLimitInKb);
let startDate: number;
res.pipe(stream);
res.once("resume", () => {
startDate = Date.now();
console.log(`Started at ${new Date(startDate)}`)
})
const interval = setInterval(() => {
if (res.isPaused()) {
res.resume();
}
totalSize += size;
size = 0;
}, 1000);
res.on("data", (chunk) => {
size += chunk.length;
if (size >= speedLimit) {
res.pause();
}
});
res.on("end", () => {
clearInterval(interval);
const elapsed = (Date.now() - startDate) / 1000;
totalSize += size
stream.end();
console.log(`${bytesToMb(totalSize)} mb of data downloaded in ${elapsed} seconds with a speed of ${bytesToKb(totalSize) / elapsed}`)
resolve(undefined);
});
res.on("error", console.log);
});
})
}

附加功能：

export const bytesToKb = (bytes: number) => bytes / 1024;
export const kbToMb = (kb: number) => kb / 1024;
export const kbToBytes = (kb: number) => kb * 1024;
export const mbToKb = (mb: number) => mb * 1024;
export const mbToBytes = (mb: number) => mb * 1024 * 1024;
export const bytesToMb = (bytes: number) => bytes / 1024 / 1024;
export const bytesToGb = (bytes: number) => bytes / 1024 / 1024 / 1024;
export const secondsToMs = (seconds: number) => seconds * 1000;
export const msToSeconds = (ms: number) => ms / 1000;

我已经写了一个Python版本的我正在努力实现的目标，这适用于任何速度限制和文件大小。我想弄清楚如何在nodejs:中实现这一点

import requests
import time
def download(url, file_name, speed_limit_in_kb):
start = time.time()
size = 0
total_size = 0
with open(file_name, "wb") as f:
with requests.get(url, stream=True) as res:
last_second = time.time()
for part in res.iter_content(1024):
f.write(part)
total_size += len(part)
size += len(part)
offset = time.time() - last_second
if offset > 1:
size = 0
last_second = time.time()
elif size > (1024 * speed_limit_in_kb):
time.sleep(1 - offset)
size = 0
last_second = time.time()
elapsed = time.time() - start
print(f"{total_size / 1024 / 1024} mb of data downloaded in {elapsed} seconds with a speed of {total_size / 1024 / elapsed}")

Answer 1

这类问题必然会得到固执己见的答案。就我个人而言，我会使用nodejs内置的流功能来进行节流。使用这种方法的观察结果：

• 最小代码
• 该代码依赖于(nodejs(库代码，而不是自定义代码
• 高性能。开销最小，同时写入内存
• CON：对于那些不熟悉流的人来说，代码似乎很复杂

import fs from "fs";
import https from "https";
import stream from "stream";
import util from "util";
async function downloadWithBackpressure(url, filename, byteRate) {
let totalBytesDownloaded = 0;
const timeBeforeStart = Date.now();
await util.promisify(stream.pipeline)(
// Start the download stream
await new Promise(resolve => https.get(url, resolve)),
// Throttle data by combining setTimeout with a stream.Transform
new stream.Transform({
transform: async (chunk, encoding, next) => {
// Accumulate the total number of bytes received
totalBytesDownloaded += chunk.byteLength;
// Sleep to throttle towards desired transfer speed
const sleepMs = Math.max(0, (totalBytesDownloaded / byteRate * 1000) - Date.now() + timeBeforeStart);
sleepMs && await new Promise(resolve => setTimeout(resolve, sleepMs));
// Propagate the chunk to the stream writable
next(null, chunk);
}
}),
// Save the file to disk
fs.createWriteStream(filename)
);
}