通过Node和SSH2向SFTP服务器写入缓冲区数据

Question

我正在尝试使用SSH2模块实现向节点中的SFTP服务器写入文件。对于我的用例，文件源是Azure Blob Storage，文件相对较大(超过5gb)，因此想法是从Blob Storage中以块的形式捕获数据并将其写入服务器。不希望下载整个文件然后执行写入，因为文件很大，也不希望在运行时出现磁盘空间问题。

我通过使用可用的downloadToBuffer()和write()函数并增加'offset'直到所有字节和写入来实现这一点。如代码片段

所示

sftp.open('remoteFilePath','w', async (openError,handle) => {
if (openError) throw openError;
var blobOffset=0;
try{
while(blobOffset<file.size){
await client.downloadToBuffer(blobOffset, blobOffset+ length > file.size? file.size - blobOffset: length).then((buffer) => {
sftp.write(handle,buffer,0,blobOffset + length > file.size? buffer.length:length, blobOffset, (writeError)=>{if(writeError) throw writeError});
});
blobOffset += length;
}
}
catch(e){
console.log(e);
}
}

这个解决方案可以工作，但是感觉对于大文件不是很有效。是否有更好的方法来实现这一点?也许可以使用流而不必使用循环?

Answer 1

关于此问题，请参考以下代码

var Client = require("ssh2").Client;
var {
BlobServiceClient,
StorageSharedKeyCredential,
} = require("@azure/storage-blob");
var accountName = "andyprivate";
var accountKey =
"";
var creds = new StorageSharedKeyCredential(accountName, accountKey);
var blobServiceClient = new BlobServiceClient(
`https://${accountName}.blob.core.windows.net`,
creds
);
var containerClient = blobServiceClient.getContainerClient("output");
let blob = containerClient.getBlockBlobClient("5m Sales Records.csv");
let read = (await blob.download(0)).readableStreamBody;
var conn = new Client();
conn.connect({
host: "",
port: 22,
username: "",
password: "!",
});
conn.on("ready", async () => {
conn.sftp((err, sftp) => {
if (err) throw err;
var write = sftp.createWriteStream("/home/testqw/test.csv");
read.pipe(write);
write
.on("error", function (error) {
throw error;
})
.on("finish", () => {
console.log("All writes are now complete.");
sftp.end();
});
});
});
conn.on("end", () => {
console.log("close the connection");
});