是否有办法在大文件的nodejs中生成SHA256或类似的?

Question

我正在尝试生成一个巨大缓冲区(2.5G)的sha256，遗憾的是哈希。update已经抛出一个错误(ERR_OUT_OF_RANGE)

RangeError: data is too long
at Hash.update (node:internal/crypto/hash:113:22)

根据我的理解，我不能对createHash('sha256')进行多次更新。https://nodejs.org/api/crypto.html hashupdatedata-inputencoding

是否有另一种方法来处理大缓冲区，除了将它们写入磁盘，然后调用ie sha256sum和处理输出?

复制的简单示例:

const {createHash} = require("crypto")
const hash = createHash('sha256')
const data = new Buffer.alloc(1024 * 1024 * 1024 * 3, 1)
hash.update(data)
console.log(hash.digest('hex'))

Answer 1

Hash类扩展了stream.Transform，文档展示了如何使用Hash和管道流从文件中读取数据，将其通过Hash转换器并将结果(哈希值)写入另一个文件:

import { createReadStream } from 'node:fs';
import { stdout } from 'node:process';
const { createHash } = await import('node:crypto');
const hash = createHash('sha256');
const input = createReadStream('test.js');
input.pipe(hash).setEncoding('hex').pipe(stdout);

这个例子展示了Node.js流的强大功能，可以用来生成一个满足你需求的函数:

const fs = require('fs');
const crypto = require('crypto');
const stream = require('stream/promises');
async function computeHash(filepath) {
const input = fs.createReadStream(filepath);
const hash = crypto.createHash('sha256');

// Connect the output of the `input` stream to the input of `hash`
// and let Node.js do the streaming
await stream.pipeline(input, hash);
return hash.digest('hex');
}

用法:

const hash = await computeHash('/path-to-file');

Answer 2

谢谢@derpirscher!

散列。Update可以根据需要随时调用，散列本身将以调用digest()

结束。现在要处理更大的缓冲区，最简单的方法是将缓冲区分成更小的块，并将它们传递给哈希。

const {createHash} = require("crypto")
const hash = createHash('sha256')
const data = new Buffer.alloc(1024 * 1024 * 1024 * 3, 1)
const chunkSize = 1024 * 1024 * 1024
const chunks = Math.ceil(data.length / chunkSize)
for (let i = 0; i < chunks; i++) {
hash.update(data.subarray(i * chunkSize, (i+1) * chunkSize))
}
console.log(hash.digest('hex'))