我在nodejs中使用worker_threads
读取多个文件并处理稍后插入数据库的行(使用oracle-db
)。
大约有100多万行,每个文件有超过10,000行。
当我试图创建"binds"使用forEachAsyncParallel
或.map
中每一行的承诺插入到数据库中,得到以下错误:
误差
RangeError: Value undefined out of range for undefined options property undefined
at Set.add (<anonymous>)
at AsyncHook.init (internal/inspector_async_hook.js:23:25)
at PromiseWrap.emitInitNative (internal/async_hooks.js:182:43)
at /app/services/workers/PushMultiValueFile.worker.js:1:1
at new Promise (<anonymous>)
at /app/services/workers/PushMultiValueFile.worker.js:20:84
at Array.map (<anonymous>)
at /app/services/workers/PushMultiValueFile.worker.js:20:57
at new Promise (<anonymous>)
at AssignKeysNameOnValues (/app/services/workers/PushMultiValueFile.worker.js:16:12)
RangeError: Value undefined out of range for undefined options property undefined
at Set.add (<anonymous>)
at AsyncHook.init (internal/inspector_async_hook.js:23:25)
at PromiseWrap.emitInitNative (internal/async_hooks.js:182:43)
at Promise.then (<anonymous>)
at processTicksAndRejections (internal/process/task_queues.js:97:5)
-----------------------
***Custom code to read error:***
[2021-03-10T19:34:47.331Z]
{
captured_at: 2021-03-10T19:34:47.000Z,
messages: '"Error: Worker stopped with exit code 1"',
stacktrace: '"Error: Error: Worker stopped with exit code 1\n at /app/services/oracle_database.js:434:36\n at processTicksAndRejections (internal/process/task_queues.js:97:5)"',
execution_detail_id: 1203
}
有办法找到解决这个问题吗?
Map with Promises
function AssignKeysNameOnValues(keys, values) {
return new Promise( async (resolve, reject) => {
try {
let result = {};
if (Object.entries(keys).length > 0) {
const promises = (Object.entries(keys)).map( async (item,index) => new Promise( async (resolve, reject) => {
try{
if ([(oracledb.DB_TYPE_NUMBER)].includes(item[1].type)) {
result[item[0]] = values[index] ? parseFloat(values[index]) : null;
resolve()
}
else {
result[item[0]] = values[index];
resolve()
}
} catch (e) {
// throw await logError(new Error(error),(workerData.executionID));
reject();
}
}));
Promise.all(promises)
.then(() => {
resolve(result);
})
.catch((err) => {
reject(err);
});
}
else {
resolve();
}
} catch (err) {
reject(err);
}
});
}
参数接收:
<<p>键/strong>:{
ARRANGEMENT_ID: {
type: 2001,
maxSize: 255,
},
OWNER: {
type: 2001,
maxSize: 255,
},
LINKED_APPL: {
type: 2001,
maxSize: 255,
}
}
:
{
"0": "ARR16@#$%3423C0",
"1": "1",
"2": "42"
}
其他类型迭代相同的值,但我得到相同的错误。
forEachAsyncParallel
Array.prototype.forEachAsyncParallel = async function (fn) {
await Promise.all(this.map(fn));
}
版本:
- 节点:v12.18.4
- Npm: 6.14.6
RangeError: Value undefined out of range for undefined options property undefined
上述错误是Set或Map达到元素的最大限制时引起的。参见参考文献:最大数量的条目在Node.js地图?
如果仍然遇到这个问题,你可以尝试large-set
和large-map
,一个允许你存储和处理大量元素的包,而不用担心在Set或Map中被分割成更小的16.777.216(2^24)的限制,使它能够存储和访问比内置的Set或Map更多的元素。