AWS Sagemaker输出如何读取文件与多个json对象分布在多行

Question

我有一堆json文件像这样

{"vector": [0.017906909808516502, 0.052080217748880386, -0.1460590809583664, ], "word": "blah blah blah"}
{"vector": [0.01027186680585146, 0.04181386157870293, -0.07363887131214142, ], "word": "blah blah blah"}
{"vector": [0.011699287220835686, 0.04741542786359787, -0.07899319380521774, ], "word": "blah blah blah"}

我可以用

读取

f = open(file_name)
data = []
for line in f:
data.append(json.dumps(line))

但是我有另一个文件输出像这样

{
"predictions": [[0.875780046, 0.124219939], [0.892282844, 0.107717164], [0.887681246, 0.112318777]
]
}
{
"predictions": [[0.0, 1.0], [0.0, 1.0], [0.0, 1.0]
]
}
{
"predictions": [[0.391415, 0.608585], [0.992118478, 0.00788147748], [0.0, 1.0]
]
}

。json被格式化为多行，所以我不能简单地一行一行地读取json。有简单的方法来解析这个吗?还是我需要写一些东西将每个json对象一行一行地缝合在一起然后json。加载?

Answer 1

嗯，据我所知，不幸的是没有办法使用json.loads加载JSONL格式的数据。一种选择是提供一个帮助函数，可以将其转换为有效的JSON字符串，如下所示:

import json
string = """
{
"predictions": [[0.875780046, 0.124219939], [0.892282844, 0.107717164], [0.887681246, 0.112318777]
]
}
{
"predictions": [[0.0, 1.0], [0.0, 1.0], [0.0, 1.0]
]
}
{
"predictions": [[0.391415, 0.608585], [0.992118478, 0.00788147748], [0.0, 1.0]
]
}
"""

def json_lines_to_json(s: str) -> str:
# replace the first occurrence of '{'
s = s.replace('{', '[{', 1)
# replace the last occurrence of '}
s = s.rsplit('}', 1)[0] + '}]'
# now go in and replace all occurrences of '}' immediately followed
# by newline with a '},'
s = s.replace('}n', '},n')
return s

print(json.loads(json_lines_to_json(string)))

打印:

[{'predictions': [[0.875780046, 0.124219939], [0.892282844, 0.107717164], [0.887681246, 0.112318777]]}, {'predictions': [[0.0, 1.0], [0.0, 1.0], [0.0, 1.0]]}, {'predictions': [[0.391415, 0.608585], [0.992118478, 0.00788147748], [0.0, 1.0]]}]

注意:你的第一个例子实际上看起来不像有效的JSON(或至少JSON行从我的理解)。特别是，由于最后一个数组元素后面有一个逗号，这部分似乎是无效的:

{"vector": [0.017906909808516502, 0.052080217748880386, -0.1460590809583664, ], ...}

为了确保它在调用helper函数后有效，还需要删除后面的逗号，因此每行都是以下格式:

{"vector": [0.017906909808516502, 0.052080217748880386, -0.1460590809583664 ], ...},

---

似乎也有一个类似的问题，他们建议在换行符上分割并每行调用json.loads;实际上，在每个对象上多次调用json.loads会(稍微)降低性能，而不是在列表上调用一次，如下所示。

from timeit import timeit
import json

string = """
{"vector": [0.017906909808516502, 0.052080217748880386, -0.1460590809583664 ], "word": "blah blah blah"}
{"vector": [0.01027186680585146, 0.04181386157870293, -0.07363887131214142 ], "word": "blah blah blah"}
{"vector": [0.011699287220835686, 0.04741542786359787, -0.07899319380521774 ], "word": "blah blah blah"}
"""

def json_lines_to_json(s: str) -> str:
# Strip newlines from end, then replace all occurrences of '}' followed
# by a newline, by a '},' followed by a newline.
s = s.rstrip('n').replace('}n', '},n')
# return string value wrapped in brackets (list)
return f'[{s}]'

n = 10_000
print('string replace:        ', timeit(r'json.loads(json_lines_to_json(string))', number=n, globals=globals()))
print('json.loads each line:  ', timeit(r'[json.loads(line) for line in string.split("n")]', number=n, globals=globals()))

结果:

string replace:         0.07599360000000001
json.loads each line:   0.1078384