使用bash, sed, grep或awk从无效JSON中提取数据?

Question

我试图在bash中解析无效的JSON

x="{componentId: 00N5E000005vm9e, componentName: Field, referenceId: 0M05E0000002XbV, referenceName: RecordPageName1, referenceUrl: null, message: Component is in use by another component in your organization., reasonCode: 10}, {componentId: 00N5E000005vm9e, componentName: Field, referenceId: 0M05E0000002XbV, referenceName: RecordPageName1, referenceUrl: null, message: Component is in use by another component in your organization., reasonCode: 10}, {componentId: 00N5E000005vm9e, componentName: Field, referenceId: 0M05E0000002XbV, referenceName: RecordPageName1, referenceUrl: null, message: Component is in use by another component in your organization., reasonCode: 10}, {componentId: 0Rb5E000000BGVi, componentName: Versions, referenceId: 0M05E0000002XbV, referenceName: RecordPageName1, referenceUrl: null, message: Component is in use by another component in your organization., reasonCode: 10}, {componentId: 0Rb5E000000BGVj, componentName: Approves, referenceId: 0M05E0000002XbV, referenceName: RecordPageName1, referenceUrl: null, message: Component is in use by another component in your organization., reasonCode: 10}, {componentId: 0Rb5E000000BGVe, componentName: activityThreads, referenceId: 0M05E0000002XbV, referenceName: RecordPageName1, referenceUrl: null, message: Component is in use by another component in your organization., reasonCode: 10}, {componentId: 0Rb5E000000BGVf, componentName: Attachments, referenceId: 0M05E0000002XbV, referenceName: RecordPageName1, referenceUrl: null, message: Component is in use by another component in your organization., reasonCode: 10}, {componentId: 0Rb5E000000BGVh, componentName: Details, referenceId: 0M05E0000002XbV, referenceName: RecordPageName1, referenceUrl: null, message: Component is in use by another component in your organization., reasonCode: 10}"

使用以下脚本

for each in $(echo $x | sed 's/{componentId: /n/g' ); do
echo "Each: $each"
echo [[ $each == 0Rb* ]]
if [[ $each == 0Rb* ]]; then
component=echo $each | awk -v FS="(componentName: |,|referenceName: |,)" '{print $3}'
reference=echo $each | awk -v FS="(componentName: |,|referenceName: |,)" '{print $6}'
echo "component: $component"
echo "reference: $component"
fi
done

但它不起作用。我不明白为什么它不起作用。当我在控制台中执行这一行时，

echo $x | sed 's/{componentId: /n/g' 

我可以看到这个无效的json被正确地分成几行，但是当我试图将它传递给for循环时，每个变量接收到更小的块到它的值

Each: 00N5E000005vm9e,

我很困惑。

我要做的是提取componentName:和,之间的值，以及referenceName:和,之间的另一个值，当componentId不从00N开始时，从无效的json中提取每个项目。有办法做到这一点吗?

我也试过使用jq -n $x，但它失败了jq: error: syntax error, unexpected IDENT, expecting '}' (Unix shell quoting issues?) at <top-level>, line 1:

Answer 1

将数据作为JSON处理

使用sed将其转换回有效的json，例如:

# Remove redundant space (assuming the text is in the `x` variable)
<<<"$x" sed 's/: /:/g; s/, /,/g' |
# Quote all "words"
sed -E 's/[^"{}:,]+/"&"/g'       |
# Separate objects
sed 's/},{/}n{/g'               |
# Parse json
jq .

输出:

{
"componentId": "00N5E000005vm9e",
"componentName": "Field",
"referenceId": "0M05E0000002XbV",
"referenceName": "RecordPageName1",
"referenceUrl": "null",
"message": "Component is in use by another component in your organization.",
"reasonCode": "10"
}
{
"componentId": "00N5E000005vm9e",
"componentName": "Field",
"referenceId": "0M05E0000002XbV",
"referenceName": "RecordPageName1",
"referenceUrl": "null",
"message": "Component is in use by another component in your organization.",
"reasonCode": "10"
}
{
"componentId": "00N5E000005vm9e",
"componentName": "Field",
"referenceId": "0M05E0000002XbV",
"referenceName": "RecordPageName1",
"referenceUrl": "null",
"message": "Component is in use by another component in your organization.",
"reasonCode": "10"
}
{
"componentId": "0Rb5E000000BGVi",
"componentName": "Versions",
"referenceId": "0M05E0000002XbV",
"referenceName": "RecordPageName1",
"referenceUrl": "null",
"message": "Component is in use by another component in your organization.",
"reasonCode": "10"
}
{
"componentId": "0Rb5E000000BGVj",
"componentName": "Approves",
"referenceId": "0M05E0000002XbV",
"referenceName": "RecordPageName1",
"referenceUrl": "null",
"message": "Component is in use by another component in your organization.",
"reasonCode": "10"
}
{
"componentId": "0Rb5E000000BGVe",
"componentName": "activityThreads",
"referenceId": "0M05E0000002XbV",
"referenceName": "RecordPageName1",
"referenceUrl": "null",
"message": "Component is in use by another component in your organization.",
"reasonCode": "10"
}
{
"componentId": "0Rb5E000000BGVf",
"componentName": "Attachments",
"referenceId": "0M05E0000002XbV",
"referenceName": "RecordPageName1",
"referenceUrl": "null",
"message": "Component is in use by another component in your organization.",
"reasonCode": "10"
}
{
"componentId": "0Rb5E000000BGVh",
"componentName": "Details",
"referenceId": "0M05E0000002XbV",
"referenceName": "RecordPageName1",
"referenceUrl": "null",
"message": "Component is in use by another component in your organization.",
"reasonCode": "10"
}

要遍历componentId和referenceId，可以使用jq的@tsv格式化操作符，例如:

... | jq -r '[ .componentId, .referenceId ] | @tsv'

输出:

00N5E000005vm9e 0M05E0000002XbV
00N5E000005vm9e 0M05E0000002XbV
00N5E000005vm9e 0M05E0000002XbV
0Rb5E000000BGVi 0M05E0000002XbV
0Rb5E000000BGVj 0M05E0000002XbV
0Rb5E000000BGVe 0M05E0000002XbV
0Rb5E000000BGVf 0M05E0000002XbV
0Rb5E000000BGVh 0M05E0000002XbV

将数据视为YAML

如@ lsama所述，您可以使用yq将该字符串解析为YAML数组。在这里是我承担4.13.2迈克法拉的方法使用版本yq:

<<<"[$x]" yq e '.[] | .componentId + " " + .referenceId' -

输出:

00N5E000005vm9e 0M05E0000002XbV
00N5E000005vm9e 0M05E0000002XbV
00N5E000005vm9e 0M05E0000002XbV
0Rb5E000000BGVi 0M05E0000002XbV
0Rb5E000000BGVj 0M05E0000002XbV
0Rb5E000000BGVe 0M05E0000002XbV
0Rb5E000000BGVf 0M05E0000002XbV
0Rb5E000000BGVh 0M05E0000002XbV

解析bash循环中的变量

您可以将上述解决方案的结果管道到while read循环中，例如:

... | while read componentId referenceId; do 
: Do your processing here with $componentId and $referenceId
done

Answer 2

这个输入字符串是YAML对象数组容器的一部分。所以用YAML解析器解析它。

与Python:

#!/usr/bin/env python3
# -*- coding: utf-8 -*-
import sys
import yaml
import json
# Your input invalid JSON but valid YAML elements part of an array
x = "{componentId: 00N5E000005vm9e, componentName: Field, referenceId: 0M05E0000002XbV, referenceName: RecordPageName1, referenceUrl: null, message: Component is in use by another component in your organization., reasonCode: 10}, {componentId: 00N5E000005vm9e, componentName: Field, referenceId: 0M05E0000002XbV, referenceName: RecordPageName1, referenceUrl: null, message: Component is in use by another component in your organization., reasonCode: 10}, {componentId: 00N5E000005vm9e, componentName: Field, referenceId: 0M05E0000002XbV, referenceName: RecordPageName1, referenceUrl: null, message: Component is in use by another component in your organization., reasonCode: 10}, {componentId: 0Rb5E000000BGVi, componentName: Versions, referenceId: 0M05E0000002XbV, referenceName: RecordPageName1, referenceUrl: null, message: Component is in use by another component in your organization., reasonCode: 10}, {componentId: 0Rb5E000000BGVj, componentName: Approves, referenceId: 0M05E0000002XbV, referenceName: RecordPageName1, referenceUrl: null, message: Component is in use by another component in your organization., reasonCode: 10}, {componentId: 0Rb5E000000BGVe, componentName: activityThreads, referenceId: 0M05E0000002XbV, referenceName: RecordPageName1, referenceUrl: null, message: Component is in use by another component in your organization., reasonCode: 10}, {componentId: 0Rb5E000000BGVf, componentName: Attachments, referenceId: 0M05E0000002XbV, referenceName: RecordPageName1, referenceUrl: null, message: Component is in use by another component in your organization., reasonCode: 10}, {componentId: 0Rb5E000000BGVh, componentName: Details, referenceId: 0M05E0000002XbV, referenceName: RecordPageName1, referenceUrl: null, message: Component is in use by another component in your organization., reasonCode: 10}"
# Compose yamlstring from x by adding the missing data array container
yamlstring = "data: [" + x + "]"
# Load data from the yamlstring
data = yaml.load(yamlstring, yaml.SafeLoader)
# Output data as JSON
json.dump(data, sys.stdout, indent=2)

或者在shell中使用yq作为解析器:

#!/usr/bin/env sh
x="{componentId: 00N5E000005vm9e, componentName: Field, referenceId: 0M05E0000002XbV, referenceName: RecordPageName1, referenceUrl: null, message: Component is in use by another component in your organization., reasonCode: 10}, {componentId: 00N5E000005vm9e, componentName: Field, referenceId: 0M05E0000002XbV, referenceName: RecordPageName1, referenceUrl: null, message: Component is in use by another component in your organization., reasonCode: 10}, {componentId: 00N5E000005vm9e, componentName: Field, referenceId: 0M05E0000002XbV, referenceName: RecordPageName1, referenceUrl: null, message: Component is in use by another component in your organization., reasonCode: 10}, {componentId: 0Rb5E000000BGVi, componentName: Versions, referenceId: 0M05E0000002XbV, referenceName: RecordPageName1, referenceUrl: null, message: Component is in use by another component in your organization., reasonCode: 10}, {componentId: 0Rb5E000000BGVj, componentName: Approves, referenceId: 0M05E0000002XbV, referenceName: RecordPageName1, referenceUrl: null, message: Component is in use by another component in your organization., reasonCode: 10}, {componentId: 0Rb5E000000BGVe, componentName: activityThreads, referenceId: 0M05E0000002XbV, referenceName: RecordPageName1, referenceUrl: null, message: Component is in use by another component in your organization., reasonCode: 10}, {componentId: 0Rb5E000000BGVf, componentName: Attachments, referenceId: 0M05E0000002XbV, referenceName: RecordPageName1, referenceUrl: null, message: Component is in use by another component in your organization., reasonCode: 10}, {componentId: 0Rb5E000000BGVh, componentName: Details, referenceId: 0M05E0000002XbV, referenceName: RecordPageName1, referenceUrl: null, message: Component is in use by another component in your organization., reasonCode: 10}"
yamlstring="data: [$x]"
printf %s "$yamlstring" | yq -I 4 -o json e '.' -

Answer 3

谢谢你的评论，看来我已经弄明白了。

echo $x | sed 's/{componentId: /n/g' | while IFS=n read -r each; do
#echo "Each: $each"
#echo [[ $each == 0Rb* ]]
if [[ $each == 0Rb* ]]; then
component=$(echo $each | awk -v FS="(componentName: |,|referenceName: |,)" '{print $3}')
reference=$(echo $each | awk -v FS="(componentName: |,|referenceName: |,)" '{print $6}')
echo "component: $component"
echo "reference: $reference"
fi
done