我被难住了，也许我误解了什么，所以我会发布，看看会有什么见解。
我有一个Flask API应用程序(bjoern WSGI(，它使用JWT进行身份验证。我向flask应用程序发送一些凭据和本地URL，以访问将提供图像的API。烧瓶应用程序然后对图像运行一些ML，并用检测数据进行回复。一旦客户端从flask应用程序接收到响应，它还从API请求匹配图像(双重命中(。
我想做的是将flask应用程序中的匹配图像与JSON格式的检测数据一起传回客户端。我已经通过使用请求工具箱和MultiPartEncoder实现了所有这些。我的问题是，当我尝试在客户端使用cv2.imdecode将response.content字节编码为jpeg时。这是相关的代码，我也会发布输出和错误我不想将映像写入磁盘并将其读回numpy数组，我正在尝试在内存中完成所有操作。
这是在客户端上，这一部分是它向flask应用程序发送请求的地方。
do_raise = True
try:
from requests_toolbelt.multipart import decoder
r = requests.post(
url=ml_object_url,
headers=auth_header,
params=params,
# json doesnt send when sending files to mlapi, so we send the file and the json together
json=mlapi_json if not files else None,
files=files,
)
r.raise_for_status()
except ValueError as v_ex:
# pass as we will do a retry loop? -> urllib3 has a retry loop built in but idk if that works here
if v_ex == "BAD_IMAGE":
pass
except requests.exceptions.HTTPError as http_ex:
if http_ex.response.status_code == 400:
if args.get('file'):
g.logger.error(
f"There seems to be an error trying to send an image from zm_detect to mlapi, looking into it"
)
else:
g.logger.error(f"{http_ex.response.json()}")
elif http_ex.response.status_code == 500:
g.logger.error(f"There seems to be an Internal Error with the mlapi host, check mlapi logs!")
else:
g.logger.error(f"ERR CODE={http_ex.response.status_code} {http_ex.response.content=}")
except urllib3.exceptions.NewConnectionError as urllib3_ex:
g.logger.debug(f"{lp} {urllib3_ex.args=} {urllib3_ex.pool=}")
g.logger.error(
f"There seems to be an error while trying to start a new connection to the mlapi host -> {urllib3_ex}")
except requests.exceptions.ConnectionError as req_conn_ex:
g.logger.error(
f"There seems to be an error while trying to start a new connection to the mlapi host -> "
f"{req_conn_ex.response}"
)
except Exception as all_ex:
g.logger.error(
f"{lp} error during post to mlapi host-> {all_ex}"
)
# g.logger.debug(f"traceback-> {format_exc()}")
else:
do_raise = False
data: Optional[dict] = None
multipart_data: decoder.MultipartDecoder = decoder.MultipartDecoder.from_response(r)
part: decoder.MultipartDecoder.from_response
img: Optional[bytes] = None
for part in multipart_data.parts:
# part = part.headers.decode('utf-8')
if part.headers.get(b'Content-Type') == b'image/jpeg':
print(f' got an image')
img = part.content
print(f"{type(img)}")
print(f"{len(img)=}")
elif part.headers.get(b'Content-Type') == b'application/json':
print(f"got json data")
data = part.content.decode('utf-8')
print(data)
np_img = np.asarray(bytearray(img), dtype=np.uint8)
print(f"img after np,asarray(bytearray()) -> {type(np_img) = }")
print(f"{len(np_img)=}")
try:
new_img = cv2.imdecode(img, cv2.IMREAD_UNCHANGED)
except Exception as exc:
print(f"EXCEPTION while cv2.imdecode")
print(exc)
else:
print(f"img after cv2.imdecode -> {type(new_img) = }")
if new_img is not None:
if options.get("resize", 'no') != "no":
new_img = resize_image(img, options.get("resize"))
data["matched_data"]["image"] = new_img
else:
print(f"exiting due to image error")
g.logger.log_close(exit=1)
g.logger.log_close(exit=1)
exit(1)
return data
finally:
if do_raise:
raise ValueError('MLAPI remote detection error!')
以下是flask应用程序中的一些代码，用于处理从API获取图像并对其进行编码以传递到ML模型管道。此代码按预期工作。
r = response
img = np.asarray(bytearray(response.content), dtype="uint8")
img = cv2.imdecode(img, cv2.IMREAD_COLOR) # RGB ?
self.orig_h_w = img.shape[:2]
return img
因此，现在有一个包含img的变量，它被cv2.imdecode解码为jpg。然后，可以将此图像格式(numpy.ndarray(传递给opencv-DNN模块，或者传递给pycoral模型进行TPU推断。这是我想要发送回客户端的图像。这是我用来实现这一点的代码。
img = matched_data['image'].tobytes()
# Remove the numpy.ndarray formatted image from matched_data because it is not JSON serializable
matched_data['image'] = None
# Construct a multipart response that contains the detection data and the image
success = False
if matched_data["frame_id"]:
success = True
resp_json = {
'success': success,
'matched_data': matched_data,
'all_matches': all_matches,
}
from requests_toolbelt import MultipartEncoder
multipart_encoded_data = MultipartEncoder(
fields={
'json': (None, json.dumps(resp_json), 'application/json'),
'image': (f"event-{g.eid}-frame-{matched_data['frame_id']}.jpg", img, 'image/jpeg')
}
)
response = Response(multipart_encoded_data.to_string(), mimetype=multipart_encoded_data.content_type)
if success:
g.logger.info(
f"{lp} returning matched detection -> {matched_data}",
)
g.logger.debug(
f"{lp} returning all detections -> {all_matches}")
else:
g.logger.info(
f"{lp} no detections to return"
)
return response
现在在客户端将JSON和图像分离，并将图像转换为可用的格式->
do_raise = False
data: Optional[dict] = None
multipart_data: decoder.MultipartDecoder = decoder.MultipartDecoder.from_response(r)
part: decoder.MultipartDecoder.from_response
img: Optional[bytes] = None
for part in multipart_data.parts:
# part = part.headers.decode('utf-8')
if part.headers.get(b'Content-Type') == b'image/jpeg':
print(f' got an image')
img = part.content
print(f"{type(img)}")
print(f"{len(img)=}")
elif part.headers.get(b'Content-Type') == b'application/json':
print(f"got json data")
data = part.content.decode('utf-8')
print(data)
np_img = np.asarray(bytearray(img), dtype=np.uint8)
print(f"img after np,asarray(bytearray()) -> {type(np_img) = }")
print(f"{len(np_img)=}")
try:
new_img = cv2.imdecode(img, cv2.IMREAD_UNCHANGED)
except Exception as exc:
print(f"EXCEPTION while cv2.imdecode")
print(exc)
else:
print(f"img after cv2.imdecode -> {type(new_img) = }")
if new_img is not None:
if options.get("resize", 'no') != "no":
new_img = resize_image(img, options.get("resize"))
data["matched_data"]["image"] = new_img
else:
print(f"exiting due to image error")
g.logger.log_close(exit=1)
g.logger.log_close(exit=1)
exit(1)
return data
我收到的错误是静默的，它只返回"无"->
# Grabbing image using an http request and converting into a jpeg
11/07/21 20:44:30.623202 zm_mlapi[37535] DBG1 Media:659 ['std.out' --> image from ZM API as response.content - type(img) = <class 'bytes'> - len(img) = 205125]
11/07/21 20:44:30.627857 zm_mlapi[37535] DBG1 Media:661 ['std.out' --> after np.asarray(bytearray(img), np.uint8) - type(img) = <class 'numpy.ndarray'> - len(img) = 205125]
11/07/21 20:44:30.658582 zm_mlapi[37535] DBG1 Media:663 ['std.out' --> after cv2.imdecode(img, cv2.IMREAD_COLOR) - type(img) = <class 'numpy.ndarray'> - len(img) = 1080]
11/07/21 20:44:30.67595 zm_mlapi[37535] DBG2 pyzm_utils:386 [resize:img: success using resize=800.0 - original dimensions: 1920*1080 - resized dimensions: 450*800]
11/07/21 20:44:30.678568 zm_mlapi[37535] DBG1 Media:681 ['std.out' --> after resize - type(img) = <class 'numpy.ndarray'> - len(img) = 450]
# returned image to the class that requested it (ML Pipeline)
11/07/21 20:44:30.687835 zm_mlapi[37535] DBG1 detect_sequence:1048 ['std.out' --> DETECT STREAM: FRAME RETURNED FROM MEDIA CLASS --> type(frame) = <class 'numpy.ndarray'> - len(frame) = 450]
11/07/21 20:44:33.582062 zm_mlapi[37535] DBG1 detect_sequence:1656 ['std.out' --> before returning matched data - type(matched_data['image']) = <class 'numpy.ndarray'> - len(matched_data['image']) = 450]
# Return image to the flask app, now the flask app has to construct a response with JSON and the image
11/07/21 20:44:33.588139 zm_mlapi[37535] DBG1 mlapi:587 ['std.out' --> type(matched_data['image']) = <class 'numpy.ndarray'> - len(matched_data['image']) = 450]
11/07/21 20:44:33.591981 zm_mlapi[37535] DBG1 mlapi:590 ['std.out' --> before converting using .tobytes() - type(img) = <class 'numpy.ndarray'> - len(img) = 450]
11/07/21 20:44:33.596642 zm_mlapi[37535] DBG1 mlapi:594 ['std.out' --> after converting using .tobytes() - type(img) = <class 'bytes'> - len(img) = 1080000]
11/07/21 20:44:33.611218 zm_mlapi[37535] DBG1 mlapi:611 ['std.out' --> multipart MIME TYPE -> multipart/form-data; boundary=e7f7b825a51d4184ad7f12e7bbc6f411]
# flask app returns the response to the client
11/07/21 21:00:58.393864 zmesdetect_m4[102768] DBG1 zm_detect:418 ['std.out' --> got json data]
11/07/21 21:00:58.395459 zmesdetect_m4[102768] DBG1 zm_detect:414 ['std.out' --> got an image with Content-Type - b'application/octet']
11/07/21 21:00:58.396815 zmesdetect_m4[102768] DBG1 zm_detect:422 ['std.out' --> success = True]
11/07/21 21:00:58.398169 zmesdetect_m4[102768] DBG1 zm_detect:423 ['std.out' --> img - type(img) = <class 'bytes'> - len(img) = 1080000]
11/07/21 21:00:58.39958 zmesdetect_m4[102768] DBG1 zm_detect:424 ['std.out' --> img[:50] = b'\gu\gu\gu]hv^iw_jx`kyalzgrx80kvx84itx82itx82itx82itx82itx82itx82ju']
11/07/21 21:00:58.401012 zmesdetect_m4[102768] DBG1 zm_detect:426 ['std.out' --> img after np.frombuffer(img, dtype=np.uint8) -> type(np_img) = <class 'numpy.ndarray'>]
11/07/21 21:00:58.402911 zmesdetect_m4[102768] DBG1 zm_detect:430 ['std.out' --> img after np_img.copy() -> type(np_img) = <class 'numpy.ndarray'>]
11/07/21 21:00:58.404296 zmesdetect_m4[102768] DBG1 zm_detect:432 ['std.out' --> len(np_img)=1080000]
11/07/21 21:00:58.405619 zmesdetect_m4[102768] DBG1 zm_detect:433 ['std.out' --> attempting to decode numpy array into a jpeg]
11/07/21 21:00:58.407144 zmesdetect_m4[102768] DBG1 zm_detect:442 ['std.out' --> img after cv2.imdecode -> type(new_img) = <class 'NoneType'>]
11/07/21 21:00:58.408474 zmesdetect_m4[102768] DBG1 zm_detect:448 ['std.out' --> exiting due to image error]
任何见解都将不胜感激！我是SO的新手，所以希望这是一个恰当的问题。
根据Reddit@ES Alexander上的一位用户的说法，在使用.tobytes((转换之前，我必须使用cv2.imencode('.jpg'，img(。这是在flask应用程序端构建多部分编码的响应。
img = matched_data['image']
assert isinstance(img, np.ndarray)
succ, img = cv2.imencode('.jpg', img)
# Remove the numpy.ndarray formatted image from matched_data
# because it is not JSON serializable
matched_data['image'] = None
img = img.tobytes()
在客户端->
# img = response.content basically
img = np.frombuffer(img, dtype=np.uint8)
img = cv2.imdecode(img, cv2.IMREAD_UNCHANGED)
日志->
11/08/21 00:34:00.641026 zmesdetect_m4[124600] DBG1 zm_detect:418 ['std.out' --> got json data]
11/08/21 00:34:00.64252 zmesdetect_m4[124600] DBG1 zm_detect:414 ['std.out' --> got an image with Content-Type - b'application/octet']
11/08/21 00:34:00.643865 zmesdetect_m4[124600] DBG1 zm_detect:422 ['std.out' --> img - type(img) = <class 'bytes'> - len(img) = 187057]
11/08/21 00:34:00.645188 zmesdetect_m4[124600] DBG1 zm_detect:423 ['std.out' --> img[:50] = b'xffxd8xffxe0x00x10JFIFx00x01x01x00x00x01x00x01x00x00xffxdbx00Cx00x02x01x01x01x01x01x02x01x01x01x02x02x02x02x0
2x04x03x02x02x02x02x05x04x04x03']
11/08/21 00:34:00.646544 zmesdetect_m4[124600] DBG1 zm_detect:428 ['std.out' --> img after np.asarray(img, dtype=np.uint8) -> type(np_img) = <class 'numpy.ndarray'>]
11/08/21 00:34:00.647876 zmesdetect_m4[124600] DBG1 zm_detect:434 ['std.out' --> len(np_img)=187057]
11/08/21 00:34:00.649185 zmesdetect_m4[124600] DBG1 zm_detect:435 ['std.out' --> attempting to decode numpy array into a jpeg]
11/08/21 00:34:00.657759 zmesdetect_m4[124600] DBG1 zm_detect:444 ['std.out' --> img after cv2.imdecode -> type(new_img) = <class 'numpy.ndarray'>]
11/08/21 00:34:00.659152 zmesdetect_m4[124600] DBG1 zm_detect:447 ['std.out' --> image_shape = (450, 800, 3)]