使用curl命令将一个包含base64编码文件(大小400Mb)的json上传到springREST接口

Question

我正在尝试使用curl将一个包含json的大型base64编码文件(大小：400Mb(上传到Spring REST接口。但我收到一个内存不足错误。我知道包含该文件的json超过了JVM的堆大小。处理这么大的文件，哪种方法比较好？任何示例处理代码都将不胜感激。

Shell命令：

jsonstring='{"uuid":"111","type":"REPORT","userdata":"test defined 
data","time":3000,"wasted":120,"status":"PASS","message":"demo 
message","report":"'"$(base64 file.zip)"'"}'
curl -s --insecure -H "Content-Type: application/json;" --data "@-" 
https://localhost:443/api/v1/upload <<<"$jsonstring"

弹簧支架接口：

@RequestMapping(value = "/api/v1/upload", method = RequestMethod.POST)
public String uploadFile(@RequestBody final UploadedFile uploadedFile, final 
HttpServletRequest request,
final HttpServletResponse response) {
byte[] decoded = 
DatatypeConverter.parseBase64Binary(uploadedFile.getReport());

错误：

Jul 17 10:03:38  bash[29824]: 2018-07-17 10:03:38.686 ERROR 29829 --- [.0-31443-exec-8] o.a.c.c.C.[.[.[/].[dispatcherServlet]    : Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Handler dispatch failed; nested exception is java.lang.OutOfMemoryError: Java heap space] with root cause
Jul 17 10:03:38 chld9004852 bash[29824]: java.lang.OutOfMemoryError: Java heap space
Jul 17 10:03:38 chld9004852 bash[29824]: at java.lang.AbstractStringBuilder.<init>(AbstractStringBuilder.java:68) ~[na:1.8.0_152]
Jul 17 10:03:38 chld9004852 bash[29824]: at java.lang.StringBuilder.<init>(StringBuilder.java:101) ~[na:1.8.0_152]
Jul 17 10:03:38 chld9004852 bash[29824]: at com.fasterxml.jackson.core.util.TextBuffer.contentsAsString(TextBuffer.java:394) ~[jackson-core-2.9.6.jar!/:2.9.6]
Jul 17 10:03:38 chld9004852 bash[29824]: at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._finishAndReturnString(UTF8StreamJsonParser.java:2408) ~[jackson-core-2.9.6.jar!/:2.9.6]
Jul 17 10:03:38 chld9004852 bash[29824]: at com.fasterxml.jackson.core.json.UTF8StreamJsonParser.getText(UTF8StreamJsonParser.java:269) ~[jackson-core-2.9.6.jar!/:2.9.6]
Jul 17 10:03:38 chld9004852 bash[29824]: at com.fasterxml.jackson.databind.deser.std.StringDeserializer.deserialize(StringDeserializer.java:35) ~[jackson-databind-2.9.6.jar!/:2.9.6]
Jul 17 10:03:38 chld9004852 bash[29824]: at com.fasterxml.jackson.databind.deser.std.StringDeserializer.deserialize(StringDeserializer.java:10) ~[jackson-databind-2.9.6.jar!/:2.9.6]
Jul 17 10:03:38 chld9004852 bash[29824]: at com.fasterxml.jackson.databind.deser.impl.MethodProperty.deserializeAndSet(MethodProperty.java:127) ~[jackson-databind-2.9.6.jar!/:2.9.6]
Jul 17 10:03:38 chld9004852 bash[29824]: at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:369) ~[jackson-databind-2.9.6.jar!/:2.9.6]

Answer 1

您的主要问题是Spring使用Jackson映射器将整个事情加载到内存中。您只能通过使用较低级别的Streaming API JSON解析器来避免这种情况——比如XML SAX或StAX vs DOM。

这个想法是让您的控制器期望InputStream而不是映射对象，并手动解析它，一次一个令牌。通过这种方式，您可以只创建处理该请求所需的对象或数据结构，而不是加载整个东西，外加相当多的开销。

当然，另一种选择是增加JVM的可用堆，然后等待下一个巨大的文件使其崩溃。(