我的etl流程出了什么问题?

Question

我正在使用GoodData的CloudConnect(基于CloverETL)读取大量json文件，并将某些元素写入。csv.

不幸的是，我在控制台日志中看到下面粘贴的错误。是由于错误导致内存不足，还是实际错误是内存不足?

---

ERROR [WatchDog_0] - Component [JSONReader:JSONREADER1] finished with status ERROR.
 Java heap space
ERROR [WatchDog_0] - Error details:
org.jetel.exception.JetelRuntimeException: Component [JSONReader:JSONREADER1] finished with status ERROR.
    at org.jetel.graph.Node.createNodeException(Node.java:543)
    at org.jetel.graph.Node.run(Node.java:522)
    at java.lang.Thread.run(Thread.java:744)
Caused by: java.lang.Exception: java.lang.OutOfMemoryError: Java heap space
    at org.jetel.component.TreeReader$StreamConvertingXPathProcessor.checkThrownException(TreeReader.java:766)
    at org.jetel.component.TreeReader$StreamConvertingXPathProcessor.manageThread(TreeReader.java:757)
    at org.jetel.component.TreeReader$StreamConvertingXPathProcessor.processInput(TreeReader.java:732)
    at org.jetel.component.TreeReader.execute(TreeReader.java:412)
    at org.jetel.graph.Node.run(Node.java:493)
    ... 1 more
Caused by: java.lang.OutOfMemoryError: Java heap space
    at net.sf.saxon.tinytree.TinyTree.condense(TinyTree.java:379)
    at net.sf.saxon.tinytree.TinyBuilder.close(TinyBuilder.java:177)
    at net.sf.saxon.event.ReceivingContentHandler.endDocument(ReceivingContentHandler.java:219)
    at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.endDocument(AbstractSAXParser.java:745)
    at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:515)
    at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:848)
    at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:777)
    at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:141)
    at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1213)
    at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:649)
    at net.sf.saxon.event.Sender.sendSAXSource(Sender.java:404)
    at net.sf.saxon.event.Sender.send(Sender.java:193)
    at net.sf.saxon.event.Sender.send(Sender.java:50)
    at net.sf.saxon.Configuration.buildDocument(Configuration.java:2973)
    at net.sf.saxon.sxpath.XPathExpression.evaluate(XPathExpression.java:154)
    at org.jetel.component.tree.reader.xml.XmlXPathEvaluator.iterate(XmlXPathEvaluator.java:79)
    at org.jetel.component.tree.reader.XPathPushParser.handleContext(XPathPushParser.java:104)
    at org.jetel.component.tree.reader.XPathPushParser.parse(XPathPushParser.java:84)
    at org.jetel.component.TreeReader$StreamConvertingXPathProcessor$PipeParser.work(TreeReader.java:827)
    at org.jetel.graph.runtime.CloverWorker.run(CloverWorker.java:87)
    ... 1 more

Answer 1

这看起来像第二种情况:此错误是由任务内存不足引起的。

在评估JSONReader组件(其中一个)时发生错误。

JSON似乎真的很大，如果可能的话，你应该考虑把这个任务分成更小的任务。

您是在本地运行转换还是在gooddata服务器上运行转换?

Answer 2

如果JSONReader -它使用更少的内存，但也读取JSON文件，请尝试使用JSONExtract。

Answer 3

从相应的帮助文档:

JSONReader使用DOM，所以整个输入都存储在内存中，因此组件可以占用内存。

JSONExtract使用SAX而不是DOM，因此它比JSONReader使用更少的内存