MapReduce代码只在本地运行.如何在分布式模式下执行我的代码



几天来,我开发了一个MapReduce代码,用于在Phoenix的HBase中插入值。但是代码仅在本地运行,并且对机器过度收费。无论我进行什么更改,我都会观察到映射。系统地使用 LocalJobRunner 类。

你对这个问题有想法吗?

我将程序的执行日志附加到这篇文章中。

提前感谢您的帮助。 :)

如果有帮助,请随时向我询问更多详细信息。

[adviz@tdr-adviz-cdh-w-01 src]$ hadoop jar testDistributedAggregation.jar 
aggregation.retroactiveAggregation.RetroactiveAggregationApplication MEASURE FIVEMIN
16/05/25 16:29:42 INFO zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x33fb6021 connecting to ZooKeeperensemble=10.66.112.204:2181
16/05/25 16:29:42 INFO zookeeper.ZooKeeper: Client environment:zookeeper.version=3.4.5-cdh5.4.0--1, built on 04/21/2015 19:12 GMT
16/05/25 16:29:42 INFO zookeeper.ZooKeeper: Client environment:host.name=tdr-adviz-cdh-w-01.eolas-services.com
16/05/25 16:29:42 INFO zookeeper.ZooKeeper: Client environment:java.version=1.7.0_67
16/05/25 16:29:42 INFO zookeeper.ZooKeeper: Client environment:java.vendor=Oracle Corporation
16/05/25 16:29:42 INFO zookeeper.ZooKeeper: Client environment:java.home=/usr/java/jdk1.7.0_67-cloudera/jre
16/05/25 16:29:42 INFO zookeeper.ZooKeeper: Client environment:java.class.path=/etc/hadoop/conf:/usr/lib/hadoop/lib/jetty-util-6.1.26.cloudera.4.jar:/usr/lib/hadoop/lib/commons-configuration-1.6.jar
16/05/25 16:29:42 INFO zookeeper.ZooKeeper: Client environment:java.library.path=/usr/lib/hadoop/lib/native
16/05/25 16:29:42 INFO zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp
16/05/25 16:29:42 INFO zookeeper.ZooKeeper: Client environment:java.compiler=<NA>
16/05/25 16:29:42 INFO zookeeper.ZooKeeper: Client environment:os.name=Linux
16/05/25 16:29:42 INFO zookeeper.ZooKeeper: Client environment:os.arch=amd64
16/05/25 16:29:42 INFO zookeeper.ZooKeeper: Client environment:os.version=2.6.32-573.8.1.el6.x86_64
16/05/25 16:29:42 INFO zookeeper.ZooKeeper: Client environment:user.name=adviz
16/05/25 16:29:42 INFO zookeeper.ZooKeeper: Client environment:user.home=/home/adviz
16/05/25 16:29:42 INFO zookeeper.ZooKeeper: Client environment:user.dir=/home/adviz/lucie/modules_test_distributedAggregation/modules/src
16/05/25 16:29:42 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=10.66.112.204:2181 sessionTimeout=90000watcher=hconnection-0x33fb60210x0, quorum=10.66.112.204:2181, baseZNode=/hbase
16/05/25 16:29:42 INFO zookeeper.ClientCnxn: Opening socket connection to server tdr-adviz-cdh-w-01.eolas-services.com/10.66.112.204:2181.Will not attempt to authenticate using SASL (unknown error)
16/05/25 16:29:42 INFO zookeeper.ClientCnxn: Socket connection established to tdr-adviz-cdh-w-01.eolas-services.com/10.66.112.204:2181,initiating session
16/05/25 16:29:42 INFO zookeeper.ClientCnxn: Session establishment complete on server tdr-adviz-cdh-w-01.eolas-services.com/10.66.112.204:2181, sessionid = 0x453ec7535016c89, negotiated timeout = 90000
16/05/25 16:29:43 INFO metrics.Metrics: Initializing metrics system: phoenix
16/05/25 16:29:43 WARN impl.MetricsConfig: Cannot locate configuration: tried hadoop-metrics2-phoenix.properties,hadoop-metrics2.properties
16/05/25 16:29:43 INFO impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s).
16/05/25 16:29:43 INFO impl.MetricsSystemImpl: phoenix metrics system started
16/05/25 16:29:44 INFO Configuration.deprecation: hadoop.native.lib is deprecated. Instead, use io.native.lib.available
16/05/25 16:29:44 INFO zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x2aa8b034 connecting to ZooKeeperensemble=10.66.112.204:2181
16/05/25 16:29:44 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=10.66.112.204:2181 sessionTimeout=90000watcher=hconnection-0x2aa8b0340x0, quorum=10.66.112.204:2181, baseZNode=/hbase
16/05/25 16:29:44 INFO zookeeper.ClientCnxn: Opening socket connection to server tdr-adviz-cdh-w-01.eolas-services.com/10.66.112.204:2181.Will not attempt to authenticate using SASL (unknown error)
16/05/25 16:29:44 INFO zookeeper.ClientCnxn: Socket connection established to tdr-adviz-cdh-w-01.eolas-services.com/10.66.112.204:2181,initiating session
16/05/25 16:29:44 INFO zookeeper.ClientCnxn: Session establishment complete on server tdr-adviz-cdh-w-01.eolas-services.com/10.66.112.204:2181, sessionid = 0x453ec7535016c8a, negotiated timeout = 90000
16/05/25 16:29:45 INFO client.ConnectionManager$HConnectionImplementation: Closing master protocol: MasterService
16/05/25 16:29:45 INFO client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x453ec7535016c8a
16/05/25 16:29:45 INFO zookeeper.ZooKeeper: Session: 0x453ec7535016c8a closed
16/05/25 16:29:45 INFO zookeeper.ClientCnxn: EventThread shut down
16/05/25 16:29:45 INFO zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x3d4e158b connecting to ZooKeeperensemble=10.66.112.204:2181
16/05/25 16:29:45 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=10.66.112.204:2181 sessionTimeout=90000watcher=hconnection-0x3d4e158b0x0, quorum=10.66.112.204:2181, baseZNode=/hbase
16/05/25 16:29:45 INFO zookeeper.ClientCnxn: Opening socket connection to server tdr-adviz-cdh-w-01.eolas-services.com/10.66.112.204:2181.Will not attempt to authenticate using SASL (unknown error)
16/05/25 16:29:45 INFO zookeeper.ClientCnxn: Socket connection established to tdr-adviz-cdh-w-01.eolas-services.com/10.66.112.204:2181,initiating session
16/05/25 16:29:45 INFO zookeeper.ClientCnxn: Session establishment complete on server tdr-adviz-cdh-w-01.eolas-services.com/10.66.112.204:2181, sessionid = 0x453ec7535016c8b, negotiated timeout = 90000
16/05/25 16:29:45 INFO client.ConnectionManager$HConnectionImplementation: Closing master protocol: MasterService
16/05/25 16:29:45 INFO client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x453ec7535016c8b
16/05/25 16:29:45 INFO zookeeper.ZooKeeper: Session: 0x453ec7535016c8b closed
16/05/25 16:29:45 INFO zookeeper.ClientCnxn: EventThread shut down
16/05/25 16:29:45 INFO zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x6aff6bd5 connecting to ZooKeeperensemble=10.66.112.204:2181
16/05/25 16:29:45 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=10.66.112.204:2181 sessionTimeout=90000watcher=hconnection-0x6aff6bd50x0, quorum=10.66.112.204:2181, baseZNode=/hbase
16/05/25 16:29:45 INFO zookeeper.ClientCnxn: Opening socket connection to server tdr-adviz-cdh-w-01.eolas-services.com/10.66.112.204:2181.Will not attempt to authenticate using SASL (unknown error)
16/05/25 16:29:45 INFO zookeeper.ClientCnxn: Socket connection established to tdr-adviz-cdh-w-01.eolas-services.com/10.66.112.204:2181,initiating session
16/05/25 16:29:45 INFO zookeeper.ClientCnxn: Session establishment complete on server tdr-adviz-cdh-w-01.eolas-services.com/10.66.112.204:2181, sessionid = 0x453ec7535016c8c, negotiated timeout = 90000
16/05/25 16:29:45 INFO client.ConnectionManager$HConnectionImplementation: Closing master protocol: MasterService
16/05/25 16:29:45 INFO client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x453ec7535016c8c
16/05/25 16:29:45 INFO zookeeper.ClientCnxn: EventThread shut down
16/05/25 16:29:45 INFO zookeeper.ZooKeeper: Session: 0x453ec7535016c8c closed
16/05/25 16:29:45 INFO zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x4e42b9fd connecting to ZooKeeperensemble=10.66.112.204:2181
16/05/25 16:29:45 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=10.66.112.204:2181 sessionTimeout=90000watcher=hconnection-0x4e42b9fd0x0, quorum=10.66.112.204:2181, baseZNode=/hbase
16/05/25 16:29:45 INFO zookeeper.ClientCnxn: Opening socket connection to server tdr-adviz-cdh-w-01.eolas-services.com/10.66.112.204:2181.Will not attempt to authenticate using SASL (unknown error)
16/05/25 16:29:45 INFO zookeeper.ClientCnxn: Socket connection established to tdr-adviz-cdh-w-01.eolas-services.com/10.66.112.204:2181,initiating session
16/05/25 16:29:45 INFO zookeeper.ClientCnxn: Session establishment complete on server tdr-adviz-cdh-w-01.eolas-services.com/10.66.112.204:2181, sessionid = 0x453ec7535016c8d, negotiated timeout = 90000
16/05/25 16:29:45 INFO client.ConnectionManager$HConnectionImplementation: Closing master protocol: MasterService
16/05/25 16:29:45 INFO client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x453ec7535016c8d
16/05/25 16:29:45 INFO zookeeper.ZooKeeper: Session: 0x453ec7535016c8d closed
16/05/25 16:29:45 INFO zookeeper.ClientCnxn: EventThread shut down
16/05/25 16:29:50 INFO Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id
16/05/25 16:29:50 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=
16/05/25 16:29:50 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
16/05/25 16:29:50 INFO Configuration.deprecation: io.bytes.per.checksum is deprecated. Instead, use dfs.bytes-per-checksum
16/05/25 16:29:50 INFO Configuration.deprecation: io.bytes.per.checksum is deprecated. Instead, use dfs.bytes-per-checksum
16/05/25 16:29:50 INFO zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x4ed8d66f connecting to ZooKeeperensemble=localhost:2181
16/05/25 16:29:50 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=localhost:2181 sessionTimeout=90000watcher=hconnection-0x4ed8d66f0x0, quorum=localhost:2181, baseZNode=/hbase
16/05/25 16:29:50 INFO zookeeper.ClientCnxn: Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
16/05/25 16:29:50 INFO zookeeper.ClientCnxn: Socket connection established to localhost/127.0.0.1:2181, initiating session
16/05/25 16:29:50 INFO zookeeper.ClientCnxn: Session establishment complete on server localhost/127.0.0.1:2181, sessionid =0x453ec7535016c8f, negotiated timeout = 90000
16/05/25 16:29:50 INFO zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x506c1818 connecting to ZooKeeperensemble=localhost:2181
16/05/25 16:29:50 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=localhost:2181 sessionTimeout=90000watcher=hconnection-0x506c18180x0, quorum=localhost:2181, baseZNode=/hbase
16/05/25 16:29:50 INFO zookeeper.ClientCnxn: Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
16/05/25 16:29:50 INFO zookeeper.ClientCnxn: Socket connection established to localhost/127.0.0.1:2181, initiating session
16/05/25 16:29:50 INFO zookeeper.ClientCnxn: Session establishment complete on server localhost/127.0.0.1:2181, sessionid =0x453ec7535016c90, negotiated timeout = 90000
16/05/25 16:29:50 INFO client.ConnectionManager$HConnectionImplementation: Closing master protocol: MasterService
16/05/25 16:29:50 INFO client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x453ec7535016c90
16/05/25 16:29:50 INFO zookeeper.ZooKeeper: Session: 0x453ec7535016c90 closed
16/05/25 16:29:50 INFO zookeeper.ClientCnxn: EventThread shut down
16/05/25 16:29:51 INFO zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x77b18bb3 connecting to ZooKeeperensemble=localhost:2181
16/05/25 16:29:51 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=localhost:2181 sessionTimeout=90000watcher=hconnection-0x77b18bb30x0, quorum=localhost:2181, baseZNode=/hbase
16/05/25 16:29:51 INFO zookeeper.ClientCnxn: Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
16/05/25 16:29:51 INFO zookeeper.ClientCnxn: Socket connection established to localhost/127.0.0.1:2181, initiating session
16/05/25 16:29:51 INFO zookeeper.ClientCnxn: Session establishment complete on server localhost/127.0.0.1:2181, sessionid =0x453ec7535016c91, negotiated timeout = 90000
16/05/25 16:29:51 INFO client.ConnectionManager$HConnectionImplementation: Closing master protocol: MasterService
16/05/25 16:29:51 INFO client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x453ec7535016c91
16/05/25 16:29:51 INFO zookeeper.ZooKeeper: Session: 0x453ec7535016c91 closed
16/05/25 16:29:51 INFO zookeeper.ClientCnxn: EventThread shut down
16/05/25 16:29:51 INFO zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x57d5d80f connecting to ZooKeeperensemble=localhost:2181
16/05/25 16:29:51 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=localhost:2181 sessionTimeout=90000watcher=hconnection-0x57d5d80f0x0, quorum=localhost:2181, baseZNode=/hbase
16/05/25 16:29:51 INFO zookeeper.ClientCnxn: Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
16/05/25 16:29:51 INFO zookeeper.ClientCnxn: Socket connection established to localhost/127.0.0.1:2181, initiating session
16/05/25 16:29:51 INFO zookeeper.ClientCnxn: Session establishment complete on server localhost/127.0.0.1:2181, sessionid =0x453ec7535016c92, negotiated timeout = 90000
16/05/25 16:29:51 INFO client.ConnectionManager$HConnectionImplementation: Closing master protocol: MasterService
16/05/25 16:29:51 INFO client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x453ec7535016c92
16/05/25 16:29:51 INFO zookeeper.ZooKeeper: Session: 0x453ec7535016c92 closed
16/05/25 16:29:51 INFO zookeeper.ClientCnxn: EventThread shut down
16/05/25 16:29:51 INFO zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x3b30baff connecting to ZooKeeperensemble=localhost:2181
16/05/25 16:29:51 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=localhost:2181 sessionTimeout=90000watcher=hconnection-0x3b30baff0x0, quorum=localhost:2181, baseZNode=/hbase
16/05/25 16:29:51 INFO zookeeper.ClientCnxn: Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
16/05/25 16:29:51 INFO zookeeper.ClientCnxn: Socket connection established to localhost/127.0.0.1:2181, initiating session
16/05/25 16:29:51 INFO zookeeper.ClientCnxn: Session establishment complete on server localhost/127.0.0.1:2181, sessionid =0x453ec7535016c93, negotiated timeout = 90000
16/05/25 16:29:51 INFO client.ConnectionManager$HConnectionImplementation: Closing master protocol: MasterService
16/05/25 16:29:51 INFO client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x453ec7535016c93
16/05/25 16:29:51 INFO zookeeper.ZooKeeper: Session: 0x453ec7535016c93 closed
16/05/25 16:29:51 INFO zookeeper.ClientCnxn: EventThread shut down
16/05/25 16:29:52 INFO mapred.JobClient: Running job: job_local573864567_0001
16/05/25 16:29:52 INFO mapred.LocalJobRunner: OutputCommitter set in config null
16/05/25 16:29:52 INFO mapred.LocalJobRunner: OutputCommitter is org.apache.phoenix.mapreduce.PhoenixOutputCommitter
16/05/25 16:29:52 INFO mapred.LocalJobRunner: Waiting for map tasks
16/05/25 16:29:52 INFO mapred.LocalJobRunner: Starting task: attempt_local573864567_0001_m_000000_0
16/05/25 16:29:52 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead
16/05/25 16:29:52 INFO util.ProcessTree: setsid exited with exit code 0
16/05/25 16:29:52 INFO mapred.Task:  Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@1309712b
16/05/25 16:29:52 INFO mapred.MapTask: Processing split: org.apache.phoenix.mapreduce.PhoenixInputSplit@b4eba526
16/05/25 16:29:52 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
16/05/25 16:29:52 INFO mapred.MapTask: io.sort.mb = 100
16/05/25 16:29:52 INFO mapred.MapTask: data buffer = 79691776/99614720
16/05/25 16:29:52 INFO mapred.MapTask: record buffer = 262144/327680
16/05/25 16:29:52 INFO mapred.LocalJobRunner:
16/05/25 16:29:52 INFO mapred.MapTask: Starting flush of map output
16/05/25 16:29:52 INFO mapred.MapTask: Finished spill 0
16/05/25 16:29:52 INFO mapred.Task: Task:attempt_local573864567_0001_m_000000_0 is done. And is in the process of commiting
16/05/25 16:29:52 INFO mapred.LocalJobRunner:
16/05/25 16:29:52 INFO mapred.Task: Task attempt_local573864567_0001_m_000000_0 is allowed to commit now
16/05/25 16:29:52 INFO mapred.LocalJobRunner:
16/05/25 16:29:52 INFO mapred.Task: Task 'attempt_local573864567_0001_m_000000_0' done.
16/05/25 16:29:52 INFO mapred.LocalJobRunner: Finishing task: attempt_local573864567_0001_m_000000_0
16/05/25 16:29:52 INFO mapred.LocalJobRunner: Starting task: attempt_local573864567_0001_m_000001_0
16/05/25 16:29:52 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead
16/05/25 16:29:52 INFO mapred.Task:  Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@7e6abaff
16/05/25 16:29:52 INFO mapred.MapTask: Processing split: org.apache.phoenix.mapreduce.PhoenixInputSplit@a313a8d1
16/05/25 16:29:52 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
16/05/25 16:29:52 INFO mapred.MapTask: io.sort.mb = 100
16/05/25 16:29:52 INFO mapred.MapTask: data buffer = 79691776/99614720
16/05/25 16:29:52 INFO mapred.MapTask: record buffer = 262144/327680
16/05/25 16:29:52 INFO mapred.LocalJobRunner:
16/05/25 16:29:52 INFO mapred.MapTask: Starting flush of map output
16/05/25 16:29:52 INFO mapred.MapTask: Finished spill 0
16/05/25 16:29:52 INFO mapred.Task: Task:attempt_local573864567_0001_m_000001_0 is done. And is in the process of commiting
16/05/25 16:29:52 INFO mapred.LocalJobRunner:
16/05/25 16:29:52 INFO mapred.Task: Task attempt_local573864567_0001_m_000001_0 is allowed to commit now
16/05/25 16:29:52 INFO mapred.LocalJobRunner:
16/05/25 16:29:52 INFO mapred.Task: Task 'attempt_local573864567_0001_m_000001_0' done.
16/05/25 16:29:52 INFO mapred.LocalJobRunner: Finishing task: attempt_local573864567_0001_m_000001_0
16/05/25 16:29:52 INFO mapred.LocalJobRunner: Starting task: attempt_local573864567_0001_m_000002_0
16/05/25 16:29:52 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead
16/05/25 16:29:52 INFO mapred.Task:  Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@4af8c887
16/05/25 16:29:52 INFO mapred.MapTask: Processing split: org.apache.phoenix.mapreduce.PhoenixInputSplit@b3945ee
16/05/25 16:29:52 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
16/05/25 16:29:52 INFO mapred.MapTask: io.sort.mb = 100
16/05/25 16:29:52 INFO mapred.MapTask: data buffer = 79691776/99614720
16/05/25 16:29:52 INFO mapred.MapTask: record buffer = 262144/327680
16/05/25 16:29:52 INFO mapred.LocalJobRunner:
16/05/25 16:29:52 INFO mapred.MapTask: Starting flush of map output
16/05/25 16:29:52 INFO mapred.MapTask: Finished spill 0
16/05/25 16:29:52 INFO mapred.Task: Task:attempt_local573864567_0001_m_000002_0 is done. And is in the process of commiting
16/05/25 16:29:52 INFO mapred.LocalJobRunner:
16/05/25 16:29:52 INFO mapred.Task: Task attempt_local573864567_0001_m_000002_0 is allowed to commit now
16/05/25 16:29:52 INFO mapred.LocalJobRunner:
16/05/25 16:29:52 INFO mapred.Task: Task 'attempt_local573864567_0001_m_000002_0' done.
16/05/25 16:29:52 INFO mapred.LocalJobRunner: Finishing task: attempt_local573864567_0001_m_000002_0
16/05/25 16:29:52 INFO mapred.LocalJobRunner: Map task executor complete.
16/05/25 16:29:52 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead
16/05/25 16:29:52 INFO mapred.Task:  Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@217b10dd
16/05/25 16:29:52 INFO mapred.LocalJobRunner:
16/05/25 16:29:52 INFO mapred.Merger: Merging 3 sorted segments
16/05/25 16:29:52 INFO mapred.Merger: Down to the last merge-pass, with 3 segments left of total size: 18192 bytes
16/05/25 16:29:52 INFO mapred.LocalJobRunner:
16/05/25 16:29:52 INFO mapreduce.PhoenixInputFormat: UseUpsertColumns=true, upsertColumnList.size()=9, upsertColumnList=id,startTimestamp,timestamp,first,last,min,max,avg,nb
16/05/25 16:29:52 INFO mapreduce.PhoenixInputFormat: Phoenix Custom Upsert Statement: UPSERT  INTO "mea_aggr_5minV2" ("id","startTimestamp", "0"."timestamp", "0"."first", "0"."last", "0"."min", "0"."max", "0"."avg", "0"."nb") VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)
16/05/25 16:29:57 INFO mapred.Task: Task:attempt_local573864567_0001_r_000000_0 is done. And is in the process of commiting
16/05/25 16:29:57 INFO mapred.LocalJobRunner:
16/05/25 16:29:57 INFO mapred.Task: Task attempt_local573864567_0001_r_000000_0 is allowed to commit now
16/05/25 16:29:57 INFO mapred.LocalJobRunner: reduce > reduce
16/05/25 16:29:57 INFO mapred.Task: Task 'attempt_local573864567_0001_r_000000_0' done.
16/05/25 16:29:58 INFO mapred.JobClient:  map 0% reduce 100%
16/05/25 16:29:58 INFO mapred.JobClient: Job complete: job_local573864567_0001
16/05/25 16:29:58 INFO mapred.JobClient: Counters: 25
16/05/25 16:29:58 INFO mapred.JobClient:   File System Counters
16/05/25 16:29:58 INFO mapred.JobClient:     FILE: Number of bytes read=222451
16/05/25 16:29:58 INFO mapred.JobClient:     FILE: Number of bytes written=1002250
16/05/25 16:29:58 INFO mapred.JobClient:     FILE: Number of read operations=0
16/05/25 16:29:58 INFO mapred.JobClient:     FILE: Number of large read operations=0
16/05/25 16:29:58 INFO mapred.JobClient:     FILE: Number of write operations=0
16/05/25 16:29:58 INFO mapred.JobClient:     HDFS: Number of bytes read=0
16/05/25 16:29:58 INFO mapred.JobClient:     HDFS: Number of bytes written=0
16/05/25 16:29:58 INFO mapred.JobClient:     HDFS: Number of read operations=0
16/05/25 16:29:58 INFO mapred.JobClient:     HDFS: Number of large read operations=0
16/05/25 16:29:58 INFO mapred.JobClient:     HDFS: Number of write operations=0
16/05/25 16:29:58 INFO mapred.JobClient:   Map-Reduce Framework
16/05/25 16:29:58 INFO mapred.JobClient:     Map input records=433
16/05/25 16:29:58 INFO mapred.JobClient:     Map output records=433
16/05/25 16:29:58 INFO mapred.JobClient:     Map output bytes=17320
16/05/25 16:29:58 INFO mapred.JobClient:     Input split bytes=2024
16/05/25 16:29:58 INFO mapred.JobClient:     Combine input records=0
16/05/25 16:29:58 INFO mapred.JobClient:     Combine output records=0
16/05/25 16:29:58 INFO mapred.JobClient:     Reduce input groups=40
16/05/25 16:29:58 INFO mapred.JobClient:     Reduce shuffle bytes=0
16/05/25 16:29:58 INFO mapred.JobClient:     Reduce input records=433
16/05/25 16:29:58 INFO mapred.JobClient:     Reduce output records=58
16/05/25 16:29:58 INFO mapred.JobClient:     Spilled Records=866
16/05/25 16:29:58 INFO mapred.JobClient:     CPU time spent (ms)=0
16/05/25 16:29:58 INFO mapred.JobClient:     Physical memory (bytes) snapshot=0
16/05/25 16:29:58 INFO mapred.JobClient:     Virtual memory (bytes) snapshot=0
16/05/25 16:29:58 INFO mapred.JobClient:     Total committed heap usage (bytes)=1679818752
16/05/25 16:29:58 INFO client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x453ec7535016c89
16/05/25 16:29:58 INFO zookeeper.ZooKeeper: Session: 0x453ec7535016c89 closed
16/05/25 16:29:58 INFO zookeeper.ClientCnxn: EventThread shut down
16/05/25 16:29:58 INFO client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x453ec7535016c8f
16/05/25 16:29:58 INFO zookeeper.ClientCnxn: EventThread shut down
16/05/25 16:29:58 INFO zookeeper.ZooKeeper: Session: 0x453ec7535016c8f closed

請參閱Steven Magana-Zook的回答

在纱线站点中.xml设置应如下所示

  <property>
     <name>mapreduce.framework.name</name>
     <value>yarn</value>
    </property>

也看看 Job 在 Cloudera 5.1 下的 LocalJobRunner 中继续运行

最新更新