Spark使用newAPIHadoopRDD 直接读取Hbase遇到的问题

阳光穿透心脏的1/2处 2022-05-26 00:12 584阅读 0赞

Spark使用newAPIHadoopRDD  直接读取Hbase遇到的问题  
数据表数据量大概：1.8T  
  
  
JavaPairRDD<ImmutableBytesWritable, Result> hbaseRDD =   
             jsctx.newAPIHadoopRDD(conf, TableInputFormat.class,ImmutableBytesWritable.class, Result.class);  
  
  
  
18/04/20 21:41:18 INFO ZooKeeper: Initiating client connection, connectString=hadoopserver11:2181,hadoopserver02:2181,hadoopserver01:2181,hadoopserver00:2181,hadoopserver12:2181 sessionTimeout=90000 watcher=hconnection-0x9263c540x0, quorum=hadoopserver11:2181,hadoopserver02:2181,hadoopserver01:2181,hadoopserver00:2181,hadoopserver12:2181, baseZNode=/hbase  
18/04/20 21:41:18 INFO ClientCnxn: Opening socket connection to server hadoopserver00/192.168.22.158:2181. Will not attempt to authenticate using SASL (unknown error)  
18/04/20 21:41:18 INFO ClientCnxn: Socket connection established, initiating session, client: /192.168.22.158:56090, server: hadoopserver00/192.168.22.158:2181  
18/04/20 21:41:18 INFO ClientCnxn: Session establishment complete on server hadoopserver00/192.168.22.158:2181, sessionid = 0x462b93f7f494b36, negotiated timeout = 90000  
18/04/20 21:41:18 WARN TableInputFormatBase: initializeTable called multiple times. Overwriting connection and table reference; TableInputFormatBase will not close these old references when done.  
18/04/20 21:41:24 INFO RegionSizeCalculator: Calculating region sizes for table "Traffic\_Link\_Table".  
18/04/20 21:41:25 INFO ConnectionManager$HConnectionImplementation: Closing master protocol: MasterService  
18/04/20 21:41:25 INFO ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x462b93f7f494b36  
18/04/20 21:41:25 INFO ZooKeeper: Session: 0x462b93f7f494b36 closed  
18/04/20 21:41:25 INFO ClientCnxn: EventThread shut down  
  
  
Spark 应用程序读取Hbase 一直处于【上面这种状态】。。。。。。。。最后就出现了下面错误。。。  
Why？？？？？？？？？？？？？？？？？？  
  
  
18/04/20 21:46:44 WARN DFSClient: DFSOutputStream ResponseProcessor exception  for block BP-15450043-192.168.22.158-1464844718994:blk\_1080847780\_7152657  
java.io.EOFException: Premature EOF: no length prefix available  
at org.apache.hadoop.hdfs.protocolPB.PBHelper.vintPrefixed(PBHelper.java:2241)  
at org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:235)  
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer$ResponseProcessor.run(DFSOutputStream.java:971)  
44718994:blk\_1080847780\_7152657  
java.io.EOFException: Premature EOF: no length prefix available  
at org.apache.hadoop.hdfs.protocolPB.PBHelper.vintPrefixed(PBHelper.java:2241)  
at org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:235)  
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer$ResponseProcessor.run(DFSOutputStream.java:971)  
Exception in thread "qtp2146613458-156" java.lang.OutOfMemoryError: Java heap space  
18/04/20 21:46:44 WARN DFSClient: DataStreamer Exception  
java.io.IOException: 断开的管道  
at sun.nio.ch.FileDispatcherImpl.write0(Native Method)  
at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)  
at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)  
at sun.nio.ch.IOUtil.write(IOUtil.java:65)  
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:471)  
at org.apache.hadoop.net.SocketOutputStream$Writer.performIO(SocketOutputStream.java:63)  
at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142)  
at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:159)  
at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:117)  
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)  
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)  
at java.io.DataOutputStream.flush(DataOutputStream.java:123)  
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:736)  
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space  
at scala.collection.mutable.ResizableArray$class.$init$(ResizableArray.scala:32)  
at scala.collection.mutable.ArrayBuffer.<init>(ArrayBuffer.scala:47)  
at scala.collection.mutable.ArrayBuffer.<init>(ArrayBuffer.scala:62)  
at scala.collection.mutable.ArrayBuffer$.newBuilder(ArrayBuffer.scala:189)  
at scala.collection.generic.GenericCompanion.empty(GenericCompanion.scala:37)  
at scala.collection.generic.GenericCompanion.apply(GenericCompanion.scala:45)  
at org.apache.spark.rdd.PartitionGroup.<init>(CoalescedRDD.scala:348)  
at org.apache.spark.rdd.PartitionCoalescer$$anonfun$setupGroups$1.apply(CoalescedRDD.scala:244)  
at org.apache.spark.rdd.PartitionCoalescer$$anonfun$setupGroups$1.apply(CoalescedRDD.scala:244)  
at scala.collection.immutable.Range.foreach(Range.scala:141)  
at org.apache.spark.rdd.PartitionCoalescer.setupGroups(CoalescedRDD.scala:244)  
at org.apache.spark.rdd.PartitionCoalescer.run(CoalescedRDD.scala:341)  
at org.apache.spark.rdd.CoalescedRDD.getPartitions(CoalescedRDD.scala:87)  
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:239)  
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:237)  
at scala.Option.getOrElse(Option.scala:120)  
at org.apache.spark.rdd.RDD.partitions(RDD.scala:237)  
at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:35)  
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:239)  
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:237)  
at scala.Option.getOrElse(Option.scala:120)  
at org.apache.spark.rdd.RDD.partitions(RDD.scala:237)  
at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:35)  
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:239)  
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:237)  
at scala.Option.getOrElse(Option.scala:120)  
at org.apache.spark.rdd.RDD.partitions(RDD.scala:237)  
at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:35)  
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:239)  
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:237)  
at scala.Option.getOrElse(Option.scala:120)  
at org.apache.spark.rdd.RDD.partitions(RDD.scala:237)  
18/04/20 21:57:44 WARN AbstractNioSelector: Unexpected exception in the selector loop.  
java.lang.OutOfMemoryError: Java heap space  
18/04/20 21:57:44 ERROR ErrorMonitor: Uncaught fatal error from thread \[sparkDriver-akka.actor.default-dispatcher-28\] shutting down ActorSystem \[sparkDriver\]  
java.lang.OutOfMemoryError: Java heap space  
at akka.util.CompactByteString$.apply(ByteString.scala:438)  
at akka.util.ByteString$.apply(ByteString.scala:38)  
at akka.remote.transport.AkkaPduProtobufCodec$.decodePdu(AkkaPduCodec.scala:169)  
at akka.remote.transport.ProtocolStateActor.akka$remote$transport$ProtocolStateActor$$decodePdu(AkkaProtocolTransport.scala:513)  
at akka.remote.transport.ProtocolStateActor$$anonfun$5.applyOrElse(AkkaProtocolTransport.scala:357)  
at akka.remote.transport.ProtocolStateActor$$anonfun$5.applyOrElse(AkkaProtocolTransport.scala:352)  
at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:33)  
at akka.actor.FSM$class.processEvent(FSM.scala:595)  
at akka.remote.transport.ProtocolStateActor.processEvent(AkkaProtocolTransport.scala:220)  
at akka.actor.FSM$class.akka$actor$FSM$$processMsg(FSM.scala:589)  
at akka.actor.FSM$$anonfun$receive$1.applyOrElse(FSM.scala:583)  
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:498)  
at akka.actor.ActorCell.invoke(ActorCell.scala:456)  
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:237)  
at akka.dispatch.Mailbox.run(Mailbox.scala:219)  
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386)  
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)  
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)  
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)  
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)  
18/04/20 21:57:44 INFO RetryInvocationHandler: Exception while invoking renewLease of class ClientNamenodeProtocolTranslatorPB over hadoopserver02/192.168.22.150:8020. Trying to fail over immediately.  
java.io.IOException: Failed on local exception: java.io.IOException: Couldn't set up IO streams; Host Details : local host is: "hadoopserver00/192.168.22.158"; destination host is: "hadoopserver02":8020;   
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:772)  
at org.apache.hadoop.ipc.Client.call(Client.java:1476)  
at org.apache.hadoop.ipc.Client.call(Client.java:1403)  
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230)  
at com.sun.proxy.$Proxy15.renewLease(Unknown Source)  
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.renewLease(ClientNamenodeProtocolTranslatorPB.java:571)  
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)  
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)  
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)  
at java.lang.reflect.Method.invoke(Method.java:497)  
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:252)  
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:104)  
at com.sun.proxy.$Proxy16.renewLease(Unknown Source)  
at org.apache.hadoop.hdfs.DFSClient.renewLease(DFSClient.java:921)  
at org.apache.hadoop.hdfs.LeaseRenewer.renew(LeaseRenewer.java:423)  
at org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:448)  
at org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71)  
at org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:304)  
at java.lang.Thread.run(Thread.java:745)  
Caused by: java.io.IOException: Couldn't set up IO streams  
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:790)  
at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:370)  
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1525)  
at org.apache.hadoop.ipc.Client.call(Client.java:1442)  
... 17 more  
Caused by: java.lang.OutOfMemoryError: Java heap space  
at java.io.BufferedOutputStream.<init>(BufferedOutputStream.java:76)  
at java.io.BufferedOutputStream.<init>(BufferedOutputStream.java:59)  
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:767)  
... 20 more  
18/04/20 21:57:44 WARN DefaultChannelPipeline: An exception was thrown by a user handler while handling an exception event (\[id: 0x1e529e4d, /192.168.22.162:51400 => /192.168.22.158:37407\] EXCEPTION: java.lang.OutOfMemoryError: Java heap space)  
java.lang.OutOfMemoryError: Java heap space  
at org.jboss.netty.buffer.HeapChannelBuffer.<init>(HeapChannelBuffer.java:42)  
at org.jboss.netty.buffer.BigEndianHeapChannelBuffer.<init>(BigEndianHeapChannelBuffer.java:34)  
at org.jboss.netty.buffer.ChannelBuffers.buffer(ChannelBuffers.java:134)  
at org.jboss.netty.buffer.HeapChannelBufferFactory.getBuffer(HeapChannelBufferFactory.java:68)  
at org.jboss.netty.buffer.AbstractChannelBufferFactory.getBuffer(AbstractChannelBufferFactory.java:48)  
at org.jboss.netty.handler.codec.frame.FrameDecoder.extractFrame(FrameDecoder.java:566)  
at org.jboss.netty.handler.codec.frame.LengthFieldBasedFrameDecoder.decode(LengthFieldBasedFrameDecoder.java:391)  
at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:425)  
at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)  
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)  
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)  
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)  
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:109)  
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312)  
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:90)  
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)  
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)  
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)  
at java.lang.Thread.run(Thread.java:745)  
18/04/20 21:57:44 WARN DFSClient: Error Recovery for block BP-15450043-192.168.22.158-1464844718994:blk\_1080847780\_7152657 in pipeline DatanodeInfoWithStorage\[192.168.22.153:50010,DS-9e523c00-0b33-4cee-8ce9-0e3a5be949bb,DISK\], DatanodeInfoWithStorage\[192.168.22.172:50010,DS-fbe9fa38-7c84-418a-b94a-45d8383d29e7,DISK\], DatanodeInfoWithStorage\[192.168.22.151:50010,DS-735335cb-ae15-4fdf-a056-f3944db05ffc,DISK\]: bad datanode DatanodeInfoWithStorage\[192.168.22.153:50010,DS-9e523c00-0b33-4cee-8ce9-0e3a5be949bb,DISK\]  
18/04/20 21:57:44 ERROR ActorSystemImpl: Uncaught fatal error from thread \[sparkDriver-akka.actor.default-dispatcher-28\] shutting down ActorSystem \[sparkDriver\]  
java.lang.OutOfMemoryError: Java heap space  
at akka.util.CompactByteString$.apply(ByteString.scala:438)  
at akka.util.ByteString$.apply(ByteString.scala:38)  
at akka.remote.transport.AkkaPduProtobufCodec$.decodePdu(AkkaPduCodec.scala:169)  
at akka.remote.transport.ProtocolStateActor.akka$remote$transport$ProtocolStateActor$$decodePdu(AkkaProtocolTransport.scala:513)  
at akka.remote.transport.ProtocolStateActor$$anonfun$5.applyOrElse(AkkaProtocolTransport.scala:357)  
at akka.remote.transport.ProtocolStateActor$$anonfun$5.applyOrElse(AkkaProtocolTransport.scala:352)  
at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:33)  
at akka.actor.FSM$class.processEvent(FSM.scala:595)  
at akka.remote.transport.ProtocolStateActor.processEvent(AkkaProtocolTransport.scala:220)  
at akka.actor.FSM$class.akka$actor$FSM$$processMsg(FSM.scala:589)  
at akka.actor.FSM$$anonfun$receive$1.applyOrElse(FSM.scala:583)  
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:498)  
at akka.actor.ActorCell.invoke(ActorCell.scala:456)  
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:237)  
at akka.dispatch.Mailbox.run(Mailbox.scala:219)  
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386)  
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)  
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)  
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)  
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)  
18/04/20 21:57:44 WARN AkkaRpcEndpointRef: Error sending message \[message = BlockManagerHeartbeat(BlockManagerId(6, hadoopserver06, 36838))\] in 1 attempts  
org.apache.spark.rpc.RpcTimeoutException: Recipient\[Actor\[akka://sparkDriver/user/BlockManagerMaster\#408486509\]\] had already been terminated.. This timeout is controlled by BlockManagerHeartbeat  
at org.apache.spark.rpc.RpcTimeout.org$apache$spark$rpc$RpcTimeout$$createRpcTimeoutException(RpcEnv.scala:214)  
at org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcEnv.scala:229)  
at org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcEnv.scala:225)  
at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:33)  
at scala.util.Failure$$anonfun$recover$1.apply(Try.scala:185)  
at scala.util.Try$.apply(Try.scala:161)  
at scala.util.Failure.recover(Try.scala:185)  
at scala.concurrent.Future$$anonfun$recover$1.apply(Future.scala:324)  
at scala.concurrent.Future$$anonfun$recover$1.apply(Future.scala:324)  
at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)  
at org.spark-project.guava.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:293)  
at scala.concurrent.impl.ExecutionContextImpl$$anon$1.execute(ExecutionContextImpl.scala:133)  
at scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:40)  
at scala.concurrent.impl.Promise$DefaultPromise.scala$concurrent$impl$Promise$DefaultPromise$$dispatchOrAddCallback(Promise.scala:280)  
at scala.concurrent.impl.Promise$DefaultPromise.onComplete(Promise.scala:270)  
at scala.concurrent.Future$class.recover(Future.scala:324)  
at scala.concurrent.impl.Promise$DefaultPromise.recover(Promise.scala:153)  
at org.apache.spark.rpc.akka.AkkaRpcEndpointRef.ask(AkkaRpcEnv.scala:319)  
at org.apache.spark.rpc.RpcEndpointRef.askWithRetry(RpcEndpointRef.scala:100)  
at org.apache.spark.scheduler.DAGScheduler.executorHeartbeatReceived(DAGScheduler.scala:194)  
at org.apache.spark.scheduler.TaskSchedulerImpl.executorHeartbeatReceived(TaskSchedulerImpl.scala:386)  
at org.apache.spark.HeartbeatReceiver$$anonfun$receiveAndReply$1$$anon$2$$anonfun$run$2.apply$mcV$sp(HeartbeatReceiver.scala:127)  
at org.apache.spark.util.Utils$.tryLogNonFatalError(Utils.scala:1185)  
at org.apache.spark.HeartbeatReceiver$$anonfun$receiveAndReply$1$$anon$2.run(HeartbeatReceiver.scala:126)  
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)  
at java.util.concurrent.FutureTask.run(FutureTask.java:266)  
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)  
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)  
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)  
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)  
at java.lang.Thread.run(Thread.java:745)  
Caused by: akka.pattern.AskTimeoutException: Recipient\[Actor\[akka://sparkDriver/user/BlockManagerMaster\#408486509\]\] had already been terminated.  
at akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)  
at org.apache.spark.rpc.akka.AkkaRpcEndpointRef.ask(AkkaRpcEnv.scala:307)  
... 13 more  
18/04/20 21:57:44 ERROR ActorSystemImpl: Uncaught fatal error from thread \[sparkDriver-akka.actor.default-dispatcher-18\] shutting down ActorSystem \[sparkDriver\]  
java.lang.OutOfMemoryError: Java heap space  
18/04/20 21:57:44 INFO RemoteActorRefProvider$RemotingTerminator: Shutting down remote daemon.  
18/04/20 21:57:44 INFO RetryInvocationHandler: Exception while invoking renewLease of class ClientNamenodeProtocolTranslatorPB over hadoopserver00/192.168.22.158:8020 after 1 fail over attempts. Trying to fail over after sleeping for 1332ms.  
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException): Operation category WRITE is not supported in state standby. Visit https://s.apache.org/sbnn-error  
at org.apache.hadoop.hdfs.server.namenode.ha.StandbyState.checkOperation(StandbyState.java:88)  
at org.apache.hadoop.hdfs.server.namenode.NameNode$NameNodeHAContext.checkOperation(NameNode.java:1781)  
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkOperation(FSNamesystem.java:1402)  
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.renewLease(FSNamesystem.java:4977)  
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.renewLease(NameNodeRpcServer.java:874)  
at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.renewLease(AuthorizationProviderProxyClientProtocol.java:356)  
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:631)  
at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)  
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617)  
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1060)  
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2086)  
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2082)  
at java.security.AccessController.doPrivileged(Native Method)  
at javax.security.auth.Subject.doAs(Subject.java:422)  
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)  
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2080)  
  
  
at org.apache.hadoop.ipc.Client.call(Client.java:1472)  
at org.apache.hadoop.ipc.Client.call(Client.java:1403)  
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230)  
at com.sun.proxy.$Proxy15.renewLease(Unknown Source)  
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.renewLease(ClientNamenodeProtocolTranslatorPB.java:571)  
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)  
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)  
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)  
at java.lang.reflect.Method.invoke(Method.java:497)  
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:252)  
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:104)  
at com.sun.proxy.$Proxy16.renewLease(Unknown Source)  
at org.apache.hadoop.hdfs.DFSClient.renewLease(DFSClient.java:921)  
at org.apache.hadoop.hdfs.LeaseRenewer.renew(LeaseRenewer.java:423)  
at org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:448)  
at org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71)  
at org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:304)  
at java.lang.Thread.run(Thread.java:745)  
18/04/20 21:57:44 INFO SparkContext: Invoking stop() from shutdown hook  
18/04/20 21:57:44 INFO RetryInvocationHandler: Exception while invoking getAdditionalDatanode of class ClientNamenodeProtocolTranslatorPB over hadoopserver00/192.168.22.158:8020. Trying to fail over immediately.  
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException): Operation category READ is not supported in state standby. Visit https://s.apache.org/sbnn-error  
at org.apache.hadoop.hdfs.server.namenode.ha.StandbyState.checkOperation(StandbyState.java:88)  
at org.apache.hadoop.hdfs.server.namenode.NameNode$NameNodeHAContext.checkOperation(NameNode.java:1781)  
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkOperation(FSNamesystem.java:1402)  
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalDatanode(FSNamesystem.java:3497)  
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getAdditionalDatanode(NameNodeRpcServer.java:710)  
at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getAdditionalDatanode(AuthorizationProviderProxyClientProtocol.java:228)  
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getAdditionalDatanode(ClientNamenodeProtocolServerSideTranslatorPB.java:506)  
at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)  
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617)  
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1060)  
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2086)  
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2082)  
at java.security.AccessController.doPrivileged(Native Method)  
at javax.security.auth.Subject.doAs(Subject.java:422)  
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)  
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2080)  
  
  
at org.apache.hadoop.ipc.Client.call(Client.java:1472)  
at org.apache.hadoop.ipc.Client.call(Client.java:1403)  
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230)  
at com.sun.proxy.$Proxy15.getAdditionalDatanode(Unknown Source)  
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getAdditionalDatanode(ClientNamenodeProtocolTranslatorPB.java:424)  
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)  
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)  
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)  
at java.lang.reflect.Method.invoke(Method.java:497)  
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:252)  
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:104)  
at com.sun.proxy.$Proxy16.getAdditionalDatanode(Unknown Source)  
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1221)  
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1375)  
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:1119)  
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:622)  
18/04/20 21:57:44 INFO RemoteActorRefProvider$RemotingTerminator: Remote daemon shut down; proceeding with flushing remote transports.  
18/04/20 21:57:44 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler\{/metrics/json,null\}  
18/04/20 21:57:44 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler\{/stages/stage/kill,null\}  
18/04/20 21:57:44 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler\{/api,null\}  
18/04/20 21:57:44 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler\{/,null\}  
18/04/20 21:57:44 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler\{/static,null\}  
18/04/20 21:57:44 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler\{/executors/threadDump/json,null\}  
18/04/20 21:57:44 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler\{/executors/threadDump,null\}  
18/04/20 21:57:44 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler\{/executors/json,null\}  
18/04/20 21:57:44 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler\{/executors,null\}  
18/04/20 21:57:44 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler\{/environment/json,null\}  
18/04/20 21:57:44 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler\{/environment,null\}  
18/04/20 21:57:44 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler\{/storage/rdd/json,null\}  
18/04/20 21:57:44 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler\{/storage/rdd,null\}  
18/04/20 21:57:44 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler\{/storage/json,null\}  
18/04/20 21:57:44 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler\{/storage,null\}  
18/04/20 21:57:44 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler\{/stages/pool/json,null\}  
18/04/20 21:57:44 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler\{/stages/pool,null\}  
18/04/20 21:57:44 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler\{/stages/stage/json,null\}  
18/04/20 21:57:44 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler\{/stages/stage,null\}  
18/04/20 21:57:44 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler\{/stages/json,null\}  
18/04/20 21:57:44 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler\{/stages,null\}  
18/04/20 21:57:44 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler\{/jobs/job/json,null\}  
18/04/20 21:57:44 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler\{/jobs/job,null\}  
18/04/20 21:57:44 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler\{/jobs/json,null\}  
18/04/20 21:57:44 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler\{/jobs,null\}  
18/04/20 21:57:44 INFO SparkUI: Stopped Spark web UI at http://192.168.22.158:4069  
18/04/20 21:57:44 INFO DAGScheduler: Stopping DAGScheduler  
18/04/20 21:57:44 INFO YarnClientSchedulerBackend: Shutting down all executors  
18/04/20 21:57:44 INFO YarnClientSchedulerBackend: Interrupting monitor thread  
18/04/20 21:57:44 WARN AkkaRpcEndpointRef: Error sending message \[message = StopExecutors\] in 1 attempts  
org.apache.spark.rpc.RpcTimeoutException: Recipient\[Actor\[akka://sparkDriver/user/CoarseGrainedScheduler\#1480151067\]\] had already been terminated.. This timeout is controlled by spark.rpc.askTimeout  
at org.apache.spark.rpc.RpcTimeout.org$apache$spark$rpc$RpcTimeout$$createRpcTimeoutException(RpcEnv.scala:214)  
at org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcEnv.scala:229)  
at org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcEnv.scala:225)  
at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:33)  
at scala.util.Failure$$anonfun$recover$1.apply(Try.scala:185)  
at scala.util.Try$.apply(Try.scala:161)  
at scala.util.Failure.recover(Try.scala:185)  
at scala.concurrent.Future$$anonfun$recover$1.apply(Future.scala:324)  
at scala.concurrent.Future$$anonfun$recover$1.apply(Future.scala:324)  
at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)  
at org.spark-project.guava.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:293)  
at scala.concurrent.impl.ExecutionContextImpl$$anon$1.execute(ExecutionContextImpl.scala:133)  
at scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:40)  
at scala.concurrent.impl.Promise$DefaultPromise.scala$concurrent$impl$Promise$DefaultPromise$$dispatchOrAddCallback(Promise.scala:280)  
at scala.concurrent.impl.Promise$DefaultPromise.onComplete(Promise.scala:270)  
at scala.concurrent.Future$class.recover(Future.scala:324)  
at scala.concurrent.impl.Promise$DefaultPromise.recover(Promise.scala:153)  
at org.apache.spark.rpc.akka.AkkaRpcEndpointRef.ask(AkkaRpcEnv.scala:319)  
at org.apache.spark.rpc.RpcEndpointRef.askWithRetry(RpcEndpointRef.scala:100)  
at org.apache.spark.rpc.RpcEndpointRef.askWithRetry(RpcEndpointRef.scala:77)  
at org.apache.spark.scheduler.cluster.CoarseGrainedSchedulerBackend.stopExecutors(CoarseGrainedSchedulerBackend.scala:274)  
at org.apache.spark.scheduler.cluster.CoarseGrainedSchedulerBackend.stop(CoarseGrainedSchedulerBackend.scala:283)  
at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.stop(YarnClientSchedulerBackend.scala:180)  
at org.apache.spark.scheduler.TaskSchedulerImpl.stop(TaskSchedulerImpl.scala:439)  
at org.apache.spark.scheduler.DAGScheduler.stop(DAGScheduler.scala:1450)  
at org.apache.spark.SparkContext$$anonfun$stop$7.apply$mcV$sp(SparkContext.scala:1724)  
at org.apache.spark.util.Utils$.tryLogNonFatalError(Utils.scala:1185)  
at org.apache.spark.SparkContext.stop(SparkContext.scala:1723)  
at org.apache.spark.SparkContext$$anonfun$3.apply$mcV$sp(SparkContext.scala:587)  
at org.apache.spark.util.SparkShutdownHook.run(ShutdownHookManager.scala:264)  
at org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(ShutdownHookManager.scala:234)  
at org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1$$anonfun$apply$mcV$sp$1.apply(ShutdownHookManager.scala:234)  
at org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1$$anonfun$apply$mcV$sp$1.apply(ShutdownHookManager.scala:234)  
at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1699)  
at org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1.apply$mcV$sp(ShutdownHookManager.scala:234)  
at org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1.apply(ShutdownHookManager.scala:234)  
at org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1.apply(ShutdownHookManager.scala:234)  
at scala.util.Try$.apply(Try.scala:161)  
at org.apache.spark.util.SparkShutdownHookManager.runAll(ShutdownHookManager.scala:234)  
at org.apache.spark.util.SparkShutdownHookManager$$anon$2.run(ShutdownHookManager.scala:216)  
at org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:54)  
Caused by: akka.pattern.AskTimeoutException: Recipient\[Actor\[akka://sparkDriver/user/CoarseGrainedScheduler\#1480151067\]\] had already been terminated.  
at akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)  
at org.apache.spark.rpc.akka.AkkaRpcEndpointRef.ask(AkkaRpcEnv.scala:307)  
... 23 more  
18/04/20 21:57:45 INFO Remoting: Remoting shut down  
18/04/20 21:57:45 INFO RemoteActorRefProvider$RemotingTerminator: Remoting shut down.  
18/04/20 21:57:45 WARN RetryInvocationHandler: A failover has occurred since the start of this method invocation attempt.  
18/04/20 21:57:47 WARN AkkaRpcEndpointRef: Error sending message \[message = StopExecutors\] in 2 attempts  
org.apache.spark.rpc.RpcTimeoutException: Recipient\[Actor\[akka://sparkDriver/user/CoarseGrainedScheduler\#1480151067\]\] had already been terminated.. This timeout is controlled by spark.rpc.askTimeout  
at org.apache.spark.rpc.RpcTimeout.org$apache$spark$rpc$RpcTimeout$$createRpcTimeoutException(RpcEnv.scala:214)  
at org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcEnv.scala:229)  
at org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcEnv.scala:225)  
at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:33)  
at scala.util.Failure$$anonfun$recover$1.apply(Try.scala:185)  
at scala.util.Try$.apply(Try.scala:161)  
at scala.util.Failure.recover(Try.scala:185)  
at scala.concurrent.Future$$anonfun$recover$1.apply(Future.scala:324)  
at scala.concurrent.Future$$anonfun$recover$1.apply(Future.scala:324)  
at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)  
at org.spark-project.guava.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:293)  
at scala.concurrent.impl.ExecutionContextImpl$$anon$1.execute(ExecutionContextImpl.scala:133)  
at scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:40)  
at scala.concurrent.impl.Promise$DefaultPromise.scala$concurrent$impl$Promise$DefaultPromise$$dispatchOrAddCallback(Promise.scala:280)  
at scala.concurrent.impl.Promise$DefaultPromise.onComplete(Promise.scala:270)  
at scala.concurrent.Future$class.recover(Future.scala:324)  
at scala.concurrent.impl.Promise$DefaultPromise.recover(Promise.scala:153)  
at org.apache.spark.rpc.akka.AkkaRpcEndpointRef.ask(AkkaRpcEnv.scala:319)  
at org.apache.spark.rpc.RpcEndpointRef.askWithRetry(RpcEndpointRef.scala:100)  
at org.apache.spark.rpc.RpcEndpointRef.askWithRetry(RpcEndpointRef.scala:77)  
at org.apache.spark.scheduler.cluster.CoarseGrainedSchedulerBackend.stopExecutors(CoarseGrainedSchedulerBackend.scala:274)  
at org.apache.spark.scheduler.cluster.CoarseGrainedSchedulerBackend.stop(CoarseGrainedSchedulerBackend.scala:283)  
at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.stop(YarnClientSchedulerBackend.scala:180)  
at org.apache.spark.scheduler.TaskSchedulerImpl.stop(TaskSchedulerImpl.scala:439)  
at org.apache.spark.scheduler.DAGScheduler.stop(DAGScheduler.scala:1450)  
at org.apache.spark.SparkContext$$anonfun$stop$7.apply$mcV$sp(SparkContext.scala:1724)  
at org.apache.spark.util.Utils$.tryLogNonFatalError(Utils.scala:1185)  
at org.apache.spark.SparkContext.stop(SparkContext.scala:1723)  
at org.apache.spark.SparkContext$$anonfun$3.apply$mcV$sp(SparkContext.scala:587)  
at org.apache.spark.util.SparkShutdownHook.run(ShutdownHookManager.scala:264)  
at org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(ShutdownHookManager.scala:234)  
at org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1$$anonfun$apply$mcV$sp$1.apply(ShutdownHookManager.scala:234)  
at org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1$$anonfun$apply$mcV$sp$1.apply(ShutdownHookManager.scala:234)  
at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1699)  
at org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1.apply$mcV$sp(ShutdownHookManager.scala:234)  
at org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1.apply(ShutdownHookManager.scala:234)  
at org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1.apply(ShutdownHookManager.scala:234)  
at scala.util.Try$.apply(Try.scala:161)  
at org.apache.spark.util.SparkShutdownHookManager.runAll(ShutdownHookManager.scala:234)  
at org.apache.spark.util.SparkShutdownHookManager$$anon$2.run(ShutdownHookManager.scala:216)  
at org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:54)  
Caused by: akka.pattern.AskTimeoutException: Recipient\[Actor\[akka://sparkDriver/user/CoarseGrainedScheduler\#1480151067\]\] had already been terminated.  
at akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)  
at org.apache.spark.rpc.akka.AkkaRpcEndpointRef.ask(AkkaRpcEnv.scala:307)  
... 23 more  
18/04/20 21:57:50 WARN AkkaRpcEndpointRef: Error sending message \[message = StopExecutors\] in 3 attempts  
org.apache.spark.rpc.RpcTimeoutException: Recipient\[Actor\[akka://sparkDriver/user/CoarseGrainedScheduler\#1480151067\]\] had already been terminated.. This timeout is controlled by spark.rpc.askTimeout  
at org.apache.spark.rpc.RpcTimeout.org$apache$spar