MapReduce运行任务报错

Dear 丶 2022-06-11 08:25 371阅读 0赞

MapReduce Error：

Error: java.io.IOException: Failing write.

Tried pipeline recovery 5 times without success.   
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:1113)   
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:622)  
attempt\_1501490104699\_0091\_r\_000014\_0 100.00 FAILED reduce > reduce hadoopserver16:8042 logs   
Wed Aug 2 12:15:24 +0800 2017 Wed Aug 2 17:32:55 +0800 2017   
5hrs, 17mins, 31sec Error:   
java.io.IOException: Failing write. Tried pipeline recovery 5 times without success.   
  
  
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:1113)   
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:622)  
  
  
  
  
  
  
java.io.IOException: Bad response ERROR for block BP-15450043-192.168.22.158-1464844718994:blk\_1077470611\_3762901 from datanode DatanodeInfoWithStorage\[192.168.22.164:50010,DS-e4a0570e-6077-447e-8299-de701eb33a1b,DISK\]  
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer$ResponseProcessor.run(DFSOutputStream.java:1002)  
2017-08-03 04:47:23,860 WARN \[DataStreamer for file /user/hive/warehouse/transinfo\_db.db/gps\_srcdata\_orc\_middle\_timeid/\_SCRATCH0.6625733093777507/supplier=huoyun/logdate=20160830/timeid=001\_288/\_temporary/1/\_temporary/attempt\_1501681898109\_0001\_r\_000007\_0/part-r-00007 block BP-15450043-192.168.22.158-1464844718994:blk\_1077470611\_3762901\] org.apache.hadoop.hdfs.DFSClient: Error Recovery for block BP-15450043-192.168.22.158-1464844718994:blk\_1077470611\_3762901 in pipeline DatanodeInfoWithStorage\[192.168.22.163:50010,DS-47fcf3cc-f3db-4475-b98e-d08178cb97a8,DISK\], DatanodeInfoWithStorage\[192.168.22.164:50010,DS-e4a0570e-6077-447e-8299-de701eb33a1b,DISK\], DatanodeInfoWithStorage\[192.168.22.151:50010,DS-735335cb-ae15-4fdf-a056-f3944db05ffc,DISK\]: bad datanode DatanodeInfoWithStorage\[192.168.22.164:50010,DS-e4a0570e-6077-447e-8299-de701eb33a1b,DISK\]  
2017-08-03 05:06:07,338 WARN \[ResponseProcessor for block BP-15450043-192.168.22.158-1464844718994:blk\_1077470611\_3762904\] org.apache.hadoop.hdfs.DFSClient: DFSOutputStream ResponseProcessor exception  for block BP-15450043-192.168.22.158-1464844718994:blk\_1077470611\_3762904  
java.io.EOFException: Premature EOF: no length prefix available  
at org.apache.hadoop.hdfs.protocolPB.PBHelper.vintPrefixed(PBHelper.java:2241)  
at org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:235)  
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer$ResponseProcessor.run(DFSOutputStream.java:971)  
2017-08-03 05:06:07,340 WARN \[DataStreamer for file /user/hive/warehouse/transinfo\_db.db/gps\_srcdata\_orc\_middle\_timeid/\_SCRATCH0.6625733093777507/supplier=huoyun/logdate=20160830/timeid=001\_288/\_temporary/1/\_temporary/attempt\_1501681898109\_0001\_r\_000007\_0/part-r-00007 block BP-15450043-192.168.22.158-1464844718994:blk\_1077470611\_3762904\] org.apache.hadoop.hdfs.DFSClient: Error Recovery for block BP-15450043-192.168.22.158-1464844718994:blk\_1077470611\_3762904 in pipeline DatanodeInfoWithStorage\[192.168.22.163:50010,DS-47fcf3cc-f3db-4475-b98e-d08178cb97a8,DISK\], DatanodeInfoWithStorage\[192.168.22.151:50010,DS-735335cb-ae15-4fdf-a056-f3944db05ffc,DISK\]: bad datanode DatanodeInfoWithStorage\[192.168.22.163:50010,DS-47fcf3cc-f3db-4475-b98e-d08178cb97a8,DISK\]  
2017-08-03 05:25:01,890 WARN \[ResponseProcessor for block BP-15450043-192.168.22.158-1464844718994:blk\_1077470611\_3762933\] org.apache.hadoop.hdfs.DFSClient: DFSOutputStream ResponseProcessor exception  for block BP-15450043-192.168.22.158-1464844718994:blk\_1077470611\_3762933  
java.io.EOFException: Premature EOF: no length prefix available  
at org.apache.hadoop.hdfs.protocolPB.PBHelper.vintPrefixed(PBHelper.java:2241)  
at org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:235)  
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer$ResponseProcessor.run(DFSOutputStream.java:971)  
2017-08-03 05:25:01,891 WARN \[DataStreamer for file /user/hive/warehouse/transinfo\_db.db/gps\_srcdata\_orc\_middle\_timeid/\_SCRATCH0.6625733093777507/supplier=huoyun/logdate=20160830/timeid=001\_288/\_temporary/1/\_temporary/attempt\_1501681898109\_0001\_r\_000007\_0/part-r-00007 block BP-15450043-192.168.22.158-1464844718994:blk\_1077470611\_3762933\] org.apache.hadoop.hdfs.DFSClient: Error Recovery for block BP-15450043-192.168.22.158-1464844718994:blk\_1077470611\_3762933 in pipeline DatanodeInfoWithStorage\[192.168.22.151:50010,DS-735335cb-ae15-4fdf-a056-f3944db05ffc,DISK\], DatanodeInfoWithStorage\[192.168.22.152:50010,DS-7ec6ea23-21fd-4447-a066-af8a1b2fb967,DISK\]: bad datanode DatanodeInfoWithStorage\[192.168.22.151:50010,DS-735335cb-ae15-4fdf-a056-f3944db05ffc,DISK\]  
2017-08-03 05:43:40,302 WARN \[ResponseProcessor for block BP-15450043-192.168.22.158-1464844718994:blk\_1077470611\_3762955\] org.apache.hadoop.hdfs.DFSClient: Slow ReadProcessor read fields took 66624ms (threshold=30000ms); ack: seqno: -2 reply: 0 reply: 1 downstreamAckTimeNanos: 0, targets: \[DatanodeInfoWithStorage\[192.168.22.152:50010,DS-7ec6ea23-21fd-4447-a066-af8a1b2fb967,DISK\], DatanodeInfoWithStorage\[192.168.22.153:50010,DS-9e523c00-0b33-4cee-8ce9-0e3a5be949bb,DISK\]\]  
2017-08-03 05:43:40,303 WARN \[ResponseProcessor for block BP-15450043-192.168.22.158-1464844718994:blk\_1077470611\_3762955\] org.apache.hadoop.hdfs.DFSClient: DFSOutputStream ResponseProcessor exception  for block BP-15450043-192.168.22.158-1464844718994:blk\_1077470611\_3762955  
java.io.IOException: Bad response ERROR for block BP-15450043-192.168.22.158-1464844718994:blk\_1077470611\_3762955 from datanode DatanodeInfoWithStorage\[192.168.22.153:50010,DS-9e523c00-0b33-4cee-8ce9-0e3a5be949bb,DISK\]  
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer$ResponseProcessor.run(DFSOutputStream.java:1002)  
2017-08-03 05:43:40,304 WARN \[DataStreamer for file /user/hive/warehouse/transinfo\_db.db/gps\_srcdata\_orc\_middle\_timeid/\_SCRATCH0.6625733093777507/supplier=huoyun/logdate=20160830/timeid=001\_288/\_temporary/1/\_temporary/attempt\_1501681898109\_0001\_r\_000007\_0/part-r-00007 block BP-15450043-192.168.22.158-1464844718994:blk\_1077470611\_3762955\] org.apache.hadoop.hdfs.DFSClient: Error Recovery for block BP-15450043-192.168.22.158-1464844718994:blk\_1077470611\_3762955 in pipeline DatanodeInfoWithStorage\[192.168.22.152:50010,DS-7ec6ea23-21fd-4447-a066-af8a1b2fb967,DISK\], DatanodeInfoWithStorage\[192.168.22.153:50010,DS-9e523c00-0b33-4cee-8ce9-0e3a5be949bb,DISK\]: bad datanode DatanodeInfoWithStorage\[192.168.22.153:50010,DS-9e523c00-0b33-4cee-8ce9-0e3a5be949bb,DISK\]  
2017-08-03 06:26:37,039 WARN \[ResponseProcessor for block BP-15450043-192.168.22.158-1464844718994:blk\_1077470611\_3762977\] org.apache.hadoop.hdfs.DFSClient: Slow ReadProcessor read fields took 65551ms (threshold=30000ms); ack: seqno: -2 reply: 0 reply: 1 downstreamAckTimeNanos: 0, targets: \[DatanodeInfoWithStorage\[192.168.22.152:50010,DS-7ec6ea23-21fd-4447-a066-af8a1b2fb967,DISK\], DatanodeInfoWithStorage\[192.168.22.173:50010,DS-b3903c92-4732-42b4-a83f-546d260d137a,DISK\]\]  
2017-08-03 06:26:37,040 WARN \[ResponseProcessor for block BP-15450043-192.168.22.158-1464844718994:blk\_1077470611\_3762977\] org.apache.hadoop.hdfs.DFSClient: DFSOutputStream ResponseProcessor exception  for block BP-15450043-192.168.22.158-1464844718994:blk\_1077470611\_3762977  
java.io.IOException: Bad response ERROR for block BP-15450043-192.168.22.158-1464844718994:blk\_1077470611\_3762977 from datanode DatanodeInfoWithStorage\[192.168.22.173:50010,DS-b3903c92-4732-42b4-a83f-546d260d137a,DISK\]  
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer$ResponseProcessor.run(DFSOutputStream.java:1002)  
2017-08-03 06:26:37,045 WARN \[DataStreamer for file /user/hive/warehouse/transinfo\_db.db/gps\_srcdata\_orc\_middle\_timeid/\_SCRATCH0.6625733093777507/supplier=huoyun/logdate=20160830/timeid=001\_288/\_temporary/1/\_temporary/attempt\_1501681898109\_0001\_r\_000007\_0/part-r-00007 block BP-15450043-192.168.22.158-1464844718994:blk\_1077470611\_3762977\] org.apache.hadoop.hdfs.DFSClient: Error Recovery for block BP-15450043-192.168.22.158-1464844718994:blk\_1077470611\_3762977 in pipeline DatanodeInfoWithStorage\[192.168.22.152:50010,DS-7ec6ea23-21fd-4447-a066-af8a1b2fb967,DISK\], DatanodeInfoWithStorage\[192.168.22.173:50010,DS-b3903c92-4732-42b4-a83f-546d260d137a,DISK\]: bad datanode DatanodeInfoWithStorage\[192.168.22.173:50010,DS-b3903c92-4732-42b4-a83f-546d260d137a,DISK\]  
2017-08-03 06:30:13,838 WARN \[ResponseProcessor for block BP-15450043-192.168.22.158-1464844718994:blk\_1077470611\_3763028\] org.apache.hadoop.hdfs.DFSClient: Slow ReadProcessor read fields took 60247ms (threshold=30000ms); ack: seqno: -2 reply: 0 reply: 1 downstreamAckTimeNanos: 0, targets: \[DatanodeInfoWithStorage\[192.168.22.152:50010,DS-7ec6ea23-21fd-4447-a066-af8a1b2fb967,DISK\], DatanodeInfoWithStorage\[192.168.22.161:50010,DS-aca2eb5f-e9b9-45a3-af21-bb66c55b0f3a,DISK\]\]  
2017-08-03 06:30:13,838 WARN \[ResponseProcessor for block BP-15450043-192.168.22.158-1464844718994:blk\_1077470611\_3763028\] org.apache.hadoop.hdfs.DFSClient: DFSOutputStream ResponseProcessor exception  for block BP-15450043-192.168.22.158-1464844718994:blk\_1077470611\_3763028  
java.io.IOException: Bad response ERROR for block BP-15450043-192.168.22.158-1464844718994:blk\_1077470611\_3763028 from datanode DatanodeInfoWithStorage\[192.168.22.161:50010,DS-aca2eb5f-e9b9-45a3-af21-bb66c55b0f3a,DISK\]  
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer$ResponseProcessor.run(DFSOutputStream.java:1002)  
2017-08-03 06:30:13,839 WARN \[DataStreamer for file /user/hive/warehouse/transinfo\_db.db/gps\_srcdata\_orc\_middle\_timeid/\_SCRATCH0.6625733093777507/supplier=huoyun/logdate=20160830/timeid=001\_288/\_temporary/1/\_temporary/attempt\_1501681898109\_0001\_r\_000007\_0/part-r-00007 block BP-15450043-192.168.22.158-1464844718994:blk\_1077470611\_3763028\] org.apache.hadoop.hdfs.DFSClient: Error recovering pipeline for writing BP-15450043-192.168.22.158-1464844718994:blk\_1077470611\_3763028. Already retried 5 times for the same packet.  
2017-08-03 08:09:07,644 WARN \[communication thread\] org.apache.hadoop.yarn.util.ProcfsBasedProcessTree: Error reading the stream java.io.IOException: 没有那个进程

Looks like you are reaching the limit set for max open file handles in your system.

Is your application opening lot of files at same time? If you expect your application to open lot many files at same time, you can increase max open files limit in hadoop by changing some settings. (Check command " ulimit -n <limit>" to update your open files handles limit.)

If your application is not supposed to open many files at same time, please check why so many file handles are open at same time.

Chiranjeevi,

You will need to set the limit on each server in the cluster ->  [http://man7.org/linux/man-pages/man3/ulimit.3.html][http_man7.org_linux_man-pages_man3_ulimit.3.html]

Run ulimit on each server in the cluster to set the value. Once you decide what the number should be, do remember to (a) set it using ulimit, & (b) put this command in a script that is invoked during reboot. To decide the number, you will need to know max number of open files the app has at any given time, and then add a buffer of say 2x to it. This user is the one used for Hadoop.

http://apache-apex-users-list.78494.x6.nabble.com/java-io-IOException-All-datanodes-DatanodeInfoWithStorage-td1049.html\#a1059

[http_man7.org_linux_man-pages_man3_ulimit.3.html]: http://man7.org/linux/man-pages/man3/ulimit.3.html