hadoop執行測試寫性能的jar包報錯失敗

想測試一下hadoop寫的性能，向hdfs上傳10個128M的數據，失敗，在此記錄一下

我認爲的原因

日誌太長了，我寫在前面了

我的三臺ecs兩個在北京一個在深圳，我認爲可能是帶寬太小（1Mbps）導致的失敗，但是我看了下後臺記錄根本沒有佔滿，而且趨近於0，後來就想着能不能把三臺服務器內網連通，但是使用企業內網不同地域（深圳-北京）要買流量包，比服務器都貴。。。最後解決的辦法是又買了一臺配置稍微差的北京地域的服務器，內網連通，真香。。。

環境

使用的是3臺阿里雲ecs服務器（102,103,104）搭建的集羣，其中102爲namenode服務器，三臺均爲DataNode服務器，副本數爲2，爲防止暴露公網地址對地址進行了替換

結點	公網ip	內網ip
102		102n.102n.102n
103	103w.103w.103w
104	104w.104w.104w

完整的執行過程

[root@hadoop102 ~]# hadoop jar /opt/module/hadoop-2.7.2/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.7.2-tests.jar TestDFSIO -write -nrFiles 10 -fileSize 128MB
20/05/24 19:58:20 INFO fs.TestDFSIO: TestDFSIO.1.8
20/05/24 19:58:20 INFO fs.TestDFSIO: nrFiles = 10
20/05/24 19:58:20 INFO fs.TestDFSIO: nrBytes (MB) = 128.0
20/05/24 19:58:20 INFO fs.TestDFSIO: bufferSize = 1000000
20/05/24 19:58:20 INFO fs.TestDFSIO: baseDir = /benchmarks/TestDFSIO
20/05/24 19:58:21 INFO fs.TestDFSIO: creating control file: 134217728 bytes, 10 files
20/05/24 19:58:22 INFO fs.TestDFSIO: created control files for: 10 files
20/05/24 19:58:22 INFO client.RMProxy: Connecting to ResourceManager at hadoop103/47.107.131.213:8032
20/05/24 19:58:23 INFO client.RMProxy: Connecting to ResourceManager at hadoop103/47.107.131.213:8032
20/05/24 19:58:34 INFO mapred.FileInputFormat: Total input paths to process : 10
20/05/24 19:58:34 INFO mapreduce.JobSubmitter: number of splits:10
20/05/24 19:58:35 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1590199135215_0002
20/05/24 19:58:35 INFO impl.YarnClientImpl: Submitted application application_1590199135215_0002
20/05/24 19:58:35 INFO mapreduce.Job: The url to track the job: http://hadoop103:8088/proxy/application_1590199135215_0002/
20/05/24 19:58:35 INFO mapreduce.Job: Running job: job_1590199135215_0002
20/05/24 19:59:55 INFO mapreduce.Job: Job job_1590199135215_0002 running in uber mode : false
20/05/24 19:59:55 INFO mapreduce.Job:  map 0% reduce 0%
20/05/24 20:00:15 INFO mapreduce.Job:  map 7% reduce 0%
20/05/24 20:00:16 INFO mapreduce.Job:  map 20% reduce 0%
20/05/24 20:00:17 INFO mapreduce.Job:  map 40% reduce 0%
20/05/24 20:10:24 INFO mapreduce.Job: Task Id : attempt_1590199135215_0002_m_000009_0, Status : FAILED
AttemptID:attempt_1590199135215_0002_m_000009_0 Timed out after 600 secs
20/05/24 20:10:24 INFO mapreduce.Job: Task Id : attempt_1590199135215_0002_m_000002_0, Status : FAILED
AttemptID:attempt_1590199135215_0002_m_000002_0 Timed out after 600 secs
20/05/24 20:10:24 INFO mapreduce.Job: Task Id : attempt_1590199135215_0002_m_000000_0, Status : FAILED
AttemptID:attempt_1590199135215_0002_m_000000_0 Timed out after 600 secs
20/05/24 20:10:24 INFO mapreduce.Job: Task Id : attempt_1590199135215_0002_m_000001_0, Status : FAILED
AttemptID:attempt_1590199135215_0002_m_000001_0 Timed out after 600 secs
20/05/24 20:13:45 INFO mapreduce.Job:  map 33% reduce 0%
20/05/24 20:13:45 INFO mapreduce.Job: Task Id : attempt_1590199135215_0002_m_000007_0, Status : FAILED
Error: java.io.IOException: All datanodes DatanodeInfoWithStorage[47.107.131.213:50010,DS-9931cc42-d694-4e79-b2a2-1fc2af193328,DISK] are bad. Aborting...
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1113)
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:876)
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:402)

20/05/24 20:13:56 INFO mapreduce.Job:  map 40% reduce 0%
20/05/24 20:20:54 INFO mapreduce.Job:  map 80% reduce 0%
20/05/24 20:20:54 INFO mapreduce.Job: Task Id : attempt_1590199135215_0002_m_000001_1, Status : FAILED
AttemptID:attempt_1590199135215_0002_m_000001_1 Timed out after 600 secs
20/05/24 20:20:54 INFO mapreduce.Job: Task Id : attempt_1590199135215_0002_m_000002_1, Status : FAILED
AttemptID:attempt_1590199135215_0002_m_000002_1 Timed out after 600 secs
20/05/24 20:20:54 INFO mapreduce.Job: Task Id : attempt_1590199135215_0002_m_000009_1, Status : FAILED
AttemptID:attempt_1590199135215_0002_m_000009_1 Timed out after 600 secs
20/05/24 20:20:54 INFO mapreduce.Job: Task Id : attempt_1590199135215_0002_m_000000_1, Status : FAILED
AttemptID:attempt_1590199135215_0002_m_000000_1 Timed out after 600 secs
20/05/24 20:20:55 INFO mapreduce.Job:  map 40% reduce 0%
20/05/24 20:31:24 INFO mapreduce.Job: Task Id : attempt_1590199135215_0002_m_000009_2, Status : FAILED
AttemptID:attempt_1590199135215_0002_m_000009_2 Timed out after 600 secs
20/05/24 20:31:24 INFO mapreduce.Job: Task Id : attempt_1590199135215_0002_m_000000_2, Status : FAILED
AttemptID:attempt_1590199135215_0002_m_000000_2 Timed out after 600 secs
20/05/24 20:31:24 INFO mapreduce.Job: Task Id : attempt_1590199135215_0002_m_000001_2, Status : FAILED
AttemptID:attempt_1590199135215_0002_m_000001_2 Timed out after 600 secs
20/05/24 20:31:24 INFO mapreduce.Job: Task Id : attempt_1590199135215_0002_m_000002_2, Status : FAILED
AttemptID:attempt_1590199135215_0002_m_000002_2 Timed out after 600 secs
20/05/24 20:41:55 INFO mapreduce.Job:  map 100% reduce 100%
20/05/24 20:43:05 INFO mapreduce.Job: Job job_1590199135215_0002 failed with state FAILED due to: Task failed task_1590199135215_0002_m_000001
Job failed as tasks failed. failedMaps:1 failedReduces:0

20/05/24 20:43:06 INFO mapreduce.Job: Counters: 17
	Job Counters 
		Failed map tasks=17
		Killed map tasks=6
		Killed reduce tasks=1
		Launched map tasks=23
		Other local map tasks=13
		Data-local map tasks=10
		Total time spent by all maps in occupied slots (ms)=25130288
		Total time spent by all reduces in occupied slots (ms)=0
		Total time spent by all map tasks (ms)=25130288
		Total time spent by all reduce tasks (ms)=0
		Total vcore-milliseconds taken by all map tasks=25130288
		Total vcore-milliseconds taken by all reduce tasks=0
		Total megabyte-milliseconds taken by all map tasks=25733414912
		Total megabyte-milliseconds taken by all reduce tasks=0
	Map-Reduce Framework
		CPU time spent (ms)=0
		Physical memory (bytes) snapshot=0
		Virtual memory (bytes) snapshot=0
java.io.IOException: Job failed!
	at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:865)
	at org.apache.hadoop.fs.TestDFSIO.runIOTest(TestDFSIO.java:457)
	at org.apache.hadoop.fs.TestDFSIO.writeTest(TestDFSIO.java:439)
	at org.apache.hadoop.fs.TestDFSIO.run(TestDFSIO.java:819)
	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
	at org.apache.hadoop.fs.TestDFSIO.main(TestDFSIO.java:712)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:71)
	at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:144)
	at org.apache.hadoop.test.MapredTestDriver.run(MapredTestDriver.java:130)
	at org.apache.hadoop.test.MapredTestDriver.main(MapredTestDriver.java:138)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
	at org.apache.hadoop.util.RunJar.main(RunJar.java:136)

NameNode日誌文件

2020-05-24 19:50:30,461 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 55 Total time for transactions(ms): 2 Number of transactions batched in Syncs: 0 Number of syncs: 36 SyncTimes(ms): 48 
2020-05-24 19:50:30,464 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1073742640_1830{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-efb40abd-a64b-413d-b9af-4f62d7390c0a:NORMAL:102n.102n.102n.102n:50010|RBW], ReplicaUC[[DISK]DS-42b062e0-208e-4f03-bf28-0b4ba31c3f1b:NORMAL:104w.104w.104w.104w:50010|RBW]]} for /HBase/MasterProcWALs/state-00000000000000000068.log
2020-05-24 19:50:30,497 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap updated: 104w.104w.104w.104w:50010 is added to blk_1073742640_1830{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-efb40abd-a64b-413d-b9af-4f62d7390c0a:NORMAL:102n.102n.102n.102n:50010|RBW], ReplicaUC[[DISK]DS-42b062e0-208e-4f03-bf28-0b4ba31c3f1b:NORMAL:104w.104w.104w.104w:50010|RBW]]} size 0
2020-05-24 19:50:30,498 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap updated: 102n.102n.102n.102n:50010 is added to blk_1073742640_1830{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-efb40abd-a64b-413d-b9af-4f62d7390c0a:NORMAL:102n.102n.102n.102n:50010|RBW], ReplicaUC[[DISK]DS-42b062e0-208e-4f03-bf28-0b4ba31c3f1b:NORMAL:104w.104w.104w.104w:50010|RBW]]} size 0
2020-05-24 19:50:30,500 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /HBase/MasterProcWALs/state-00000000000000000068.log is closed by DFSClient_NONMAPREDUCE_2066682487_1
2020-05-24 19:50:30,503 INFO BlockStateChange: BLOCK* addToInvalidates: blk_1073742640_1830 102n.102n.102n.102n:50010 104w.104w.104w.104w:50010 
2020-05-24 19:50:30,568 INFO BlockStateChange: BLOCK* BlockManager: ask 102n.102n.102n.102n:50010 to delete [blk_1073742640_1830]
2020-05-24 19:50:33,568 INFO BlockStateChange: BLOCK* BlockManager: ask 104w.104w.104w.104w:50010 to delete [blk_1073742640_1830]
2020-05-24 19:58:21,360 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 60 Total time for transactions(ms): 2 Number of transactions batched in Syncs: 0 Number of syncs: 40 SyncTimes(ms): 51 
2020-05-24 19:58:21,487 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1073742641_1831{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-efb40abd-a64b-413d-b9af-4f62d7390c0a:NORMAL:102n.102n.102n.102n:50010|RBW], ReplicaUC[[DISK]DS-42b062e0-208e-4f03-bf28-0b4ba31c3f1b:NORMAL:104w.104w.104w.104w:50010|RBW]]} for /benchmarks/TestDFSIO/io_control/in_file_test_io_0
2020-05-24 19:58:21,620 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap updated: 104w.104w.104w.104w:50010 is added to blk_1073742641_1831{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-efb40abd-a64b-413d-b9af-4f62d7390c0a:NORMAL:102n.102n.102n.102n:50010|RBW], ReplicaUC[[DISK]DS-42b062e0-208e-4f03-bf28-0b4ba31c3f1b:NORMAL:104w.104w.104w.104w:50010|RBW]]} size 0
2020-05-24 19:58:21,622 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap updated: 102n.102n.102n.102n:50010 is added to blk_1073742641_1831{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-efb40abd-a64b-413d-b9af-4f62d7390c0a:NORMAL:102n.102n.102n.102n:50010|RBW], ReplicaUC[[DISK]DS-42b062e0-208e-4f03-bf28-0b4ba31c3f1b:NORMAL:104w.104w.104w.104w:50010|RBW]]} size 0
2020-05-24 19:58:21,627 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /benchmarks/TestDFSIO/io_control/in_file_test_io_0 is closed by DFSClient_NONMAPREDUCE_1115390510_1
2020-05-24 19:58:21,635 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1073742642_1832{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-efb40abd-a64b-413d-b9af-4f62d7390c0a:NORMAL:102n.102n.102n.102n:50010|RBW], ReplicaUC[[DISK]DS-42b062e0-208e-4f03-bf28-0b4ba31c3f1b:NORMAL:104w.104w.104w.104w:50010|RBW]]} for /benchmarks/TestDFSIO/io_control/in_file_test_io_1
2020-05-24 19:58:21,660 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap updated: 104w.104w.104w.104w:50010 is added to blk_1073742642_1832{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-efb40abd-a64b-413d-b9af-4f62d7390c0a:NORMAL:102n.102n.102n.102n:50010|RBW], ReplicaUC[[DISK]DS-42b062e0-208e-4f03-bf28-0b4ba31c3f1b:NORMAL:104w.104w.104w.104w:50010|RBW]]} size 0
2020-05-24 19:58:21,662 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap updated: 102n.102n.102n.102n:50010 is added to blk_1073742642_1832{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-efb40abd-a64b-413d-b9af-4f62d7390c0a:NORMAL:102n.102n.102n.102n:50010|RBW], ReplicaUC[[DISK]DS-42b062e0-208e-4f03-bf28-0b4ba31c3f1b:NORMAL:104w.104w.104w.104w:50010|RBW]]} size 0
2020-05-24 19:58:21,664 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /benchmarks/TestDFSIO/io_control/in_file_test_io_1 is closed by DFSClient_NONMAPREDUCE_1115390510_1
2020-05-24 19:58:21,669 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1073742643_1833{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-efb40abd-a64b-413d-b9af-4f62d7390c0a:NORMAL:102n.102n.102n.102n:50010|RBW], ReplicaUC[[DISK]DS-42b062e0-208e-4f03-bf28-0b4ba31c3f1b:NORMAL:104w.104w.104w.104w:50010|RBW]]} for /benchmarks/TestDFSIO/io_control/in_file_test_io_2
2020-05-24 19:58:21,694 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap updated: 104w.104w.104w.104w:50010 is added to blk_1073742643_1833{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-efb40abd-a64b-413d-b9af-4f62d7390c0a:NORMAL:102n.102n.102n.102n:50010|RBW], ReplicaUC[[DISK]DS-42b062e0-208e-4f03-bf28-0b4ba31c3f1b:NORMAL:104w.104w.104w.104w:50010|RBW]]} size 0
2020-05-24 19:58:21,697 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap updated: 102n.102n.102n.102n:50010 is added to blk_1073742643_1833{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-efb40abd-a64b-413d-b9af-4f62d7390c0a:NORMAL:102n.102n.102n.102n:50010|RBW], ReplicaUC[[DISK]DS-42b062e0-208e-4f03-bf28-0b4ba31c3f1b:NORMAL:104w.104w.104w.104w:50010|RBW]]} size 0
2020-05-24 19:58:21,700 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /benchmarks/TestDFSIO/io_control/in_file_test_io_2 is closed by DFSClient_NONMAPREDUCE_1115390510_1
2020-05-24 19:58:21,705 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1073742644_1834{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-efb40abd-a64b-413d-b9af-4f62d7390c0a:NORMAL:102n.102n.102n.102n:50010|RBW], ReplicaUC[[DISK]DS-9931cc42-d694-4e79-b2a2-1fc2af193328:NORMAL:103w.103w.103w.103w:50010|RBW]]} for /benchmarks/TestDFSIO/io_control/in_file_test_io_3
2020-05-24 19:58:21,868 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap updated: 103w.103w.103w.103w:50010 is added to blk_1073742644_1834{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-efb40abd-a64b-413d-b9af-4f62d7390c0a:NORMAL:102n.102n.102n.102n:50010|RBW], ReplicaUC[[DISK]DS-9931cc42-d694-4e79-b2a2-1fc2af193328:NORMAL:103w.103w.103w.103w:50010|RBW]]} size 0
2020-05-24 19:58:21,870 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap updated: 102n.102n.102n.102n:50010 is added to blk_1073742644_1834{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-efb40abd-a64b-413d-b9af-4f62d7390c0a:NORMAL:102n.102n.102n.102n:50010|RBW], ReplicaUC[[DISK]DS-9931cc42-d694-4e79-b2a2-1fc2af193328:NORMAL:103w.103w.103w.103w:50010|RBW]]} size 0
2020-05-24 19:58:21,873 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /benchmarks/TestDFSIO/io_control/in_file_test_io_3 is closed by DFSClient_NONMAPREDUCE_1115390510_1
2020-05-24 19:58:21,878 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1073742645_1835{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-efb40abd-a64b-413d-b9af-4f62d7390c0a:NORMAL:102n.102n.102n.102n:50010|RBW], ReplicaUC[[DISK]DS-9931cc42-d694-4e79-b2a2-1fc2af193328:NORMAL:103w.103w.103w.103w:50010|RBW]]} for /benchmarks/TestDFSIO/io_control/in_file_test_io_4
2020-05-24 19:58:22,024 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap updated: 103w.103w.103w.103w:50010 is added to blk_1073742645_1835{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-efb40abd-a64b-413d-b9af-4f62d7390c0a:NORMAL:102n.102n.102n.102n:50010|RBW], ReplicaUC[[DISK]DS-9931cc42-d694-4e79-b2a2-1fc2af193328:NORMAL:103w.103w.103w.103w:50010|RBW]]} size 0
2020-05-24 19:58:22,025 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap updated: 102n.102n.102n.102n:50010 is added to blk_1073742645_1835{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-efb40abd-a64b-413d-b9af-4f62d7390c0a:NORMAL:102n.102n.102n.102n:50010|RBW], ReplicaUC[[DISK]DS-9931cc42-d694-4e79-b2a2-1fc2af193328:NORMAL:103w.103w.103w.103w:50010|RBW]]} size 0
2020-05-24 19:58:22,028 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /benchmarks/TestDFSIO/io_control/in_file_test_io_4 is closed by DFSClient_NONMAPREDUCE_1115390510_1
2020-05-24 19:58:22,033 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1073742646_1836{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-efb40abd-a64b-413d-b9af-4f62d7390c0a:NORMAL:102n.102n.102n.102n:50010|RBW], ReplicaUC[[DISK]DS-9931cc42-d694-4e79-b2a2-1fc2af193328:NORMAL:103w.103w.103w.103w:50010|RBW]]} for /benchmarks/TestDFSIO/io_control/in_file_test_io_5
2020-05-24 19:58:22,179 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap updated: 103w.103w.103w.103w:50010 is added to blk_1073742646_1836{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-efb40abd-a64b-413d-b9af-4f62d7390c0a:NORMAL:102n.102n.102n.102n:50010|RBW], ReplicaUC[[DISK]DS-9931cc42-d694-4e79-b2a2-1fc2af193328:NORMAL:103w.103w.103w.103w:50010|RBW]]} size 0
2020-05-24 19:58:22,181 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap updated: 102n.102n.102n.102n:50010 is added to blk_1073742646_1836{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-efb40abd-a64b-413d-b9af-4f62d7390c0a:NORMAL:102n.102n.102n.102n:50010|RBW], ReplicaUC[[DISK]DS-9931cc42-d694-4e79-b2a2-1fc2af193328:NORMAL:103w.103w.103w.103w:50010|RBW]]} size 0
2020-05-24 19:58:22,183 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /benchmarks/TestDFSIO/io_control/in_file_test_io_5 is closed by DFSClient_NONMAPREDUCE_1115390510_1
2020-05-24 19:58:22,188 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1073742647_1837{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-efb40abd-a64b-413d-b9af-4f62d7390c0a:NORMAL:102n.102n.102n.102n:50010|RBW], ReplicaUC[[DISK]DS-9931cc42-d694-4e79-b2a2-1fc2af193328:NORMAL:103w.103w.103w.103w:50010|RBW]]} for /benchmarks/TestDFSIO/io_control/in_file_test_io_6
2020-05-24 19:58:22,346 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap updated: 103w.103w.103w.103w:50010 is added to blk_1073742647_1837{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-efb40abd-a64b-413d-b9af-4f62d7390c0a:NORMAL:102n.102n.102n.102n:50010|RBW], ReplicaUC[[DISK]DS-9931cc42-d694-4e79-b2a2-1fc2af193328:NORMAL:103w.103w.103w.103w:50010|RBW]]} size 0
2020-05-24 19:58:22,349 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap updated: 102n.102n.102n.102n:50010 is added to blk_1073742647_1837{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-efb40abd-a64b-413d-b9af-4f62d7390c0a:NORMAL:102n.102n.102n.102n:50010|RBW], ReplicaUC[[DISK]DS-9931cc42-d694-4e79-b2a2-1fc2af193328:NORMAL:103w.103w.103w.103w:50010|RBW]]} size 0
2020-05-24 19:58:22,351 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /benchmarks/TestDFSIO/io_control/in_file_test_io_6 is closed by DFSClient_NONMAPREDUCE_1115390510_1
2020-05-24 19:58:22,355 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1073742648_1838{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-efb40abd-a64b-413d-b9af-4f62d7390c0a:NORMAL:102n.102n.102n.102n:50010|RBW], ReplicaUC[[DISK]DS-9931cc42-d694-4e79-b2a2-1fc2af193328:NORMAL:103w.103w.103w.103w:50010|RBW]]} for /benchmarks/TestDFSIO/io_control/in_file_test_io_7
2020-05-24 19:58:22,504 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap updated: 103w.103w.103w.103w:50010 is added to blk_1073742648_1838{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-efb40abd-a64b-413d-b9af-4f62d7390c0a:NORMAL:102n.102n.102n.102n:50010|RBW], ReplicaUC[[DISK]DS-9931cc42-d694-4e79-b2a2-1fc2af193328:NORMAL:103w.103w.103w.103w:50010|RBW]]} size 0
2020-05-24 19:58:22,508 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap updated: 102n.102n.102n.102n:50010 is added to blk_1073742648_1838{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-efb40abd-a64b-413d-b9af-4f62d7390c0a:NORMAL:102n.102n.102n.102n:50010|RBW], ReplicaUC[[DISK]DS-9931cc42-d694-4e79-b2a2-1fc2af193328:NORMAL:103w.103w.103w.103w:50010|RBW]]} size 0
2020-05-24 19:58:22,511 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /benchmarks/TestDFSIO/io_control/in_file_test_io_7 is closed by DFSClient_NONMAPREDUCE_1115390510_1
2020-05-24 19:58:22,516 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1073742649_1839{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-efb40abd-a64b-413d-b9af-4f62d7390c0a:NORMAL:102n.102n.102n.102n:50010|RBW], ReplicaUC[[DISK]DS-9931cc42-d694-4e79-b2a2-1fc2af193328:NORMAL:103w.103w.103w.103w:50010|RBW]]} for /benchmarks/TestDFSIO/io_control/in_file_test_io_8
2020-05-24 19:58:22,663 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap updated: 103w.103w.103w.103w:50010 is added to blk_1073742649_1839{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-efb40abd-a64b-413d-b9af-4f62d7390c0a:NORMAL:102n.102n.102n.102n:50010|RBW], ReplicaUC[[DISK]DS-9931cc42-d694-4e79-b2a2-1fc2af193328:NORMAL:103w.103w.103w.103w:50010|RBW]]} size 0
2020-05-24 19:58:22,666 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap updated: 102n.102n.102n.102n:50010 is added to blk_1073742649_1839{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-efb40abd-a64b-413d-b9af-4f62d7390c0a:NORMAL:102n.102n.102n.102n:50010|RBW], ReplicaUC[[DISK]DS-9931cc42-d694-4e79-b2a2-1fc2af193328:NORMAL:103w.103w.103w.103w:50010|RBW]]} size 0
2020-05-24 19:58:22,668 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /benchmarks/TestDFSIO/io_control/in_file_test_io_8 is closed by DFSClient_NONMAPREDUCE_1115390510_1
2020-05-24 19:58:22,675 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1073742650_1840{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-efb40abd-a64b-413d-b9af-4f62d7390c0a:NORMAL:102n.102n.102n.102n:50010|RBW], ReplicaUC[[DISK]DS-42b062e0-208e-4f03-bf28-0b4ba31c3f1b:NORMAL:104w.104w.104w.104w:50010|RBW]]} for /benchmarks/TestDFSIO/io_control/in_file_test_io_9
2020-05-24 19:58:22,693 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap updated: 104w.104w.104w.104w:50010 is added to blk_1073742650_1840{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-efb40abd-a64b-413d-b9af-4f62d7390c0a:NORMAL:102n.102n.102n.102n:50010|RBW], ReplicaUC[[DISK]DS-42b062e0-208e-4f03-bf28-0b4ba31c3f1b:NORMAL:104w.104w.104w.104w:50010|RBW]]} size 0
2020-05-24 19:58:22,695 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap updated: 102n.102n.102n.102n:50010 is added to blk_1073742650_1840{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-efb40abd-a64b-413d-b9af-4f62d7390c0a:NORMAL:102n.102n.102n.102n:50010|RBW], ReplicaUC[[DISK]DS-42b062e0-208e-4f03-bf28-0b4ba31c3f1b:NORMAL:104w.104w.104w.104w:50010|RBW]]} size 0
2020-05-24 19:58:22,697 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /benchmarks/TestDFSIO/io_control/in_file_test_io_9 is closed by DFSClient_NONMAPREDUCE_1115390510_1
2020-05-24 19:58:23,403 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1073742651_1841{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-efb40abd-a64b-413d-b9af-4f62d7390c0a:NORMAL:102n.102n.102n.102n:50010|RBW], ReplicaUC[[DISK]DS-42b062e0-208e-4f03-bf28-0b4ba31c3f1b:NORMAL:104w.104w.104w.104w:50010|RBW]]} for /tmp/hadoop-yarn/staging/root/.staging/job_1590199135215_0002/job.jar
2020-05-24 19:58:34,136 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap updated: 104w.104w.104w.104w:50010 is added to blk_1073742651_1841{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-efb40abd-a64b-413d-b9af-4f62d7390c0a:NORMAL:102n.102n.102n.102n:50010|RBW], ReplicaUC[[DISK]DS-42b062e0-208e-4f03-bf28-0b4ba31c3f1b:NORMAL:104w.104w.104w.104w:50010|RBW]]} size 0
2020-05-24 19:58:34,137 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap updated: 102n.102n.102n.102n:50010 is added to blk_1073742651_1841{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-efb40abd-a64b-413d-b9af-4f62d7390c0a:NORMAL:102n.102n.102n.102n:50010|RBW], ReplicaUC[[DISK]DS-42b062e0-208e-4f03-bf28-0b4ba31c3f1b:NORMAL:104w.104w.104w.104w:50010|RBW]]} size 0
2020-05-24 19:58:34,142 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /tmp/hadoop-yarn/staging/root/.staging/job_1590199135215_0002/job.jar is closed by DFSClient_NONMAPREDUCE_1115390510_1
2020-05-24 19:58:34,145 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Increasing replication from 2 to 10 for /tmp/hadoop-yarn/staging/root/.staging/job_1590199135215_0002/job.jar
2020-05-24 19:58:34,188 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Increasing replication from 2 to 10 for /tmp/hadoop-yarn/staging/root/.staging/job_1590199135215_0002/job.split
2020-05-24 19:58:34,194 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1073742652_1842{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-efb40abd-a64b-413d-b9af-4f62d7390c0a:NORMAL:102n.102n.102n.102n:50010|RBW], ReplicaUC[[DISK]DS-9931cc42-d694-4e79-b2a2-1fc2af193328:NORMAL:103w.103w.103w.103w:50010|RBW], ReplicaUC[[DISK]DS-42b062e0-208e-4f03-bf28-0b4ba31c3f1b:NORMAL:104w.104w.104w.104w:50010|RBW]]} for /tmp/hadoop-yarn/staging/root/.staging/job_1590199135215_0002/job.split
2020-05-24 19:58:34,466 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap updated: 104w.104w.104w.104w:50010 is added to blk_1073742652_1842{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-efb40abd-a64b-413d-b9af-4f62d7390c0a:NORMAL:102n.102n.102n.102n:50010|RBW], ReplicaUC[[DISK]DS-9931cc42-d694-4e79-b2a2-1fc2af193328:NORMAL:103w.103w.103w.103w:50010|RBW], ReplicaUC[[DISK]DS-42b062e0-208e-4f03-bf28-0b4ba31c3f1b:NORMAL:104w.104w.104w.104w:50010|RBW]]} size 0
2020-05-24 19:58:34,503 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap updated: 103w.103w.103w.103w:50010 is added to blk_1073742652_1842{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-efb40abd-a64b-413d-b9af-4f62d7390c0a:NORMAL:102n.102n.102n.102n:50010|RBW], ReplicaUC[[DISK]DS-9931cc42-d694-4e79-b2a2-1fc2af193328:NORMAL:103w.103w.103w.103w:50010|RBW], ReplicaUC[[DISK]DS-42b062e0-208e-4f03-bf28-0b4ba31c3f1b:NORMAL:104w.104w.104w.104w:50010|RBW]]} size 0
2020-05-24 19:58:34,505 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap updated: 102n.102n.102n.102n:50010 is added to blk_1073742652_1842{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-efb40abd-a64b-413d-b9af-4f62d7390c0a:NORMAL:102n.102n.102n.102n:50010|RBW], ReplicaUC[[DISK]DS-9931cc42-d694-4e79-b2a2-1fc2af193328:NORMAL:103w.103w.103w.103w:50010|RBW], ReplicaUC[[DISK]DS-42b062e0-208e-4f03-bf28-0b4ba31c3f1b:NORMAL:104w.104w.104w.104w:50010|RBW]]} size 0
2020-05-24 19:58:34,507 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /tmp/hadoop-yarn/staging/root/.staging/job_1590199135215_0002/job.split is closed by DFSClient_NONMAPREDUCE_1115390510_1
2020-05-24 19:58:34,516 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1073742653_1843{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-efb40abd-a64b-413d-b9af-4f62d7390c0a:NORMAL:102n.102n.102n.102n:50010|RBW], ReplicaUC[[DISK]DS-9931cc42-d694-4e79-b2a2-1fc2af193328:NORMAL:103w.103w.103w.103w:50010|RBW]]} for /tmp/hadoop-yarn/staging/root/.staging/job_1590199135215_0002/job.splitmetainfo
2020-05-24 19:58:34,682 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap updated: 103w.103w.103w.103w:50010 is added to blk_1073742653_1843{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-efb40abd-a64b-413d-b9af-4f62d7390c0a:NORMAL:102n.102n.102n.102n:50010|RBW], ReplicaUC[[DISK]DS-9931cc42-d694-4e79-b2a2-1fc2af193328:NORMAL:103w.103w.103w.103w:50010|RBW]]} size 0
2020-05-24 19:58:34,684 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap updated: 102n.102n.102n.102n:50010 is added to blk_1073742653_1843{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-efb40abd-a64b-413d-b9af-4f62d7390c0a:NORMAL:102n.102n.102n.102n:50010|RBW], ReplicaUC[[DISK]DS-9931cc42-d694-4e79-b2a2-1fc2af193328:NORMAL:103w.103w.103w.103w:50010|RBW]]} size 0
2020-05-24 19:58:34,687 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /tmp/hadoop-yarn/staging/root/.staging/job_1590199135215_0002/job.splitmetainfo is closed by DFSClient_NONMAPREDUCE_1115390510_1
2020-05-24 19:58:34,797 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1073742654_1844{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-efb40abd-a64b-413d-b9af-4f62d7390c0a:NORMAL:102n.102n.102n.102n:50010|RBW], ReplicaUC[[DISK]DS-42b062e0-208e-4f03-bf28-0b4ba31c3f1b:NORMAL:104w.104w.104w.104w:50010|RBW]]} for /tmp/hadoop-yarn/staging/root/.staging/job_1590199135215_0002/job.xml
2020-05-24 19:58:35,050 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap updated: 104w.104w.104w.104w:50010 is added to blk_1073742654_1844{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-efb40abd-a64b-413d-b9af-4f62d7390c0a:NORMAL:102n.102n.102n.102n:50010|RBW], ReplicaUC[[DISK]DS-42b062e0-208e-4f03-bf28-0b4ba31c3f1b:NORMAL:104w.104w.104w.104w:50010|RBW]]} size 0
2020-05-24 19:58:35,053 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap updated: 102n.102n.102n.102n:50010 is added to blk_1073742654_1844{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-efb40abd-a64b-413d-b9af-4f62d7390c0a:NORMAL:102n.102n.102n.102n:50010|RBW], ReplicaUC[[DISK]DS-42b062e0-208e-4f03-bf28-0b4ba31c3f1b:NORMAL:104w.104w.104w.104w:50010|RBW]]} size 0
2020-05-24 19:58:35,055 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /tmp/hadoop-yarn/staging/root/.staging/job_1590199135215_0002/job.xml is closed by DFSClient_NONMAPREDUCE_1115390510_1
2020-05-24 19:58:37,289 WARN org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: PendingReplicationMonitor timed out blk_1073741835_1011
2020-05-24 19:58:37,290 WARN org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: PendingReplicationMonitor timed out blk_1073741837_1013
2020-05-24 19:58:37,290 WARN org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: PendingReplicationMonitor timed out blk_1073741839_1015
2020-05-24 19:58:37,290 WARN org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: PendingReplicationMonitor timed out blk_1073741841_1017
2020-05-24 19:58:37,290 WARN org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: PendingReplicationMonitor timed out blk_1073741844_1020
2020-05-24 19:58:39,628 INFO BlockStateChange: BLOCK* ask 104w.104w.104w.104w:50010 to replicate blk_1073742651_1841 to datanode(s) 103w.103w.103w.103w:50010
2020-05-24 19:58:45,628 INFO BlockStateChange: BLOCK* ask 102n.102n.102n.102n:50010 to replicate blk_1073742651_1841 to datanode(s) 103w.103w.103w.103w:50010
2020-05-24 19:58:45,628 INFO BlockStateChange: BLOCK* ask 103w.103w.103w.103w:50010 to replicate blk_1073741835_1011 to datanode(s) 102n.102n.102n.102n:50010
2020-05-24 19:58:45,628 INFO BlockStateChange: BLOCK* ask 104w.104w.104w.104w:50010 to replicate blk_1073741837_1013 to datanode(s) 102n.102n.102n.102n:50010
2020-05-24 19:58:45,628 INFO BlockStateChange: BLOCK* ask 104w.104w.104w.104w:50010 to replicate blk_1073741839_1015 to datanode(s) 102n.102n.102n.102n:50010
2020-05-24 19:58:48,629 INFO BlockStateChange: BLOCK* ask 104w.104w.104w.104w:50010 to replicate blk_1073741844_1020 to datanode(s) 102n.102n.102n.102n:50010
2020-05-24 19:58:52,839 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap updated: 103w.103w.103w.103w:50010 is added to blk_1073742651_1841 size 1534215
2020-05-24 19:58:54,636 INFO BlockStateChange: BLOCK* ask 103w.103w.103w.103w:50010 to replicate blk_1073741841_1017 to datanode(s) 102n.102n.102n.102n:50010
2020-05-24 19:59:53,413 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 144 Total time for transactions(ms): 5 Number of transactions batched in Syncs: 3 Number of syncs: 92 SyncTimes(ms): 109 
2020-05-24 19:59:53,715 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1073742655_1845{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-9931cc42-d694-4e79-b2a2-1fc2af193328:NORMAL:103w.103w.103w.103w:50010|RBW], ReplicaUC[[DISK]DS-efb40abd-a64b-413d-b9af-4f62d7390c0a:NORMAL:102n.102n.102n.102n:50010|RBW]]} for /tmp/hadoop-yarn/staging/root/.staging/job_1590199135215_0002/job_1590199135215_0002_1_conf.xml
2020-05-24 20:00:11,714 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1073742656_1846{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-9931cc42-d694-4e79-b2a2-1fc2af193328:NORMAL:103w.103w.103w.103w:50010|RBW], ReplicaUC[[DISK]DS-42b062e0-208e-4f03-bf28-0b4ba31c3f1b:NORMAL:104w.104w.104w.104w:50010|RBW]]} for /benchmarks/TestDFSIO/io_data/test_io_5
2020-05-24 20:00:12,663 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1073742657_1847{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-9931cc42-d694-4e79-b2a2-1fc2af193328:NORMAL:103w.103w.103w.103w:50010|RBW], ReplicaUC[[DISK]DS-42b062e0-208e-4f03-bf28-0b4ba31c3f1b:NORMAL:104w.104w.104w.104w:50010|RBW]]} for /benchmarks/TestDFSIO/io_data/test_io_6
2020-05-24 20:00:12,668 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1073742658_1848{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-9931cc42-d694-4e79-b2a2-1fc2af193328:NORMAL:103w.103w.103w.103w:50010|RBW], ReplicaUC[[DISK]DS-42b062e0-208e-4f03-bf28-0b4ba31c3f1b:NORMAL:104w.104w.104w.104w:50010|RBW]]} for /benchmarks/TestDFSIO/io_data/test_io_3
2020-05-24 20:00:12,941 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1073742659_1849{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-9931cc42-d694-4e79-b2a2-1fc2af193328:NORMAL:103w.103w.103w.103w:50010|RBW], ReplicaUC[[DISK]DS-efb40abd-a64b-413d-b9af-4f62d7390c0a:NORMAL:102n.102n.102n.102n:50010|RBW]]} for /benchmarks/TestDFSIO/io_data/test_io_7
2020-05-24 20:00:12,948 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1073742660_1850{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-9931cc42-d694-4e79-b2a2-1fc2af193328:NORMAL:103w.103w.103w.103w:50010|RBW], ReplicaUC[[DISK]DS-42b062e0-208e-4f03-bf28-0b4ba31c3f1b:NORMAL:104w.104w.104w.104w:50010|RBW]]} for /benchmarks/TestDFSIO/io_data/test_io_8
2020-05-24 20:00:13,183 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1073742661_1851{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-9931cc42-d694-4e79-b2a2-1fc2af193328:NORMAL:103w.103w.103w.103w:50010|RBW], ReplicaUC[[DISK]DS-42b062e0-208e-4f03-bf28-0b4ba31c3f1b:NORMAL:104w.104w.104w.104w:50010|RBW]]} for /benchmarks/TestDFSIO/io_data/test_io_4
2020-05-24 20:00:20,779 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from 104w.104w.104w.104w
2020-05-24 20:00:20,779 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
2020-05-24 20:00:20,779 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 8087
2020-05-24 20:00:20,780 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 177 Total time for transactions(ms): 5 Number of transactions batched in Syncs: 3 Number of syncs: 109 SyncTimes(ms): 128 
2020-05-24 20:00:20,781 INFO org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits file /opt/module/hadoop-2.7.2/data/tmp/dfs/name/current/edits_inprogress_0000000000000008087 -> /opt/module/hadoop-2.7.2/data/tmp/dfs/name/current/edits_0000000000000008087-0000000000000008263
2020-05-24 20:00:20,781 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at 8264
2020-05-24 20:00:20,870 INFO org.apache.hadoop.hdfs.server.namenode.TransferFsImage: Transfer took 0.01s at 5000.00 KB/s
2020-05-24 20:00:20,870 INFO org.apache.hadoop.hdfs.server.namenode.TransferFsImage: Downloaded file fsimage.ckpt_0000000000000008263 size 26182 bytes.
2020-05-24 20:00:20,874 INFO org.apache.hadoop.hdfs.server.namenode.NNStorageRetentionManager: Going to retain 2 images with txid >= 8086
2020-05-24 20:00:20,874 INFO org.apache.hadoop.hdfs.server.namenode.NNStorageRetentionManager: Purging old image FSImageFile(file=/opt/module/hadoop-2.7.2/data/tmp/dfs/name/current/fsimage_0000000000000008038, cpktTxId=0000000000000008038)
2020-05-24 20:00:56,927 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1073742662_1852{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-9931cc42-d694-4e79-b2a2-1fc2af193328:NORMAL:103w.103w.103w.103w:50010|RBW], ReplicaUC[[DISK]DS-42b062e0-208e-4f03-bf28-0b4ba31c3f1b:NORMAL:104w.104w.104w.104w:50010|RBW]]} for /tmp/hadoop-yarn/staging/root/.staging/job_1590199135215_0002/job_1590199135215_0002_1_conf.xml
2020-05-24 20:01:16,318 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1073742663_1853{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-9931cc42-d694-4e79-b2a2-1fc2af193328:NORMAL:103w.103w.103w.103w:50010|RBW], ReplicaUC[[DISK]DS-42b062e0-208e-4f03-bf28-0b4ba31c3f1b:NORMAL:104w.104w.104w.104w:50010|RBW]]} for /benchmarks/TestDFSIO/io_data/test_io_7
2020-05-24 20:01:53,606 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 10 Total time for transactions(ms): 4 Number of transactions batched in Syncs: 0 Number of syncs: 6 SyncTimes(ms): 15 
2020-05-24 20:01:53,613 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1073742664_1854{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-42b062e0-208e-4f03-bf28-0b4ba31c3f1b:NORMAL:104w.104w.104w.104w:50010|RBW], ReplicaUC[[DISK]DS-efb40abd-a64b-413d-b9af-4f62d7390c0a:NORMAL:102n.102n.102n.102n:50010|RBW]]} for /HBase/WALs/hadoop104,16020,1590147107908/hadoop104%2C16020%2C1590147107908.1590321713592
2020-05-24 20:02:14,266 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: updatePipeline(blk_1073742656_1846, newGS=1855, newLength=387072, newNodes=[104w.104w.104w.104w:50010], client=DFSClient_attempt_1590199135215_0002_m_000005_0_-62252113_1)
2020-05-24 20:02:14,266 INFO BlockStateChange: BLOCK* Removing stale replica from location: [DISK]DS-9931cc42-d694-4e79-b2a2-1fc2af193328:NORMAL:103w.103w.103w.103w:50010
2020-05-24 20:02:14,266 INFO BlockStateChange: BLOCK* Removing stale replica from location: [DISK]DS-42b062e0-208e-4f03-bf28-0b4ba31c3f1b:NORMAL:104w.104w.104w.104w:50010
2020-05-24 20:02:14,267 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: updatePipeline(blk_1073742656_1846 => blk_1073742656_1855) success
2020-05-24 20:02:42,894 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: updatePipeline(blk_1073742662_1852, newGS=1856, newLength=64512, newNodes=[104w.104w.104w.104w:50010], client=DFSClient_NONMAPREDUCE_-899541460_1)
2020-05-24 20:02:42,895 INFO BlockStateChange: BLOCK* Removing stale replica from location: [DISK]DS-9931cc42-d694-4e79-b2a2-1fc2af193328:NORMAL:103w.103w.103w.103w:50010
2020-05-24 20:02:42,895 INFO BlockStateChange: BLOCK* Removing stale replica from location: [DISK]DS-42b062e0-208e-4f03-bf28-0b4ba31c3f1b:NORMAL:104w.104w.104w.104w:50010
2020-05-24 20:02:42,896 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: updatePipeline(blk_1073742662_1852 => blk_1073742662_1856) success
2020-05-24 20:02:44,567 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap updated: 104w.104w.104w.104w:50010 is added to blk_1073742662_1856{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-42b062e0-208e-4f03-bf28-0b4ba31c3f1b:NORMAL:104w.104w.104w.104w:50010|RBW]]} size 64512
2020-05-24 20:02:44,604 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /tmp/hadoop-yarn/staging/root/.staging/job_1590199135215_0002/job_1590199135215_0002_1_conf.xml is closed by DFSClient_NONMAPREDUCE_-899541460_1
2020-05-24 20:02:45,676 INFO BlockStateChange: BLOCK* ask 104w.104w.104w.104w:50010 to replicate blk_1073742662_1856 to datanode(s) 103w.103w.103w.103w:50010
2020-05-24 20:02:48,509 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap updated: 103w.103w.103w.103w:50010 is added to blk_1073742662_1856 size 116369
2020-05-24 20:02:56,634 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 19 Total time for transactions(ms): 4 Number of transactions batched in Syncs: 0 Number of syncs: 13 SyncTimes(ms): 22 
2020-05-24 20:02:56,639 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1073742665_1857{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-42b062e0-208e-4f03-bf28-0b4ba31c3f1b:NORMAL:104w.104w.104w.104w:50010|RBW], ReplicaUC[[DISK]DS-9931cc42-d694-4e79-b2a2-1fc2af193328:NORMAL:103w.103w.103w.103w:50010|RBW]]} for /HBase/WALs/hadoop104,16020,1590147107908/hadoop104%2C16020%2C1590147107908.1590321713592
2020-05-24 20:02:56,763 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* fsync: /HBase/WALs/hadoop104,16020,1590147107908/hadoop104%2C16020%2C1590147107908.1590321713592 for DFSClient_NONMAPREDUCE_681596530_1
2020-05-24 20:02:56,854 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap updated: 103w.103w.103w.103w:50010 is added to blk_1073742634_1824{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-42b062e0-208e-4f03-bf28-0b4ba31c3f1b:NORMAL:104w.104w.104w.104w:50010|RBW], ReplicaUC[[DISK]DS-9931cc42-d694-4e79-b2a2-1fc2af193328:NORMAL:103w.103w.103w.103w:50010|RBW]]} size 83
2020-05-24 20:02:56,865 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap updated: 104w.104w.104w.104w:50010 is added to blk_1073742634_1824{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-42b062e0-208e-4f03-bf28-0b4ba31c3f1b:NORMAL:104w.104w.104w.104w:50010|RBW], ReplicaUC[[DISK]DS-9931cc42-d694-4e79-b2a2-1fc2af193328:NORMAL:103w.103w.103w.103w:50010|RBW]]} size 83
2020-05-24 20:02:56,867 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /HBase/WALs/hadoop104,16020,1590147107908/hadoop104%2C16020%2C1590147107908.1590318110255 is closed by DFSClient_NONMAPREDUCE_681596530_1
2020-05-24 20:04:01,388 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 27 Total time for transactions(ms): 4 Number of transactions batched in Syncs: 0 Number of syncs: 19 SyncTimes(ms): 27 
2020-05-24 20:04:01,518 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: updatePipeline(blk_1073742660_1850, newGS=1858, newLength=1870848, newNodes=[104w.104w.104w.104w:50010], client=DFSClient_attempt_1590199135215_0002_m_000008_0_-1520775049_1)
2020-05-24 20:04:01,518 INFO BlockStateChange: BLOCK* Removing stale replica from location: [DISK]DS-9931cc42-d694-4e79-b2a2-1fc2af193328:NORMAL:103w.103w.103w.103w:50010
2020-05-24 20:04:01,518 INFO BlockStateChange: BLOCK* Removing stale replica from location: [DISK]DS-42b062e0-208e-4f03-bf28-0b4ba31c3f1b:NORMAL:104w.104w.104w.104w:50010
2020-05-24 20:04:01,520 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: updatePipeline(blk_1073742660_1850 => blk_1073742660_1858) success
2020-05-24 20:07:48,601 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 29 Total time for transactions(ms): 4 Number of transactions batched in Syncs: 0 Number of syncs: 21 SyncTimes(ms): 29 
2020-05-24 20:07:48,655 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: updatePipeline(blk_1073742663_1853, newGS=1859, newLength=1999872, newNodes=[103w.103w.103w.103w:50010], client=DFSClient_attempt_1590199135215_0002_m_000007_0_815438823_1)
2020-05-24 20:07:48,655 INFO BlockStateChange: BLOCK* Removing stale replica from location: [DISK]DS-9931cc42-d694-4e79-b2a2-1fc2af193328:NORMAL:103w.103w.103w.103w:50010
2020-05-24 20:07:48,655 INFO BlockStateChange: BLOCK* Removing stale replica from location: [DISK]DS-42b062e0-208e-4f03-bf28-0b4ba31c3f1b:NORMAL:104w.104w.104w.104w:50010
2020-05-24 20:07:48,656 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: updatePipeline(blk_1073742663_1853 => blk_1073742663_1859) success
2020-05-24 20:08:37,290 WARN org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: PendingReplicationMonitor timed out blk_1073741835_1011
2020-05-24 20:08:37,290 WARN org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: PendingReplicationMonitor timed out blk_1073741837_1013
2020-05-24 20:08:37,290 WARN org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: PendingReplicationMonitor timed out blk_1073741839_1015
2020-05-24 20:08:37,290 WARN org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: PendingReplicationMonitor timed out blk_1073741841_1017
2020-05-24 20:08:37,290 WARN org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: PendingReplicationMonitor timed out blk_1073741844_1020
2020-05-24 20:08:37,290 WARN org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: PendingReplicationMonitor timed out blk_1073742651_1841
2020-05-24 20:08:42,748 INFO BlockStateChange: BLOCK* ask 103w.103w.103w.103w:50010 to replicate blk_1073741835_1011 to datanode(s) 102n.102n.102n.102n:50010
2020-05-24 20:08:42,748 INFO BlockStateChange: BLOCK* ask 104w.104w.104w.104w:50010 to replicate blk_1073741837_1013 to datanode(s) 102n.102n.102n.102n:50010
2020-05-24 20:08:42,748 INFO BlockStateChange: BLOCK* ask 103w.103w.103w.103w:50010 to replicate blk_1073741839_1015 to datanode(s) 102n.102n.102n.102n:50010
2020-05-24 20:08:45,748 INFO BlockStateChange: BLOCK* ask 103w.103w.103w.103w:50010 to replicate blk_1073741844_1020 to datanode(s) 102n.102n.102n.102n:50010
2020-05-24 20:08:51,756 INFO BlockStateChange: BLOCK* ask 104w.104w.104w.104w:50010 to replicate blk_1073741841_1017 to datanode(s) 102n.102n.102n.102n:50010
2020-05-24 20:10:23,550 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 30 Total time for transactions(ms): 4 Number of transactions batched in Syncs: 0 Number of syncs: 23 SyncTimes(ms): 32 
2020-05-24 20:10:23,625 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1073742666_1860{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-9931cc42-d694-4e79-b2a2-1fc2af193328:NORMAL:103w.103w.103w.103w:50010|RBW], ReplicaUC[[DISK]DS-efb40abd-a64b-413d-b9af-4f62d7390c0a:NORMAL:102n.102n.102n.102n:50010|RBW]]} for /tmp/hadoop-yarn/staging/root/.staging/job_1590199135215_0002/job_1590199135215_0002_1.jhist
2020-05-24 20:11:26,670 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 34 Total time for transactions(ms): 4 Number of transactions batched in Syncs: 4 Number of syncs: 24 SyncTimes(ms): 34 
2020-05-24 20:11:26,709 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1073742667_1861{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-9931cc42-d694-4e79-b2a2-1fc2af193328:NORMAL:103w.103w.103w.103w:50010|RBW], ReplicaUC[[DISK]DS-42b062e0-208e-4f03-bf28-0b4ba31c3f1b:NORMAL:104w.104w.104w.104w:50010|RBW]]} for /tmp/hadoop-yarn/staging/root/.staging/job_1590199135215_0002/job_1590199135215_0002_1.jhist
2020-05-24 20:11:28,566 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* fsync: /tmp/hadoop-yarn/staging/root/.staging/job_1590199135215_0002/job_1590199135215_0002_1.jhist for DFSClient_NONMAPREDUCE_-899541460_1
2020-05-24 20:13:43,840 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 38 Total time for transactions(ms): 4 Number of transactions batched in Syncs: 4 Number of syncs: 27 SyncTimes(ms): 38 
2020-05-24 20:13:48,950 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1073742668_1862{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-9931cc42-d694-4e79-b2a2-1fc2af193328:NORMAL:103w.103w.103w.103w:50010|RBW], ReplicaUC[[DISK]DS-42b062e0-208e-4f03-bf28-0b4ba31c3f1b:NORMAL:104w.104w.104w.104w:50010|RBW]]} for /benchmarks/TestDFSIO/io_data/test_io_7
2020-05-24 20:13:54,825 INFO BlockStateChange: BLOCK* addToInvalidates: blk_1073742634_1824 104w.104w.104w.104w:50010 103w.103w.103w.103w:50010 
2020-05-24 20:13:57,806 INFO BlockStateChange: BLOCK* BlockManager: ask 104w.104w.104w.104w:50010 to delete [blk_1073742634_1824]
2020-05-24 20:14:00,807 INFO BlockStateChange: BLOCK* BlockManager: ask 103w.103w.103w.103w:50010 to delete [blk_1073742634_1824]
2020-05-24 20:16:37,667 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 44 Total time for transactions(ms): 4 Number of transactions batched in Syncs: 6 Number of syncs: 30 SyncTimes(ms): 41 
2020-05-24 20:16:37,786 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: updatePipeline(blk_1073742661_1851, newGS=1863, newLength=10644480, newNodes=[104w.104w.104w.104w:50010], client=DFSClient_attempt_1590199135215_0002_m_000004_0_-45857330_1)
2020-05-24 20:16:37,786 INFO BlockStateChange: BLOCK* Removing stale replica from location: [DISK]DS-9931cc42-d694-4e79-b2a2-1fc2af193328:NORMAL:103w.103w.103w.103w:50010
2020-05-24 20:16:37,786 INFO BlockStateChange: BLOCK* Removing stale replica from location: [DISK]DS-42b062e0-208e-4f03-bf28-0b4ba31c3f1b:NORMAL:104w.104w.104w.104w:50010
2020-05-24 20:16:37,787 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: updatePipeline(blk_1073742661_1851 => blk_1073742661_1863) success
2020-05-24 20:18:37,290 WARN org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: PendingReplicationMonitor timed out blk_1073741835_1011
2020-05-24 20:18:37,290 WARN org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: PendingReplicationMonitor timed out blk_1073741837_1013
2020-05-24 20:18:37,290 WARN org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: PendingReplicationMonitor timed out blk_1073741839_1015
2020-05-24 20:18:37,290 WARN org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: PendingReplicationMonitor timed out blk_1073741841_1017
2020-05-24 20:18:37,290 WARN org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: PendingReplicationMonitor timed out blk_1073741844_1020
2020-05-24 20:18:45,833 INFO BlockStateChange: BLOCK* ask 104w.104w.104w.104w:50010 to replicate blk_1073741835_1011 to datanode(s) 102n.102n.102n.102n:50010
2020-05-24 20:18:45,833 INFO BlockStateChange: BLOCK* ask 104w.104w.104w.104w:50010 to replicate blk_1073741837_1013 to datanode(s) 102n.102n.102n.102n:50010
2020-05-24 20:18:45,833 INFO BlockStateChange: BLOCK* ask 103w.103w.103w.103w:50010 to replicate blk_1073741839_1015 to datanode(s) 102n.102n.102n.102n:50010
2020-05-24 20:18:48,834 INFO BlockStateChange: BLOCK* ask 103w.103w.103w.103w:50010 to replicate blk_1073741844_1020 to datanode(s) 102n.102n.102n.102n:50010
2020-05-24 20:18:54,834 INFO BlockStateChange: BLOCK* ask 103w.103w.103w.103w:50010 to replicate blk_1073741841_1017 to datanode(s) 102n.102n.102n.102n:50010
2020-05-24 20:20:53,514 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 45 Total time for transactions(ms): 4 Number of transactions batched in Syncs: 6 Number of syncs: 32 SyncTimes(ms): 43 
2020-05-24 20:24:54,269 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 46 Total time for transactions(ms): 4 Number of transactions batched in Syncs: 10 Number of syncs: 32 SyncTimes(ms): 43 
2020-05-24 20:24:54,389 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: updatePipeline(blk_1073742657_1847, newGS=1864, newLength=11418624, newNodes=[104w.104w.104w.104w:50010], client=DFSClient_attempt_1590199135215_0002_m_000006_0_-1645513451_1)
2020-05-24 20:24:54,389 INFO BlockStateChange: BLOCK* Removing stale replica from location: [DISK]DS-9931cc42-d694-4e79-b2a2-1fc2af193328:NORMAL:103w.103w.103w.103w:50010
2020-05-24 20:24:54,389 INFO BlockStateChange: BLOCK* Removing stale replica from location: [DISK]DS-42b062e0-208e-4f03-bf28-0b4ba31c3f1b:NORMAL:104w.104w.104w.104w:50010
2020-05-24 20:24:54,390 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: updatePipeline(blk_1073742657_1847 => blk_1073742657_1864) success
2020-05-24 20:28:37,291 WARN org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: PendingReplicationMonitor timed out blk_1073741835_1011
2020-05-24 20:28:37,291 WARN org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: PendingReplicationMonitor timed out blk_1073741837_1013
2020-05-24 20:28:37,291 WARN org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: PendingReplicationMonitor timed out blk_1073741839_1015
2020-05-24 20:28:37,291 WARN org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: PendingReplicationMonitor timed out blk_1073741841_1017
2020-05-24 20:28:37,291 WARN org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: PendingReplicationMonitor timed out blk_1073741844_1020
2020-05-24 20:28:42,954 INFO BlockStateChange: BLOCK* ask 103w.103w.103w.103w:50010 to replicate blk_1073741835_1011 to datanode(s) 102n.102n.102n.102n:50010
2020-05-24 20:28:42,954 INFO BlockStateChange: BLOCK* ask 104w.104w.104w.104w:50010 to replicate blk_1073741837_1013 to datanode(s) 102n.102n.102n.102n:50010
2020-05-24 20:28:42,954 INFO BlockStateChange: BLOCK* ask 104w.104w.104w.104w:50010 to replicate blk_1073741839_1015 to datanode(s) 102n.102n.102n.102n:50010
2020-05-24 20:28:45,954 INFO BlockStateChange: BLOCK* ask 104w.104w.104w.104w:50010 to replicate blk_1073741844_1020 to datanode(s) 102n.102n.102n.102n:50010
2020-05-24 20:28:51,955 INFO BlockStateChange: BLOCK* ask 103w.103w.103w.103w:50010 to replicate blk_1073741841_1017 to datanode(s) 102n.102n.102n.102n:50010
2020-05-24 20:29:11,971 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 48 Total time for transactions(ms): 4 Number of transactions batched in Syncs: 10 Number of syncs: 34 SyncTimes(ms): 45 
2020-05-24 20:29:12,098 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: updatePipeline(blk_1073742658_1848, newGS=1865, newLength=15482880, newNodes=[104w.104w.104w.104w:50010], client=DFSClient_attempt_1590199135215_0002_m_000003_0_417532496_1)
2020-05-24 20:29:12,098 INFO BlockStateChange: BLOCK* Removing stale replica from location: [DISK]DS-9931cc42-d694-4e79-b2a2-1fc2af193328:NORMAL:103w.103w.103w.103w:50010
2020-05-24 20:29:12,098 INFO BlockStateChange: BLOCK* Removing stale replica from location: [DISK]DS-42b062e0-208e-4f03-bf28-0b4ba31c3f1b:NORMAL:104w.104w.104w.104w:50010
2020-05-24 20:29:12,100 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: updatePipeline(blk_1073742658_1848 => blk_1073742658_1865) success
2020-05-24 20:31:23,522 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 49 Total time for transactions(ms): 4 Number of transactions batched in Syncs: 10 Number of syncs: 36 SyncTimes(ms): 47 
2020-05-24 20:32:15,795 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1073742669_1866{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-efb40abd-a64b-413d-b9af-4f62d7390c0a:NORMAL:102n.102n.102n.102n:50010|RBW], ReplicaUC[[DISK]DS-42b062e0-208e-4f03-bf28-0b4ba31c3f1b:NORMAL:104w.104w.104w.104w:50010|RBW]]} for /HBase/WALs/hadoop102,16020,1590147108899/hadoop102%2C16020%2C1590147108899.1590323535789
2020-05-24 20:32:15,816 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* fsync: /HBase/WALs/hadoop102,16020,1590147108899/hadoop102%2C16020%2C1590147108899.1590323535789 for DFSClient_NONMAPREDUCE_-1294288672_1
2020-05-24 20:32:15,827 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap updated: 104w.104w.104w.104w:50010 is added to blk_1073742635_1825{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-efb40abd-a64b-413d-b9af-4f62d7390c0a:NORMAL:102n.102n.102n.102n:50010|RBW], ReplicaUC[[DISK]DS-42b062e0-208e-4f03-bf28-0b4ba31c3f1b:NORMAL:104w.104w.104w.104w:50010|RBW]]} size 83
2020-05-24 20:32:15,828 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap updated: 102n.102n.102n.102n:50010 is added to blk_1073742635_1825{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-efb40abd-a64b-413d-b9af-4f62d7390c0a:NORMAL:102n.102n.102n.102n:50010|RBW], ReplicaUC[[DISK]DS-42b062e0-208e-4f03-bf28-0b4ba31c3f1b:NORMAL:104w.104w.104w.104w:50010|RBW]]} size 83
2020-05-24 20:32:15,834 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /HBase/WALs/hadoop102,16020,1590147108899/hadoop102%2C16020%2C1590147108899.1590319935605 is closed by DFSClient_NONMAPREDUCE_-1294288672_1
2020-05-24 20:33:34,089 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 58 Total time for transactions(ms): 4 Number of transactions batched in Syncs: 14 Number of syncs: 42 SyncTimes(ms): 56 
2020-05-24 20:33:34,134 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1073742670_1867{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-9931cc42-d694-4e79-b2a2-1fc2af193328:NORMAL:103w.103w.103w.103w:50010|RBW], ReplicaUC[[DISK]DS-42b062e0-208e-4f03-bf28-0b4ba31c3f1b:NORMAL:104w.104w.104w.104w:50010|RBW]]} for /HBase/WALs/hadoop103,16020,1590147108408/hadoop103%2C16020%2C1590147108408.meta.1590323613940.meta
2020-05-24 20:33:34,330 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* fsync: /HBase/WALs/hadoop103,16020,1590147108408/hadoop103%2C16020%2C1590147108408.meta.1590323613940.meta for DFSClient_NONMAPREDUCE_1857522783_1
2020-05-24 20:33:34,412 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap updated: 104w.104w.104w.104w:50010 is added to blk_1073742638_1828{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-9931cc42-d694-4e79-b2a2-1fc2af193328:NORMAL:103w.103w.103w.103w:50010|RBW], ReplicaUC[[DISK]DS-42b062e0-208e-4f03-bf28-0b4ba31c3f1b:NORMAL:104w.104w.104w.104w:50010|RBW]]} size 83
2020-05-24 20:33:34,447 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap updated: 103w.103w.103w.103w:50010 is added to blk_1073742638_1828{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-9931cc42-d694-4e79-b2a2-1fc2af193328:NORMAL:103w.103w.103w.103w:50010|RBW], ReplicaUC[[DISK]DS-42b062e0-208e-4f03-bf28-0b4ba31c3f1b:NORMAL:104w.104w.104w.104w:50010|RBW]]} size 83
2020-05-24 20:33:34,456 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /HBase/WALs/hadoop103,16020,1590147108408/hadoop103%2C16020%2C1590147108408.meta.1590320010153.meta is closed by DFSClient_NONMAPREDUCE_1857522783_1
2020-05-24 20:33:44,458 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1073742671_1868{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-9931cc42-d694-4e79-b2a2-1fc2af193328:NORMAL:103w.103w.103w.103w:50010|RBW], ReplicaUC[[DISK]DS-efb40abd-a64b-413d-b9af-4f62d7390c0a:NORMAL:102n.102n.102n.102n:50010|RBW]]} for /HBase/WALs/hadoop103,16020,1590147108408/hadoop103%2C16020%2C1590147108408.1590323624348
2020-05-24 20:34:47,568 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 70 Total time for transactions(ms): 4 Number of transactions batched in Syncs: 14 Number of syncs: 50 SyncTimes(ms): 67 
2020-05-24 20:34:47,606 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1073742672_1869{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-9931cc42-d694-4e79-b2a2-1fc2af193328:NORMAL:103w.103w.103w.103w:50010|RBW], ReplicaUC[[DISK]DS-42b062e0-208e-4f03-bf28-0b4ba31c3f1b:NORMAL:104w.104w.104w.104w:50010|RBW]]} for /HBase/WALs/hadoop103,16020,1590147108408/hadoop103%2C16020%2C1590147108408.1590323624348
2020-05-24 20:34:47,763 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* fsync: /HBase/WALs/hadoop103,16020,1590147108408/hadoop103%2C16020%2C1590147108408.1590323624348 for DFSClient_NONMAPREDUCE_1857522783_1
2020-05-24 20:34:47,838 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap updated: 104w.104w.104w.104w:50010 is added to blk_1073742639_1829{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-9931cc42-d694-4e79-b2a2-1fc2af193328:NORMAL:103w.103w.103w.103w:50010|RBW], ReplicaUC[[DISK]DS-42b062e0-208e-4f03-bf28-0b4ba31c3f1b:NORMAL:104w.104w.104w.104w:50010|RBW]]} size 83
2020-05-24 20:34:47,874 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap updated: 103w.103w.103w.103w:50010 is added to blk_1073742639_1829{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-9931cc42-d694-4e79-b2a2-1fc2af193328:NORMAL:103w.103w.103w.103w:50010|RBW], ReplicaUC[[DISK]DS-42b062e0-208e-4f03-bf28-0b4ba31c3f1b:NORMAL:104w.104w.104w.104w:50010|RBW]]} size 83
2020-05-24 20:34:47,878 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /HBase/WALs/hadoop103,16020,1590147108408/hadoop103%2C16020%2C1590147108408.1590320020617 is closed by DFSClient_NONMAPREDUCE_1857522783_1
2020-05-24 20:38:37,291 WARN org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: PendingReplicationMonitor timed out blk_1073741835_1011
2020-05-24 20:38:37,291 WARN org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: PendingReplicationMonitor timed out blk_1073741837_1013
2020-05-24 20:38:37,291 WARN org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: PendingReplicationMonitor timed out blk_1073741839_1015
2020-05-24 20:38:37,292 WARN org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: PendingReplicationMonitor timed out blk_1073741841_1017
2020-05-24 20:38:37,292 WARN org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: PendingReplicationMonitor timed out blk_1073741844_1020
2020-05-24 20:38:46,074 INFO BlockStateChange: BLOCK* ask 104w.104w.104w.104w:50010 to replicate blk_1073741835_1011 to datanode(s) 102n.102n.102n.102n:50010
2020-05-24 20:38:46,074 INFO BlockStateChange: BLOCK* ask 104w.104w.104w.104w:50010 to replicate blk_1073741837_1013 to datanode(s) 102n.102n.102n.102n:50010
2020-05-24 20:38:46,074 INFO BlockStateChange: BLOCK* ask 103w.103w.103w.103w:50010 to replicate blk_1073741839_1015 to datanode(s) 102n.102n.102n.102n:50010
2020-05-24 20:38:49,075 INFO BlockStateChange: BLOCK* ask 104w.104w.104w.104w:50010 to replicate blk_1073741844_1020 to datanode(s) 102n.102n.102n.102n:50010
2020-05-24 20:38:55,075 INFO BlockStateChange: BLOCK* ask 104w.104w.104w.104w:50010 to replicate blk_1073741841_1017 to datanode(s) 102n.102n.102n.102n:50010
2020-05-24 20:41:53,524 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 77 Total time for transactions(ms): 4 Number of transactions batched in Syncs: 14 Number of syncs: 56 SyncTimes(ms): 75 
2020-05-24 20:41:56,449 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap updated: 104w.104w.104w.104w:50010 is added to blk_1073742667_1861{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-9931cc42-d694-4e79-b2a2-1fc2af193328:NORMAL:103w.103w.103w.103w:50010|RBW], ReplicaUC[[DISK]DS-42b062e0-208e-4f03-bf28-0b4ba31c3f1b:NORMAL:104w.104w.104w.104w:50010|RBW]]} size 16765
2020-05-24 20:41:56,485 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap updated: 103w.103w.103w.103w:50010 is added to blk_1073742667_1861{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-9931cc42-d694-4e79-b2a2-1fc2af193328:NORMAL:103w.103w.103w.103w:50010|RBW], ReplicaUC[[DISK]DS-42b062e0-208e-4f03-bf28-0b4ba31c3f1b:NORMAL:104w.104w.104w.104w:50010|RBW]]} size 16765
2020-05-24 20:41:56,491 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /tmp/hadoop-yarn/staging/root/.staging/job_1590199135215_0002/job_1590199135215_0002_1.jhist is closed by DFSClient_NONMAPREDUCE_-899541460_1
2020-05-24 20:41:56,567 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1073742673_1870{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-9931cc42-d694-4e79-b2a2-1fc2af193328:NORMAL:103w.103w.103w.103w:50010|RBW], ReplicaUC[[DISK]DS-efb40abd-a64b-413d-b9af-4f62d7390c0a:NORMAL:102n.102n.102n.102n:50010|RBW]]} for /tmp/hadoop-yarn/staging/history/done_intermediate/root/job_1590199135215_0002.summary_tmp
2020-05-24 20:42:54,884 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 84 Total time for transactions(ms): 4 Number of transactions batched in Syncs: 24 Number of syncs: 60 SyncTimes(ms): 80 
2020-05-24 20:42:54,885 INFO BlockStateChange: BLOCK* addToInvalidates: blk_1073742635_1825 102n.102n.102n.102n:50010 104w.104w.104w.104w:50010 
2020-05-24 20:42:55,122 INFO BlockStateChange: BLOCK* BlockManager: ask 102n.102n.102n.102n:50010 to delete [blk_1073742635_1825]
2020-05-24 20:42:58,130 INFO BlockStateChange: BLOCK* BlockManager: ask 104w.104w.104w.104w:50010 to delete [blk_1073742635_1825]
2020-05-24 20:42:59,746 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1073742674_1871{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-9931cc42-d694-4e79-b2a2-1fc2af193328:NORMAL:103w.103w.103w.103w:50010|RBW], ReplicaUC[[DISK]DS-42b062e0-208e-4f03-bf28-0b4ba31c3f1b:NORMAL:104w.104w.104w.104w:50010|RBW]]} for /tmp/hadoop-yarn/staging/history/done_intermediate/root/job_1590199135215_0002.summary_tmp
2020-05-24 20:42:59,911 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap updated: 104w.104w.104w.104w:50010 is added to blk_1073742674_1871{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-9931cc42-d694-4e79-b2a2-1fc2af193328:NORMAL:103w.103w.103w.103w:50010|RBW], ReplicaUC[[DISK]DS-42b062e0-208e-4f03-bf28-0b4ba31c3f1b:NORMAL:104w.104w.104w.104w:50010|RBW]]} size 0
2020-05-24 20:42:59,946 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap updated: 103w.103w.103w.103w:50010 is added to blk_1073742674_1871{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-9931cc42-d694-4e79-b2a2-1fc2af193328:NORMAL:103w.103w.103w.103w:50010|RBW], ReplicaUC[[DISK]DS-42b062e0-208e-4f03-bf28-0b4ba31c3f1b:NORMAL:104w.104w.104w.104w:50010|RBW]]} size 0
2020-05-24 20:42:59,955 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /tmp/hadoop-yarn/staging/history/done_intermediate/root/job_1590199135215_0002.summary_tmp is closed by DFSClient_NONMAPREDUCE_-899541460_1
2020-05-24 20:43:00,320 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1073742675_1872{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-9931cc42-d694-4e79-b2a2-1fc2af193328:NORMAL:103w.103w.103w.103w:50010|RBW], ReplicaUC[[DISK]DS-42b062e0-208e-4f03-bf28-0b4ba31c3f1b:NORMAL:104w.104w.104w.104w:50010|RBW]]} for /tmp/hadoop-yarn/staging/history/done_intermediate/root/job_1590199135215_0002-1590321515269-root-hadoop%2Dmapreduce%2Dclient%2Djobclient%2D2.7.2%2Dtests.jar-1590324113573-0-0-FAILED-default-1590321593373.jhist_tmp
2020-05-24 20:43:00,974 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap updated: 104w.104w.104w.104w:50010 is added to blk_1073742675_1872{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-9931cc42-d694-4e79-b2a2-1fc2af193328:NORMAL:103w.103w.103w.103w:50010|RBW], ReplicaUC[[DISK]DS-42b062e0-208e-4f03-bf28-0b4ba31c3f1b:NORMAL:104w.104w.104w.104w:50010|RBW]]} size 0
2020-05-24 20:43:01,008 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap updated: 103w.103w.103w.103w:50010 is added to blk_1073742675_1872{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-9931cc42-d694-4e79-b2a2-1fc2af193328:NORMAL:103w.103w.103w.103w:50010|RBW], ReplicaUC[[DISK]DS-42b062e0-208e-4f03-bf28-0b4ba31c3f1b:NORMAL:104w.104w.104w.104w:50010|RBW]]} size 0
2020-05-24 20:43:01,019 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /tmp/hadoop-yarn/staging/history/done_intermediate/root/job_1590199135215_0002-1590321515269-root-hadoop%2Dmapreduce%2Dclient%2Djobclient%2D2.7.2%2Dtests.jar-1590324113573-0-0-FAILED-default-1590321593373.jhist_tmp is closed by DFSClient_NONMAPREDUCE_-899541460_1
2020-05-24 20:43:01,572 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1073742676_1873{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-9931cc42-d694-4e79-b2a2-1fc2af193328:NORMAL:103w.103w.103w.103w:50010|RBW], ReplicaUC[[DISK]DS-42b062e0-208e-4f03-bf28-0b4ba31c3f1b:NORMAL:104w.104w.104w.104w:50010|RBW]]} for /tmp/hadoop-yarn/staging/history/done_intermediate/root/job_1590199135215_0002_conf.xml_tmp
2020-05-24 20:43:03,673 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap updated: 104w.104w.104w.104w:50010 is added to blk_1073742676_1873{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-9931cc42-d694-4e79-b2a2-1fc2af193328:NORMAL:103w.103w.103w.103w:50010|RBW], ReplicaUC[[DISK]DS-42b062e0-208e-4f03-bf28-0b4ba31c3f1b:NORMAL:104w.104w.104w.104w:50010|RBW]]} size 0
2020-05-24 20:43:03,712 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap updated: 103w.103w.103w.103w:50010 is added to blk_1073742676_1873{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-9931cc42-d694-4e79-b2a2-1fc2af193328:NORMAL:103w.103w.103w.103w:50010|RBW], ReplicaUC[[DISK]DS-42b062e0-208e-4f03-bf28-0b4ba31c3f1b:NORMAL:104w.104w.104w.104w:50010|RBW]]} size 0
2020-05-24 20:43:03,720 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /tmp/hadoop-yarn/staging/history/done_intermediate/root/job_1590199135215_0002_conf.xml_tmp is closed by DFSClient_NONMAPREDUCE_-899541460_1
2020-05-24 20:43:04,981 INFO BlockStateChange: BLOCK* addToInvalidates: blk_1073742651_1841 102n.102n.102n.102n:50010 104w.104w.104w.104w:50010 103w.103w.103w.103w:50010 
2020-05-24 20:43:04,981 INFO BlockStateChange: BLOCK* addToInvalidates: blk_1073742652_1842 102n.102n.102n.102n:50010 103w.103w.103w.103w:50010 104w.104w.104w.104w:50010 
2020-05-24 20:43:04,981 INFO BlockStateChange: BLOCK* addToInvalidates: blk_1073742653_1843 102n.102n.102n.102n:50010 103w.103w.103w.103w:50010 
2020-05-24 20:43:04,981 INFO BlockStateChange: BLOCK* addToInvalidates: blk_1073742654_1844 102n.102n.102n.102n:50010 104w.104w.104w.104w:50010 
2020-05-24 20:43:04,981 INFO BlockStateChange: BLOCK* addToInvalidates: blk_1073742667_1861 103w.103w.103w.103w:50010 104w.104w.104w.104w:50010 
2020-05-24 20:43:04,981 INFO BlockStateChange: BLOCK* addToInvalidates: blk_1073742662_1856 104w.104w.104w.104w:50010 103w.103w.103w.103w:50010 
2020-05-24 20:43:07,131 INFO BlockStateChange: BLOCK* BlockManager: ask 102n.102n.102n.102n:50010 to delete [blk_1073742651_1841, blk_1073742652_1842, blk_1073742653_1843, blk_1073742654_1844]
2020-05-24 20:43:10,131 INFO BlockStateChange: BLOCK* BlockManager: ask 104w.104w.104w.104w:50010 to delete [blk_1073742662_1856, blk_1073742667_1861, blk_1073742651_1841, blk_1073742652_1842, blk_1073742654_1844]
2020-05-24 20:43:10,915 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1073742677_1874{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-9931cc42-d694-4e79-b2a2-1fc2af193328:NORMAL:103w.103w.103w.103w:50010|RBW], ReplicaUC[[DISK]DS-efb40abd-a64b-413d-b9af-4f62d7390c0a:NORMAL:102n.102n.102n.102n:50010|RBW]]} for /tmp/logs/root/logs/application_1590199135215_0002/hadoop103_34169.tmp
2020-05-24 20:43:11,616 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1073742678_1875{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-42b062e0-208e-4f03-bf28-0b4ba31c3f1b:NORMAL:104w.104w.104w.104w:50010|RBW], ReplicaUC[[DISK]DS-efb40abd-a64b-413d-b9af-4f62d7390c0a:NORMAL:102n.102n.102n.102n:50010|RBW]]} for /tmp/logs/root/logs/application_1590199135215_0002/hadoop104_37475.tmp
2020-05-24 20:43:13,131 INFO BlockStateChange: BLOCK* BlockManager: ask 103w.103w.103w.103w:50010 to delete [blk_1073742662_1856, blk_1073742667_1861, blk_1073742651_1841, blk_1073742652_1842, blk_1073742653_1843]
2020-05-24 20:43:55,046 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 115 Total time for transactions(ms): 5 Number of transactions batched in Syncs: 24 Number of syncs: 81 SyncTimes(ms): 104 
2020-05-24 20:43:55,047 INFO BlockStateChange: BLOCK* addToInvalidates: blk_1073742638_1828 103w.103w.103w.103w:50010 104w.104w.104w.104w:50010 
2020-05-24 20:43:55,133 INFO BlockStateChange: BLOCK* BlockManager: ask 104w.104w.104w.104w:50010 to delete [blk_1073742638_1828]
2020-05-24 20:43:58,142 INFO BlockStateChange: BLOCK* BlockManager: ask 103w.103w.103w.103w:50010 to delete [blk_1073742638_1828]
2020-05-24 20:44:14,003 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1073742679_1876{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-9931cc42-d694-4e79-b2a2-1fc2af193328:NORMAL:103w.103w.103w.103w:50010|RBW], ReplicaUC[[DISK]DS-42b062e0-208e-4f03-bf28-0b4ba31c3f1b:NORMAL:104w.104w.104w.104w:50010|RBW]]} for /tmp/logs/root/logs/application_1590199135215_0002/hadoop103_34169.tmp
2020-05-24 20:44:14,647 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1073742680_1877{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-42b062e0-208e-4f03-bf28-0b4ba31c3f1b:NORMAL:104w.104w.104w.104w:50010|RBW], ReplicaUC[[DISK]DS-9931cc42-d694-4e79-b2a2-1fc2af193328:NORMAL:103w.103w.103w.103w:50010|RBW]]} for /tmp/logs/root/logs/application_1590199135215_0002/hadoop104_37475.tmp
2020-05-24 20:44:15,437 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap updated: 103w.103w.103w.103w:50010 is added to blk_1073742680_1877{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-42b062e0-208e-4f03-bf28-0b4ba31c3f1b:NORMAL:104w.104w.104w.104w:50010|RBW], ReplicaUC[[DISK]DS-9931cc42-d694-4e79-b2a2-1fc2af193328:NORMAL:103w.103w.103w.103w:50010|RBW]]} size 0
2020-05-24 20:44:15,440 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap updated: 104w.104w.104w.104w:50010 is added to blk_1073742680_1877{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-42b062e0-208e-4f03-bf28-0b4ba31c3f1b:NORMAL:104w.104w.104w.104w:50010|RBW], ReplicaUC[[DISK]DS-9931cc42-d694-4e79-b2a2-1fc2af193328:NORMAL:103w.103w.103w.103w:50010|RBW]]} size 0
2020-05-24 20:44:15,446 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /tmp/logs/root/logs/application_1590199135215_0002/hadoop104_37475.tmp is closed by DFSClient_NONMAPREDUCE_1775561001_225
2020-05-24 20:44:17,756 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap updated: 104w.104w.104w.104w:50010 is added to blk_1073742679_1876{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-9931cc42-d694-4e79-b2a2-1fc2af193328:NORMAL:103w.103w.103w.103w:50010|RBW], ReplicaUC[[DISK]DS-42b062e0-208e-4f03-bf28-0b4ba31c3f1b:NORMAL:104w.104w.104w.104w:50010|RBW]]} size 0
2020-05-24 20:44:17,791 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap updated: 103w.103w.103w.103w:50010 is added to blk_1073742679_1876{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-9931cc42-d694-4e79-b2a2-1fc2af193328:NORMAL:103w.103w.103w.103w:50010|RBW], ReplicaUC[[DISK]DS-42b062e0-208e-4f03-bf28-0b4ba31c3f1b:NORMAL:104w.104w.104w.104w:50010|RBW]]} size 0
2020-05-24 20:44:17,796 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /tmp/logs/root/logs/application_1590199135215_0002/hadoop103_34169.tmp is closed by DFSClient_NONMAPREDUCE_-717655908_208
2020-05-24 20:44:54,871 INFO BlockStateChange: BLOCK* addToInvalidates: blk_1073742639_1829 103w.103w.103w.103w:50010 104w.104w.104w.104w:50010 
2020-05-24 20:44:55,145 INFO BlockStateChange: BLOCK* BlockManager: ask 104w.104w.104w.104w:50010 to delete [blk_1073742639_1829]
2020-05-24 20:44:58,154 INFO BlockStateChange: BLOCK* BlockManager: ask 103w.103w.103w.103w:50010 to delete [blk_1073742639_1829]

DataNode日誌文件

2020-05-24 19:58:21,563 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-1776586901-102n.102n.102n.102n-1588561313229:blk_1073742641_1831 src: /102n.102n.102n.102n:42504 dest: /102n.102n.102n.102n:50010
2020-05-24 19:58:21,622 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /102n.102n.102n.102n:42504, dest: /102n.102n.102n.102n:50010, bytes: 112, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_1115390510_1, offset: 0, srvID: ee66c1dc-68e4-4053-9f91-6b3f6570945f, blockid: BP-1776586901-102n.102n.102n.102n-1588561313229:blk_1073742641_1831, duration: 46735377
2020-05-24 19:58:21,622 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-1776586901-102n.102n.102n.102n-1588561313229:blk_1073742641_1831, type=HAS_DOWNSTREAM_IN_PIPELINE terminating
2020-05-24 19:58:21,638 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-1776586901-102n.102n.102n.102n-1588561313229:blk_1073742642_1832 src: /102n.102n.102n.102n:42508 dest: /102n.102n.102n.102n:50010
2020-05-24 19:58:21,661 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /102n.102n.102n.102n:42508, dest: /102n.102n.102n.102n:50010, bytes: 112, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_1115390510_1, offset: 0, srvID: ee66c1dc-68e4-4053-9f91-6b3f6570945f, blockid: BP-1776586901-102n.102n.102n.102n-1588561313229:blk_1073742642_1832, duration: 13037339
2020-05-24 19:58:21,661 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-1776586901-102n.102n.102n.102n-1588561313229:blk_1073742642_1832, type=HAS_DOWNSTREAM_IN_PIPELINE terminating
2020-05-24 19:58:21,676 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-1776586901-102n.102n.102n.102n-1588561313229:blk_1073742643_1833 src: /102n.102n.102n.102n:42512 dest: /102n.102n.102n.102n:50010
2020-05-24 19:58:21,697 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /102n.102n.102n.102n:42512, dest: /102n.102n.102n.102n:50010, bytes: 112, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_1115390510_1, offset: 0, srvID: ee66c1dc-68e4-4053-9f91-6b3f6570945f, blockid: BP-1776586901-102n.102n.102n.102n-1588561313229:blk_1073742643_1833, duration: 14589379
2020-05-24 19:58:21,697 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-1776586901-102n.102n.102n.102n-1588561313229:blk_1073742643_1833, type=HAS_DOWNSTREAM_IN_PIPELINE terminating
2020-05-24 19:58:21,708 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-1776586901-102n.102n.102n.102n-1588561313229:blk_1073742644_1834 src: /102n.102n.102n.102n:42516 dest: /102n.102n.102n.102n:50010
2020-05-24 19:58:21,870 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /102n.102n.102n.102n:42516, dest: /102n.102n.102n.102n:50010, bytes: 112, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_1115390510_1, offset: 0, srvID: ee66c1dc-68e4-4053-9f91-6b3f6570945f, blockid: BP-1776586901-102n.102n.102n.102n-1588561313229:blk_1073742644_1834, duration: 82004877
2020-05-24 19:58:21,871 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-1776586901-102n.102n.102n.102n-1588561313229:blk_1073742644_1834, type=HAS_DOWNSTREAM_IN_PIPELINE terminating
2020-05-24 19:58:21,881 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-1776586901-102n.102n.102n.102n-1588561313229:blk_1073742645_1835 src: /102n.102n.102n.102n:42520 dest: /102n.102n.102n.102n:50010
2020-05-24 19:58:22,025 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /102n.102n.102n.102n:42520, dest: /102n.102n.102n.102n:50010, bytes: 112, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_1115390510_1, offset: 0, srvID: ee66c1dc-68e4-4053-9f91-6b3f6570945f, blockid: BP-1776586901-102n.102n.102n.102n-1588561313229:blk_1073742645_1835, duration: 72915703
2020-05-24 19:58:22,025 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-1776586901-102n.102n.102n.102n-1588561313229:blk_1073742645_1835, type=HAS_DOWNSTREAM_IN_PIPELINE terminating
2020-05-24 19:58:22,037 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-1776586901-102n.102n.102n.102n-1588561313229:blk_1073742646_1836 src: /102n.102n.102n.102n:42524 dest: /102n.102n.102n.102n:50010
2020-05-24 19:58:22,180 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /102n.102n.102n.102n:42524, dest: /102n.102n.102n.102n:50010, bytes: 112, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_1115390510_1, offset: 0, srvID: ee66c1dc-68e4-4053-9f91-6b3f6570945f, blockid: BP-1776586901-102n.102n.102n.102n-1588561313229:blk_1073742646_1836, duration: 73498438
2020-05-24 19:58:22,180 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-1776586901-102n.102n.102n.102n-1588561313229:blk_1073742646_1836, type=HAS_DOWNSTREAM_IN_PIPELINE terminating
2020-05-24 19:58:22,191 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-1776586901-102n.102n.102n.102n-1588561313229:blk_1073742647_1837 src: /102n.102n.102n.102n:42528 dest: /102n.102n.102n.102n:50010
2020-05-24 19:58:22,348 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /102n.102n.102n.102n:42528, dest: /102n.102n.102n.102n:50010, bytes: 112, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_1115390510_1, offset: 0, srvID: ee66c1dc-68e4-4053-9f91-6b3f6570945f, blockid: BP-1776586901-102n.102n.102n.102n-1588561313229:blk_1073742647_1837, duration: 79678376
2020-05-24 19:58:22,348 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-1776586901-102n.102n.102n.102n-1588561313229:blk_1073742647_1837, type=HAS_DOWNSTREAM_IN_PIPELINE terminating
2020-05-24 19:58:22,358 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-1776586901-102n.102n.102n.102n-1588561313229:blk_1073742648_1838 src: /102n.102n.102n.102n:42532 dest: /102n.102n.102n.102n:50010
2020-05-24 19:58:22,507 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /102n.102n.102n.102n:42532, dest: /102n.102n.102n.102n:50010, bytes: 112, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_1115390510_1, offset: 0, srvID: ee66c1dc-68e4-4053-9f91-6b3f6570945f, blockid: BP-1776586901-102n.102n.102n.102n-1588561313229:blk_1073742648_1838, duration: 75701274
2020-05-24 19:58:22,507 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-1776586901-102n.102n.102n.102n-1588561313229:blk_1073742648_1838, type=HAS_DOWNSTREAM_IN_PIPELINE terminating
2020-05-24 19:58:22,520 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-1776586901-102n.102n.102n.102n-1588561313229:blk_1073742649_1839 src: /102n.102n.102n.102n:42536 dest: /102n.102n.102n.102n:50010
2020-05-24 19:58:22,666 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /102n.102n.102n.102n:42536, dest: /102n.102n.102n.102n:50010, bytes: 112, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_1115390510_1, offset: 0, srvID: ee66c1dc-68e4-4053-9f91-6b3f6570945f, blockid: BP-1776586901-102n.102n.102n.102n-1588561313229:blk_1073742649_1839, duration: 73928730
2020-05-24 19:58:22,666 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-1776586901-102n.102n.102n.102n-1588561313229:blk_1073742649_1839, type=HAS_DOWNSTREAM_IN_PIPELINE terminating
2020-05-24 19:58:22,678 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-1776586901-102n.102n.102n.102n-1588561313229:blk_1073742650_1840 src: /102n.102n.102n.102n:42540 dest: /102n.102n.102n.102n:50010
2020-05-24 19:58:22,694 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /102n.102n.102n.102n:42540, dest: /102n.102n.102n.102n:50010, bytes: 112, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_1115390510_1, offset: 0, srvID: ee66c1dc-68e4-4053-9f91-6b3f6570945f, blockid: BP-1776586901-102n.102n.102n.102n-1588561313229:blk_1073742650_1840, duration: 9135944
2020-05-24 19:58:22,694 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-1776586901-102n.102n.102n.102n-1588561313229:blk_1073742650_1840, type=HAS_DOWNSTREAM_IN_PIPELINE terminating
2020-05-24 19:58:23,406 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-1776586901-102n.102n.102n.102n-1588561313229:blk_1073742651_1841 src: /102n.102n.102n.102n:42546 dest: /102n.102n.102n.102n:50010
2020-05-24 19:58:23,899 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Slow BlockReceiver write packet to mirror took 473ms (threshold=300ms)
2020-05-24 19:58:24,542 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Slow BlockReceiver write packet to mirror took 643ms (threshold=300ms)
2020-05-24 19:58:25,032 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Slow BlockReceiver write packet to mirror took 490ms (threshold=300ms)
2020-05-24 19:58:25,517 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Slow BlockReceiver write packet to mirror took 485ms (threshold=300ms)
2020-05-24 19:58:26,251 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Slow BlockReceiver write packet to mirror took 734ms (threshold=300ms)
2020-05-24 19:58:26,741 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Slow BlockReceiver write packet to mirror took 490ms (threshold=300ms)
2020-05-24 19:58:27,236 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Slow BlockReceiver write packet to mirror took 495ms (threshold=300ms)
2020-05-24 19:58:27,932 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Slow BlockReceiver write packet to mirror took 696ms (threshold=300ms)
2020-05-24 19:58:28,420 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Slow BlockReceiver write packet to mirror took 488ms (threshold=300ms)
2020-05-24 19:58:28,916 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Slow BlockReceiver write packet to mirror took 496ms (threshold=300ms)
2020-05-24 19:58:29,470 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Slow BlockReceiver write packet to mirror took 554ms (threshold=300ms)
2020-05-24 19:58:30,222 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Slow BlockReceiver write packet to mirror took 752ms (threshold=300ms)
2020-05-24 19:58:30,713 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Slow BlockReceiver write packet to mirror took 491ms (threshold=300ms)
2020-05-24 19:58:31,254 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Slow BlockReceiver write packet to mirror took 541ms (threshold=300ms)
2020-05-24 19:58:31,705 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Slow BlockReceiver write packet to mirror took 450ms (threshold=300ms)
2020-05-24 19:58:32,403 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Slow BlockReceiver write packet to mirror took 698ms (threshold=300ms)
2020-05-24 19:58:32,893 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Slow BlockReceiver write packet to mirror took 490ms (threshold=300ms)
2020-05-24 19:58:34,137 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /102n.102n.102n.102n:42546, dest: /102n.102n.102n.102n:50010, bytes: 1534215, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_1115390510_1, offset: 0, srvID: ee66c1dc-68e4-4053-9f91-6b3f6570945f, blockid: BP-1776586901-102n.102n.102n.102n-1588561313229:blk_1073742651_1841, duration: 10721625384
2020-05-24 19:58:34,137 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-1776586901-102n.102n.102n.102n-1588561313229:blk_1073742651_1841, type=HAS_DOWNSTREAM_IN_PIPELINE terminating
2020-05-24 19:58:34,198 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-1776586901-102n.102n.102n.102n-1588561313229:blk_1073742652_1842 src: /102n.102n.102n.102n:42552 dest: /102n.102n.102n.102n:50010
2020-05-24 19:58:34,504 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /102n.102n.102n.102n:42552, dest: /102n.102n.102n.102n:50010, bytes: 1237, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_1115390510_1, offset: 0, srvID: ee66c1dc-68e4-4053-9f91-6b3f6570945f, blockid: BP-1776586901-102n.102n.102n.102n-1588561313229:blk_1073742652_1842, duration: 158542292
2020-05-24 19:58:34,504 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-1776586901-102n.102n.102n.102n-1588561313229:blk_1073742652_1842, type=HAS_DOWNSTREAM_IN_PIPELINE terminating
2020-05-24 19:58:34,521 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-1776586901-102n.102n.102n.102n-1588561313229:blk_1073742653_1843 src: /102n.102n.102n.102n:42556 dest: /102n.102n.102n.102n:50010
2020-05-24 19:58:34,683 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /102n.102n.102n.102n:42556, dest: /102n.102n.102n.102n:50010, bytes: 256, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_1115390510_1, offset: 0, srvID: ee66c1dc-68e4-4053-9f91-6b3f6570945f, blockid: BP-1776586901-102n.102n.102n.102n-1588561313229:blk_1073742653_1843, duration: 82715573
2020-05-24 19:58:34,683 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-1776586901-102n.102n.102n.102n-1588561313229:blk_1073742653_1843, type=HAS_DOWNSTREAM_IN_PIPELINE terminating
2020-05-24 19:58:34,803 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-1776586901-102n.102n.102n.102n-1588561313229:blk_1073742654_1844 src: /102n.102n.102n.102n:42560 dest: /102n.102n.102n.102n:50010
2020-05-24 19:58:35,052 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /102n.102n.102n.102n:42560, dest: /102n.102n.102n.102n:50010, bytes: 97850, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_1115390510_1, offset: 0, srvID: ee66c1dc-68e4-4053-9f91-6b3f6570945f, blockid: BP-1776586901-102n.102n.102n.102n-1588561313229:blk_1073742654_1844, duration: 239271419
2020-05-24 19:58:35,052 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-1776586901-102n.102n.102n.102n-1588561313229:blk_1073742654_1844, type=HAS_DOWNSTREAM_IN_PIPELINE terminating
2020-05-24 19:58:46,013 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(102n.102n.102n.102n:50010, datanodeUuid=ee66c1dc-68e4-4053-9f91-6b3f6570945f, infoPort=50075, infoSecurePort=0, ipcPort=50020, storageInfo=lv=-56;cid=CID-1cc14dea-3668-45ca-a983-da294aafb0e7;nsid=1994222639;c=0) Starting thread to transfer BP-1776586901-102n.102n.102n.102n-1588561313229:blk_1073742651_1841 to 103w.103w.103w.103w:50010 
2020-05-24 19:58:46,091 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(102n.102n.102n.102n:50010, datanodeUuid=ee66c1dc-68e4-4053-9f91-6b3f6570945f, infoPort=50075, infoSecurePort=0, ipcPort=50020, storageInfo=lv=-56;cid=CID-1cc14dea-3668-45ca-a983-da294aafb0e7;nsid=1994222639;c=0):Failed to transfer BP-1776586901-102n.102n.102n.102n-1588561313229:blk_1073742651_1841 to 103w.103w.103w.103w:50010 got 
java.net.SocketException: Original Exception : java.io.IOException: Connection reset by peer
	at sun.nio.ch.FileChannelImpl.transferTo0(Native Method)
	at sun.nio.ch.FileChannelImpl.transferToDirectlyInternal(FileChannelImpl.java:428)
	at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:493)
	at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:608)
	at org.apache.hadoop.net.SocketOutputStream.transferToFully(SocketOutputStream.java:223)
	at org.apache.hadoop.hdfs.server.datanode.BlockSender.sendPacket(BlockSender.java:579)
	at org.apache.hadoop.hdfs.server.datanode.BlockSender.doSendBlock(BlockSender.java:759)
	at org.apache.hadoop.hdfs.server.datanode.BlockSender.sendBlock(BlockSender.java:706)
	at org.apache.hadoop.hdfs.server.datanode.DataNode$DataTransfer.run(DataNode.java:2123)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: Connection reset by peer
	... 10 more
2020-05-24 19:58:47,094 INFO org.apache.hadoop.hdfs.server.datanode.VolumeScanner: VolumeScanner(/opt/module/hadoop-2.7.2/data/tmp/dfs/data, DS-efb40abd-a64b-413d-b9af-4f62d7390c0a): no suitable block pools found to scan.  Waiting 54219882 ms.
2020-05-24 20:32:15,799 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-1776586901-102n.102n.102n.102n-1588561313229:blk_1073742669_1866 src: /102n.102n.102n.102n:42930 dest: /102n.102n.102n.102n:50010
2020-05-24 20:32:15,828 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /102n.102n.102n.102n:42222, dest: /102n.102n.102n.102n:50010, bytes: 91, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-1294288672_1, offset: 0, srvID: ee66c1dc-68e4-4053-9f91-6b3f6570945f, blockid: BP-1776586901-102n.102n.102n.102n-1588561313229:blk_1073742635_1825, duration: 3600205599615
2020-05-24 20:32:15,828 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-1776586901-102n.102n.102n.102n-1588561313229:blk_1073742635_1825, type=HAS_DOWNSTREAM_IN_PIPELINE terminating
2020-05-24 20:42:55,539 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Scheduling blk_1073742635_1825 file /opt/module/hadoop-2.7.2/data/tmp/dfs/data/current/BP-1776586901-102n.102n.102n.102n-1588561313229/current/finalized/subdir0/subdir3/blk_1073742635 for deletion
2020-05-24 20:42:55,539 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Deleted BP-1776586901-102n.102n.102n.102n-1588561313229 blk_1073742635_1825 file /opt/module/hadoop-2.7.2/data/tmp/dfs/data/current/BP-1776586901-102n.102n.102n.102n-1588561313229/current/finalized/subdir0/subdir3/blk_1073742635
2020-05-24 20:43:07,540 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Scheduling blk_1073742651_1841 file /opt/module/hadoop-2.7.2/data/tmp/dfs/data/current/BP-1776586901-102n.102n.102n.102n-1588561313229/current/finalized/subdir0/subdir3/blk_1073742651 for deletion
2020-05-24 20:43:07,540 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Scheduling blk_1073742652_1842 file /opt/module/hadoop-2.7.2/data/tmp/dfs/data/current/BP-1776586901-102n.102n.102n.102n-1588561313229/current/finalized/subdir0/subdir3/blk_1073742652 for deletion
2020-05-24 20:43:07,540 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Scheduling blk_1073742653_1843 file /opt/module/hadoop-2.7.2/data/tmp/dfs/data/current/BP-1776586901-102n.102n.102n.102n-1588561313229/current/finalized/subdir0/subdir3/blk_1073742653 for deletion
2020-05-24 20:43:07,540 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Scheduling blk_1073742654_1844 file /opt/module/hadoop-2.7.2/data/tmp/dfs/data/current/BP-1776586901-102n.102n.102n.102n-1588561313229/current/finalized/subdir0/subdir3/blk_1073742654 for deletion
2020-05-24 20:43:07,540 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Deleted BP-1776586901-102n.102n.102n.102n-1588561313229 blk_1073742651_1841 file /opt/module/hadoop-2.7.2/data/tmp/dfs/data/current/BP-1776586901-102n.102n.102n.102n-1588561313229/current/finalized/subdir0/subdir3/blk_1073742651
2020-05-24 20:43:07,540 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Deleted BP-1776586901-102n.102n.102n.102n-1588561313229 blk_1073742652_1842 file /opt/module/hadoop-2.7.2/data/tmp/dfs/data/current/BP-1776586901-102n.102n.102n.102n-1588561313229/current/finalized/subdir0/subdir3/blk_1073742652
2020-05-24 20:43:07,540 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Deleted BP-1776586901-102n.102n.102n.102n-1588561313229 blk_1073742653_1843 file /opt/module/hadoop-2.7.2/data/tmp/dfs/data/current/BP-1776586901-102n.102n.102n.102n-1588561313229/current/finalized/subdir0/subdir3/blk_1073742653
2020-05-24 20:43:07,540 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Deleted BP-1776586901-102n.102n.102n.102n-1588561313229 blk_1073742654_1844 file /opt/module/hadoop-2.7.2/data/tmp/dfs/data/current/BP-1776586901-102n.102n.102n.102n-1588561313229/current/finalized/subdir0/subdir3/blk_1073742654
2020-05-24 20:50:30,509 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-1776586901-102n.102n.102n.102n-1588561313229:blk_1073742681_1878 src: /102n.102n.102n.102n:43124 dest: /102n.102n.102n.102n:50010
2020-05-24 20:50:30,526 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /102n.102n.102n.102n:43124, dest: /102n.102n.102n.102n:50010, bytes: 30, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_2066682487_1, offset: 0, srvID: ee66c1dc-68e4-4053-9f91-6b3f6570945f, blockid: BP-1776586901-102n.102n.102n.102n-1588561313229:blk_1073742681_1878, duration: 9024235
2020-05-24 20:50:30,526 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-1776586901-102n.102n.102n.102n-1588561313229:blk_1073742681_1878, type=HAS_DOWNSTREAM_IN_PIPELINE terminating
2020-05-24 20:50:31,624 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Scheduling blk_1073742681_1878 file /opt/module/hadoop-2.7.2/data/tmp/dfs/data/current/BP-1776586901-102n.102n.102n.102n-1588561313229/current/finalized/subdir0/subdir3/blk_1073742681 for deletion
2020-05-24 20:50:31,625 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Deleted BP-1776586901-102n.102n.102n.102n-1588561313229 blk_1073742681_1878 file /opt/module/hadoop-2.7.2/data/tmp/dfs/data/current/BP-1776586901-102n.102n.102n.102n-1588561313229/current/finalized/subdir0/subdir3/blk_1073742681
2020-05-24 21:32:16,030 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-1776586901-102n.102n.102n.102n-1588561313229:blk_1073742683_1880 src: /102n.102n.102n.102n:43572 dest: /102n.102n.102n.102n:50010
2020-05-24 21:32:16,161 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /102n.102n.102n.102n:42930, dest: /102n.102n.102n.102n:50010, bytes: 91, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-1294288672_1, offset: 0, srvID: ee66c1dc-68e4-4053-9f91-6b3f6570945f, blockid: BP-1776586901-102n.102n.102n.102n-1588561313229:blk_1073742669_1866, duration: 3600350870539
2020-05-24 21:32:16,161 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-1776586901-102n.102n.102n.102n-1588561313229:blk_1073742669_1866, type=HAS_DOWNSTREAM_IN_PIPELINE terminating
2020-05-24 21:42:56,259 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Scheduling blk_1073742669_1866 file /opt/module/hadoop-2.7.2/data/tmp/dfs/data/current/BP-1776586901-102n.102n.102n.102n-1588561313229/current/finalized/subdir0/subdir3/blk_1073742669 for deletion
2020-05-24 21:42:56,260 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Deleted BP-1776586901-102n.102n.102n.102n-1588561313229 blk_1073742669_1866 file /opt/module/hadoop-2.7.2/data/tmp/dfs/data/current/BP-1776586901-102n.102n.102n.102n-1588561313229/current/finalized/subdir0/subdir3/blk_1073742669
2020-05-24 21:50:30,541 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-1776586901-102n.102n.102n.102n-1588561313229:blk_1073742688_1891 src: /102n.102n.102n.102n:43766 dest: /102n.102n.102n.102n:50010
2020-05-24 21:50:30,707 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /102n.102n.102n.102n:43766, dest: /102n.102n.102n.102n:50010, bytes: 30, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_2066682487_1, offset: 0, srvID: ee66c1dc-68e4-4053-9f91-6b3f6570945f, blockid: BP-1776586901-102n.102n.102n.102n-1588561313229:blk_1073742688_1891, duration: 82744103
2020-05-24 21:50:30,707 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-1776586901-102n.102n.102n.102n-1588561313229:blk_1073742688_1891, type=HAS_DOWNSTREAM_IN_PIPELINE terminating
2020-05-24 21:50:32,347 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Scheduling blk_1073742688_1891 file /opt/module/hadoop-2.7.2/data/tmp/dfs/data/current/BP-1776586901-102n.102n.102n.102n-1588561313229/current/finalized/subdir0/subdir3/blk_1073742688 for deletion
2020-05-24 21:50:32,347 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Deleted BP-1776586901-102n.102n.102n.102n-1588561313229 blk_1073742688_1891 file /opt/module/hadoop-2.7.2/data/tmp/dfs/data/current/BP-1776586901-102n.102n.102n.102n-1588561313229/current/finalized/subdir0/subdir3/blk_1073742688