1.现象描述

hdfs fsck / 检查HDFS状态时发现:

Missing replicas:              996 (4.563992 %)

Target Replicas is 10 but found 4 replica(s)

[hadoop@SZC-L0055746 logs]$ hdfs fsck / Connecting to namenode via http://master2:50070FSCK started by hadoop (auth:SIMPLE) from /30.16.94.72 for path / at Thu May 25 09:52:20 CST 2017 .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... ................................................ /data0/hadoop/dfs/hive/scratchdir/hadoop/1aa6d7af-2c53-42c8-b7ba-c500780b98c4/hive2017-05-2509-43-438952972602983038145037-126/-mr-10007/94b5a7d2-f8d1-4271-9ec1-111357abbbdf/map.xml:  Under replicated BP-631829430-30.16.94.72-1492782671063:blk107379061849855. Target Replicas is 10 but found 4 replica(s). . /data0/hadoop/dfs/hive/scratchdir/hadoop/1aa6d7af-2c53-42c8-b7ba-c500780b98c4/hive2017-05-2509-43-438952972602983038145037-126/-mr-10007/94b5a7d2-f8d1-4271-9ec1-111357abbbdf/reduce.xml:  Under replicated BP-631829430-30.16.94.72-1492782671063:blk107379061949856. Target Replicas is 10 but found 4 replica(s). .....

. /tmp/hadoop-yarn/staging/hadoop/.staging/job14955232649130228/libjars/solr-solrj-5.5.3.jar:  Under replicated BP-631829430-30.16.94.72-1492782671063:blk107378702646247. Target Replicas is 10 but found 4 replica(s). . /tmp/hadoop-yarn/staging/hadoop/.staging/job14955232649130228/libjars/stax2-api-3.1.4.jar:  Under replicated BP-631829430-30.16.94.72-1492782671063:blk107378702746248. Target Replicas is 10 but found 4 replica(s). . /tmp/hadoop-yarn/staging/hadoop/.staging/job14955232649130228/libjars/woodstox-core-asl-4.4.1.jar:  Under replicated BP-631829430-30.16.94.72-1492782671063:blk107378702846249. Target Replicas is 10 but found 4 replica(s). . /tmp/hadoop-yarn/staging/hadoop/.staging/job14955232649130234/job.jar:  Under replicated BP-631829430-30.16.94.72-1492782671063:blk107378742646647. Target Replicas is 10 but found 4 replica(s). . /tmp/hadoop-yarn/staging/hadoop/.staging/job14955232649130234/job.split:  Under replicated BP-631829430-30.16.94.72-1492782671063:blk107378742846649. Target Replicas is 10 but found 4 replica(s). ... /tmp/hadoop-yarn/staging/hadoop/.staging/job14955232649130234/libjars/Hive2Solr-2.1.1-5.5.3.jar:  Under replicated BP-631829430-30.16.94.72-1492782671063:blk107378741646637. Target Replicas is 10 but found 4 replica(s). . /tmp/hadoop-yarn/staging/hadoop/.staging/job14955232649130234/libjars/httpclient-4.4.1.jar:  Under replicated BP-631829430-30.16.94.72-1492782671063:blk107378741746638. Target Replicas is 10 but found 4 replica(s). . /tmp/hadoop-yarn/staging/hadoop/.staging/job14955232649130234/libjars/httpcore-4.4.1.jar:  Under replicated BP-631829430-30.16.94.72-1492782671063:blk107378741846639. Target Replicas is 10 but found 4 replica(s). . /tmp/hadoop-yarn/staging/hadoop/.staging/job14955232649130234/libjars/httpmime-4.4.1.jar:  Under replicated BP-631829430-30.16.94.72-1492782671063:blk107378741946640. Target Replicas is 10 but found 4 replica(s). . /tmp/hadoop-yarn/staging/hadoop/.staging/job14955232649130234/libjars/jcl-over-slf4j-1.7.7.jar:  Under replicated BP-631829430-30.16.94.72-1492782671063:blk107378742046641. Target Replicas is 10 but found 4 replica(s). . /tmp/hadoop-yarn/staging/hadoop/.staging/job14955232649130234/libjars/noggit-0.6.jar:  Under replicated BP-631829430-30.16.94.72-1492782671063:blk107378742146642. Target Replicas is 10 but found 4 replica(s). . /tmp/hadoop-yarn/staging/hadoop/.staging/job14955232649130234/libjars/slf4j-api-1.7.10.jar:  Under replicated BP-631829430-30.16.94.72-1492782671063:blk107378742246643. Target Replicas is 10 but found 4 replica(s). . /tmp/hadoop-yarn/staging/hadoop/.staging/job14955232649130234/libjars/solr-solrj-5.5.3.jar:  Under replicated BP-631829430-30.16.94.72-1492782671063:blk107378742346644. Target Replicas is 10 but found 4 replica(s). .

/tmp/hadoop-yarn/staging/hadoop/.staging/job14955232649130234/libjars/stax2-api-3.1.4.jar:  Under replicated BP-631829430-30.16.94.72-1492782671063:blk107378742446645. Target Replicas is 10 but found 4 replica(s). . /tmp/hadoop-yarn/staging/hadoop/.staging/job14955232649130234/libjars/woodstox-core-asl-4.4.1.jar:  Under replicated BP-631829430-30.16.94.72-1492782671063:blk107378742546646. Target Replicas is 10 but found 4 replica(s). . /tmp/hadoop-yarn/staging/hadoop/.staging/job14955232649130235/job.jar:  Under replicated BP-631829430-30.16.94.72-1492782671063:blk107378745346674. Target Replicas is 10 but found 4 replica(s). . /tmp/hadoop-yarn/staging/hadoop/.staging/job14955232649130235/job.split:  Under replicated BP-631829430-30.16.94.72-1492782671063:blk107378745446675. Target Replicas is 10 but found 4 replica(s). ... /tmp/hadoop-yarn/staging/hadoop/.staging/job14955232649130235/libjars/Hive2Solr-2.1.1-5.5.3.jar:  Under replicated BP-631829430-30.16.94.72-1492782671063:blk107378744346664. Target Replicas is 10 but found 4 replica(s). . /tmp/hadoop-yarn/staging/hadoop/.staging/job14955232649130235/libjars/httpclient-4.4.1.jar:  Under replicated BP-631829430-30.16.94.72-1492782671063:blk107378744446665. Target Replicas is 10 but found 4 replica(s). . /tmp/hadoop-yarn/staging/hadoop/.staging/job14955232649130235/libjars/httpcore-4.4.1.jar:  Under replicated BP-631829430-30.16.94.72-1492782671063:blk107378744546666. Target Replicas is 10 but found 4 replica(s). . /tmp/hadoop-yarn/staging/hadoop/.staging/job14955232649130235/libjars/httpmime-4.4.1.jar:  Under replicated BP-631829430-30.16.94.72-1492782671063:blk107378744646667. Target Replicas is 10 but found 4 replica(s). . /tmp/hadoop-yarn/staging/hadoop/.staging/job14955232649130235/libjars/jcl-over-slf4j-1.7.7.jar:  Under replicated BP-631829430-30.16.94.72-1492782671063:blk107378744746668. Target Replicas is 10 but found 4 replica(s). . /tmp/hadoop-yarn/staging/hadoop/.staging/job14955232649130235/libjars/noggit-0.6.jar:  Under replicated BP-631829430-30.16.94.72-1492782671063:blk107378744846669. Target Replicas is 10 but found 4 replica(s). . /tmp/hadoop-yarn/staging/hadoop/.staging/job14955232649130235/libjars/slf4j-api-1.7.10.jar:  Under replicated BP-631829430-30.16.94.72-1492782671063:blk107378744946670. Target Replicas is 10 but found 4 replica(s). . /tmp/hadoop-yarn/staging/hadoop/.staging/job14955232649130235/libjars/solr-solrj-5.5.3.jar:  Under replicated BP-631829430-30.16.94.72-1492782671063:blk107378745046671. Target Replicas is 10 but found 4 replica(s). . /tmp/hadoop-yarn/staging/hadoop/.staging/job14955232649130235/libjars/stax2-api-3.1.4.jar:  Under replicated BP-631829430-30.16.94.72-1492782671063:blk107378745146672. Target Replicas is 10 but found 4 replica(s). . /tmp/hadoop-yarn/staging/hadoop/.staging/job14955232649130235/libjars/woodstox-core-asl-4.4.1.jar:  Under replicated BP-631829430-30.16.94.72-1492782671063:blk107378745246673. Target Replicas is 10 but found 4 replica(s). . /tmp/hadoop-yarn/staging/hadoop/.staging/job14955232649130236/job.jar:  Under replicated BP-631829430-30.16.94.72-1492782671063:blk107378753746759. Target Replicas is 10 but found 4 replica(s). . /tmp/hadoop-yarn/staging/hadoop/.staging/job14955232649130236/job.split:  Under replicated BP-631829430-30.16.94.72-1492782671063:blk107378753846760. Target Replicas is 10 but found 4 replica(s). ... /tmp/hadoop-yarn/staging/hadoop/.staging/job14955232649130236/libjars/Hive2Solr-2.1.1-5.5.3.jar:  Under replicated BP-631829430-30.16.94.72-1492782671063:blk107378752746749. Target Replicas is 10 but found 4 replica(s). . /tmp/hadoop-yarn/staging/hadoop/.staging/job14955232649130236/libjars/httpclient-4.4.1.jar:  Under replicated BP-631829430-30.16.94.72-1492782671063:blk107378752846750. Target Replicas is 10 but found 4 replica(s). . /tmp/hadoop-yarn/staging/hadoop/.staging/job14955232649130236/libjars/httpcore-4.4.1.jar:  Under replicated BP-631829430-30.16.94.72-1492782671063:blk107378752946751. Target Replicas is 10 but found 4 replica(s). . /tmp/hadoop-yarn/staging/hadoop/.staging/job14955232649130236/libjars/httpmime-4.4.1.jar:  Under replicated BP-631829430-30.16.94.72-1492782671063:blk107378753046752. Target Replicas is 10 but found 4 replica(s). . /tmp/hadoop-yarn/staging/hadoop/.staging/job14955232649130236/libjars/jcl-over-slf4j-1.7.7.jar:  Under replicated BP-631829430-30.16.94.72-1492782671063:blk107378753146753. Target Replicas is 10 but found 4 replica(s). . /tmp/hadoop-yarn/staging/hadoop/.staging/job14955232649130236/libjars/noggit-0.6.jar:  Under replicated BP-631829430-30.16.94.72-1492782671063:blk107378753246754. Target Replicas is 10 but found 4 replica(s). . /tmp/hadoop-yarn/staging/hadoop/.staging/job14955232649130236/libjars/slf4j-api-1.7.10.jar:  Under replicated BP-631829430-30.16.94.72-1492782671063:blk107378753346755. Target Replicas is 10 but found 4 replica(s). . /tmp/hadoop-yarn/staging/hadoop/.staging/job14955232649130236/libjars/solr-solrj-5.5.3.jar:  Under replicated BP-631829430-30.16.94.72-1492782671063:blk107378753446756. Target Replicas is 10 but found 4 replica(s). . /tmp/hadoop-yarn/staging/hadoop/.staging/job14955232649130236/libjars/stax2-api-3.1.4.jar:  Under replicated BP-631829430-30.16.94.72-1492782671063:blk107378753546757. Target Replicas is 10 but found 4 replica(s). . /tmp/hadoop-yarn/staging/hadoop/.staging/job14955232649130236/libjars/woodstox-core-asl-4.4.1.jar:  Under replicated BP-631829430-30.16.94.72-1492782671063:blk107378753646758. Target Replicas is 10 but found 4 replica(s). . /tmp/hadoop-yarn/staging/hadoop/.staging/job14955232649130238/job.jar:  Under replicated BP-631829430-30.16.94.72-1492782671063:blk107378832147543. Target Replicas is 10 but found 4 replica(s). . /tmp/hadoop-yarn/staging/hadoop/.staging/job14955232649130238/job.split:  Under replicated BP-631829430-30.16.94.72-1492782671063:blk107378832247544. Target Replicas is 10 but found 4 replica(s). ... /tmp/hadoop-yarn/staging/hadoop/.staging/job14955232649130238/libjars/Hive2Solr-2.1.1-5.5.3.jar:  Under replicated BP-631829430-30.16.94.72-1492782671063:blk107378831147533. Target Replicas is 10 but found 4 replica(s). . /tmp/hadoop-yarn/staging/hadoop/.staging/job14955232649130238/libjars/hiveUDF.jar:  Under replicated BP-631829430-30.16.94.72-1492782671063:blk107378831047532. Target Replicas is 10 but found 4 replica(s). . /tmp/hadoop-yarn/staging/hadoop/.staging/job14955232649130238/libjars/httpclient-4.4.1.jar:  Under replicated BP-631829430-30.16.94.72-1492782671063:blk107378831247534. Target Replicas is 10 but found 4 replica(s). . /tmp/hadoop-yarn/staging/hadoop/.staging/job14955232649130238/libjars/httpcore-4.4.1.jar:  Under replicated BP-631829430-30.16.94.72-1492782671063:blk107378831347535. Target Replicas is 10 but found 4 replica(s). . /tmp/hadoop-yarn/staging/hadoop/.staging/job14955232649130238/libjars/httpmime-4.4.1.jar:  Under replicated BP-631829430-30.16.94.72-1492782671063:blk107378831447536. Target Replicas is 10 but found 4 replica(s). . /tmp/hadoop-yarn/staging/hadoop/.staging/job14955232649130238/libjars/jcl-over-slf4j-1.7.7.jar:  Under replicated BP-631829430-30.16.94.72-1492782671063:blk107378831547537. Target Replicas is 10 but found 4 replica(s). . /tmp/hadoop-yarn/staging/hadoop/.staging/job14955232649130238/libjars/noggit-0.6.jar:  Under replicated BP-631829430-30.16.94.72-1492782671063:blk107378831647538. Target Replicas is 10 but found 4 replica(s). . /tmp/hadoop-yarn/staging/hadoop/.staging/job14955232649130238/libjars/slf4j-api-1.7.10.jar:  Under replicated BP-631829430-30.16.94.72-1492782671063:blk107378831747539. Target Replicas is 10 but found 4 replica(s). . /tmp/hadoop-yarn/staging/hadoop/.staging/job14955232649130238/libjars/solr-solrj-5.5.3.jar:  Under replicated BP-631829430-30.16.94.72-1492782671063:blk107378831847540. Target Replicas is 10 but found 4 replica(s). . /tmp/hadoop-yarn/staging/hadoop/.staging/job14955232649130238/libjars/stax2-api-3.1.4.jar:  Under replicated BP-631829430-30.16.94.72-1492782671063:blk107378831947541. Target Replicas is 10 but found 4 replica(s). . /tmp/hadoop-yarn/staging/hadoop/.staging/job14955232649130238/libjars/woodstox-core-asl-4.4.1.jar:  Under replicated BP-631829430-30.16.94.72-1492782671063:blk107378832047542. Target Replicas is 10 but found 4 replica(s). . /tmp/hadoop-yarn/staging/hadoop/.staging/job14955232649130242/job.jar:  Under replicated BP-631829430-30.16.94.72-1492782671063:blk107379063149868. Target Replicas is 10 but found 4 replica(s). . /tmp/hadoop-yarn/staging/hadoop/.staging/job14955232649130242/job.split:  Under replicated BP-631829430-30.16.94.72-1492782671063:blk107379063249869. Target Replicas is 10 but found 4 replica(s). ..... /tmp/hadoop-yarn/staging/hadoop/.staging/job14955232649130242/libjars/Hive2Solr-2.1.1-5.5.3.jar:  Under replicated BP-631829430-30.16.94.72-1492782671063:blk107379062149858. Target Replicas is 10 but found 4 replica(s). . /tmp/hadoop-yarn/staging/hadoop/.staging/job14955232649130242/libjars/hiveUDF.jar:  Under replicated BP-631829430-30.16.94.72-1492782671063:blk107379062049857. Target Replicas is 10 but found 4 replica(s). . /tmp/hadoop-yarn/staging/hadoop/.staging/job14955232649130242/libjars/httpclient-4.4.1.jar:  Under replicated BP-631829430-30.16.94.72-1492782671063:blk107379062249859. Target Replicas is 10 but found 4 replica(s). . /tmp/hadoop-yarn/staging/hadoop/.staging/job14955232649130242/libjars/httpcore-4.4.1.jar:  Under replicated BP-631829430-30.16.94.72-1492782671063:blk107379062349860. Target Replicas is 10 but found 4 replica(s). . /tmp/hadoop-yarn/staging/hadoop/.staging/job14955232649130242/libjars/httpmime-4.4.1.jar:  Under replicated BP-631829430-30.16.94.72-1492782671063:blk107379062449861. Target Replicas is 10 but found 4 replica(s). . /tmp/hadoop-yarn/staging/hadoop/.staging/job14955232649130242/libjars/jcl-over-slf4j-1.7.7.jar:  Under replicated BP-631829430-30.16.94.72-1492782671063:blk107379062549862. Target Replicas is 10 but found 4 replica(s). . /tmp/hadoop-yarn/staging/hadoop/.staging/job14955232649130242/libjars/noggit-0.6.jar:  Under replicated BP-631829430-30.16.94.72-1492782671063:blk107379062649863. Target Replicas is 10 but found 4 replica(s). . /tmp/hadoop-yarn/staging/hadoop/.staging/job14955232649130242/libjars/slf4j-api-1.7.10.jar:  Under replicated BP-631829430-30.16.94.72-1492782671063:blk107379062749864. Target Replicas is 10 but found 4 replica(s). . /tmp/hadoop-yarn/staging/hadoop/.staging/job14955232649130242/libjars/solr-solrj-5.5.3.jar:  Under replicated BP-631829430-30.16.94.72-1492782671063:blk107379062849865. Target Replicas is 10 but found 4 replica(s). . /tmp/hadoop-yarn/staging/hadoop/.staging/job14955232649130242/libjars/stax2-api-3.1.4.jar:  Under replicated BP-631829430-30.16.94.72-1492782671063:blk107379062949866. Target Replicas is 10 but found 4 replica(s). . /tmp/hadoop-yarn/staging/hadoop/.staging/job14955232649130242/libjars/woodstox-core-asl-4.4.1.jar:  Under replicated BP-631829430-30.16.94.72-1492782671063:blk107379063049867. Target Replicas is 10 but found 4 replica(s). ....................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... ...............................................................Status: HEALTHY Total size:    693673280704 B (Total open files size: 16511955286 B) Total dirs:    2014 Total files:   2563 Total symlinks:                0 (Files currently being written: 34) Total blocks (validated):      6887 (avg. block size 100722125 B) (Total open file blocks (not validated): 154) Minimally replicated blocks:   6887 (100.0 %) Over-replicated blocks:        0 (0.0 %) Under-replicated blocks:       166 (2.4103384 %) Mis-replicated blocks:         0 (0.0 %) Default replication factor:    3 Average block replication:     3.0241034 Corrupt blocks:                0 Missing replicas:              996 (4.563992 %) Number of data-nodes:          4 Number of racks:               1 FSCK ended at Thu May 25 09:52:20 CST 2017 in 72 milliseconds

The filesystem under path '/' is HEALTHY

原因分析

集群4个节点，mapred-site.xml 没有设置mapreduce.client.submit.file.replication，该参数默认值为10， 因此客户端提交的Job默认都要求10 replication，但集群总节点数才4个，每个节点副本，总数为4，因此集群会有提示信息。

PS：该信息不影响集群状态

处理办法

在mapred-site.xml上增加 mapreduce.client.submit.file.replication设置

    <property>
    <name>mapreduce.client.submit.file.replication</name>
    <value>4</value>
</property>