关于hadoop的dfs.replication

xjc2694

首先<span class="Apple-style-span" style="font-family: verdana, geneva, helvetica, arial, sans-serif; font-size: 13px; line-height: normal; background-color: rgb(255, 255, 255); "><span>&nbsp;dfs.replication这个参数是个client参数，即node level参数。需要在每台datanode上设置。</span></span><div><font class="Apple-style-span" face="verdana, geneva, helvetica, arial, sans-serif" size="2"><span class="Apple-style-span" style="line-height: normal;">其实默认为3个副本已经够用了，设置太多也没什么用。</span></font></div><div><font class="Apple-style-span" face="verdana, geneva, helvetica, arial, sans-serif" size="2"><span class="Apple-style-span" style="line-height: normal;"><br></span></font></div><div><font class="Apple-style-span" face="verdana, geneva, helvetica, arial, sans-serif" size="2"><span class="Apple-style-span" style="line-height: normal;">一个文件，上传到hdfs上时指定的是几个副本就是几个。以后你修改了副本数，对已经上传了的文件也不会起作用。可以再上传文件的同时指定创建的副本数</span></font></div><div><font class="Apple-style-span" face="verdana, geneva, helvetica, arial, sans-serif" size="2"><span class="Apple-style-span" style="line-height: normal;"><div id="codeText" class="codeText"><ol start="1" class="dp-css"><li>hadoop dfs -D dfs.replication=1 -put 70M  logs/2</li></ol></div></span></font></div><div><font class="Apple-style-span" face="verdana, geneva, helvetica, arial, sans-serif" size="2"><span class="Apple-style-span" style="line-height: normal; "><br></span></font></div><div><font class="Apple-style-span" face="verdana, geneva, helvetica, arial, sans-serif" size="2"><span class="Apple-style-span" style="line-height: normal;">可以通过命令来更改已经上传的文件的副本数：</span></font></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, simsun, u5b8bu4f53; line-height: 23px; background-color: rgb(255, 255, 255); font-size: small; ">hadoop fs -setrep -R 3 /</span></div><div><span class="Apple-style-span" style="color: rgb(100, 100, 100); font-family: Arial, Helvetica, simsun, u5b8bu4f53; line-height: 23px; background-color: rgb(255, 255, 255); font-size: small; "><br></span></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, simsun, u5b8bu4f53; line-height: 23px; background-color: rgb(255, 255, 255); font-size: small; ">查看当前hdfs的副本数</span></div><div><span class="Apple-style-span" style="background-color: rgb(255, 255, 255); line-height: 23px;"><font class="Apple-style-span" face="Arial, Helvetica, simsun, u5b8bu4f53" size="2">hadoop fsck -locations</font></span></div><div><span class="Apple-style-span" style="background-color: rgb(255, 255, 255); line-height: 23px;"><font class="Apple-style-span" face="Arial, Helvetica, simsun, u5b8bu4f53" size="2"><div id="codeText" class="codeText"><ol start="1" class="dp-css"><li>FSCK started by hadoop from /172.18.6.112 for path / at Thu Oct 27 13:24:25 CST 2011</li><li>
....................Status: HEALTHY</li><li>
Total size:    4834251860 B</li><li>
Total dirs:    21</li><li>
Total files:   20</li><li>
Total blocks (validated):      82 (avg. block size 58954290 B)</li><li>
Minimally replicated blocks:   82 (100.0 %)</li><li>
Over-replicated blocks:        0 (0.0 %)</li><li>
Under-replicated blocks:       0 (0.0 %)</li><li>
Mis-replicated blocks:         0 (0.0 %)</li><li>
Default replication factor:    3</li><li>
Average block replication:     3.0</li><li>
Corrupt blocks:                0</li><li>
Missing replicas:              0 (0.0 %)</li><li>
Number of data-nodes:          3</li><li>
Number of racks:               1</li><li>
FSCK ended at Thu Oct 27 13:24:25 CST 2011 in 10 milliseconds</li><li>
</li><li>
</li><li>
The filesystem under path '/' is HEALTHY</li></ol></div></font></span></div><div><span class="Apple-style-span" style="background-color: rgb(255, 255, 255); line-height: 23px; "><font class="Apple-style-span" face="Arial, Helvetica, simsun, u5b8bu4f53" size="2"><br></font></span></div><div><span class="Apple-style-span" style="background-color: rgb(255, 255, 255); line-height: 23px; "><font class="Apple-style-span" face="Arial, Helvetica, simsun, u5b8bu4f53" size="2">某个文件的副本数，可以通过ls中的文件描述符看到</font></span></div><div><span class="Apple-style-span" style="background-color: rgb(255, 255, 255); line-height: 23px; "><font class="Apple-style-span" face="Arial, Helvetica, simsun, u5b8bu4f53" size="2">hadoop dfs -ls</font></span></div><div><span class="Apple-style-span" style="background-color: rgb(255, 255, 255); line-height: 23px;"><font class="Apple-style-span" face="Arial, Helvetica, simsun, u5b8bu4f53" size="2">-rw-r--r-- &nbsp; 3 hadoop supergroup &nbsp;153748148 2011-10-27 16:11 /user/hadoop/logs/201108/impression_witspixel2011080100.thin.log.gz</font></span></div><div><span class="Apple-style-span" style="background-color: rgb(255, 255, 255); line-height: 23px; "><font class="Apple-style-span" face="Arial, Helvetica, simsun, u5b8bu4f53" size="2"><br></font></span></div><div><span class="Apple-style-span" style="background-color: rgb(255, 255, 255); line-height: 23px; "><font class="Apple-style-span" face="Arial, Helvetica, simsun, u5b8bu4f53" size="2"><br></font></span></div><div><span class="Apple-style-span" style="background-color: rgb(255, 255, 255); line-height: 23px;"><font class="Apple-style-span" face="Arial, Helvetica, simsun, u5b8bu4f53" size="2">如果你只有3个datanode，但是你却指定副本数为4，是不会生效的，因为每个datanode上只能存放一个副本。</font></span></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, simsun, u5b8bu4f53; line-height: 23px; background-color: rgb(255, 255, 255); font-size: small; ">hadoop fsck -locations 可以看到相应的提示信息，可以看到副本丢失率为33.33%：</span></div><div><span class="Apple-style-span" style="background-color: rgb(255, 255, 255); "><font class="Apple-style-span"><div><div id="codeText" class="codeText"><ol start="1" class="dp-css"><li style="font-family: Arial, Helvetica, simsun, u5b8bu4f53; font-size: small; line-height: 23px; ">/user/hadoop/logs/test.log:  Under replicated blk_-45151128047308146_1147. Target Replicas is 4 but found 3 replica(s).</li><li></li><li>Status: HEALTHY</li><li>&nbsp;Total size: &nbsp; &nbsp;4834251860 B</li><li>&nbsp;Total dirs: &nbsp; &nbsp;21</li><li>&nbsp;Total files: &nbsp; 20</li><li>&nbsp;Total blocks (validated): &nbsp; &nbsp; &nbsp;82 (avg. block size 58954290 B)</li><li>&nbsp;Minimally replicated blocks: &nbsp; 82 (100.0 %)</li><li>&nbsp;Over-replicated blocks: &nbsp; &nbsp; &nbsp; &nbsp;0 (0.0 %)</li><li>&nbsp;Under-replicated blocks: &nbsp; &nbsp; &nbsp; 82 (100.0 %)</li><li>&nbsp;Mis-replicated blocks: &nbsp; &nbsp; &nbsp; &nbsp; 0 (0.0 %)</li><li>&nbsp;Default replication factor: &nbsp; &nbsp;3</li><li>&nbsp;Average block replication: &nbsp; &nbsp; 3.0</li><li>&nbsp;Corrupt blocks: &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;0</li><li>&nbsp;Missing replicas: &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;82 (33.333332 %)</li><li>&nbsp;Number of data-nodes: &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;3</li><li>&nbsp;Number of racks: &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 1</li><li>FSCK ended at Thu Oct 27 13:22:14 CST 2011 in 12 milliseconds</li></ol></div></div></font></span></div><div><font class="Apple-style-span" face="Arial, Helvetica, simsun, u5b8bu4f53" size="2"><span class="Apple-style-span" style="line-height: 23px;">参考：hdfs_design</span></font></div><div><font class="Apple-style-span" face="Arial, Helvetica, simsun, u5b8bu4f53" size="2"><span class="Apple-style-span" style="line-height: 23px;"><a href="http://hadoop.apache.org/common/docs/r0.20.204.0/hdfs_design.pdf" target="_blank" target="_blank">http://hadoop.apache.org/common/docs/r0.20.204.0/hdfs_design.pdf</a></span></font></div><div><font class="Apple-style-span" face="Arial, Helvetica, simsun, u5b8bu4f53" size="2"><a href="http://hadoop.apache.org/common/docs/r0.20.204.0/hdfs_design.html" target="_blank" target="_blank">http://hadoop.apache.org/common/docs/r0.20.204.0/hdfs_design.html</a></font></div><div><font class="Apple-style-span" face="Arial, Helvetica, simsun, u5b8bu4f53" size="2"><span class="Apple-style-span" style="line-height: 23px; ">当一个文件上传时，client并不立刻联系namenode，而是先在本地缓存数据，当&nbsp;HDFS block size时，联系namenode，namenode将文件名插入到文件系统结构中，并为期分配一个数据块。namenode以datanode主机名和数据块的位置来相应client的请求。客户端从本地临时文件中将数据刷新到指定的datanode。当file关闭时，未刷新的临时文件将传输到datanode，client通知namenode 文件关闭。此时，namenode将文件创建操作提交到永久存储。如果namenode在file closes之前die，则文件丢失。</span></font></div><div><font class="Apple-style-span" face="Arial, Helvetica, simsun, u5b8bu4f53" size="2"><span class="Apple-style-span" style="line-height: 23px;"><br></span></font></div><div><font class="Apple-style-span" face="Arial, Helvetica, simsun, u5b8bu4f53" size="2"><span class="Apple-style-span" style="line-height: 23px;">创建副本</span></font></div><div><font class="Apple-style-span" face="Arial, Helvetica, simsun, u5b8bu4f53" size="2"><span class="Apple-style-span" style="line-height: 23px;">当client写文件到hdfs时，像前面提到的，先写文件到本地临时文件，假设设定hdfs的副本系数为3.当缓存的文件达到hdfs block size时，client从namenode检索一个datanode的列表。该列表包含将host该副本的datanode列表。client刷新数据到列表中的第一个datanode。第一个datanode以4kb为单位接收数据，将数据写到本地并传输到列表中的第二个datanode，第二个datanode也做同样操作。一个datanode可以从上一个数据管道获取数据，并同时将数据发送到下一个数据管道。</span></font></div><div><font class="Apple-style-span" face="Arial, Helvetica, simsun, u5b8bu4f53" size="2"><span class="Apple-style-span" style="line-height: 23px;"><br></span></font></div><div><font class="Apple-style-span" face="Arial, Helvetica, simsun, u5b8bu4f53" size="2"><span class="Apple-style-span" style="line-height: 23px;"><br></span></font></div><div><font class="Apple-style-span" face="Arial, Helvetica, simsun, u5b8bu4f53" size="2"><span class="Apple-style-span" style="line-height: 23px;">关于namenode如何选择副本存放位置：</span></font></div><div><font class="Apple-style-span" face="Arial, Helvetica, simsun, u5b8bu4f53" size="2"><span class="Apple-style-span" style="line-height: 23px;"><a href="http://blog.chinaunix.net/space.php?uid=1838361&amp;do=blog&amp;id=2981485" target="_blank" target="_blank">http://blog.chinaunix.net/space.php?uid=1838361&amp;do=blog&amp;id=2981485</a></span></font></div><div><span class="Apple-style-span" style="background-color: rgb(255, 255, 255); line-height: 23px; "><font class="Apple-style-span" face="Arial, Helvetica, simsun, u5b8bu4f53" size="2"><br></font></span></div><div><span class="Apple-style-span" style="color: rgb(100, 100, 100); font-family: Arial, Helvetica, simsun, u5b8bu4f53; line-height: 23px; background-color: rgb(255, 255, 255); font-size: small; "><br></span></div>