[Hadoop] 殺掉正在運作的job

查詢任務列表
hadoop job -list

殺掉job
hadoop job -kill jobId

Cluster監控的tool (持續更新中)

Sematext - SPM

我覺得它的UI非常好看，但是要收費

http://sematext.com/spm/

ambari包含了Ganglia & Nagios

Installing a Hadoop Cluster with three Commands

Ambari (the graphical monitoring and management environment for Hadoop)

https://blog.codecentric.de/en/2014/04/hadoop-cluster-automation/

ambari安裝經驗分享

http://saintya.blogspot.tw/2013/04/ambari.html

使用Ambari快速部署Hadoop大数据环境

http://www.cnblogs.com/scotoma/archive/2013/05/18/3085248.html

Ganglia介紹
http://www.ascc.sinica.edu.tw/iascc/articals.php?_section=2.4&_op=?articalID:5134

RPi-Monitor

專門監控Raspberry Pi

• CPU Loads
• Network
• Disk Boot
• Disk Root
• Swap
• Memory
• Uptime
• Temperature

http://rpi-experiences.blogspot.tw/p/rpi-monitor.html

http://www.raspberry-pi-geek.com/Archive/2013/01/Monitoring-software-for-Raspberry-Pi

2015/05/17
我要尋找監控Hadoop和Spark效能以及cluster功率消耗的tool
目前還沒找到最理想的解決方法 

[Hadoop] Browse the filesystem無法連結

點選Browse the filesystem出現錯誤無法連結



[ Solution 1] 在master01執行

cd /opt/hadoop/etc/hadoop
vi hdfs-site.xml

在hdfs-site.xml中加入以下，結果無效

<property>
<name>dfs.datanode.http.address</name>
<value>10.0.0.234:50075</value>
</property>



[ Solution 2 ] 在本機端執行

cd /etc
sudo vi hosts

在檔案的最後面加入(記得換成自己的ip喲)，就可以瀏覽了

192.168.70.101 master01
192.168.70.102 slave01
192.168.70.103 slave02

[參考資料]

http://www.cnblogs.com/hzmark/p/hadoop_browsethefilesystem.html

http://kurthung1224.pixnet.net/blog/post/170147913

[Hadoop] Word count 範例實做教學

Hadoop on cloudera quickstart vm test example 01 wordcount

mkdir temp

cd temp

ls -ltr

echo "this is huiming and you can call me juiming or killniu i am good at statistical modeling and data analysis" > wordcount.txt



hdfs dfs -mkdir /user/cloudera/input

hdfs dfs -ls /user/cloudera/input

hdfs dfs -put /home/cloudera/temp/wordcount.txt /user/cloudera/input

hdfs dfs -ls /user/cloudera/input

應該會出現剛剛創造的wordcount.txt



ls -ltr /usr/lib/hadoop-mapreduce/

hadoop jar /usr/lib/hadoop-mapreduce/hadoop-mapreduce-example.jar

hadoop jar /usr/lib/hadoop-mapreduce/hadoop-mapreduce-example.jar wordcount /user/cloudera/input/wordcount.txt /user/cloudera/output



hdfs dfs -ls /user/cloudera/output

hdfs dfs -cat /user/cloudera/output/part-r-00000

最後就會跑出word count囉

[Hadoop] 開啟或關閉hadoop

開啟或關閉hadoop



把hadoop關掉(master)
stop-all.sh

刪掉hadoop暫存檔
\rm -r /opt/hadoop/tmp



登入slave01
ssh slave01

\rm -r /opt/hadoop/tmp
exit //離開slave01

登入slave02
ssh slave02

\rm -r /opt/hadoop/tmp
exit //離開slave02回到master



把namenode格式化，讓master重新認識namenode
hadoop namenode -format

啟動hadoop
start-all.sh



確認有沒有開啟hadoop
http://[master01 IP]:50070/

確認MapReduce的狀況
http://[master01 IP]:8088/

jps //類似工作管理員，看有哪些東西在run

hadoop dfsadmin  -report  //查看群集狀態



[參考資料]

Hadoop 2.2.0 Single Cluster 安裝 (二)(CentOS 6.4 x64)
http://shaurong.blogspot.tw/2013/11/hadoop-220-single-cluster-centos-64-x64_7.html

CentOS 安装 hadoop(伪分布模式)
http://zhans52.iteye.com/blog/1102649