Saturday, May 30, 2015

Steps to compile 64-bit Hadoop 2.3.0 under Ubuntu 14.04


All rights reserved Please keep the author name: KiwenLau and original blog link :

http://kiwenlau.blogspot.com/2015/05/steps-to-compile-hadoop-230-under.html


hadoop-2.3.0.tar.gz provided by Hadoop official website is compiled under 32 bit machine,you will get some trouble if you use it under 64 bit machine, for example:

WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

So, you have to compile hadoop source code by your self. I compiled hadoop-2.3.0 under 64 bit ubuntu 14.04 and run wordcount program successfully, you can download it from this: http://1drv.ms/1HZ1TSV


1. update package lists
apt-get update

2. install dependencies
apt-get install -y openjdk-7-jdk libprotobuf-dev protobuf-compiler maven cmake build-essential pkg-config libssl-dev zlib1g-dev llvm-gcc automake autoconf make

3. download hadoop source file

4. extract hadoop source file
tar -xzvf hadoop-2.3.0-src.tar.gz

5. enter hadoop directory
cd hadoop-2.3.0-src

6. compile hadoop 2.3.0
mvn package -Pdist,native -DskipTests -Dtar

output when it success

[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Apache Hadoop Main ................................ SUCCESS [1:11.968s]
[INFO] Apache Hadoop Project POM ......................... SUCCESS [30.393s]
[INFO] Apache Hadoop Annotations ......................... SUCCESS [18.398s]
[INFO] Apache Hadoop Assemblies .......................... SUCCESS [0.246s]
[INFO] Apache Hadoop Project Dist POM .................... SUCCESS [20.372s]
[INFO] Apache Hadoop Maven Plugins ....................... SUCCESS [23.721s]
[INFO] Apache Hadoop MiniKDC ............................. SUCCESS [1:41.836s]
[INFO] Apache Hadoop Auth ................................ SUCCESS [22.303s]
[INFO] Apache Hadoop Auth Examples ....................... SUCCESS [7.052s]
[INFO] Apache Hadoop Common .............................. SUCCESS [2:29.466s]
[INFO] Apache Hadoop NFS ................................. SUCCESS [11.604s]
[INFO] Apache Hadoop Common Project ...................... SUCCESS [0.073s]
[INFO] Apache Hadoop HDFS ................................ SUCCESS [1:30.230s]
[INFO] Apache Hadoop HttpFS .............................. SUCCESS [17.976s]
[INFO] Apache Hadoop HDFS BookKeeper Journal ............. SUCCESS [19.927s]
[INFO] Apache Hadoop HDFS-NFS ............................ SUCCESS [3.304s]
[INFO] Apache Hadoop HDFS Project ........................ SUCCESS [0.032s]
[INFO] hadoop-yarn ....................................... SUCCESS [0.033s]
[INFO] hadoop-yarn-api ................................... SUCCESS [36.284s]
[INFO] hadoop-yarn-common ................................ SUCCESS [33.912s]
[INFO] hadoop-yarn-server ................................ SUCCESS [0.213s]
[INFO] hadoop-yarn-server-common ......................... SUCCESS [8.193s]
[INFO] hadoop-yarn-server-nodemanager .................... SUCCESS [41.181s]
[INFO] hadoop-yarn-server-web-proxy ...................... SUCCESS [2.768s]
[INFO] hadoop-yarn-server-resourcemanager ................ SUCCESS [13.923s]
[INFO] hadoop-yarn-server-tests .......................... SUCCESS [0.904s]
[INFO] hadoop-yarn-client ................................ SUCCESS [4.363s]
[INFO] hadoop-yarn-applications .......................... SUCCESS [0.120s]
[INFO] hadoop-yarn-applications-distributedshell ......... SUCCESS [2.262s]
[INFO] hadoop-yarn-applications-unmanaged-am-launcher .... SUCCESS [1.615s]
[INFO] hadoop-yarn-site .................................. SUCCESS [0.086s]
[INFO] hadoop-yarn-project ............................... SUCCESS [2.703s]
[INFO] hadoop-mapreduce-client ........................... SUCCESS [0.132s]
[INFO] hadoop-mapreduce-client-core ...................... SUCCESS [18.951s]
[INFO] hadoop-mapreduce-client-common .................... SUCCESS [14.320s]
[INFO] hadoop-mapreduce-client-shuffle ................... SUCCESS [3.330s]
[INFO] hadoop-mapreduce-client-app ....................... SUCCESS [9.664s]
[INFO] hadoop-mapreduce-client-hs ........................ SUCCESS [7.678s]
[INFO] hadoop-mapreduce-client-jobclient ................. SUCCESS [9.263s]
[INFO] hadoop-mapreduce-client-hs-plugins ................ SUCCESS [1.549s]
[INFO] Apache Hadoop MapReduce Examples .................. SUCCESS [5.748s]
[INFO] hadoop-mapreduce .................................. SUCCESS [2.880s]
[INFO] Apache Hadoop MapReduce Streaming ................. SUCCESS [7.080s]
[INFO] Apache Hadoop Distributed Copy .................... SUCCESS [14.648s]
[INFO] Apache Hadoop Archives ............................ SUCCESS [2.602s]
[INFO] Apache Hadoop Rumen ............................... SUCCESS [5.706s]
[INFO] Apache Hadoop Gridmix ............................. SUCCESS [3.649s]
[INFO] Apache Hadoop Data Join ........................... SUCCESS [2.483s]
[INFO] Apache Hadoop Extras .............................. SUCCESS [2.678s]
[INFO] Apache Hadoop Pipes ............................... SUCCESS [6.359s]
[INFO] Apache Hadoop OpenStack support ................... SUCCESS [5.088s]
[INFO] Apache Hadoop Client .............................. SUCCESS [4.534s]
[INFO] Apache Hadoop Mini-Cluster ........................ SUCCESS [0.433s]
[INFO] Apache Hadoop Scheduler Load Simulator ............ SUCCESS [7.757s]
[INFO] Apache Hadoop Tools Dist .......................... SUCCESS [4.099s]
[INFO] Apache Hadoop Tools ............................... SUCCESS [0.428s]
[INFO] Apache Hadoop Distribution ........................ SUCCESS [18.045s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 14:59.240s
[INFO] Finished at: Thu Jan 15 18:51:59 JST 2015
[INFO] Final Memory: 168M/435M
[INFO] ------------------------------------------------------------------------


you can get the compiled hadoop-2.3.0 under this directory:
hadoop-2.3.0-src/hadoop-dist/target/hadoop-2.3.0.tar.gz


The methods is the same when you want to compile hadoop-2.5.2, hadoop-2.6.0 and hadoop-2.7.0

42 comments:

  1. Website Design Virginia I am sure this post has helped me save many hours of browsing other related posts just to find what I was looking for. Many thanks!

    ReplyDelete
  2. Thanks for providing this informative information. it is very useful you may also refer-http://www.s4techno.com/blog/2016/08/13/installing-a-storm-cluster/

    ReplyDelete
  3. Thanks for providing this informative information…..
    You may also refer-
    http://www.s4techno.com/blog/category/hadoop/

    ReplyDelete
  4. This is by far the most concise guide I've seen so far. Very helpful. Thanks.

    ReplyDelete
  5. I am really happy to say it’s an interesting post to read . I learn new information from your article , you are doing a great job . Keep it up
    Hadoop Training in Hyderabad
    Data Science Training in Hyderabad

    ReplyDelete
  6. This comment has been removed by the author.

    ReplyDelete
  7. This comment has been removed by the author.

    ReplyDelete
  8. I know you feel more happy when you get things done and best of all those things are your most precious treasure.
    selenium Training in chennai


    amazon web services Training in chennai


    Block Chain Training in velachery

    ReplyDelete
  9. This comment has been removed by the author.

    ReplyDelete
  10. Wow it is really wonderful and awesome thus it is very much useful for me to understand many concepts and helped me a lot. it is really explainable very well and i got more information from your blog.


    rpa training in chennai | best rpa training in chennai | rpa training in chennai | rpa training in bangalore
    rpa training in pune | rpa online training

    ReplyDelete
  11. A good blog always comes-up with new and exciting information and while reading I have feel that this blog is really have all those quality that qualify a blog to be a one.I wanted to leave a little comment to support you and wish you a good continuation. Wishing you the best of luck for all your blogging efforts read this.
    Data Science training in kalyan nagar | Data Science training in OMR
    Data Science training in chennai | Data science training in velachery
    Data science training in tambaram | Data science training in jaya nagar

    ReplyDelete
  12. It's interesting that many of the bloggers to helped clarify a few things for me as well as giving.Most of ideas can be nice content.The people to give them a good shake to get your point and across the command
    java training in chennai | java training in bangalore

    java online training | java training in pune

    ReplyDelete
  13. Good Post, I am a big believer in posting comments on sites to let the blog writers know that they ve added something advantageous to the world wide web.
    Best Devops Training in pune

    ReplyDelete
  14. Whoa! I’m enjoying the template/theme of this website. It’s simple, yet effective. A lot of times it’s very hard to get that “perfect balance” between superb usability and visual appeal. I must say you’ve done a very good job with this.

    AWS Training in Bangalore | Amazon Web Services Training in bangalore , india

    AWS Training in pune | Amazon Web Services Training in Pune, india

    AWS Training in Chennai|Amazon Web Services Training in Chennai,India


    aws online training and certification | amazon web services online training ,india


    ReplyDelete
  15. Thank you a lot for providing individuals with a very spectacular possibility to read critical reviews from this site.
    iosh course in chennai

    ReplyDelete
  16. Your very own commitment to getting the message throughout came to be rather powerful and have consistently enabled employees just like me to arrive at their desired goals.
    Data science Course Training in Chennai | Data Science Training in Chennai
    RPA Course Training in Chennai | RPA Training in Chennai
    AWS Course Training in Chennai | AWS Training in Chennai

    ReplyDelete
  17. Excellent blog information
    Sanjary Kids is one of the best play school and preschool in Hyderabad,India. Give your child the best preschool experience by choosing the best playschool of Hyderabad in Abids. we provide programs like Play group,Nursery,Junior KG,Senior KG,and provides Teacher Training Program.
    early childhood teacher training course in hyderabad

    ReplyDelete
  18. Great blog information of the author provided

    Sanjary Academy is the best Piping Design institute in Hyderabad, Telangana. It is the best Piping design Course in India and we have offer professional Engineering Courses like Piping design Course, QA/QC Course, document controller course, Pressure Vessel Design Course, Welding Inspector Course, Quality Management Course and Safety Officer Course.
    best Piping Design Course
    piping design course with placement
    Piping Design Course
    Piping Design Course in Hyderabad ­
    Piping Design Course in India­
    best Piping Design institute
    best institute of Piping Design Course in India

    ReplyDelete
  19. This comment has been removed by the author.

    ReplyDelete
  20. Thanks for sharing the updated information and understandable blog clearly

    Sanjary Academy provide pressure vessel design,quality management system course, piping design course, qa/qc course and document controller course.
    Welding Inspector Course
    Safety officer course
    Quality Management Course
    Quality Management Course in India

    ReplyDelete
  21. Your info is really amazing with impressive content..Excellent blog with informative concept. Really I feel happy to see this useful blog, Thanks for sharing such a nice blog..
    If you are looking for any Data science Related information please visit our website Data science courses in Pune page!

    ReplyDelete
  22. I like this topic.This site has lots of advantage.I found many interesting things from this site. It helps me in many ways.Thanks for posting this again.
    Agra University BCom 1st, 2nd & Final Year Time Table 2020

    ReplyDelete
  23. This site helps to clear your all query. shekhauni ba final result
    Mizoram university bcom exam result This is really worth reading. nice informative article.

    ReplyDelete
  24. Looking forward to reading more. Great article post. Thanks Again. Fantastic.

    B Com final year time table

    ReplyDelete
  25. I have not any word to appreciate this post…..Really i am impressed from this post….

    ReplyDelete
  26. Informative and engaging. Can't wait to read more from this author. We are Swaran Homoeopathic Clinic Provide homeopathy treatment and medicine in dwarka, delhi if you have any health realted issue please contact us.

    ReplyDelete