tag:blogger.com,1999:blog-6019741717555751737.post8286486856739020650..comments2023-08-13T03:31:05.204-07:00Comments on Low Level Manager: Hadoop Streaming - running a jobAndrew Grangaardhttp://www.blogger.com/profile/14818383405782029025noreply@blogger.comBlogger7125tag:blogger.com,1999:blog-6019741717555751737.post-26041326371053341732012-10-21T04:45:16.555-07:002012-10-21T04:45:16.555-07:00I installed hadoop in usr/local/hadoop
and I copie...I installed hadoop in usr/local/hadoop<br />and I copied range_mapper.py ,range_reducer.py , and two required files <br /><br />My hadoop system contains these files<br /><br />hadoop fs -ls /user/hduser/map<br />Warning: $HADOOP_HOME is deprecated.<br /><br />Found 5 items<br />drwxr-xr-x - hduser supergroup 0 2012-10-21 16:16 /user/hduser/map/a-output<br />-rw-r--r-- 1 hduser supergroup 146 2012-10-21 14:08 /user/hduser/map/input<br />-rw-r--r-- 1 hduser supergroup 4 2012-10-21 14:08 /user/hduser/map/range<br />-rw-r--r-- 1 hduser supergroup 170 2012-10-21 14:08 /user/hduser/map/range_mapper.py<br />-rw-r--r-- 1 hduser supergroup 353 2012-10-21 14:08 /user/hduser/map/range_reducer.py<br /><br />And when I run this command <br />hadoop jar contrib/streaming/hadoop-*streaming*.jar -input /user/hduser/map/* -output /user/hduser/map/a-output -mapper /home/hduser/range_mapreduce/range_mapper.py -reducer /home/hduser/range_mapreduce/range_reducer.py -file range_mapper.py -file range_reducer.py<br /><br />It gives following error<br /><br /><br />Warning: $HADOOP_HOME is deprecated.<br /><br />packageJobJar: [range_mapper.py, range_reducer.py, /app/hadoop/tmp/hadoop-unjar4576261370031348165/] [] /tmp/streamjob5718955918410559810.jar tmpDir=null<br />12/10/21 16:16:16 INFO util.NativeCodeLoader: Loaded the native-hadoop library<br />12/10/21 16:16:16 WARN snappy.LoadSnappy: Snappy native library not loaded<br />12/10/21 16:16:16 INFO mapred.FileInputFormat: Total input paths to process : 4<br />12/10/21 16:16:16 INFO streaming.StreamJob: getLocalDirs(): [/app/hadoop/tmp/mapred/local]<br />12/10/21 16:16:16 INFO streaming.StreamJob: Running job: job_201210211419_0020<br />12/10/21 16:16:16 INFO streaming.StreamJob: To kill this job, run:<br />12/10/21 16:16:16 INFO streaming.StreamJob: /usr/local/hadoop/libexec/../bin/hadoop job -Dmapred.job.tracker=localhost:54311 -kill job_201210211419_0020<br />12/10/21 16:16:16 INFO streaming.StreamJob: Tracking URL: http://localhost:50030/jobdetails.jsp?jobid=job_201210211419_0020<br />12/10/21 16:16:17 INFO streaming.StreamJob: map 0% reduce 0%<br />12/10/21 16:16:53 INFO streaming.StreamJob: map 100% reduce 100%<br />12/10/21 16:16:53 INFO streaming.StreamJob: To kill this job, run:<br />12/10/21 16:16:53 INFO streaming.StreamJob: /usr/local/hadoop/libexec/../bin/hadoop job -Dmapred.job.tracker=localhost:54311 -kill job_201210211419_0020<br />12/10/21 16:16:53 INFO streaming.StreamJob: Tracking URL: http://localhost:50030/jobdetails.jsp?jobid=job_201210211419_0020<br />12/10/21 16:16:53 ERROR streaming.StreamJob: Job not successful. Error: # of failed Map Tasks exceeded allowed limit. FailedCount: 1. LastFailedTask: task_201210211419_0020_m_000000<br />12/10/21 16:16:53 INFO streaming.StreamJob: killJob...<br />Streaming Command Failed!<br /><br /><br />Please help me to fix itVijuPoonthottamhttps://www.blogger.com/profile/12460338614332426045noreply@blogger.comtag:blogger.com,1999:blog-6019741717555751737.post-18216016921046815662012-10-21T04:41:50.467-07:002012-10-21T04:41:50.467-07:00I installed hadoop in usr/local/hadoop
and I copie...I installed hadoop in usr/local/hadoop<br />and I copied range_mapper.py ,range_reducer.py , and two required files <br /><br />My hadoop system contains these files<br /><br />hadoop fs -ls /user/hduser/map<br />Warning: $HADOOP_HOME is deprecated.<br /><br />Found 5 items<br />drwxr-xr-x - hduser supergroup 0 2012-10-21 16:16 /user/hduser/map/a-output<br />-rw-r--r-- 1 hduser supergroup 146 2012-10-21 14:08 /user/hduser/map/input<br />-rw-r--r-- 1 hduser supergroup 4 2012-10-21 14:08 /user/hduser/map/range<br />-rw-r--r-- 1 hduser supergroup 170 2012-10-21 14:08 /user/hduser/map/range_mapper.py<br />-rw-r--r-- 1 hduser supergroup 353 2012-10-21 14:08 /user/hduser/map/range_reducer.py<br /><br />And when I run this command <br />hadoop jar contrib/streaming/hadoop-*streaming*.jar -input /user/hduser/map/* -output /user/hduser/map/a-output -mapper /home/hduser/range_mapreduce/range_mapper.py -reducer /home/hduser/range_mapreduce/range_reducer.py -file range_mapper.py -file range_reducer.py<br /><br />It gives following error<br /><br /><br />Warning: $HADOOP_HOME is deprecated.<br /><br />packageJobJar: [range_mapper.py, range_reducer.py, /app/hadoop/tmp/hadoop-unjar4576261370031348165/] [] /tmp/streamjob5718955918410559810.jar tmpDir=null<br />12/10/21 16:16:16 INFO util.NativeCodeLoader: Loaded the native-hadoop library<br />12/10/21 16:16:16 WARN snappy.LoadSnappy: Snappy native library not loaded<br />12/10/21 16:16:16 INFO mapred.FileInputFormat: Total input paths to process : 4<br />12/10/21 16:16:16 INFO streaming.StreamJob: getLocalDirs(): [/app/hadoop/tmp/mapred/local]<br />12/10/21 16:16:16 INFO streaming.StreamJob: Running job: job_201210211419_0020<br />12/10/21 16:16:16 INFO streaming.StreamJob: To kill this job, run:<br />12/10/21 16:16:16 INFO streaming.StreamJob: /usr/local/hadoop/libexec/../bin/hadoop job -Dmapred.job.tracker=localhost:54311 -kill job_201210211419_0020<br />12/10/21 16:16:16 INFO streaming.StreamJob: Tracking URL: http://localhost:50030/jobdetails.jsp?jobid=job_201210211419_0020<br />12/10/21 16:16:17 INFO streaming.StreamJob: map 0% reduce 0%<br />12/10/21 16:16:53 INFO streaming.StreamJob: map 100% reduce 100%<br />12/10/21 16:16:53 INFO streaming.StreamJob: To kill this job, run:<br />12/10/21 16:16:53 INFO streaming.StreamJob: /usr/local/hadoop/libexec/../bin/hadoop job -Dmapred.job.tracker=localhost:54311 -kill job_201210211419_0020<br />12/10/21 16:16:53 INFO streaming.StreamJob: Tracking URL: http://localhost:50030/jobdetails.jsp?jobid=job_201210211419_0020<br />12/10/21 16:16:53 ERROR streaming.StreamJob: Job not successful. Error: # of failed Map Tasks exceeded allowed limit. FailedCount: 1. LastFailedTask: task_201210211419_0020_m_000000<br />12/10/21 16:16:53 INFO streaming.StreamJob: killJob...<br />Streaming Command Failed!<br /><br /><br />Please help me to fix itVijuPoonthottamhttps://www.blogger.com/profile/12460338614332426045noreply@blogger.comtag:blogger.com,1999:blog-6019741717555751737.post-20015169126598071502012-04-11T02:51:03.003-07:002012-04-11T02:51:03.003-07:00Thanks you save my day tooThanks you save my day tooAnonymousnoreply@blogger.comtag:blogger.com,1999:blog-6019741717555751737.post-92090706536803030982011-09-16T13:42:28.695-07:002011-09-16T13:42:28.695-07:00Shrish,
When you use -file to include a file in y...Shrish,<br /><br />When you use -file to include a file in your jar, the file will be placed in the working directory. Use a local path to access it.<br /><br />e.g.<br />-file /home/hadoop/codes/MapperTst.class<br />-file /home/hadoop/codes/ReducerTst.class<br />-mapper MapperTst.class<br />-reducer ReducerTst.classAndrew Grangaardhttps://www.blogger.com/profile/14818383405782029025noreply@blogger.comtag:blogger.com,1999:blog-6019741717555751737.post-21631604239024680742011-07-22T13:57:14.486-07:002011-07-22T13:57:14.486-07:00Hi,
Whenever I am trying to use Java class files ...Hi,<br /><br />Whenever I am trying to use Java class files as my mapper and/or reducer I am getting the following error:<br /><br />java.io.IOException: Cannot run program "MapperTst.class": java.io.IOException: error=2, No such file or directory<br /><br />I executed the following command on the terminal:<br /><br />hadoop@ubuntu:/usr/local/hadoop$ bin/hadoop jar contrib/streaming/hadoop-streaming-0.20.203.0.jar -file /home/hadoop/codes/MapperTst.class -mapper /home/hadoop/codes/MapperTst.class -file /home/hadoop/codes/ReducerTst.class -reducer /home/hadoop/codes/ReducerTst.class -input gutenberg/* -output gutenberg-outputtstch27<br /><br />Please let me know if I am going wrong.<br /><br />Regards<br /><br />Shrishshrishhttps://www.blogger.com/profile/11634543342846322490noreply@blogger.comtag:blogger.com,1999:blog-6019741717555751737.post-34019724569038386152011-01-13T10:00:46.450-08:002011-01-13T10:00:46.450-08:00Nicolas,
I'm glad I was able to save your day...Nicolas,<br /><br />I'm glad I was able to save your day! I'd love to hear more about what you're working on, what's your hadoop project? Are you using my Hadoop::Streaming perl module?Andrew Grangaardhttps://www.blogger.com/profile/14818383405782029025noreply@blogger.comtag:blogger.com,1999:blog-6019741717555751737.post-92119158432492326552011-01-12T02:07:47.206-08:002011-01-12T02:07:47.206-08:00you saved my day!you saved my day!nicolasnoreply@blogger.com