Lambda Architecture

aka "Runaway complexity in Big Data, and a plan to stop it."

Nathan Marz's talk tonight at Strangeloop coined the term "Lambda Architecture" to describe a hybrid batch+realtime data engine built on functions running over immutable data. This builds on themes from his "Big Data" book.

The pieces all exist, but there's no simple packaging over all of them : distributed raw data store, map-reduce for batch (hadoop/mapr with pig, hive, etc) to precompute views that are stored in fast-read, map-reduce-writable DBs (voldemort, elephantdb), storm for streams, high throughput/small volume db for the storm output (cassandra, risk, hbase), and a custom query merge on top of both. There's no pre-made piece for the custom query merge, possibly storm works there.

Exciting and awesome!

slides and a HackerNews discussion

Runaway complexity in Big Data... and a plan to stop it from nathanmarz

"Big Data" book by Nathan Marz

Big Data: Principle and best practices of scalable realtime data systems by Nathan Marz and Samuel E. Ritchie is now available in MEAP/Roughcut edition from Manning.

The early access edition of my book "Big Data" is now available. Use code bd50 for 50% off http://www.manning.com/marz/
---- @nathanmarz on twitter (link)

Questions about the book? Ask me here: news.ycombinator.com/item?id=3444300
---- @nathanmarz on twitter (link)

Nathan tweeted a discount code 'bd50' for 50% off, which is a nice bonus. I ordered a ebook+print bundle my copy on Monday. Looking forward to digging in and updating you all with a review. Excited to read the Appendix on storm.

Table of Contents:

1. A new paradigm for Big Data - FREE
2. Data model for Big Data - AVAILABLE
3. Data storage on the batch layer
4. MapReduce and batch processing
5. Batch processing with Cascading
6. Basics of the serving layer
7. Storm and the speed layer
8. Incremental batch processing
9. Layered architecture in-depth
10. Piping the system together
11. Future of NoSQL and Big Data processing
    • Appendix A: Hadoop
    • Appendix B: Thrift
    • Appendix C: Storm

Storm is upon us!

A storm (from Proto-Germanic *sturmaz "noise, tumult") is any disturbed state of an astronomical body's atmosphere, especially affecting its surface, and strongly implying severe weather.
-- wikipedia:storm

Storm is open-source: distributed and fault-tolerant realtime computation
-- Nathan Marz via twitter

Twitter has open sourced Storm, a distributed real-time processing engine ( Announcement, Github ). Hadoop is to batch as Storm is to stream. This is huge. So huge that it seems to be mostly ignored?!

Storm's primary author, Nathan Marz, is the same guy who created cascalog for hadoop queries (a marriage of Clojure and Datalog running queries in hadoop) and ElephantDB. You'll remember he was working at BackType (their first non-founder hire. Good choice, guys!). Twitter bought BackType and now they are sharing storm with us. Thanks twitter!

Since Storm's release seems to have flown in under the radar and is a difficult generic search term, I've collected up links to the relevant resources.

Resources & Announcements:

StrangeLoop/Infoq presentation

http://www.infoq.com/presentations/Storm

Nathan opensourced storm in the middle of his StrangeLoop 2011 presentation.

The slides below the image update as the video plays.

Storm Slides

http://www.slideshare.net/nathanmarz/storm-distributed-and-faulttolerant-realtime-computation

StrangeLoop 2011 video links

https://thestrangeloop.com/news/strange-loop-2011-video-schedule

A Storm is Coming more details from twitter.

http://engineering.twitter.com/2011/08/storm-is-coming-more-details-and-plans.html

Storm Source on Github

https://github.com/nathanmarz/storm

Storm Wiki on Github

Wiki Front Page

Rationale

Tutorial

Mailing List

http://groups.google.com/group/storm-user

0.6.1 released

http://groups.google.com/group/storm-user/browse_thread/thread/72b3d4a4aebdebea

Testing Storm Topologies(in Clojure)

http://www.pixelmachine.org/2011/12/17/Testing-Storm-Topologies.html

Overview of Storm Presentation at BashoChats 001

http://basho.com/blog/technical/2011/12/20/Basho-Chats-001-Talk-Videos/

BackType techtalks

http://tech.backtype.com/pages/presentations-8

Thrift!

Sweet, a new module popped onto the CPAN this weekend, Thrift::XS, an XS version of Thrift. This is doubly nice -- one, it's a faster XS version. two, it's available directly on cpan. The current module from the Apache Thrift project requires finding and downloading the package.

Thrift is a streaming serialization format. See also Protocol Buffers and Avro.

DESCRIPTION

Thrift::XS provides faster versions of Thrift::BinaryProtocol and Thrift::MemoryBuffer. On average it is about 4-6 times faster.

Thrift compact protocol support is also available, just replace Thrift::XS::BinaryProtocol with Thrift::XS::CompactProtocol.

To use, simply replace your Thrift initialization code with the appropriate Thrift::XS version.

LA Hadoop

Great attendance at the Los Angeles Hadoop Users Group (LA Hug) meetup last night on "Productizing Hadoop." Cloudera provided a great speaker to discuss the do's and don't's of migrating hadoop from play/development to full enterprise mode ( from hunter gatherer to modern city). The Hadoop infrastructure has come a long way since my first LA hadoop meetup 1+ year ago -- better support for multi-tenancy with auth and authz, more tools built on top of hadoop, and less need to roll-your-own scripts for everything.

Props to Shopzilla for hosting.

This was a much shyer crowd than we see at LA Perl Mongers (LA.PM). Only one other person asked a question at the end. At PM, we tend to pepper questions and feedback all along the presentation making everything a group production.

New Year, New Releases

I opened my cpan mail today and received a lovely email from a user of one of my CPAN modules, Hadoop::Streaming. Reading a nice comment was a wonderful way to start the first Monday of this New Year. Included with the praise was a bug report -- double plus good!

You have absolutely no idea (or perhaps you do) how happy I was to see that there is a hadoop streaming module for perl. So I thank you for making this available! I wonder if you are still working on it or have plans to continue working on it? Are there many users to your knowledge? Finally, I've tried to run the example code myself under perl 5.12.2 and receive bareword errors when running the mapper locally.

---- Frank S Fejes III

Looking at the package and the error output in his email I realized that my hastily pushed out synopsis example had not been code reviewed -- it wouldn't compile as I wasn't quoting the arguments to the Moose keyword 'with.'

It was a small matter to fix and a breeze to push to cpan via the magic of Dist::Zilla. Thanks Ricardo!

1. clone code from github repo : git clone git@github.com:spazm/hadoop-streaming-frontend.git
2. edit lib/Hadoop/Streaming.pm to fix the Synopsis pod
3. add comment to Changes file
4. commit the change locally and back to github: git commit && git push origin master
5. magic Dist::Zilla command, dzil release, which took care of:
   1. checking for file modified files not checked in to git
   2. running pod weaver,
   3. running tests,
   4. updating the release version in Changes file,
   5. git commit Changes,
   6. git push origin master,
   7. git branch,
   8. tar+gz release,
   9. push release to CPAN

While checking my CPAN mail, I also found a CPANTS fail notice for Net::HTTP::Factual which is built on Net::HTTP::Spore. Spore v0.3 came out and changed the spec format again from v0.2 which was in turn different from v0.1.

I tweaked my factual .spec to work with Sport v0.2 or v0.3 and pushed it up to cpan. Same magic Dist::Zilla command.

Freshly available on CPAN:

• Hadoop Steaming v0.110030
• Net::HTTP::Factual v0.110030

Hadoop::Streaming PAUSE registration submitted

Submitted a PAUSE (Perl Authors Upload SErver) request to register Hadoop::Streamingin the User Interface tree at CPAN. I wasn't really sure which top-level category to put it in, but settled on UI as it provides a simple adaption of the Streaming interface of Hadoop.

Woo, my first registered module space. Update: oooh, brian d foy!

On Wed, Sep 08, 2010 at 06:50:34AM +0200, Perl Authors Upload Server wrote:
> 
> The next version of the Module List will list the following module:
> 
>   modid:       Hadoop::Streaming
>   DSLIP:       RdpOp
>   description: simple interface to Hadoop Streaming
>   userid:      SPAZM (Andrew Grangaard)
>   chapterid:   8 (User_Interfaces)
>   enteredby:   BDFOY (brian d foy)
>   enteredon:   Wed Sep  8 04:50:33 2010 GMT
> 
> The resulting entry will be:
> 
> Hadoop::
> ::Streaming       RdpOp simple interface to Hadoop Streaming         SPAZM

Hadoop::Streaming 0.102490 pushed to CPAN

I've pushed a new release of Hadoop::Streaming to CPAN. It should be available in a couple of hours, depending on how long it takes your CPAN mirror to do the mirror update dance.

The release includes expanded documentation in the base Hadoop::Streaming placeholder file. Also included is a Hadoop::Streaming::Combiner role, for creating combiners. Combiners are like reducers that run post-map, per-merge. Once can reuse the reducer as combiner, if the reducer produces the same key/value format on output as input.

After writing my new documentation, test and code, I tested it with dzil test. After passing tests, it's a simple 1 step push to CPAN and github via dzil release. AWESOME! Dist::Zilla makes maintaining CPAN modules brilliantly easy.

Happy Labor Day!

Links

CPAN - Comprehensive Perl Archive Network

http://search.cpan.org

Hadoop::Streaming perl modules

http://search.cpan.org/perldoc?Hadoop::Streaming

Dist::Zilla

http://search.cpan.org/perldoc?Dist::Zilla

There's no Step Two!

[andrew@mini]% dzil release                                1 ~/src/hadoop-streaming-frontend
[DZ] beginning to build Hadoop-Streaming
[DZ] guessing dist's main_module is lib/Hadoop/Streaming.pm
[DZ] extracting distribution abstract from lib/Hadoop/Streaming.pm
[DZ] writing Hadoop-Streaming in Hadoop-Streaming-0.102490
[DZ] writing archive to Hadoop-Streaming-0.102490.tar.gz
[@Basic/TestRelease] Extracting /home/andrew/src/hadoop-streaming-frontend/Hadoo
p-Streaming-0.102490.tar.gz to .build/dVEDcaew44
Checking if your kit is complete...
Looks good
Writing Makefile for Hadoop::Streaming
cp lib/Hadoop/Streaming.pm blib/lib/Hadoop/Streaming.pm
cp lib/Hadoop/Streaming/Combiner.pm blib/lib/Hadoop/Streaming/Combiner.pm
cp lib/Hadoop/Streaming/Role/Emitter.pm blib/lib/Hadoop/Streaming/Role/Emitter.p
m
cp lib/Hadoop/Streaming/Reducer/Input/ValuesIterator.pm blib/lib/Hadoop/Streamin
g/Reducer/Input/ValuesIterator.pm
cp lib/Hadoop/Streaming/Reducer.pm blib/lib/Hadoop/Streaming/Reducer.pm
cp lib/Hadoop/Streaming/Reducer/Input/Iterator.pm blib/lib/Hadoop/Streaming/Redu
cer/Input/Iterator.pm
cp lib/Hadoop/Streaming/Role/Iterator.pm blib/lib/Hadoop/Streaming/Role/Iterator.pm
cp lib/Hadoop/Streaming/Reducer/Input.pm blib/lib/Hadoop/Streaming/Reducer/Input.pm
cp lib/Hadoop/Streaming/Mapper.pm blib/lib/Hadoop/Streaming/Mapper.pm
Manifying blib/man3/Hadoop::Streaming::Combiner.3pm
Manifying blib/man3/Hadoop::Streaming.3pm
Manifying blib/man3/Hadoop::Streaming::Role::Emitter.3pm
Manifying blib/man3/Hadoop::Streaming::Reducer::Input::ValuesIterator.3pm
Manifying blib/man3/Hadoop::Streaming::Reducer::Input::Iterator.3pm
Manifying blib/man3/Hadoop::Streaming::Reducer.3pm
Manifying blib/man3/Hadoop::Streaming::Role::Iterator.3pm
Manifying blib/man3/Hadoop::Streaming::Reducer::Input.3pm
Manifying blib/man3/Hadoop::Streaming::Mapper.3pm
PERL_DL_NONLAZY=1 /usr/bin/perl "-MExtUtils::Command::MM" "-e" "test_harness(0, 'blib/lib', 'blib/arch')" t/*.t
t/00-load.t ....... ok
t/01-wordcount.t .. 8/? # perl path -> /usr/bin/perl
t/01-wordcount.t .. ok
t/02-analog.t ..... ok
All tests successful.
Files=3, Tests=19,  3 wallclock secs ( 0.05 usr  0.01 sys +  2.34 cusr  0.20 csys =  2.60 CPU) 
Result: PASS   

[@Basic/TestRelease] all's well; removing .build/dVEDcaew44
*** Preparing to upload Hadoop-Streaming-0.102490.tar.gz to CPAN ***

Do you want to continue the release process? [y/N]: y
[@Git/Check] branch master is in a clean state
[@Basic/UploadToCPAN] registering upload with PAUSE web server
[@Basic/UploadToCPAN] POSTing upload for Hadoop-Streaming-0.102490.tar.gz
[@Basic/UploadToCPAN] PAUSE add message sent ok [200]
[@Git/Commit] Committed Changes
[@Git/Tag] Tagged v0.102490
[@Git/Push] pushing to origin

Hadoop::Streaming module updated and awesome

Verison 0.101881 of my Hadoop::Streaming module is released on CPAN.

This is a bug fix release -- fixing bugs in my test suite! I had problems with errors from the smoke testers, even though it all "worked for me." To fix this I wrote my tests in a sane manner, so they weren't held together by "gravity and good luck." See Bug #59164 for the gory details. 

I'm happy and proud to report that version 0.101881 has 81 PASS and 0 FAIL results from cpants!

Now all I need is a well written tutorial on how to use it.

links

Bug #59164 for Hadoop-Streaming: Test failures on a variety of platforms

https://rt.cpan.org/Public/Bug/Display.html?id=59164

Hadoop::Streaming Module

Documentation

http://search.cpan.org/perldoc?Hadoop::Streaming

Module Overview:

http://search.cpan.org/search?q=Hadoop::Streaming

Test Reports:

http://www.cpantesters.org/show/Hadoop-Streaming.html#Hadoop-Streaming-0.101881

Open Bugs (0!):

https://rt.cpan.org/Public/Dist/Display.html?Name=Hadoop-Streaming

Git Repo:

http://github.com/spazm/hadoop-streaming-frontend

LA Perl Mongers, January Recap

We had a great meeting this month! Thank you guests and presenters. Twenty mongers in attendance at the start with a few more filing in later. I thought I had over ordered by getting 5 large and one XL pizza, but they disappeared within 20 minutes.

Looking forward to seeing you all again on Wed February 24, 2010.

My talk on Hadoop::Streaming had to cover too much breadth of background to spend much time getting into the code. I look forward to doing a part two where I display a non-trivial example. The talk and slides covered an overview of Hadoop, Hadoop streaming, writing streaming jobs manually, writing streaming jobs using Hadoop::Streaming from CPAN, testing jobs locally, and finally running jobs on the cluster.

The slides will be posted soon. There was a hardware failure on the *.pm.org hosting box on Tuesday into Wednesday, so I didn't push an update. You can see them in raw vroom format in the Git Repo.

Aran was kind enough to come down from Thousand Oaks and present a nice talk on "Coding Concisely in Perl." This was a fun, interactive presentation discussing various ways to make your code (or your inherited code) cleaner and more representative of its core ideas. From simple changes like using ternaries for setting variables, using map/grep to replace cut-and-paste code, and finally a look at the ValueClick method of mocking the production DB. Their DB Mock layer uses a shim to create a local sqlite3 db by autoloading the schemas from the production databases. It's quite clever.

The slides on DB testing have company code on them, so we are waiting on his CTO to approve releasing them for publishing. I expecte to have them up on la.pm.org soon.

CPAN upload, ONE!

My very first CPAN upload is happening *RIGHT NOW*. go go gadget CPAN upload! And now off to sleep, really.


2010-01-05 13:26:15 $$23346 v1048: Info: Need to get uriid[S/SP/SPAZM/Hadoop-Streaming-0.100050.tar.gz] (paused:333)
2010-01-05 13:26:15 $$23346 v1048: Info: Going to fetch uriid[S/SP/SPAZM/Hadoop-Streaming-0.100050.tar.gz] (paused:619)
2010-01-05 13:26:15 $$23346 v1048: Info: Requesting a GET on uri [ftp://pause.perl.org/incoming/Hadoop-Streaming-0.100050.tar.gz] (paused:641)
2010-01-05 13:26:16 $$23346 v1048: Debug: ls[-rw-rw-r-- 1 root root 16457 2010-01-05 13:26 /home/ftp/tmp/S/SP/SPAZM/Hadoop-Streaming-0.100050.tar.gz
]zcat[/bin/zcat]tpath[/home/ftp/tmp/S/SP/SPAZM/Hadoop-Streaming-0.100050.tar.gz]ret[]stat[2057 470407906 33204 1 0 0 0 16457 1262694376 1262694375 1262694375 4096 40]: No child processes (paused:696)
2010-01-05 13:26:16 $$23346 v1048: Info: Got S/SP/SPAZM/Hadoop-Streaming-0.100050.tar.gz (size 16457) (paused:492)
2010-01-05 13:26:18 $$23346 v1048: Info: Sent 'has entered' email about uriid[S/SP/SPAZM/Hadoop-Streaming-0.100050.tar.gz] (paused:555)
2010-01-05 13:27:43 $$23346 v1048: Info: Verified S/SP/SPAZM/Hadoop-Streaming-0.100050.tar.gz (paused:304)
2010-01-05 13:27:43 $$23346 v1048: Info: Started mldistwatch for lpath[/home/ftp/pub/PAUSE/authors/id/S/SP/SPAZM/Hadoop-Streaming-0.100050.tar.gz] with pid[24954] (paused:309)
2010-01-05 13:27:53 $$23346 v1048: Debug: Reaped child[24954] (paused:64)

I think it should show up soon on my cpan page or perhaps the Hadoop::Streaming search.cpan page?

Hadoop Streaming - running a job

Logs from running a hadoop streaming job in a freshly set up environment

1. copy input files
2. run streaming jar (now located in contrib/streaming)
3. hope.

1) copy input files to dfs:
hadoop dfs -copyFromLocal examples/wordcount wordcount

2) run streaming job:
hadoop jar /usr/lib/hadoop/contrib/streaming/hadoop-0.20.1+152-streaming.jar \
-input wordcount \
-output wordcountout \
-mapper examples/wordcount/map.pl \
-reducer examples/wordcount/reduce.pl \


packageJobJar: [/home/hadoop/tmp/hadoop-hadoop/hadoop-unjar5876487782773207253/] [] /tmp/streamjob4555454909817451366.jar tmpDir=null
10/01/05 03:29:34 INFO mapred.FileInputFormat: Total input paths to process : 1
10/01/05 03:29:35 INFO streaming.StreamJob: getLocalDirs(): [/home/hadoop/tmp/hadoop-hadoop/mapred/local]
10/01/05 03:29:35 INFO streaming.StreamJob: Running job: job_201001050303_0003
10/01/05 03:29:35 INFO streaming.StreamJob: To kill this job, run:
10/01/05 03:29:35 INFO streaming.StreamJob: /usr/lib/hadoop-0.20/bin/hadoop job -Dmapred.job.tracker=localhost:54311 -kill job_201001050303_0003
10/01/05 03:29:35 INFO streaming.StreamJob: Tracking URL: http://localhost:50030/jobdetails.jsp?jobid=job_201001050303_0003
10/01/05 03:29:36 INFO streaming.StreamJob: map 0% reduce 0%
10/01/05 03:30:19 INFO streaming.StreamJob: map 100% reduce 100%
10/01/05 03:30:19 INFO streaming.StreamJob: To kill this job, run:
10/01/05 03:30:19 INFO streaming.StreamJob: /usr/lib/hadoop-0.20/bin/hadoop job -Dmapred.job.tracker=localhost:54311 -kill job_201001050303_0003
10/01/05 03:30:19 INFO streaming.StreamJob: Tracking URL: http://localhost:50030/jobdetails.jsp?jobid=job_201001050303_0003
10/01/05 03:30:19 ERROR streaming.StreamJob: Job not Successful!
10/01/05 03:30:19 INFO streaming.StreamJob: killJob...
Streaming Command Failed!
Sigh. Failure. Time to debug the job. Definitely needs the -file flag to bundle the executables to the remote machine.
-file map.pl
-file reducer.pl

Step back and run a simpler job.
hadoop jar /usr/lib/hadoop/contrib/streaming/hadoop-0.20.1+152-streaming.jar -input wordcount -output wordcountout3 -mapper /bin/cat -reducer /bin/wc


10/01/05 03:36:17 INFO streaming.StreamJob: Tracking URL: http://localhost:50030/jobdetails.jsp?jobid=job_201001050303_0004
10/01/05 03:36:17 ERROR streaming.StreamJob: Job not Successful!
10/01/05 03:36:17 INFO streaming.StreamJob: killJob...
Streaming Command Failed!

This is the example code. Why did it fail again? Is there a problem with my input file?

I'll pick this back up tomorrow (or later). A little birdie just told me it was 3:40am. Well past my bedtime. I'd like to finish this up, but that's what I've been saying since I started at midnight. I have the Hadoop::Streaming::Mapper and ::Reducer modules all packaged up and ready to make an initial push to CPAN, but first I need to get an example running under hadoop. I did finish writing the tests for the non-hadoop case, and those are clean and ready. Feel free to follow along at my github repository.

Update

or maybe I'll just try a few more times...

hadoop jar /usr/lib/hadoop/contrib/streaming/hadoop-0.20.1+152-streaming.jar -input wordcount -output wordcountout7 -mapper map.pl -reducer reduce.pl -file examples/wordcount/map.pl -file examples/wordcount/reduce.pl


packageJobJar: [examples/wordcount/map.pl, examples/wordcount/reduce.pl, /home/hadoop/tmp/hadoop-hadoop/hadoop-unjar390944251948922559/] [] /tmp/streamjob7610913425753318391.jar tmpDir=null
10/01/05 03:59:11 INFO mapred.FileInputFormat: Total input paths to process : 1
10/01/05 03:59:11 INFO streaming.StreamJob: getLocalDirs(): [/home/hadoop/tmp/hadoop-hadoop/mapred/local]
10/01/05 03:59:11 INFO streaming.StreamJob: Running job: job_201001050303_0010
10/01/05 03:59:11 INFO streaming.StreamJob: To kill this job, run:
10/01/05 03:59:11 INFO streaming.StreamJob: /usr/lib/hadoop-0.20/bin/hadoop job -Dmapred.job.tracker=localhost:54311 -kill job_201001050303_0010
10/01/05 03:59:11 INFO streaming.StreamJob: Tracking URL: http://localhost:50030/jobdetails.jsp?jobid=job_201001050303_0010
10/01/05 03:59:12 INFO streaming.StreamJob: map 0% reduce 0%
10/01/05 03:59:23 INFO streaming.StreamJob: map 100% reduce 0%
10/01/05 03:59:35 INFO streaming.StreamJob: map 100% reduce 100%
10/01/05 03:59:38 INFO streaming.StreamJob: Job complete: job_201001050303_0010
10/01/05 03:59:38 INFO streaming.StreamJob: Output: wordcountout7

Success!


hadoop dfs -ls wordcountout7;

Found 2 items
drwxr-xr-x - hadoop supergroup 0 2010-01-05 03:59 /user/hadoop/wordcountout7/_logs
-rw-r--r-- 1 hadoop supergroup 125 2010-01-05 03:59 /user/hadoop/wordcountout7/part-00000


hadoop dfs -cat wordcountout7/part*

apple 2
bar 2
baz 1
c 1
c++ 2
cpan 9
foo 2
haskell 4
lang 1
lisp 1
ocaml 2
orange 2
perl 9
python 1
ruby 4
scheme 1
search 1

installing hadoop on ubuntu karmic

Mixing and matching a couple of guides, I've installed a local hadoop instance on my netbook. Here are my notes from the install process.

I'll refer to the guides by number later. Doc 1 is the current #1 hit for 'ubuntu hadoop' on google, so it seemed a good spot to start.

Documents:

1. http://www.michael-noll.com/wiki/Running_Hadoop_On_Ubuntu_Linux_%28Single-Node_Cluster%29
2. http://archive.cloudera.com/docs/_apt.html
3. http://github.com/spazm/config/tree/master/hadoop/conf/

1) created a hadoop user and group, as per document 1. Also ssh-key for hadoop user. (currently no-password, will check that soon).

2) added jaunty-testing repo from cloudera, see doc 2. They don't have a jaunty package yet. Add /etc/apt/souces.list.d/cloudera.list


#deb http://archive.cloudera.com/debian karmic-testing contrib
#deb-src http://archive.cloudera.com/debian karmic-testing contrib
#no packages for karmic yet, trying jaunty-testing, jaunty-stable, jaunty-cdh1 or jaunty-cdh2
deb http://archive.cloudera.com/debian jaunty-testing contrib
deb-src http://archive.cloudera.com/debian jaunty-testing contrib

3) install hadoop:

[andrew@mini]% sudo aptitude install hadoop                                                                   0 ~/src
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Reading extended state information      
Initializing package states... Done
"hadoop" is a virtual package provided by:
  hadoop-0.20 hadoop-0.18 
You must choose one to install.
No packages will be installed, upgraded, or removed.
0 packages upgraded, 0 newly installed, 0 to remove and 25 not upgraded.
Need to get 0B of archives. After unpacking 0B will be used.
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Reading extended state information      
Initializing package states... Done

3b) sudo aptitude update, sudo aptitude install hadoop-0.20

[andrew@mini]% sudo aptitude install hadoop-0.20                                                              0 ~/src
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Reading extended state information      
Initializing package states... Done
The following NEW packages will be installed:
  hadoop-0.20 hadoop-0.20-native{a} 
0 packages upgraded, 2 newly installed, 0 to remove and 25 not upgraded.
Need to get 20.1MB of archives. After unpacking 41.9MB will be used.
Do you want to continue? [Y/n/?] Y
Writing extended state information... Done
[... snip ...]
Initializing package states... Done
Writing extended state information... Done

4) this has setup our config information in /etc/hadoop-0.20, also symlinked as /etc/hadoop/
hadoop-env.sh is loaded from /etc/hadoop/conf/hadoop-env.sh (aka /etc/hadoop-0.20/conf.empty/hadoop-envb.sh)

Modify hadoop-env.sh to point to our jvm. Since I installed sun java 1.6 (aka Java6), I updated it to: export JAVA_HOME=/usr/lib/jvm/java-6-sun

5) update rest of configs.
Snapshotted conf.empty to ~/config/hadoop/conf, and started making edits, as per doc 1. Symlinked into /etc/hadoop/conf

files available at document #3, my github config project, hadoop/conf subidr.

6) switch to hadoop user
sudo -i -u hadoop

7) initiale hdfs (as hadoop user)
mkdir ~hadoop/tmp
chmod a+rwx ~hadoop/tmp
hadoop namenode -format

8) fire it up: (as hadoop user)

/usr/lib/hadoop/bin/start-all.sh
hadoop@mini:/usr/lib/hadoop/logs$ /usr/lib/hadoop/bin/stop-all.sh
stopping jobtracker
localhost: stopping tasktracker
stopping namenode
localhost: stopping datanode
localhost: stopping secondarynamenode
hadoop@mini:/usr/lib/hadoop/logs$ /usr/lib/hadoop/bin/start-all.sh
starting namenode, logging to /usr/lib/hadoop/bin/../logs/hadoop-hadoop-namenode-mini.out
localhost: starting datanode, logging to /usr/lib/hadoop/bin/../logs/hadoop-hadoop-datanode-mini.out
localhost: starting secondarynamenode, logging to /usr/lib/hadoop/bin/../logs/hadoop-hadoop-secondarynamenode-mini.out
starting jobtracker, logging to /usr/lib/hadoop/bin/../logs/hadoop-hadoop-jobtracker-mini.out localhost: starting tasktracker, logging to /usr/lib/hadoop/bin/../logs/hadoop-hadoop-tasktracker-mini.out

8) Check that it is running via jps
hadoop@mini:/usr/lib/hadoop/logs$ jps
12001 NameNode
12166 DataNode
12684 Jps
12568 TaskTracker
12409 JobTracker
12332 SecondaryNameNode

(note to self, why don't we have hadoop completion in zsh? Must rectify)

9) Run example. See doc 1:
hadoop jar hadoop-0.20.0-examples.jar wordcount gutenberg gutenberg-output

hadoop@mini:~/install$ hadoop jar hadoop-0.20.1+152-examples.jar wordcount gutenberg gutenberg-output
09/12/25 23:24:19 INFO input.FileInputFormat: Total input paths to process : 3
09/12/25 23:24:20 INFO mapred.JobClient: Running job: job_200912252310_0001
09/12/25 23:24:21 INFO mapred.JobClient:  map 0% reduce 0%
09/12/25 23:24:33 INFO mapred.JobClient:  map 66% reduce 0%
09/12/25 23:24:39 INFO mapred.JobClient:  map 100% reduce 0%
09/12/25 23:24:42 INFO mapred.JobClient:  map 100% reduce 33%
09/12/25 23:24:48 INFO mapred.JobClient:  map 100% reduce 100%
09/12/25 23:24:50 INFO mapred.JobClient: Job complete: job_200912252310_0001
...

hadoop@mini:~/install$ hadoop dfs -ls gutenberg-output
Found 2 items
drwxr-xr-x   - hadoop supergroup          0 2009-12-25 23:24 /user/hadoop/gutenberg-output/_logs
-rw-r--r--   1 hadoop supergroup      21356 2009-12-25 23:24 /user/hadoop/gutenberg-output/part-r-00000

It Lives!