Tuesday, January 17, 2012

The Grail of Efficiency : premature optimization

Premature optimization is the root of all evil ... most of the time. You're only going to know that it's time to optimize after you've built. This doesn't get us off the hook for building slow systems nor for adding nonessential complexity into our solutions.

Build it first, then measure, then improve. And PS. you're bad at guessing what's wrong.

I like this longer version of knuth's quotation (though he claims he didn't coin the phrase) than the one that shows up on the wikipedia page on program optimization.

There is no doubt that the grail of efficiency leads to abuse. Programmers waste enormous amounts of time thinking about, or worrying about, the speed of noncritical parts of their programs, and these attempts at efficiency actually have a strong negative impact when debugging and maintenance are considered. We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil.

Yet we should not pass up our opportunities in that critical 3%. A good programmer will not be lulled into complacency by such reasoning, he will be wise to look carefully at the critical code; but only after that code has been identified. It is often a mistake to make a priori judgments about what parts of a program are really critical, since the universal experience of programmers who have been using measurement tools has been that their intuitive guesses fail.
-- Donald Knuth, ACM: Structured programing with go to statements

Wednesday, January 11, 2012

"Big Data" book by Nathan Marz

Big Data: Principle and best practices of scalable realtime data systems by Nathan Marz and Samuel E. Ritchie is now available in MEAP/Roughcut edition from Manning.

The early access edition of my book "Big Data" is now available. Use code bd50 for 50% off http://www.manning.com/marz/
---- @nathanmarz on twitter (link)
Questions about the book? Ask me here: news.ycombinator.com/item?id=3444300
---- @nathanmarz on twitter (link)

Nathan tweeted a discount code 'bd50' for 50% off, which is a nice bonus. I ordered a ebook+print bundle my copy on Monday. Looking forward to digging in and updating you all with a review. Excited to read the Appendix on storm.

Table of Contents:

  1. A new paradigm for Big Data - FREE
  2. Data model for Big Data - AVAILABLE
  3. Data storage on the batch layer
  4. MapReduce and batch processing
  5. Batch processing with Cascading
  6. Basics of the serving layer
  7. Storm and the speed layer
  8. Incremental batch processing
  9. Layered architecture in-depth
  10. Piping the system together
  11. Future of NoSQL and Big Data processing
    • Appendix A: Hadoop
    • Appendix B: Thrift
    • Appendix C: Storm

Tuesday, January 10, 2012

Building a jabber bot with Bot::Backend

There are many ways to write a (jabber/xmpp) chat bot. A quick search for "Jabber Bot" turns up Net::Jabber::Bot, Bot::JabberBot, Bot::Backbone::Service::JabberChat, Bot::Jabbot, IM::Engine, AnyEvent::XMPP and more. You'll see even more if you search for XMPP instead.

How to choose?

I started with Net::Jabber::Bot, but ran into problems connecting to google-talk based chat. Jabber modules built on AnyEvent::XMPP (seem to) connect to google-auth better than those built on Net::XMPP. This is moot now that I have a normal jabber server, but it still tripped me up. The pod doc lays out funny, so I patched and filed a change-request at github.

Knowing that I would eventually want an event-loop based app, I focused on AnyEvent::XMPP packages. Bot::Backbone and IM::Engine are both interesting abstractions. IM::Engine is quote "currently alpha quality with serious features missing and is rife with horrible bugs." Perhaps I should be happy that he admits this up front? On to Bot::Backbone and sugary Moose!

Once I realized that the group_domain wouldn't be automatically filled in as "conference.$domain" in Bot::Backbone::Service::JabberChat, I was able to connect to a group chat room on my jabber server!

Follow along with my bot, App::Sulla on github. Thus far I have a working connection that responds to "!time" requests -- I've even fixed the part where it threw warnings on all other messages!

#this code in my dispatch table:
    also not_command not_to_me run_this {
            my ( $self, $message ) = @_;
            respond { "hello world" };
    };

# lead to the following error on every other chat message:
unhandled callback exception on event (message, AnyEvent::XMPP::Ext::MUC=HASH(0x3746818), AnyEvent::XMPP::Ext::MUC::Room=HASH(0x3a6d700) ANY_STRING ): Can't call method "add_predicate_or_return" on an undefined value at perl5/lib/perl5/Bot/Backbone/DispatchSugar.pm line 124.

I decided I wanted to pass the auth and channel information into App::Sulla as parameters to the constructor. This didn't play very well with the service sugar. service executes at compile time and squirrels away the configuration hash for the service into services. I added a before modifier to construct_services which is called during run just before initializing each service.

Monday, January 2, 2012

Storm is upon us!

A storm (from Proto-Germanic *sturmaz "noise, tumult") is any disturbed state of an astronomical body's atmosphere, especially affecting its surface, and strongly implying severe weather.
-- wikipedia:storm

Storm is open-source: distributed and fault-tolerant realtime computation
-- Nathan Marz via twitter
Twitter has open sourced Storm, a distributed real-time processing engine ( Announcement, Github ). Hadoop is to batch as Storm is to stream. This is huge. So huge that it seems to be mostly ignored?!

Storm's primary author, Nathan Marz, is the same guy who created cascalog for hadoop queries (a marriage of Clojure and Datalog running queries in hadoop) and ElephantDB. You'll remember he was working at BackType (their first non-founder hire. Good choice, guys!). Twitter bought BackType and now they are sharing storm with us. Thanks twitter!

Since Storm's release seems to have flown in under the radar and is a difficult generic search term, I've collected up links to the relevant resources.

Resources & Announcements:

StrangeLoop/Infoq presentation
http://www.infoq.com/presentations/Storm
Nathan opensourced storm in the middle of his StrangeLoop 2011 presentation.
The slides below the image update as the video plays.
Storm Slides
http://www.slideshare.net/nathanmarz/storm-distributed-and-faulttolerant-realtime-computation
StrangeLoop 2011 video links
https://thestrangeloop.com/news/strange-loop-2011-video-schedule
A Storm is Coming more details from twitter.
http://engineering.twitter.com/2011/08/storm-is-coming-more-details-and-plans.html
Storm Source on Github
https://github.com/nathanmarz/storm
Storm Wiki on Github
Wiki Front Page
Rationale
Tutorial
Mailing List
http://groups.google.com/group/storm-user
0.6.1 released
http://groups.google.com/group/storm-user/browse_thread/thread/72b3d4a4aebdebea
Testing Storm Topologies(in Clojure)
http://www.pixelmachine.org/2011/12/17/Testing-Storm-Topologies.html
Overview of Storm Presentation at BashoChats 001
http://basho.com/blog/technical/2011/12/20/Basho-Chats-001-Talk-Videos/
BackType techtalks
http://tech.backtype.com/pages/presentations-8

Saturday, December 31, 2011

28c3: Effective Denial of Service attacks against web application platforms

Synopsis

Hour long presentation from the C3 security conference, on hash collision based attacks against web apps. The idea being that most web stacks automatically grab the query params and stick them into a hash of key-value pairs and by exploiting hash collisions an attacker can waste huge amounts of CPU time by a simple HTTP POST. In languages that don't randomly perturb their hashing functions (mostly DJBX33A or DJBX33X), collisions can be easily found and exploited.

This affects node.js/v8, php, python, ASP, ruby, java and etc. Only perl (5.8.1 ~2003) and cruby (1.9 ~2008) are patched for randomized hashing functions. The v8 devs are unconcerned "because v8 is a client side language," which caused a bit of alarm for the node.js folks.

The presenters started their investigation after reading a mention in the perl security faq (perldoc perlsec), in the section Algorithmic Complexity Attacks, that before 5.8.1 perl had a security flaw where hash collisions could be exploited. 5.8.1 was released in 2003.

           In Perls before 5.8.1 one could rather easily generate data that as
           hash keys would cause Perl to consume large amounts of time because
           internal structure of hashes would badly degenerate.  In Perl 5.8.1
           the hash function is randomly perturbed by a pseudorandom seed
           which makes generating such naughty hash keys harder.  See
           "PERL_HASH_SEED" in perlrun for more information.
Thread from the node.js mailing list: "HOLY CRAP. nearly all nodejs http servers are vulnerable to DoS and apparently, the V8 guys seem to not care much"

Related Node.js followup:

Friday, December 30, 2011

Talent, Bias and Diversity

Inc.com has a nice article on talent: You Can't Predict Talent; Foster It . His tips for fostering talent: open atmosphere, extravagant diversity, time didn't matter, stretch goals were just the start. The article leads off with this quote:
In his recent book Thinking Fast and Slow, behavioral economist Daniel Kahneman tells the story of observing army recruits out on exercises and his belief that he could spot the potential leaders amongst them. Years later, it turned out he'd been almost entirely wrong. His confident judgment had been a morass of bias, heuristics, and narrative fallacies.

This got me thinking about a more-or-less completely non-sequiter, vaguely related to open atmostphere and extravagant diversity.

As humans, we have brains built for pattern matching, so we find patterns. We apply these patterns all day long into systems of heuristics: "have I been in this position before? What did I do then? Did it work?" Actions that match our heuristics feel right because they work as expected and match previous experience.

We're making more of these patterns every day, and we are awesome in our ability to match situations against our patterns.

Problem: we're not good at evaluating whether our past decisions were correct. This introduces systemic bias. Bias expands directly with the homogeneity of your peer group.

Fixes? Think different. Use DATA to evaluate your decisions. Interact outside of your comfortable peer group. Be aware that you're using short cuts, and take the long way every-so-often to see if it really is longer. think.

Thursday, December 1, 2011

Perl Advent Calendars, 2011 edition.

It's Advent Calendar time in the perl ecosystem! Start each day with a delicious treat of knowledge.

I've found a half dozen english language perl advent calendars, starting with the original perl advent calendar. For extra fun I've included another half dozen Japanese language calendars -- I can still read the perl it's just the prose that is lost in translation.

Ricardo (RJBS) has taken over the Perl Advent calendar this year, which is awesome. Sadly, that means he won't be doing his own "month of rjbs" calendar. I've added a link to his 2010 calendar, in case you missed it the first time around. He's starting the month with Day 1: cpanm and local::lib.

For a second year, Miyagawa has skipped updating plack advent calendar. Check out the 2009 edition linked below. He has given us plenty of other presents this year: Carton, etc.

Perl Advent
http://perladvent.org/2011/
(Formerly the perladvent.pm.org calendar)
Perl Dancer -- the dancer mini web framework
http://advent.perldancer.org/2011/
Catalyst Advent Calendar -- The Catalyst Web Framework
http://www.catalystframework.org/calendar/
Perl 6
http://perl6advent.wordpress.com/
For the adventurous: Japanese Perl Advent Calendars, 8 different tracks!
http://perl-users.jp/articles/advent-calendar/2011/
AnySan Track
Casual Track
dbix Track
English Track
Hacker Track
Test Track
Acme Track
Teng Track
Amon2 Track

Ricardo's 2010 advent calendar -- a month of RJBS
http://advent.rjbs.manxome.org/2010/
2009 Plack calendar
http://advent.plackperl.org/

One bonus list, for the sysadmin in your life:

SysAdvent - The Sysadmin Advent Calendar.
http://sysadvent.blogspot.com/
Evil: If I were creating the world I wouldn't mess about with butterflies and daffodils. I would have started with lasers, eight o'clock, Day One!
-- Time Bandits

Saturday, November 12, 2011

Git::CPAN::Patch

Just saw an announcement of Git::CPAN::Patch over at Yanick's blog.

This is an awesome tool to create patches for other people's modules (OPM(tm)). It will create a git repo from current sources so you can get straight to patching. New in v0.7.0, if the module has a public repo, it'll pull directly from that. Rockstar!

Git::CPAN::Patch could already seed a local repository with the latest distribution of a module, or its whole BackPAN history, or its GitPAN mirror. But with version 0.7.0, it can now go straight for the meat and clone the distribution's officil git repository, provided that it's specified in its META.json or META.yml.
-- Read more: http://babyl.dyndns.org/techblog/entry/new-and-improved-git-cpan-patch-0.7.0

Provides two tools git cpan-sources and git cpan-init

HackDay! 11/12/11

David (DDubs!) came over for a HACKDAY today. Fun times!

He's in a maze of twisty little passages, all alike. He's installing SVN on localhost, with the full webDAV thing. Why? So he can practice migrating SVN to git. Yes, crazy land. I'm more looking forward to the actual project, a Bayesian Classifier to practice with Moose.

I've updated my App::PM::Website tool for maintaining the Los Angeles Perl Mongers website. It now uses Lingua::EN::Numbers::Ordinate to render dates like "Wednesday the 7th" instead of "Wednesday the 7." That's one more piece of manual hard coding replaced with software. Woot! Come on out on December 7th for Mike's talk on VOIP!

I really do want to get this code into a state where it's releasable to CPAN and usable by other monger groups. Maybe by making my vapor App::PM::Toolbox into reality? Why am I so tentative on releasing that?

I also updated and released v0.113160 of Hadoop::Streaming perl module, mostly to use Any::Moose. I made the changes a few months ago at a user's request. I sent him a beta version for testing, and it got lost in the weeds. I merged the code over today from feature/mouse and pushed it out. While I was in there, I updated the documentation in the main package to show how to use the '-archive' flag to hadoop.

I love typing 'dzil release' and having my Changes file updated, checked into git, release git tagged, release built and bundle pushed to PAUSE for CPAN.

If I had checked github first, I would have seen this lovely change request waiting where a user had made the Any::Moose conversion for me. My first incoming github change request. ROCKSTAR!

This post brought to you by the number 30 and the letter D (eltron).

2011-11-13 00:52:18 $$5747 v1049: Info: Need to get uriid[S/SP/SPAZM/Hadoop-Streaming-0.113160.tar.gz] (paused:337)
2011-11-13 00:52:18 $$5747 v1049: Info: Going to fetch uriid[S/SP/SPAZM/Hadoop-Streaming-0.113160.tar.gz] (paused:625)
2011-11-13 00:52:18 $$5747 v1049: Info: Requesting a GET on uri [ftp://pause.perl.org/incoming/Hadoop-Streaming-0.113160.tar.gz] (paused:647)
2011-11-13 00:52:19 $$5747 v1049: Info: renamed '/home/ftp/tmp/S/SP/SPAZM/Hadoop-Streaming-0.113160.tar.gz' to '/home/ftp/pub/PAUSE/authors/id/S/SP/SPAZM/Hadoop-Streaming-0.113160.tar.gz' (paused:760)
2011-11-13 00:52:19 $$5747 v1049: Info: Got S/SP/SPAZM/Hadoop-Streaming-0.113160.tar.gz (size 28088) (paused:496)
2011-11-13 00:52:20 $$5747 v1049: Info: Sent 'has entered' email about uriid[S/SP/SPAZM/Hadoop-Streaming-0.113160.tar.gz] (paused:561)
2011-11-13 00:53:46 $$5747 v1049: Info: Verified S/SP/SPAZM/Hadoop-Streaming-0.113160.tar.gz (paused:308)
2011-11-13 00:53:46 $$5747 v1049: Info: Started mldistwatch for lpath[/home/ftp/pub/PAUSE/authors/id/S/SP/SPAZM/Hadoop-Streaming-0.113160.tar.gz] with pid[11168] (paused:313)

Wednesday, October 26, 2011

Perl for newbies

I've heard the tutorials at Perl-Begin.org are a good way to get started with perl. Current, modern perl. Not some 15-year-old script kiddie intro.

Speaking of, it might be interesting to make a learnperlthehardway book, ala Write your own LxTHW and the companion site LearnCodeTheHardWay.org.

Sunday, October 16, 2011

Homework!

It's been a long time since I've had schoolwork. Some habits die hard -- here I am 1 week in and rushing/procrastinating/just-plain-late with my first assignments!

I'm taking two courses at Stanford this fall. Maybe you know someone else in one of them? There are one hundred thousand people in my AI class. 100,000. Even our largest freshman intro courses at Caltech had fewer than 215 people, so this is three decimal orders of magnitude larger. Wow. I'm not sure how many people are signed up for the Machine Learning class, but it's probably similarly large.

Back to work!

Saturday, October 15, 2011

Author dependencies in Dist::Zilla

Wanna edit/tweak/build/play-with a Dist::Zilla based perl module you've checked out? Seems daunting because "the module installer isn't included" or "what if I don't have the same helper modules that the author uses?" ? Worry Not!

It's easy to get the minimal pieces installed, so let's get to it.

Basic steps for building/using a Dist::Zilla based module from raw source:

  1. Check out module source
  2. install Dist::Zilla:
    cpanm Dist::Zilla
  3. install author deps:
    dzil authordeps | cpanm
  4. install module deps:
    dzil listdeps | cpanm
  5. build module with dzil:
    dzil build

1. Check out module source

For my example, I'm migrating my own App::PM::Website sources from an old laptop to a new one.

Looking for example code to checkout? Try searching for dist.ini on github to find an interesting perl module. ;)

% git clone git@github.com:spazm/app-pm-website 
% cd app-pm-website

Check the code out of the repository and cd into the top level.

2. Install Dist::Zilla

On this relatively clean perl 5.12.3 install, cpanm Dist::Zilla brought in 81 packages.
% cpan Dist::Zilla
[andrew@fred]% cpanm Dist::Zilla           127 (git)-[dev] ~/src/app-pm-website--> Working on Dist::ZillaFetching http://search.cpan.org/CPAN/authors/id/R/RJ/RJBS/Dist-Zilla-4.300002.tar.gz ... OK
[... snip ...]
Building and testing Dist-Zilla-4.300002 ... OKSuccessfully installed Dist-Zilla-4.30000291 distributions installed

3. Install author dependencies

dzil authordeps will show the modules necessary for Dist::Zilla to build the module from raw source into a built module. Pipe this to cpanm to install the modules.
% dzil authordeps
Dist::Zilla::Plugin::MetaResources
Dist::Zilla::Plugin::AutoPrereqs
Dist::Zilla::Plugin::Repository
Dist::Zilla::Plugin::NextRelease
Dist::Zilla::PluginBundle::Basic
Dist::Zilla::Plugin::AutoVersion
Dist::Zilla::PluginBundle::Git
Dist::Zilla::Plugin::PkgVersion
Dist::Zilla::Plugin::MetaJSON
Dist::Zilla::Plugin::PodWeaver

% dzil authordeps | cpanm
Dist::Zilla::Plugin::MetaResources is up to date. (4.300002)
--> Working on Dist::Zilla::Plugin::Repository
[... snip ...]
Successfully installed Dist-Zilla-Plugin-PodWeaver-3.101641
11 distributions installed

4. Install module dependencies

Install the authordeps before module dependencies, in case authordeps are required for dzil to calculate the module dependencies. E.g. I needed PodWeaver installed via authordeps before I could run dzil listdeps to see the module dependencies.
% dzil listdeps
App::Cmd
App::Cmd::Command
App::Cmd::Tester
base
Config::YAML
Data::Dumper
Date::Parse
DateTime
DateTime::Format::Strptime
ExtUtils::MakeMaker
HTTP::DAV
Net::Netrc
POSIX
strict
Template
Test::Class
Test::Class::Load
Test::More
warnings

% dzil listdeps | cpanm
App::Cmd is up to date. (0.312)
base is up to date. (2.15)
--> Working on Config::YAML
[...snip...]
Successfully installed Test-Class-0.36
Test::More is up to date. (0.98)
10 distributions installed

5. Build module

dzil build will build the module into a directory and tar it up ready for cpan.

Similarly, dzil test will build the code and run the tests, you'll use this to verify your changes to the target module.

% dzil build
[DZ] beginning to build App-PM-Website
[DZ] guessing dist's main_module is lib/App/PM/Website.pm
[DZ] extracting distribution abstract from lib/App/PM/Website.pm
[DZ] writing App-PM-Website in App-PM-Website-0.112890
[DZ] building archive with Archive::Tar; install Archive::Tar::Wrapper for improved speed
[DZ] writing archive to App-PM-Website-0.112890.tar.gz
And now I can get back to the task at hand, improving this module. I'll let you know how that goes too.

Friday, September 16, 2011

Fix linux DNS issues with .local addresses on MS domain

B.L.U.F.:

Microsoft uses .local as the recommended root of internal domains, and serves them via unicast dns. Linux uses .local as the root of multicast dns. If you're stuck on a broken MS network like this, reconfigure your linux multicast DNS to use a different domain like .alocal.

To do this, add a "domain-name=.alocal" line to the "[server]" section of "/etc/avahi/avahi-daemon.conf", then restart avahi-daemon: "sudo service avahi-daemon restart".

#/etc/avahi/avahi-daemon.conf
[server]
domain-name=.alocal

You may need to flush the DNS,mDNS and resolver cache, as well as restart your web browsers to clear their internal cache.

Background.

I was seeing the strangest behavior on my work linux box. I could look up local addresses, but not contact them in my browser. Turns out I could look them up but not ping them, either.
% host foo
foo.corp.local is an alias for bar.corp.local
bar.corp.local has address 10.1.2.3

% host foo.corp.local
foo.corp.local is an alias for bar.corp.local
bar.corp.local has address 10.1.2.3

% ping foo -q -c 1
PING bar.corp.local (10.1.2.3) 56(84) bytes of data.

--- bar.corp.local ping statistics ---
1 packets transmitted, 1 recieved, 0% packet loss, time 0ms

% ping foo.corp.local
unknown host foo.corp.local
I spent a while thinking this was a resolver issue in /etc/resolv.conf, since I knew that was getting modified by the VPN. Everything was fine in the resolver. What I'd forgotten about was /etc/nsswitch.conf! The hosts line in /etc/nsswitch.conf put mdns4_minimal before dns AND set a reply of "NOTFOUND" from mdns to propagate back directly without hitting DNS.
# /etc/nsswitch.conf hosts line:
hosts: files mdns4_minimal [NOTFOUND=return] dns mdns4
We could side-step the problem by removing mdns4_minimal from the hosts search path, but this will lead to potentially long dns timeouts from mistyped .local addresses. (Ok, that's not a very bad side effect, but still let's fix it correctly).

Dig a little deeper into .local and mdns, and you'll find Avahi. Avahi "facilitates service discovery on a local network via the mDNS/DNS-SD protocol suite," what Apple calls Bonjour or Zeroconf. They have a warning page about unicast .local DNS zones that gets to the crux of the problem : linux has mdns (multicast dns) support configured for .local, but Microsoft support suggests using .local with unicast DNS. The two don't get along at all. mDNS/DNS-SD is inherently incompatible with unicast DNS zones .local. We strongly recommend not to use Avahi or nss-mdns in such a network setup. N.B.: nss-mdns is not typically bundled with Avahi and requires a separate download and install.
-- Avahi and Unicast Dot Local wiki page

Fixes:

  1. move avahi mdns from .local to a different name (e.g. .alocal)
  2. or Remove mdns from /etc/nsswitch.conf or remove mdns module.
For the former, add a domain-name=.alocal line to the [server] section of /etc/avahi/avahi-daemon.conf, then restart avahi-daemon: sudo service avahi-daemon restart.

If that doesn't work (and you restarted your browsers, with their insidious dns cache, right?) you can try removing mdns from the hosts entry in /etc/nsswitch.conf. replace this line:

hosts: files mdns4_minimal [NOTFOUND=return] dns mdns4
with this line:
hosts: files dns
Links:

Tuesday, August 9, 2011

Demand Media buys RSS Graffiti

Demand Media is announcing today their purchase of RSS Graffiti. This is the last week of our El Segundo office. Next monday I'll be working in Santa Monica again. Super exciting.

"Demand Media Acquires RSS Graffiti RSS Graffiti's Talented Engineering Team and Popular Facebook Application to Help Accelerate Demand Media's Social Publishing Strategy"
-- http://www.marketwatch.com/story/demand-media-acquires-rss-graffiti-2011-08-09
"In addition to releasing its Q2 earnings today, content aggregator Demand Media (NYSE: DMD) has made two acquisitions: it’s bought a long-running partner, the blog ad network IndieClick and RSS Graffiti, a Los Angeles-based developer of social media products, primarily Facebook. "
-- http://paidcontent.org/article/419-demand-media-buys-indieclick-for-14-million-expands-google-ad-deal/

Friday, August 5, 2011

irrational markets

Two quotes to ponder from economist John Maynard Keynes. I searched this morning and found this first one, a variant of which I've been saying for years. I really wanted to think it was my own creation. I'm glad I looked it up, as I found this second quotation updating opinions to follow facts rather than updating "facts" to follow opinions.

Markets can remain irrational a lot longer than you and I can remain solvent.
---- John Maynard Keynes

When the facts change, I change my mind. What do you do, sir?
---- John Maynard Keynes

Thank you wikiquote!

Sunday, July 17, 2011

Wedding Dance

My latest Dancer project is up at SweetieBeast.us, a website for my pending wedding. T-minus-13 days and counting!

The source is up at github. Look in the wedding-dancer directory for the dancer project.

The dancer code is very short, because I'm not really using any dancer bits. I'm using dancer to provide an interface around templating code and page layout wrapping. As such, the pages are all pure templates.

Version .1 was very close to the following code snippet. Auto_page turns my template files into corresponding routes. I then created a template for each of my pages, spent a day futzing with CSS and poof: wedding website!

package sweetiebeast;
use Dancer ':syntax';
our $VERSION = '0.1';
set auto_page => 1;
get '/' => sub { template 'index'; };
1

When I launched, I put the site behind an apache proxypass directive. So I added Dancer::Plugin::Proxy to create links relative to the external proxy. I query Dancer::Plugin::Proxy to explicitly create link targets in the model to simplify my views.

package sweetiebeast;
use Dancer ':syntax';
use Dancer::Plugin::ProxyPath;
our $VERSION = '0.2';

set auto_page => 1;
get '/' => sub { template 'index'; };

before_template sub {
    my $tokens = shift;
    $tokens->{uri_base}           = proxy->uri_for("/");
    $tokens->{uri_for_index}      = proxy->uri_for("/");
    $tokens->{uri_for_faq}        = proxy->uri_for("/faq");
    $tokens->{uri_for_location}   = proxy->uri_for("/location");
    $tokens->{uri_for_travel}     = proxy->uri_for("/travel");
    $tokens->{uri_for_guests}     = proxy->uri_for("/guests");
    $tokens->{uri_for_images}     = proxy->uri_for("/images");
};
true;

Monday, June 27, 2011

Macintosh Fork Failure

Random failures on my macbook air today. Oh I see, fork failures. The default maxproc setting is ridiculously low (512).

Let's double that:

% sudo sysctl -w kern.maxproc=1024
kern.maxproc: 532 -> 1024
% sudo sysctl -w kern.maxprocperuid=1024
kern.maxprocperuid: 512 -> 1024
Now let's check the ulimits, now that only my shells are affected.
% ulimit -a
-t: cpu time (seconds)         unlimited
-f: file size (blocks)         unlimited
-d: data seg size (kbytes)     unlimited
-s: stack size (kbytes)        8192
-c: core file size (blocks)    0
-v: address space (kb)         unlimited
-l: locked-in-memory size (kb) unlimited
-u: processes                  266
-n: file descriptors           2560
% ulimit -a 512
Now, why am I still getting fork errors when opening a new terminal? What process has this 266 locked in?

Friday, June 17, 2011

my first yapc

I'm looking forward to my trip to Asheville, NC for YAPC::na. June 26-30. Yet Another Perl Conference, North America.

I was supposed to go last year while I was at Rubicon Project, but my trip got canceled (some mix of busy, crunch time, and politics).

I haven't been to a focused perl conference since the short lived "O'Reilly University" circa 2000 in nyc. MJD was the bomb! It'll be interesting to compare with our <shameless_plugs> excellently run LA pm</shameless_plug> and all the other LA tech meet-ups.

what do I need to do to prep? Rest my brain and liver, I suppose.

Grub prompt after upgrade to ubuntu 11-04

I got bit by this when I upgraded "mini", my 11" acer to ubuntu 11.04. Rebooting left me at a grub prompt. The exercise has left me with an improved feel for the grub2 interface and boot prompt -- it's actually pretty slick with the TAB completion. Improved in-grub help would have been nifty, instead I found myself doing research on my phone.

Using information from aaron kelleys blog[1] and information available from tab completion I performed the following steps which led to a successful boot. After the root command, filename tab completion looks at that drive.

Boot from grub prompt:

root (hd0.5)
linux /vmlinuz root=/dev/sda ro
initrd /initrd.img
boot

Alternative, equivalent boot from grub prompt:

linux (hd0,5)/vmlinuz root=/dev/sda ro
initrd (hd0,5)/initrd.img
boot
After booting, these steps updated grub on the MBR for my root drive:
sudo grub-install /dev/sda
sudo update-grub

For a taste of the flip side, I restarted into windows for windows update. Looks like it's been 6 months since I booted windows7, so 300Meg to download. Question: Will it be finished or pwned when I get home tonight?

links:

  1. http://aaron-kelley.net/blog/2011/04/grub-prompt-after-upgrade-to-ubuntu-11-04/

Saturday, June 4, 2011

migrate Subversion repository/dump to Git & Github, with tags and branches

A simple and complete method for migrating from subersion to git, bringing over past tags and branches.

I migrated a work repository from Subversion to Git last week. It went surprisingly smoothly. There were a more steps than I expected after reading various, selected, sources.

My repository was in "standard layout" containing only a single tree/project. I was sent a dump of the repository created with svndump. If you have direct access to your svn repository, skip step one. Similarly you can skip the github steps or replace them with your own "primary git server."

Note, if you don't want to maintain the svn tags and branches, then there is a simple single step solution using git-svn. This works if you don't have tags and branches to keep, or you just want a personal git checkout of a remote svn repository.
git svn clone --no-metadata --stdlayout --A users.txt http://svn/repo/here/ new_checkout_name/

Overview:

  1. import svn dump into svn repository
  2. create mapping file of svn committer to email address
  3. create empty git repository
  4. cd to git repository
  5. import from svn repo to git repo
  6. pull branch and tag information from svn to git
  7. create github repository
  8. add remote repository
  9. push all tags and branches to github repository.
Steps:
  1. import svn dump into svn repository from dump file svn_repo.dump
    % svnadmin create svnrepo
    % svnadmin load svnrepo < svn_repo.dump
  2. create mapping file of svn committer to email address
    Create a file called authors.txt and fill it with line separated “svn-name = git-name ” pairs.
    #example:
    % cat authors.txt
    user_a = Alpha <alpha@example.com%gt;
    user_b = B Dawg <bdawg@github.example.com%gt;
  3. create empty git repository
    % mkdir gitrepo
    % git init gitrepo
  4. import from svn repo to git repo
    #   git svn clone file:///pathto/svnrepo /pathto/gitrepo –no-metadata -A authors.txt --stdlayout
    svn repo path must be an absolute path when using file://
    % git svn clone file://$(pwd)/svnrepo gitrepo/ --no-metadata -A authors.txt --stdlayout
  5. cd to the git repository
    % cd gitrepo
  6. pull branch and tag information from svn to git
    tags and branches are both viewed as remote branches, tags are prefixed with "tag/".
    For each branch you want to migrate, make a local branch.
    For each tag, make a local tag.
    % git svn fetch                          # fetch remote branches and tags from svn
    % git branch -r                          # view remote branches
    % git branch local_branch remote_branch  # for each branch to migrate
    % git tag local_tag tags/remote_tag      # for each tag to migrate
    % git branch                             # check work by viewing local branches and ...
    % git tag -l                             # ... local tags 
  7. create github repository
    ( github repo ui, create yourrepo )
  8. add remote repository
    % git remote add origin git@github.com:yourname/yourrepo
  9. push all tags and branches to github repository.
    % git push origin --tags   # push all tags
    % git push origin --all    # push all branches