Dzil plugins: GitHub::Meta vs GithubMeta

"Dear lazyweb: how do I add a github repository link to my dist-zilla dist.ini?"

Curtis Poe (@ovidperl) via twitter

Ovid asked how to add git repo information to a dist.ini file. Both Dist::Zilla::Plugin::GitHub::Meta and Dist::Zilla::Plugin::GithubMeta were suggested, along with the old-school Dist::Zilla::Plugin::MetaResources.

I've used MetaResources to include repository information in my dist.ini files. Which plugin should I use now: GitHub::Meta GithubMeta?

My MetaResources usage is pretty simple: homepage + repository information. The GitHub::Meta and GithubMeta versions are nearly the same from the dist.ini side, both have the same format for manually overriding the default homepage:

#MetaResources Github example:
[MetaResources]
homepage        = http://lowlevelmanager.com/
repository.web  = http://github.com/spazm/app-pm-website
repository.url  = http://github.com/spazm/app-pm-website.git
repository.type = git

## GithubMeta version
[GithubMeta]
homepage = http://lowlevelmanager.com/

## GitHub::Meta version
[GitHub::Meta]
homepage = http://lowlevelmanager.com/

There are differences if we look inside the modules. GithubMeta makes a system call to git to get remote url(s). The plugin will only run if called from within a git repository and with git in $PATH. If it finds a github.com url, it uses that as the remote. This url is then parsed to get the github user name and project name.

Conversely, GitHub::Meta shells out to git to get the github.user entry from git config via git config github.user and then makes an API call to github to get the project name. It does have an interesting feature where it can check if the checkout is a fork and use the upstream information instead.

GitHub::Meta could be nice if you want to go all-in and auto-create the repository with GitHub::Create and push updates with GitHub::Update (actually, this just seems to change the homepage url?), all of them will require the github.user configuration set in git. The docs suggest using GitHub::Create with Git::Init. I must admit, I'm behind the DZil times, as I'm not even sure where to store the default plugin options to apply when running dzil new, to actually trigger these plugins.

GithubMeta does what you'd expect and does it with a minimum of fuss. No extra options to set in your git configuration, but you do have to have the github remote added to your repo by other means. This seems a much slimmer change for existing repositories.

Zsh: merge stdout and stderr with |&

|& provides a zsh-only shortcut for merging STDOUT and STDERR when piping. It is the same as 2>&1 | , just a lot shorter to type. I find this useful for commands that feel like dumping --help output to STDERR.

command --help |& less

Pro Tip: Don't confuse this with &| which backgrounds the final command of the pipeline.

Find out more in the "Simple commands & pipelines" section of zshmisc:

A pipeline is either a simple command, or a sequence of two or more simple commands where each command is separated from the next by `|' or `|&'. Where commands are separated by `|', the standard output of the first command is connected to the standard input of the next. `|&' is shorthand for `2>&1 |', which connects both the standard output and the standard error of the command to the standard input of the next.

Update: |& is borrowed from csh. I don't know if it originated in csh or prior.

The standard error output may be directed through a pipe with the standard output. Simply use the form |& instead of just |.

-- man csh(1)

Zsh brace expansion and inline for loops

Two tips for today: zsh brace expansions [1] [2] and short for loops.

Brace Expansion:
{x..y} will expand to the integers x,x+1,...y. Prepend the numbers with padding for padded output. This will work in bash as well.

echo {3..5}     # 3 4 5
echo {3..100}   # 3 4 5 ... 10 11 12 ... 98 99 100
echo {03..100}  # 03 04 05 ... 10 11 12 ... 98 99 100
echo {003..100} # 003 004 005 ... 010 011 012 ... 098 099 100

Up until now, I've used seq to create integer lists like this. Using seq in commands requires a subshell. Do you know how to zero-pad the output of seq?

echo $(seq 3 5)   # 3 4 5
echo $(seq -w 3 5) # 3 4 5
echo $(seq -w 8 10) # 08 09 10

But today is about moving beyond seq. Let's move on to for loops.

For Loops:

% for i in {3..5}; do echo $i; done
3
4
5

Zsh has two shortened forms of this standard for loop. The short forms only support a single command. If you really want multiple commands, you can put them in {}, but then you might as well use "do ... done" in that case. These won't work in bash (woo!).

I never remember this when I need it and can't find references to this zsh specific "single line forloop" -- but it's right there in zshmisc,"Alternate Forms For Complex Commands."

for ... in list; comand
for ... (list) command

# short form with "in" and ";"
% for i in {3..5}; echo "line $i"; date
line 3
line 4
line 5
Fri May 11 15:52:32 PDT 2012

# short form with parens:
% for i ({3..5}) echo "line $i"; date
line 3
line 4
line 5
Fri May 11 15:52:32 PDT 2012

# short form with multiple commands in {}
% for i in {3..5}; { echo "line $i"; date }
line 3
Fri May 11 15:52:43 PDT 2012
line 4
Fri May 11 15:52:43 PDT 2012
line 5
Fri May 11 15:52:43 PDT 2012

Complex Commands
for name ... [ in word ... ] term do list done

where term is at least one newline or ;. Expand the list of words, and set the parameter name to each of them in turn, executing list each time. If the in word is omitted, use the positional parameters instead of the words.

More than one parameter name can appear before the list of words. If N names are given, then on each execution of the loop the next N words are assigned to the corresponding parameters. If there are more names than remaining words, the remaining parameters are each set to the empty string. Execution of the loop ends when there is no remaining word to assign to the first name. It is only possible for in to appear as the first name in the list, else it will be treated as marking the end of the list.

Alternate Forms For Complex Commands

for name ... ( word ... ) sublist

A short form of for.

for name ... [ in word ... ] term sublist

where term is at least one newline or ;. Another short form of for.

Zsh history expansion

You can refer back to prior elements of your command history, modify them, and make new commands. With Zsh expansion, history expansions can be expanded in-line before executing! Start by trying to learn a couple entries and slowly expand while building up your memory of useful mappings.

!$ and !* to reference args from prior commands

I've been using !$ (bang-last) for a while now. It expands to the last arg of the previous command. Use this when you are taking multiple actions with the same argument:

    % touch newfile
    % chmod a+x !$
    % vim !$

I'm now working !* into my command-line-fu, which pulls all the args from the previous command rather than just the last.

History Expansion:

! initiates history expansion, pulling items from your previous commands. The most commonly known/used abbreviation being !! to repeat the last command in full. Similarly !n repeats item n from your history and !-n goes back n commands. !-1 is synonymous with !!. !str is the most recent command to start with str. !# is current command line up to this point.

"Word Designators" can be applied to the selected history item, as we do above with $ and * . 0 is the command, n is the nth argument, x-y is args x through y. x* is like x-$ while x- is like x-$ excluding the last item.

I haven't even gotten started with the modifiers, like G for global on ^search^replace, a and A for prepending the current dir and crazy others.

Examples:

!!                # last command
!$                # final word of prior command
!*                # all words of prior command (excluding the command itself)
!3                # third command from history
!-3:*             # all words from three commands back 
^foo^bar          #last command, with the first substring foo replaced by bar
!-2:s^foo^bar     #full command from 2 back, with the first foo replaced by bar
!-2:s^foo^bar^:G  #full command from 2 back, with all "foo"s replaced by "bar"s

man zshexpn for all the details.

History Expansion

History expansion allows you to use words from previous command lines in the command line you are typing. This simplifies spelling corrections and the repetition of complicated commands or arguments. Immediately before execution, each command is saved in the history list, the size of which is controlled by the HISTSIZE parameter. The one most recent command is always retained in any case. Each saved command in the history list is called a history event and is assigned a number, beginning with 1 (one) when the shell starts up. The history number that you may see in your prompt (see EXPANSION OF PROMPT SEQUENCES in zshmisc(1)) is the number that is to be assigned to the next command.

Overview

A history expansion begins with the first character of the histchars parameter, which is '!' by default, and may occur anywhere on the command line; history expansions do not nest. The '!' can be escaped with '\' or can be enclosed between a pair of single quotes ('') to suppress its special meaning. Double quotes will not work for this. Following this history character is an optional event designator (see the section 'Event Designators') and then an optional word designator (the section 'Word Designators'); if neither of these designators is present, no history expansion occurs.
Input lines containing history expansions are echoed after being expanded, but before any other expansions take place and before the command is executed. It is this expanded form that is recorded as the history event for later references.

By default, a history reference with no event designator refers to the same event as any preceding history reference on that command line; if it is the only history reference in a command, it refers to the previous command. However, if the option CSH_JUNKIE_HISTORY is set, then every history reference with no event specification always refers to the previous command.

For example, '!' is the event designator for the previous command, so '!!:1' always refers to the first word of the previous command, and '!!$' always refers to the last word of the previous command. With CSH_JUNKIE_HISTORY set, then '!:1' and '!$' function in the same manner as '!!:1' and '!!$', respectively. Conversely, if CSH_JUNKIE_HISTORY is unset, then '!:1' and '!$' refer to the first and last words, respectively, of the same event referenced by the nearest other history reference preceding them on the current command line, or to the previous command if there is no preceding reference.

The character sequence '^foo^bar' (where '^' is actually the second character of the histchars parameter) repeats the last command, replacing the string foo with bar. More precisely, the sequence '^foo^bar^' is synonymous with '!!:s^foo^bar^', hence other modifiers (see the section 'Modifiers') may follow the final '^'. In particular, '^foo^bar:G' performs a global substitution.

If the shell encounters the character sequence '!"' in the input, the history mechanism is temporarily disabled until the current list (see zshmisc(1)) is fully parsed. The '!"' is removed from the input, and any subsequent '!' characters have no special significance.

A less convenient but more comprehensible form of command history support is provided by the fc builtin.

Event Designators

An event designator is a reference to a command-line entry in the history list. In the list below, remember that the initial '!' in each item may be changed to another character by setting the histchars parameter.
!
Start a history expansion, except when followed by a blank, newline, '=' or '('. If followed immediately by a word designator (see the section 'Word Designators'), this forms a history reference with no event designator (see the section 'Overview').

!!

Refer to the previous command. By itself, this expansion repeats the previous command.

!n

Refer to command-line n.

!-n

Refer to the current command-line minus n.

!str

Refer to the most recent command starting with str.

!?str[?]
Refer to the most recent command containing str. The trailing '?' is necessary if this reference is to be followed by a modifier or followed by any text that is not to be considered part of str.
!#
Refer to the current command line typed in so far. The line is treated as if it were complete up to and including the word before the one with the '!#' reference.

!{...}

Insulate a history reference from adjacent characters (if necessary).

Word Designators

A word designator indicates which word or words of a given command line are to be included in a history reference. A ':' usually separates the event specification from the word designator. It may be omitted only if the word designator begins with a '^', '$', '*', '-' or '%'. Word designators include:
0
The first input word (command).

n

The nth argument.

^

The first argument. That is, 1.

$

The last argument.

%

The word matched by (the most recent) ?str search.

x-y

A range of words; x defaults to 0.

*

All the arguments, or a null value if there are none.

x*

Abbreviates 'x-$'.

x-

Like 'x*' but omitting word $.

Note that a '%' word designator works only when used in one of '!%', '!:%' or '!?str?:%', and only when used after a !? expansion (possibly in an earlier command). Anything else results in an error, although the error may not be the most obvious one.
Modifiers

After the optional word designator, you can add a sequence of one or more of the following modifiers, each preceded by a ':'. These modifiers also work on the result of filename generation and parameter expansion, except where noted.
a
Turn a file name into an absolute path: prepends the current directory, if necessary, and resolves any use of '..' and '.' in the path. Note that the transformation takes place even if the file or any intervening directories do not exist.

A

As 'a', but also resolve use of symbolic links where possible. Note that resolution of '..' occurs before resolution of symbolic links. This call is equivalent to a unless your system has the realpath system call (modern systems do).

c

Resolve a command name into an absolute path by searching the command path given by the PATH variable. This does not work for commands containing directory parts. Note also that this does not usually work as a glob qualifier unless a file of the same name is found in the current directory.

e

Remove all but the extension.

h

Remove a trailing pathname component, leaving the head. This works like 'dirname'.

l

Convert the words to all lowercase.

p

Print the new command but do not execute it. Only works with history expansion.

q

Quote the substituted words, escaping further substitutions. Works with history expansion and parameter expansion, though for parameters it is only useful if the resulting text is to be re-evaluated such as by eval.

Q

Remove one level of quotes from the substituted words.

r

Remove a filename extension of the form '.xxx', leaving the root name.

s/l/r[/]
Substitute r for l as described below. The substitution is done only for the first string that matches l. For arrays and for filename generation, this applies to each word of the expanded text. See below for further notes on substitutions.
The forms 'gs/l/r' and 's/l/r/:G' perform global substitution, i.e. substitute every occurrence of r for l. Note that the g or :G must appear in exactly the position shown.
&
Repeat the previous s substitution. Like s, may be preceded immediately by a g. In parameter expansion the & must appear inside braces, and in filename generation it must be quoted with a backslash.

t

Remove all leading pathname components, leaving the tail. This works like 'basename'.

u

Convert the words to all uppercase.

x

Like q, but break into words at whitespace. Does not work with parameter expansion.

debugging nagios remote nrpe commands

Nagios NRPE debugging steps:

1. run the command manually on the target host
   
2. enable debugging in nrpe.cfg and watch syslog
   
3. dig in deeper with debug jobs.
   

Debugging nagios remote nrpe commands can feel very opaque. Normally I find my issue in step 1 of the debug escalation. Today I had to hit all three steps while debugging a test that wrapped around s3cmd. I eventually found that HOME is not set in the environment used to run nrpe commands.

check_nrpe -H 10.7.202.92 -c check_ui_s3_backup                                 
NRPE: Unable to read output

I run my check remotely and receive the dreaded general error "unable to read output." This means the script failed to run and didn't produce any output to STDOUT. STDERR seems to be ignored, even with logging enabled.

Step 1a: go to the server and verify the command being run by nrpe.

[andrew@ip-10-7-202-92]% grep check_ui_s3_backup /etc/nagios/nrpe.d/herbie.cfg
command[check_ui_s3_backup]=HOME=~postgres /usr/lib/nagios/plugins/herbie/check_ui_s3_backup

Step 1b: run the command manually. Here I find that the script fails if I don't have the config file:

[andrew@ip-10-7-202-92]% /usr/lib/nagios/plugins/herbie/check_ui_s3_backup
ERROR: /home/andrew/.s3cfg: No such file or directory
ERROR: Configuration file not available.
ERROR: Consider using --configure parameter to create one.

That should be a simple fix. Find the user running the nrpe command and give them a .s3cfg. Easy-Peasy.

cp .s3cfg ~nagios/
sudo -u nagios -H /usr/lib/nagios/plugins/herbie/check_ui_s3_backup
OK - Last backup 0 days ago.

Ok, it works locally. Recheck it remotely. It fails?!!?! This is where we start the gnashing of teeth and pulling of hair

[andrew@ip-10-7-203-10]% check_nrpe -H 10.7.202.92 -c check_ui_s3_backup
NRPE: Unable to read output

Step 2: Enable logging in nrpe.cfg, run the remote check and inspect the logs. Surprise, nothing useful.

May  7 19:08:29 ip-10-7-202-92 nrpe[17159]: Connection from 10.7.203.10 port 15286
May  7 19:08:29 ip-10-7-202-92 nrpe[17159]: Host address is in allowed_hosts
May  7 19:08:29 ip-10-7-202-92 nrpe[17159]: Handling the connection...
May  7 19:08:29 ip-10-7-202-92 nrpe[17159]: Host is asking for command 'check_ui_s3_backup' to be run...
May  7 19:08:29 ip-10-7-202-92 nrpe[17159]: Running command: /usr/lib/nagios/plugins/herbie/check_ui_s3_backup
May  7 19:08:29 ip-10-7-202-92 nrpe[17159]: Command completed with return code 3 and output: 
May  7 19:08:29 ip-10-7-202-92 nrpe[17159]: Return Code: 3, Output: NRPE: Unable to read output
May  7 19:08:29 ip-10-7-202-92 nrpe[17159]: Connection from 10.7.203.10 closed.

Step 3: debug jobs (aka printf aka "hail marry" debugging). I create two new nrpe entries and restart nagios-nrpe-server. The first will show me the user running the command and the second will show the environment, using whoami and env respectively.

command[check_ui_test]=whoami
command[check_ui_test2]=env

[andrew@ip-10-7-203-10]% check_nrpe -H 10.7.202.92 -c check_ui_test
nagios
[andrew@ip-10-7-203-10]% check_nrpe -H 10.7.202.92 -c check_ui_test2
NRPE_PROGRAMVERSION=2.12
TERM=screen-256color-bce
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
LANG=en_US.UTF-8
NRPE_MULTILINESUPPORT=1
PWD=/

Yep, the tests are running as the expected user, nagios.

Holy Schmoly! Look at that environment! HOME is not set. Simple enough to fix for my check, but wow was I not expecting that. Also useful to note the minimal PATH and that the working directory is /.

Update check to explicitly set HOME in the environment and restart nrpe:

command[check_ui_s3_backup]=HOME=~nagios /usr/lib/nagios/plugins/herbie/check_ui_s3_backup

Restart nrpe:

[andrew@ip-10-7-202-92]% sudo service nagios-nrpe-server restart
 * Stopping nagios-nrpe nagios-nrpe
   ...done.
 * Starting nagios-nrpe nagios-nrpe
   ...done.

Check:

[andrew@ip-10-7-203-10]% check_nrpe -H 10.7.202.92 -c check_ui_s3_backup
OK - Last backup 0 days ago.

zsh history: extend and persist

Zsh can do so much awesome stuff with your history. Yet by default it doesn't save to file and only stores 30 items.

A history mechanism for retrieving previously typed lines (most simply with the Up or Down arrow keys) is available; note that, unlike other shells, zsh will not save these lines when the shell exits unless you set appropriate variables, and the number of history lines retained by default is quite small (30 lines). See the description of the shell variables (referred to in the documentation as parameters) HISTFILE, HISTSIZE and SAVEHIST in zshparam(1).

Zsh can do so much if you ask it: save lots of lines, store them on exit, merge multiple concurrent histories on save, store and merge incrementally! A dizzying array of options.

Bottom Line Up Front: Having done the following research, I'm adding some new options to my .zshrc:

# .zshrc
## History
HISTFILE=$HOME/.zhistory       # enable history saving on shell exit
setopt APPEND_HISTORY          # append rather than overwrite history file.
HISTSIZE=1200                  # lines of history to maintain memory
SAVEHIST=1000                  # lines of history to maintain in history file.
setopt HIST_EXPIRE_DUPS_FIRST  # allow dups, but expire old ones when I hit HISTSIZE
setopt EXTENDED_HISTORY        # save timestamp and runtime information

Let's start digging! Checking the zshparam manpage we see:

HISTFILE
The file to save the history in when an interactive shell exits. If unset, the history is not saved.
HISTSIZE <S>
The maximum number of events stored in the internal history list. If you use the HIST_EXPIRE_DUPS_FIRST option, setting this value larger than the SAVEHIST size will give you the difference as a cushion for saving duplicated history events.
SAVEHIST
The maximum number of history events to save in the history file.

As we continue digging, we find some of the other exciting options by looking at the HISTORY section of the zshoptions manpage: APPEND_HISTORY, EXTENDED_HISTORY, SHARE_HISTORY, INC_APPEND_HISTORY, HIST_SAVE_NO_DUPS, HIST_SAVE_BY_COPY, HIST_REDUCE_BLANKS, .

Once you start merging histories (APPEND_HISTORY, INC_APPEND_HISTORY or SHARE_HISTORY), you'll want to limit the duplications ( HIST_EXPIRE_DUPS_FIRST, HIST_IGNORE_DUPS, HIST_IGNORE_ALL_DUPS, HIST_SAVE_NO_DUPS).

I found INC_APPEND_HISTORY to be too confusing, merging all my current shells into one shared history, live as I go. I want !! or up-arrow to repeat the last command in THIS VERY SHELL.

APPEND_HISTORY
If this is set, zsh sessions will append their history list to the
history file, rather than replace it. Thus, multiple parallel zsh ses‐
sions will all have the new entries from their history lists added to
the history file, in the order that they exit. The file will still be
periodically re-written to trim it when the number of lines grows 20%
beyond the value specified by $SAVEHIST (see also the
HIST_SAVE_BY_COPY option).
INC_APPEND_HISTORY
This options works like APPEND_HISTORY except that new history lines
are added to the $HISTFILE incrementally (as soon as they are
entered), rather than waiting until the shell exits. The file will
still be periodically re-written to trim it when the number of lines
grows 20% beyond the value specified by $SAVEHIST (see also the
HIST_SAVE_BY_COPY option).
EXTENDED_HISTORY
Save each command's beginning timestamp (in seconds since the epoch)
and the duration (in seconds) to the history file. The format of this
prefixed data is:

`: :;'.
SHARE_HISTORY
This option both imports new commands from the history file, and also
causes your typed commands to be appended to the history file (the
latter is like specifying INC_APPEND_HISTORY). The history lines are
also output with timestamps ala EXTENDED_HISTORY (which makes it eas‐
ier to find the spot where we left off reading the file after it gets
re-written).
HIST_EXPIRE_DUPS_FIRST
If the internal history needs to be trimmed to add the current command
line, setting this option will cause the oldest history event that has
a duplicate to be lost before losing a unique event from the list.
You should be sure to set the value of HISTSIZE to a larger number
than SAVEHIST in order to give you some room for the duplicated
events, otherwise this option will behave just like
HIST_IGNORE_ALL_DUPS once the history fills up with unique events.
HIST_IGNORE_ALL_DUPS
If a new command line being added to the history list duplicates an
older one, the older command is removed from the list (even if it is
not the previous event).
HIST_IGNORE_DUPS (-h)
Do not enter command lines into the history list if they are dupli‐
cates of the previous event.
HIST_SAVE_NO_DUPS
When writing out the history file, older commands that duplicate newer
ones are omitted.

Data-Driven Documents with d3.js and cubism.js

Cubism.js: "A D3 plugin for visualizing time-series data".

Cubism can pull stats from graphite to produce pretty dashboard monitors. It is built on top of d3.js : Data-Driven Documents.

We're slowing putting stats into graphite at work. It'd be great fun to find the time to play with something like this , even better for someone to build me something shiny. Any takers? :)

remove a file from the most recent git commit with 'git reset'

How to quickly and easily remove a file from your most recent git commit while maintaing the changes in the working directory. Use 'git reset HEAD^' followed by 'git commit --amend'.

Caveats: Don't modify your commits if they've been pushed upstream or shared (unless you know why and how and ... no, just don't).

git reset HEAD^ -- $file
git commit --amend

Notes:

1. This technique works nicely if you need to modify a previous commit by wrapping it inside a 'git rebase -i ...' and then 'edit'ing the commit you want.
2. My right-hand prompt shows the VCS system (git), branch and git status. zsh's vcs_info is AWESOME.
3. Yes, I do have a smiley face in my prompt. It goes from a green happy face to a red sad face if the last command failed.

Short Example:

% git add a b c
% git commit -m "adds a and c"
# realize I didn't mean to include b!
% git reset HEAD^ -- b
% git commit --amend

Full Example:

[vm53@vm53]% git status                                 :) (git)-[master] ~/src/bar
# On branch master
nothing to commit (working directory clean)
[vm53@vm53]% echo "a" > a; echo "b">b; echo "c">c       :) (git)-[master] ~/src/bar
[vm53@vm53]% git status                                 :) (git)-[master] ~/src/bar
# On branch master
# Untracked files:
#   (use "git add ..." to include in what will be committed)
#
#       a
#       b
#       c
nothing added to commit but untracked files present (use "git add" to track)
[vm53@vm53]% git add a b c                              :) (git)-[master] ~/src/bar
[vm53@vm53]% git commit                                 :) (git)-[master] ~/src/bar
### editor opens with the old git message, modify if desired
[master 18d73bd] adds a and c
 3 files changed, 3 insertions(+), 0 deletions(-)
 create mode 100644 a
 create mode 100644 b
 create mode 100644 c
[vm53@vm53]% git status                                 :) (git)-[master] ~/src/bar
# On branch master  
nothing to commit (working directory clean)
[vm53@vm53]% git reset HEAD^ -- b                       :) (git)-[master] ~/src/bar
[vm53@vm53]% git status                                 :) (git)-[master] ~/src/bar
# On branch master
# Changes to be committed:
#   (use "git reset HEAD ..." to unstage)
#
#       deleted:    b
#
# Untracked files:
#   (use "git add ..." to include in what will be committed)
#
#       b
[vm53@vm53]% git commit --amend -m "adds a and c"       :) (git)-[master] ~/src/bar
[master c15220b] adds a and c
 2 files changed, 2 insertions(+), 0 deletions(-)
 create mode 100644 a
 create mode 100644 c
[vm53@vm53]% git status                                 :) (git)-[master] ~/src/bar
# On branch master
# Untracked files:
#   (use "git add ..." to include in what will be committed)
#
#       b
nothing added to commit but untracked files present (use "git add" to track)
[vm53@vm53]%                                            :) (git)-[master] ~/src/bar

Email for the High D (DiSC model)

What does a High D want in her email?

Direct. No salutation, no contact info. Bottom line up front. Short: 4 paragraphs of 4 sentences is an upper bound, one to two sentences preferred. Actionable. No story, no background. NO ATTACHMENTS unless you are specifically sending them something they asked for. BEAM INFORMATION INTO YOUR HEAD.

A high S/high C would call the same email too short, underspecified and cold/angry.

You've got a huge corpus to train from, read your boss's emails and reply using her style. Improves communication (ie, the actual ability to get your thought into their head) and appreciation.

Email and the High D from the Career Tools podcast from the Manager Tools guys.

See also Email and the High I. Looking forward to "Email and the High C" and "Email and the High S," I hope they get recorded and shared.

More Storm Videos

Six videos covering an hour-and-a-half of Nathan Marz and Ted Dunning talking about Storm at SF DataMining.

Smile! with a Zsh prompt happy/sad face

UPDATE: Now in color!

%(?,%F{green}:%),%F{yellow}%? %F{red}:()%f

% RPROMPT='%(?,%F{green}:%),%F{yellow}%? %F{red}:()%f'
% /bin/true                                                                     :)
% /bin/false                                                                    :)
%                                                                             1 :(

Wouldn't you like a more descriptive shell prompt, that would show you the return value of the last command in a visually intuitive way? Sure, you could use %? to get the return int, zero for success and non-zero for failure. But that's just going to make your prompt more bizarre to your (pair-programming) partner.

Add this expansion sequence %(?.:%).:() to your PROMPT or RPOMPT to add a smiley/frowny face to your zsh prompt based on the return status of the previous command.

Let's see it in action! "Before" vs "After" of putting this in my RPROMPT to set the right prompt, to replace the raw %? return value I had previously.

#BEFORE
% RPROMPT='%? %~'                                                0 ~
% /bin/true                                                      0 ~
% /bin/false                                                     0 ~
%                                                                1 ~

#AFTER
% RPROMPT='%(?,:%),:() %~'                                       1 ~
% /bin/true                                                     :) ~
% /bin/false                                                    :) ~
%                                                               :( ~

Right, you're wondering how you'll live not knowing the actual failure int return code, right? Well, you don't have to give that up. Let's modify the false-text to prepend the failure int %? to the sad face:

#Add this to your PROMPT or RPROMPT: %(?,:%),%? :()

[vm53@vm53]% RPROMPT='%(?,:%),%? :() %~'                                   :) ~
[vm53@vm53]% /bin/true                                                     :) ~
[vm53@vm53]% /bin/false                                                    :) ~
[vm53@vm53]%                                                             1 :( ~

How does this work? We use the built-in conditional form, %(x.true-text.false-text), as documented in the zsh man pages under "CONDITIONAL SUBSTRINGS IN PROMPTS". Closing parenthesis must be encoded to appear in the true-text or false-text. ? checks against the return code of the previous command. Our true text makes a happy face :) and our false-text makes a sad face :(. Our updated sad face prepends the return value%? :(. I chose to prepend as the is the first item in my right prompt, this makes the smiley/frowny faces align.

Inspired by Selena Deckelmann's Use This Interview. Thanks Clark for showing me the post and then putting up the challenge to do it in zsh (and do it more cleanly).

CONDITIONAL SUBSTRINGS IN PROMPTS:

%(x.true-text.false-text)
              Specifies a ternary expression.  The character following  the  x
              is  arbitrary;  the  same character is used to separate the text
              for the `true' result from that for the  `false'  result.   This
              separator  may  not appear in the true-text, except as part of a
              %-escape sequence.  A `)' may appear in the false-text as  `%)'.
              true-text  and  false-text  may  both contain arbitrarily-nested
              escape sequences, including further ternary expressions.

              The left parenthesis may be preceded or followed by  a  positive
              integer  n,  which defaults to zero.  A negative integer will be
              multiplied by -1.  The test character x may be any of  the  fol‐
              lowing:

              !      True if the shell is running with privileges.
              #      True if the effective uid of the current process is n.
              ?      True if the exit status of the last command was n.
              _      True if at least n shell constructs were started.
              C
              /      True if the current absolute path has at least n elements
                     relative to the root directory, hence / is counted  as  0
                     elements.
              c
              .
              ~      True if the current path, with prefix replacement, has at
                     least n elements relative to the root directory, hence  /
                     is counted as 0 elements.
              D      True if the month is equal to n (January = 0).
              d      True if the day of the month is equal to n.
              g      True if the effective gid of the current process is n.
              j      True if the number of jobs is at least n.
              L      True if the SHLVL parameter is at least n.
              l      True  if  at least n characters have already been printed
                     on the current line.
              S      True if the SECONDS parameter is at least n.
              T      True if the time in hours is equal to n.
              t      True if the time in minutes is equal to n.
              v      True if the array psvar has at least n elements.
              V      True  if  element  n  of  the  array  psvar  is  set  and
                     non-empty.
              w      True if the day of the week is equal to n (Sunday = 0).

The Grail of Efficiency : premature optimization

Premature optimization is the root of all evil ... most of the time. You're only going to know that it's time to optimize after you've built. This doesn't get us off the hook for building slow systems nor for adding nonessential complexity into our solutions.

Build it first, then measure, then improve. And PS. you're bad at guessing what's wrong.

I like this longer version of knuth's quotation (though he claims he didn't coin the phrase) than the one that shows up on the wikipedia page on program optimization.

There is no doubt that the grail of efficiency leads to abuse. Programmers waste enormous amounts of time thinking about, or worrying about, the speed of noncritical parts of their programs, and these attempts at efficiency actually have a strong negative impact when debugging and maintenance are considered. We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil.

Yet we should not pass up our opportunities in that critical 3%. A good programmer will not be lulled into complacency by such reasoning, he will be wise to look carefully at the critical code; but only after that code has been identified. It is often a mistake to make a priori judgments about what parts of a program are really critical, since the universal experience of programmers who have been using measurement tools has been that their intuitive guesses fail.
-- Donald Knuth, ACM: Structured programing with go to statements

"Big Data" book by Nathan Marz

Big Data: Principle and best practices of scalable realtime data systems by Nathan Marz and Samuel E. Ritchie is now available in MEAP/Roughcut edition from Manning.

The early access edition of my book "Big Data" is now available. Use code bd50 for 50% off http://www.manning.com/marz/
---- @nathanmarz on twitter (link)

Questions about the book? Ask me here: news.ycombinator.com/item?id=3444300
---- @nathanmarz on twitter (link)

Nathan tweeted a discount code 'bd50' for 50% off, which is a nice bonus. I ordered a ebook+print bundle my copy on Monday. Looking forward to digging in and updating you all with a review. Excited to read the Appendix on storm.

Table of Contents:

1. A new paradigm for Big Data - FREE
2. Data model for Big Data - AVAILABLE
3. Data storage on the batch layer
4. MapReduce and batch processing
5. Batch processing with Cascading
6. Basics of the serving layer
7. Storm and the speed layer
8. Incremental batch processing
9. Layered architecture in-depth
10. Piping the system together
11. Future of NoSQL and Big Data processing
    • Appendix A: Hadoop
    • Appendix B: Thrift
    • Appendix C: Storm

Building a jabber bot with Bot::Backend

There are many ways to write a (jabber/xmpp) chat bot. A quick search for "Jabber Bot" turns up Net::Jabber::Bot, Bot::JabberBot, Bot::Backbone::Service::JabberChat, Bot::Jabbot, IM::Engine, AnyEvent::XMPP and more. You'll see even more if you search for XMPP instead.

How to choose?

I started with Net::Jabber::Bot, but ran into problems connecting to google-talk based chat. Jabber modules built on AnyEvent::XMPP (seem to) connect to google-auth better than those built on Net::XMPP. This is moot now that I have a normal jabber server, but it still tripped me up. The pod doc lays out funny, so I patched and filed a change-request at github.

Knowing that I would eventually want an event-loop based app, I focused on AnyEvent::XMPP packages. Bot::Backbone and IM::Engine are both interesting abstractions. IM::Engine is quote "currently alpha quality with serious features missing and is rife with horrible bugs." Perhaps I should be happy that he admits this up front? On to Bot::Backbone and sugary Moose!

Once I realized that the group_domain wouldn't be automatically filled in as "conference.$domain" in Bot::Backbone::Service::JabberChat, I was able to connect to a group chat room on my jabber server!

Follow along with my bot, App::Sulla on github. Thus far I have a working connection that responds to "!time" requests -- I've even fixed the part where it threw warnings on all other messages!

#this code in my dispatch table:
    also not_command not_to_me run_this {
            my ( $self, $message ) = @_;
            respond { "hello world" };
    };

# lead to the following error on every other chat message:
unhandled callback exception on event (message, AnyEvent::XMPP::Ext::MUC=HASH(0x3746818), AnyEvent::XMPP::Ext::MUC::Room=HASH(0x3a6d700) ANY_STRING ): Can't call method "add_predicate_or_return" on an undefined value at perl5/lib/perl5/Bot/Backbone/DispatchSugar.pm line 124.

I decided I wanted to pass the auth and channel information into App::Sulla as parameters to the constructor. This didn't play very well with the service sugar. service executes at compile time and squirrels away the configuration hash for the service into services. I added a before modifier to construct_services which is called during run just before initializing each service.

Storm is upon us!

A storm (from Proto-Germanic *sturmaz "noise, tumult") is any disturbed state of an astronomical body's atmosphere, especially affecting its surface, and strongly implying severe weather.
-- wikipedia:storm

Storm is open-source: distributed and fault-tolerant realtime computation
-- Nathan Marz via twitter

Twitter has open sourced Storm, a distributed real-time processing engine ( Announcement, Github ). Hadoop is to batch as Storm is to stream. This is huge. So huge that it seems to be mostly ignored?!

Storm's primary author, Nathan Marz, is the same guy who created cascalog for hadoop queries (a marriage of Clojure and Datalog running queries in hadoop) and ElephantDB. You'll remember he was working at BackType (their first non-founder hire. Good choice, guys!). Twitter bought BackType and now they are sharing storm with us. Thanks twitter!

Since Storm's release seems to have flown in under the radar and is a difficult generic search term, I've collected up links to the relevant resources.

Resources & Announcements:

StrangeLoop/Infoq presentation

http://www.infoq.com/presentations/Storm

Nathan opensourced storm in the middle of his StrangeLoop 2011 presentation.

The slides below the image update as the video plays.

Storm Slides

http://www.slideshare.net/nathanmarz/storm-distributed-and-faulttolerant-realtime-computation

StrangeLoop 2011 video links

https://thestrangeloop.com/news/strange-loop-2011-video-schedule

A Storm is Coming more details from twitter.

http://engineering.twitter.com/2011/08/storm-is-coming-more-details-and-plans.html

Storm Source on Github

https://github.com/nathanmarz/storm

Storm Wiki on Github

Wiki Front Page

Rationale

Tutorial

Mailing List

http://groups.google.com/group/storm-user

0.6.1 released

http://groups.google.com/group/storm-user/browse_thread/thread/72b3d4a4aebdebea

Testing Storm Topologies(in Clojure)

http://www.pixelmachine.org/2011/12/17/Testing-Storm-Topologies.html

Overview of Storm Presentation at BashoChats 001

http://basho.com/blog/technical/2011/12/20/Basho-Chats-001-Talk-Videos/

BackType techtalks

http://tech.backtype.com/pages/presentations-8

28c3: Effective Denial of Service attacks against web application platforms

Synopsis

Hour long presentation from the C3 security conference, on hash collision based attacks against web apps. The idea being that most web stacks automatically grab the query params and stick them into a hash of key-value pairs and by exploiting hash collisions an attacker can waste huge amounts of CPU time by a simple HTTP POST. In languages that don't randomly perturb their hashing functions (mostly DJBX33A or DJBX33X), collisions can be easily found and exploited.

This affects node.js/v8, php, python, ASP, ruby, java and etc. Only perl (5.8.1 ~2003) and cruby (1.9 ~2008) are patched for randomized hashing functions. The v8 devs are unconcerned "because v8 is a client side language," which caused a bit of alarm for the node.js folks.

The presenters started their investigation after reading a mention in the perl security faq (perldoc perlsec), in the section Algorithmic Complexity Attacks, that before 5.8.1 perl had a security flaw where hash collisions could be exploited. 5.8.1 was released in 2003.

           In Perls before 5.8.1 one could rather easily generate data that as
           hash keys would cause Perl to consume large amounts of time because
           internal structure of hashes would badly degenerate.  In Perl 5.8.1
           the hash function is randomly perturbed by a pseudorandom seed
           which makes generating such naughty hash keys harder.  See
           "PERL_HASH_SEED" in perlrun for more information.

Thread from the node.js mailing list: "HOLY CRAP. nearly all nodejs http servers are vulnerable to DoS and apparently, the V8 guys seem to not care much"

Related Node.js followup:

• https://plus.google.com/103939784105159993090/posts/Sh2Nxw8p3WB
• https://plus.google.com/116910304844117268718/posts/BfTdM25d23y

Talent, Bias and Diversity

Inc.com has a nice article on talent: You Can't Predict Talent; Foster It . His tips for fostering talent: open atmosphere, extravagant diversity, time didn't matter, stretch goals were just the start. The article leads off with this quote:

In his recent book Thinking Fast and Slow, behavioral economist Daniel Kahneman tells the story of observing army recruits out on exercises and his belief that he could spot the potential leaders amongst them. Years later, it turned out he'd been almost entirely wrong. His confident judgment had been a morass of bias, heuristics, and narrative fallacies.

This got me thinking about a more-or-less completely non-sequiter, vaguely related to open atmostphere and extravagant diversity.

As humans, we have brains built for pattern matching, so we find patterns. We apply these patterns all day long into systems of heuristics: "have I been in this position before? What did I do then? Did it work?" Actions that match our heuristics feel right because they work as expected and match previous experience.

We're making more of these patterns every day, and we are awesome in our ability to match situations against our patterns.

Problem: we're not good at evaluating whether our past decisions were correct. This introduces systemic bias. Bias expands directly with the homogeneity of your peer group.

Fixes? Think different. Use DATA to evaluate your decisions. Interact outside of your comfortable peer group. Be aware that you're using short cuts, and take the long way every-so-often to see if it really is longer. think.

Perl Advent Calendars, 2011 edition.

It's Advent Calendar time in the perl ecosystem! Start each day with a delicious treat of knowledge.

I've found a half dozen english language perl advent calendars, starting with the original perl advent calendar. For extra fun I've included another half dozen Japanese language calendars -- I can still read the perl it's just the prose that is lost in translation.

Ricardo (RJBS) has taken over the Perl Advent calendar this year, which is awesome. Sadly, that means he won't be doing his own "month of rjbs" calendar. I've added a link to his 2010 calendar, in case you missed it the first time around. He's starting the month with Day 1: cpanm and local::lib.

For a second year, Miyagawa has skipped updating plack advent calendar. Check out the 2009 edition linked below. He has given us plenty of other presents this year: Carton, etc.

Perl Advent

http://perladvent.org/2011/

(Formerly the perladvent.pm.org calendar)

Perl Dancer -- the dancer mini web framework

http://advent.perldancer.org/2011/

Catalyst Advent Calendar -- The Catalyst Web Framework

http://www.catalystframework.org/calendar/

Perl 6

http://perl6advent.wordpress.com/

For the adventurous: Japanese Perl Advent Calendars, 8 different tracks!

http://perl-users.jp/articles/advent-calendar/2011/

AnySan Track

Casual Track

dbix Track

English Track

Hacker Track

Test Track

Acme Track

Teng Track

Amon2 Track

Ricardo's 2010 advent calendar -- a month of RJBS

http://advent.rjbs.manxome.org/2010/

2009 Plack calendar

http://advent.plackperl.org/

One bonus list, for the sysadmin in your life:

SysAdvent - The Sysadmin Advent Calendar.

http://sysadvent.blogspot.com/

Evil: If I were creating the world I wouldn't mess about with butterflies and daffodils. I would have started with lasers, eight o'clock, Day One!
-- Time Bandits

Git::CPAN::Patch

Just saw an announcement of Git::CPAN::Patch over at Yanick's blog.

This is an awesome tool to create patches for other people's modules (OPM(tm)). It will create a git repo from current sources so you can get straight to patching. New in v0.7.0, if the module has a public repo, it'll pull directly from that. Rockstar!

Git::CPAN::Patch could already seed a local repository with the latest distribution of a module, or its whole BackPAN history, or its GitPAN mirror. But with version 0.7.0, it can now go straight for the meat and clone the distribution's officil git repository, provided that it's specified in its META.json or META.yml.
-- Read more: http://babyl.dyndns.org/techblog/entry/new-and-improved-git-cpan-patch-0.7.0

Provides two tools git cpan-sources and git cpan-init

HackDay! 11/12/11

David (DDubs!) came over for a HACKDAY today. Fun times!

He's in a maze of twisty little passages, all alike. He's installing SVN on localhost, with the full webDAV thing. Why? So he can practice migrating SVN to git. Yes, crazy land. I'm more looking forward to the actual project, a Bayesian Classifier to practice with Moose.

I've updated my App::PM::Website tool for maintaining the Los Angeles Perl Mongers website. It now uses Lingua::EN::Numbers::Ordinate to render dates like "Wednesday the 7th" instead of "Wednesday the 7." That's one more piece of manual hard coding replaced with software. Woot! Come on out on December 7th for Mike's talk on VOIP!

I really do want to get this code into a state where it's releasable to CPAN and usable by other monger groups. Maybe by making my vapor App::PM::Toolbox into reality? Why am I so tentative on releasing that?

I also updated and released v0.113160 of Hadoop::Streaming perl module, mostly to use Any::Moose. I made the changes a few months ago at a user's request. I sent him a beta version for testing, and it got lost in the weeds. I merged the code over today from feature/mouse and pushed it out. While I was in there, I updated the documentation in the main package to show how to use the '-archive' flag to hadoop.

I love typing 'dzil release' and having my Changes file updated, checked into git, release git tagged, release built and bundle pushed to PAUSE for CPAN.

If I had checked github first, I would have seen this lovely change request waiting where a user had made the Any::Moose conversion for me. My first incoming github change request. ROCKSTAR!

This post brought to you by the number 30 and the letter D (eltron).

2011-11-13 00:52:18 $$5747 v1049: Info: Need to get uriid[S/SP/SPAZM/Hadoop-Streaming-0.113160.tar.gz] (paused:337)
2011-11-13 00:52:18 $$5747 v1049: Info: Going to fetch uriid[S/SP/SPAZM/Hadoop-Streaming-0.113160.tar.gz] (paused:625)
2011-11-13 00:52:18 $$5747 v1049: Info: Requesting a GET on uri [ftp://pause.perl.org/incoming/Hadoop-Streaming-0.113160.tar.gz] (paused:647)
2011-11-13 00:52:19 $$5747 v1049: Info: renamed '/home/ftp/tmp/S/SP/SPAZM/Hadoop-Streaming-0.113160.tar.gz' to '/home/ftp/pub/PAUSE/authors/id/S/SP/SPAZM/Hadoop-Streaming-0.113160.tar.gz' (paused:760)
2011-11-13 00:52:19 $$5747 v1049: Info: Got S/SP/SPAZM/Hadoop-Streaming-0.113160.tar.gz (size 28088) (paused:496)
2011-11-13 00:52:20 $$5747 v1049: Info: Sent 'has entered' email about uriid[S/SP/SPAZM/Hadoop-Streaming-0.113160.tar.gz] (paused:561)
2011-11-13 00:53:46 $$5747 v1049: Info: Verified S/SP/SPAZM/Hadoop-Streaming-0.113160.tar.gz (paused:308)
2011-11-13 00:53:46 $$5747 v1049: Info: Started mldistwatch for lpath[/home/ftp/pub/PAUSE/authors/id/S/SP/SPAZM/Hadoop-Streaming-0.113160.tar.gz] with pid[11168] (paused:313)