Thursday, July 29, 2010

App::cpanoutofdate for keeping local::lib cpan up-to-date

A new version of App::cpanoutdated was released yesterday which ads a new -l and -L option to the bundled binary,cpan-outdated. These flags work the same as -l and -L in cpanminus, pointing to a local::lib controlled directory.

--compare-changes shows the difff to the changes file between releases. That's awesome.

This code will update all of the out of date modules in /apps/perl5 which is a local::lib controlled directory on my system: cpan-outdated -l /apps/perl5 | xargs cpanm -l /apps/perl5

--compare-changes example

cpan-outdated --compare-changes| head -20
S/SA/SARTAK/Any-Moose-0.13.tar.gz
0a1,23
> 0.13  Wed 19 May 2010
>   * Add load_first_existing_class (gfx)
> 
> 0.12  Fri 02 Apr 2010
>   * t/000-version.t for better diagnostics (tokuhirom)
>   * Slight performance improvements for is_class_loaded,
>       lazily loading Carp, etc (Sartak)
>   * Start some real documentation (Sartak)
>   * Document $ENV{ANY_MOOSE} (Sartak)
>     - fixes [rt.cpan.org #52339]
>   * Test that Moose is loaded, not CMOP (Sartak)
>     - fixes [rt.cpan.org #56093]
>   * Alias class_of and more functions (Sartak)
>     - requested by [rt.cpan.org #52275]

Usage:

cpan-outdated --help                
Usage:
        # print list of outdated modules
        % cpan-outdated

        # verbose
        % cpan-outdated --verbose

        # output changes diff
        % cpan-outdated --compare-changes

        # alternate mirrors
        % cpan-outdated --mirror file:///home/user/minicpan/

        # additional module path
        % cpan-outdated -I extlib/

        # install with cpan
        % cpan-outdated | xargs cpan -i

        # install with cpanm
        % cpan-outdated | xargs cpanm

Wednesday, July 28, 2010

July LA.pm meeting recap

Los Angeles Perl Mongers meeting for July 28th, 2010.

We had two speakers, Troy Will and Guy Shaw. This was Troy's first visit to LA.pm. Thank you both for presenting. Guy was a last minute fill to cover when Aran took ill.

Troy talked about his weight tracking project Getfit, written in perl. He also talked about GNU Stow, a perl system for packaging.

Guy is talking about pe-cpp, a partial c-pre-processor evaluator for making simplified header files. pe-cpp produces "sliced" cpp header files and does partial evaluation of arithmetic expressions. It does not simplifiy data structures. This is a source-to-source transformation for c header files to create simpler c header files that have been compressed by collapsing the known options. Yacc/Bison style!

After a thorough analysis of the available projects in this space, he found he needed to write his own to work around the limitations of the other. Originally written to assist C+asm integration and uses Sun's CTF for definitions of offsets and the like. "It's less scary than looking into h2ph," says Guy.

A good question: what happened to preprocessors? They seem to have fallen out of fashion, one camp views preprocessors as needed "only to cover up bad language design." Perhaps with Common Lisp and pure functional languages getting a resurgence of interest we will see more use of source-to-source transforms of code like this.

Convert::Binary::C may be useful to a similar audience, if one wants to use C data structure definitions within perl.

Tuesday, July 20, 2010

Hadoop::Streaming module updated and awesome

Verison 0.101881 of my Hadoop::Streaming module is released on CPAN.

This is a bug fix release -- fixing bugs in my test suite! I had problems with errors from the smoke testers, even though it all "worked for me." To fix this I wrote my tests in a sane manner, so they weren't held together by "gravity and good luck." See Bug #59164 for the gory details. 

I'm happy and proud to report that version 0.101881 has 81 PASS and 0 FAIL results from cpants!

Now all I need is a well written tutorial on how to use it.

links
Bug #59164 for Hadoop-Streaming: Test failures on a variety of platforms
https://rt.cpan.org/Public/Bug/Display.html?id=59164
Hadoop::Streaming Module
Documentation
http://search.cpan.org/perldoc?Hadoop::Streaming
Module Overview:
http://search.cpan.org/search?q=Hadoop::Streaming
Test Reports:
http://www.cpantesters.org/show/Hadoop-Streaming.html#Hadoop-Streaming-0.101881
Open Bugs (0!):
https://rt.cpan.org/Public/Dist/Display.html?Name=Hadoop-Streaming
Git Repo:
http://github.com/spazm/hadoop-streaming-frontend

morning reading: Getopt::Long::Descriptive (GLD) and App::Cmd

"Getopt::Long::Descriptive - Getopt::Long, but simpler and more powerful"
given a descriptive getopt argument, returns a usage summary and an object containing set options.
I am a fan of this less is more approach of "simpler AND more powerful."
Getopt::Long::Descriptive
"App::Cmd - write command line apps with less suffering"
Simplify writing command line apps by breaking functionality into module(s) with a minimal runner script. Makes extending command line apps easy by adding additional sub-modules.
Uses Getopt::Long::Descriptive.
App::Cmd

Have you ever asked "why do I have to document my command line options twice? once in the usage command and once in the Getopt statement?" I know I was asking just last week at the Thousand Oaks Perl Mongers meeting. If you don't ask, is it because you consider your Getopt command documentation? We discussed and compared various Getopt alternatives: Getopt::Long, Getopt::Euclid, Getop::Clade, etc.

One interesting solution is Getopt::Euclid. Euclid parses specially formatted pod commands to create the Getopt configuration. This is an awesome idea, but leaves me with two problems: 1) There is an extra step to create this parsing setup, and that means I can't test the exact code that my user will see. 2) it employs a source filter which freaks me out. Perhaps the source filter has been upgraded to use one of the PPI parsing modules instead, that would be less scary than the black magic happening in source filters.

GetOpt::Long::Descriptive (GLD) is built on Getopt::Long and takes a more verbose/descriptive input format. It outputs a usage command and an object containing options and values. Unlike Euclid, it doesn't read or create perldoc/POD.

I think adding a podweaver+Getopt::Long::Descriptive plugin would be interesting. Creating a nicely documented POD section during the dzil build step of my Dist::Zilla based modules would be a big win. I just browsed the code for Getopt::Long::Descriptive::Usage, and it looks like adding a weaver transform would be possible, if I could figure out how to extract the usage code from the module during the build step. This may be easier to visualize once I start investigating App::Cmd.

Example
From the SYNOPSIS section:

use Getopt::Long::Descriptive;

my ($opt, $usage) = describe_options(
    'my-program %o ',
    [ 'server|s=s', "the server to connect to"                  ],
    [ 'port|p=i',   "the port to connect to", { default => 79 } ],
    [],
    [ 'verbose|v',  "print extra stuff"            ],
    [ 'help',       "print usage message and exit" ],
);
print($usage->text), exit if $opt->help;
Client->connect( $opt->server, $opt->port );
print "Connected!\n" if $opt->verbose;

...and running my-program --help will produce:

  my-program [-psv] [long options...] 
    -s --server     the server to connect to
    -p --port       the port to connect to
                  
    -v --verbose    print extra stuff
    --help          print usage message and exit

I'm quite fond of how s|server became -s --server, showing both the long and short form of the arguments.

Go play with these modules and let me know if you start using them. Many thanks to their Authors for sharing their work on CPAN. You guys rock. I'm going to start using these modules and will report back when I have a better feel for them. This will likely become a future topic for the Los Angeles Perl Mongers (LAPM).

Links:

Getopt::Long::Descriptive
http://search.cpan.org/perldoc?Getopt::Long::Descriptive
App::Cmd
http://search.cpan.org/perldoc?App::Cmd
Getopt::Euclid
http://search.cpan.org/perldoc?Getopt::Euclid
Dist::Zilla
http://search.cpan.org/perldoc?Dist::Zilla
Pod::Weaver
http://search.cpan.org/perldoc?Pod::Weaver
PPI documentation
http://search.cpan.org/perldoc?PPI
Los Angeles Perl Mongers
la.pm.org
Thousand Oaks Perl Mongers
http://thousand-oaks-perl.org

Sunday, July 18, 2010

cpanminus with CPAN::mini?

At the July Thousand Oaks Perl Mongers [1] open-discussion meeting, we discussed both CPAN::Mini [2],[3] and App::cpanminus [4],[5]. They both work well individually, but can they be combined?

I did an example install of CPAN::Mini[2] for the meeting. This was a learning experience for me, as I'd only briefly heard of it before and this was my first install. CPAN::Mini makes a minimal private mirror of CPAN[6], of all the current release versions of all the CPAN modules! The idea is you keep this mirror up-to-date and anytime you need to install a module, you have it available. To keep the repository more manageable only the current version of each module is mirrored. Installation was super simple, the hardest being polling the room for a preferred CPAN mirror. We settled on ASK's develooper.com.
% cpan CPAN::Mini
% mkdir -p $HOME/mirrors/cpan
% minicpan -L $HOME/mirrors/cpan -R http://cpan.develooper.com/

Tommy then discussed his four favorite modules-of-the-moment for perl5 development and usage, one of which was App::cpanminus[4]. I'd heard of this previously, but hadn't gotten around to testing it. It installs a binary, cpanm[5], that is used to install modules. It tries to just do-the-right-thing, and install modules and keep the output to a minimum. I especially liked the flag to install to a local::lib maintained directory, which is useful in a sudo case where environment variables are annoying to pass.

I then tried to merge the two. First I installed App::cpanminus from my local CPAN::Mini mirror using cpan with the network unplugged. Success!

Then I tried to install a module with cpanm, and found that it was going to the network even though my repo was at a file:// url. Bummer!

I haven't gotten back to testing this, and a brief search found one comment from a user who reverts to cpan to use his CPAN::Mini mirror when he's away from the network. Does anyone have this working? Is it just part of the cpanminus magic that it looks online for the CPAN meta data?

My only reason to teach cpan.pm after that is to support CPAN::Mini and a local CPAN mirror. Thats the only reason I still use CPAN, I'm usually offline.
-- Pedro [7],[8]

Links

Thousand Oaks Perl Mongers
http://thousand-oaks-perl.org
CPAN::Mini
href=http://search.cpan.org/search?q=CPAN::Mini
cpanmini
href=http://search.cpan.org/search?q=minicpan
App::cpanminus
href=http://search.cpan.org/search?q=App::cpanminus
cpanm
href=http://search.cpan.org/search?q=cpanm
C.P.A.N. -- the Comprehensive Perl Archive Network.
http://cpan.org
Pedro
http://www.simplicidade.org/
Modern perl comment
http://www.modernperlbooks.com/mt/2010/04/should-novices-prefer-cpanminus.html#comment-362

Saturday, July 17, 2010

ironman badges, I miss you

I miss my smiling ironman badge. It is a beautiful thing.

I know it's been months, but I still miss it. Am I the only one? It was so good at reminding me to write. And not just self referential ironman related posts.

mmm skyscraper I love you.
mmm skyscraper I love you.
mmm skyscraper I love you.
mmm skyscraper I love you.

thirty thousand feet above the earth. it's a beautiful thing.
and you're a beautiful thing.
thirty thousand feet above the earth. it's a beautiful thing.
everybody's a beautiful thing.

---- Underworld, Skyscraper, I love you

Lot's of exciting happenings of late. I have two la.pm mongers write-ups that are in the todo list. Last week I hit two events. Wednesday was thousand-oaks.pm last week for an open rap session. Last Tuesday was a great talk at The Hammer museum at UCLA on the question of what is current internet use doing to our brains, which left me rejuvenated and ready to spend some quality time focusing rather than skimming back and forth.

Thursday, July 8, 2010

DavMail gateway provides LDAP and WebDav interface to MS Exchange

Goal: Get calendar and ldap-lookup working in thunderbird to my corporate hosted exchange service.

Status: WORKING!

Tools:

DavMail
DavMail provides a gateway from open protocols to MS Exchange
davmail.sf.net
Lightning
Thunderbird plugin for calendaring, built on sunbird
Current version requires Thunderbird 3.1.
http://www.mozilla.org/projects/calendar/lightning/
Thunderbird
Email client.
Calendar Client (with Lightening installed)
http://www.mozillamessaging.com/en-US/thunderbird/
Exchange Server
My office uses an exchange server hosted by AppRiver's Shoreline system.
http://www.appriver.com/exchange
Installation
  1. Install DavMail
    On my ubuntu system, I downloaded the deb from sourceforge and installed it manually with dpkg. I ran into a problem where I didn't have the proper swing libaries installed and the partially installed dpkg blocked the install of the swing libraries. In reality it was as simple as answering "Y" to "delete DavMail" and letting the swing install go through.
    1. #manually download deb from http://sourceforge.net/projects/davmail/files/
    2. sudo aptitude install sun-java-6 libswt-gtk-3.5-java
    3. sudo dpkg -i davmail_3.6.6-1032-1_all.deb

    The install integrated into the menu system, providing a menu entry to launch the app: Applications->Internet->DavMail.
  2. Upgrade/Install Thunderbird 3.1
    3.1 is currently required for lightning. I installed via the ubuntuzilla repository. Caveat: only 32 bit builds are available. I installed via the ubuntu repository, as per the Instructions.
    1. Create a new sources file /etc/apt/sources.d/ubuntuzilla.list containing:
      deb http://downloads.sourceforge.net/project/ubuntuzilla/mozilla/apt all main
    2. Then add the key, update and install:
      1. sudo apt-key adv --recv-keys --keyserver keyserver.ubuntu.com C1289A29
      2. sudo apt-get update
      3. sudo apt-get install thunderbird-mozilla-build
  3. Install Lightning into Thunderbird.
    This is most easily done directly through Thunderbird:
    1. Tools->Add-ons menu item.
    2. [Get Add-ons] button
    3. Type Lightning in the search box and press enter to search.
    4. push [Add to Thunderbird] button on the Lightning item.
    5. confirm install
    6. Restart Thunderbird
Configuration The devil is in the details. Getting the correct configuration from our hosted exchange server took quite a bit of trial and error. The email system predates the current company, so I have at least three logins:
My primary email address
username @ rubiconproject.com
This is the address used to send and receive mail.
Same account at a different domain (the founder's vanity domain).
username @ addante.com
This is the address used to authenticate with the pop/imap servers and to access Outlook WebMail.
my username with internally assigned id number attached. No domain.
username_12345
Used internally. Will be used in our ldap setup.
  1. Launch and Configure DavMail
    1. Launch from menus Applications->Internet->DavMail, which launches to a gnome icone in by the clock.
    2. Open settings right-click DavMail icon->Settings...
    3. Set the OWA (Exchange) URL : https://exg3.exghost.com/Exchange
    4. Make note of the ports for CalDav (1080) and LDAP (1389), change if desired.
  2. Configure calendar in Thunderbird-lightning
    1. File->New->Calendar...
    2. choose [*] On the Network and [Next]
      1. choose [*] CalDav
      2. Location: http://localhost:1080/users/username@addante.com/calendar
      3. [Next]
    3. Configure the name, color and alarm settings as desired. Email should by your work email, aka username@rubiconproject. [Next]
    4. Provide login credentials, the same as if you were using the webmail tool, for for me that is the username@addante.com account.
  3. Configure LDAP
    1. Thunderbird Edit->Preferences->Composition->Addressing
    2. [Edit Directories]
    3. [Add]
    4. Configuration settings. Base DN and Bind DN were tricky.
      • Name: proxy ldap
      • Hostname: localhost
      • BaseDN: ou=people
      • Port: 1389
      • Bind DN: username_12345
      • [OK]
    5. Test the connection:
      1. select proxy ldap and press Enter
      2. [Offline] tab
      3. [Download Now].
      4. enter username@addante.com as username and correct password in the authentication box.
      5. if the download completes and "Replication Succeeded" appears, all is working.

And there you have it, working exchange integration through open standards. Yay! It'll only take 15 minutes or so total to install now (after my hour putzing it around and 3 hours writing it up.)

I now have ldap lookup and caldav alerts functioning. I haven't delved that far into the system, but it's a big step up from my previous interaction lightning. I'll see what happens when I get my next meeting invite and attempt to reply via thunderbird.

Tuesday, July 6, 2010

Do you have CPAN::Reporter installed? Go do it!

Do you have CPAN::Reporter installed? Did you maybe not know what it is, what it does, and why it's the easiest way to help the perl+cpan community? CPAN::Reporter sends back test results every time you download a module from CPAN. This gives module authors something tangible to know that people are using their modules and to automatically report errors.
Now anyone with an up-to-date CPAN.pm can contribute to CPAN Testers -- no smoke server needed.
...
Becoming a CPAN Tester with CPAN::Reporter is easy. From the CPAN shell prompt:

cpan> install CPAN::Reporter
cpan> reload cpan
cpan> o conf init test_report
cpan> o conf commit

  --  http://use.perl.org/articles/06/11/08/1256207.shtml

My Hadoop::Streaming module has all sorts of crazy errors on esoteric platforms. I only know this because of automated smoke tests run by CPANTesters. It would be nice to have some reports from actual users mingled in with the automated runs.

In the latest release I pushed out over Independence Day weekend, I added monitoring on STDERR from my sub-command tests. Now I have some test results that capture error messages from external apps I try to run from my tests. At least now I have a chance of figuring them out... not sure I understand these error messages yet. Maybe they'll make more sense tomorrow. I'd like to make it so all my users see this output from cpan:


All tests successful.
Files=5, Tests=12, 2 wallclock secs ( 0.05 usr 0.00 sys + 2.04 cusr 0.20 csys = 2.29 CPU)
Result: PASS
(/usr/bin/make test exited with 0)
CPAN::Reporter: Test result is 'pass', All tests successful.
CPAN::Reporter: preparing a CPAN Testers report for Hadoop-Streaming-0.100270
CPAN::Reporter: sending test report with 'pass' to cpan-testers@perl.org