cpanm and perlbrew should not run tests by default.

This may sound heretical. Perl has always had a strong testing culture, and end-user testing may have once played a valuable role in testing a distribution under many systems. But we now have a CPAN Testers network which will run tests on countless systems and Perl versions, and report failures back to the author promptly and automatically. Distributions can be sent through the Testers’ gauntlet before ever being officially released. In this environment, it’s hard to see much additional value in ad hoc end-user testing.

As Dave Rolsky points out, we install most other software without running tests and rarely give it a thought.

None of this would matter if end-user testing was free. But it is not.

The costs of end-user testing

Slower installs. On my system, a fresh install of Moose and its dependencies takes three times longer with tests (2 minutes versus 41 seconds). A fresh install of Catalyst and its dependencies takes nearly four times longer with tests (9.5 minutes versus 2.5 minutes).

How many new Perl users find CPAN installs much slower than they need to be? How many would choose a 3-4 times speedup if they knew it was an option? It’s like having a turbo button and leaving it unpressed by default.

False positives. The more tests CPAN authors write, the more likely an occasional false-positive failure sneaks through (as in “Failed 1/1746 tests.”) In most such cases the module will still work for the user’s purposes. But the default behavior is to prevent the module from being installed at all. If your module depends on other modules, then any failure up the dependency chain likewise prevents your module from being installed, even if the failure has no bearing on your module’s efficacy.

How many new Perl users have unnecessarily failed to install a module like Moose or Catalyst because of an obscure, temporary failure deep in the dependency chain?

Fear of dependencies and code reuse. Slower installs and false positives are the main reasons why people complain about distributions having “too many dependencies”. (If the dependencies installed quickly and reliably, as they do with –notest and apt-get and yum, would anyone complain or even notice?) These complaints in turn encourage module authors to reduce or eliminate their dependencies, thus reinventing where they could be reusing.

It’s the wrong default for new users

Some veteran Perl folks may like tests to run on every install. That’s fine. But I suspect new Perl users just want things to install quickly and reliably, and in any event don’t have the experience to evaluate or take action on a test failure (especially one in an obscure dependency). For these users running tests is simply the wrong default.

I turn on –notest on each system I administer and preach it enthusiastically to every new Perl user I encounter. But I wish I didn’t have to mention it at all.

I hardly ever choose Tie interfaces — too much magic — but occasionally they do produce pretty code. In this case, we have a watchdog script that sends USR2 signals to httpd processes that grow too large (so that they’ll log their call stack). Sometimes these processes stick around for a while, so I only want to send a certain number of signals per process.

    my %kill_count;
    ...
    if ( $vsize > $max_vsize ) {
        if ( $kill_count{$pid} < $max_kills ) {
            kill( 'USR2', $pid );
            $kill_count{$pid}++;
            $log->warn(sprintf( "pid %d vsize %dmb > %dmb, sending USR2",
                $pid, $vsize, $max_vsize ));
        }
    }

Then it occurred to me that the watchdog restarts frequently, so I need to keep the kill counts persistent; and I should only limit on an hourly basis, because the same pid will eventually come around again. We already have a custom CHI subclass that we use for caching all over our application, so it was easy to plug it in:

    use Tie::CHI;
    my $cache = HM::Cache->new
       (namespace => 'watchdog/kill_count', expires_in => '1 hour');
    my %kill_count;
    tie %kill_count, 'Tie::CHI', $cache;

And voila, %kill_count is persistent, and its values decay after an hour.

   my $cache = CHI->new( driver => 'Memory', global => 1 );
   my $lst = ['foo'];
   $cache->set('key' => $lst);   # serializes $lst before storing
   $cache->get('key');   # returns ['foo']
   $lst->[0] = 'bar';
   $cache->get('key');   # returns ['foo']

   my $cache = CHI->new( driver => 'RawMemory', global => 1 );
   my $lst = ['foo'];
   $cache->set('key' => $lst);   # stores $lst directly
   $cache->get('key');   # returns ['foo']
   $lst->[0] = 'bar';
   $cache->get('key');   # returns ['bar']!

It should work well as a short-lived L1 cache in front of memcached, for example.

I was motivated to create this by Yuval Kogman’s Cache::Ref, which is still a little faster (not having some of the overhead of CHI’s metadata and features). See the CHI benchmarks, also new with this release.

Let’s say we want to support URLs like

    /archives/2010/05

where the second and third part of the path are year and month parameters respectively.

Using the new plugin:

    %% route "{year:[0-9]{4}}/{month:[0-9]{2}}";

    Archives for the month of <% $.month %>/<% $.year %>:
    ...

    <%init>
    # Use $.month and $.year to fetch the archives
    </%init>

The route command adds a route indicating how the path_info (the remainder of the path beyond /archives) should be parsed.

Any named captured arguments are placed in component attributes, which are declared automatically if you don’t do so manually. So above, we get auto-declared attributes $.year and $.month that are then set to ‘2010′ and ‘05′ for that URL.

You can add multiple routes and they’ll each be tried in turn. If the URL does not match any route, the component will decline, which generally results in a 404.

Other examples of Router::Simple routes:

    route "wiki/:page";     # sets $.page
    route "download/*.*";   # sets $.splat to a 2-element arrayref

I chose Router::Simple because I liked the compact declaration syntax, but one could create similar plugins (with some shared common code) to support other routers like Path::Dispatcher and Path::Router.

For those not familiar with it, Mason is a templating framework for generating web pages and other dynamic content. Mason 2 has been rearchitected and reimplemented from the ground up, to take advantage of modern Perl techniques (Moose, Plack/PSGI) and to correct long-standing feature and syntax inadequacies. Its new foundations should allow its performance and flexibility to far exceed Mason 1.

Though little original code or documentation remains, Mason’s core philosophy is intact; it should still “feel like Mason” to existing users.

I’ve talked about plans for Mason 2 here before, but as things have changed in the past year and a half, here’s an updated summary:

• Name. The name is now Mason, instead of HTML::Mason.

• Component classes. Each component is represented by its own (Moose) class, rather than just an instance of a common class. This means that components have their own namespaces, subroutines, methods, and attributes, and can truly inherit from one other. See Mason::Manual::Components.

• Filters. A single powerful filter syntax and mechanism consolidates three separate filter mechanisms from Mason 1 (filter blocks, components with content, and escape flags). See Mason::Manual::Filters.

• Plugins. Moose roles are utilized to create a flexible plugin system that can modify nearly every aspect of Mason’s operation. Previously core features such as caching can now be implemented in plugins. See Mason::Manual::Plugins.

• Web integration. Mason 1’s bulky custom web handling code (ApacheHandler, CGIHandler) has been replaced with a simple PSGI handler and with plugins for web frameworks like Catalyst and Dancer. The core Mason distribution is now completely web-agnostic. See Mason::Plugin::PSGIHandler.

• File naming. Mason now facilitates and enforces (in a customizable way) standard file extensions for components: .m (top-level components), .mi (internal components), and .pm (pure-perl components).

See Mason::Manual::UpgradingFromMason1 for a more detailed list of changes.

Mason 2 is obviously still in alpha status, but it has a fair sized test suite and I’m eager to start building web projects with it. I hope you’ll give it a try too! Post feedback here or on the Mason user’s list.

As just one example, I initially added a cache feature to Mason 2, just because it was there in Mason 1 and seemed generally useful. But now I realize it can be cleanly separated into its own plugin.

There’s no real performance or footprint advantage for those that omit the plugin. What it does do is put all the cache related code, documentation, and tests in a single place. As small as it is, the cache feature touched many of the main Mason modules in some way; now it is all in its own directory. It also makes me more amenable to expanding the feature (e.g. with new parameters or methods), because I’m not worried about disproportionately “polluting” the main code and API.

Of course, deciding what is a plugin versus a built-in feature (plus that third category, a plugin-in-name that really everyone expects to install) is the age-old question. Too many plugins will of course hurt performance, because each one adds a layer of method modifiers, and will also hurt useability and readability. I’m still quite confused about where and how Dist::Zilla performs many of its tasks, and part of that is the sheer number of plugins one needs for a typical installation. It’s a virtual certainty that in my new enthusiasm I’ll over-plugin somewhere.

In Mason 2 we aim to unify these with a single implementation and a more consistent syntax.

A set of standard filters are automatically available in components, and other filter packages can be loaded via plugins.

Invoking filters (block syntax)

Here’s the typical way of invoking a filter:

<% $.Trim { %>
This string will be trimmed
</%>

Things to note here:

• An { at the end of a <% %> tag denotes a filter call.
• The </%> tag marks the end of the filtered content.
• The expression $.Trim, aka $self->Trim, is a method call on the component object which returns a filter. In general everything before the brace is evaluated and is expected to return a filter or list of filters.
• Filter names use CamelCase to distinguish themselves from other methods in the Mason::Component namespace. Another option was to use a standard prefix or suffix, e.g. trim_filter or filter_trim, but I thought this was the least ugly option and made filters look a little more like built-in tags.

Filters can take arguments:

<% $.Repeat(3) { %>
  There's no place like home.
</%>

  ==>  There's no place like home.
 There's no place like home.
 There's no place like home.

Again, the expression $.Repeat(3) returns a filter, meaning that it can
be curried:

% my $repeat_three = $.Repeat(3);
<% $repeat_three { %>
  There's no place like home.
</%>

A simple filter is just a subroutine that takes text as input and return the
new text. Thus you can create one-off filters with anonymous subroutines:

<% sub { reverse($_[0]) } { %>Hello</%>

     ==> olleH

Filters can be nested, with separate tags:

<% $.Trim { %>
  <% sub { uc($_[0]) } { %>
    This string will be trimmed and uppercased
  </%>
</%>

or within a single tag:

<% $.Trim, sub { uc($_[0]) } { %>
  This will be trimmed and uppercased
</%>

Multiple filters within the same tag are applied, intuitively, in reverse order with the last one being innermost. e.g. in this block

% my $i = 1;
<% $.Repeat(3), $.Cache($key, '1 hour') { %> <% $i++ %> </%>

  => 1 1 1

the output of <% $i++ %> is cached, and then repeated three times, whereas in this block

% my $i = 1;
<% $.Cache($key, '1 hour'), $.Repeat(3) { %> <% $i++ %> </%>

  => 1 2 3

<% $i++ %> is executed and output three times, and then the whole thing cached.

Invoking filters (pipe syntax)

Filters can also appear in a limited way inside a regular <% %> tag; this replaces Mason 1’s escape flags.

<% $content | NoBlankLines,Trim %>

The filter list contains one or more comma-separated names, which are treated as methods on the current component class. With this syntax you cannot use anonymous subroutines or variables as filters, or pass arguments to filters (but you can define local filter methods to get around this).

One common use of this form is to escape HTML strings:

<% $message_body | H %>

The H filter is provided along with other web-related filters in Mason::Plugin::HTMLFilters.

Applying filter to current component

Mason 1’s <%filter> section was good for filtering the content of the current component. e.g. to make it uppercase:

<%filter>
$_ = uc($_);
</%filter>

In Mason 2 the most succinct replacement is currently an around modifier:

<%around main>
<% sub { uc($_[0]) } { %>
% $self->$orig();
</%>
</%around>

Creating filters

Here’s a filter package that implements two filters, Upper and Lower:

package MyApp::Filters;
use Method::Signatures::Simple;
use Moose::Role;

method Upper () {
    return sub { uc($_[0]) }
}

method Lower () {
    return sub { lc($_[0]) }
}

1;

To use these in a component:

<%class>
with 'MyApp::Filters';
</%class>

<% $.Upper { %>
...
</%>

Or if you want them available to all components, put them in Base.pm at the top of your component hierarchy, or in your application’s Mason::Component subclass.

Simple vs. dynamic filters

A simple filter is a code ref which takes a string and returns the output, like Upper and Lower above.

A dynamic filter (a Mason::DynamicFilter object) contains a code ref which takes a yield block and returns the output. A yield block is a zero-argument code ref that returns a content string.

The power of dynamic filters is that they can choose if and when to execute the yield block. For example, here’s the standard Cache filter in action:

<% $.Cache($key, '1 hour') { %>
... some computed content ...
</%>

This caches the inner content for one hour based on cache key $key. Here’s the implementation:

method Cache ( $key, $set_options ) {
    Mason::DynamicFilter->new(
        filter => sub {
            my $yield = shift;
            $self->cmeta->cache->compute( $key, $yield, $set_options );
        }
    );
}

Using CHI’s compute method, we check the cache first and return the output immediately if it is available. Only on a cache miss do we actually execute the (presumably expensive) yield block. This could not be implemented with a simple filter since the content would be computed every time.

Perl::Tidy::perltidy(
    ...,
    prefilter => sub {
        $_ = $_[0];
        s/^method (.*)/sub $1 \#__METHOD/gm;
        return $_;
    },
    postfilter => sub {
        $_ = $_[0];
        s/^sub (.*?)\s* \#__METHOD/method $1/gm;
        return $_;
    }
);

• The prefilter code substitutes method for sub, and adds a comment
  so we’ll be able to find and convert these back afterwards.
• When perltidy operates on the code, it will just see regular subs and treat them accordingly.
• The postfilter code converts the sub back to method wherever the special comment appears.

I put this in my private Perl::Tidy subclass, along with other tweaks, such as telling perltidy to leave my Moose ‘has’ lines alone (I prefer them to always be on one line).

Then I create my own perltidy script which uses this subclass:

#!/usr/local/bin/perl
package main;
use JS::Perl::Tidy;
JS::Perl::Tidy::perltidy();

A little messy, but it works. I’m a devoted perltidy user, with enforced perltidy-on-commit policy on every project, so I’m glad I don’t have to choose between perltidy and my favorite Perl tweaks.

Anyway.

In Mason 1, content wrapping (the practice of wrapping HTML in one or more surrounding templates) was achieved via autohandlers. Each autohandler would print its content and then call $m->call_next, which would pass control to the next component down the subclass chain. This was all special-purpose code written against Mason’s already hacky inheritance scheme.

But in Mason 2, Moose gives us a natural solution for this pattern: inner and augment.

At the start of a Mason 2 request, after determining the top-level component (aka “page component”), Mason will call that component’s render() method. By default, render() will just call main(), which outputs the main body of the component. However, each ancestor of the page component has an opportunity to wrap render() with its own template. For example:

# /Base.m - top-most template
#
<%augment render>
  <html><body>
      <% inner() %>
      <div class="footer">Copyright 2010 McHuffy Inc.</div>
    </body></html>
</%augment>

# /product/Base.m - product-specific template
#
<%augment render>
  <h2>Products</h2>
  <& product_nav.mi &>
  <div class="products_body">
    <% inner() %>
  </div>
</%augment>

# /product/sales/Base.pm - utility methods, no additional template
#
<%method some_utility_method>
   ...
</%method>

# /product/sales/display.m:
#
<h3><% $product->title %></h3>
...

The page /product/sales/display will be wrapped by two templates: the outer one
in /Base.pm, and the inner one in /product/Base.pm.

A couple of new things here for Mason 1 users:

• ‘autohandler’ has changed to ‘Base.m’. A component of this name becomes the automatic superclass of every component in its directory and subdirectories.
• There are sections for declaring Moose methods and method modifiers with embedded-Perl syntax: <%after>, <%around>, <%augment>, <%before>, and <%method>. (Why we “need” all these explicit sections is a subject for another post.)

Now here’s an implementation snag. Notice that neither /product/sales/display nor /product/sales/Base.pm have a render method. For the former, we want to automatically create a render that just calls main, which will display the main body of the component. For the latter, we want render to just fall through to the next level. But there’s no way to know in advance whether a component will be used as the page component or as one of its superclasses, so we have to create a default render for every component that does the right thing at runtime. In pseudo-code:

if (we are a superclass of the page component) {
    # Go to the next level.
    inner();
} else {
    # We are the page component. Draw the main content.
    $self->main();
}

Unfortunately there’s no official way to ask Moose whether we’re at the bottom of the inner() chain. So I’ve got a couple of choices here:

1. Check ref($self):

if ( ref($self) ne __PACKAGE__ ) {
    inner();
} else {
    $self->main();
}

As long as we provide a default render for every component, I think this should correctly distinguish the bottom of the chain. But I’m not entirely sure – Moose does a lot of with auto-generated subclasses.

2. Steal the code out of Moose::inner:

if ( $Moose::INNER_BODY ) {
    inner();
} else {
    $self->main();
}

Of course this is awful, as it relies on implementation details that are subject to change.

For now I’ll be going with #1.

In Mason 2, plugins are based on Moose roles, allowing you to modify behavior in each of Mason’s main subclasses — Interp, Request, Compiler, Compilation, and Component — in a composable way.

Here’s my first Mason 2 plugin, to perltidy object files after components are compiled – useful in development. First, a nominal base class:

package Mason::Plugin::TidyObjectFiles;
use Moose;
extends 'Mason::Plugin';

1;

(This has to exist for Mason to recognize the plugin, but there isn’t much to do there right now.)

Then, a role for each of Mason’s subclasses that we want to modify. In this case we only need to modify Mason::Compiler:

package Mason::Plugin::TidyObjectFiles::Compiler;
use Moose::Role;
use Perl::Tidy;
use strict;
use warnings;

has 'tidy_options' => ( is => 'ro' );

around 'write_object_file' => sub {
    my ( $orig, $self, $object_file, $object_contents ) = @_;

    my $argv = $self->tidy_options || '';
    my $source = $object_contents;
    Perl::Tidy::perltidy(
        'perltidyrc' => '/dev/null',
        source       => \$source,
        destination  => \$object_contents,
        argv         => $argv
    );
    $self->$orig( $object_file, $object_contents );
};

1;

To use this, I simply list ‘TidyObjectFiles’ in my plugins:

my $mason = Mason->new(..., plugins=>[ 'TidyObjectFiles' ]);

or, if I want to specify perltidy options:

my $mason = Mason->new(..., plugins=>[ 'TidyObjectFiles' ], tidy_options => '-noll -l=100');

The Mason::Plugin:: prefix is automatically added to plugin names. You can also specify a full path with a leading ‘+’, e.g.

plugins => ['+MyApp::Plugin::MyPlugin']

Now my object files go from looking like this:

no warnings 'redefine';
sub _comp_info { return {comp_dir_path => '/',comp_is_external => 1,comp_path => '/hi.m'} }
sub main {
my $self = shift;
my $m = $self->m;

my $_buffer = $m->current_buffer;
 #line 1 "/Users/swartz/git/mason.git/tmp/comps/hi.m"
$$_buffer .= 'Hi there! The time is ';
 #line 1 "/Users/swartz/git/mason.git/tmp/comps/hi.m"
{ $$_buffer .=  scalar(localtime)  if defined( scalar(localtime) ) }
 #line 1 "/Users/swartz/git/mason.git/tmp/comps/hi.m"
$$_buffer .= '.
';

;return;
}

to this:

no warnings 'redefine';

sub _comp_info {
    return {
        comp_dir_path    => '/',
        comp_is_external => 1,
        comp_path        => '/hi.m'
    };
}

sub main {
    my $self = shift;
    my $m    = $self->m;

    my $_buffer = $m->current_buffer;

    #line 1 "/Users/swartz/git/mason.git/tmp/comps/hi.m"
    $$_buffer .= 'Hi there! The time is ';

    #line 1 "/Users/swartz/git/mason.git/tmp/comps/hi.m"
    { $$_buffer .= scalar(localtime) if defined( scalar(localtime) ) }

    #line 1 "/Users/swartz/git/mason.git/tmp/comps/hi.m"
    $$_buffer .= '.
';

    return;
}