Dec 19

I wrote previously about the inability of perltidy to handle the method keyword of Method::Signatures::Simple. Now, Steve Hancock has graciously accepted my patches in the latest Perl::Tidy, including prefilter and postfilter options. This allows me to easily add method support:

Perl::Tidy::perltidy(
    ...,
    prefilter => sub {
        $_ = $_[0];
        s/^method (.*)/sub $1 \#__METHOD/gm;
        return $_;
    },
    postfilter => sub {
        $_ = $_[0];
        s/^sub (.*?)\s* \#__METHOD/method $1/gm;
        return $_;
    }
);
  • The prefilter code substitutes method for sub, and adds a comment
    so we’ll be able to find and convert these back afterwards.
  • When perltidy operates on the code, it will just see regular subs and treat them accordingly.
  • The postfilter code converts the sub back to method wherever the special comment appears.

I put this in my private Perl::Tidy subclass, along with other tweaks, such as telling perltidy to leave my Moose ‘has’ lines alone (I prefer them to always be on one line).

Then I create my own perltidy script which uses this subclass:

#!/usr/local/bin/perl
package main;
use JS::Perl::Tidy;
JS::Perl::Tidy::perltidy();

A little messy, but it works. I’m a devoted perltidy user, with enforced perltidy-on-commit policy on every project, so I’m glad I don’t have to choose between perltidy and my favorite Perl tweaks.

Dec 15

(These posts are starting to follow a familiar pattern: “In Mason 1 we did X, but in Mason 2, Moose makes it so much easier…)”

Anyway.

In Mason 1, content wrapping (the practice of wrapping HTML in one or more surrounding templates) was achieved via autohandlers. Each autohandler would print its content and then call $m->call_next, which would pass control to the next component down the subclass chain. This was all special-purpose code written against Mason’s already hacky inheritance scheme.

But in Mason 2, Moose gives us a natural solution for this pattern: inner and augment.

At the start of a Mason 2 request, after determining the top-level component (aka “page component”), Mason will call that component’s render() method. By default, render() will just call main(), which outputs the main body of the component. However, each ancestor of the page component has an opportunity to wrap render() with its own template. For example:

# /Base.m - top-most template
#
<%augment render>
  <html><body>
      <% inner() %>
      <div class="footer">Copyright 2010 McHuffy Inc.</div>
    </body></html>
</%augment>

# /product/Base.m - product-specific template
#
<%augment render>
  <h2>Products</h2>
  <& product_nav.mi &>
  <div class="products_body">
    <% inner() %>
  </div>
</%augment>

# /product/sales/Base.pm - utility methods, no additional template
#
<%method some_utility_method>
   ...
</%method>

# /product/sales/display.m:
#
<h3><% $product->title %></h3>
...

The page /product/sales/display will be wrapped by two templates: the outer one
in /Base.pm, and the inner one in /product/Base.pm.

A couple of new things here for Mason 1 users:

  • ‘autohandler’ has changed to ‘Base.m’. A component of this name becomes the automatic superclass of every component in its directory and subdirectories.
  • There are sections for declaring Moose methods and method modifiers with embedded-Perl syntax: <%after>, <%around>, <%augment>, <%before>, and <%method>. (Why we “need” all these explicit sections is a subject for another post.)

Now here’s an implementation snag. Notice that neither /product/sales/display nor /product/sales/Base.pm have a render method. For the former, we want to automatically create a render that just calls main, which will display the main body of the component. For the latter, we want render to just fall through to the next level. But there’s no way to know in advance whether a component will be used as the page component or as one of its superclasses, so we have to create a default render for every component that does the right thing at runtime. In pseudo-code:

if (we are a superclass of the page component) {
    # Go to the next level.
    inner();
} else {
    # We are the page component. Draw the main content.
    $self->main();
}

Unfortunately there’s no official way to ask Moose whether we’re at the bottom of the inner() chain. So I’ve got a couple of choices here:

1. Check ref($self):

if ( ref($self) ne __PACKAGE__ ) {
    inner();
} else {
    $self->main();
}

As long as we provide a default render for every component, I think this should correctly distinguish the bottom of the chain. But I’m not entirely sure – Moose does a lot of with auto-generated subclasses.

2. Steal the code out of Moose::inner:

if ( $Moose::INNER_BODY ) {
    inner();
} else {
    $self->main();
}

Of course this is awful, as it relies on implementation details that are subject to change.

For now I’ll be going with #1.

Dec 14

In Mason 1, extending Mason’s behavior means either using the limited plugin API (allowing you to take action before and after a request or component call), or else writing subclasses, which is problematic when you try to use more than one together.

In Mason 2, plugins are based on Moose roles, allowing you to modify behavior in each of Mason’s main subclasses — Interp, Request, Compiler, Compilation, and Component — in a composable way.

Here’s my first Mason 2 plugin, to perltidy object files after components are compiled – useful in development. First, a nominal base class:

package Mason::Plugin::TidyObjectFiles;
use Moose;
extends 'Mason::Plugin';

1;

(This has to exist for Mason to recognize the plugin, but there isn’t much to do there right now.)

Then, a role for each of Mason’s subclasses that we want to modify. In this case we only need to modify Mason::Compiler:

package Mason::Plugin::TidyObjectFiles::Compiler;
use Moose::Role;
use Perl::Tidy;
use strict;
use warnings;

has 'tidy_options' => ( is => 'ro' );

around 'write_object_file' => sub {
    my ( $orig, $self, $object_file, $object_contents ) = @_;

    my $argv = $self->tidy_options || '';
    my $source = $object_contents;
    Perl::Tidy::perltidy(
        'perltidyrc' => '/dev/null',
        source       => \$source,
        destination  => \$object_contents,
        argv         => $argv
    );
    $self->$orig( $object_file, $object_contents );
};

1;

To use this, I simply list ‘TidyObjectFiles’ in my plugins:

my $mason = Mason->new(..., plugins=>[ 'TidyObjectFiles' ]);

or, if I want to specify perltidy options:

my $mason = Mason->new(..., plugins=>[ 'TidyObjectFiles' ], tidy_options => '-noll -l=100');

The Mason::Plugin:: prefix is automatically added to plugin names. You can also specify a full path with a leading ‘+’, e.g.

plugins => ['+MyApp::Plugin::MyPlugin']

Now my object files go from looking like this:

no warnings 'redefine';
sub _comp_info { return {comp_dir_path => '/',comp_is_external => 1,comp_path => '/hi.m'} }
sub main {
my $self = shift;
my $m = $self->m;

my $_buffer = $m->current_buffer;
 #line 1 "/Users/swartz/git/mason.git/tmp/comps/hi.m"
$$_buffer .= 'Hi there! The time is ';
 #line 1 "/Users/swartz/git/mason.git/tmp/comps/hi.m"
{ $$_buffer .=  scalar(localtime)  if defined( scalar(localtime) ) }
 #line 1 "/Users/swartz/git/mason.git/tmp/comps/hi.m"
$$_buffer .= '.
';

;return;
}

to this:

no warnings 'redefine';

sub _comp_info {
    return {
        comp_dir_path    => '/',
        comp_is_external => 1,
        comp_path        => '/hi.m'
    };
}

sub main {
    my $self = shift;
    my $m    = $self->m;

    my $_buffer = $m->current_buffer;

    #line 1 "/Users/swartz/git/mason.git/tmp/comps/hi.m"
    $$_buffer .= 'Hi there! The time is ';

    #line 1 "/Users/swartz/git/mason.git/tmp/comps/hi.m"
    { $$_buffer .= scalar(localtime) if defined( scalar(localtime) ) }

    #line 1 "/Users/swartz/git/mason.git/tmp/comps/hi.m"
    $$_buffer .= '.
';

    return;
}
Dec 12

In Mason 1 you can declare component arguments and then refer to them as normal lexically scoped variables:

<%args>
$address
$city
$name
</%args>

Dear <% $name %>: We will come to your house at <% $address %> in the
fair city of <% $city> to deliver your prize!

In Mason 2, components are classes rather than subroutines, so we have attributes rather than arguments. When you call a component, you create an instance of the component class and pass attributes to the constructor.

Attributes have several advantages over arguments: you can refer to them in any of the component’s methods (unlike in Mason 1, where arguments have to be awkwardly passed to each method) and you can take advantage of Moose’s powerful attribute declarations.

Here’s how this looks without any special syntax:

<%class>
has 'address' => (is => 'ro');
has 'city'    => (is => 'ro');
has 'name'    => (is => 'ro');
</%class>

Dear <% $self->name %>: We will come to your house at <% $self->address %> in the fair city of <% $self->city> to deliver your prize!

Clearly, we’ll need to pour some sugar on this.

Declaring attributes

Mason 2 will provide an <%attr> section, analagous to Mason 1’s <%args>:

<%attr>
a
b => 'foo'
c => (isa => 'Str', default => 'something')
</%attr>

This illustrates the three kinds of <%attr> declarations:

  • A name by itself: required attribute.
  • A name followed by => and a non-paren: optional attribute with a default.
  • A name followed by => a paren: specifies Moose attribute options.

You can mix and match these attributes with standard has declarations. The above is equivalent to:

<%class>
has 'a' => (required => 1);
has 'b' => (default => 'foo');
has 'c' => (isa => 'Str', default => 'something');
</%class>

All attributes will be read-write by default, via MooseX::HasDefaults::RW. Although read-write attributes are sometimes discouraged, I believe they’re appropriate and convenient for these short-lived component objects that are not generally expected to be accessed outside of their class.

Accessing attributes

To make the frequent access of attributes more convenient, we resort to some limited source filtering evil and implement Perl 6ish $. notation. Specifically, all occurrances of

$.foo

within Perl sections will be replaced with

$self->foo

where foo is any valid attribute name ([A-Za-z_]\w*).

Note that with read-write lvalue attributes, this could even be used for assignment:

$.foo = 5;

Very convenient for attributes which exist chiefly for sharing data between methods.

The $. substitution feature will be made easy to disable, for those who are against it.

Putting it all together

The first example above with attribute declaration and access sugar:

<%attr>
address
city
name
</%attr>

Dear <% $.name %>: We will come to your house at <% $.address %> in the
fair city of <% $.city> to deliver your prize!
Nov 26

I’ve been meaning to try Method::Signatures::Simple ever since the last YAPC and just finally found the time. Something about typing “my ($self) = @_” for the millionth time pushing me over the edge.

Converted a random module in Mason 2. Reduced the number of lines by 65 (10%) and I feel like I’m developing in a modern language again. I’m hooked! Hats off to Rhesa Rozendaal and those responsible for the underlying Devel::Declare.

Now I’m eager to convert Mason 2 and possibly CHI to this across the board. The question is, is it reasonable for me to use this in published CPAN modules? It seems like very few CPAN modules do, and given how clearly awesome it is :) , I’m wondering if there is a reason, other than not wanting the extra dependency or not wanting to depend on “black magic”.

One snag is that perltidy (which I use rigorously on commit) doesn’t yet recognize the ‘method’ keyword. I managed to fix by patching Perl::Tidy to allow arbitrary transformations before and after tidying – so I can replace ‘method’ with ’sub’ beforehand and convert back afterwards. Will attempt to contribute this back.

Oct 12

Server::Control allows you to control servers ala apachectl, but with better diagnostics and many more features. It includes both a drop-in replacement for apachectl (”apachectlp”) and an OO interface.

Though it was designed with Apache in mind, there are also subclasses for HTTP::Server::Simple and Net::Server, and it’s general enough to use with any server that has a pid file and listens on a port.

Features include:

  • Checks server status via both pid file and port connect
  • Detects and handles corrupt or out-of-date pid files
  • Reports what is using a port when it is busy (via lsof)
  • Tails the error log when server fails to start
  • Supports sudo usage for restricted (< 1024) port
  • Easy to customize for your environment with start/stop hooks, etc.

Coming soon: plugins for common tasks associated with starting and stopping servers, such as generating conf files from templates and auto-restarting a server on file change.

Sep 22

One of the traditional problems with mod_perl development is getting the server to recognize module changes, without restarting after every change. I’m thinking about an improved solution for this.

These are the solutions I know about:

1. Apache2::Reload / Module::Reload

These check whether modules have changed on each request, and if so, clear their symbols and reload them inside the process.

Problem: some modules fail to reload properly. Sometimes the failure is intermittent, depending on the order of module loading and other esoteric details. Moose and ORM modules seem particularly prone to reload failures. For me, this level of unpredictability makes *::Reload too frustrating to use.

2. Catalyst Auto-restart

Catalyst has an engine (Catalyst::Engine::HTTP::Prefork::Restarter) which forks off a “watcher” process that waits for your modules to change. When they change, it restarts the server. The usual effect is that, between the time you hit “save” in your editor and reload your page, the server has restarted or at least begun restarting.

Problems: Doesn’t work well if you make a few changes in a row; the restart only captures your first change. (EDIT: Dave Rolsky points out that this is fixed now under some OS’s.) Bad user experience if there’s an error in your module; you have to realize the server has died, find the error message in some shell or log, and manually start up the server again. Finally, you are still ultimately waiting for a full restart.

3. MaxRequestsPerChild=1

Perrin Harkins recently alerted me to the MaxRequestsPerChild=1 technique. That is, set MaxRequestsPerChild to 1, then load any potentially-changing modules in the child, not the parent (obviously only for development environments). Each request will hit a fresh child server, which will load all of your potentially-changing modules anew.

This is the nicest solution I’ve seen so far. The only problem I can see is its performance – each potentially-changing module has to be loaded on each request. **

4. My idea: The Pied Piper

As in 3, load any potentially-changing modules in the child. Leave MaxRequestsPerChild alone. As in 2, fork off a “watcher” process that waits for your modules to change. When they change, kill all the server’s children explicitly.

The end result is that you get reasonable performance when your modules don’t change (e.g. when you are only futzing with templates), but when modules do change, you should see the effects immediately.

This should be able to work with mod_perl, fastcgi, Net::Server, etc., as long as the parent server responds appropriately to the killing of all its children (by launching new ones). Apache, at least, seems to be ok with this.

Going to try to implement this as a plugin in Server::Control. Will see how it goes.


** – You can try to load things only on demand, but often mod_perl code is written without ‘use’ statements as it assumes everything is loaded in the parent. You can also try to minimize the number of potentially-changing modules, but then you run the risk of leaving something off and having to adjust it and restart.

Sep 01

I’ve been thinking about what version 2.0 of Mason might look like. It’s been seven years since the last major version (1.10), and a lot has changed in the Perl world since then!

This is an initial brainstorm. Refinement and details to follow, and feedback is very much encouraged.

Name

  • HTML::Mason -> Mason. I believe we are finally entitled to a top-level name. :) This will help separate the new Mason from the old.
  • MasonX:: for extensions? This is more problematic, as the MasonX:: namespace already has a number of modules that are specific to the current Mason. Regardless, it needs to be a short and convenient prefix.

Implementation

  • Moose. We will use Moose for all classes. Moose represents a fundamentally different way of developing Perl, and it was the catalyst that started me thinking about a new Mason. Many of the remaining items in this list fall out of, or are enhanced by, Moosification.
  • Components will be classes. Components started out as subroutines, then became objects. The logical next step is for each component to be its own Moose subclass. This will allow components to benefit from true OO techniques (methods, attributes, roles), instead of Mason’s current painful pseudo-OO.
    • The entire component will be placed inside a MooseX::Declare class block, with a class name auto-generated from the component path.
    • The main component body – the content and the inline Perl sections – will be placed into a main method.
    • Calling a component (via <& &> or $m->comp) will mean creating a new instance of the component class, and calling its main method. Component call parameters will be passed to the constructor. Have to watch for performance impacts here.
    • A $self variable is automatically available, assigned to the component instance.
    • One of <%args> or <%attr> will be used to generate Moose attributes, rather than subroutine parameters. There will still be some syntactic sugar, but the section will have access to the full power of attributes. e.g.
      <%args>
         foo
         bar => (default => 5)
         baz => (default => 5, isa => 'Int')
         </%args>

      The advantage of attributes over parameters is that other methods in the component will have access to those values, something that is painful to deal with right now.

  • Subcomponents will become methods. <%def> and <%method> are currently implemented as their own component objects inside the main component. There was never a particularly good reason to do this other than that it seemed like a neat idea, and over the years it has proved more frustrating than helpful – e.g. $m->current_comp and $m->caller sometimes defy expectation. Now that components will be classes, it makes sense for a subcomponent to be just another method alongside main.
  • Autohandlers will be superclasses. Components will truly inherit from their autohandlers now. e.g. they’ll be able to call methods in the autohandler, a great way to share code within a hierarchy of components.
  • Smaller core, more extensions. As with other modern Perl frameworks, we should move as much as we can from the core to extensions, where we can do so without overly sacrificing ease of use, performance, etc.

Syntax

  • Same core syntax. The core syntax will be mostly the same – <% %> blocks, <%perl> and <%init> sections, %-lines. I do sometimes regret not going with the industry-standard <%= %> for expressions, so that <% %> could be used for non-expression Perl; but at this point <% $foo %> may be too deeply embedded in my brain to change it.
  • <%once> becomes <%class>. The <%class> section will contain code to be placed in the class definition outside the main method: ‘use’ statements, pure-Perl methods, utility subroutines, pragmas etc.
  • Automatic newline removal. Obvious newlines should be removed by default, so that components can be spaced neatly without adding gratuitous newlines to the output.
  • Consolidated filter syntax. Mason has several kinds of content filters: <%filter> section, components with content, and escapes (e.g. |h). All of these filtering methods should be consolidated under a single implementation, if not a single syntax. In addition, we need a way to create new content filters with Perl methods and use them on blocks. Standard filters could be installed on the main component class, and more specific filters on individual components. Example:
  • <%class>
       filter repeat ($times) {
           my ($self, $content) = @_;
           return $content x $times;
       }
       </%class>
    
       ...
    
       <% repeat (3) { %>
       blah blah
       <% } %>

New features

  • More powerful controller mechanisms. Mason has controller mechanisms – dhandler, autohandler, and the resolver – that were innovative once but are now underpowered. I don’t believe (as some do) that Mason should get out of the controller business – I actually like having my “C” and “V” in the same place. Mason just needs to provide easy access to the same powerful routing techniques that are used in modern web frameworks, e.g. via Path::Dispatcher or Path::Router. For example, an autohandler or dhandler or other special file could indicate how paths under that directory should be routed – still have to work out the syntactic details. Of course, nothing will prevent people from using an MVC framework’s routing and treating Mason like a controller-less template language.
  • Logging. Mason will log generously with different levels and categories via Log::Any. This was supposed to be put into Mason 1.x years ago but never made it…
  • Output placeholders. Support the insertion of placeholders whose values can be supplied by code further along in the request. This is motivated by HTML head elements, like title, that you want to output in the top template but determine in a more specific page component.

New extensions

  • MasonX::Tidy and MasonX::Critic. Tidying and criticizing specific to Mason components, analagous to what perltidy and perlcritic do for Perl.
  • MasonX::Safe. Use static code analysis (PPI) to restrict the set of Perl operations that could be executed in a set of Mason components. In the spirit of, but less draconian and broken than, Safe. Useful for environments when non-Perl-programmers are modifying components, or to give the “don’t do too much in components” philosophy some teeth. Intended for friendly policy enforcement, not airtight security.

What can be deprecated/eliminated

  • Web features. Mason has always ambiguously straddled the line between template language and web framework. This was out of necessity — when Mason was created, mod_perl was the only “web framework” in Perl. With the advent of web frameworks like Catalyst, Jifty and CGI::Application, and interfaces like PSGI/Plack, Mason no longer needs to occupy the web space. It should strive to simply be a great templating system that is easy to plug into various web frameworks. This means we can eliminate, among other things, the ApacheHandler and CGIHandler classes from the core.
  • Plugin API. In 1.3 we added a custom Mason plugin API, but Moose method modifiers and roles are a far better and more industry-standard way of supporting plugins.
  • <%cleanup> section. <%cleanup> has always been of dubious value because it is not guaranteed to run in case of an abnormal exit from the component. Now that components are classes, a DESTROY method in the <%class> section will be a more appropriate and functional way to perform cleanup.
  • <%shared> section. Since the various methods in a component are now part of the same class and share attributes, there is no reason to have <%shared>.
  • <%attr> or <%args> section. We only need to retain one of these to represent class attributes; the other can be eliminated.
  • <%method> or <%def> section. Perl methods will now be easy to define in the <%class> section. It will still be useful to define methods that produce output, but we only need one of these sections.
  • Non-CHI cache APIs. Eliminate the Cache::Cache and 1.0 APIs, and just support CHI.

What will be kept, begrudgingly

  • Components returning values. There are some cleanliness arguments for eliminating the ability of components to return values – one should use methods instead, etc. But I think this feature is too frequently used to discard. This is something to discourage in the docs and for MasonX::Critic to complain about. :)

What do you think?

Mason developers, let me hear your feedback and ideas.

Jul 13

Log::Any has finally been released to CPAN. It allows CPAN modules to log messages in a generic way, while letting the application choose (or decline to choose) a logging mechanism such as Log::Dispatch or Log::Log4perl.

This is something I’ve been meaning to implement for nearly two years – talk about procrastination! What pushed me over the edge was that three of my distributions – Mason, CHI, and an upcoming Server::Control – all needed to log various things, and had no good generic way to do it.

As part of this effort, I surveyed the various general-purpose logging packages on CPAN to see which ones Log::Any should interface with. Log::Dispatch and Log::Log4perl were the heavyweights and obvious first choices, but I was unprepared for the sheer variety of other distributions. Logging, like templating systems, seems to be one of those things that everyone wants to try their hand at. Here’s a partial list of logging packages that all share roughly the same mission – dispatching logs to various outputs – along with the number of distributions that depend on each:

Log::Log4perl 176
Log::Dispatch 40
Log::Agent 18
Log::Report 16
Log::Trace 13
Log::Dispatch::Config 9
Log::Message 5
Log::Channel 3
Log::Dump 3
Log::Handler 2
Log::Info 2
Log::LogLite 1
Log::FileSimple 1
Log::Message::Simple 1
Log::StdLog 1
Log::Simple -
Log::Simplest -
Log::Tiny -
Log::Trivial -

I particularly like that there are seven packages named with a variation of “Simple”, “Trivial”, etc. Trying to choose between these would be the opposite of simple.

To be fair, Log::Dispatch, presumably the easier of the two main logging packages to configure, requires this just to log normally to a file:

   my $dispatcher = Log::Dispatch->new();
   $dispatcher->add(
       Log::Dispatch::File->new(
           name      => 'foo',
           min_level => 'info',
           filename  => "$dir/test.log",
           mode      => 'append',
           callbacks => sub { my %params = @_; "$params{message}\n" },
       )
   );
   $dispatcher->info('this is a message');

It’s no wonder people tried this and yearned for a simpler alternative. (I’ve gotten permission from Dave to simplify a few of these things with a maintenance release.)

Jun 26

I finally got to my first YAPC this year – coincidences and conflicts kept me away in previous years. So excuse me for getting all dewy eyed, but it was a great experience! Haven’t felt that close to a development community since the early Oreilly Perl conferences. I feel all energized about Perl again and it makes me want to host a winter workshop in the bay area someday.

Here are some cool things I learned about:

  • distroprefs – A way to improve the automatability of CPAN installs; namely, dealing with those pesky distributions that insist on asking you questions. We spent a bunch of time at Hearst writing expect scripts to deal with these so that we could automatically install our large CPAN base on different servers. Wish I’d known about distroprefs!
  • Dist::Zilla – I’ve always been vaguely bothered by the amount of boilerplate in my CPAN distributions (copyright, author, version, etc.) but Dist::Zilla has finally provided the means and motivation to do something about it.
  • KiokuDB – Stevan Little’s talk on KiokuDB got me excited about leaving behind RDBMSs and the Impedence Mismatch on my next web project. Just store Moose objects! Of course people have been getting excited about object databases since the 80s, and there’s probably a reason that Oracle is still around. But allow me to enjoy my dream before it smashes into reality.
  • Memory persistence – As Steven Lembark gleefully hammered home, Perl never shrinks a variable. Once $foo has had a million bytes or @foo has a million entries, it’ll reserve that space forever, even if you empty it. Makes me want to look real close at any globals in persistent web environments.
  • CGI::Inspect – Brock Wilcox wins the award for most satisfying feature demo. Insert CGI::Inspect’s inspect() somewhere in a web template. When it runs, a dashboard launches in a new browser window, with a stack trace and the ability to inspect and change lexical state before the page resumes its render. Of course, other environments like JSP/Eclipse have had live debugging of web pages for years, but in Perl this still seems groundbreaking :) , and the possibilities for extending it are intriguing.

Here are the pages for the talks referenced above:

  • CPAN – A big enough lever to install the world
  • Dist::Zilla – Automating quality since 2008
  • KiokuDB – A Real World Introduction
  • Memory Manglement
  • Drop-In Web-Based REPL for CGI Applications
  • preload preload preload