Jan 31

chromatic shows users how to run tests faster on cpanm and perlbrew installs. This strikes me as well-meaning advice that misses a much more basic point:

cpanm and perlbrew should not run tests by default.

This may sound heretical. Perl has always had a strong testing culture, and end-user testing may have once played a valuable role in testing a distribution under many systems. But we now have a CPAN Testers network which will run tests on countless systems and Perl versions, and report failures back to the author promptly and automatically. Distributions can be sent through the Testers’ gauntlet before ever being officially released. In this environment, it’s hard to see much additional value in ad hoc end-user testing.

As Dave Rolsky points out, we install most other software without running tests and rarely give it a thought.

None of this would matter if end-user testing was free. But it is not.

The costs of end-user testing

Slower installs. On my system, a fresh install of Moose and its dependencies takes three times longer with tests (2 minutes versus 41 seconds). A fresh install of Catalyst and its dependencies takes nearly four times longer with tests (9.5 minutes versus 2.5 minutes).

How many new Perl users find CPAN installs much slower than they need to be? How many would choose a 3-4 times speedup if they knew it was an option? It’s like having a turbo button and leaving it unpressed by default.

False positives. The more tests CPAN authors write, the more likely an occasional false-positive failure sneaks through (as in “Failed 1/1746 tests.”) In most such cases the module will still work for the user’s purposes. But the default behavior is to prevent the module from being installed at all. If your module depends on other modules, then any failure up the dependency chain likewise prevents your module from being installed, even if the failure has no bearing on your module’s efficacy.

How many new Perl users have unnecessarily failed to install a module like Moose or Catalyst because of an obscure, temporary failure deep in the dependency chain?

Fear of dependencies and code reuse. Slower installs and false positives are the main reasons why people complain about distributions having “too many dependencies”. (If the dependencies installed quickly and reliably, as they do with –notest and apt-get and yum, would anyone complain or even notice?) These complaints in turn encourage module authors to reduce or eliminate their dependencies, thus reinventing where they could be reusing.

It’s the wrong default for new users

Some veteran Perl folks may like tests to run on every install. That’s fine. But I suspect new Perl users just want things to install quickly and reliably, and in any event don’t have the experience to evaluate or take action on a test failure (especially one in an obscure dependency). For these users running tests is simply the wrong default.

I turn on –notest on each system I administer and preach it enthusiastically to every new Perl user I encounter. But I wish I didn’t have to mention it at all.

preload preload preload