Jan 31

chromatic shows users how to run tests faster on cpanm and perlbrew installs. This strikes me as well-meaning advice that misses a much more basic point:

cpanm and perlbrew should not run tests by default.

This may sound heretical. Perl has always had a strong testing culture, and end-user testing may have once played a valuable role in testing a distribution under many systems. But we now have a CPAN Testers network which will run tests on countless systems and Perl versions, and report failures back to the author promptly and automatically. Distributions can be sent through the Testers’ gauntlet before ever being officially released. In this environment, it’s hard to see much additional value in ad hoc end-user testing.

As Dave Rolsky points out, we install most other software without running tests and rarely give it a thought.

None of this would matter if end-user testing was free. But it is not.

The costs of end-user testing

Slower installs. On my system, a fresh install of Moose and its dependencies takes three times longer with tests (2 minutes versus 41 seconds). A fresh install of Catalyst and its dependencies takes nearly four times longer with tests (9.5 minutes versus 2.5 minutes).

How many new Perl users find CPAN installs much slower than they need to be? How many would choose a 3-4 times speedup if they knew it was an option? It’s like having a turbo button and leaving it unpressed by default.

False positives. The more tests CPAN authors write, the more likely an occasional false-positive failure sneaks through (as in “Failed 1/1746 tests.”) In most such cases the module will still work for the user’s purposes. But the default behavior is to prevent the module from being installed at all. If your module depends on other modules, then any failure up the dependency chain likewise prevents your module from being installed, even if the failure has no bearing on your module’s efficacy.

How many new Perl users have unnecessarily failed to install a module like Moose or Catalyst because of an obscure, temporary failure deep in the dependency chain?

Fear of dependencies and code reuse. Slower installs and false positives are the main reasons why people complain about distributions having “too many dependencies”. (If the dependencies installed quickly and reliably, as they do with –notest and apt-get and yum, would anyone complain or even notice?) These complaints in turn encourage module authors to reduce or eliminate their dependencies, thus reinventing where they could be reusing.

It’s the wrong default for new users

Some veteran Perl folks may like tests to run on every install. That’s fine. But I suspect new Perl users just want things to install quickly and reliably, and in any event don’t have the experience to evaluate or take action on a test failure (especially one in an obscure dependency). For these users running tests is simply the wrong default.

I turn on –notest on each system I administer and preach it enthusiastically to every new Perl user I encounter. But I wish I didn’t have to mention it at all.

21 Responses to “Stop running tests on install!”

  1. Christian Walde Says:

    > This may sound heretical.

    You mean EXTRA heretical ( http://bit.ly/yPmsum ). To use an anecdote to validate the idea of test on install:

    Just yesterday i tried to install Pod::WikiDoc for a new perl developer at $work. Turns out it fails pretty spectacularly. ( http://bit.ly/yTTOxL ) Further inspection led to the discovery that a recent release of a newer version of a dependency had broken Pod::WikiDoc. Without the tests and other test reports to compare against, it would’ve taken a LOT longer to find the issue here.

  2. ambrus Says:

    It’s testing perl at build time on my individual home machine that has started an investigation where khw has eventually uncovered a bug in GNU libc. My configuration was not broken, just unusual. The story is complicated, and partly documented at “https://rt.perl.org/rt3//Ticket/Display.html?id=88822″ and “http://sourceware.org/bugzilla/show_bug.cgi?id=12788″ .

  3. ambrus Says:

    Though on the other hand, I also manually skip some tests for perl modules when they carry too much dependencies. And I don’t even try to run the full testsuites of gcc and ghc because they seem to take infinite time.

  4. ilmari Says:

    The problem is that CPAN Testers doesn’t stop versions that fail their tests from being installed by default on a user’s system. If you disable running tests you should use the appropriate cpX.X.Xan, to only get modules that pass on your version of perl: http://cpxxxan.barnyard.co.uk/

  5. Ed Avis Says:

    You don’t run tests if installing a binary package built for your system. Whoever made that package will already have tested it. But if *compiling* from source, you normally would do ‘make test’ before ‘make install’.

    Which of the two is more analogous to installing Perl modules using tools like ‘cpan’? It’s a bit of both, but since you don’t have exactly the same environment on your system as the module’s author, I would err on the side of treating it as ‘compilation’ and run the tests. It would be great, though, if testing could be done after installation. Then you could install quickly and get on with your work, while a nightly cron job runs the test suite for all installed modules and lets you know if something broke.

    For a quick end-user install without tests, I recommend using a package manager such as rpm or dpkg. Here somebody else has done the ‘compilation’ step, and if the package is provided by your OS vendor, it has been tested to work with the particular version of perl and other CPAN modules also shipped by that vendor.

  6. Tatsuhiko Miyagawa Says:

    Thanks for writing this.

    I actually agree with you on making notest the default behavior. It is probably the last bit of what I wanted to add to cpanm’s “sane” defaults behavior, but couldn’t, because of all the backlash from the existing developers user base at that time.

    What they want instead currently is to add a support for CPAN testers reporting, which is kind of reasonable since if you run tests but doesn’t have an ability to report that test failure, it is just a waste of resources and a burden to prevent installs, like you point out. I has been declared the patch is welcome for that, and I have never received one yet.

    Anyway, for Carton, the CPAN dependencies manager software that uses cpanm internally, the default is set to notest. There may be an option for developers to turn on tests during the development, but for the default use case of deployments, I think notest makes much more sense.

  7. Jonathan Swartz Says:

    Christian: Not to downplay your story but it seems like the first ‘use Pod::WikiDoc’ on your developer’s system would have failed pretty spectacularly as well, which then presumably would have led you to manually run the tests. So is this an argument for running every test on install, or just an argument for having tests that you can run as needed?

  8. Jonathan Swartz Says:

    ambrus: You are using bleadperl, so you are definitely not the intended target of this post. :) Of course you should be running tests by default if you choose; you’ll be able to react to and handle failures appropriately. I’m concerned with the default behavior for new Perl users.

  9. Jonathan Swartz Says:

    > It would be great, though, if testing could be done after installation. Then you could
    > install quickly and get on with your work, while a nightly cron job runs the test suite
    > for all installed modules and lets you know if something broke.

    Ed: could not agree more. Even without the cron job, it should be easy to run tests later if a module is not working as you expect.

  10. Robert Sedlacek Says:

    The thing is: It isn’t really easier at the moment to run the tests afterwards. Plus this would be even harder for new Perl users in my opinion. If package A depends on package B, and package B has a problem, but A doesn’t exercise that part, or doesn’t trigger the behavior in B with its tests, the new user would have to retest the whole dependency tree, and that might not even give them a solution. This is a step that can be currently skipped for (truthful) users by asking if they installed normally.

    I’d rather have this as a feature for power users. Personally, I’d even favor a default where a system asks a user if we wants to rerun the tests of all installed modules that depend on the one I just updated :)

  11. Christian Walde Says:

    Jonathan: The main issue here is that without test reports from *users* (not dedicated smoke boxes) it would’ve been a lot harder to figure out what was going wrong. And if everyone installs without tests, we will see less of those. It’s fine to skip testing for deployment. I use Carton a lot myself, but for development tests and reporting are a must, if only for the community.

    Then there’s also this question: What if the failure had been not spectacular, but rather insiduous? A while ago DBD::mysql had the weird error on windows that it swallowed all errors sent by the DB. This was almost unnoticable in production until i started writing tests to ensure the web app handled things right when the db messed up.

  12. Billy Vierra Says:

    I disagree completely. You are building from source. Yes you are right, it takes longer, but users want to know it works when it is done.

    In some instances you have a hard crash when you run it if the tests were to fail, but what about the other instances? What if it passed back a wrong # from a mathematical operation? Imagine trying to track that down?

    The time tests add on are less of an issue than a user not fully understanding Perl and trying to figure out why his program doesn’t work when he is positive his code works.

  13. Steven Haryanto Says:

    @Christian Walde: “The main issue here is that without test reports from *users* (not dedicated smoke boxes) it would’ve been a lot harder to figure out what was going wrong.” Then when the user reports error, ask him/her to run the test.

    @Billy Viera: “What if it passed back a wrong # from a mathematical operation? Imagine trying to track that down?” Yes, we can start dreaming up all sorts of hypothetical weird errors, and yes there’s Murphy’s Law and all that. But first of all we must ask ourselves, have *we* (the developers) written good tests to catch such errors? As much as we hate to admit it, a large significant portion of CPAN is crap, including the tests. Most module authors don’t test enough edge cases, yet sometimes they do stupid things like trying to send email to a real SMTP port and hangs (the case of Mail::Sendmail, for several years).

  14. Steven Haryanto Says:

    Since we are “very obsessed with testing” (BTW, these days I’m hearing this statement from the Ruby folks), why don’t we *test* the default notest on some CPAN clients like cpanminus? And see whether the sky indeed will fall down.

    If anything, this will motivate CPAN authors and CPAN Testers to broaden the test coverage so potential problems with end users are minimized.

  15. Jonathan Swartz Says:

    Tatsuhiko: Thanks for your reply, it saves me the trouble of appealing to you directly. :) I guess we’re in the minority, though I actually didn’t realize my stance was that controversial before I posted it.

  16. Perrin Harkins Says:

    I just want to register my support for not running the tests at install by default. The vast majority of failures I get from CPAN shell installs are not real failures but rather problems with the tests, and newbies are not confident enough to charge ahead and force install when this happens.

  17. Chris Hutchinson Says:

    It sounds like a useful default to speed up installs for app dependencies, particularly when we’ve already run the tests on each module individually.

    How do you set -notest on your systems, or per-CPAN invocation? In MyConfig.pm, or elsewhere?

  18. Nanu Says:

    I think having it enabled would really be nice, if the tests would be reported. Since this isn’t the case the chance of anyone benefiting from that is really small. I really hope that cpanm gets CPAN Testers support soon. That would be great, since cpanm is really a great tool, especially when using it together with perlbrew.

  19. Jonathan Swartz Says:

    @Chris Hutchinson: I have this in my .bashrc:

    export PERL_CPANM_OPT=”–skip-installed –notest –auto-cleanup=0″

    If you’re using the regular CPAN client, I don’t know of a way to automate this – I just got used to typing “notest install …”. But I recommend you switch to the great cpanm if you can. :)

  20. Andrew Grangaard Says:

    A binary install option of pre-tested modules would be pretty neat.

    This would be unfeasible for the wide range of supported platforms that perl supports, but perhaps a popular subset? This gets us back to the dpkg example, but then we’d need to keep our debs up to date, and we don’t. How many of use maintain debs for our modules <sfx: not raising hands>? This also brings us back to system perl, and no one is using that, right?

    At previous $work, we were maintaining a local::lib directory in source control, but then ran into 32 vs 64 bit issues.

    What’s a nice solution for the $work space? Running full tests in my qa environment and using these to create approved binary bundles to push to production? (maybe fpm packages https://github.com/jordansissel/fpm ) But that doesn’t even help in the local::lib multiple incompatible projects space, does it?

    Seems like it takes us back to deploying without tests if we want speed?

  21. Andrew Grangaard Says:

    Interesting conversation on travis-ci issue 72: https://github.com/travis-ci/travis-cookbooks/issues/72 .

    It started at “Can you pre-install Moose and Dist::Zilla?” “No, we tried, it took too long,” and progressed beyond to local Dist::Zilla mirror and cpanm options for faster installs.

    As of one month ago, travis-ci preinstalls:
    Dist::Zilla Moose Test::Pod Test::Pod::Coverage Test::Exception Test::Kwalitee Dist::Zilla::Plugin::Bootstrap::lib Catalyst LWP Module::Install Test::Most

Leave a Reply

preload preload preload