Stefil

Simplicity is power!

After extensively using some of the other test frameworks we wanted something that is much more out of the way then usual unit test libraries and better fits our daily routine with Slime.

The interesting primitives of Stefil are deftest, defsuite, defixture and some assertion macros like is, signals and finishes, and probably that's about it. The novel (ehhh) idea in Stefil is that tests are basically instrumented defun's that ensure a runtime dynamic environment where the assertion macros can record various information about the assertions and failures, and provide useful restarts when dealing with errors and/or failures. The process of the dynamic environment setup properly handles nesting, so you can call any of your tests from the REPL (assuming that you properly defined your fixtures if you happen to need them). Also, instead of textual descriptions of the test results, we customized Slime's inspector which include things like actions to rerun tests.

As you can see, we look at (unit) testing as writing an ordinary application using slightly modified versions of the well known Lisp primitives, that uses as much functionality of the base lib with as little user interaction as possible. And to be a bit more useful, in the process this test application executes as much assertions as it can. Stefil provides ways to annotate your program with constructs that help storing the results of those assertions together with other contextual information (e.g. the test backtrace) that helps finding regressions and failures.

Now if I mention a few more features then I think we've given a full documentation of Stefil:

• The assertion macros work without any test environment in which case they just print to *debug-io* (think of copy-pasting code fragments to the REPL)
• defsuite directly maps to deftest with a body that executes all sub-tests that have no mandatory arguments (except when they are created with :auto-call nil). Tests can be C-c C-c'd and they'll be smart enough to know their original suite.
• Fixtures (see defixture and with-fixture) are also mapped to simple defun's (hint: REPL use) that are guaranteed to be setup and teared down as few times as possible
• The :compile-before-run keyword lets you specify that tests should be freshly compiled for each test session:

  (deftest (x :compile-before-run nil/t) ())

  Tests that need to have lexical bindings should not be recompiled independent from the default, which is disabled.
• There are various restarts registered when the tests are running (e.g. to skip all the other unexpected errors and/or assertion failures, or to rerun the failed test, etc)
• The topmost invocation of a deftest returns the test results as the nth + 1 value in the return values for easy inspection in Slime
• *test-result-history* and *last-test-result* together with run-failed-tests
• Simple time measurement for tests that measure the real time spent with tests. Available in the inspected test results.

See the demo page or self-tests.lisp for more examples. There's also an extensive test framework comparision at Phil Gregory's blog.

• Stefil-devel for any discussion about Stefil

Attila Lendvai

Tamás Borbély

Levente Mészáros

You can browse the Stefil darcs repository or get the tree with

darcs get http://common-lisp.net/project/stefil/darcs/stefil

We are using a branched Slime, but from time to time I test with the official Slime and guard features that need our branch with #+slime-wdim-branch. If you are interested, you can get that branch from the link below, but Stefil should be fully functional with the official Slime assuming it's recent enough. Please note that the relevant differences between the two Slime branches are subtle and mostly aesthetic ones in the inspector.

darcs get --partial http://common-lisp.net/project/cl-wdim/darcs/slime

BSD / Public Domain