# Alternative clojure.test Integration With test.check

I’ve enjoyed using test.check lately, but its integration with clojure.test doesn’t fit with how I want to use it very well. In this post we’re going to explore a different approach, which I’m going to try to get into test.chuck.

## The Problem

To demonstrate my difficulty, consider these basic tests in clojure.test:

Instead of checking hard-coded integers, I would prefer test.check to do its magic. To do this using defspec (the existing integration option), I would have to do something like this:

There’s a lot of boilerplate here, and it’s not communicating that these two properties are related. Alternatively, I could join everything into one big condition:

This is also unsatisfying, as I wouldn’t be able to tell easily what failed.

Setting aside that issue for now, let’s look at the output:

Unlike deftest, defspec prints a report even when it passes. If I start using properties liberally my test output will quickly get too noisy.

The transition path is also difficult. Look at the original deftest code and notice that moving to defspec feels like a complete rewrite, as opposed to an upgrade from hard-coded to generated values.

Also, consider the case where we’re performing multiple assertions stepping through a stateful piece of code.

Moving this to defspec can be tricky:

## The Alternative

What I’d really like to say is exactly the same thing as the original deftest code, with just a little bit of variation for the generated values:

Notice how simple it is to move from testing to checking. With defspec the code screams generative testing and mentions the assertions as an aside. With checking generative testing becomes the aside and the assertions becomes the focus.

## Naive Implementation

A first pass on checking looks something like this:

This works because the is macro completely bypasses the tc/quick-check construct. While this gets us limping along, there are a couple problems.

First, let’s force a failure:

Now look at the output for this failure:

We’re seeing a failure for every attempt when tc/quick-check tries to narrow down to the smallest failure. All we really want to know about is the result of this search.

The other problem is that tc/quick-check only traces when the final sexp is false:

Which becomes obvious in the test output:

## Intercepting Reporting

We’re seeing the failure, but getting a lot of noise as well. What we really need here is an alternative test-reporting framework that can be nested within the checking macro. Fortunately, clojure.test is designed to replace that framework by overriding the report multimethod. Let’s look at what one of these reports looks like:

So the condition quick-check is looking for is actually if :type is not :pass. If we could catch that investigations would work correctly:

Output:

## Tracking Reports

This guarantees that the investigation happens, but we’ve lost the individual assertions in the process. What we need is a way to get at only those reports which were generated in the final failing execution. There’s not a mechanism for passing information out of a failure, so we’ll have to use a closure with some state to simulate the effect:

And the output:

## Including Results

So things are functioning correctly, and the failures are easy to read. In more complex scenarios the value may have been transformed heavily before the assertion is made, in which case we want to present the tc/quick-check return value.

Output:

## Cleaning Up

I’m happy with how things are reported now, but the code is a pretty big mess. Decomposing the logic should help with that:

## Number of Tests

One final touch is that the number of tests really shouldn’t be hard-coded. This adds to the checking footprint slightly, but that seems worth it to simplify our ability to control the number of tests being run.

# Update

The macro has been accepted to test.chuck in release 0.1.12. Gary Fredericks helped me work out some concurrency problems with the above implementation.

First, with-redefs rebinds things globally, which means that we can end up saving to the wrong atom. It’s much better to use binding:

Second, using reset! in save-to-final-reports causes a race condition between checking the condition and assigning to the atom. To get around this we can rearrange the arguments of save-to-final-reports and call swap! instead: