To illustrate this concept, I’m going to provide an example from a recent project I’ve been working on. On a Go board, adjacent points in same state are considered connected:

So the groups marked 1, 2 and 3 are each distinct in this regard. I’m also going to refer to 4 and 5 as connected groups, this is useful for the logic I’m developing even though Go players don’t refer to them in this way.

I want to develop a cache of these groups, which can be accessed efficiently using a javascript Map. As new stones are added and removed from the board, this will need to be updated accordingly.

Smaller is Better

A point on the board is comprised of x and y coordinates, which feels natural to represent as [x, y]. It’s tempting to also the state of that position, but this is the first lesson: the more sprawling the abstraction the harder it is to reason about it.

To illustrate this point, let me share with you a simple utility function I use in several places in this project:

function neighbors([x, y]: Point): Point[] {
  return [[1, 0], [-1, 0], [0, 1], [0, -1]]
    .map(([ox, oy]) => [x + ox, y + oy])
}

Because it doesn’t need to be aware of the state of the board, it can remain self contained and more generic. If it needs the state, how do we figure out what to include there? One option would be to put some sort of dummy value:

function neighbors([x, y, _]: Point): Point[] {
  return [[1, 0], [-1, 0], [0, 1], [0, -1]]
    .map(([ox, oy]) => [x + ox, y + oy, " "])
}

This would be functional, but adds unnecessary weight to the abstraction. Now, whenever you use the results of this function, you have to keep in mind that the state of that point should be disregarded. Eventually, you’ll accidentally use that state and have a confusing bug to track down when it leaks into other parts of the code.

The other option is to provide enough context to provide meaningful state:

function neighbors([x, y]: Point, board: Board): Point[] {
  return [[1, 0], [-1, 0], [0, 1], [0, -1]]
    .map(([ox, oy]) => {
      const nx = x + ox
      const ny = x + oy
      return [nx, ny, board.get(nx, ny)]
    })
}

Again, this would be functional, but introduces different problems. To start, when writing the tests for this function, a Board has to be constructed to provide this state. Next, knowledge of the state is helpful in some situations but can become a burden in others, which I’ll come back to when I discuss sets of points.

Grouping Reduces Complexity

Discussing the entire Board class is beyond the scope of this article, but notice that because Point doesn’t include state we can have a signature like this: get(point: Point): State | undefined. In the previous example, because state was part of the type, we had to call a version that looked like this: get(x: number, y: number): State | undefined.

To understand the consequence of this, consider the situation where you only want neighbors that are valid board positions:

class Board {
  ...
  neighbors(point: Point): Point[] {
    return neighbors(point).filter((n) => this.get(n))
  }
  ...
}

Notice that this function doesn’t need to understand anything about how Point is structured. If it was aware of state, then to call get we would have to break into the individual x and y coordinates. Moreover, it ceases to make sense to decompose the work into two simple functions.

class Board {
  ...
  neighbors([x, y, _]: Point): Point[] {
    return [[1, 0], [-1, 0], [0, 1], [0, -1]]
      .map(([ox, oy]) => board.get(x + ox, y + oy))
      .filter(n => n)
  }
  ...
}

What you’re witnessing here is how balls of mud begin to form. The wrong abstraction was chosen for Point, so logic has begun accumulating instead of being decomposed. The former version was compact and bootstrapped from a more general concept, whereas the latter must know about everything to accomplish its goal.

Also notice how much harder it is to understand the latter version, instead of taking it in at a glance you have to understand the goal of the low level transformations. Genies might be able to determine that rapidly, but it takes more tokens to do so and over an entire codebase this can lead to slower and more expensive development costs.

Unnecessary Weight

Because Javascript’s Set implementation would not work as expected (the abstraction is leaky), I had to develop PointSet to handle these. The implementation is rather boring, so I won’t share it here, but it’s a class I use all over the place.

Part of what makes it so useful is that it conforms to the Single Responsibility Principle. It has one job and does it well, and I rarely think about how it accomplishes its goal. Keeping Point to x/y coordinates is the secret sauce of why I never have to think about it, to understand why let’s explore what would happen if it included state.

If we include state as part of what makes a point unique, it could be put in a state where the same coordinates can be in two states simultaneously. Even if you never care about the state, it’s still vulnerable to this kind of problem:

const set = new PointSet()
set.add([0, 0, "B"])
set.add([0, 0, "W"])
set.remove([0, 0, "W"])

When remove is called, it’s reasonable to expect that (0, 0) is no longer in the set. If you iterate over the members, though, you’ll find that a phantom [0, 0, "B"] is still hanging around.

To address this, we could consider ignoring the state when it comes to uniqueness. As long as you’re the only one to ever interact with the class, it might be feasible to keep that detail in your head. Now that we have genies, though, this is virtually never true. In my experience, this manifests as a hallucination assuming that uniqueness extends to the state. As a result, every interaction with the class grows defensive logic, guarding against the possibility that an existing point exists with a different state.

Both of these options are sub-optimal, and the consequences add weight to interacting with the class. It’s a detail that needs to be tracked, adding to either your mental or the genie’s context. If you try to shed that weight, it results in subtle errors or superfluous code, both of which simply shift the weight into debugging time or the codebase.

Levels of Abstraction

Now we can finally discuss the group cache, which will use a two tiered Map to provide rapid lookup.

export class GroupCache {
  groups: Groups
  constructor(starting_groups?: Groups) {
    if (starting_groups) this.groups = starting_groups
    else this.groups = new Map()
  }

  at(point: Point): PointSet | undefined {
    return this.get(point)!.points
  }
  ...
}

To add members to this cache, an update function is defined:

update(point: Point, state: State) {
  this.remove_from_existing_group(point, state)
  this.set(point, {
    state,
    points: this.add_to_new_group(point, state),
  })
}

What’s important to understand here is that all of this logic operates at the same level. No part of at or update looks inside of point, they delegate to get and set to operate on those:

private get([x, y]: Point): PointGroup | undefined {
  return this.groups.get(x)?.get(y)
}

private set([x, y]: Point, point_group: PointGroup) {
  if (!this.groups.has(x)) this.groups.set(x, new Map())
  this.groups.get(x)!.set(y, point_group)
}

What’s interesting is that, although these operations aren’t complex, they have some nuance protecting against common errors. For example, set first ensures that the map exists for x before attempting to call set on it. None of that matters from the perspective of update, though, it can focus on what it means to update instead of having to muck about with data structure massaging.

Genies

As I’ve alluded to a few times, this kind of thinking and design skills are precisely what’s needed when working with coding assistants. My first attempt at creating this cache was done by Codex, which tightly bound the cache to capturing rules from the game. Sometimes that’s good enough, but in this case it fell apart when I started tracking captures for scoring purposes.

When you take care to form clean abstractions, though, these assistants can really sing. When I was working on remove_from_existing_group, I needed to break apart a single PointSet into subsets that were connected. This logic is a bit hairy, but because the goal had such a clean definition, I was able to generate that functionality rapidly.

The takeaway isn’t that genies are awful or wonderful, it’s to better understand what they are good and bad at. This article explores that boundary to help you better understand how to navigate it.

function equals(x, y) {
    return x == y;
}

How would I know that this property is related to that function:

property(
    integer(), integer(),
    (x, y) => equals(x, y) == equals(y, x),
)

Obviously it’s calling the function, but how do we figure out if the call is incidental or an attribute of the function itself. Static analysis is an option, but that can be challenging to build and sequencing the steps would be somewhat complex to orchestrate.

Other languages provide a way to annotate functions and classes, but in the Javascript world the only conversations I can find are around documentation. Javascript is a pretty dynamic language, though, so maybe there’s something clever we can do there. Here’s the property, definition and test reorganized a bit; it doesn’t get us where we want to go, but helps organize things a little differently.

const equals_property = property(
    integer(), integer(),
    (x, y) => equals(x, y) == equals(y, x),
)
function equals(x, y) {
    return x == y;
}

test("symmetric", () => {
    assert(
        equals_property
    )
})

One possible idea is to use first order functions to wrap function creation in a way that attaches a provide property:

function def(property, func) {
    func.property = property;
    return func;
}

const equals = def(
    property(
        integer(), integer(),
        (x, y) => equals(x, y) == equals(y, x),
    ),
    (x, y) => x == y,
)

test("symmetric", () => {
    assert(
        equals.property
    )
})

So then it does seem possible to attach a property directly to the associated function. It might get a little complicated when we begin looking at defining objects like this, but this is enough to demonstrate that it’s possible to establish a direct link between a piece of logic and a property.

function equals(x, y) {
    return x == y;
}

Let’s also take an extremely simple property related to this expression:

property(
    integer(), integer(),
    (x, y) => equals(x, y) == equals(y, x),
)

Our goal is to identify some mechanism which can determine that such a property can be inferred in a function calling equals:

function double_equals(x, y) {
    return equals(2*x, 2*y);
}

Building this example, though, I’m beginning to see a counterexample to the original premise. Of course, double_equals retains the symmetry of equals, but something like this wouldn’t:

function is_double(x, y) {
    return equals(x, 2*y);
}

Perhaps there’s a special circumstance when the structure of both arguments are identical, but this feels like it’s diving too heavily into the specifics of mathematics. It’s also possible there’s more advanced things to explore in this area, but I’m looking to tackle something less complex.

So instead of confidentally inferring all properties from lower level abstractions, I’m envisioning a system which can extrapolate candidates properties. It would propose symmentry for both double_equals and is_double, but leave it up to the user to determine its relevance. It also gives the user an opportunity to define a revised version of the property such as this:

property(
    integer(), integer(),
    (x, y) => is_double(x, y) == is_double(y*2, x/2)
)

I didn’t achieve what I set out to at the beginning of this article, diving into the practical mechanics of substitution, but I was able to identify a flaw in the original approach. The original idea might eventually be possible, but this revised approach proves to be a much more tractable problem.

I’ll use some simple mathematical properties as a way to explain what I mean. Consider, for example, a property true of the > relationship:

property(
    integer(), integer(), integer(),
    (a, b, c) => (a > b && b > c) ? a > c : true,
)

Here’s a simple function that could be built on top of that relationship:

function positive(x) {
    return x > 0;
}

We of course know that, if j > k and k is positive, then j must also be positive. What I’m interesting in determining is if, because I defined the transitive property as a generative test, such a property could be automatically determined.

Before I embark on trying to pry apart the mechanics of how to figure something like that out, though, I need to validate this makes logical sense. To accomplish this, we somehow must infer that c == 0 in the original property; perhaps easier said than done, but assuming it is determined leads to the following property:

property(
    integer(), integer(),
    (a, b) => (a > b && b > 0) ? a > 0 : true,
)

Then we can transform this again by replacing with the definition of positive:

property(
    integer(), integer(),
    (a, b) => (a > b && positive(b)) ? positive(a) : true,
)

These transformations are far from trivial in most languages, but shows it’s at least possible to make logical transformations from base properties to more elaborate ones. This kind of meta-analysis might be more than Javascript is capable of in a practical way, but now I at least know it’s logically possible to reach such conclusions.

First, though, let me talk about what I’m looking to explore. A lot of people talk about how unit testing is a reasonable approximation for proving the correctness of a program, but what’s always sat uncomfortably with me is how easy it is to overlook corner cases. Generative testing offers the potential to more comprehensively cover a piece of code, and I wish to examine how it can be used to more comprehensively cover the contract it fulfills.

TestCheck.js

A basic property is defined like this:

const { check, gen, property } = require('testcheck');

const result = check(
  property(
    gen.int,
    x => x - x === 0
  )
)

I suppose I could write something to look at that result, but that can be a little clunky when I’m looking to use this frequently. It does offer test framework integration, but because it was last updated 6 years ago it doesn’t support Vitest which appears to be the current front runner.

I can still wrap it in a test with an expectation, though, so still pretty easy to play around with:

test("x - x", () => {
    const result = check(
        property(
            gen.int,
            x => x - x === 0
        )
    );
    expect(result.result).toBeTruthy();
});

Here’s the basic call signature, stripped of specific examples:

check(
    property(
        gen.<type>,
        <some function returning a boolean>
    ),
    <options>
)

There are a ton of different generators, enough that most conceivable cases would be covered. It’s also possible to construct new generators by composing the ones provided, which gives a reasonable amount of extensibility.

fast-check

Fast Check appears to be more actively maintained and feature rich, but the documentation isn’t as well organized. The basic usage is pretty much identical:

test("x - x == 0", () => {
    fc.assert(
        fc.property(
            fc.integer(),
            (x) => x - x == 0
        )
    )
})

Because it’s actively maintained, though, it supports modern testing libraries and so I didn’t have to examine the result to get the test to fail.

This library also seems to provide many generators which can be extended to check most things I could imagine.

Conclusion

The basic footprint of all JS generative testing libraries is the same, although there only appears to be one library actively maintained. Even though it’s not too cumbersome, the footprint in both cases is a bit clunky and leads to a lot of extra code to transition between a simple test and a property based test.