And the existing validation logic, defined as validateSignup. We won’t dig into to the implementation yet, only its type signature:

It’s a pure function, taking SignupForm data as an argument, and returning a Validation value. In case the form data is valid, it returns a Signup data structure. This data type resembles SignupForm in its structure, but refines the age as a Natural when valid:

In case the form data is invalid, validateSignup returns a non-empty list of SignupError values. SignupError is a union type of the possible validation errors:

The Validation Type

Also, we’ll need to import the implementation module:

We’re now ready to define some property tests.

A Positive Property Test

The first property test we’ll add is a positive test. That is, a test using only valid input data. This way, we know the form validation should always be successful. We define prop_valid_signup_form_succeeds:

First, we define genForm (➊), a generator producing form data with valid names and ages. Next, we generate form values from our defined generator (➋). Finally, we apply the validateSignup function and pattern match on the result (➌):

• In case it’s successful, we have the test pass with pure ()
• In case it fails, we print the failure' and fail the test

The validName and validAge generators are defined as follows:

Recall the validation rules (eq. 1). The ranges in these generators yielding valid form data are defined precisely in terms of the validation rules.

The character generator used for names is alphaNum, meaning we’ll only generate names with alphabetic letters and numbers. If you’re comfortable with regular expressions, you can think of genValidName as producing values matching [a-zA-Z0-9]+.

Let’s run some tests:

λ> check prop_valid_signup_form_succeeds
  ✓ <interactive> passed 100 tests.

Hooray, it works.

Negative Property Tests

In addition to the positive test, we’ll add negative tests for the name and age, respectively. Opposite to positive tests, our negative tests will only use invalid input data. We can then expect the form validation to always fail.

First, let’s test invalid names.

Similar to our the positive property test, we define a generator genForm (➊). Note that we use invalidName instead of validName.

Again, we pattern match on the result of applying validateSignup (➋). In this case we expect failure. Both NameTooLong and NameTooShort are expected failures. If we get anything else, the test fails (➌).

The test for invalid age is similar, expect we use the invalidAge generator, and expect only InvalidAge validation failures:

The invalidName and invalidAge generators are also defined in terms of the validation rules (eq. 1), but with ranges ensuring no overlap with valid data:

Let’s run our new property tests:

λ> check prop_invalid_name_fails
  ✓ <interactive> passed 100 tests.

λ> check prop_invalid_age_fails
  ✓ <interactive> passed 100 tests.

All good? Maybe not. The astute reader might have noticed a problem with one of our generators. We’ll get back to that later.

Accumulating All Failures

When validating the form data, we want all failures returned to the user posting the form, rather than returning only one at a time. The Validation type accumulates failures when combined with Applicative, which is exactly what we want. Yet, while the hard work is handled by Validation, we still need to test that we’re correctly combining validations in validateSignup.

We define a property test generating form data, where all fields are invalid (➊). It expects the form validation to fail, returning two failures (➋).

This property is weak. It states nothing about which failures should be returned. We could assert that the validation failures are equal to some expected list. But how do we know if the name is too long or too short? I’m sure you’d be less thrilled if we replicated all of the validation logic in this test.

Let’s define a slightly stronger property. We pattern match, extract the two failures (➊), and check that they’re not equal (➋).

We’re still not being specific about which failures should be returned. But unlike prop_two_failures_are_returned, this property at least makes sure there are no duplicate failures.

The Value of a Property

Is there a faulty behaviour that would slip past prop_two_different_failures_are_returned? Sure. The implementation could have a typo or copy-paste error, and always return NameTooLong failures, even if the name is too short. Does this mean our property is bad? Broken? Useless?

In itself, this property doesn’t give us strong confidence in the correctness of validateSignup. In conjuction with our other properties, however, it provides value. Together they make up a stronger test suite.

Let’s look at it in another way. What are the benefits of weaker properties over stronger ones? In general, weak properties are beneficial in that they are:

1. easier to define
2. likely to catch simple mistakes early
3. less coupled to the SUT

A small investment in a set of weak property tests might catch a lot of mistakes. While they won’t precisely specify your system and catch the trickiest of edge cases, their power-to-weight ratio is compelling. Moreover, a set of weak properties is better than no properties at all. If you can’t formulate the strong property you’d like, instead start simple. Lure out some bugs, and improve the strength and specificity of your properties over time.

Coming up with good properties is a skill. Practice, and you’ll get better at it.

Testing Generators

We could use this technique to check that all values generated by validAge are valid. How about invalidAge? Can we check that it generates values such that all boundaries of our validation function are hit? No, not using this technique. Testing the correctness of a generator using a property can only find problems with individual generated values. It can’t perform assertions over all generated values. In that sense, it’s a local assertion.

Instead, we’ll find the generator problem by capturing statistics on the generated values and performing global assertions. Hedgehog, and a few other PBT frameworks, can measure the occurences of user-defined labels. A label in Hedgehog is a Text value, declared with an associated condition. When Hedgehog runs the tests, it records the percentage of tests in which the condition evaluates to True. After the test run is complete, we’re presented with a listing of percentages per label.

We can even have Hedgehog fail the test unless a certain percentage is met. This way, we can declare mininum coverage requirements for the generators used in our property tests.

Adding Coverage Checks

Let’s check that we generate values covering enough cases, based on the validation rules in eq. 1 . In prop_invalid_age_fails, we use cover to ensure we generate values outside the boundaries of valid ages. 5% is enough for each, but realistically they could both get close to 50%.

Let’s run some tests again.

λ> check prop_invalid_age_fails
  ✗ <interactive> failed
    after 100 tests.
    too young 100% ████████████████████ ✓ 5%
  ⚠ too old     0% ···················· ✗ 5%

       ┏━━ test/Validation/V1Test.hs ━━━
    63 ┃ prop_invalid_age_fails = property $ do
    64 ┃   let genForm = SignupForm <$> validName <*> invalidAge
    65 ┃   form <- forAll genForm
    66 ┃   cover 5 "too young" (formAge form <= 17)
    67 ┃   cover 5 "too old"   (formAge form >= 151)
       ┃   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
       ┃   │ Failed (0% coverage)
    68 ┃   case validateSignup form of
    69 ┃     Failure (InvalidAge{} :| []) -> pure ()
    70 ┃     other                        -> do
    71 ┃       annotateShow other
    72 ┃       failure

    Insufficient coverage.

100% too young and 0% too old. The invalidAge generator is clearly not good enough. Let’s have a look at its definition again:

We’re only generating invalid ages between the minimum bound of Int and 17. Let’s fix that, by using Gen.choice and another generator for ages greater than 150:

Running tests again, the coverage check stops complaining. But there’s another problem:

λ> check prop_invalid_age_fails
  ✗ <interactive> failed at test/Validation/V1Test.hs:75:7
    after 3 tests and 2 shrinks.
    too young 67% █████████████▎······ ✓ 5%
  ⚠ too old    0% ···················· ✗ 5%

       ┏━━ test/Validation/V1Test.hs ━━━
    66 ┃ prop_invalid_age_fails = property $ do
    67 ┃   let genForm = SignupForm <$> validName <*> invalidAge
    68 ┃   form <- forAll genForm
       ┃   │ SignupForm { formName = "a" , formAge = 151 }
    69 ┃   cover 5 "too young" (formAge form <= 17)
    70 ┃   cover 5 "too old"   (formAge form >= 151)
    71 ┃   case validateSignup form of
    72 ┃     Failure (InvalidAge{} :| []) -> pure ()
    73 ┃     other                        -> do
    74 ┃       annotateShow other
       ┃       │ Success Signup { name = "a" , age = 151 }
    75 ┃       failure
       ┃       ^^^^^^^

OK, we have an actual bug. When the age is 151 or greater, the form is deemed valid. It should cause a validation failure. Looking closer at the implementation, we see that a pattern guard is missing the upper bound check:

If we change it to age' >= 18 && age' <= 150, and rerun the tests, they pass.

λ> check prop_invalid_age_fails
  ✓ <interactive> passed 100 tests.
    too young 53% ██████████▌········· ✓ 5%
    too old   47% █████████▍·········· ✓ 5%

We’ve fixed the bug.

Measuring and declaring requirements on coverage is a powerful tool in Hedgehog. It gives us visibility into the generative tests we run, making it practical to debug generators. It ensures our tests meet our coverage requirements, even as implementation and tests evolve over time.

From Ages to Birth Dates

So far, our efforts have been successful. We’ve fixed real issues in both implementation and tests. Management is pleased. They’re now asking us to modify the signup system, and use our testing skills to ensure quality remains high.

Instead of entering their age, users will enter their birth date. Let’s suppose this information is needed for something important, like sending out birthday gifts. The form validation function must be modified to check, based on the supplied birth date date, if the user signing up is old enough.

First, we import the Calendar module from the time package:

Next, we modify the SignupForm data type to carry a formBirthDate of type Date, rather than an Int.

And we make the corresponding change to the Signup data type:

We’ve also been requested to improve the validation errors. Instead of just InvalidAge, we define three constructors for various invalid birthdates:

Finally, we need to modify the validateSignup function. Here, we’re faced with an important question. How should the validation function obtain today’s date?

Keeping Things Deterministic

We could make validateSignup a non-deterministic action, which in Haskell would have the following type signature:

Note the use of IO. It means we could retrieve the current time from the system clock, and extract the Day value representing today’s date. But this approach has severe drawbacks.

If validateSignup uses IO to retrieve the current date, we can’t test it with other dates. What it there’s a bug that causes validation to behave incorrectly only on a particular date? We’d have to run the tests on that specific date to trigger it. If we introduce a bug, we want to know about it immediately. Not weeks, months, or even years after the bug was introduced. Furthermore, if we find such a bug with our tests, we can’t easily reproduce it on another date. We’d have to rewrite the implementation code to trigger the bug again.

Instead of using IO, we’ll use a simply technique for keeping our function pure: take all the information the function needs as arguments. In the case of validateSignup, we’ll pass today’s date as the first argument:

Again, let’s not worry about the implementation just yet. We’ll focus on the tests.

Generating Dates

In order to test the new validateSignup implementation, we need to generate Day values. We’re going to use a few functions from a separate module called Data.Time.Gen, previously written by some brilliant developer in our team. Let’s look at their type signatures. The implementations are not very interesting.

The generator, day, generates a day within the given range:

A day range is constructed with linearDay:

Alternatively, we might use exponentialDay:

The linearDay and exponentialDay range functions are analoguous to Hedgehog’s linear and exponential ranges for integral numbers.

To use the generator functions from Data.Time.Gen, we first add an import, qualified as Time:

Next, we define a generator anyDay:

The date range $[\text{1900-01-01}, \text{2100-12-31}]$ is arbitrary. We could pick any centuries we like, provided the time package supports the range. But why not make it somewhat realistic?

Rewriting Existing Properties

Now, it’s time to rewrite our existing property tests. Let’s begin with the one testing that validating a form with all valid data succeeds:

A few new things are going on here. We’re generating a date representing today (➊), and generating a form with a birth date based on today’s date (➋). Generating today’s date, we’re effectively time travelling and running the form validation on that date. This means our validBirthDate generator must know which date is today, in order to pick a valid birth date. We pass today’s date as a parameter, and generate a date within the range of 18 to 150 years earlier:

We define the helper function yearsBefore in the test suite. It offsets a date backwards in time by a given number of years:

The Data.Time.Calendar module exports the addGregorianYearsClip function. It adds a number of years, clipping February 29th (leap days) to February 28th where necessary.

Let’s run tests:

λ> check prop_valid_signup_form_succeeds
  ✓ <interactive> passed 100 tests.

Let’s move on to the next property, checking that invalid birth dates do not pass validation. Here, we use the same pattern as before, generating today’s date, but use invalidBirthDate instead:

Notice that we’ve also adjusted the coverage checks. There’s a new label, “not born yet,” for birth dates in the future. Running tests, we see the label in action:

λ> check prop_invalid_age_fails
  ✓ <interactive> passed 100 tests.
    not yet born 18% ███▌················ ✓ 5%
    too young    54% ██████████▊········· ✓ 5%
    too old      46% █████████▏·········· ✓ 5%

Good coverage, all tests passing. We’re not quite done, though. There’s a particular set of dates that we should be sure to cover: “today” dates and birth dates that are close to, or exactly, 18 years apart.

Within our current property test for invalid ages, we’re only sure that generated birth dates include at least 5% too old, and at least 5% too young. We don’t know how far away from the “18 years” validation boundary they are.

We could tweak our existing generators to produce values close to that boundary. Given a date $T$, exactly 18 years before today’s date, then:

• invalidBirthDate would need to produce birth dates just after but not equal to $T$
• validBirthDate would need to produce birth dates just before or equal to $T$

There’s another option, though. Instead of defining separate properties for valid and invalid ages, we’ll use a single property for all cases. This way, we only need a single generator.

A Single Validation Property

There are a few new things going on here:

1. Instead of generating exclusively invalid or valid birth dates, we’re now generating any birth date based on today’s date
2. The boolean expressions are used both in coverage checks and in asserting, so we separate them in a let binding
3. We add three new labels for the valid cases
4. Finally, we assert on both valid and invalid cases, based on the same expressions used in coverage checks

Note that our assertions are more specific than in prop_invalid_age_fails. The failure cases only pass if the corresponding label expressions are true. The oldEnough case covers all valid birth dates. Any result other than the four expected cases is considered incorrect.

The anyBirthDate generator is based on today’s date:

We defines helper functions (➊) for generating dates in the past, in the future, and close to 18 years ago. Using those helper functions, we combine four generators, with different date ranges, using a Gen.frequency distribution (➋). The weights we use are selected to give us a good coverage.

Let’s run some tests:

λ> check prop_validates_age
  ✓ <interactive> passed 100 tests.
    too young    62% ████████████▍······· ✓ 10%
    not yet born 20% ████················ ✓  1%
    too old       4% ▊··················· ✓  1%
    old enough   38% ███████▌············ ✓ 20%
    exactly 18   16% ███▏················ ✓  1%
    close to 18  21% ████▏··············· ✓  5%

Looks good! We’ve gone from testing positive and negative cases separately, to instead have a single property covering all cases, based on a single generator. It’s now easier to generate values close to the valid/invalid boundary of our SUT, i.e. around 18 years from today’s date.

February 29th

For the fun of it, let’s run some more tests. We’ll crank it up to 20000.

λ> check (withTests 20000 prop_validates_age)
  ✗ <interactive> failed at test/Validation/V3Test.hs:141:64
    after 17000 tests and 25 shrinks.
    too young    60% ████████████········ ✓ 10%
    not yet born 20% ███▉················ ✓  1%
    too old       9% █▉·················· ✓  1%
    old enough   40% ███████▉············ ✓ 20%
    exactly 18   14% ██▊················· ✓  1%
    close to 18  21% ████▎··············· ✓  5%

        ┏━━ test/Validation/V3Test.hs ━━━
    114 ┃ prop_validates_age = property $ do
    115 ┃   today <- forAll anyDay
        ┃   │ 1956 - 02 - 29
    116 ┃   form  <- forAll (SignupForm <$> validName <*> anyBirthDate today) ➊
        ┃   │ SignupForm { formName = "aa" , formBirthDate = 1938 - 03 - 01 }
    117 ┃
    118 ┃   let tooYoung        = formBirthDate form > 18 `yearsBefore` today ➋
    119 ┃       notYetBorn      = formBirthDate form > today
    120 ┃       tooOld          = formBirthDate form < 150 `yearsBefore` today
    121 ┃       oldEnough       = formBirthDate form <= 18 `yearsBefore` today
    122 ┃       exactlyEighteen = formBirthDate form == 18 `yearsBefore` today
    123 ┃       closeToEighteen =
    124 ┃         let diff' =
    125 ┃                 diffDays (formBirthDate form) (18 `yearsBefore` today)
    126 ┃         in  abs diff' `elem` [0 .. 2]
    127 ┃
    128 ┃   cover 10 "too young"    tooYoung
    129 ┃   cover 1  "not yet born" notYetBorn
    130 ┃   cover 1  "too old"      tooOld
    131 ┃
    132 ┃   cover 20 "old enough"   oldEnough ➌
    133 ┃   cover 1  "exactly 18"   exactlyEighteen
    134 ┃   cover 5  "close to 18"  closeToEighteen
    135 ┃
    136 ┃   case validateSignup today form of ➍
    137 ┃     Failure (NotYetBorn{} :| []) | notYetBorn -> pure ()
    138 ┃     Failure (TooYoung{} :| []) | tooYoung -> pure ()
    139 ┃     Failure (TooOld{} :| []) | tooOld -> pure ()
    140 ┃     Success{} | oldEnough             -> pure ()
    141 ┃     other                             -> annotateShow other >> failure
        ┃     │ Success Signup { name = "aa" , birthDate = 1938 - 03 - 01 }
        ┃     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Failure! Chaos! What’s going on here? Let’s examine the test case:

• Today’s date is 1956-02-29
• The birth date is 1938-03-01
• The validation function considers this valid (it returns a Success value)
• The test does considers this invalid (oldEnough is False)

Arbitrary values for today’s date and the birth date are drawn most frequently (➊), with a weight of 5. Next, with weights of 2, are generators for cases close to the boundaries of the validation function (➋). Finally, with weights of 1, are generators for special cases involving leap days as today’s date (➌) and leap days as birth date (➍).

Note that these generators return pairs of dates. For most of these generators, there’s a strong relation between today’s date and the birth date. For example, we can’t first generate any today’s date, pass that into a generator function, and expect it to always generate a leap day that occured 18 years ago. Such a generator would have to first generate the leap day and then today’s date.

Let’s define the generators. The first one, anyDayAndBirthDate, picks any today’s date within a wide date range. It also picks a birth date from an even wider date range, resulting in some future birth dates and some ages above 150.

Writing automated tests with a hard-coded year 2020 might scare you. Won’t these tests fail when run in the future? No, not these tests. Remember, the validation function is deterministic. We control today’s date. The actual date on which we run these tests doesn’t matter.

Similar to the previous generator is anyDayAndBirthDateAroundYearsAgo. First, it generates any date as today’s date (➊). Next, it generates an arbitrary date approximately some number of years ago (➋), where the number of years is an argument of the generator.

The addingApproxYearsAgo generator adds a number of years to a date, and offsets it between two days back and two days forward in time.

The last two generators used in our frequency distribution cover leap day edge cases. First, let’s define the leapDayAndBirthDateAroundYearsAgo generator. It generates a leap day used as today’s date, and a birth date close to the given number of years ago.

The leapDay generator uses mod to only generate years divisible by 4 and constructs dates on February 29th. That alone isn’t enough to only generate valid leap days, though. Years divisible by 100 but not by 400 are not leap years. To keep the generator simple, we discard those years using the already existing isLeapDay predicate as a filter.

In general, we should be careful about discarding generated values using filter. If we discard too much, Hedgehog gives up and complains loudly. In this particular case, discarding a few generated dates is fine. Depending on the year range we pass it, we might not discard any date.

Finally, we define the commonDayAndLeaplingBirthDateAroundYearsAgo generator. It first generates a leap day used as the birth date, and then a today’s date approximately the given number of years after the birth date.

That’s it for the generators. Now, how do we know that we’re covering the edge cases well enough? With coverage checks!

We add new checks to the property test, checking that we hit both leap day and regular day cases around the 18th birthday (➊) and the 150th birthday (➋). Notice that we had similar checks before, but we were not discriminating between leap days and common days.

The use of addGregorianYearsRollOver together with adding a negative number of years is the problem, rolling over to March 1st instead of clipping to February 28th. Instead, we should use addGregorianYearsClip:

Running 100 tests again, we see that they all pass, and that our coverage requirements are met.

λ> Hedgehog.check prop_validates_age
  ✓ <interactive> passed 100 tests.
    too young                             17% ███▍················ ✓ 10%
    not yet born                           7% █▍·················· ✓  1%
    too old                               19% ███▊················ ✓  1%
    old enough                            83% ████████████████▌··· ✓ 20%
    close to 18, validated on common day  30% ██████·············· ✓  5%
    close to 18, validated on leap day     2% ▍··················· ✓  1%
    close to 150, validated on common day 31% ██████▏············· ✓  5%
    close to 150, validated on leap day    6% █▏·················· ✓  1%
    exactly 18 today, born on common day  17% ███▍················ ✓  5%
    legally 18 today, born on leap day     5% █··················· ✓  1%

Summary

In this tutorial, we started with a simple form validation function, checking the name and age of a person signing up for an online service. We defined property tests for positive and negative tests, learned how to test generators with coverage checks, and found bugs in both the test suite and the implementation.

When requirements changed, we had to start working with dates. In order to keep the validation function deterministic, we had to pass in today’s date. This enabled us to simulate the validation running on any date, in combination with any reported birth date, and trigger bugs that could otherwise take years to find, if ever. Had we not made it deterministic, we would likely not have found the leap day bug later on.

To generate inputs that sufficiently test the validation function’s boundaries, we rewrote our separate positive and negative properties into a single property, and used coverage checks to ensure the quality of our generators. The trade-off between multiple disjoint properties and a single more complicated property is hard.

With multiple properties, for example split between positive and negative tests, both generators and assertions can be simpler and more targeted. On the other hand, you run a risk of missing certain inputs. The set of properties might not cover the entire space of inputs. Furthermore, performing coverage checks across multiple properties, using multiple targeted generators, can be problematic.

Ensuring coverage of generators in a single property is easier. You might even get away with a naive generator, depending on the system you’re testing. If not, you’ll need to combine more targeted generators, for example with weighted probabilities. The drawback of using a single property is that the assertion not only becomes more complicated, it’s also likely to mirror the implementation of the SUT. As we saw with our single property testing the validation function, the assertion duplicated the validation rules. You might be able to reuse the coverage expressions in assertions, but still, there’s a strong coupling.

The choice between single or multiple properties comes down to how you want to cover the boundaries of the SUT. Ultimately, both approaches can achieve the same coverage, in different ways. They both suffer from the classic problem of a test suite mirroring the system it’s testing.

Finally, running a larger number of tests, we found a bug related to leap days. Again, without having made the validation function deterministic, this could’ve only been found on a leap day. We further refined our generators to cover leap day cases, and found the bug reliably with as few as 100 tests. The bug was easy to find and fix when we had the inputs pointing directly towards it.

That’s it for this tutorial. Thanks for reading, and happy property testing and time travelling!