Elixir Community Tools: StreamData

In his ElixirConf US 2018 keynote, José Valim announced that what the core team had set out to do with Elixir was now present in the language, and that new projects, ideas, and developments belong to the ecosystem, and should be explored by the community. In this blog series, we will highlight some of the interesting packages that have emerged in the Elixir ecosystem, describe why they exist and how they can help us build Elixir applications.

It is important to stress that a mention of a package is not necessarily an approval from Erlang Solutions. There might be occasions where a given package might not be the right choice for your application, but we do provide code review services and are able to augment your team with highly skilled consultants, so do get in touch.

With that out of the way; In this first instalment, I will take a look at the StreamData project started by Andrea Leopardi. StreamData will help us generate random data that we can use to test our applications.

The purpose of StreamData

StreamData is a framework for generating pseudo-random data based on specifications, which are defined using “generators” that can be combined together to create more complex generators. Generators are created using the functions found in the StreamData toolbox.

That truly is a mouthful, and one might ask: Why would I need randomly generated data for test purposes? After all, I am interested in testing based on a known input for my application, or algorithm, to produce an expected output! To answer that we have to look at the second part that StreamData provides us, ExUnitProperties, which gives us ExUnit helpers for defining test cases that fall within the category of “property based testing.”

So, we have a framework for defining generators that will produce random data based on specifications, and helpers that enable us to define property based tests in ExUnit.

Let us explore that; first we will explore data generation by defining a data generator, and then we will use that data generator to define a test case.

Generating data

StreamData is available on Hex, and as I write this, the current version is “0.5.0.” Let us create a “Playground” application using mix, and add {:stream_data, "~> 0.5.0"} as a dependency. Once we run our trusty mix deps.get it should be available to us.

Running our Playground project in an interactive Elixir shell we will see that typing StreamData. (dot) and pressing tab will produce a long list of functions, and some of them have names such as integer, float, string, etc, mapping to data types we are familiar with from Elixir.

So let us try to generate an integer:

iex> StreamData.integer()  
#StreamData<45.14801001/2 in StreamData.integer/0>  

Hmm, not the output we expected. Instead of getting an integer, we got some kind of data structure; This is a generator! The generators in StreamData implement the Enumerable protocol, meaning we can use functions from the Enum-module to produce our data.

iex> StreamData.integer() |> Enum.take(5)  
[-1, -2, -2, -4, 1]  

As the name “StreamData” suggests we can also use functions found in the Elixir Stream module. Applying a Stream.map to our pipeline we could, for instance, get rid of all the negative numbers:

iex> StreamData.integer() |> Stream.map(&abs(&1)) |> Enum.take(5)  
[1, 2, 2, 3, 3]  

abs/1 will take the arithmetical absolute value of number; a fancy way of saying it will ignore the minus if the number is negative.

NB: If we used the map function from the Enum module our interactive Elixir shell would hang. This is because it will work on the entire list, eagerly grabbing all the elements from the stream, which is infinite! The map function found in Stream will work on one element at a time, so this combined with the Enum.take function, which takes a finite number of elements resulting in a finite list of elements.

One interesting thing to notice is that the values produced get more “crazy” and “extreme” the more we ask from our generator. If we experiment with dropping a lot of elements we can see this in action:

iex> StreamData.integer() |> Stream.drop(100_000) |> Enum.take(1)  
[-52]    

This is true for all the data types StreamData can produce;

iex> StreamData.float() |> Stream.drop(1_000) |> Enum.take(1)  
[-1.243273519341557e30]  
iex> StreamData.string() |> Stream.drop(1_000) |> Enum.take(1)  
["9DE^nC*:?k4S~\\7xWdW lt`Y_6HC]a>R@pkRXX96iw7K~~*Z\\1"]    

NB: I have reduced the number of elements we drop for these examples, because it is a bit more computation intensive to produce floats and strings than it is to produce an integer. Even though we are dropping the values they are still computed.

The fact that the values start out small and become bigger, and more “crazy”, is very important for when we get into the property based testing aspect of StreamData. But first, let us explore composing generators together.

Elixir has some compound data types such as lists, maps, and tuples. They can all contain keys and values consisting of other terms, so being able to generate these data types and specifying what type their keys and what type the values will have will be very helpful; to do this in StreamData we have some functions that will work together with other StreamData functions to “compose” the desired output. For instance, we got the StreamData.list_of-function, that will take a generator and generate lists containing elements produced by the given generator.

iex> StreamData.list_of(StreamData.integer()) |> Enum.take(5)  
[[1], [], [2, 2, 1], [-1], []]  

As we can see this will produce lists of integers of varying length, and sometimes it will be empty. The list_of function takes the integer function and composes it into a generator that will produce exactly that, a list of integers. We could compose that with the nonempty function and get a new generator that will never produce an empty list. Very handy in some situations!

iex> list_of_integers = StreamData.list_of(StreamData.integer())
#StreamData<54.14801001/2 in StreamData.list_of/1>
iex> StreamData.nonempty(list_of_integers) |> Enum.take(5)
[[0], [-1], [3, -1], [2], [-4, -3, 2, -3, 0]]  

Again; notice that the output gets “crazier” the more data we request from the stream, and in my “crazy” terminology: If you are a list, to get crazy, means that you will grow in size. StreamData makes it possible to adjust how much a list should grow by passing in options, but that is outside the scope of this article. Notice though that the inside of the list generators gets “crazy” as well. It is craziness all the way down.

Property based testing

Let us put all this randomness to good use. So far, we have a generator that can produce random lists containing random integers. We know the list will start out with small values, and get a greater degree of complexity the more values we request from the generator. These are properties that come in very handy when we get into property based testing.

How does property-based testing differ from regular unit testing?

In unit testing, we call into our implementation with some input values that we come up with, and we will assert on the resulting values. In the unit test approach, we use a known input and test that against the expected output, which is also known. This is a good approach to testing, and it will get us so far, but it requires us to anticipate edge cases, and it can become very repetitive to set up assertions. In property based testing, we look for a “property that should hold true” for the thing we want to test. This sounds scary, and it really is, but it is very powerful and can find edge cases that would be very hard to anticipate. Let us take a list as an example, and demonstrate how to test that a reverse list function works as expected. The property of a list being reversed is…that the order of the elements in the list is in the reverse order of its input. So, if we give an input to the reverse list function and test if the result is different from the input like this:

defmodule PlaygroundTest do
  use ExUnit.Case
  use ExUnitProperties

  describe "list" do
    property "should not be the same when reversed" do
      check all input_list <- list_of(integer()) do
        refute input_list == Enum.reverse(input_list)
      end
    end
  end
end  

Will show us that our assumption and property had a minor flaw. When run, it will produce the following output:

 1) property list should not be the same when reversed (PlaygroundTest)
     test/playground_test.exs:8
     Failed with generated values (after 0 successful runs):

         * Clause:    input_list <- list_of(integer())
           Generated: []

     Refute with == failed, both sides are exactly equal
     code: refute input_list == Enum.reverse(input_list)
     left: []
     stacktrace:
       test/playground_test.exs:10: anonymous fn/2 in PlaygroundTest."property list should not be the same when reversed"/1
       (stream_data 0.5.0) lib/stream_data.ex:2102: StreamData.check_all/7
       test/playground_test.exs:9: (test)

Finished in 0.04 seconds
1 property, 1 failure

Randomized with seed 205049

Studying this we can see that we failed spectacularly at our first run. Our generator produced an empty list, and of course, the result of reversing an empty list is an empty list. We need to adjust our test model. What happens if we add the StreamData.nonempty/1 to the data generator? Well, we will learn that it sometimes generates lists containing the same value; reversing the list [0, 0] will result in [0, 0]. We need to think deeper.

List reversal

Reversing a list is a reversible operation. So if we reverse a list, and then reverse it again, then we should end up with the initial list. Let us try that instead:

defmodule PlaygroundTest do
  use ExUnit.Case
  use ExUnitProperties

  describe "list" do
    property "reversing twice should result in the initial input" do
      check all input_list <- list_of(integer()) do
        assert input_list == input_list |> Enum.reverse() |> Enum.reverse()
      end
    end
  end
end

Presto. This should result in a successful test run. The observant reader would notice that the identity function (a function that simply returns its input, fn (x) -> x end) would pass this property as well!
We have a couple of options for tackling this problem, where the easiest might be to use the StreamData.uniq_list_of generator instead of the StreamData.list_of, as it would ensure the elements in the lists will never repeat. As a result, reversing the list should result in a list that is different from the input. Another option could be to mix and match property based testing, and regular unit testing, and write a test that makes the assertion that a list with two different elements, :foo and :bar, indeed becomes :bar and :foo when reversed—the two approaches to testing lives perfectly well side-by-side, and accommodate each other quite well. We encourage you to copy the example into a project and make it more robust, and please share your findings with the community.

Additional things to note

A couple of things to notice. Once we have used the ExUnitProperties module (provided by StreamData) a “property”-macro will be available to us in the test DSL (domain specific language). It is similar to the test-macro, but it knows about the check macro, that we in this case instructs to check “all” (meaning a lot of different input lists), and all this will be tested in the assert (or refute) in the check-body. Also, in the examples we use list_of and integer without specifying the StreamData module; these have been imported when we used ExUnitProperties.

There is one last thing I would like to show you. I have been talking a lot about data getting crazier the more we ask from a given data generator. Would that make our test output “crazy” if our property test finds a failure deep within the run? Not necessarily! StreamData supports “shrinking,” which means that once it has found a failure it will attempt to “shrink” the input that made the test fail, so it can present the minimal input needed to break the property. Let me demonstrate with a crazy example, where we use our own reverse implementation that has a weird bug which ignores instances of the number 15:

defmodule PlaygroundTest do
  use ExUnit.Case
  use ExUnitProperties

  # An intentionally broken reverse function
  defp broken_reverse(list) when is_list(list) do
    # kick off the recursion
    broken_reverse(list, [])
  end

  # Base case
  defp broken_reverse([], acc), do: acc

  # Recursive cases:

  # BUG: For some reason we throw away the value 15 when we see it!
  # Thanks to jlouis for providing this very random integer:
  # - https://twitter.com/jlouis666/status/1260563927534112768
  defp broken_reverse([15 | remaining], acc) do
    broken_reverse(remaining, acc)
  end

  # put the head into the accumulator and recurse
  defp broken_reverse([head | remaining], acc) do
    broken_reverse(remaining, [head | acc])
  end

  describe "crazy list" do
    property "input should be the same if reversed twice" do
      check all input_list <- list_of(integer()) do
        assert input_list == input_list |> broken_reverse() |> broken_reverse()
      end
    end
  end
end

Which will result in this helpful test report:

 1) property crazy list reversing twice should result in the initial input (PlaygroundTest)
     test/playground_test.exs:29
     Failed with generated values (after 20 successful runs):

         * Clause:    input_list <- list_of(integer())
           Generated: [15]

     Assertion with == failed
     code:  assert input_list == input_list |> broken_reverse() |> broken_reverse()
     left:  [15]
     right: []
     stacktrace:
       test/playground_test.exs:31: anonymous fn/2 in PlaygroundTest."property crazy list reversing twice should result in the initial input"/1
       (stream_data 0.5.0) lib/stream_data.ex:2148: StreamData.shrink_failure/6
       (stream_data 0.5.0) lib/stream_data.ex:2108: StreamData.check_all/7
       test/playground_test.exs:30: (test)

Finished in 0.05 seconds
1 property, 1 failure

In this case it had 20 successful runs, and on the twenty-first run it produced a list that resulted in an error. It shrunk the input until it found the smallest possible input that can reproduce the error, a list containing the number 15, [15]. This makes it much easier to find the bug and correct it. Feel free to play around with it—it really is magic—and if you need random values a bunch of helpful people have provided some here:
https://twitter.com/gausby/status/1260561299181838336

Notice that a value which is too high, such as 115, in our “bug” might go unnoticed. This is because the generator will not get to produce a value anywhere near that range before it decides it has generated enough tests to be confident that the property holds true. Luckily we can adjust the size of the test space, but keep in mind that a bigger test space will result in a longer run time. A good way to get around that is to have a small test space on the local test environment, such that tests run very fast while we develop, and go a bit crazier on a continuous integration server, where we can allow the test suite to run a bit longer and be a bit more thorough. How to set this all is described in the StreamData documentation.

Thoughtfulness behind testing

As we see, creating property-based testing requires some deep thought, and it is a skill that needs to be learned, but it is a very strong testing strategy when applied correctly. StreamData is a framework that integrates very well with the Elixir ecosystem but is only one of the many options out there. Another option is PropEr (and PropCheck, which provides a wrapper for Elixir).

The creator of StreamData, Andrea Leopardi gave the talk Property-Based testing is a mindset at ElixirConf EU 2018, and this is a very good introduction to both topics. If property-based testing has caught your interest we would like to suggest the book Property-Based Testing with PropEr, Erlang, and Elixir by Fred Hebert from the Pragmatic Bookshelf.

Find out more about our Elixir programming language projects.