Unit Testing In Elixir
Note that all of this was written at the time of Elixir 0.12.5-dev, so things might have changed since.
Since the tutorial covers the mechanics pretty well, this post is more about my impressions of Elixir's approach to this topic.
Unit testing is built into the language and the tools, as well as the tutorial - I love this approach. When you create a new project with mix, this is the output you get:
$ mix new myproject ... Your mix project was created successfully. You can use mix to compile it, test it, and more: cd myproject mix compile mix test Run `mix help` for more information.
The test skeleton is not beginner friendly
The skeleton unit test suffers the same problems like the Django equivalent - it is aimed at those who get unit testing already. If you are new to automated testing, seeing code like
defmodule FooTest do use ExUnit.Case test "the truth" do assert(true) end end
won't really make you say "yes, this makes me understand and love testing!". Granted, the whole Elixir tutorial is aimed at people who can already program.
Simplified, yet flexible assertions
Unlike classic unit testing frameworks, which come bundled with numerous assertWhatever and assertNotWhatever etc. methods, (various overloads for each type), ExUnit mostly only uses assert and assert_raise (though there are some more - see ExUnit.Assertions). Rather, it relies on pattern matching and the assert macro being smart enough to figure out how to provide a good (enough) failure message - in this aspect, it reminds me of py.test
I'm curious to see whether it will stay this way - custom assertions are superb for writing DSL-like code that experts can read (e.g.: Hamcrest), and I really like the protocol-based extensibility model used in Elixir elsewhere.
Constraint of the failure messages
Your opinion of this feature might differ from mine, but it's worth pointing this out. While the assertions are pretty flexible, the actual error message will become the values on the left- and right hand side of the pattern matching. This takes away the information of exactly which method was called with what parameters, which I have grown to rely on in Python. Consider the following tests and the resulting failure messages - which one is more helpful?
test "survival - a living cell with 2 or 3 neighbours survives" do assert Gofl.Rules.next_state(:alive, 2) == :alive assert Gofl.Rules.next_state(:alive, 3) == :alive assert Gofl.Rules.next_state(:dead, 3) == :alive assert Gofl.Rules.next_state(:dead, 2) == :dead end
1) test survival - a living cell with 2 or 3 neighbours survives (GoflRulesTest) ** (ExUnit.ExpectationError) expected: :alive to be equal to (==): :dead at unit_testing_with_elixir_gofl.exs:18 Finished in 0.05 seconds (0.04s on load, 0.01s on tests) 1 tests, 1 failures
def test_survival_a_living_cell_with_2_or_3_neighbours_survives(self): self.assertEquals(ALIVE, cell_next_state(curr_state=ALIVE, live_neighbour_cnt=2)) self.assertEquals(ALIVE, cell_next_state(curr_state=ALIVE, live_neighbour_cnt=3)) self.assertEquals(ALIVE, cell_next_state(curr_state=DEAD, live_neighbour_cnt=3)) self.assertEquals(DEAD, cell_next_state(curr_state=DEAD, live_neighbour_cnt=2))
F ====================================================================== FAIL: test_survival_a_living_cell_with_2_or_3_neighbours_survives (__main__.GoflRulesTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "unit_testing_with_elixir_gofl.py", line 22, in test_survival_a_living_cell_with_2_or_3_neighbours_survives self.assertEquals(DEAD, cell_next_state(curr_state=DEAD, live_neighbour_cnt=2)) AssertionError: False != True ---------------------------------------------------------------------- Ran 1 test in 0.000s FAILED (failures=1)
Granted, Python is unique in this regard, IIRC I wouldn't get such useful stacktraces in C# or Java either, but since I've been mostly working in Python lately, it certainly was a surprise - not necessarily for discouraging multiple assertions per test (it's a valid approach), but more the hiding of the call site information.
However, since assert itself is a macro, its technically possible to change the assertion - added to my list of things to try :)
Test isolation is almost always hard, and these issues tend to only come out from hiding once tests need to run in parallel for speed - good things never come alone :) So having concurrent test running built in from the start is a great way to have the one big (future) problem broken into many minor inconveniences.
I also like that instead of a test runner command line switch, it is declared on the TestCase level whether it's OK to run it in parallel - giving finer control to the test author.
defmodule GoflRulesTest do use ExUnit.Case, async: true
I just wish it was true by default!
Cool things I would like to explore further
- This short exploration hasn't allowed me figure what kind of
differences there are between testing functional- and object oriented programs. My gut feeling is there shouldn't be many: black box, transformation (state) based should be the same. Interaction testing (mocking) seems an odd fit, however it might just turn out to be as easy as passing in a "mock" function as an argument.
- Property-based testing has been on my radar for a while to be
tried - and its use seems to be more common in functional programming, and it's appropriate I should try it in a context where it's set up to succeed. This is when I'm thankful for Elixir being built/run on top of a mature platform - otherwise there wouldn't yet be an available library.