Numerous TDD experts have made the suggestion that there should only be one assertion per test:
The theory goes that each test should only test one thing, and that should be the name of the test.
I don’t do a very good job of following that rule. If I wanted to test a method that returned books for an author I would write tests like so:
1 [Test]
2 public void GetBooksForAuthorSuccessfull()
3 {
4 List<Book> books = AuthorDA.GetBooks(1);
5 Assert.IsNotNull(books, "Null value returned for books");
6 Assert.IsTrue(books.Count > 0, "No books returned");
7 Assert.AreEqual(books.Count, 3, "Incorrect number of books returned");
8 Assert.AreEqual(books[1].ID, 1, "First book returned is incorrect");
9 Assert.AreEqual(books[2].ID, 2, "Second book returned is incorrect");
10 Assert.AreEqual(books[3].ID, 3, "Third book returned is incorrect");
11 }
12
13 [Test]
14 public void GetBooksForAuthorNoBooksReturned()
15 {
16 List<Book> books = AuthorDA.GetBooks(122);
17 Assert.IsNotNull(books, "Null value returned for books");
18 Assert.AreEqual(books.Count, 0, "Books are returned when they shouldn't be");
19 }
20
21 [Test]
22 [ExpectedException(typeof(ArgumentException))]
23 public void GetBooksForAuthorBadAuthorID()
24 {
25 AuthorDA.GetBooks(-1);
26 }
I have three different tests that each test different things, but two of them contain more than one assertion and one of them contains seven assertions. If I wanted to write these tests using the one assertion per test guideline I would have this:
1 [Test]
2 public void GetBooksForAuthorSuccessfullNotNull()
3 {
4 List<Book> books = AuthorDA.GetBooks(1);
5 Assert.IsNotNull(books, "Null value returned for books");
6 }
7
8 [Test]
9 public void GetBooksForAuthorSuccessfullCountGreaterThanZero()
10 {
11 List<Book> books = AuthorDA.GetBooks(1);
12 Assert.IsTrue(books.Count > 0, "No books returned");
13 }
14
15 [Test]
16 public void GetBooksForAuthorSuccessfullCountEqualsThree()
17 {
18 List<Book> books = AuthorDA.GetBooks(1);
19 Assert.AreEqual(books.Count, 3, "Incorrect number of books returned");
20 }
21
22 [Test]
23 public void GetBooksForAuthorSuccessfullFirstBookReturnedCorrect()
24 {
25 List<Book> books = AuthorDA.GetBooks(1);
26 Assert.AreEqual(books[1].ID, 1, "First book returned is incorrect");
27 }
28
29 [Test]
30 public void GetBooksForAuthorSuccessfullSecondBookReturnedCorrect()
31 {
32 List<Book> books = AuthorDA.GetBooks(1);
33 Assert.AreEqual(books[2].ID, 2, "Second book returned is incorrect");
34 }
35
36 [Test]
37 public void GetBooksForAuthorSuccessfullThirdBookReturnedCorrect()
38 {
39 List<Book> books = AuthorDA.GetBooks(1);
40 Assert.AreEqual(books[3].ID, 3, "Third book returned is incorrect");
41 }
42
43 [Test]
44 public void GetBooksForAuthorNoBooksReturnedNotNull()
45 {
46 List<Book> books = AuthorDA.GetBooks(122);
47 Assert.AreEqual(books.Count, 0, "Books are returned when they shouldn't be");
48 }
49
50 [Test]
51 public void GetBooksForAuthorNoBooksReturnedCountGreaterThan0()
52 {
53 List<Book> books = AuthorDA.GetBooks(122);
54 Assert.AreEqual(books.Count, 0, "Books are returned when they shouldn't be");
55 }
56
57 [Test]
58 [ExpectedException(typeof(ArgumentException))]
59 public void GetBooksForAuthorBadAuthorID()
60 {
61 AuthorDA.GetBooks(-1);
62 }
I have a couple problems with this change:
- I think it’s actually harder to read since the assertions are scattered around in separate methods.
- It would increase the number of tests. On my current project we have 1800 tests, if we followed the one assertion rule we would have over 6,000 I am sure.
- If my method breaks and starts returning null then I have 8 tests failing instead of just 2, this means I have to know the dependency tree of my tests to find the real issue.
- Any code I have to write to setup the data for my test has to be duplicated 8 times. (if I move that setup data to the setup method than I am effectively limiting my fixtures to one fixture per test)
- I now have over double the amount of code. I am constantly trying to reduce the amount of code in my project, whether test or production, and anything that doubles it better add a ton of value.
The main drawback to having multiple asserts in a test is that all of the unit testing frameworks I have used fail the test on the first assert that fails. This means that if my first test fails on the third assert (line 82), the remaining assertions are never run:
76 [Test]
77 public void GetBooksForAuthorSuccessfull()
78 {
79 List<Book> books = AuthorDA.GetBooks(1);
80 Assert.IsNotNull(books, "Null value returned for books");
81 Assert.IsTrue(books.Count > 0, "No books returned");
82 Assert.AreEqual(books.Count, 3, "Incorrect number of books returned"); ‘ This Fails
83 Assert.AreEqual(books[1].ID, 1, "First book returned is incorrect"); ‘ Never Run
84 Assert.AreEqual(books[2].ID, 2, "Second book returned is incorrect"); ‘ Never Run
85 Assert.AreEqual(books[3].ID, 3, "Third book returned is incorrect"); ‘ Never Run
86 }
This is the main reason Roy gives in his MSDN article on why you should limit tests to a single assert. My idea is that we should have an attribute we could use to tell the framework that we want it to run all the asserts, something like this:
76 [MultipleAssertTest]
77 public void GetBooksForAuthorSuccessfull()
78 {
79 List<Book> books = AuthorDA.GetBooks(1);
80 Assert.IsNotNull(books, "Null value returned for books", false);
81 Assert.IsTrue(books.Count > 0, "No books returned", false);
82 Assert.AreEqual(books.Count, 3, "Incorrect number of books returned", true);
83 Assert.AreEqual(books[1].ID, 1, "First book returned is incorrect", true);
84 Assert.AreEqual(books[2].ID, 2, "Second book returned is incorrect", true);
85 Assert.AreEqual(books[3].ID, 3, "Third book returned is incorrect", true);
86 }
The new test type would allow me to add an additional parameter to all of my assertions that tells the harness whether or not it should continue with the test. If either of the first two asserts fails I want to abort and not evaluate the other assertions since they will all fail. The last four assertions are not dependent on each other so I want to continue running the rest of the assertions in my test if one of them fails.
You would then need to be able to see each of the failure in the GUI, something like a tree would work:
+ GetBooksForAuthorSuccessfull() Failed
– 2 Failures
* Incorrect number of books returned
* Third book returned is incorrect
This would give me the benefits of one assertion per test without the additional code or huge increase in the number of tests. This would be a great feature for MbUnit.
-James
