Given failures not aborting tests

May 2, 2011 at 4:06 PM

It appears that Given() is catching exceptions thrown by the associated action, then continuing on with When() and Then(). If either Given or When fail, shouldn't the test abort rather than continuing on? If the pre-conditions aren't met, for example, then it doesn't seem to make sense to continue with the test.

Coordinator
May 2, 2011 at 7:39 PM

Hi Jestr

That's by design. Sometimes it's interesting to know if the Then() still passes even though one of the precondition steps fails (usually it's a bad code smell - meaning that your assertions aren't very good).

The test itself will still fail, but each step will have been run

Hope that helps - Rob

May 2, 2011 at 7:47 PM

It helps. I'll have to think about whether I entirely agree, and whether I think a "abort if prior step fails" mode would be useful.

May 3, 2011 at 5:57 PM
robfe wrote:

Hi Jestr

That's by design. Sometimes it's interesting to know if the Then() still passes even though one of the precondition steps fails (usually it's a bad code smell - meaning that your assertions aren't very good).

The test itself will still fail, but each step will have been run

Hope that helps - Rob

OK I'm going to go ahead and disagree with you. I am relying on my Given method to set up state that my Whens/Thens rely on. If the Given fails, the Whens/Thens crash and burn. Yes, I could program my Whens/Thens defensively, but it seems that part of the point of Given is to set up a reliable initial state. If the givens aren't given, then the test has no meaning. So you may be right from a test-debugging perspective (did I write a good test), but I think from a test-execution perspective, if Given doesn't hold then all bets are off and there's no point going further.

 

Coordinator
May 3, 2011 at 8:45 PM

Why would you bother programming your whens and thens defensively though? If they fail, how is that worse than being skipped?

You've convinced me that in your case it's *not good* to keep running, but i don't see why it's *bad*.

May 3, 2011 at 8:55 PM
robfe wrote:

Why would you bother programming your whens and thens defensively though? If they fail, how is that worse than being skipped?

You've convinced me that in your case it's *not good* to keep running, but i don't see why it's *bad*.


In my particular case, I need to integrate my tests with an external test-results reporting system. If I let the Whens/Thens run after Given fails, the Thens make assert calls, which fail, and which report Failure to the external system. According to that system's rules, though, I should be reporting Inconclusive. Inconclusive reports happen automatically if I don't make the assert calls (i.e., don't run the Thens).

I am looking at a hack where Given sets an error flag, then all the Whens and Thens check that flag before running. Major ugh. Thinking about using extension methods to hook directly into the StoryQ fluent interface, but haven't drunk enough coffee yet :-).

Coordinator
May 3, 2011 at 9:06 PM

Could you just add a TakeUntil to the LINQ statement in

void IStepContainer.Execute(params IRenderer[] renderers)

of http://storyq.codeplex.com/SourceControl/changeset/view/3ddb165d8ded#src%2fStoryQ%2fInfrastructure%2fFragmentBase.cs ?

 

Personally, I forward the results from storyq onward by parsing the generated XML report and pushing that information to an API (such as JIRA or TFS), rather than having some code in the asserts do this. I can imagine that this isn't always applicable though.

Aug 24, 2011 at 8:07 PM

Finally getting around to trying to implement this. Have some questions. I'm definitely a LINQ novice, so appreciate your patience:

  1. I don't see a TakeUntil method in IEnumerable or anywhere else. Do I understand correctly that I have to implement it myself? Could I accomplish the same thing using TakeWhile?
  2. I'm not sure what TakeUntil or TakeWhile should be testing
Nov 3, 2011 at 8:06 PM

I created an issue for this here: http://storyq.codeplex.com/workitem/16120

This behavior makes it very difficult to implement tests (deeper than just writing non-implemented exceptions for every test method) before writing the actual implementation.  This problem is removing much of the possible usefulness of this framework for me as it makes it impossible to tell the difference between when one of us actually breaks a test with a change and when the build is failing because the test was simply written before the implementation.

Coordinator
Nov 4, 2011 at 11:04 AM

So what exactly are you looking for, bjmarte?

As soon as a test step pends (throws a not implemented exception), the following test steps are skipped?

 

I'm more open to this than having a failing test step skip the following steps...

Nov 4, 2011 at 2:38 PM

Yes, my preferred behavior is exactly as you described.  Once a NotImplementedException is thrown then test execution stops and the result is Pending.  

I would also be fine with a test giving a fail result if an assert fails or a different exception is thrown before any NotImplementedExceptions are thrown.

Nov 10, 2011 at 8:45 PM
robfe wrote:

Sometimes it's interesting to know if the Then() still passes even though one of the precondition steps fails (usually it's a bad code smell - meaning that your assertions aren't very good).

I was on the verge of changing my mind and agreeing with you, but decided not to. In principle your statement is accurate. In practice, though, if I look at my test output report in the morning and say "Oh goodie, all my tests passed", I will miss the case where a test passed that should have failed.

Nov 10, 2011 at 8:48 PM

Just to be clear... For what I am asking for, you will not miss a case that should have failed.  It would have shown as pending rather than passed or failed.

Nov 10, 2011 at 9:14 PM
bjmarte wrote:

Just to be clear... For what I am asking for, you will not miss a case that should have failed.  


Yes, we may be mixing different cases in the same thread. I responded as I did because you made me think about the topic again.