This project is read-only.

Data Driven stories / scenarios

Nov 14, 2010 at 6:13 PM

I am considering a new StoryQ feature and would like some feedback on the concepts.

Give a data driven test that looks like this in nunit:

[TestCase(1, 2, 3)]
[TestCase(2, 3, 5)]
public void CalculatorExample(int a, int b, int result)
{
	new Story("Calculator")
		.InOrderTo("Add two integers")
		.AsA("User")
		.IWant("The numbers are added")
		.WithScenario("adding two integers")
		.Given(ASimpleIntegerCalculator)
		.When(ISetTheValueTo, a)
			.And(IInvokeAdd)
			.And(ISetTheValueTo, b)
			.And(IInvokeEquals)
		.Then(TheValueIs, result)
		.ExecuteWithReport();
}

When run, will result in the following StoryQ report, a block of text for each run:

Story is Calculator
  In order to Add two integers
  As a User
  I want The numbers are added

      With scenario adding two integers
        Given a simple integer calculator => Passed
        When I set the value to(1)        => Passed
          And I invoke add                => Passed
          And I set the value to(2)       => Passed
          And I invoke equals             => Passed
        Then the value is(3)              => Passed

Story is Calculator
  In order to Add two integers
  As a User
  I want The numbers are added

      With scenario adding two integers
        Given a simple integer calculator => Passed
        When I set the value to(2)        => Passed
          And I invoke add                => Passed
          And I set the value to(3)       => Passed
          And I invoke equals             => Passed
        Then the value is(5)              => Passed

This can be quite cumbersome to go through all this of the StoryQ output when there is a large number of data driven tests.

What I would like is to have the report look something like this:

Story is Calculator
  In order to Add two integers
  As a User
  I want The numbers are added

      With scenario adding two integers
        Given a simple integer calculator => Passed
        When I set the value to {a}       => Passed
          And I invoke add                => Passed
          And I set the value to {b}      => Passed
          And I invoke equals             => Passed
        Then the value is {result}        => Passed

| a | b | result |
| 1 | 2 | 3      |
| 2 | 3 | 5      |

This is just a first initial sketch. Thinking about the "=> passed" indicators could be aggregate. So for instance, if a certain TestCase failed the output could look like:

Story is Calculator
  In order to Add two integers
  As a User
  I want The numbers are added

      With scenario adding two integers
        Given a simple integer calculator => All passed
        When I set the value to {a}       => All passed
          And I invoke add                => All passed
          And I set the value to {b}      => All passed
          And I invoke equals             => All passed
        Then the value is {result}        => Failed

| a | b | result     |
| 1 | 2 | 3          |
| 2 | 3 | 4 [Failed] |

So before I tease this out any further what are people's opinions on this? Barking mad? Has potential? Should I even be using StoryQ for this? If not, why not? :)

Cheers,

- Damian

 

Nov 14, 2010 at 6:48 PM
Edited Nov 14, 2010 at 6:49 PM

In the past, i've always relied on the host test runner (MSTest or nunit) to do data-driven scenarios. As you say, it's a little verbose in it's output. I've just been putting up with that.

I quite like your proposal for how output should look. If there were multiple scenarios in a story, then would the data summary appear at the bottom of each scenario? 

What kind of API are you evisaging? It would have to be something you'd call directly in the StoryQ api, we shouldn't depend on the host test runner's data-driven framework(s). Some key tenets that i've tried to stick to with StoryQ are that the API must be:

  • Discoverable (often with xml docs showing up in intellisense).
  • Strongly typed (so that the compiler can catch any silly errors or mistypes you've made)
  • DRY (or at least it allows you to be DRY when you use it)
  • Refactorable (related to the two above)
  • Portable (I *much* prefer to have the values right there in the source file than depend on a related CSV file, although by making it fully programmatic you could always read the CSV in yourself).

Cheers - Rob

Nov 14, 2010 at 7:30 PM

My initial consideration was to use the host runner data-driven api. detect that the test method running had parameters and was, well, being run multiple times. Also try to do this in a test runner / framework agnostic way. This way, developers can use the data driven api they are most familiar with and perhaps re-purpose existing data-driven tests.

Altering the StoryQ api I haven't yet considered in any depth. First thought... One problem perhaps, is that instead of running each TestCase as separated isolated test as test runners would (and they would may run them in random order), it would be one uber test. Not entirely sure if that is desirable. (Similarly I prefer the method-per-scenario approach)

Will think further about both approaches. I think I'll initially spike out the first approach before attempting to chang / extend the storyq api.

As to your tenents, seems perfectly reasonable to me. ;)

Nov 14, 2010 at 7:38 PM

hmm - I guess if it's possible to format that way from within the host, then that would be pretty awesome. I can't think of a clean way to do that though! 

Nov 14, 2010 at 11:01 PM

I'm mainly thinking about tackling via the XmlFileManagerBase \ XmlRenderer but not going near the text renderer. i.e. you would only see this format when viewing the report.

Will see what comes of it. Won't be clean first time around ;)

Nov 15, 2010 at 5:07 AM
This proposal seems a great idea to me. It is what is used in SpecFlow and RSpec (or cucumber - I can't remember). I was planning on spiking it at some stage but am getting slammed at the moment.
todd

On Mon, Nov 15, 2010 at 12:01 PM, damianh <notifications@codeplex.com> wrote:

From: damianh

I'm mainly thinking about tackling via the XmlFileManagerBase \ XmlRenderer but not going near the text renderer. i.e. you would only see this format when viewing the report.

Will see what comes of it. Won't be clean first time around ;)

Read the full discussion online.

To add a post to this discussion, reply to this email (storyq@discussions.codeplex.com)

To start a new discussion for this project, email storyq@discussions.codeplex.com

You are receiving this email because you subscribed to this discussion on CodePlex. You can unsubscribe or change your settings on codePlex.com.

Please note: Images and attachments will be removed from emails. Any posts to this discussion will also be available online at codeplex.com


Dec 3, 2010 at 7:16 PM

This is exactly the functionality I have been considering wiring in. Your vision of the output looks a lot better than what I was thinking of though. 

This would really help me evangelize StoryQ in my organization. Our chief architect is promoting Fitnesse as the solution for documenting data-driven acceptance criteria but he's too disconnected from our development experience to appreciate the real cost of Fitnesse. 

I'm interested in contributing to this feature.

Dec 3, 2010 at 11:35 PM

I did some research on what it would take to implement this feature as layed out above. Unfortunately there is no (simple) way to use reflection to get the test method's parameter meta data along with the actual parameter values from the stack frame. If there was then we could just output a table containing the test method parameter values along with some meta data about the parameters.

However, with one small tweek in the actual Given/When/Then delegates, we could get this behavior fairly easily... How does the syntax below look?

Notice the new TestDataColumnAttribute and how it is applied in the Operands and ResultIs methods. It should be possible to use (and extend) this attribute to build out the tabular data without much trouble.

Once I get this test passing I'll add another test to make multiple runs of the same test method with different parameters merge their results into one Story.TestData node. And then I'll update the text for each of the steps to display parameter names instead of parameter values (as Damian illustrated with the curly braces).

The TestDataColumnAttribute might be redundant - the code could automatically detect if the same set of steps have been executed multiple times (just with different parameters) and always display a table with every parameter. There might be value in hiding some parameters or specifying meta data about them though.

Thoughts?


    public class TestDataColumnAttribute : Attribute
    { }

    [TestClass]
    public class DataDrivenTableTest
    {
        [TestMethod]
        [TestCase(1, 2, 3)]
        public void TestTabularDataOutput(int a, int b, int result)
        {
            var outcome = new Story("").InOrderTo("").AsA("").IWant("").WithScenario("")
                .Given(Operands, a, b)
                .When(Adding)
                .Then(ResultIs, result);

            var xml = GetXmlForOutcomeOfStory(outcome);

            StringAssert.Contains(@"
                <TestData>
                    <Test>
                        <Parameter name="operand1" Type=""passed"">1</Parameter>
                        <Parameter name="operand2" Type=""passed"">2</Parameter>
                        <Parameter name="result" Type=""passed"">3</Parameter>
                    </Test>
                </TestData>

                "
, xml.ToString());
        }

        private void Operands([TestDataColumn]int operand1, [TestDataColumn]int operand2)
        {
            _operand1 = operand1;
            _operand2 = operand2;
        }

        private void Adding()
        {
            _result = _operand1 + _operand2;
        }

        private void ResultIs([TestDataColumn]int result)
        {
            Assert.That(_result, Is.EqualTo(result));
        }

        private static XElement GetXmlForOutcomeOfStory(Outcome outcome)
        { // copied from XmlRenderTest
            XElement element = new XElement("Story", new XAttribute("Name", "Story"));
            var results = ((IStepContainer)outcome).SelfAndAncestors().Reverse().Select(x => x.Step.Execute());
            new XmlRenderer(element).Render(results);
            return element;
        }

        private int _result, _operand1, _operand2;
    }

Dec 4, 2010 at 5:39 AM

Caley,

> Our chief architect is promoting Fitnesse as the solution for documenting data-driven acceptance criteria but he's too disconnected from our development experience to appreciate the real cost of Fitnesse.

Agreed that Fitnesse is "expensive" but that to me is because of where the sweetspot for Fitnesse is. If you really do have business involved who are maintaining the tests then it is the tool for the job. If not, then I wouldn't go there. Interestingly, there isn't a good Fixture in Fitnesse or Slim (for .net) for writing Given/When/Then either. That's not surprising to me because this rubric in not the type tests that business would maintain. Well, that has been my experience over the last 5 years. Hence the key difference of StoryQ is its reporting back out to business - and with StoryQ GUI to get tests in. I think that the real cost is training business (or even analysts) to make abstractions of the type that can be related to the code which both Fitnesse and StoryQ require (actually, there are loads of developers who can't).

Anyway, my point is that probably cost is only one cost. Fit for purpose is another. I have the wider argument that in LOB apps it is very unlikely that you are going to have only one type of test too. So, if there are only Fitnesse tests in the code base I somewhat dubious. I find that if the business is involveed I need some Fitnesse tests or some storyQ tests - these are going to be system level acceptance tests. Regardless,  I am going to also have classical (and BDD-style) tests for unit and integration tests.

I wonder what does the shape of your code base look like in this respect?

--todd

Dec 6, 2010 at 5:38 PM

Hi Todd,

I agree that the benefits could possibly outweigh the cost of a tool like Fitnesse if business owners are actually creating and maintaining the tests. Unfortunately that has not been my experience - development has owned the tool and tests and business has had no interest in taking it over. It has become just a way to communicate requirements back to business. StoryQ accomplishes this with much less overhead for the development staff. This is why I am enthusiastic about StoryQ as a replacement for us (where business is not writing the automated tests) - it is a much more natural tool for development to use.

Back to the Data Driven Stories discussion - after introducing our architect to StoryQ he pointed out that Fitnesse does have a more readable and less cumbersome way to express Data Driven stories. I agree with him, but Damien proposes a good solution to this.

- Caley

Dec 10, 2010 at 3:10 PM

Just a heads up, been very busy lately, but hope to spike this out soon.

Dec 15, 2010 at 5:30 PM
damianh wrote:

Just a heads up, been very busy lately, but hope to spike this out soon.

Hey Damian - check out the DataDrivenStories fork. The functionality isn't complete there yet but it is as far as including the parameter info in the Step and Result (by composition) classes so that data is available to the XmlRenderer and it is doing some parameter output in the xml now. I haven't tackled the Step Text updates to support the parameter names their yet. Is this the route you were considering? Suggestions?

Thanks,
Caley

Jan 4, 2011 at 12:22 AM

I finally got some images on photobucket illustrating the enhancement in the report. The individual rows in the data table are clickable and bring up the story in the context of that set of data. Also, there is a template row which just displays placeholders for variables. The variable names would ideally be - operand:1, operand:2 and result but I haven't made that enhancement yet.

This is available on the DataDrivenStories fork.

        [Test]
        [TestCase(1, 2, 3)]
        [TestCase(1, -2, 0)]
        [TestCase(5, -2, 3)]
        public void Addition(int operand1, int operand2, int result)
        {
            new Story("Calculator").Tag("sprint 1")
                .InOrderTo("Add two integers")
                .AsA("User")
                .IWant("The numbers are added")
                .WithScenario("adding two integers")
                .Given(ASimpleIntegerCalculator)
                .When(ISetTheValueTo, operand1)
                .And(IInvokeAdd)
                .And(ISetTheValueTo, operand2)
                .And(IInvokeEquals)
                .Then(TheValueIs, result)
                .ExecuteWithReport();
        }




Jan 6, 2011 at 1:13 AM

I'm still liking the idea of data driven stories. So, I'm about to be critical but don't have any solutions as yet. I wonder then if this is a moment where the we are copying current approaches but really we need to find a solution that builds on the others.

My criticism is that the solution still doesn't quite feel right. A couple of things:

  • I'm not sure that attributes are the best solution here - we want tabular data and it sort of works
  • I think that we need to stay as close to MSTest, NUnit test runner idioms as possible - does TestCase help with this cf with Sequential?
  • the parameters on the test method are a code smell ;-)
  • the reporting looks great but wonder about how well it will work with large tables


I am wondering if there are any idioms with lambdas/expressions here that could help with the syntax? As I look at the code we have not only separated out the data but also moved it to a different scope. What if you brought the data into the scope of the Story?

p.s. don't forget my bias is not use StoryQ for BDD unit or integration testing. I use it primarily for system testing. As such, I would less likely to use tables in the first place.

Jan 7, 2011 at 8:08 PM

Thanks for reviewing this, Todd.

The parameters on the test method and the TestCase attributes are actually NUnit constructs - see http://www.nunit.org/index.php?p=testCase&r=2.5. We are just leveraging the NUnit functionality to run the same Story with different parameters. This is a construct that NUnit has created, I don't think it's a code smell.

However, unfortunately MSTest does not yet have a similar construct. This is where the flexibility of this implementation helps out.

Basically, anytime the same story gets executed ("same story" means that all text and action methods match but the parameters to the step actions can differ) with the same MethodBase, then the stories will fold up together and be presented as a data-driven story.

The issue with executing multiple stories in a single test method in MSTest is that if the first execution fails, the following stories will not get executed. I just checked in a solution to this problem.

Here are a couple examples of the usage:

Option A (leveraging IDisposable)

        [TestMethod]
        public void ProgramaticAdditionDemo2()
        {
            using (DataDrivenStoryExecuter.CreateContext())
            {
                foreach (var data in new[]
                                        {
                                            new {Operand1=1, Operand2=2, Result=3},
                                            new {Operand1=1, Operand2=-2, Result=0},
                                            new {Operand1=5, Operand2=-2, Result=3},
                                        })
                {
                    new Story("Calculator").Tag("sprint 1")
                        .InOrderTo("Add two integers")
                        .AsA("User")
                        .IWant("The numbers are added")
                        .WithScenario("adding two integers")
                        .Given(ASimpleIntegerCalculator)
                        .When(ISetTheValueTo, data.Operand1)
                        .And(IInvokeAdd)
                        .And(ISetTheValueTo, data.Operand2)
                        .And(IInvokeEquals)
                        .Then(TheValueIs, data.Result)
                        .ExecuteWithReport();
                }
            }
        }

Option B (leveraging test setup/teardown methods)

        [TestMethod]
        public void ProgramaticAdditionDemo_NoUsingBlock()
        {
            foreach (var data in
                new[]
                    {
                        new {Operand1 = 1, Operand2 = 2, Result = 3},
                        new {Operand1 = 1, Operand2 = -2, Result = 0},
                        new {Operand1 = 5, Operand2 = -2, Result = 3},
                    })
            {
                new Story("Calculator").Tag("sprint 1")
                    .InOrderTo("Add two integers")
                    .AsA("User")
                    .IWant("The numbers are added")
                    .WithScenario("adding two integers")
                    .Given(ASimpleIntegerCalculator)
                    .When(ISetTheValueTo, data.Operand1)
                    .And(IInvokeAdd)
                    .And(ISetTheValueTo, data.Operand2)
                    .And(IInvokeEquals)
                    .Then(TheValueIs, data.Result)
                    .ExecuteWithReport();
            }
        }

        [TestInitialize]
        public void SetUp()
        {
            DataDrivenStoryExecuter.CreateContext();
        }

        [TestCleanup]
        public void TearDown()
        {
            DataDrivenStoryExecuter.ExecuteStoriesInContext();
        }


Both options produce the same report output as the NUNit-TestCase example above. I think the second option is cleaner, but both are supported with the code in the DataDrivenStories fork.

This is very flexible. As Rob pointed out - if this is fully programmable, we can get the parameter values any way we like: inline, external CSV, TestCase attributes... I do think I prefer iterating over the anonymous type array to the NUnit TestCase construct though (it has some limitations).

The implementation of DataDrivenStoryExecuter is checked into the DataDrivenStories fork.

 

Jan 16, 2011 at 9:41 PM

Caley,

So, this response has sat drafted for about a week now!

Just as I was wrong about TestCase because I forgot it is part of NUnit, I am wrong that MSTest also has this ability through attributes. MSTest doesn't although it does have its own through DataSource (see http://codeclimber.net.nz/archive/2008/01/18/how-to-simulate-rowtest-with-ms-test.aspx). This is so funny because I would have sworn that I have used the MSTest version somewhere along the line but clearly can't have - I did go back to the API and have a quick search.

In terms of TestCase being a code smell. I still think that it is albeit a small one. I would draw a comparison to the MVC pattern. I find that the C is often a code smell in many applications and it most often turns up with a fat controller pattern. So, my point is - even with well established patterns - that there are often smells within established and ubiquitous patterns. Unfortunately, I'm not smart enough to provide a solution (although I made an attempt on the fat controller problem in MVC Contrib with the fluent controller helper).

Back to your problem at hand, if you will bare with me. Given now that you are going to do the looping yourself, you now have duplication and redundancy that I wondering about if you can refactor out (back into the StoryQ conventions).

The worst duplication is in the "Operand1", "Operand2", and "Result" and is distracting from the data. It is tabular data so we know that there are conventions to follow here - wiki markup has taught us that.

new {Operand1 = 1, Operand2 = 2, Result = 3},
new {Operand1 = 1, Operand2 = -2, Result = 0},
new {Operand1 = 5, Operand2 = -2, Result = 3},

The redundancy lies in the IDisposable

  --> using (DataDrivenStoryExecuter.CreateContext())
 
Given that the ExecuteWithReport uses a vistor pattern, can't we delegate (or hide) at that point? I know that I should provide a code sample now!

Rob: you know the code (and language syntax) better than I, any thoughts on syntactic sugar that can aid us?

Here's the simplest first step to remove the iterator and setup/teardown which can be dealt with by the library:

[TestMethod]
public void ProgramaticAdditionDemo_NoUsingBlock()
{

   var data = new[]
          {
              new {Operand1 = 1, Operand2 = 2, Result = 3},
              new {Operand1 = 1, Operand2 = -2, Result = 0},
              new {Operand1 = 5, Operand2 = -2, Result = 3},
          });

    new Story("Calculator", data)
        .InOrderTo("Add two integers")
        .AsA("User")
        .IWant("The numbers are added")

        .WithScenario("adding two integers")
        .Given(ASimpleIntegerCalculator)
        .When(ISetTheValueTo, data.Operand1)
          .And(IInvokeAdd)
          .And(ISetTheValueTo, data.Operand2)
          .And(IInvokeEquals)
        .Then(TheValueIs, data.Result)

        .Execute();
    }
}


But I still want something even simpler that brings the data forward. Some pseudo code which is pretty much the wiki syntax (we could write this with | instead of commas).

[TestMethod]
public void ProgramaticAdditionDemo_NoUsingBlock()
{

   var data =

   Operand1, Operand2, Result
   1,        2,        3
   1,       -2,        0
   5,       -2,        3

    new Story("Calculator", data)
        .InOrderTo("Add two integers")
        .AsA("User")
        .IWant("The numbers are added")

        .WithScenario("adding two integers")
        .Given(ASimpleIntegerCalculator)
        .When(ISetTheValueTo, data.Operand1)
          .And(IInvokeAdd)
          .And(ISetTheValueTo, data.Operand2)
          .And(IInvokeEquals)
        .Then(TheValueIs, data.Result)

        .Execute();
    }
}

So the problem is getting IDE help on data.XXXX. Perhaps then, we need to add an abstraction as a struct (ouch).

private struct AdditionDemo{
    int Operand1
    int Operand2
    int Result
}

[TestMethod]
public void ProgramaticAdditionDemo_NoUsingBlock()
{

   var data = AdditionDemo.with("
               1,        2,        3
               1,       -2,        0
               5,       -2,        3");

    new Story("Calculator", data)
        .InOrderTo("Add two integers")
        .AsA("User")
        .IWant("The numbers are added")

        .WithScenario("adding two integers")
        .Given(ASimpleIntegerCalculator)
        .When(ISetTheValueTo, data.Operand1)
          .And(IInvokeAdd)
          .And(ISetTheValueTo, data.Operand2)
          .And(IInvokeEquals)
        .Then(TheValueIs, data.Result)

        .Execute();
    }
}


If that worked then perhaps we can tinker a little. Now, you'll note that I have a "with" helper that probably returns a type so that we can get some casting done under the hood.
[TestMethod]
public void ProgramaticAdditionDemo_NoUsingBlock()
{
    new Story("Calculator", with.AdditionDemo)
        .InOrderTo("Add two integers")
        .AsA("User")
        .IWant("The numbers are added")

        .WithScenario("adding two integers")
        .Given(ASimpleIntegerCalculator)
        .When(ISetTheValueTo, x => x.Operand1)
          .And(IInvokeAdd)
          .And(ISetTheValueTo, x => x.Operand2)
          .And(IInvokeEquals)
        .Then(TheValueIs, x => x.Result)

        .Execute(
                      "1,   2,  3
                       1,  -2,  0
                       5,  -2,  3"
                        );
    }
}


To do this there will be a class Story with constructor Story(string text), and a class Story<TData> with a constructor Story(string text, IEnumerable<TData> data).
   
  class AdditionDemo : IEnumerable<AdditionDemo>
    {
        int Operand1
        int Operand2
        int Result
    }
   
Alternatively, using a builder pattern, we could go somewhere like this:
[TestMethod]
public void ProgramaticAdditionDemo_NoUsingBlock()
{
      var data = AdditionDemo.with{
          "1,  2, 3
           1, -2, 0
           5, -2, 3"};
       
    new Story("Calculator", data)
        .InOrderTo("Add two integers")
        .AsA("User")
        .IWant("The numbers are added")

        .WithScenario("adding two integers")
        .Given(ASimpleIntegerCalculator)
        .When(ISetTheValueTo, x => x.Operand1)
          .And(IInvokeAdd)
          .And(ISetTheValueTo, x => x.Operand2)
          .And(IInvokeEquals)
        .Then(TheValueIs, x => x.Result)

        .Execute();
    }
}

If you really want to do inline tables, this suggests to me that SpecFlow and Gherkin syntax is what you want! But here is how StoryQ could go.

Feb 7, 2011 at 5:32 PM
This discussion has been copied to a work item. Click here to go to the work item and continue the discussion.