Randomize execution of sibling scopes/resets

[mdwhatcott]

This is to drive out temporal couplings between tests. Technically, the setup and teardown methods should be enough to do this but you just never know...

I think this will be a command line option that is enabled by default and can be disabled (which is how some of the unit tests can avoid being broken by this new feature).

[mdwhatcott]

The big question for me is how this might affect existing tests that depend of consistent execution order...

Also problematic is that fact that I can't create new command line flags. So, the only option is to use invalid values for existing ones (like I've done with json reporting).

[mholt]

@mdwhatcott Now that tests run concurrently, I've noticed a sort of pseudo-randomness in test execution. Is this good enough, or do we want to provide random seeds and let the user specify one, etc, etc...?

[mdwhatcott]

It's more about making sure that unit tests run in any order without any effect on the end result. I've been in situations with frameworks that do randomize execution and have uncovered subtle bugs that hide when you only see the tests executed in deterministic order. The purist in me says we have to have this right now ("How dare we call this a robust testing framework without it!?"). The pragmatist says wait until we get burned by not having it (and I'm still hung up about the command line flag hack that will be required--see my last comment).

[mdwhatcott]

This issue is somewhat related to #70. If we can resolve that issue successfully we could rely on go test running the specs concurrently to root out any issues with the tests that would be masked by synchronous, deterministic execution.

[mdwhatcott]

Not quite sure how to go about this yet so we are deferring this issue. It will be referenced on the wiki for future discussion.

[riannucci]

Out of curiosity, would it be enough to:

• for every suite of N !complete Convey blocks
  • for each Convey in the suite
    • if expectChildRun is true
      • with 1/N probability execute the suite

It gets a bit tricky to do with the way that it's doing greedy suite execution right now (e.g. it will execute the first !complete Convey in every suite if expectChildRun is true). If we did this randomized execution, there'd have to be more passes through the test in order to do breadth-first evaluation of Conveys.. since every suite can also have initialization side effects, this would mean that we'd have to restart execution from the top-most suite every time we encountered a new suite that we hadn't seen before.

[mdwhatcott]

Having just rewritten most of the convey package you probably have a much better feel for how this could work at this point. :)

The tests would certainly have to be re-thought--they count on deterministic execution order pretty heavily. Maybe there could be a setting that enables random execution? (sigh...)

I'd be interested to try it out though.

[riannucci]

Probably will need options like -randomize-test-order (bool) and -random-seed (int). The tests for the random execution would probably set the seed at the top of the test (like they do for some other global stuff in prepare()). I would probably have each Convey(..., t, ...) create a new random generator starting at seed so that t.Parallel() doesn't goof things up :).

edited because github thought I had a weird html tag in there