Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Using t.Parallel() with Convey dies #70

Closed
augustoroman opened this issue Nov 5, 2013 · 17 comments
Closed

Using t.Parallel() with Convey dies #70

augustoroman opened this issue Nov 5, 2013 · 17 comments

Comments

@augustoroman
Copy link

For example, modify the tests in examples to all use t.Parallel(), then run:

go test -parallel 5 github.com/smartystreets/...

The following panic comes up:

panic: Only the top-level call to Convey(...) needs a reference to the *testing.T. (See /.../src/github.com/smartystreets/goconvey/examples/bowling_game_test.go:71) [recovered]
    panic: Only the top-level call to Convey(...) needs a reference to the *testing.T. (See /.../src/github.com/smartystreets/goconvey/examples/bowling_game_test.go:71)
@ghost ghost assigned mdwhatcott Nov 5, 2013
@mdwhatcott
Copy link
Collaborator

@augustoroman I'm having trouble replicating that error message. Can you paste the code you used in the bowling_game_test.go file? It looks like you might have added a t parameter where it wasn't needed (hence the error message). You only pass t to Convey when the Convey is not nested within any other Convey calls.

I was able to add a single call to t.Parallel() at the beginning of each test function in the examples package and saw that tests were run concurrently. This currently isn't supported when using the Web UI but it probably will be soon. We tend to run tests in separate packages in parallel, rather than tests within a package in parallel. But this is something I'd like to see if we can fix in order to more seamlessly integrate with "go test". Thanks for reporting this. Again, I'd like to see the exact code you used to produce the panic message you reported...

@augustoroman
Copy link
Author

Please see https://github.com/augustoroman/goconvey/commit/6330da9ee523ae253aaa80e22b563e98eae007ca

I verified that this occurs for me with go1.2rc2 and go1.2rc3, but not go1.1.2. Might be a regression.

@mdwhatcott
Copy link
Collaborator

@augustoroman Ok, that's just like what I tried. I'm on go1.1.2 right now and there is an open issue related to getting go 1.2 working. Right now I'm working on some internal refactoring on the server but the Go 1.2 stuff will be addressed quickly. Stay tuned...

@augustoroman
Copy link
Author

It's not related to the server, but rather to core functionality in GoConvey. The problem seems to be that there's a single global runner variable that is not safe for concurrent access. Running the tests with the race detector exposed the problem:

$ go test -race -parallel 5 github.com/augustoroman/goconvey/examples
==================
WARNING: DATA RACE
Read by goroutine 5:
  github.com/smartystreets/goconvey/execution.(*runner).ensureStoryCanBegin()
      /Users/aroman/go/src/github.com/smartystreets/goconvey/execution/runner.go:23 +0x48
  github.com/smartystreets/goconvey/execution.(*runner).Begin()
      /Users/aroman/go/src/github.com/smartystreets/goconvey/execution/runner.go:18 +0x34
  github.com/smartystreets/goconvey/convey.register()
      /Users/aroman/go/src/github.com/smartystreets/goconvey/convey/doc.go:44 +0x7a
  github.com/smartystreets/goconvey/convey.Convey()
      /Users/aroman/go/src/github.com/smartystreets/goconvey/convey/doc.go:30 +0x56
  github.com/augustoroman/goconvey/examples.TestScoring()
      /Users/aroman/go/src/github.com/augustoroman/goconvey/examples/bowling_game_test.go:71 +0x14d
  testing.tRunner()
      /usr/local/go/src/pkg/testing/testing.go:389 +0x10f

Previous write by goroutine 3:
  github.com/smartystreets/goconvey/execution.(*runner).ensureStoryCanBegin()
      /Users/aroman/go/src/github.com/smartystreets/goconvey/execution/runner.go:24 +0x68
  github.com/smartystreets/goconvey/execution.(*runner).Begin()
      /Users/aroman/go/src/github.com/smartystreets/goconvey/execution/runner.go:18 +0x34
  github.com/smartystreets/goconvey/convey.register()
      /Users/aroman/go/src/github.com/smartystreets/goconvey/convey/doc.go:44 +0x7a
  github.com/smartystreets/goconvey/convey.Convey()
      /Users/aroman/go/src/github.com/smartystreets/goconvey/convey/doc.go:30 +0x56
  github.com/augustoroman/goconvey/examples.TestAssertions()
      /Users/aroman/go/src/github.com/augustoroman/goconvey/examples/assertion_examples_test.go:86 +0x14d
  testing.tRunner()
      /usr/local/go/src/pkg/testing/testing.go:389 +0x10f

Goroutine 5 (running) created at:
  testing.RunTests()
      /usr/local/go/src/pkg/testing/testing.go:469 +0xb3c
  testing.Main()
      /usr/local/go/src/pkg/testing/testing.go:401 +0xa2
  main.main()
      github.com/augustoroman/goconvey/examples/_test/_testmain.go:51 +0xdc

Goroutine 3 (running) created at:
  testing.RunTests()
      /usr/local/go/src/pkg/testing/testing.go:469 +0xb3c
  testing.Main()
      /usr/local/go/src/pkg/testing/testing.go:401 +0xa2
  main.main()
      github.com/augustoroman/goconvey/examples/_test/_testmain.go:51 +0xdc
==================

In goconvey/convey/wiring.go:49 the global vars runner and reporter are created, then accessed without any locking in goconvey/convey/doc.go:44. When multiple tests are run in parallel, the top-level Convey calls are interleaved to the global runner, causing the spurious failure suggesting that testing.T is passed in multiple times.

I'm not sure how extensive your refactoring is. The globals should be removed if possible.

@mdwhatcott
Copy link
Collaborator

Thanks for your detective work here @augustoroman. You're absolutely right about the globals and I've known they would pose a problem where concurrency is concerned. We won't be able to remove them because this code is imported as a library and is subordinate to go test, which generates its own main() function which we can't hook into (at least I can't think of a way...). I'm not sure where else I would store that state. The solution might be to try to lock on those variables whenever they are being acted upon but I'm not even sure that would work in a concurrent model because of the way GoConvey uses them to register callbacks during execution. I'll try it out sometime soon (you're welcome to tinker with it if you're feeling brave). Stay tuned...

@mdwhatcott
Copy link
Collaborator

Not quite sure how to go about this yet so we are deferring this issue. It will be referenced on the wiki for future discussion.

@mdwhatcott
Copy link
Collaborator

Ok, I've got some ideas on how to solve this. It's really been bugging me that we don't support this feature. Making it work would involve a few trade-offs:

  1. The public DSL would change ever so slightly, requiring anyone with existing GoConvey tests to run a script or update their tests with the change (this would be GoConvey 2.0). See the code below for an example.
  2. Because of the change mentioned in item 1, the console reporter would no longer be able to display the number of assertions executed so far. Basically, I would remove the package level runner/reporter and create a new one for each Test* function.

Here's what the code would look like:

package examples

import (
    . "github.com/smartystreets/goconvey/convey"
    "testing"
)

func TestSpec(t *testing.T) {
    Convey("Subject: Integer incrementation and decrementation", t, func(so Assert) {
        var x int

        Convey("Given a starting integer value", func() {
            x = 42

            Convey("When incremented", func() {
                x++

                Convey("The value should be greater by one", func() {
                    so(x, ShouldEqual, 43)
                })
                Convey("The value should NOT be what it used to be", func() {
                    so(x, ShouldNotEqual, 42)
                })
            })
            Convey("When decremented", func() {
                x--

                Convey("The value should be lesser by one", func() {
                    so(x, ShouldEqual, 41)
                })
                Convey("The value should NOT be what it used to be", func() {
                    so(x, ShouldNotEqual, 42)
                })
            })
            Reset(func() {
                x = 0
            })
        })
    })
}

Notice that the top-level Convey now receives a func with a single parameter, which itself is a func that matches the current So method. This new method would be provided by the framework and would be a closure over the reporter/runner for that Test* function.

Thoughts?

@mdwhatcott mdwhatcott reopened this Dec 26, 2013
@mdwhatcott
Copy link
Collaborator

As I've mused about this alternate syntax I'm realizing that there are some intricacies like (SkipConvey, SkipSo, Reset, etc...) that will have to be worked through as well. All of those public methods are leveraging the package-level resources which I will be trying to get rid of.

@mholt
Copy link
Contributor

mholt commented Dec 26, 2013

The change to the DSL is minor, albeit more complex/more to remember. (Even though we have a code generator, it's not used all the time.) Is it worth it?

@mdwhatcott
Copy link
Collaborator

It's true that I've gone back and forth on this. Here are the current reasons for moving forward with the proposed change:

  1. It eats at me that we don't support some of the built-in go test functionality. It's something we say we do but there's this little caveat/exception.
  2. Apart from preventing us from running concurrent Test functions, having the runner and the reporter stored as package-level variables has always felt dirty.
  3. If the reporter and the runner are isolated by top-level Convey calls then that simplifies the internal code in several places and I think it could make some of our other long-standing issues easier to resolve without feeling like we've just hacked the framework (like randomization of sibling specs, consolidated Convey-So calls, etc...).

Reasons against the change:

  1. Any change to the DSL is a big deal. I'd probably want to create a script or a go fmt refactor command that people could run to automatically fix their existing test suites. But still, people want to feel like they can depend on the API of a framework. This would, no doubt, be a version bump to 2.0.
  2. It bloats the top-level Convey call. Not by much, but it is more verbose.
  3. We have to rethink Skip, SkipSo, and Reset as they now use the ambient runner and reporter.

@mdwhatcott
Copy link
Collaborator

Alternative DSL that makes skipping and resets easy (addresses item number 3 in my last comment):

Convey("Stuff", t, func(c C) {
    c.Convey("More stuff", func() {
        c.So(1, ShouldEqual, 1)
    })

    c.SkipConvey("Skipped stuff", func() {
        c.So(1, ShouldEqual, 2)
    })

    c.Reset(func() {
        // stuff
    })
})

So this introduces a new struct C that is provided by the framework for all nested behavior. As for the name of the struct, which here is C, I'm not really a fan of this name--it would be great if there were something that contributed to the flow of how the tests read, not just noisy boilerplate. I'm not settled on doing anything just yet. Just trying to balance my desire for a more robust feature set with my desire for a clean, readable DSL.

@mdwhatcott
Copy link
Collaborator

Here's another version that's arguably cleaner:

Convey("Stuff", t, func(c Context, so Assert) {
    Convey("More stuff", c, func() {
        so(1, ShouldEqual, 1)
    })

    SkipConvey("Skipped stuff", c, func() {
        so(1, ShouldEqual, 2)
    })

    Reset(c, func() {
        // stuff
    })
})

Benefits:

  • There are always three arguments to Convey. Top-level receives a t, nested ones receive c.
  • The DSL is still just function calls--no c.Convey and c.So everywhere
  • I think the so Assert parameter is just a method on the Context under the hood.
  • Having the Context struct on a per-test function basis allows us to do accomplish a lot more (as stated in the previous comments).

@mdwhatcott
Copy link
Collaborator

Ok, I'm closing this issue to start another regarding the changes we are considering with the DSL. In the event that a change is implemented it will include a fix and a reference for this issue.

@mdwhatcott
Copy link
Collaborator

@augustoroman @mholt @joliver - I've fixed it! And it didn't require any changes to the DSL. I'm still hacking on things but you are welcome to check out the "issue70" branch to see the fix. The package-level state still exists but instead of one runner and one reporter there are now runners and reporters for each test function so there's no competing for resources (where was my brain 3 months ago?). I modified the example tests to use t.Parallel() so all you have to do is run go test -parallel=3 and it works.

@augustoroman
Copy link
Author

Awesome!! I tried it out and it was working really well, and I was about to call this done when I hit this:

$ go test --parallel 30 -race github.com/smartystreets/goconvey/examples
..............

5 assertions thus far

.==================
WARNING: DATA RACE
Write by goroutine 7:
  runtime.mapassign1()
      /usr/local/go/src/pkg/runtime/hashmap.c:1134 +0x0
  github.com/smartystreets/goconvey/convey.register()
      /Users/aroman/go/src/github.com/smartystreets/goconvey/convey/doc.go:50 +0x2ac
  github.com/smartystreets/goconvey/convey.Convey()
      /Users/aroman/go/src/github.com/smartystreets/goconvey/convey/doc.go:31 +0x56
  github.com/smartystreets/goconvey/examples.TestSpec()
      /Users/aroman/go/src/github.com/smartystreets/goconvey/examples/simple_example_test.go:44 +0x26e
  testing.tRunner()
      /usr/local/go/src/pkg/testing/testing.go:391 +0x10f

Previous read by goroutine 3:
  runtime.mapaccess1_faststr()
      /usr/local/go/src/pkg/runtime/hashmap_fast.c:17 +0x0
  github.com/smartystreets/goconvey/convey.So()
      /Users/aroman/go/src/github.com/smartystreets/goconvey/convey/doc.go:83 +0xc6
  github.com/smartystreets/goconvey/examples.func·001()
      /Users/aroman/go/src/github.com/smartystreets/goconvey/examples/assertion_examples_test.go:29 +0x976
  github.com/smartystreets/goconvey/execution.(*Action).Invoke()
      /Users/aroman/go/src/github.com/smartystreets/goconvey/execution/action.go:6 +0x42
  github.com/smartystreets/goconvey/execution.(*scope).visit()
      /Users/aroman/go/src/github.com/smartystreets/goconvey/execution/scope.go:37 +0x75
  github.com/smartystreets/goconvey/execution.(*scope).visitChild()
      /Users/aroman/go/src/github.com/smartystreets/goconvey/execution/scope.go:52 +0xb6
  github.com/smartystreets/goconvey/execution.(*scope).visitChildren()
      /Users/aroman/go/src/github.com/smartystreets/goconvey/execution/scope.go:47 +0x64
  github.com/smartystreets/goconvey/execution.(*scope).visit()
      /Users/aroman/go/src/github.com/smartystreets/goconvey/execution/scope.go:38 +0x83
  github.com/smartystreets/goconvey/execution.(*scope).visitChild()
      /Users/aroman/go/src/github.com/smartystreets/goconvey/execution/scope.go:52 +0xb6
  github.com/smartystreets/goconvey/execution.(*scope).visitChildren()
      /Users/aroman/go/src/github.com/smartystreets/goconvey/execution/scope.go:47 +0x64
  github.com/smartystreets/goconvey/execution.(*scope).visit()
      /Users/aroman/go/src/github.com/smartystreets/goconvey/execution/scope.go:38 +0x83
  github.com/smartystreets/goconvey/execution.(*runner).Run()
      /Users/aroman/go/src/github.com/smartystreets/goconvey/execution/runner.go:91 +0xbf
  github.com/smartystreets/goconvey/convey.register()
      /Users/aroman/go/src/github.com/smartystreets/goconvey/convey/doc.go:53 +0x2d5
  github.com/smartystreets/goconvey/convey.Convey()
      /Users/aroman/go/src/github.com/smartystreets/goconvey/convey/doc.go:31 +0x56
  github.com/smartystreets/goconvey/examples.TestAssertions()
      /Users/aroman/go/src/github.com/smartystreets/goconvey/examples/assertion_examples_test.go:92 +0x14d
  testing.tRunner()
      /usr/local/go/src/pkg/testing/testing.go:391 +0x10f

Goroutine 7 (running) created at:
  testing.RunTests()
      /usr/local/go/src/pkg/testing/testing.go:471 +0xb3c
  testing.Main()
      /usr/local/go/src/pkg/testing/testing.go:403 +0xa2
  main.main()
      github.com/smartystreets/goconvey/examples/_test/_testmain.go:51 +0xdc

Goroutine 3 (running) created at:
  testing.RunTests()
      /usr/local/go/src/pkg/testing/testing.go:471 +0xb3c
  testing.Main()
      /usr/local/go/src/pkg/testing/testing.go:403 +0xa2
  main.main()
      github.com/smartystreets/goconvey/examples/_test/_testmain.go:51 +0xdc
==================
.....

4 assertions thus far

....................................

47 assertions thus far

PASS
Found 1 data race(s)
FAIL    github.com/smartystreets/goconvey/examples  1.623s

@mholt
Copy link
Contributor

mholt commented Jan 31, 2014

That doesn't look too bad; not anything a channel can't fix, I'd imagine? We just have to protect that map.

@mdwhatcott
Copy link
Collaborator

Yup, not hard to fix. I've implemented a simple sync.Mutex for now. Thanks @augustoroman for bringing up -race, that's a great tool. Pull in the latest on the issue70 branch--everything should be working now. (Note that I've removed statistical aggregation from the console output.)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants