Latest Specflow caused "Stopped Working" dialog popup .Net Core 2.2

[jronay]

SpecFlow Version:

• 3.0
• 2.4
• 2.3
• 2.2
• 2.1
• 2.0
• 1.9

Used Test Runner

• SpecFlow+Runner
• MSTest
• NUnit
• Xunit

Version number: NUnt3TestAdapter 3.13.0

Visual Studio Version

• VS 2017 v15.9.8
• VS 2015
• VS 2013

Are the latest Visual Studio updates installed?

• Yes
• No

.NET Framework:

• >= .NET 4.5
• before .NET 4.5
• .NET Core 2.0
• .NET Core 2.1
• .NET Core 2.2
• .NET Core 3.0

Test Execution Method:

• Visual Studio Test Explorer
• TFS/VSTS/Azure DevOps – Task – PLEASE SPECIFY THE NAME OF THE TASK
• Command line – PLEASE SPECIFY THE FULL COMMAND LINE

<SpecFlow> Section in app.config

{
  "language": {
    "feature": "en-US"
  }
}

Repro Project

Issue Description

This dialog box pops up frequently"
"TechTalk.SpecFlow.VisualStudioCodeBehind has stopped working"

Steps to Reproduce

This dialog is still popping up with all the recommended configurations in place.
I believe this is a bug as it has been present for the last 3 versions of Visual Studio.

Running Visual Studio Enterprise 2017 v15.9.8
Installed dependencies:

<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <TargetFramework>netcoreapp2.2</TargetFramework>
  </PropertyGroup>
  <ItemGroup>
    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="16.0.1" />
    <PackageReference Include="NUnit3TestAdapter" Version="3.13.0" />
    <PackageReference Include="SpecFlow" Version="3.0.177-beta" />
    <PackageReference Include="SpecFlow.NUnit" Version="3.0.177-beta" />
    <PackageReference Include="SpecFlow.Tools.MsBuild.Generation" Version="3.0.177-beta" />
  </ItemGroup>
  <ItemGroup>
    <ProjectReference Include="..\CalTesting.Calc\CalTesting.Calc.csproj" />
  </ItemGroup>
</Project>

Project configuration:

<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <TargetFramework>netcoreapp2.2</TargetFramework>
  </PropertyGroup>
  <ItemGroup>
    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="16.0.1" />
    <PackageReference Include="NUnit3TestAdapter" Version="3.13.0" />
    <PackageReference Include="SpecFlow" Version="3.0.177-beta" />
    <PackageReference Include="SpecFlow.NUnit" Version="3.0.177-beta" />
    <PackageReference Include="SpecFlow.Tools.MsBuild.Generation" Version="3.0.177-beta" />
  </ItemGroup>
  <ItemGroup>
    <ProjectReference Include="..\CalTesting.Calc\CalTesting.Calc.csproj" />
  </ItemGroup>
</Project>

specflow.json contents:

 {
  "language": {
    "feature": "en-US"
  }
}

Note: The tests would not run / be detected unless the Microsoft.NET.Test.Sdk was installed as shown above. The error without the sdk installed indicated “Unable to run NUnit unit test. Unable to find testhost.dll Please publish your test project and retry”
See the referenced post here: microsoft/vstest#1870

I would like to know if others to see the “Stopped Working“ dialogue mentioned here and if there is any recommended workaround to prevent this.

[SabotageAndi]

1. We don't see having a dependency on Microsoft.NET.Test.Sdk as our job, because we don't need anything from it. I think xUnit is doint it, but NUnit not.

2. When you use the preview extension mentioned in https://specflow.org/2018/specflow-3-public-preview-now-available/ the version has an option Enable SpecFlowSingleFileGenerator.
   

If you set this to false, you shouldn't get the error message.

[SabotageAndi]

possible a duplicate of #1445

[jronay]

1. The tests would not be detected with NUnit and Specflow unless the mentioned SDK was referenced.
2. The setting you mentioned was set as you indicated, and yet the popup still occurs
   

[jronay]

Thanks for the quick reply :)

[SabotageAndi]

@jronay when do you get the error then?

[jronay]

If I load the project and click on the Calculator.feature file then the stopped working popup occurs. The calculator feature uses the basic calculator add tests.

[jronay]

p.s. I'm not sure I follow your comment about not having a dependency on the test SDK, because for those of us that use NUnit, the specflow tests will not run without it -- seems to me if you support NUnit, then there is a "dependency" on the test SDK, intended or not.

[SabotageAndi]

The Test SDK dependency story is "complicated". It is needed for the unit test project that VSTest finds tests. No test runner needs the dependency. It's only needed because of the tooling. And I think it's only needed for .NET Core. Not for .NET Desktop Framework.

Additionally the different test frameworks have different approaches to it:

SpecFlow+Runner: the SpecRun.Runner package has the dependency (https://www.nuget.org/packages/SpecRun.Runner/3.0.246-beta)
xUnit: The VisualStudio Runner has the dependency (https://www.nuget.org/packages/xunit.runner.visualstudio/)
NUnit: Nothing has the dependency on the SDK (I found nothing)
MSTest: same as NUnit

As SpecFlow sits on top of the test frameworks/Runner, I don't think it is it's job to make sure, that you can also execute the tests. We also don't have dependencies on the various adapter/runner packages from xUnit/MSTest/NUnit.

I understand that it makes it easier for the user, but it is one more dependency, we have to care about. Perhaps we will change this sometime in the future. Depends how much issues we get about it. But currently I want to hold the dependencies as low as possible.

You would also need to add this dependency yourself, if you use NUnit without SpecFlow.
I hope this makes our decision to not have this dependency clear.

[jronay]

@SabotageAndi I understand part of your reasoning. But just to be clear, in .Net Core 2.2, specflow with NUnit does not work at all (you can't run the tests because they are not detected) unless you install the test SDK mentioned.
As developers using the specflow product in .Net Core, (hopefully there will be many soon) we should not have to "discover" this by researching the reported error, which is part of the reason for this post.
"As SpecFlow sits on top of the test frameworks/Runner, I don't think it is it's job to make sure, that you can also execute the tests." -- But why would we use SpecFlow to write tests that cannot be executed?? Of course it is SpecFlow's job :) It is stated that SpecFlow works with NUnit in the product specifications for .Net Core, and this particular version is targeting .Net Core.

[SabotageAndi]

Some provoking questions, (only for thought process, please take no offense in them):

Can you run your NUnit Tests without NUnit3TestAdapter in VS? - No
Did NUnit or SpecFlow has a dependency on it? - No
Did you add the dependency to the project yourself? - Yes

I have the view, it is the same with Microsoft.NET.Test.Sdk.

Needing this dependency is sadly and the user has to know it.

The expectation from Microsoft is, that you start with one of their unit test projects. These contain the reference to the test sdk.

Have a look at the documentation:

• https://docs.microsoft.com/en-us/dotnet/core/tools/dotnet-test?tabs=netcore21#description
• https://docs.microsoft.com/en-us/dotnet/core/testing/unit-testing-with-nunit

You see in both pages, the Microsoft.NET.Test.Sdk is referenced.
We will do that too. @Stephen-Mc, please make sure, that we will mention the need for Microsoft.NET.Test.Sdk in our documentation.

[SabotageAndi]

Closed because of inactivity

[github-actions]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.