Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Work item tests crash on OSX x64 Debug #87287

Closed
eerhardt opened this issue Jun 8, 2023 · 8 comments
Closed

Work item tests crash on OSX x64 Debug #87287

eerhardt opened this issue Jun 8, 2023 · 8 comments
Labels
area-Infrastructure blocking-clean-ci Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms' Known Build Error Use this to report build issues in the .NET Helix tab

Comments

@eerhardt
Copy link
Member

eerhardt commented Jun 8, 2023

Build Information

Build: https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=300158
Build error leg or test failing: System.IO.FileSystem.Watcher.Tests.WorkItemExecution
Pull request: #86974

Error Message

Fill the error message using step by step known issues guidance.

{
  "ErrorMessage": "END OF WORK ITEM LOG: Command exited with 127",
  "ErrorPattern": "",
  "BuildRetry": false,
  "ExcludeConsoleLog": false
}

Report

Build Definition Test Pull Request
475355 dotnet/runtime System.IO.FileSystem.Tests.WorkItemExecution
474849 dotnet/runtime System.IO.FileSystem.Tests.WorkItemExecution
474532 dotnet/runtime System.IO.FileSystem.Tests.WorkItemExecution
473727 dotnet/runtime System.IO.FileSystem.Tests.WorkItemExecution
473597 dotnet/runtime System.IO.FileSystem.Manual.Tests.WorkItemExecution
473431 dotnet/runtime System.IO.FileSystem.Tests.WorkItemExecution
473397 dotnet/runtime System.IO.FileSystem.Tests.WorkItemExecution
472898 dotnet/runtime System.IO.FileSystem.Tests.WorkItemExecution
472852 dotnet/runtime System.IO.Compression.Tests.WorkItemExecution
472282 dotnet/runtime System.IO.FileSystem.Tests.WorkItemExecution
471895 dotnet/runtime System.IO.FileSystem.Tests.WorkItemExecution
471740 dotnet/runtime System.IO.Compression.Tests.WorkItemExecution
471580 dotnet/runtime System.IO.FileSystem.Manual.Tests.WorkItemExecution
471389 dotnet/runtime System.IO.FileSystem.Tests.WorkItemExecution
470637 dotnet/runtime System.IO.FileSystem.Tests.WorkItemExecution
470321 dotnet/runtime System.IO.FileSystem.Tests.WorkItemExecution
469012 dotnet/runtime System.IO.FileSystem.Tests.WorkItemExecution
468964 dotnet/runtime System.IO.FileSystem.Primitives.Tests.WorkItemExecution
468841 dotnet/runtime System.IO.FileSystem.Manual.Tests.WorkItemExecution
468587 dotnet/runtime System.IO.FileSystem.Primitives.Tests.WorkItemExecution
468089 dotnet/runtime System.IO.FileSystem.Primitives.Tests.WorkItemExecution
467619 dotnet/runtime System.IO.FileSystem.Tests.WorkItemExecution
467146 dotnet/runtime System.IO.FileSystem.Tests.WorkItemExecution
466691 dotnet/runtime System.IO.FileSystem.Tests.WorkItemExecution
465768 dotnet/runtime System.IO.FileSystem.Tests.WorkItemExecution
465695 dotnet/runtime System.IO.FileSystem.Primitives.Tests.WorkItemExecution
465642 dotnet/runtime XUnitLogChecker.WorkItemExecution #93906
465587 dotnet/runtime XUnitLogChecker.WorkItemExecution #93906
465447 dotnet/runtime XUnitLogChecker.WorkItemExecution #93906
465319 dotnet/runtime System.IO.FileSystem.Primitives.Tests.WorkItemExecution
465327 dotnet/runtime System.IO.FileSystem.Primitives.Tests.WorkItemExecution
464592 dotnet/runtime System.IO.FileSystem.Tests.WorkItemExecution
463142 dotnet/runtime System.IO.FileSystem.Tests.WorkItemExecution
462608 dotnet/runtime System.IO.FileSystem.Manual.Tests.WorkItemExecution
461771 dotnet/runtime System.IO.FileSystem.Tests.WorkItemExecution
460596 dotnet/runtime System.IO.FileSystem.Tests.WorkItemExecution
460238 dotnet/runtime System.IO.FileSystem.Tests.WorkItemExecution
459941 dotnet/runtime System.IO.FileSystem.Tests.WorkItemExecution
459223 dotnet/runtime System.IO.FileSystem.Tests.WorkItemExecution
458289 dotnet/runtime System.IO.FileSystem.Tests.WorkItemExecution
457386 dotnet/runtime System.IO.FileSystem.Tests.WorkItemExecution
456786 dotnet/runtime System.IO.FileSystem.Tests.WorkItemExecution
456243 dotnet/runtime System.IO.FileSystem.Tests.WorkItemExecution
456245 dotnet/runtime System.IO.FileSystem.Manual.Tests.WorkItemExecution
455009 dotnet/runtime System.IO.FileSystem.Tests.WorkItemExecution
454543 dotnet/runtime System.IO.FileSystem.Manual.Tests.WorkItemExecution
454533 dotnet/runtime System.IO.FileSystem.Tests.WorkItemExecution
454134 dotnet/runtime System.IO.FileSystem.Tests.WorkItemExecution
452981 dotnet/runtime System.IO.FileSystem.Tests.WorkItemExecution
451747 dotnet/runtime System.IO.FileSystem.Tests.WorkItemExecution
450630 dotnet/runtime System.IO.FileSystem.Tests.WorkItemExecution
449186 dotnet/runtime System.IO.FileSystem.Tests.WorkItemExecution
447511 dotnet/runtime System.IO.FileSystem.Tests.WorkItemExecution
446965 dotnet/runtime System.IO.FileSystem.Tests.WorkItemExecution

Summary

24-Hour Hit Count 7-Day Hit Count 1-Month Count
1 21 54

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=323604
Error message validated: Work item System.IO.FileSystem.Watcher.Tests in job
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 6/28/2023 9:26:54 PM UTC

@eerhardt eerhardt added blocking-clean-ci Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms' Known Build Error Use this to report build issues in the .NET Helix tab labels Jun 8, 2023
@ghost ghost added the untriaged New issue has not been triaged by the area owner label Jun 8, 2023
@ghost
Copy link

ghost commented Jun 8, 2023

Tagging subscribers to this area: @dotnet/area-infrastructure-libraries
See info in area-owners.md if you want to be subscribed.

Issue Details

Build Information

Build: https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=300158
Build error leg or test failing: System.IO.FileSystem.Watcher.Tests.WorkItemExecution
Pull request: #86974

Error Message

Fill the error message using step by step known issues guidance.

{
  "ErrorMessage": "Work item System.IO.FileSystem.Watcher.Tests in job",
  "ErrorPattern": "",
  "BuildRetry": false,
  "ExcludeConsoleLog": false
}
Author: eerhardt
Assignees: -
Labels:

area-Infrastructure-libraries, blocking-clean-ci, untriaged, Known Build Error

Milestone: -

@ghost
Copy link

ghost commented Jun 8, 2023

Tagging subscribers to this area: @dotnet/area-system-io
See info in area-owners.md if you want to be subscribed.

Issue Details

Build Information

Build: https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=300158
Build error leg or test failing: System.IO.FileSystem.Watcher.Tests.WorkItemExecution
Pull request: #86974

Error Message

Fill the error message using step by step known issues guidance.

{
  "ErrorMessage": "Work item System.IO.FileSystem.Watcher.Tests in job",
  "ErrorPattern": "",
  "BuildRetry": false,
  "ExcludeConsoleLog": false
}
Author: eerhardt
Assignees: -
Labels:

area-System.IO, blocking-clean-ci, untriaged, Known Build Error

Milestone: -

@jozkee
Copy link
Member

jozkee commented Jul 21, 2023

The logs not only show FileSystem.Watcher.Tests. I think this is a broader issue.

https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_apis/build/builds/334444/logs/123

e.g:

2023-07-11T00:12:22.4993970Z /Users/runner/work/1/s/.packages/microsoft.dotnet.helix.sdk/7.0.0-beta.23313.4/tools/Microsoft.DotNet.Helix.Sdk.MultiQueue.targets(90,5): error : Work item Common.Tests in job 9199670f-aa4c-4e8e-8150-b650ad0bb59f has failed. [/Users/runner/work/1/s/src/libraries/sendtohelixhelp.proj]
2023-07-11T00:12:22.4995970Z /Users/runner/work/1/s/.packages/microsoft.dotnet.helix.sdk/7.0.0-beta.23313.4/tools/Microsoft.DotNet.Helix.Sdk.MultiQueue.targets(90,5): error : Failure log: https://helix.dot.net/api/2019-06-17/jobs/9199670f-aa4c-4e8e-8150-b650ad0bb59f/workitems/Common.Tests/console [/Users/runner/work/1/s/src/libraries/sendtohelixhelp.proj]
2023-07-11T00:12:22.5108980Z ##[error].packages/microsoft.dotnet.helix.sdk/7.0.0-beta.23313.4/tools/Microsoft.DotNet.Helix.Sdk.MultiQueue.targets(90,5): error : (NETCORE_ENGINEERING_TELEMETRY=Test) Work item Common.Tests in job 9199670f-aa4c-4e8e-8150-b650ad0bb59f has failed.
Failure log: https://helix.dot.net/api/2019-06-17/jobs/9199670f-aa4c-4e8e-8150-b650ad0bb59f/workitems/Common.Tests/console
2023-07-11T00:12:22.5122420Z /Users/runner/work/1/s/.packages/microsoft.dotnet.helix.sdk/7.0.0-beta.23313.4/tools/Microsoft.DotNet.Helix.Sdk.MultiQueue.targets(90,5): error : Work item IcuAppLocal.Tests in job 9199670f-aa4c-4e8e-8150-b650ad0bb59f has failed. [/Users/runner/work/1/s/src/libraries/sendtohelixhelp.proj]
2023-07-11T00:12:22.5126220Z /Users/runner/work/1/s/.packages/microsoft.dotnet.helix.sdk/7.0.0-beta.23313.4/tools/Microsoft.DotNet.Helix.Sdk.MultiQueue.targets(90,5): error : Failure log: https://helix.dot.net/api/2019-06-17/jobs/9199670f-aa4c-4e8e-8150-b650ad0bb59f/workitems/IcuAppLocal.Tests/console [/Users/runner/work/1/s/src/libraries/sendtohelixhelp.proj]

@ghost
Copy link

ghost commented Jul 21, 2023

Tagging subscribers to this area: @dotnet/runtime-infrastructure
See info in area-owners.md if you want to be subscribed.

Issue Details

Build Information

Build: https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=300158
Build error leg or test failing: System.IO.FileSystem.Watcher.Tests.WorkItemExecution
Pull request: #86974

Error Message

Fill the error message using step by step known issues guidance.

{
  "ErrorMessage": "Work item System.IO.FileSystem.Watcher.Tests in job",
  "ErrorPattern": "",
  "BuildRetry": false,
  "ExcludeConsoleLog": false
}

Report

Build Definition Step Name Console log Pull Request
347502 dotnet/runtime Send to Helix Log
347514 dotnet/runtime Send to Helix Log
347516 dotnet/runtime Send to Helix Log
347506 dotnet/runtime Send to Helix Log
347504 dotnet/runtime Send to Helix Log
347527 dotnet/runtime Send to Helix Log
346179 dotnet/runtime Send to Helix Log
346184 dotnet/runtime Send to Helix Log
346183 dotnet/runtime Send to Helix Log
346189 dotnet/runtime Send to Helix Log
346180 dotnet/runtime Send to Helix Log
346188 dotnet/runtime Send to Helix Log
345385 dotnet/runtime Send to Helix Log #89145
344799 dotnet/runtime Send to Helix Log
344802 dotnet/runtime Send to Helix Log
344773 dotnet/runtime Send to Helix Log
344775 dotnet/runtime Send to Helix Log
344801 dotnet/runtime Send to Helix Log
344791 dotnet/runtime Send to Helix Log
344313 dotnet/runtime Send to Helix Log #89145
344014 dotnet/runtime Send to Helix Log #89061
343534 dotnet/runtime Send to Helix Log
343456 dotnet/runtime Send to Helix Log #89074
343307 dotnet/runtime Send to Helix Log
343300 dotnet/runtime Send to Helix Log
343299 dotnet/runtime Send to Helix Log
343301 dotnet/runtime Send to Helix Log
343298 dotnet/runtime Send to Helix Log
343296 dotnet/runtime Send to Helix Log
343319 dotnet/runtime Send to Helix Log #89074
342971 dotnet/runtime Send to Helix Log
342902 dotnet/runtime Send to Helix Log #87847
342704 dotnet/runtime Send to Helix Log #88974
342426 dotnet/runtime Send to Helix Log
342101 dotnet/runtime Send to Helix Log
341764 dotnet/runtime Send to Helix Log
341747 dotnet/runtime Send to Helix Log
341763 dotnet/runtime Send to Helix Log
341766 dotnet/runtime Send to Helix Log
341752 dotnet/runtime Send to Helix Log
341745 dotnet/runtime Send to Helix Log
341422 dotnet/runtime Send to Helix Log #88502
341117 dotnet/runtime Send to Helix Log
341130 dotnet/runtime Send to Helix Log
341115 dotnet/runtime Send to Helix Log
341116 dotnet/runtime Send to Helix Log
341137 dotnet/runtime Send to Helix Log
341114 dotnet/runtime Send to Helix Log
340730 dotnet/runtime Send to Helix Log
340706 dotnet/runtime Send to Helix Log
340711 dotnet/runtime Send to Helix Log
340713 dotnet/runtime Send to Helix Log
340722 dotnet/runtime Send to Helix Log
340731 dotnet/runtime Send to Helix Log
340606 dotnet/runtime Send to Helix Log #74623
339887 dotnet/runtime Send to Helix Log
339546 dotnet/runtime Send to Helix Log
339567 dotnet/runtime Send to Helix Log
339550 dotnet/runtime Send to Helix Log
339557 dotnet/runtime Send to Helix Log
339565 dotnet/runtime Send to Helix Log
339554 dotnet/runtime Send to Helix Log
339453 dotnet/runtime Send to Helix Log #88892
339194 dotnet/runtime Send to Helix Log #74623
339190 dotnet/runtime Send to Helix Log
338975 dotnet/runtime Send to Helix Log
338958 dotnet/runtime Send to Helix Log
338949 dotnet/runtime Send to Helix Log
338278 dotnet/runtime Send to Helix Log
338279 dotnet/runtime Send to Helix Log
338284 dotnet/runtime Send to Helix Log
338258 dotnet/runtime Send to Helix Log
338277 dotnet/runtime Send to Helix Log
338291 dotnet/runtime Send to Helix Log
338281 dotnet/runtime Send to Helix Log
337928 dotnet/runtime Send to Helix Log
337971 dotnet/runtime Send to Helix Log
336905 dotnet/runtime Send to Helix Log
336900 dotnet/runtime Send to Helix Log
336626 dotnet/runtime Send to Helix Log
336635 dotnet/runtime Send to Helix Log
336627 dotnet/runtime Send to Helix Log
336647 dotnet/runtime Send to Helix Log
336638 dotnet/runtime Send to Helix Log
336646 dotnet/runtime Send to Helix Log
336353 dotnet/runtime Send to Helix Log
336347 dotnet/runtime Send to Helix Log
335895 dotnet/runtime Send to Helix Log #88626
335021 dotnet/runtime Send to Helix Log
335024 dotnet/runtime Send to Helix Log
335003 dotnet/runtime Send to Helix Log
335004 dotnet/runtime Send to Helix Log
334953 dotnet/runtime Send to Helix Log
335012 dotnet/runtime Send to Helix Log
335002 dotnet/runtime Send to Helix Log
334959 dotnet/runtime Send to Helix Log
334772 dotnet/runtime Send to Helix Log
334717 dotnet/runtime Send to Helix Log
334699 dotnet/runtime Send to Helix Log
334444 dotnet/runtime Send to Helix Log
Displaying 100 of 262 results

Summary

24-Hour Hit Count 7-Day Hit Count 1-Month Count
6 55 262

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=323604
Error message validated: Work item System.IO.FileSystem.Watcher.Tests in job
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 6/28/2023 9:26:54 PM UTC

Author: eerhardt
Assignees: -
Labels:

blocking-clean-ci, area-Infrastructure, untriaged, Known Build Error

Milestone: -

@carlossanlop
Copy link
Member

carlossanlop commented Jul 21, 2023

Yea, doesn't seem to be a FileSystemWatcher-only issue. The jobs are getting unexpectedly killed, according to the console log outputs. Example:

2023-07-11T00:05:26.562Z	INFO   	run.py	run(48)	main	Beginning reading of test results.
2023-07-11T00:05:26.563Z	INFO   	run.py	__init__(42)	read_results	Searching '/private/tmp/helix/working/A8D4097B/w/9393087D/e' for test results files
2023-07-11T00:05:26.563Z	INFO   	run.py	__init__(42)	read_results	Searching '/tmp/helix/working/A8D4097B/w/9393087D/uploads' for test results files
2023-07-11T00:05:26.563Z	WARNING	run.py	__init__(55)	read_results	No results file found in any of the following formats: xunit, junit, trx
2023-07-11T00:05:26.563Z	INFO   	run.py	packing_test_reporter(30)	report_results	Packing 0 test reports to '/tmp/helix/working/A8D4097B/w/9393087D/e/__test_report.json'
2023-07-11T00:05:26.564Z	INFO   	run.py	packing_test_reporter(33)	report_results	Packed 1523 bytes
+ exit 127
['Microsoft.Bcl.AsyncInterfaces.Tests' END OF WORK ITEM LOG: Command exited with 127]

@carlossanlop carlossanlop added Known Build Error Use this to report build issues in the .NET Helix tab and removed Known Build Error Use this to report build issues in the .NET Helix tab labels Jul 21, 2023
@jozkee jozkee changed the title System.IO.FileSystem.Watcher.Tests crash on osx x64 Debug Work item tests crash on OSX x64 Debug Jul 22, 2023
@akoeplinger
Copy link
Member

akoeplinger commented Nov 20, 2023

A couple observations from looking through a couple results:

  1. Only happens on the 7.0 branches
  2. Only happens on runtime-libraries-coreclr outerloop pipelines, across all OSes
  3. The test doesn't get killed, instead it fails because dotnet couldn't be found:

./RunTests.sh: line 168: /tmp/helix/working/B2D409A4/p/dotnet: No such file or directory

I downloaded the test-runtime-net7.0-OSX-Release-arm64.zip correlation payload and it is indeed missing the dotnet binary and host, it only contains the shared framework.

@akoeplinger
Copy link
Member

The 7.0 branch is missing this fix: #79652

@ghost ghost added the in-pr There is an active PR which will close this issue when it is merged label Nov 20, 2023
akoeplinger added a commit that referenced this issue Nov 20, 2023
@ghost ghost removed the in-pr There is an active PR which will close this issue when it is merged label Nov 20, 2023
@akoeplinger
Copy link
Member

Fixed by #95011

@ghost ghost removed the untriaged New issue has not been triaged by the area owner label Nov 20, 2023
@github-actions github-actions bot locked and limited conversation to collaborators Dec 21, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
area-Infrastructure blocking-clean-ci Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms' Known Build Error Use this to report build issues in the .NET Helix tab
Projects
None yet
Development

No branches or pull requests

4 participants