Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make sure process parameters are correctly quoted #10676

Merged
merged 2 commits into from
Oct 1, 2024

Conversation

ffried
Copy link
Contributor

@ffried ffried commented Sep 26, 2024

What are you trying to accomplish?

With #9678, the nuget updated gained support for package lock files.
However, it fails when the csproj path contains spaces (see #9678 (comment)).

This now changes the ProcessEx helper to accept a list of args (instead of just a string). This should fix this issue for the lock file update as well as several other places.

Anything you want to highlight for special attention from reviewers?

My initial thought was to simply escape the strings myself, but ProcessStartInfo has a constructor that also takes an argument list, so I opted to use that one and leave the escaping to the system.
Also, I didn't just want to fix the lock file updater, because I found other places where paths (which can possibly contains spaces) weren't correctly escaped. With that API change, it should no longer be an issue, also for new usages of the ProcessEx helper.

How will you know you've accomplished your goal?

I've updated all the call sites of the old extension by passing the args as array. One was even creating an array and joining it with a string.

Checklist

  • I have run the complete test suite to ensure all tests and linters pass.
  • I have thoroughly tested my code changes to ensure they work as expected, including adding additional tests for new functionality.
  • I have written clear and descriptive commit messages.
  • I have provided a detailed description of the changes in the pull request, including the problem it addresses, how it fixes the problem, and any relevant details about the implementation.
  • I have ensured that the code is well-documented and easy to understand.

Unfortunately, I was unable to run the tests. Starting the development shell fails with the following error:

E: Unable to locate package powershell

@ffried ffried requested a review from a team as a code owner September 26, 2024 06:56
@github-actions github-actions bot added the L: dotnet:nuget NuGet packages via nuget or dotnet label Sep 26, 2024
@brettfo
Copy link
Contributor

brettfo commented Sep 26, 2024

Good catch!

@ffried
Copy link
Contributor Author

ffried commented Sep 26, 2024

@brettfo Thanks for the review. Do I need to do anything else to get this merged?

@brettfo
Copy link
Contributor

brettfo commented Sep 26, 2024

@ffried

anything else to get this merged?

You're good from your end, I'm talking to folks on my side to get this merged. Should be soon.

Starting the development shell fails with the following error:
E: Unable to locate package powershell

What was your process to see this error? I suspect it's related to #10521 that was merged yesterday that adds a PowwerShell dependency to the NuGet updater image, but only in certain circumstances. Building the dev shell with bin/docker-dev-shell nuget --rebuild succeeds for me and even has PowerShell installed.

@ffried
Copy link
Contributor Author

ffried commented Sep 26, 2024

@brettfo

You're good from your end, I'm talking to folks on my side to get this merged. Should be soon.

Cool, thanks!

What was your process to see this error? I suspect it's related to #10521 that was merged yesterday that adds a PowwerShell dependency to the NuGet updater image, but only in certain circumstances. Building the dev shell with bin/docker-dev-shell nuget --rebuild succeeds for me and even has PowerShell installed.

Yea, I just ran bin/docker-dev-shell nuget and it started building the docker image, eventually failing with the aforementioned error.
I just tried it again with --rebuild leading to the same error:

=> [ 2/15] RUN source /etc/os-release  && curl --location --output /tmp/packages-microsoft-prod.deb "https://packages.microsoft.com/config/ubuntu/$VERSION_ID/packages-microsoft-prod.deb"  && d  0.6s
 => ERROR [ 3/15] RUN apt-get update  && apt-get install -y --no-install-recommends     jq     libicu-dev=70.1-2     powershell  && rm -rf /var/lib/apt/lists/*                                    4.2s 
------                                                                                                                                                                                                  
 > [ 3/15] RUN apt-get update  && apt-get install -y --no-install-recommends     jq     libicu-dev=70.1-2     powershell  && rm -rf /var/lib/apt/lists/*:                                               
0.187 Get:1 https://packages.microsoft.com/ubuntu/22.04/prod jammy InRelease [3,632 B]                                                                                                                  
0.244 Get:2 https://packages.microsoft.com/ubuntu/22.04/prod jammy/main all Packages [1,261 B]                                                                                                          
0.258 Get:3 http://ports.ubuntu.com/ubuntu-ports jammy InRelease [270 kB]                                                                                                                               
0.270 Get:4 https://packages.microsoft.com/ubuntu/22.04/prod jammy/main amd64 Packages [173 kB]                                                                                                         
0.310 Get:5 https://packages.microsoft.com/ubuntu/22.04/prod jammy/main armhf Packages [15.6 kB]
0.312 Get:6 https://packages.microsoft.com/ubuntu/22.04/prod jammy/main arm64 Packages [43.5 kB]
0.401 Get:7 http://ports.ubuntu.com/ubuntu-ports jammy-updates InRelease [128 kB]
0.439 Get:8 http://ports.ubuntu.com/ubuntu-ports jammy-backports InRelease [127 kB]
0.483 Get:9 http://ports.ubuntu.com/ubuntu-ports jammy-security InRelease [129 kB]
0.521 Get:10 http://ports.ubuntu.com/ubuntu-ports jammy/universe arm64 Packages [17.2 MB]
1.692 Get:11 http://ports.ubuntu.com/ubuntu-ports jammy/restricted arm64 Packages [24.2 kB]
1.693 Get:12 http://ports.ubuntu.com/ubuntu-ports jammy/main arm64 Packages [1,758 kB]
1.755 Get:13 http://ports.ubuntu.com/ubuntu-ports jammy/multiverse arm64 Packages [224 kB]
1.764 Get:14 http://ports.ubuntu.com/ubuntu-ports jammy-updates/universe arm64 Packages [1,387 kB]
1.812 Get:15 http://ports.ubuntu.com/ubuntu-ports jammy-updates/main arm64 Packages [2,265 kB]
1.887 Get:16 http://ports.ubuntu.com/ubuntu-ports jammy-updates/restricted arm64 Packages [2,475 kB]
2.026 Get:17 http://ports.ubuntu.com/ubuntu-ports jammy-updates/multiverse arm64 Packages [29.5 kB]
2.083 Get:18 http://ports.ubuntu.com/ubuntu-ports jammy-backports/main arm64 Packages [110 kB]
2.135 Get:19 http://ports.ubuntu.com/ubuntu-ports jammy-backports/universe arm64 Packages [35.0 kB]
2.140 Get:20 http://ports.ubuntu.com/ubuntu-ports jammy-security/multiverse arm64 Packages [24.1 kB]
2.145 Get:21 http://ports.ubuntu.com/ubuntu-ports jammy-security/restricted arm64 Packages [2,405 kB]
2.426 Get:22 http://ports.ubuntu.com/ubuntu-ports jammy-security/universe arm64 Packages [1,104 kB]
2.430 Get:23 http://ports.ubuntu.com/ubuntu-ports jammy-security/main arm64 Packages [1,994 kB]
2.645 Fetched 31.9 MB in 3s (12.6 MB/s)
2.645 Reading package lists...
3.398 Reading package lists...
3.979 Building dependency tree...
4.098 Reading state information...
4.109 E: Unable to locate package powershell
------
Dockerfile:12
--------------------
  11 |      && rm /tmp/packages-microsoft-prod.deb
  12 | >>> RUN apt-get update \
  13 | >>>  && apt-get install -y --no-install-recommends \
  14 | >>>     jq \
  15 | >>>     libicu-dev=70.1-2 \
  16 | >>>     powershell \
  17 | >>>  && rm -rf /var/lib/apt/lists/*
  18 |     
--------------------
ERROR: failed to solve: process "/bin/bash -o pipefail -c apt-get update  && apt-get install -y --no-install-recommends     jq     libicu-dev=70.1-2     powershell  && rm -rf /var/lib/apt/lists/*" did not complete successfully: exit code: 100

@brettfo
Copy link
Contributor

brettfo commented Sep 26, 2024

@ffried I wonder if something is happening during the download and install of packages-microsoft-prod.deb. Can you run this and look up a few lines from your previous log snippet? I want to make sure that deb file is getting installed correctly:

DOCKER_BUILD_ARGS="--progress=plain" bin/docker-dev-shell nuget --rebuild

@brettfo brettfo mentioned this pull request Sep 26, 2024
5 tasks
@ffried
Copy link
Contributor Author

ffried commented Sep 26, 2024

@brettfo I just cleaned my build cache and ran the command you provided. It first builds the dependabot-updater-core image (successfully). Then it fails when building the dependabot-updater-nuget:

#0 building with "desktop-linux" instance using docker driver

#1 [internal] load build definition from Dockerfile
#1 transferring dockerfile: 2.06kB done
#1 DONE 0.0s

#2 [internal] load metadata for ghcr.io/dependabot/dependabot-updater-core:latest
#2 DONE 0.0s

#3 [internal] load .dockerignore
#3 transferring context: 398B done
#3 DONE 0.0s

#4 [internal] load build context
#4 DONE 0.0s

#5 [ 1/15] FROM ghcr.io/dependabot/dependabot-updater-core:latest@sha256:abfdbe2f038d34ae0a36da57403f679703e19b4279fa1da620780dfc0a21ab36
#5 resolve ghcr.io/dependabot/dependabot-updater-core:latest@sha256:abfdbe2f038d34ae0a36da57403f679703e19b4279fa1da620780dfc0a21ab36 done
#5 DONE 0.0s

#6 importing cache manifest from ghcr.io/dependabot/dependabot-updater-nuget
#6 inferred cache manifest type: application/vnd.docker.distribution.manifest.v2+json done
#6 DONE 0.8s

#4 [internal] load build context
#4 transferring context: 15.08MB 0.2s done
#4 DONE 0.2s

#5 [ 1/15] FROM ghcr.io/dependabot/dependabot-updater-core:latest@sha256:abfdbe2f038d34ae0a36da57403f679703e19b4279fa1da620780dfc0a21ab36
#5 DONE 0.2s

#7 [ 2/15] RUN source /etc/os-release  && curl --location --output /tmp/packages-microsoft-prod.deb "https://packages.microsoft.com/config/ubuntu/$VERSION_ID/packages-microsoft-prod.deb"  && dpkg -i /tmp/packages-microsoft-prod.deb  && rm /tmp/packages-microsoft-prod.deb
#7 0.091   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
#7 0.091                                  Dload  Upload   Total   Spent    Left  Speed
100  3692  100  3692    0     0  16451      0 --:--:-- --:--:-- --:--:-- 16408
#7 0.333 Selecting previously unselected package packages-microsoft-prod.
#7 0.346 (Reading database ... 15049 files and directories currently installed.)
#7 0.346 Preparing to unpack .../packages-microsoft-prod.deb ...
#7 0.348 Unpacking packages-microsoft-prod (1.0-ubuntu22.04.1) ...
#7 0.357 Setting up packages-microsoft-prod (1.0-ubuntu22.04.1) ...
#7 DONE 0.5s

#8 [ 3/15] RUN apt-get update  && apt-get install -y --no-install-recommends     jq     libicu-dev=70.1-2     powershell  && rm -rf /var/lib/apt/lists/*
#8 0.189 Get:1 https://packages.microsoft.com/ubuntu/22.04/prod jammy InRelease [3,632 B]
#8 0.213 Get:2 http://ports.ubuntu.com/ubuntu-ports jammy InRelease [270 kB]
#8 0.246 Get:3 https://packages.microsoft.com/ubuntu/22.04/prod jammy/main all Packages [1,261 B]
#8 0.271 Get:4 https://packages.microsoft.com/ubuntu/22.04/prod jammy/main amd64 Packages [173 kB]
#8 0.316 Get:5 https://packages.microsoft.com/ubuntu/22.04/prod jammy/main arm64 Packages [43.5 kB]
#8 0.319 Get:6 https://packages.microsoft.com/ubuntu/22.04/prod jammy/main armhf Packages [15.6 kB]
#8 0.357 Get:7 http://ports.ubuntu.com/ubuntu-ports jammy-updates InRelease [128 kB]
#8 0.394 Get:8 http://ports.ubuntu.com/ubuntu-ports jammy-backports InRelease [127 kB]
#8 0.430 Get:9 http://ports.ubuntu.com/ubuntu-ports jammy-security InRelease [129 kB]
#8 0.466 Get:10 http://ports.ubuntu.com/ubuntu-ports jammy/multiverse arm64 Packages [224 kB]
#8 0.493 Get:11 http://ports.ubuntu.com/ubuntu-ports jammy/universe arm64 Packages [17.2 MB]
#8 1.385 Get:12 http://ports.ubuntu.com/ubuntu-ports jammy/main arm64 Packages [1,758 kB]
#8 1.423 Get:13 http://ports.ubuntu.com/ubuntu-ports jammy/restricted arm64 Packages [24.2 kB]
#8 1.423 Get:14 http://ports.ubuntu.com/ubuntu-ports jammy-updates/restricted arm64 Packages [2,475 kB]
#8 1.544 Get:15 http://ports.ubuntu.com/ubuntu-ports jammy-updates/main arm64 Packages [2,265 kB]
#8 1.567 Get:16 http://ports.ubuntu.com/ubuntu-ports jammy-updates/universe arm64 Packages [1,387 kB]
#8 1.624 Get:17 http://ports.ubuntu.com/ubuntu-ports jammy-updates/multiverse arm64 Packages [29.5 kB]
#8 1.679 Get:18 http://ports.ubuntu.com/ubuntu-ports jammy-backports/main arm64 Packages [110 kB]
#8 1.733 Get:19 http://ports.ubuntu.com/ubuntu-ports jammy-backports/universe arm64 Packages [35.0 kB]
#8 1.739 Get:20 http://ports.ubuntu.com/ubuntu-ports jammy-security/multiverse arm64 Packages [24.1 kB]
#8 1.745 Get:21 http://ports.ubuntu.com/ubuntu-ports jammy-security/universe arm64 Packages [1,104 kB]
#8 1.870 Get:22 http://ports.ubuntu.com/ubuntu-ports jammy-security/restricted arm64 Packages [2,405 kB]
#8 2.141 Get:23 http://ports.ubuntu.com/ubuntu-ports jammy-security/main arm64 Packages [1,994 kB]
#8 2.309 Fetched 31.9 MB in 2s (14.6 MB/s)
#8 2.309 Reading package lists...
#8 2.879 Reading package lists...
#8 3.439 Building dependency tree...
#8 3.576 Reading state information...
#8 3.588 E: Unable to locate package powershell
#8 ERROR: process "/bin/bash -o pipefail -c apt-get update  && apt-get install -y --no-install-recommends     jq     libicu-dev=70.1-2     powershell  && rm -rf /var/lib/apt/lists/*" did not complete successfully: exit code: 100
------
 > [ 3/15] RUN apt-get update  && apt-get install -y --no-install-recommends     jq     libicu-dev=70.1-2     powershell  && rm -rf /var/lib/apt/lists/*:
1.679 Get:18 http://ports.ubuntu.com/ubuntu-ports jammy-backports/main arm64 Packages [110 kB]
1.733 Get:19 http://ports.ubuntu.com/ubuntu-ports jammy-backports/universe arm64 Packages [35.0 kB]
1.739 Get:20 http://ports.ubuntu.com/ubuntu-ports jammy-security/multiverse arm64 Packages [24.1 kB]
1.745 Get:21 http://ports.ubuntu.com/ubuntu-ports jammy-security/universe arm64 Packages [1,104 kB]
1.870 Get:22 http://ports.ubuntu.com/ubuntu-ports jammy-security/restricted arm64 Packages [2,405 kB]
2.141 Get:23 http://ports.ubuntu.com/ubuntu-ports jammy-security/main arm64 Packages [1,994 kB]



3.588 E: Unable to locate package powershell
------
Dockerfile:12
--------------------
  11 |      && rm /tmp/packages-microsoft-prod.deb
  12 | >>> RUN apt-get update \
  13 | >>>  && apt-get install -y --no-install-recommends \
  14 | >>>     jq \
  15 | >>>     libicu-dev=70.1-2 \
  16 | >>>     powershell \
  17 | >>>  && rm -rf /var/lib/apt/lists/*
  18 |     
--------------------
ERROR: failed to solve: process "/bin/bash -o pipefail -c apt-get update  && apt-get install -y --no-install-recommends     jq     libicu-dev=70.1-2     powershell  && rm -rf /var/lib/apt/lists/*" did not complete successfully: exit code: 100```

@sachin-sandhu
Copy link
Contributor

Hi @ffried , I think your fork seems out-of-date with dependabot core main branch, can you please trying rebasing with dependabot-core so that i can try to do a release.

@ffried
Copy link
Contributor Author

ffried commented Sep 26, 2024

@sachin-sandhu I just rebased both main and the branch for this PR.

@sachin-sandhu
Copy link
Contributor

@ffried , i can see some issues while testing #10676 (comment) ,i have deferred deploy and merge for now, Please let me know once you are good and we can resume.

@ffried
Copy link
Contributor Author

ffried commented Sep 27, 2024

@sachin-sandhu Not sure what I can do here. The failure seems completely unrelated to my changes.

@ffried
Copy link
Contributor Author

ffried commented Sep 27, 2024

@sachin-sandhu I'm also getting this error on main - so it's definitively not related to my changes.

@sachin-sandhu
Copy link
Contributor

sachin-sandhu commented Sep 27, 2024

HI @brettfo, Can you please provide inputs on #10676 (comment)

linking #9678 (comment) in case it is related.

cc @ffried

@brettfo
Copy link
Contributor

brettfo commented Sep 27, 2024

@sachin-sandhu I think both of those things are unrelated to these changes.

In the first case (unable to build Dockerfile): I can build it locally and the CI is passing which means it can be built on more than my machine. There must be some weird caching issue with the other user's Docker instance, but I can't fathom what it could possibly be.

In the second case, that looks like we're trying to update a file that we didn't report earlier, but from reading the code that should be impossible: we only check things previously reported as a dependency_file and I've even added a smoke-test to verify that package.lock.json files work: dependabot/smoke-tests#231

In short, I don't see any blockers to this getting merged and deployed.

@ffried
Copy link
Contributor Author

ffried commented Sep 27, 2024

@brettfo @sachin-sandhu One thing that just came to my mind is that I'm working on an Apple Silicon MacBook Pro. So it's ARM architecture. Maybe that's the issue with the Docker build.

@brettfo
Copy link
Contributor

brettfo commented Sep 27, 2024

@brettfo @sachin-sandhu One thing that just came to my mind is that I'm working on an Apple Silicon MacBook Pro. So it's ARM architecture. Maybe that's the issue with the Docker build.

Good call, that's exactly it. I checked the feed and there is no arm64 version of the powershell package, just amd64 which is what I'm using locally and the images used in production.

Edit: it appears there is a .tar.gz binary release for arm64, but not an installable .deb file. I'll see how difficult it would be to detect the architecture in the Dockerfile and unpack the binaries in that case.

@ffried
Copy link
Contributor Author

ffried commented Sep 27, 2024

Edit: it appears there is a .tar.gz binary release for arm64, but not an installable .deb file. I'll see how difficult it would be to detect the architecture in the Dockerfile and unpack the binaries in that case.

One option could be to use the predefined build arguments to determine the arch: https://docs.docker.com/build/building/variables/#multi-platform-build-arguments

@brettfo
Copy link
Contributor

brettfo commented Sep 27, 2024

@ffried I've filed issue #10689 to track enabling arm64 support for the updater image.

@brettfo
Copy link
Contributor

brettfo commented Oct 1, 2024

@ffried We're getting ready to merge this, could you please update your fork/branch to the latest in main?

@ffried
Copy link
Contributor Author

ffried commented Oct 1, 2024

@brettfo Done

@kbukum1 kbukum1 merged commit 073100c into dependabot:main Oct 1, 2024
51 checks passed
@ffried ffried deleted the space-for-everyone branch October 2, 2024 05:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
L: dotnet:nuget NuGet packages via nuget or dotnet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants