Add debug flag to arm deployment command #1188

benbp · 2020-11-10T23:40:53Z

There are some ARM deployment errors that do not surface adequate messages for debugging in the default error output. For example issues related to template validation (that can be a result of account quotas, invalid names, etc.):

WARNING: Attempt 1 failed: Exception calling "Invoke" with "0" argument(s): "The running command stopped because the preference variable "ErrorActionPreference" or common parameter is set to Stop:
07:13:02 - Error: Code=InvalidTemplateDeployment; Message=The template deployment 'benbp-foobar' is not valid according to the validation procedure. The tracking id is '182f1217-fa54-4fa8-bff3-ef74605e2fe9'. See inner errors for details.

By updating the $DebugPreference for the New-AzResourceGroupDeployment command, we print the inner error messages without having to plug correlation IDs into the ARM internal kusto to get the actual error.

DEBUG: ============================ HTTP RESPONSE ============================

Status Code:
BadRequest

Headers:
Cache-Control                 : no-cache
Pragma                        : no-cache
x-ms-failure-cause            : gateway
x-ms-ratelimit-remaining-subscription-writes: 1199
x-ms-request-id               : 11a8a92c-21ba-43dc-b504-84b92524c71b
x-ms-correlation-request-id   : 11a8a92c-21ba-43dc-b504-84b92524c71b
x-ms-routing-request-id       : EASTUS:20201110T225406Z:11a8a92c-21ba-43dc-b504-84b92524c71b
Strict-Transport-Security     : max-age=31536000; includeSubDomains
X-Content-Type-Options        : nosniff
Date                          : Tue, 10 Nov 2020 22:54:06 GMT

Body:
{
  "error": {
    "code": "InvalidTemplateDeployment",
    "message": "The template deployment 'benbp-foobar' is not valid according to the validation procedure. The tracking id is '11a8a92c-21ba-43dc-b504-84b92524c71b'. See inner errors for details.",
    "details": [
      {
        "code": "PreflightValidationCheckFailed",
        "message": "Preflight validation failed. Please refer to the details for the specific errors.",
        "details": [
          {
            "code": "AccountNameInvalid",
            "target": "benbp-basenameprim",
            "message": "benbp-basenameprim is not a valid storage account name. Storage account name must be between 3 and 24 characters in length and use numbers and lower-case letters only."
          },

An example full log with failures can be found here: https://dev.azure.com/azure-sdk/internal/_build/results?buildId=614171&view=logs&j=011e1ec8-6569-5e69-4f06-baf193d1351e&t=5431112d-2b61-5a5f-7042-ef698f761043

benbp · 2020-11-11T00:05:45Z

I could also have used Test-AzResourceGroupDeployment, but I like that this prints out the whole ARM template too for debugging (with secret values removed).

azure-sdk · 2020-11-11T00:08:40Z

The following pipelines have been queued for testing:
java - template
js - template
net - template
python - template
You can sign off on the approval gate to test the release stage of each pipeline.
See eng/common workflow

eng/common/TestResources/New-TestResources.ps1

azure-sdk · 2020-11-11T01:04:29Z

The following pipelines have been queued for testing:
java - template
js - template
net - template
python - template
You can sign off on the approval gate to test the release stage of each pipeline.
See eng/common workflow

eng/common/TestResources/New-TestResources.ps1

azure-sdk · 2020-11-11T01:52:22Z

The following pipelines have been queued for testing:
java - template
js - template
net - template
python - template
You can sign off on the approval gate to test the release stage of each pipeline.
See eng/common workflow

ghost · 2020-11-11T21:36:20Z

Hello @azure-sdk!

Because this pull request has the auto-merge label, I will be glad to assist with helping to merge this pull request once all check-in policies pass.

p.s. you can customize the way I help with merging this pull request, such as holding this pull request until a specific person approves. Simply @mention me (`@msftbot`) and give me an instruction to get started! Learn more here.

weshaggard · 2020-11-11T23:52:43Z

eng/common/TestResources/New-TestResources.ps1

@@ -385,7 +385,15 @@ foreach ($templateFile in $templateFiles) {

    Log "Deploying template '$templateFile' to resource group '$($resourceGroup.ResourceGroupName)'"
    $deployment = Retry {
-        New-AzResourceGroupDeployment -Name $BaseName -ResourceGroupName $resourceGroup.ResourceGroupName -TemplateFile $templateFile -TemplateParameterObject $templateFileParameters
+        $lastDebugPreference = $DebugPreference
+        try {


Why can't you simply pass -Debug do this cmdlet? Doesn't that equate to the same thing without the need for the try/finally block?

In my testing, adding -Debug sets $DebugPreference to "Inquire" for the rest of the script context. This does two undesired things: we don't want debug output for the rest of the script, and the Inquire setting requires manual confirmation, as opposed to "Continue".

https://docs.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_commonparameters?view=powershell-7.1#debug

That seems to suggest that it will set it to Continue in non-interactive mode. I'm not even sure what Inquire does.

It seems weird that it isn't scoped to just the one cmdlet call but if that is the case what you have works.

weshaggard · 2020-11-11T23:55:04Z

This is definitely an improvement does it also make sense to provide a link to the portal where we can see this deployment failure? Or is all that failure information included in the error message?

benbp · 2020-11-12T00:16:06Z

This is definitely an improvement does it also make sense to provide a link to the portal where we can see this deployment failure? Or is all that failure information included in the error message?

All the failure information is included in the error, based on comparing what gets dumped out vs. what shows up in properties in the ARM kusto log. The portal activity log actually DOESN'T provide this level of information, frustratingly enough. While activity logs are queryable via a command, I couldn't find a way to parameterize the kusto queries by URL, but we could pass a link to some of the wiki debugging steps I added at https://dev.azure.com/azure-sdk/internal/_wiki/wikis/internal.wiki/51/Testing-Guidelines?anchor=debugging-live-tests ?

weshaggard · 2020-11-12T00:31:52Z

Yeah maybe that link to more info would be good. However we should set up an aka.ms link if we do that.

benbp · 2020-11-12T01:43:57Z

Yeah maybe that link to more info would be good. However we should set up an aka.ms link if we do that.

@weshaggard #1202

There are some ARM deployment errors that do not surface adequate messages for debugging in the default error output. For example issues related to template validation (that can be a result of account quotas, invalid names, etc.): ``` WARNING: Attempt 1 failed: Exception calling "Invoke" with "0" argument(s): "The running command stopped because the preference variable "ErrorActionPreference" or common parameter is set to Stop: 07:13:02 - Error: Code=InvalidTemplateDeployment; Message=The template deployment 'benbp-foobar' is not valid according to the validation procedure. The tracking id is '182f1217-fa54-4fa8-bff3-ef74605e2fe9'. See inner errors for details. ``` By updating the `$DebugPreference` for the `New-AzResourceGroupDeployment` command, we print the inner error messages without having to plug correlation IDs into the ARM internal kusto to get the actual error. ``` DEBUG: ============================ HTTP RESPONSE ============================ Status Code: BadRequest Headers: Cache-Control : no-cache Pragma : no-cache x-ms-failure-cause : gateway x-ms-ratelimit-remaining-subscription-writes: 1199 x-ms-request-id : 11a8a92c-21ba-43dc-b504-84b92524c71b x-ms-correlation-request-id : 11a8a92c-21ba-43dc-b504-84b92524c71b x-ms-routing-request-id : EASTUS:20201110T225406Z:11a8a92c-21ba-43dc-b504-84b92524c71b Strict-Transport-Security : max-age=31536000; includeSubDomains X-Content-Type-Options : nosniff Date : Tue, 10 Nov 2020 22:54:06 GMT Body: { "error": { "code": "InvalidTemplateDeployment", "message": "The template deployment 'benbp-foobar' is not valid according to the validation procedure. The tracking id is '11a8a92c-21ba-43dc-b504-84b92524c71b'. See inner errors for details.", "details": [ { "code": "PreflightValidationCheckFailed", "message": "Preflight validation failed. Please refer to the details for the specific errors.", "details": [ { "code": "AccountNameInvalid", "target": "benbp-basenameprim", "message": "benbp-basenameprim is not a valid storage account name. Storage account name must be between 3 and 24 characters in length and use numbers and lower-case letters only." }, ``` An example full log with failures can be found here: https://dev.azure.com/azure-sdk/internal/_build/results?buildId=614171&view=logs&j=011e1ec8-6569-5e69-4f06-baf193d1351e&t=5431112d-2b61-5a5f-7042-ef698f761043

benbp added 2 commits November 10, 2020 07:59

Add debug flag to arm deployment command

f683e60

Only set debug preference when $CI is true

dc567c8

benbp requested review from danieljurek, heaths and weshaggard as code owners November 10, 2020 23:40

heaths requested changes Nov 11, 2020

View reviewed changes

eng/common/TestResources/New-TestResources.ps1 Outdated Show resolved Hide resolved

benbp requested a review from heaths November 11, 2020 00:40

heaths approved these changes Nov 11, 2020

View reviewed changes

eng/common/TestResources/New-TestResources.ps1 Outdated Show resolved Hide resolved

azure-sdk added the auto-merge label Nov 11, 2020

ghost merged commit b9c55de into Azure:master Nov 11, 2020

benbp deleted the benbp/debug-template-deployment branch November 11, 2020 21:44

weshaggard reviewed Nov 11, 2020

View reviewed changes

This pull request was closed.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add debug flag to arm deployment command #1188

Add debug flag to arm deployment command #1188

benbp commented Nov 10, 2020

benbp commented Nov 11, 2020

azure-sdk commented Nov 11, 2020

azure-sdk commented Nov 11, 2020

azure-sdk commented Nov 11, 2020

ghost commented Nov 11, 2020

weshaggard Nov 11, 2020

benbp Nov 12, 2020

weshaggard Nov 12, 2020

weshaggard commented Nov 11, 2020

benbp commented Nov 12, 2020

weshaggard commented Nov 12, 2020

benbp commented Nov 12, 2020

Add debug flag to arm deployment command #1188

Add debug flag to arm deployment command #1188

Conversation

benbp commented Nov 10, 2020

benbp commented Nov 11, 2020

azure-sdk commented Nov 11, 2020

azure-sdk commented Nov 11, 2020

azure-sdk commented Nov 11, 2020

ghost commented Nov 11, 2020

p.s. you can customize the way I help with merging this pull request, such as holding this pull request until a specific person approves. Simply @mention me (@msftbot) and give me an instruction to get started! Learn more here.

weshaggard Nov 11, 2020

Choose a reason for hiding this comment

benbp Nov 12, 2020

Choose a reason for hiding this comment

weshaggard Nov 12, 2020

Choose a reason for hiding this comment

weshaggard commented Nov 11, 2020

benbp commented Nov 12, 2020

weshaggard commented Nov 12, 2020

benbp commented Nov 12, 2020

p.s. you can customize the way I help with merging this pull request, such as holding this pull request until a specific person approves. Simply @mention me (`@msftbot`) and give me an instruction to get started! Learn more here.