Cannot connect to S3 storage backend using IAM role

[jacobtomlinson]

I'm running vault 0.7.3 using the official docker container. The ec2 instance the container is running on has an IAM role allowing access to an S3 bucket. When starting vault I get a 400 Bad Request error and vault fails to start.

Error initializing storage of type s3: unable to access bucket 'redacted-bucket-name': BadRequest: Bad Request

Vault config

storage "s3" {
  bucket     = "redacted-bucket-name"
}

listener "tcp" {
  address     = "127.0.0.1:8200"
  tls_disable = 1
}

default_lease_ttl = "168h",
max_lease_ttl = "720h"

IAM Role Config

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "s3:ListAllMyBuckets",
      "Resource": "arn:aws:s3:::*"
    },
    {
      "Effect": "Allow",
      "Action": "s3:*",
      "Resource": [
        "arn:aws:s3:::redacted-bucket-namet",
        "arn:aws:s3:::redacted-bucket-name/*"
      ]
    }
  ]
}

[AndriMar]

Same here, using Vault v0.7.3 on EC2 instance.

Vault config

storage "s3" {
  access_key = ""
  secret_key = ""
  bucket     = "my-vault-secrets"
}
listener "tcp" {
  address = "127.0.0.1:8200"
  tls_disable = 1
}

IAM Role Config

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["s3:ListBucket"],
      "Resource": ["arn:aws:s3:::my-vault-secrets"]
    },
    {
      "Effect": "Allow",
      "Action": [
        "s3:PutObject",
        "s3:GetObject",
        "s3:DeleteObject"
      ],
      "Resource": ["arn:aws:s3:::my-vault-secrets/*"]
    }
  ]
}

Installed the aws-cli on the machine and aws s3 ls s3://my-vault-secrets and aws s3 cp file.txt s3://my-vault-secrets works so the IAM Role Config should be ok.

[jefferai]

Does 0.7.2 work okay? Could be a regression in the official aws library if so.

[AndriMar]

@jefferai I did a test with binary versions 0.7.0, 0.7.1, 0.7.2 and 0.7.3 same result BadRequest

[AndriMar]

I removed the IAM role from the machine and set my personal access_key and secret_key that has AdministratorAccess policy and still getting Error initializing storage of type s3: unable to access bucket 'my-vault-secrets': BadRequest: Bad Request

Edit:
Added the region option to the server config and then I get another message: Forbidden: Forbidden status code: 403
And with out region I get BadRequest: Bad Request status code: 400.
And still using the IAM role with AdministratorAccess policy and aws s3 ls s3://my-vault-secrets and aws s3 cp file.txt s3://my-vault-secrets works so Forbidden is a strange message except if it cant find the bucket, but that would be strange.

[AndriMar]

This was not an problem with the code, at least for me.
First thing is that I was missing one character in the end of my access_key 💀 after finding that out I was still getting Bad Request and after adding loghandler to aws I saw this 20170619/us-east-1/s3/aws4_reques and added the region to the config and all worked. 👍

But after this I have two suggestions.

1. Have region required in config or env.

	region := os.Getenv("AWS_DEFAULT_REGION")
	if region == "" {
		region = conf["region"]
		if region == "" {
			return nil, fmt.Errorf("'region' must be set")
		}
	}

2. Add check to see if the bucket can be found.

 result, err := s3conn.ListBuckets(nil)
	if err != nil {
		fmt.Println("Failed to list buckets", err)
	}
	bucketFound := false
	for _, b := range result.Buckets {
		if aws.StringValue(b.Name) == bucket {
			bucketFound = true
		}
	}
	if !bucketFound {
		return nil, fmt.Errorf("Cant find bucket '%s' in region '%s'", bucket, region)
	}

@jefferai what do you think ?

[jefferai]

Hi Andri,

I would have thought the IAM credentials would specify the region that should be used but I guess not. I'm in favor of option 2, not option 1, but: does Vault create the buckets in the first place? If not, would they then get an error merely because they're trying to run for the first time?

[AndriMar]

No the vault is not creating the bucket and I think that is the right way to go about this.
If it was to create the bucket you would need to give the IAM more access than it really need.
Creating a IAM with read and write to a single vault bucket is better in my opinion, but it requires the bucket to exist.
It might be a good idea to document and give some nice examples how the policy should look like and maybe some aws-cli commands, but that's another issue. :)

I think it is really strange the errors from the 's3conn.HeadBucket()' command.
If you put some random bucket name that does not exist it gives you 'NotFound'.
But if it does exist but you put the wrong region it gives you 'BadRequest'.

Option 2 would cover this strange error messages from aws, but little changes are needed because the IAM should not need to have the access to list all buckets.

[AndriMar]

It ended to be really simple, use 'ListObjects' when validating bucket not 'HeadBucket' it is probably bigger request but the error messages are much better.
If the bucket dose not exist we get:
Error initializing storage of type s3: unable to access bucket 'my-vault-secrets2' in region us-east-1: NoSuchBucket: The specified bucket does not exist status code: 404
If it is the wrong region we get:
Error initializing storage of type s3: unable to access bucket 'my-vault-secrets' in region us-east-1: AuthorizationHeaderMalformed: The authorization header is malformed; the region 'us-east-1' is wrong; expecting 'eu-west-1' status code: 400
We even get wat region it is expecting :)

[jefferai]

Cool. Can you PR?

[AndriMar]

Done: #2892

[AndriMar]

Travis is failing because of cassandra_test.go:220: err: error verifying connection 😞

[jefferai]

Yeah, the cassandra tests are really hit and miss for some reason, having to do with how Docker is feeling on Travis at any given time. No worries about that :-)

[jefferai]

Closed via #2892

[subramanianravi]

Whats was the answer for IAM role ? Is there any resolution ? I am not using docker but stable version of vault 0.7.2

[subramanianravi]

No worries seems like its looking for region :) all good now.