Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(iot): add Action to put objects in S3 Buckets #17307

Merged
merged 10 commits into from
Nov 10, 2021
2 changes: 2 additions & 0 deletions package.json
Original file line number Diff line number Diff line change
Expand Up @@ -107,6 +107,8 @@
"@aws-cdk/aws-eks/yaml/**",
"@aws-cdk/aws-events-targets/aws-sdk",
"@aws-cdk/aws-events-targets/aws-sdk/**",
"@aws-cdk/aws-iot-actions/case",
"@aws-cdk/aws-iot-actions/case/**",
"@aws-cdk/aws-s3-deployment/case",
"@aws-cdk/aws-s3-deployment/case/**",
"@aws-cdk/cloud-assembly-schema/jsonschema",
Expand Down
30 changes: 30 additions & 0 deletions packages/@aws-cdk/aws-iot-actions/NOTICE
Original file line number Diff line number Diff line change
@@ -1,2 +1,32 @@
AWS Cloud Development Kit (AWS CDK)
Copyright 2018-2021 Amazon.com, Inc. or its affiliates. All Rights Reserved.

-------------------------------------------------------------------------------

The AWS CDK includes the following third-party software/licensing:

** case - https://www.npmjs.com/package/case
Copyright (c) 2013 Nathan Bubna

Permission is hereby granted, free of charge, to any person
obtaining a copy of this software and associated documentation
files (the "Software"), to deal in the Software without
restriction, including without limitation the rights to use,
copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the
Software is furnished to do so, subject to the following
conditions:

The above copyright notice and this permission notice shall be
included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES
OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
OTHER DEALINGS IN THE SOFTWARE.

----------------
19 changes: 19 additions & 0 deletions packages/@aws-cdk/aws-iot-actions/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,25 @@ new iot.TopicRule(this, 'TopicRule', {
});
```

## Put objects to a S3 bucket

The code snippet below creates an AWS IoT Rule that put objects to a S3 bucket
when it is triggered.

```ts
import * as iot from '@aws-cdk/aws-iot';
import * as actions from '@aws-cdk/aws-iot-actions';
import * as s3 from '@aws-cdk/aws-s3';

const bucket = new s3.Bucket(this, 'MyBucket');

new iot.TopicRule(this, 'TopicRule', {
sql: iot.IotSql.fromStringAsVer20160323("SELECT topic(2) as device_id FROM 'device/+/data'"),
actions: [new actions.S3Action(bucket)],
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you illustrate the optional properties of S3Action in this example? Especially how to use the substitution templates.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh. It is important for S3Action! I will illustrate it!

});
```


## Put logs to CloudWatch Logs

The code snippet below creates an AWS IoT Rule that put logs to CloudWatch Logs
Expand Down
1 change: 1 addition & 0 deletions packages/@aws-cdk/aws-iot-actions/lib/index.ts
Original file line number Diff line number Diff line change
@@ -1,2 +1,3 @@
export * from './cloudwatch-logs-action';
export * from './lambda-function-action';
export * from './s3-action';
77 changes: 77 additions & 0 deletions packages/@aws-cdk/aws-iot-actions/lib/s3-action.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,77 @@
import * as iam from '@aws-cdk/aws-iam';
import * as iot from '@aws-cdk/aws-iot';
import * as s3 from '@aws-cdk/aws-s3';
import { kebab as toKebabCase } from 'case';
import { singletonActionRole } from './private/role';

/**
* Configuration properties of an action for s3.
*/
export interface S3ActionProps {
/**
* The Amazon S3 canned ACL that controls access to the object identified by the object key.
* @see https://docs.aws.amazon.com/AmazonS3/latest/userguide/acl-overview.html#canned-acl
*
* @default None
*/
readonly cannedAcl?: s3.BucketAccessControl;

/**
* The path to the file where the data is written.
*
* Supports substitution templates.
* @see https://docs.aws.amazon.com/iot/latest/developerguide/iot-substitution-templates.html
*
* @default '${topic()}/${timestamp()}'
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we have static, string-typed constants for these substitution templates? So that it's clear which ones can be used by which Actions?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we cannot define all static, string-typed constants. Because users can use MQTT payload property defined by themselves in template.

For example, there are following MQTT payload:

{
  "year": "2021",
  "month": "Nov",
  "day": "04",
  "data": 123.4
}

The user can define the bucket key as ${year}/${month}/${day}/${timestamp()}.

Are we on the same page?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think we need to have all of them - but you have a few that are always available, right? Like ${timestamp()} and ${topic()}. Maybe it's worth it to include those that are always available, and just document the other ones?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, I also want to make it as user-friendly as possible. So I'd like to be on same page!

  1. there are about 70 functions, and they take arguments
  • So we need to use functions, instead of static constants.
  1. there are also Operators.
  • So it is better that our functions return without ${} as topic().
  1. in many cases, they are used embedded in strings without any regularity. For example, cloudwatch metric name could be used substitution templates.

And I thought 3 plans.

PLAN1, implement a set of functions

In light of the above, it might be better for us to implement a set of functions and for users to use them in template literals, as following:

key: `\${${iot.Template.topic(2)}}/\${${iot.Template.timestamp()}}`

PLAN2, just validation

Or, is it better to implement validation as JsonPath of @aws-cdk/aws-stepfunctions
https://github.com/aws/aws-cdk/tree/master/packages/@aws-cdk/aws-stepfunctions/lib/fields.ts#L216-L226

For substitution templates, I think we will be using a lot of regular expressions.

PLAN3, implement Expression and Template

Or, we implement Expression and Template?

I mean Expression as following:

iot.Expression.substring(iot.Expression.topic(2), 1)
// return type of Expression

I mean Template as following:

iot.Template.format(
  '{}/{}',
  iot.Expression.substring(iot.Expression.topic(2), 5, 10),
  iot.Expression.timestamp(),
)
// return type of Template

And, properties that can be used substitution templates will have the type Template.

{
  key: iot.Template,
}

The above may help to naturally communicate to users that substitution templates are available.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Before I answer, I have one more clarifying question.

Are any of the substitutions Action-dependent? So, are there any substitutions that can only be used with certain Action(s)?

In any case, I see now this is a much more complicated topic than I initially thought, and should be tackled in a separate PR. Let's continue the discussion on how to best model them - that's fine, but I don't want to block this PR on the discussion. Let's proceed with this PR without anything around substitutions, like it is now.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are any of the substitutions Action-dependent? So, are there any substitutions that can only be used with certain Action(s)?

As far as I can tell from the documentation, no. And from the documentation, they all seem to be common functions and not Action-dependent. 🙂

Let's proceed with this PR without anything around substitutions, like it is now.

Sounds good!

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are any of the substitutions Action-dependent? So, are there any substitutions that can only be used with certain Action(s)?

As far as I can tell from the documentation, no. And from the documentation, they all seem to be common functions and not Action-dependent. 🙂

Great! Given that, I would go with Plan 1 from #17307 (comment).

*/
readonly key?: string;

/**
* The IAM role that allows access to the S3.
*
* @default a new role will be created
*/
readonly role?: iam.IRole;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will every Action have a role property? Because if the answer is "yes", we should introduce a common interface that all Action prop interfaces extend. Like the CommonActionProps in @aws-cdk/aws-codepipeline.

Copy link
Contributor Author

@yamatatsu yamatatsu Nov 4, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@skinny85
Not every Action is need a role, but almost. 2 actions in all actions will not have a role property.

  • HTTP Action
  • Lambda Action

For example, LambdaFunctionAction constructor (most simple one) is as following:

constructor(private readonly func: lambda.IFunction) {}

So I thought there are no common property for all actions. But the way to align the arguments of the action's constructor is what I was also struggling with.

What do you think? I'm open to any and all refactoring!

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, so if all of actions besides two will need role, I think introducing a separate common interface makes sense. And make S3ActionProps (soon to be changed to S3PutObjectActionProps I hope 😉) extend this new interface.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK! I will! 🚀

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@skinny85
If we create CommonActionProps and CommonAwsActionProps, should these constructors be unified in the following form?

interface CommonActionProps {
}

intercace CommonAwsActionProps extends CommonActionProps {
  role?: iam.IRole;
}

class Action {
  constructor(props: CommonActionProps) {}
}

Now, three Actions LambdaFunctionAction, CloudWatchLogsAction and S3PutObjectActionProps have following constructor, and first argument of them constroctor is not Props:

LambdaFunctionAction

constructor(func: lambda.IFunction) {}

CloudWatchLogsAction

constructor(logGroup: logs.ILogGroup, props: CloudWatchLogsActionProps) {}

S3PutObjectActionProps

constructor(bucket: s3.IBucket, props: S3PutObjectActionProps) {}

So if first question is "yes", I will do this refactoring in another PR!

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No. Let's leave the constructors as they are now.

BTW, there's no point in having both CommonActionProps and CommonAwsActionProps for IoT Actions - you only need one common superinterface (it's a different story with CodePipeline Actions, but I don't want to get into the details of why that is here).

}

/**
* The action to write the data from an MQTT message to an Amazon S3 bucket.
*/
export class S3Action implements iot.IAction {
yamatatsu marked this conversation as resolved.
Show resolved Hide resolved
private readonly cannedAcl?: string;
private readonly key?: string;
private readonly role?: iam.IRole;

/**
* @param bucket The Amazon S3 bucket to which to write data.
* @param props Optional properties to not use default
*/
constructor(private readonly bucket: s3.IBucket, props: S3ActionProps = {}) {
this.cannedAcl = props.cannedAcl;
this.key = props.key;
this.role = props.role;
}

bind(rule: iot.ITopicRule): iot.ActionConfig {
const role = this.role ?? singletonActionRole(rule);
role.addToPrincipalPolicy(this.putEventStatement(this.bucket));

return {
configuration: {
s3: {
bucketName: this.bucket.bucketName,
cannedAcl: this.cannedAcl && toKebabCase(this.cannedAcl.toString()),
key: this.key ?? '${topic()}/${timestamp()}',
roleArn: role.roleArn,
},
},
};
}

private putEventStatement(bucket: s3.IBucket) {
return new iam.PolicyStatement({
actions: ['s3:PutObject'],
skinny85 marked this conversation as resolved.
Show resolved Hide resolved
resources: [bucket.arnForObjects('*')],
});
}
}
6 changes: 6 additions & 0 deletions packages/@aws-cdk/aws-iot-actions/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -83,7 +83,9 @@
"@aws-cdk/aws-iot": "0.0.0",
"@aws-cdk/aws-lambda": "0.0.0",
"@aws-cdk/aws-logs": "0.0.0",
"@aws-cdk/aws-s3": "0.0.0",
"@aws-cdk/core": "0.0.0",
"case": "1.6.3",
"constructs": "^3.3.69"
},
"homepage": "https://github.com/aws/aws-cdk",
Expand All @@ -92,9 +94,13 @@
"@aws-cdk/aws-iot": "0.0.0",
"@aws-cdk/aws-lambda": "0.0.0",
"@aws-cdk/aws-logs": "0.0.0",
"@aws-cdk/aws-s3": "0.0.0",
"@aws-cdk/core": "0.0.0",
"constructs": "^3.3.69"
},
"bundledDependencies": [
"case"
],
"engines": {
"node": ">= 10.13.0 <13 || >=13.7.0"
},
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,85 @@
{
"Resources": {
"TopicRule40A4EA44": {
"Type": "AWS::IoT::TopicRule",
"Properties": {
"TopicRulePayload": {
"Actions": [
{
"S3": {
"BucketName": {
"Ref": "MyBucketF68F3FF0"
},
"Key": "${topic()}/${timestamp()}",
"RoleArn": {
"Fn::GetAtt": [
"TopicRuleTopicRuleActionRole246C4F77",
"Arn"
]
}
}
}
],
"AwsIotSqlVersion": "2016-03-23",
"Sql": "SELECT topic(2) as device_id FROM 'device/+/data'"
}
}
},
"TopicRuleTopicRuleActionRole246C4F77": {
"Type": "AWS::IAM::Role",
"Properties": {
"AssumeRolePolicyDocument": {
"Statement": [
{
"Action": "sts:AssumeRole",
"Effect": "Allow",
"Principal": {
"Service": "iot.amazonaws.com"
}
}
],
"Version": "2012-10-17"
}
}
},
"TopicRuleTopicRuleActionRoleDefaultPolicy99ADD687": {
"Type": "AWS::IAM::Policy",
"Properties": {
"PolicyDocument": {
"Statement": [
{
"Action": "s3:PutObject",
"Effect": "Allow",
"Resource": {
"Fn::Join": [
"",
[
{
"Fn::GetAtt": [
"MyBucketF68F3FF0",
"Arn"
]
},
"/*"
]
]
}
}
],
"Version": "2012-10-17"
},
"PolicyName": "TopicRuleTopicRuleActionRoleDefaultPolicy99ADD687",
"Roles": [
{
"Ref": "TopicRuleTopicRuleActionRole246C4F77"
}
]
}
},
"MyBucketF68F3FF0": {
"Type": "AWS::S3::Bucket",
"UpdateReplacePolicy": "Delete",
"DeletionPolicy": "Delete"
}
}
}
25 changes: 25 additions & 0 deletions packages/@aws-cdk/aws-iot-actions/test/s3/integ.s3-action.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
/// !cdk-integ pragma:ignore-assets
import * as iot from '@aws-cdk/aws-iot';
import * as s3 from '@aws-cdk/aws-s3';
import * as cdk from '@aws-cdk/core';
import * as actions from '../../lib';

const app = new cdk.App();

class TestStack extends cdk.Stack {
constructor(scope: cdk.App, id: string, props?: cdk.StackProps) {
super(scope, id, props);

const topicRule = new iot.TopicRule(this, 'TopicRule', {
sql: iot.IotSql.fromStringAsVer20160323("SELECT topic(2) as device_id FROM 'device/+/data'"),
});

const bucket = new s3.Bucket(this, 'MyBucket', {
removalPolicy: cdk.RemovalPolicy.DESTROY,
});
topicRule.addAction(new actions.S3Action(bucket));
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Again, let's use a more sophisticated example of S3Action here, instead of just the minimal one.

}
}

new TestStack(app, 'test-stack');
app.synth();
Loading