The Internet of Things on AWS – Official Blog

The AWS IoT platform enables you to connect your internet-enabled devices to the AWS cloud via MQTT/HTTP/Websockets protocol. Once connected, the devices can send data to MQTT topic(s). Data ingested on MQTT topics can be routed into AWS services (like Amazon S3, Amazon SQS, Amazon DynamoDB, Amazon Lambda etc.), by configuring rules in AWS IoT Rules Engine.

This blog post explains how to set up rules for cross-account data ingestion, from an MQTT topic in one account, to a destination in another account. We will focus on the cross-account access from an MQTT topic (the source) to Lambda and SQS (the destinations).

The blog has been written with the assumption that you are familiar with AWS IoT and the Rules Engine, and have a fair understanding of AWS IAM concepts like users, role and resource-based permission.

We are going to use the AWS CLI to setup cross-account rules. If you don’t have AWS CLI installed, you can follow these steps. If you have the AWS CLI installed, make sure you are using the most recent version.

Why do you need cross-account access via rules engine?

Rules with cross-account access allow you to ingest data published on an MQTT topic in one account to a destination (S3, SQS etc.) in another account. For example, Weather Corp collects weather data using its network of sensors and then publishes that data on MQTT topics in its AWS account. Now, if Weather Corp wishes to publish this data to an Amazon SQS queue of its partner, Forecast Corp’s, AWS account, they can do so by enabling cross-account access via the AWS IoT Rules Engine.

How can you configure a cross-account rule?

Cross-account rules can be configured using the resource-based permissions on the destination resource.

Thus, for Weather Corp to create a rule in their account to ingest weather data into an Amazon SQS queue in Forecast Corp’s AWS account, the cross account access can be set up by means of the two step method stated below:

1. Forecast Corp creates a resource policy on their Amazon SQS queue, allowing Weather Corp’s AWS account to sqs:SendMessage action.
2. Weather Corp configures a rule with the Forecast Corp queue URL as its destination.

Note: Cross-account access, via AWS IoT Rules Engine, needs resource-based permissions. Hence, only destinations that support resource-based permission can be enabled for the cross-account access via AWS IoT Rules Engine. Following is the list of such destinations:

Amazon Simple Queue Service (SQS)
Amazon Simple Notification Service (SNS)
Amazon Simple Storage Service (S3)
AWS Lambda

Configure a cross-account rule

In this section, configuration of a cross account rule to access an AWS Lambda function and Amazon SQS queue in a different account has been explained. We used the AWS CLI for this configuration.

Steps to configure a cross-account rule for AWS Lambda is different when compared to other AWS services that support resource policy.

For Lambda:

The AWS IoT Rules Engine, mandatorily requires resource-based policy to access Lambda functions; so a cross-account Lambda function invocation is configured just like any other IoT-Lambda rule. The process of enabling cross-account access for Lambda can be understood from the following example:

Assume that Weather Corp, using AWS account# 123456789012, wishes to trigger a Lambda function (LambdaForWeatherCorp) in Forecast Corp’s account (AWS account# 987654321012) via the Rules Engine. Further, Weather Corp wishes to trigger this rule when a message arrives on Weather/Corp/Temperature MQTT topic.

To do this, Weather Corp would need to create a rule (WeatherCorpRule) which will be attached to Weather/Corp/Temperature topic. To create this rule, Weather Corp would need to call the CreateTopicRule API. Here is an example of this API call via AWS CLI:

aws iot create-topic-rule --rule-name WeatherCorpRule --topic-rule-payload file://./lambdaRule

Contents of the lambdaRule file:

{
       "sql": "SELECT * FROM 'Weather/Corp/Temperature'", 
       "ruleDisabled": false, 
       "actions": [{
           "lambda": {
               "functionArn": "arn:aws:lambda:us-east-1:987654321012:function:LambdaForWeatherCorp"   //Cross account lambda
            }
       }]
}

Forecast Corp will also have to give the AWS IoT Rules Engine permission to trigger LambdaForWeatherCorp Lambda function. Also, it is very important for Forecast Corp to make sure that only the AWS IoT Rules Engine is able to trigger the Lambda function and that it is done so only on behalf of Weather Corp’s WeatherCorpRule (created above) rule.

To do this, Forecast Corp would need to use Lambda’s AddPermission API. Here is an example of this API call via AWS CLI:

aws lambda add-permission --function-name LambdaForWeatherCorp --region us-east-1 --principal iot.amazonaws.com --source-arn arn:aws:iot:us-east-1:123456789012:rule/WeatherCorpRule --source-account 123456789012 --statement-id "unique_id" --action "lambda:InvokeFunction"

Options:
–principal: This field gives permission to AWS IoT (represented by iot.amazonaws.com) to call the Lambda function.

–source-arn: This field makes sure that only arn:aws:iot:us-east-1:123456789012:rule/WeatherCorpRule rule in AWS IoT triggers this Lambda (no other rule in the same or different account can trigger this Lambda).

–source-account: This field makes sure that AWS IoT triggers this Lambda function only on behalf of 123456789012 account.

Note: To run the above command, IAM user/role should have permission to lambda:AddPermission action.

For Other Services

As of today, the Rules Engine does not use resource policy to access non-Lambda AWS resources (Amazon SQS, Amazon S3, Amazon SNS ). Instead, it uses IAM role to access these resources in an account. Additionally, AWS IoT rules can only be configured with roles from the same account. This implies, that a rule cannot be created in one account that uses a role from another account.

While, a role from another account cannot be used in a rule, a role can be set up in an account to access resources in another account. Also, for a cross-account role to work, you need a resource policy on the resource that has to be accessed across the account.

The process of rule creation with access to cross-account resources can be understood from the below example:

Let’s assume that Weather Corp, using AWS account# 123456789012, wishes to send some data to Amazon SQS (SqsForWeatherCorp) in Forecast Corp’s account (AWS account# 987654321012) via rules engine. If Weather Corp wishes to trigger this rule when a message arrives on Weather/Corp/Temperature MQTT topic.

To do this, Weather Corp would need to do the following things:

Step 1: Create an IAM policy (PolicyWeatherCorp) that defines cross-account access to SqsForWeatherCorp SQS queue. To do this, Weather Corp would need to call IAM’s CreatePolicy API. Here is an example of this API call via AWS CLI:

aws iam create-policy --policy-name PolicyWeatherCorp --policy-document file://./crossAccountSQSPolicy

Where the contents of crossAccountSQSPolicy file are below:

{
   "Version": "2012-10-17",
   "Statement": [
       {
           "Sid": “unique”,
           "Effect": "Allow",
           "Action": [
               "sqs:SendMessage"
           ],
           "Resource": [
               "arn:aws:sqs:us-east-1:987654321012:SqsForWeatherCorp" //Cross account SQS queue
           ]
       }
   ]
}

Step 2: Create a role (RoleWeatherCorp) that defines iot.amazonaws.com as a trusted entity. To do this Weather Corp would need to call IAM’s CreateRole API. Here is an example of this API call via AWS CLI:

aws iam create-role --role-name RoleWeatherCorp  --assume-role-policy-document file://./roleTrustPolicy

Where the contents of roleTrustPolicy file are below:

{
 "Version": "2012-10-17",
 "Statement": [
   {
     "Sid": "",
     "Effect": "Allow",
     "Principal": {
       "Service": "iot.amazonaws.com"
     },
     "Action": "sts:AssumeRole"
   }
 ]
}

Step 3: Attach policy to role. To do this, Weather Corp would need to call AttachRolePolicy API. Here is an example of this API call via AWS CLI:

aws iam attach-role-policy --role-name RoleWeatherCorp --policy-arn  arn:aws:iam::123456789012:policy/PolicyWeatherCorp

Step 4: Create a rule (WeatherCorpRule) that is attached to Weather/Corp/Temperature topic. To create this rule, Weather Corp would need to call CreateRule API. Here is an example of this API call via AWS CLI:

aws iot create-topic-rule --rule-name WeatherCorpRule --topic-rule-payload file://./sqsRule

Where the contents of sqsRule file are below:

{
       "sql": "SELECT * FROM 'Weather/Corp/Temperature'", 
       "ruleDisabled": false, 
       "actions": [{
           "sqs": {
               "queueUrl": "https://sqs.us-east-1.amazonaws.com/987654321012/SqsForWeatherCorp",
               "roleArn": "arn:aws:iam::123456789012:role/RoleWeatherCorp”, 
               "useBase64": false
           }
       }]
}

Note: To run the above command, IAM user/role should have permission to iot:CreateTopicRule with rule arn as resource. Also, it needs to have permission to iam:PassRole action with resource as role arn.

Further, Forecast Corp would need to give permission on SqsForWeatherCorp to Weather Corp’s account, using resource policy. This can be done using SQS’s add-permission API. Here is an example of this API call via AWS CLI:

aws sqs add-permission --queue-url https://sqs.us-east-1.amazonaws.com/987654321012/SqsForWeatherCorp --label SendMessagesToMyQueue --aws-account-ids 123456789012 --actions SendMessage

It is important to note, that by adding this resource policy, Forecast Corp not only allows AWS IoT rules engine to send messages to SqsForWeatherCorp, but also permits all users/roles in Weather Corp’s account (which have the policy to allow sqs:SendMessage to SqsForWeatherCorp) to send messages to SqsForWeatherCorp.

Once the above setup is done, all messages sent to Weather/Corp/Temperature (which is in WeatherCorp’s account) will be sent to SqsForWeatherCorp (which is in Forecast Corp’s account) using the rules engine.

Conclusion

In this blog, the process of creating AWS IoT rules with cross account destination has been explained. With the help of simple case scenarios, the process of creating rules for Lambda and SQS destinations, using AWS CLI, has been detailed in a step by step manner.

We hope you found this walkthrough useful. Feel free to leave your feedback in the comments.

AWS provides essential building blocks to help virtually any company build and deploy an Internet of Things (IoT) solution.  Building on AWS, you have access to a broad array of services including AWS IoT, a managed cloud platform that lets connected devices easily and securely interact with cloud applications and other devices, and low-cost data storage with high durability and availability for backup, archiving, and disaster recovery options to meet virtually an infinite number of scenarios and use cases. For example, Amazon S3 provides scalable storage in the cloud, Amazon Glacier provides low cost archive storage, and AWS Snowball enables large volume data transfers.  No solution is complete without information being generated from the system data collected. Here, you can utilize Amazon Machine Learning for predictive capabilities which can enable you to gain business insights from the data you’ve collected. We strive to offer services commonly used to build solutions today, and regularly release new services purposely built to help you meet your IoT business needs today and in the future.

We are currently witnessing a major shift in how customers view their business. Customers across industries, including Financial Services, Manufacturing, Energy, Transportation, Industrial and Banking, are on a business transformation journey and are seeking guidance to help transform from product-centric to service-orientated companies, taking advantage of actionable insights they can drive through IoT.  Early adopters have already deployed a wide range of cloud-based IoT solutions, and many are seeking to optimize existing solutions. Some companies are just getting started.  Regardless of where your company is in your IoT journey, working with industry-leading AWS Partner Network (APN) Partners who offer value-added services and solutions on AWS can help you accelerate your success.

Today, we launched the AWS IoT Competency to help you easily connect to APN Partners with proven expertise and customer success to help meet your specific business needs.

What’s the value of the AWS IoT Competency for your firm?

The IoT value chain is complex, and has many “actors.” Successful IoT implementations require services and technologies not traditionally part of the Enterprise DNA. As you seek to find best-in-breed partners for your specific needs, whether they be identifying edge or gateway devices or software, a platform to acquire, analyze, and act on IoT data, connectivity for edge and gateway devices, or consulting services to help you architect and deploy your solution, we want to make sure we help you easily connect with Consulting and Technology Partners who can help.

APN Partners who have achieved the AWS IoT Competency have been vetted by AWS solutions architects, and have passed a high bar of requirements such as providing evidence of deep technical and consulting expertise helping enterprises adopt, develop, and deploy complex IoT projects and solutions. IoT Competency Partners provide proven technology and/or implementation capabilities for a variety of use cases including (though not limited to) intelligent factories, smart cities, energy, automotive, transportation, and healthcare.  Lastly, public customer references and proven customer success are a core requirement for any APN Partner to achieve the AWS IoT Competency.

Use Cases and Launch Partners

Congratulations to our launch IoT Technology Competency Partners in the following categories:

Edge:  Partners who provide hardware and software ingredients used to build IoT devices, or finished products used in IoT solutions or applications.  Examples include: sensors, microprocessors and microcontrollers, operating systems, secure communication modules, evaluation and demo kits.

• Intel
• Microchip Technology

Gateway: Partners who provide data aggregation hardware and/or software connecting edge devices to the cloud and providing on premise intelligence as well as connecting to enterprise IT systems.  Examples include hardware gateways, software components to translate protocols, and platforms running on-premises to support local decision making.

• MachineShop

Platform Providers: Independent software vendors (ISVs) who’ve developed a cloud-based platform to acquire, analyze, and act on IoT data. Examples include device management systems, visualization tools, predictive maintenance applications, data analytics, and machine learning software.

• Bsquare Corporation
• C3 IoT
• Splunk
• PTC
• Thinglogix

Connectivity: Partners who provide systems to manage wide-area connectivity for edge and gateway devices.  Examples include device and subscription management platforms, billing and rating systems, device provisioning systems, and Mobile Network Operators (MNOs) and Mobile Virtual Network Operators (MVNOs)

• Amdocs, Inc.
• Asavie
• Eseye
• SORACOM

Congratulations to our launch IoT Consulting Competency Partners!

• Accenture
• Aricent
• Cloud Technology Partners
• Mobiquity, Inc.
• Luxoft
• Solstice
• Sturdy
• Trek10

Learn More

Hear from two of our launch AWS IoT Competency Partners, MachineShop and C3 IoT, as they discuss why they work with AWS, and the value of the AWS IoT Competency for customers:

C3 IoT:

MachineShop:

Want to learn more about the different IoT Partner Solutions? Click here.

For many customers, the management console is the primary tool for interacting with and monitoring AWS IoT. This includes connecting a first device, diving into thing details, finding key resources, and testing with the MQTT client. Over the past year, we received feedback from customers that drove the redesign of a new AWS IoT console available today.

In the new console, you will see:

• New visual design for improved usability and navigation
• Improved navigation to things, types, certificates, policies, and rules, making them easier to find
• A new dashboard with account-level metrics
• Streamlined MQTT web client to troubleshoot your IoT solutions
• A new wizard to connect devices in fewer steps
• A real-time feed of things’ lifecycle events and shadow activity

To try the new console experience, sign in to the console.

Your feedback is important to us as we continue to improve the AWS IoT console experience. To send feedback, please use the Feedback button in the footer of the console.

Fig 1 – New dashboard with account-level metrics.

Fig 2 – Things, types, certificates, policies, and rules all have their own areas.

Fig 3 – Drill in to resource details and take action.

You can connect securely millions of objects to AWS IoT using our AWS SDKs or the AWS IoT Device SDKs. In the context of industrial IoT, objects are usually connected to a gateway for multiple reasons: sensors can be very constrained and not able to directly connect to the cloud, sensors are only capable of using other protocols than MQTT or you might might need to perform analytics and processing locally on the gateway.

One feature of local MQTT broker is called ‘Bridge’ and will enable you to connect your local MQTT broker to AWS IoT so they can exchange MQTT messages. This will enable your objects to communicate in a bi-directional fashion with AWS IoT and benefit from the power of the AWS Cloud.

In this article we are going to explain use cases where this feature can be very useful and show you how to implement it.

Why Bridge your MQTT Broker to AWS IoT

Security is paramount in IoT and the AWS IoT broker has a high level of security built-in to authenticate and authorize devices base on standards like TLS 1.2 with client certificates.

If you have legacy IoT deployments, you might already have objects connected to an MQTT broker using other authentication mechanism like username and passwords. Your MQTT broker can be very close to where your sensors are deployed (local MQTT broker) or in a remote location like the Cloud.

If you plan to upgrade your current security standards to match those of AWS IoT but want to benefit from the scalability and Rule Engine of AWS IoT today, you can bridge your legacy MQTT broker to AWS IoT. This represents an easy transient solution that you can deploy quickly without having to wait for your current system’s upgrade. Scaling beyond a single broker is not in the scope of this post, we will focus on the bridging feature of Mosquitto MQTT Broker.

Open source MQTT broker like Mosquitto can be installed on many operating systems like Linux for example. For those wishing to deploy a local gateway quickly without developing extra code to send data to AWS IoT, installing Mosquitto on a local device can represent an attractive solution as well as you will benefit locally from Mosquitto boker’s features (persist messages locally, log activity locally, …).

How to Install Mosquitto MQTT Broker

The first step will be to install Mosquitto broker on your device/virtual machine, you can go to Mosquitto download page for instructions.

Typically, you should install this on your local gateway. Mosquitto supports a wide range of platforms including many distributions of Linux. Therefore, you can run your local gateway on low powered devices as well as on a full-fledged server/virtual machine.

In our case we will install Mosquitto on an EC2 Amazon Linux instance which would be equivalent to having a local gateway running a Linux distribution.

If you are not planning on using an Amazon EC2 Instance you can skip to the section “How to configure the bridge to AWS IoT”

Launching and Configuring the EC2 Instance

Before launching an EC2 Amazon Linux instance to host the Mosquitto broker, we are going to create an IAM Role so we’ll be able to use the CLI on the instance to create keys and certificate in AWS IoT for the bridge.

1. Go to the AWS Web Console and access the IAM service (Fig. 1)
   
2. Click on Roles
3. Click on Create New Role (Fig. 2)
   
4. Name the role AWSIoTConfigAccess (Fig. 3)
   
5. Click Next Step
6. Select Amazon EC2 (Fig. 4)
   
7. Filter with the value AWSIoTConfigAccess (Fig. 5)
   
8. Select the policy AWSIoTConfigAccess and click on Next Step
9. Review the Role and click on Create Role (Fig. 6)
   
10. Now that the Role has been created you can go to Amazon EC2. Choose a region, preferably where AWS IoT is available, in this article I am using Frankfurt.
11. Click on Launch Instance.
12. Select Amazon Linux AMI 2016.03.1 (Fig. 7)
    
13. Select the t2.micro instance type (Fig. 8)
    
14. Click on Next: Configure Instance Details
15. In the IAM Role, select AWSIoTConfigAccess (Fig. 9)
    
16. Leave default parameters as shown in the picture and click on Next: Add Storage
17. Leave everything as is and click on Next: Tag Instance
18. Give a name to your instance ‘MosquittoBroker’
19. Click on Next: Configure Security Groups
20. Create a new security group (Fig. 10)
    
21. Review and launch the EC2 instance
22. Follow instructions to connect to the EC2 instance once it is running.
23. Once logged in type the following commands:

#Update the list of repositories with one containing Mosquitto
sudo wget http://download.opensuse.org/repositories/home:/oojah:/mqtt/CentOS_CentOS-7/home:oojah:mqtt.repo -O /etc/yum.repos.d/mqtt.repo
#Install Mosquitto broker and Mosquitto command line tools
sudo yum install mosquitto mosquitto-clients

How to Configure the Bridge to AWS IoT

Now that we have installed Mosquitto onto our EC2 instance (or local gateway), we will need to configure the bridge so that the Mosquitto broker can create a connection to AWS IoT. We will first use the AWS CLI to create the necessary resources on AWS IoT side.

Enter the following commands in your terminal:

#Configure the CLI with your region, leave access/private keys blank
aws configure

#Create an IAM policy for the bridge
aws iot create-policy --policy-name bridge --policy-document '{"Version": "2012-10-17","Statement": [{"Effect": "Allow","Action": "iot:*","Resource": "*"}]}'

#Place yourself in Mosquitto directory
#And create certificates and keys, note the certificate ARN
cd /etc/mosquitto/certs/
sudo aws iot create-keys-and-certificate --set-as-active --certificate-pem-outfile cert.crt --private-key-outfile private.key --public-key-outfile public.key –region eu-central-1

#List the certificate and copy the ARN in the form of
# arn:aws:iot:eu-central-1:0123456789:cert/xyzxyz
aws iot list-certificates

#Attach the policy to your certificate
aws iot attach-principal-policy --policy-name bridge --principal <ARN_OF_CERTIFICATE>

#Add read permissions to private key and client cert
sudo chmod 644 private.key
sudo chmod 644 cert.crt

#Download root CA certificate
sudo wget https://www.symantec.com/content/en/us/enterprise/verisign/roots/VeriSign-Class%203-Public-Primary-Certification-Authority-G5.pem -O rootCA.pem

We now have a client certificate for our bridge, this certificate is associated with an IAM policy that will give all permissions to the bridge (this policy must be restricted for your usage). The bridge will have everything it needs to connect, we just need to edit the configuration file with our specific parameters for Mosquitto.

#Create the configuration file
sudo nano /etc/mosquitto/conf.d/bridge.conf

Edit the following by replacing the value address with your own AWS IoT endpoint. You can use the AWS CLI to find it with ‘aws iot describe-endpoint’ as mentioned below. Then copy the content and paste it in the nano editor, finally save the file.

#Copy paste the following in the nano editor:
# =================================================================
# Bridges to AWS IOT
# =================================================================

# AWS IoT endpoint, use AWS CLI 'aws iot describe-endpoint'
connection awsiot
address XXXXXXXXXX.iot.eu-central-1.amazonaws.com:8883

# Specifying which topics are bridged
topic awsiot_to_localgateway in 1
topic localgateway_to_awsiot out 1
topic both_directions both 1

# Setting protocol version explicitly
bridge_protocol_version mqttv311
bridge_insecure false

# Bridge connection name and MQTT client Id,
# enabling the connection automatically when the broker starts.
cleansession true
clientid bridgeawsiot
start_type automatic
notifications false
log_type all

# =================================================================
# Certificate based SSL/TLS support
# -----------------------------------------------------------------
#Path to the rootCA
bridge_cafile /etc/mosquitto/certs/rootCA.pem

# Path to the PEM encoded client certificate
bridge_certfile /etc/mosquitto/certs/cert.crt

# Path to the PEM encoded client private key
bridge_keyfile /etc/mosquitto/certs/private.key

Now we can start the Mosquitto broker with this new configuration:

#Starts Mosquitto in the background
sudo mosquitto -c /etc/mosquitto/conf.d/bridge.conf –d
#Enable Mosquitto to run at startup automatically
sudo chkconfig --level 345 scriptname on

Making Sure Everything is Working

The broker has now started and has already connected to AWS IoT in the background. In our configuration we have bridged 3 topics:

• awsiot_to_localgateway: any message received by AWS IoT from this topic will be forwarded to the local gateway
• localgateway_to_awsiot: any message received by the local gateway will be forwarded to AWS IoT
• both_directions: any message received on this topic by one broker will be forwarded to the other broker

We will check that the topic localgateway_to_awsiot is working, feel free to check the whole configuration.

• Go to the AWS IoT Console and click on MQTT Client
• Click on Generate Client Id and Connect
• Click on Subscribe to topic and enter localgateway_to_awsiot, click on Subscribe (Fig. 11)/>

Now that we have subscribed to this topic on AWS IoT side you can publish an MQTT message from your terminal (so from the local gateway) to see if it gets forwarded.

#Publish a message to the topic
mosquitto_pub -h localhost -p 1883 -q 1 -d -t localgateway_to_awsiot  -i clientid1 -m "{\"key\": \"helloFromLocalGateway\"}"

You should now get this message on your screen, delivered by AWS IoT thanks to the bridge.

If you are done testing with an Amazon EC2 Instance you can do this with your own local/remote MQTT broker!

Next Steps

The bridge between your local broker and AWS IoT is up and running, you might want to fine tune some parameters of the bridge connection. Please consult the Bridge section of the official Mosquitto documentation if you need additional details.

Now that your data is flowing through AWS IoT you can create new IoT applications using other AWS Services for Machine Learning, Analytics, Real-Time Dashboarding and much more so do not hesitate to read our blog, documentation and additional developer resources.

Welcome to another installment in the Bites of IoT blog series.

In this bite, we will use the AWS IoT rules engine to select and route a message to Amazon Simple Notification Service (Amazon SNS). Specifically, we’ll send a text message to a phone number when someone rings a virtual doorbell.

Concepts Covered in This Bite

The rules engine is often the first point at which your IoT solution can take actions on a device’s messages (for example, message filtering, routing to other services, and even direct processing of messages in your solution).

By using  AWS IoT and Amazon SNS together, you can send near real-time notifications in response to changes in the physical world as reported by devices in your IoT solution. These notifications can be:

• sent to system operators or technicians for human intervention.
• added to your existing ticketing system.
• used to kick off an automated workflow.

Configure the CLI

As with all Bites of IoT posts, we are using the AWS IoT ELF client available in AWS Labs on GitHub. If you aren’t familiar with the ELF, check out the first post in this series.

We’ll use the AWS CLI to create the Amazon SNS topic and AWS IoT rule. In addition to the permissions required by the AWS IoT ELF, you will need to configure the profile used by your CLI with, at minimum, the following permissions. Make sure to replace the value 012345678901 in the policy with your AWS account ID:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "sns:CreateTopic",
                "sns:SetTopicAttributes",
                "sns:Subscribe",
                "sns:Publish",
                "sns:DeleteTopic",
                "iot:CreateTopicRule",
                "iot:DeleteTopicRule",
                "iam:AttachRolePolicy",
                "iam:CreatePolicy",
                "iam:CreateRole",
                "iam:DetachRolePolicy",
                "iam:DeletePolicy",
                "iam:DeleteRole"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "iam:PassRole"
            ],
            "Resource": [
                "arn:aws:iam::012345678901:role/*"
            ]
        }
    ]
}

Create a Bites Directory

If you followed the steps in Getting Started, you should already have a local directory structure of ~/dev/aws-iot-elf (where “~” is shorthand for your computer user’s home directory).

Each Bite of IoT post will require the creation of local files. Start by creating your Bites directory. On Linux or Mac OS, execute the following commands:

cd ~/dev
mkdir bites
cd ~/dev/bites

Now, in your Bites directory, execute these commands to make a directory for the artifacts used in this post, and then change into that directory.

mkdir sns
cd ~/dev/bites/sns

Now that the basic dev environment is set up, let’s get into the details.

Create the SNS Topic

Create the Amazon SNS topic that the AWS IoT rules engine will be connecting with to send notifications. Use the following AWS CLI commands to create and name the topic. Remember to replace 012345678901 in the --topic-arn value with your AWS account ID.

aws sns create-topic --region us-west-2 --name door-alert
aws sns set-topic-attributes --region us-west-2 --topic-arn arn:aws:sns:us-west-2:012345678901:door-alert --attribute-name DisplayName --attribute-value DOORBELL

Note: The DisplayName value DOORBELL will appear as the “from” address when you receive a notification on your mobile phone.

Now subscribe your mobile phone to the topic using the following CLI command. Replace 1-012-345-6789 with your phone number. This command enables your mobile phone to receive the notification when the virtual doorbell rings.

aws sns subscribe --region us-west-2 --topic-arn arn:aws:sns:us-west-2:012345678901:door-alert --protocol sms --notification-endpoint 1-012-345-6789

You can use this command to validate the subscription:

aws sns publish --region us-west-2 --message "A Bite of Hello" --topic-arn arn:aws:sns:us-west-2:012345678901:door-alert

Create Permissions to Use Amazon SNS

Before we create the IoT rule, we need to create an IAM role that the AWS IoT rules engine will assume and use to securely publish a message to Amazon SNS.

The first step to creating the role is to make a trust policy that grants AWS IoT permission to assume the role. Copy and paste the following trust policy document into a temporary file named iot-role-trust.json:

{
    "Version":"2012-10-17",
    "Statement":[{
        "Effect": "Allow",
        "Principal": {
            "Service": "iot.amazonaws.com"
        },
        "Action": "sts:AssumeRole"
    }]
}

Run the following CLI command to create the role:

aws iam create-role --role-name iot_sns_role --assume-role-policy-document file://iot-role-trust.json

AWS IoT now needs permission to publish to the Amazon SNS topic we created. Copy and paste the following IAM policy document to create a file named iot-policy.json:

{
    "Version": "2012-10-17",
    "Statement": {
        "Effect": "Allow",
        "Action": "sns:Publish",
        "Resource": "arn:aws:sns:us-west-2:012345678901:door-alert"
    }
}

Run the following CLI command to create the policy:

aws iam create-policy --policy-name iot_sns_policy --policy-document file://iot-policy.json

Then run this CLI command to attach the policy to the role:

aws iam attach-role-policy --role-name iot_sns_role --policy-arn arn:aws:iam::012345678901:policy/iot_sns_policy

Create the AWS IoT Rule

Now we can set up the AWS IoT rule that will trigger a text alert when the bell rings. The rule will perform this task by sending every message that comes across the doorbell topic to Amazon SNS. Copy and paste the following rule definition into a file named sns-rule.json:

{
  "sql": "SELECT * FROM 'doorbell/+'",
  "description": "Sends a message to SNS when a message comes across the 'doorbell' topic",
  "actions": [
    {
      "sns": {
        "targetArn":"arn:aws:sns:us-west-2:012345678901:door-alert",
        "roleArn":"arn:aws:iam::012345678901:role/iot_sns_role",
        "messageFormat": "RAW"
      }
    }
  ],
  "ruleDisabled": false
}

Remember to replace 012345678901 in the rule with your AWS account ID.

Finally, use this CLI command to create the rule:

aws iot create-topic-rule --region us-west-2 --rule-name sendToSNS --topic-rule-payload file://sns-rule.json

Simulate the Doorbell

Everything is now ready to ask the ELF to push the virtual doorbell button. Use the following commands to switch to the AWS IoT ELF directory and get the latest version:

cd ~/dev/aws-iot-elf

git pull

Use this command to activate the ELF’s virtual environment:

source ~/dev/aws-iot-elf/venv/bin/activate

On Windows:

.\venv\Scripts\activate

Use this command to create a thing in the AWS IoT service to serve as our doorbell:

python elf.py --region us-west-2 create 1

The following command will now send a message on the topic as if the ELF pushed the virtual doorbell button.

python elf.py --region us-west-2 send --duration 1 --topic 'doorbell' 'Somebody is at the door!'

Very shortly, your mobile phone should receive a text message similar to this:

DOORBELL> {"msg": "Somebody is at the door!", "ts": "1468958915.41"}

Clean Up

You can use the following commands to clean up your environment:

python elf.py --region us-west-2 clean

aws iam detach-role-policy --role-name iot_sns_role --policy-arn arn:aws:iam::012345678901:policy/iot_sns_policy

aws iam delete-role --role-name iot_sns_role

aws iam delete-policy --policy-arn arn:aws:iam::012345678901:policy/iot_sns_policy

aws iot delete-topic-rule --region us-west-2 --rule-name sendToSNS

aws sns delete-topic --region us-west-2 --topic-arn arn:aws:sns:us-west-2:012345678901:door-alert

If you’d like to clean up the files created in this post, you can remove them from the ~dev/bites/sns directory.

Done!

In this bite, you’ve learned how to use AWS IoT and Amazon SNS together to send near real-time notifications in response to messages that come from a thing.

We hope you found this walk-through useful. Feel free to leave your feedback in the comments.

Stay tuned for the next post in the Bite of IoT series. Until then, may your success be swift and your solution scalable.

In an earlier blog post about certificates, we discussed how use-your-own-certificate support in AWS IoT lets customers use device certificates signed and issued by their own certificate authority (CA) to connect and authenticate with AWS IoT. This is an alternative to using certificates generated by AWS IoT.

Using your own certificate with AWS IoT is a two-step process:

• The first step is to register the CA certificate that signed and issued the device certificates.
• After registration, any device certificate that was signed by the CA can be registered with AWS IoT and used for authentication thereafter.

Now, with support for just-in-time registration (JITR) of CA-signed certificates, AWS IoT eliminates the second step.

In this blog post, I will explain how JITR works and how it can be used to set up a workflow that activates device certificates and attaches policies to them automatically. I will also walk you through steps for deactivating a CA certificate and revoking device certificates.

You will do the following:

1. Create, register, and activate a CA certificate that will be used to sign your device certificate.
2. Enable auto-registration of certificates.
3. Create device certificates signed by the CA and install them on your device.
4. Create and attach a rule with an AWS Lambda action that activates the certificate, and then creates and attaches policies to the certificate.
5. Connect to AWS IoT using the device certificate.

When you connect to AWS IoT with the device certificate for the first time, the service will detect an unknown certificate signed by a registered CA and will auto-register the device certificate. On successful registration, AWS IoT will publish a registration message on a reserved MQTT topic and disconnect the client. This MQTT registration event will trigger the attached AWS Lambda rules engine action, which will complete the provisioning of the certificate. After these steps, your device certificate will be ready to connect and authenticate with AWS IoT.

In this blog post, I assume you are familiar with AWS IoT and the process of creating an AWS IoT certificate. You will use the AWS CLI and OpenSSL to perform the procedures. If you don’t have the AWS CLI installed, follow these steps. If you already have the AWS CLI, make sure you are using the most recent version.

For information about authentication in AWS IoT or how to use AWS IoT-generated certificates, see the AWS IoT Developer Guide.

Registering Your CA Certificate

If you are a manufacturer, you have purchased CA certificates from vendors like Symantec or Verisign or you have your own CA. To use your own X.509 certificates that have been signed by your CA certificate, AWS IoT must first verify that you not only own the CA certificate, but that you also have access to its private key. The process for validating ownership of a CA certificate is done through a challenge and response workflow.

Let’s start by using openssl in a terminal to create your sample CA certificate. In the real world, the signing or intermediate certificates would be issued by your CA vendor. This sample CA certificate is used later in the walkthrough to sign a device certificate that you register with AWS IoT:

$ openssl genrsa -out sampleCACertificate.key 2048
$ openssl req -x509 -new -nodes -key sampleCACertificate.key -sha256 -days 365 -out sampleCACertificate.pem

For simplicity, we are creating and registering the root CA certificate. In reality, the intermediate CA certificate would be signed by the root CA that signs the device certificates. In that case, you register the intermediate CA certificate with AWS IoT.

Now that you’ve created a sample CA certificate, you will register it with AWS IoT. When you register a CA certificate with AWS IoT, you follow a workflow to verify that you have access to both the CA certificate and the private key associated with the CA certificate. To verify ownership of the private key, you generate a verification certificate using the CA certificate, the private key, and a registration code that you generate from AWS IoT.

The registration workflow first requires retrieving the registration code. You can use the AWS CLI or the Register Certificate section in the AWS IoT console to get the registration code.

To use the AWS CLI, run the following command:

$ aws iot get-registration-code

This command will return a randomly generated, unique registration code that is bound to your AWS account. This registration code is long-lived. It does not expire until you delete it.

Next, you will use the registration code to create a CSR:

$ openssl genrsa -out privateKeyVerification.key 2048
$ openssl req -new -key privateKeyVerification.key -out privateKeyVerification.csr

During the CSR creation process, you will be prompted for information. Enter the registration code into the Common Name field of the verification certificate:

...
Organization Name (eg, company) []:
Organizational Unit Name (eg, section)
Common Name (e.g. server FQDN or YOUR name) []: XXXXXSAMPLEREGISTRATIONCODEXXXXX
EMAIL ADDRESS []:

The registration code establishes that the generated verification certificate was created specifically for registering the CA certificate with AWS IoT, and that the verification certificate is not a previously issued certificate.

Now that you have a CSR that includes the registration code, use your first sample CA certificate and the CSR to create a new certificate:

$ openssl x509 -req -in privateKeyVerification.csr -CA sampleCACertificate.pem -CAkey sampleCACertificate.key -CAcreateserial -out privateKeyVerification.crt -days 365 -sha256

When you register your CA certificate with AWS IoT, the combination of the registration code, verification certificate signed with the CA private key, and the CA certificate are used to verify ownership of the CA private key.

Next, you will use the verification certificate to register your sample CA certificate:

$ aws iot register-ca-certificate --ca-certificate file://sampleCACertificate.pem --verification-certificate file://privateKeyVerification.crt

You can make a describe-ca-certificate call to get the information on the registered CA certificate. You will use the certificate ID of the registered CA returned in the response of the previous CLI command.

$ aws iot describe-ca-certificate --certificate-id <certificateId>

Next, you will activate the CA certificate. By default, the CA certificate will be registered in an INACTIVE state. At the time of the registration of the device certificate, AWS IoT will consult the status of its registered CA certificate and will allow the registration of the device certificate only if the CA certificate is in an ACTIVE state. Use the update-ca-certificate CLI command to change the status of the CA certificate. Alternatively, you can register the CA certificate in the ACTIVE state by passing the set-as-active flag at the time of registration:

$ aws iot update-ca-certificate --certificate-id <certificateId> --new-status ACTIVE

By default, the auto-registration-status of the registered CA certificate is disabled. That means any device certificate issued by the registered CA will not be auto-registered by default when it first connects to the AWS IoT service. However, you can use the register-certificate CLI command to explicitly register the device certificate. If the auto-registration-status is enabled for a CA certificate, the device certificate issued by that CA will be auto-registered, if not already registered, when it connects to AWS IoT for the first time. To opt in for the auto-registration of the device certificates issued by a registered CA, set the auto-registration-status for the CA certificate to ENABLE.

You can enable the auto-registration-status through the update-ca-certificate CLI command. Alternatively, you can register the CA certificate with the auto-registration-status enabled by passing the --allow-auto-registration flag to the register-ca-certificate CLI command.

$ aws iot update-ca-certificate --certificate-id <caCertificateId> --new-auto-registration-status ENABLE

Device Certificate Registration Event and Action

When a device attempts to connect with an X.509 certificate that is not known to AWS IoT but was signed by a CA that was registered with AWS IoT, the device certificate will be auto-registered by AWS IoT in a new PENDING_ACTIVATION state. PENDING_ACTIVATION means that the device certificate was auto-registered and is awaiting activation. Only AWS IoT can mark the status of a certificate as PENDING_ACTIVATION. If you connect with a certificate in PENDING_ACTIVATION state, a TLS handshake failure will occur because only ACTIVE certificates are authenticated with AWS IoT. For this reason, you need to change the status of the registered certificate from PENDING_ACTIVATION to ACTIVE so that it can be successfully authenticated.

When AWS IoT auto-registers a certificate or when a certificate in PENDING_ACTIVATION status connects, it publishes a message to the following MQTT topic:

$aws/events/certificates/registered/<caCertificateID>

where the caCertificateId is the ID of the CA certificate that issued the device certificate.

The message published to this topic has the following structure:

{
  "certificateId": "<certificateID>",
  "caCertificateId": "<caCertificateId>",
  "timestamp": "<timestamp>",
  "certificateStatus": "PENDING_ACTIVATION",
  "awsAccountId": "<awsAccountId>",
  "certificateRegistrationTimestamp": "<certificateRegistrationTimestamp>"
}

You can subscribe or attach any AWS IoT rule to the registration topic. The attached AWS IoT rules can then take some action based on the messages received. For example, an AWS IoT rule in your account can listen on the $aws/events/certificates/registered/+ topic to build an Amazon DynamoDB table of all the registered certificates. The general recommendation is to attach an AWS IoT rules engine action to the registration topic that will perform the bootstrapping or provisioning steps (like consulting your CRLs) and then activate/deactivate/revoke the certificate, create and attach the policies to the certificate, and so on.

In the following example, we will create a topic rule with an AWS Lambda action on the registration topic that will activate the certificate and create and attach a policy.

Use the AWS Lambda console to create the AWS Lambda function:

1. Sign in to the AWS Management Console and open the AWS Lambda console at https://console.aws.amazon.com/lambda/home?region=us-east-1.
2. Choose Create an AWS Lambda function.
3. On the Configure function page, type a name and description for the AWS Lambda function. In Runtime, choose Node.js 4.3.
4. Scroll down to the AWS Lambda function code section of the page. Replace the existing code with this sample code.
5. Scroll down to the AWS Lambda function handler and role section of the page. For Role, choose Create a custom role. When the IAM console opens, you can create an IAM role that AWS Lambda can assume when executing the AWS Lambda function.
6. In the navigation pane, choose Create New Role.
7. For Role name, type a role name.To edit the role’s policy to give it permission to update the certificate and create and attach the policy:
   1. Choose View Policy Document.
   2. Replace the policy document with the following:

      {  
         "Version":"2012-10-17",
         "Statement":[  
            {  
               "Effect":"Allow",
               "Action":[  
                  "logs:CreateLogGroup",
                  "logs:CreateLogStream",
                  "logs:PutLogEvents"
               ],
               "Resource":"arn:aws:logs:*:*:*"
            },
            {  
               "Effect":"Allow",
               "Action":[  
                  "iot:UpdateCertificate",
                  "iot:CreatePolicy",
                  "iot:AttachPrincipalPolicy"
               ],
               "Resource":"*"
            }
         ]
      }

   3. Choose Allow.
8. Leave the settings on the Advanced settings page at their defaults, and choose Next.
9. On the Review page, choose Create function.

Creating an AWS Lambda Rule

Now that you have created an AWS Lambda function, you can create a rule that invokes the function.

1. In the AWS IoT console, choose Create a resource.
2. Choose Create a rule.
3. Type a name and description for the rule.
4. Enter the following settings for the rule:
   SQL version: 2016-03-23-beta
   Attribute: *
   Topic filter: $aws/events/certificates/registered/<caCertificateID> Note: Replace <caCertificateId> with the ID of the registered CA certificate.
5. For Choose an action, choose Insert this message into a code function and execute it (AWS Lambda).
6. From Function name, choose your AWS Lambda function name, and then choose Add action.
7. Choose Create to create your AWS Lambda function.

You also need to grant permissions to the AWS IoT service principal to invoke the AWS Lambda function on your behalf when the MQTT message is published on the registration topic.

Note: If you created the rule through the AWS IoT console, you can skip this step. The console does this for you when you create the AWS Lambda rule.

You will use the AWS Lambda AddPermission API to grant permissions:

aws lambda add-permission --function-name <lambda-function-name> --region us-east-1 --principal iot.amazonaws.com --source-arn <rule-arn> --source-account <your-aws-account> --statement-id Id-123 --action "lambda:InvokeFunction"

Auto-registration of a Device Certificate Signed by Your CA Certificate

Now that you’ve created, registered, and activated a sample CA certificate with auto-registration-status enabled and configured a rule with an AWS Lambda action to activate the certificate and attach policies, use the CA certificate to create a new device certificate and auto-register it when it first connects to AWS IoT.

Enter the following commands in your terminal to create a device certificate:

$ openssl genrsa -out deviceCert.key 2048
$ openssl req -new -key deviceCert.key -out deviceCert.csr
$ openssl x509 -req -in deviceCert.csr -CA sampleCACertificate.pem -CAkey sampleCACertificate.key -CAcreateserial -out deviceCert.crt -days 365 -sha256

Next, try to connect to AWS IoT using the device certificate. Because you’ve completed the preceding steps, your device certificates will be auto-registered during the TLS handshake when it connects to AWS IoT for the first time. At the time of connection, you need to send both the device certificate and its registered CA certificate.

Create a certificate file that contains the device certificate and its registered CA certificate. Here is the Linux command:

$ cat deviceCert.crt sampleCACertificate.pem > deviceCertAndCACert.crt

In the following example, you will use the MQTT Mosquitto client to connect and publish to AWS IoT using the device certificate:

$ mosquitto_pub --cafile root.cert --cert deviceCertAndCACert.crt --key deviceCert.key -h <prefix>.iot.us-east-1.amazonaws.com -p 8883 -q 1 -t  foo/bar -i  anyclientID --tls-version tlsv1.2 -m "Hello" -d

Note: The root.cert is the AWS IoT root certificate. The AWS IoT root CA certificate is used by a device to verify the identity of the AWS IoT servers. Click here to download the root certificate. Save this file to your desktop and name it “root.cert”.

You will see a TLS failure when you run the command because AWS IoT disconnects the connection after the registration of the device certificate. AWS IoT has registered the certificate in the PENDING_ACTIVATION state and won’t let it connect and authenticate unless the certificate is marked ACTIVE. The client or device should implement an automatic reconnect strategy when it is disconnected due to the TLS error. The device should also implement a back-off strategy (that is, increase the time between retries) to avoid unnecessary network traffic and allow your own activation logic to complete before devices attempt to reconnect.

The commands to connect to AWS IoT with the device certificate whose CA has been registered will lead to the auto-registration of the device certificate during the TLS handshake. An MQTT registration event will be published on the registration topic and trigger the attached Lambda rule action to activate the certificate and attach the policy to it. This will complete the registration and provisioning of the device certificate. It can now be used for authentication and authorization with AWS IoT. After a device certificate is registered with AWS IoT, you can interact with it in a way that’s similar to a device certificate generated by AWS IoT. You can activate/deactivate /revoke certificates, attach policies, and associate things to your device certificates. In addition, you can manage the lifecycle of that device certificate in AWS IoT by deactivating, revoking, or activating the certificate in your AWS account.

Deactivate the CA Certificate

During device certificate registration, AWS IoT will check if the associated CA certificate is active. If the CA certificate is inactive, AWS IoT will not allow the device certificate to be registered. This feature provides you with the flexibility to manage the lifecycle of device certificates signed by the registered CA that are yet to be registered with AWS IoT. By marking the status INACTIVE, you prevent any new device certificates issued by the compromised CA to be registered in your account. You can run the following command to deactivate the CA certificate:

$ aws iot update-ca-certificate --certificate-id <certificateId> --new-status INACTIVE

Note: Running this command will not deactivate all of your registered device certificates. It will prevent new device certificates from being registered.

Revoke the Device Certificate

If you wish to revoke a registered device certificate, use the following command to revoke it:

$ aws iot update-certificate --certificate-id <certificateId> --new-status REVOKED

Use the list-certificates-by-ca CLI command to get a list of registered device certificates, sorted by the date of registration, that are issued by the CA certificate:

$ aws iot list-certificates-by-ca --ca-certificate-id <caCertificateId>

If an error or exception occurs during the auto-registration of the device certificates, AWS IoT will send the events or message to CloudWatch Logs. To set up CloudWatch Logs for your account, see the AWS IoT Developer Guide.

Conclusion

We hope you found these steps (using your own device certificate with AWS IoT, auto-registration of device certificates, and deactivating CA certificates) useful and will use them to provision and register your device certificates with AWS IoT. Give it a try and feel free to leave your feedback in the comments.

Welcome to Bites of IoT, the first post in a series designed to introduce developers to AWS IoT.

In this first bite, we’ll set up a very simple client that can create things in AWS IoT, send messages to those things, subscribe to topics to receive messages from things, and then clean up. The client is available through AWS Labs on GitHub. It’s used in this post to illustrate some IoT concepts, but will be used in later posts in this series, too.

Before we get into the details, let’s cover a few concepts:

AWS IoT is a managed service in the AWS Cloud that enables your devices to easily and securely connect to a rich ecosystem of cloud-based services or your own homegrown systems. The service enables this connectivity while reliably scaling to billions of devices sending trillions of messages. The service includes these building blocks:

Message Broker – Acts as a front door for your device to the AWS Cloud. The message broker exposes publish-and-subscribe messaging capabilities available for use by devices through transport protocols like MQTT, HTTP, and WebSocket.

Rules Engine – This is the point closest to the front door where your solution can make decisions about a device’s messages (for example, message filtering, routing messages to other services, and even a direct processing of messages in your solution). We will explore the facets of the rules engine in later posts.

Device Shadow – Sometimes referred to as a thing shadow, this is a logical representation of a physical device’s reported state or desired future state. Although the concept is a bit abstract, this building block provides a huge benefit for customers who build web- or mobile-based applications that need to interact with real devices at any scale. In later posts, we will explore the ways in which the device shadow is used in these interactions.

AWS APIs and the AWS Management Console – These are the mechanisms used to provision every AWS service. This post and all others in this series will include interaction with AWS through the APIs and AWS Management Console. In all Bites of IoT posts we assume you have an IAM user with developer-level permissions in an AWS account.

The relationships between these building blocks is:

AWS IoT has a message broker, rules engine, and device shadows.
AWS IoT is a part of the AWS Cloud.

Installing ELF

The AWS IoT ELF Python example client demonstrates how you can create things, send messages to things, subscribe to topics to receive messages from things, and clean up things in AWS IoT by using the AWS SDK for Python and the standard AWS IoT device client.

To ensure you have a simple working IoT client for this and subsequent blog posts, follow the Getting Started instructions for your computing environment. Make sure your AWS user can access the AWS IoT console.

Understanding the ELF Client and AWS IoT

We’ll go through the AWS IoT ELF commands to show how a device interacts with the building blocks of AWS IoT.

When you execute the create command, AWS IoT ELF will create a certificate, then a thing, and then attach the thing to the certificate. At this point, the device that possesses the certificate (in this case, IoT ELF) can use the AWS IoT service as if it actually is the logical thing.

Like a driver’s license that identifies a person and conveys the privilege to drive, a certificate identifies the thing and conveys the privilege to use the thing’s AWS IoT building blocks; specifically, the device can interact with the device shadow and message broker.

When you execute the send command, AWS IoT ELF will first create a policy that allows interaction with thing-specific topics for general messaging and device shadow messages. Then AWS IoT ELF will attach the policy to the certificate. The device can now publish and subscribe to messages on topics in the message broker and interact with the device shadow associated with the previously attached thing.

To continue the driver’s license analogy, even after drivers obtain a license, they can only use their car on roads in the country for which they have the privilege to drive. Similarly, when a device has a certificate, it can interact with the AWS IoT service, but the IoT policy dictates which topics or “lanes” the device can use to send and receive messages.

After a policy is configured for a device (or, in this case, AWS IoT ELF) the topics through which messages are sent convey the purpose of each message to a customer’s solution.

For example, messages sent or received on a topic elf/thing_0 are considered general purpose messages for the ELF in the solution. These messages might have a customer-defined format and convey telemetry or commands for use by the customer’s solution. But there are topics in AWS IoT that are reserved for interaction with building blocks of the AWS IoT service itself.

For example, a device named thing_0 can report its current state and subscribe to requests to change state through messages flowing through the thing’s device shadow topic $aws/things/thing_0/shadow/update. Messages sent and received on this topic must follow the format expected by the device shadow and are tracked as state transitions of the thing.

This topic-based interaction keeps messages flowing in the correct lanes of the solution. It is controlled by using AWS IoT to attach policies to the certificate possessed by the device.

Finally, when you execute the clean command, AWS IoT ELF detaches the IoT policy and thing from the certificate, deactivates the certificate, and then deletes the policy, thing, and certificate.

Subscribing to Topics in the AWS Console

Instead of using the subscribe command in AWS IoT ELF, we will now use the ELF to send messages to AWS IoT and then use the AWS IoT console to subscribe to those messages:

1. Sign in to the AWS Management Console, and then choose the AWS IoT icon.
2. In the AWS IoT console, choose MQTT Client.
3. In the Client ID field, type web-elf, and then choose Connect.
4. Choose Subscribe to topic and in the Subscription topic field, type elf/#. This will subscribe the console’s built-in MQTT client to the default AWS IoT ELF topic.

Note: These steps are similar to those described in View Device MQTT Messages in the AWS IoT Developer Guide.

Execute these commands from the directory in which you installed AWS IoT ELF:

python elf.py create
python elf.py send --duration 30

Go back the AWS IoT console, and for about 30 seconds, you will see messages being sent by AWS IoT ELF.

Specifically, what you’re seeing in the console is a result of:

1. The ELF creating messages to send.
2. The ELF sharing the certificate with AWS IoT to identify itself as a device mapped to a thing.
3. The ELF using the MQTT protocol to send a message to the elf/thing_0 topic.
4. The policy previously attached to the certificate being checked and, if the policy allows it, the message being sent on the elf/thing_0 topic.
5. The AWS IoT console’s built-in MQTT client indirectly subscribing to the elf/thing_0 topic. (The # in elf/# is a wildcard that means “everything from this point in the topic hierarchy and below.”)

Next Topics in the Series

Upcoming Bites of IoT posts will show simple routing examples using AWS IoT ELF and the rules engine. These examples will cover sending notifications, streaming and saving messages, republishing messages, computing on messages, and finally, searching messages.

After that, we’ll explore the not-so-shadowy world of device shadows to show how truly powerful this building block can be when added to an IoT solution designer’s toolbox.

I hope you’ve enjoyed taking your first small bite out of IoT. I look forward to offering up other tasty morsels in the near future.

May your success be swift and your solution scalable. Feel free to leave your feedback in the comments.

One of the biggest benefits of the Internet of Things (IoT) is the ability to get contextual insight from sensor data. Before you analyze sensor data, you may want to remove anomalies. Sometimes, though, you may want to analyze the anomalies or at least be notified of their presence.

In this blog post, we will focus on detecting contextual anomalies. An example of a contextual anomaly is a temperature measurement that suddenly jumps from 50 to 70 degrees. The 70-degree measurement may not be unusual, but it is unusual in a context in which previous measurements were much lower. For more information about anomaly detection, see the survey by Varun Chandola, et al. [1]

Prerequisites

You should be familiar with the AWS CLI and should have set up an Amazon Cognito identity pool. For information, see  Authenticate Users with Amazon Cognito Identity.

Although mathematical knowledge is not required, we encourage you to check out the references.

Contextual Anomaly Detection

The demo in this post uses wind velocity measurements from the San Francisco Wind Monitoring Data as the data set. You can download the formatted data set here. The algorithm we’ll use is called the probabilistic exponentially weighted moving average (PEWMA), which was proposed by Carter and Streilein [2]. The details of the algorithm are not covered in this post, but at a high level, the algorithm is calculating an average and standard deviation of the time-series data and evaluating the probability of observing the current point. If the probability is below a set threshold, then it is marked as an anomaly. A plot of the raw data and an example of the upper and lower bounds of what is considered to be normal behavior is shown here:

The data flow for this anomaly detection process is shown in the following figure:

Data is sent from our sensor to AWS IoT, where it is routed to AWS Lambda through the AWS IoT Rules Engine. Lambda executes the logic for anomaly detection and because the algorithm requires knowledge of previous measurements, uses Amazon DynamoDB as a key-value store. The Lambda function republishes the message along with parameters extracted from the PEWMA algorithm. The results can be viewed in your browser through a WebSocket connection to AWS IoT on your local machine. A variation of this flow is to route observations marked as anomalous to Amazon Elasticsearch Service (Amazon ES) or Amazon S3.

For the anomaly detection method, we are using AWS Lambda with Python 2.7. Use the following AWS CLI command to set up your DynamoDB table:

aws dynamodb create-table --table-name windDemo --attribute-definitions AttributeName=Station_Name,AttributeType=S  --key-schema AttributeName=Station_Name,KeyType=HASH --provisioned-throughput ReadCapacityUnits=50,WriteCapacityUnits=50

Now we need to create a role that will give trust permissions to Lambda and AWS IoT to execute actions on our behalf. To do this, save the following text as iot_lambda_role.json.

{
"Version": "2012-10-17",
"Statement": [{
   "Sid": "",
   "Effect": "Allow",
   "Principal": {
      "Service": ["lambda.amazonaws.com", "iot.amazonaws.com"]
   },
   "Action": "sts:AssumeRole"
 }]
}

The actions Lambda will execute on our behalf (access DynamoDB, AWS IoT, and logging) are defined in a policy document like the following. Save the following text as anom_det_policy.json.

{
 "Version": "2012-10-17",
 "Statement": [
  {
   "Sid": "Stmt1463425014235",
   "Action": [
    "dynamodb:GetItem",
    "dynamodb:PutItem"
   ],
   "Effect": "Allow",
   "Resource": "*"
  },
  {
   "Sid": "Stmt1463425053236",
   "Action": [
   "iot:Connect",
   "iot:Publish"
  ],
  "Effect": "Allow",
  "Resource": "*"
 },
 {
  "Effect": "Allow",
  "Action": [
   "logs:CreateLogGroup",
   "logs:CreateLogStream",
   "logs:PutLogEvents"
  ],
  "Resource": "arn:aws:logs:*:*:*"
  }
 ]
}

The following commands will create the role, create the policy, and attach the policy to the role.

aws iam create-role --role-name iot_lambda_role --assume-role-policy-document file://iot_lambda_role.json
aws iam create-policy --policy-name anom_det_policy --policy-document file://anom_det_policy.json
aws iam attach-role-policy --role-name iot_lambda_role --policy-arn arn:aws:iam::<your-aws-account-id-here>:policy/anom_det_policy

The next step is to create the Lambda function. Download the code for the Lambda function. The Lambda function will be executed every time a message is sent to the anom/detect topic. It will perform the PEWMA algorithm and republish the results to the anom/pred topic. Use the following command to create the Lambda function from the AWS CLI.

aws lambda create-function \
 --function-name anom_detector \
 --runtime python2.7 \
 --region us-east-1 \
 --role arn:aws:iam::<your-aws-account-id-here>:role/iot_lambda_role  \
 --handler lambda_function.lambda_handler \
 --memory-size 512 \
 --timeout 15 \
 --description "lambda function which implements probabilistic exponentially weighted moving average" \
 --zip-file fileb://lambda_function.zip

Now we need to create a rule that will send messages on a topic in AWS IoT to the Lambda function and give our rule permission to invoke Lambda. Copy and paste the following code into a JSON file and save it as anomaly_rule.json.

{
        "sql": "SELECT * FROM 'anom/detect'",
        "ruleDisabled": false,
        "actions": [{
          "lambda": {
            "functionArn": "arn:aws:lambda:us-east-1:<your-aws-account-id-here>:function:anom_detector"
           }
         }]
}

We will use this JSON message to create the rule and give it permission to call Lambda using the following AWS CLI commands.

aws iot create-topic-rule --rule-name anomaly_detection_lambda --topic-rule-payload file://anomaly_rule.json
aws lambda add-permission --function-name "anom_detector" --region "us-east-1" --principal iot.amazonaws.com --source-arn arn:aws:iot:us-east-1:<your-aws-account-id-here>:rule/anomaly_detection_lambda --source-account "<your-aws-account-id-here>" --statement-id "123123123" --action "lambda:InvokeFunction"

All of the resources required to run the demo are now created. To run the demo, download this script, and then use the following AWS CLI command to start sending data to AWS IoT.

python emit_json_data.py anom/detect SF36.json

Viewing Your Data Stream and Anomaly Boundaries

To connect to AWS IoT and view the output of the anomaly detection algorithm, download this WebSocket example. To use the websocket demo on your local machine, modify the config.js file to include your access key, secret key, Amazon Cognito pool ID, region, and AWS IoT endpoint. This websocket demo is only meant to be run on your local machine to be able to visualize the data stream and anomaly boundaries. Do not publish the html page in the demo to the public or in general any code that contains your API or secret key. If you don’t know your IoT endpoint, use the following AWS CLI command.

aws iot describe-endpoint

Now you can open the index.html file, connect to AWS IoT, and subscribe to the topic the anomaly detection algorithm is publishing to. You should start to see the plot of the raw data and the calculation of the anomaly boundaries. The anomaly boundaries set for the plot are 3.3 standard deviations away from the moving average, which corresponds to a threshold of 0.002. After the streaming plot is working, you can modify the alpha_0 and beta parameters in the Lambda function to see how those changes affect the anomaly boundary in real time.

The following figures shows what the streaming anomaly plot should look like in your browser:

Conclusion

This demo in this post has shown you how to perform anomaly detection for sensor data on a thing connected to AWS IoT. This approach could scale to many things. You can use a unique identifier for each sensor you want to monitor as a hash key in DynamoDB. A useful extension for this process would be to add a WHERE statement to the rule on anom/pred that filters the data so that only anomalous observations are routed to Amazon S3 or Amazon ES for analysis. We would love to hear your feedback!

Appendix: Tuning the PEWMA Algorithm to Your Data

The Lambda function has four parameters that you can change to tune the behavior of the algorithm. Two of these parameters are used to select pieces of the JSON messages and identify a DynamoDB table to use to persist state information for the algorithm. If you want to generalize the approach in this demo to work on a different data set, you will need to change a few parameters, including the name of the DynamoDB table (table_name) and and the JSON object (data_cols). You may also want to tweak alpha_0, beta, and threshold until the algorithm is identifying what you believe are anomalies.

alpha_0  This parameter is used to specify how much weight to give to the previous value of the mean in determining the moving average. The default value is 0.95. The closer this parameter is to 1, the smoother the moving average will be. The closer this parameter is to 0, the more quickly the moving average will adjust to changing patterns in the data.

beta  This parameter ranges from 0 to 1. It prevents anomalies from strongly shifting the mean. The larger the value, the more we limit the effect an anomalous point has on the mean. In the demo, we use a default of 0.5.

data_cols  The object of our AWS IoT JSON payload where we want to perform anomaly detection.

key_param  The object of our AWS IoT JSON payload to be used as the hash key in our DynamoDB table. Usually a device ID.

T  The number of points to use to calculate the initial value for the moving average. The default value is 30.

table_name  The name of the DynamoDB table used to persist data required during the execution of the PEWMA algorithm.

threshold  This parameter sets the limit for what we call an anomaly. The lower the value, the fewer points will be classified as anomalies.

The following plots illustrate the effect that some of the parameters have on the PEWMA algorithm.

threshold

Raising the threshold reduces the area that is considered normal.

alpha_0

You can think of alpha_0 as a smoothing parameter. The closer this parameter is to 1, the smoother the moving average will be.

beta

The closer this parameter is to 1, the more the effect of anomalies on the moving average and standard deviation is minimized. To perform a standard exponentially weighted moving average, set this parameter to 0.

References

[1] Chandola, Varun, Arindam Banerjee, and Vipin Kumar. “Anomaly detection: A survey.” ACM computing surveys (CSUR) 41.3 (2009): 15.
[2] Carter, Kevin M., and William W. Streilein. “Probabilistic reasoning for streaming anomaly detection.” Statistical Signal Processing Workshop (SSP), 2012 IEEE. IEEE, 2012.

We are proud to announce two new device SDKs for AWS IoT – the AWS IoT SDK for Java and the AWS IoT SDK for Python. These SDKs add to those already available for AWS IoT:

• Embedded C with porting guide
• JavaScript/Node.js
• Arduino Yún
• iOS (part of AWS Mobile SDK and samples for iOS)
• Android (part of AWS Mobile SDK and samples for Android)

The Java and Python SDKs help you easily and quickly connect your device or application to AWS IoT, send and receive messages using either MQTT or MQTT over the WebSocket protocol, and exchange state information with device shadows (sometimes referred to as thing shadows). All of the SDKs are open-sourced on GitHub, so you can use them as is or tweak them for your device or application.

Using the AWS IoT SDK for Java

You can get started with the SDK for Java by integrating it into an existing application or by trying the included sample applications. The SDK and samples are available both as source code in GitHub and binaries in Maven. For information about using the SDK, see the readme. In this post, we will show you how to load and run the samples from source code so you can step through and see the SDK in action.

Acquire and build the samples from source code

1. Clone the aws-iot-device-sdk-java repository to your development machine.

   $ git clone https://github.com/aws/aws-iot-device-sdk-java.git
   $ cd aws-iot-device-sdk-java

2. To build from the command line:
   a. Make sure you have Maven installed.
   b. Navigate to the aws-iot-device-sdk-java-samples folder.
   c. Run mvn clean install to build the project.
3. To build from an IDE like Eclipse:
   a. Make sure you have installed Maven support for your IDE.
   b. Navigate to the aws-iot-device-sdk-java-samples folder and load the pom.xml file in your IDE. (In Eclipse, from the File menu, choose Import, choose Maven, and then choose Existing Maven Projects.
   c. Build the project. Eclipse is configured by default to build automatically.

Run the samples

There are three samples applications included with the SDK.

• Publish/Subscribe sample: This sample consists of two publishers publishing one message per second to a test topic, sdkTest/sub. One subscriber subscribing to the same topic receives and prints the messages.
• Device Shadow sample: This example demonstrates how to keep the device connected with its shadow in the cloud. This is the recommended way of accessing a shadow and keeping it synchronized with the device. To see this in action, you can use the AWS IoT console to update the shadow and observe the corresponding state change in this sample application.
• Device Shadow Echo sample: This sample consists of a simple demo that uses shadow methods to send a shadow update and then retrieve it every second.

From the command line, use the following commands and parameters to execute the sample applications (assuming TLS mutual authentication is used) from the aws-iot-device-sdk-java-samples folder. If you are using Eclipse, specify the parameters in the Run/Debug Configurations of your project and then press F11 to launch it in debug mode. You can find the parameter values in the AWS IoT console by navigating to the thing and choosing Connect Device.

To run the Publish/Subscribe sample, use the following command:

$ mvn exec:java -Dexec.mainClass="com.amazonaws.services.iot.client.sample.pubSub.PublishSubscribeSample" -Dexec.args="-clientEndpoint .iot..amazonaws.com -clientId  -certificateFile  -privateKeyFile "

To run the Device Shadow sample, use the following command:

$ mvn exec:java -Dexec.mainClass="com.amazonaws.services.iot.client.sample.shadow.ShadowSample" -Dexec.args="-clientEndpoint .iot..amazonaws.com -clientId -thingName -certificateFile -privateKeyFile "

To run the Device Shadow Echo sample, use the following command:

$ mvn exec:java -Dexec.mainClass="com.amazonaws.services.iot.client.sample.shadowEcho.ShadowEchoSample" -Dexec.args="-clientEndpoint .iot..amazonaws.com -clientId -thingName -certificateFile -privateKeyFile "

Using the AWS IoT SDK for Python

You can get started with the SDK for Python by integrating it into an existing application or by trying the included sample applications. The SDK is available both as a package in PyPi and as source with samples on GitHub. For information about using the SDK, see the readme. In this post, we will show you how to load and run the samples.

Acquire the samples

Clone the aws-iot-device-sdk-python repository to your development machine.

$ git clone https://github.com/aws/aws-iot-device-sdk-python.git
$ cd aws-iot-device-sdk-python
$ python setup.py install

Run the samples

There are four samples applications included with the SDK:

1. BasicPubSub sample: This example demonstrates a simple MQTT publish/subscribe operation using AWS IoT. It first subscribes to a topic and registers a callback to print new messages. It then publishes to the same topic in a loop.
2. BasicPubSub with Amazon Cognito Session Token sample: This example builds on the first sample by using an Amazon Cognito Identity session token. Using the AWS SDK for Python (boto3), it first makes a request to Amazon Cognito to retrieve the access ID, the access key, and the session token for temporary authentication. Using the AWS IoT SDK for Python, it then uses these credentials to connect to AWS IoT and communicate data/messages using MQTT over the WebSocket protocol.
3. ThingShadowEcho sample: This example demonstrates how a device communicates with AWS IoT, syncing data into the device shadow in the cloud and receiving commands from another app. When there is a new command from the app to change the desired state of the device, the device receives this request on a delta callback function and applies the change by publishing it as the reported state. To see this in action, you can use the AWS IoT console to update the shadow and observe the corresponding state change in this sample application.
4. BasicShadow sample: This example demonstrates the use of basic shadow operations (update/delta). It has two scripts that perform shadow update operations and listen on delta topics, respectively. The example shows how a shadow update request triggers delta events.

From the command line, use the following commands and parameters to execute the sample applications. You can find the parameter values for the commands in the AWS IoT console by navigating to the thing and choosing Connect Device.

To run the BasicPubSub sample, use the command below that corresponds to your connection type:

$ # Certificate based mutual authentication
$ python basicPubSub.py -e -r -c -k
$ # MQTT over WebSocket
$ python basicPubSub.py -e -r -w

For the BasicPubSub with Amazon Cognito Session Token sample, you will need:

• Your Amazon Cognito identity pool ID.
• A role that allows unauthenticated identities to connect. Make sure that the policy attached to the unauthenticated role has permissions to access the required AWS IoT APIs. For more information about Amazon Cognito, see here.

To run the sample, use the following command:

$ python basicPubSub_CognitoSTS.py -e -r -C

To run the ThingShadowEcho sample, use the command below that corresponds to your connection type:

$ # Certificate based mutual authentication
$ python ThingShadowEcho.py -e -r -c -k
$ # MQTT over WebSocket
$ python ThingShadowEcho.py -e -r -w

To run the BasicShadow sample, first start the basicShadowDeltaListener. Use the command below that corresponds to your connection type:

$ # Certificate-based mutual authentication
$ python basicShadowDeltaListener.py -e -r -c -k
$ # MQTT over WebSocket
$ python basicShadowDeltaListener.py -e -r -w

Then, start the basicShadowUpdater. Use the command below that corresponds to your connection type:

$ # Certificate-based mutual authentication
$ python basicShadowUpdater.py -e -r -c -k
$ # MQTT over WebSockets
$ python basicShadowUpdater.py -e -r -w

Summary

We hope you found this blog post useful. We regularly update all of the SDKs as new features are added to AWS IoT. We will also be adding more SDKs in the coming months. Stay tuned to this blog and GitHub for updates. Feel free to leave your feedback in the comments.

The AWS IoT platform enables you to connect your devices and build scalable Internet of Things solutions with the breadth of Amazon Web Services.

When shopping for a new tool, many developers want to be able to test-drive options before making a choice. When evaluating an IoT solution, it’s not practical to do so at scale with physical devices. Building a sensor simulator is the next best choice. With a sensor simulator built on top of AWS Lambda, you can elastically generate device sensors that report their state to the cloud.

In this blog post, you will learn how to implement a simulator that feeds data sets into AWS IoT. You’ll see how to monitor the simulators in real time, visualize data, configure your own sensor, and add hooks to start simulators from your projects.

Why Simulate? Why Lambda?

One device is cheap, but you’re building at a bigger scale than that. What will data for all of your devices look like in AWS? Simulation is an inexpensive means to test a cloud platform at scale, without first making changes to your real devices.

AWS Lambda is a perfect partner for simulation. On-demand, serverless compute time lets you try out a fleet of devices on AWS IoT with ease. In the Lambda function provided in this post, you’ll be able to specify the following each time you run the function:

• number of devices to simulate
• simulation duration
• publish frequency
• message topic
• data set
• Amazon Elasticsearch Service (Amazon ES) domain

The simulator connects to AWS IoT and publishes data as messages over MQTT. You can simulate many devices with the same configuration, or invoke multiple Lambda functions with unique configurations. You can test the functioning of AWS IoT policies by making changes to your Lambda execution role.

Running the AWS CloudFormation Template

As part of this blog post, we are providing an AWS CloudFormation template to get you up and running with Lambda simulators quickly. This template creates a stack of AWS resources that do the following:

• a Lambda function that simulates devices publishing to AWS IoT.
• an Amazon ES domain for visualizing published data.
• an AWS IoT rule that forwards simulator messages to Amazon ES.
• an Amazon S3 bucket to store your simulator data sets.
• all required AWS Identity and Access Management (IAM) roles.

Note: This CloudFormation template is written to be executed in the US West (Oregon) Region. When creating Lambda functions through CloudFormation, the Lambda simulator source must be hosted in the same region as the S3 bucket. To use this template in another AWS region, download the Lambda simulator source, and you can update the CloudFormation template to point to a bucket you own with the source there.

To run the CloudFormation template and create these resources:

1. Sign in to the AWS Management Console and open the AWS CloudFormation console at https://console.aws.amazon.com/cloudformation/home?region=us-west-2.
2. Choose Create Stack.
3. In the Template section, select Specify an Amazon S3 template URL to type ot paste the following URL, and then choose Next.
   https://s3-us-west-2.amazonaws.com/iot-us-west-2-simulator-publicbucket/createSimulator.json
4.  In the Specify Details section, type a name for your stack (for example, my-iot-simulator-stack).
5. In the Parameters section, type a name for your S3 bucket (for example, YOURNAME-iot-simulator-data-sets), and then choose Next.
6. On the Options page, choose Next.
7. On the Review page, under Capabilities, select the I acknowledge that this template might cause AWS CloudFormation to create IAM resources check box, and choose Create.

CloudFormation will now create your AWS resources. It can take up to ten minutes to create an Amazon Elasticsearch Service domain.

Starting the Simulator

Until you’re familiar with the configuration parameters, you should start the simulator manually. To do so, navigate to the AWS Lambda console. The simulator will have the prefix of your CloudFormation stack (for example, my-iot-simulator-stack-AWSLambda-ABC123). In your function’s view, from the Actions drop-down menu, choose Configure test event.

This is where you will set the runtime configuration of your simulator. The following sample JSON configuration is followed by an explanation of options.

 {
  "esEndpoint": "search-my-iot-elastic-1ubx76h2qqoan-3dijocbhik332asd4sryz4au5q.us-west-2.es.amazonaws.com",
  "esDomainArn": "arn:aws:es:us-west-2:111122223333:domain/my-iot-elastic-1ubxg6r2qwean",
  "topic": "test/topic",
  "simTime": 10000,
  "interval": 500,
  "numDevice": 3
 }

• esEndpoint: your Amazon ES domain endpoint.
• esDomainArn: your Amazon ES domain.
• topic: the AWS IoT topic this simulator will publish to.
• simTime: the duration of this simulator, in milliseconds (max 240000).
• interval: the interval between publishes, in milliseconds.
• numDevice: the number of devices of this configuration to run.
• bucketName: (OPTIONAL) the S3 bucket with your simulator data set to publish. Defaults to iot-us-west-2-simulator-publicbucket.
• key: (OPTIONAL) the S3 key of your simulator data file to read for published messages. Defaults to physiological-data.csv.

Replace the values shown for esEndpoint and esDomainArn with those for your domain. You can find these values in the Amazon Elasticsearch Service console after you click on your domain.

Note: If you omit the optional bucketName and key parameters, a data set file provided with the simulator will be used. It contains records of a sleep healthcare monitoring device. You can review this data set here.

In the AWS Lambda console, when you choose Save and test, your configuration JSON will be saved and the Lambda function will start with this configuration. If you scroll down, you’ll see Execution result: succeeded. Congratulations, you’ve run your first device simulator!

Visualize the Data

There are two ways to see the simulator data in AWS:

• In real time, in the AWS IoT console.
• In a data visualization dashboard called Kibana, which is included with the Amazon ES domain.

In the AWS IoT console, navigate to the MQTT Client. Enter or generate a client ID, and then choose Connect. Choose Subscribe to topic, enter the topic you specified in your Lambda configuration, and then choose Subscribe. If your Lambda simulator is running, you should now see these messages arriving at the interval you specified, times the number of devices specified. If your simulator has already run, in the AWS Lambda console, choose Test again to restart it.

To visualize the published data from the simulator in Kibana, open the Amazon Elasticsearch Service console. Find your Amazon ES domain in the list and choose it. Open the Kibana link in a new tab.

On your first visit to Kibana, you’ll see a settings page where you can choose an index pattern. This tells Kibana which indices to surface in your queries. Clear the Index contains time-based events box (you may want this in the future for your own data sets), and in Index name or pattern, type “index_*”. Choose Create.

Choose the Discover tab to see your simulated data and start using Kibana. For more information about configuring visualizations of AWS IoT data in Kibana, see another of our blog posts.

Extending the Simulator

Now that you have a working device simulator, let’s break down the important bits of code in the simulator Lambda function in case you want to extend it. This simulator will read through the input sample data and publish one message per line, per device, until the end of file, duration, or maximum Lambda runtime is reached.

For example, if the input sample data is 10 lines long, the number of devices to simulate is 30, and the interval is 1000 ms, then over 10 seconds of runtime, this simulator will make 30 connections to AWS IoT and publish a total of 300 messages.

In index.js, the event handler calls createResources(Object, Function). This exists to ensure that your Amazon ES domain is ready and your AWS IoT rule is set up to feed it incoming published messages. It also fetches a CSV file from S3, which will be parsed as messages for your simulated devices to publish.

 createResources(event, (err, data) => {
   if (err) {
     console.log(err);
     return;
   }
   const s3 = new AWS.S3({
     region: 'us-west-2'
   });
   var bucket = event.bucketName;
   var key = decodeURIComponent(event.key).replace(/\+/g, " ");
 
   s3.getObject({
     Bucket: bucket,
     Key: key
   }, (err, data) => {
     if (err) {
       context.fail("Failed to read config file");
       return;
     }
     event.data = data.Body.toString(); // attach file content to event
     const iot = new AWS.Iot();
     iot.describeEndpoint({}, (err, data) => {
       if (err) {
         console.log(err);
         return;
       }
       event.endpoint = data.endpointAddress;
       processText(event, context);
     });
   });
 });
 

After this is complete, the processText(Object, Object) function will be called. It will ingest your device sample data with parseData(Object, Integer) and then set up one MQTT client per device to be simulated. This is where you can adjust the way in which the MQTT clients are created.

 function processText(params, context) {
   const mqttController = new mqtt.ClientControllerCache();
   const jsonData = parseData(params, params.numDevice);
   for (var i = 0; i < params.numDevice; i++) {
     var connectOpts = {
       accessKey: params.accessKey,
       clientId: `${Math.random().toString(36).substring(2,12)}`, // 10-bit random string
       endpoint: params.endpoint,
       secretKey: params.secretKey,
       sessionToken: params.sessionToken,
       regionName: params.region,
       topic: params.topic
     };
     var simOpts = {
       simTime: params.simTime,
       interval: params.interval,
       index: i
     };
     createMqttClient(connectOpts, simOpts, mqttController, jsonData, context);
   }
 }
 

The data parsing function, parseData(Object, Integer), reads in your sample data file and splits by new lines and then by comma characters. (This is the standard way of reading in a CSV file.) If you have a different file format to parse, you can edit the way this function works. To read in JSON, you’ll want to iterate over keys or a JSON array value.

 function parseData(params, numDevice) {
   var dataJSON = [];
   const lines = params.data.split('\n');
   var lineNumber = lines.length;
   for (var i = 0; i < lineNumber; i++) {
     var columns = lines[i].trim().split(',');
     var dev = [];
     for (var j = 0; j < numDevice; j++) { 
       var clientId = 'client_' + j + '@' + params.endpoint;
       dev.push({
         clientId: clientId,
         field: columns[j]
       });
     }
     dataJSON.push(dev);
   }
   return dataJSON;
 }
 

After parsing your data, the next step is to create one MQTT client per device to be simulated. This Lambda function uses MQTT over the WebSocket protocol to connect and publish messages to AWS IoT. The createMqttClient(Object, Object, Object, Array, Object) function sets up event handlers for the MQTT protocol events.

 function createMqttClient(connectOpts, simOpts, mqttController, jsonData, context) {
   var cbs = {
     onConnect: onConnect,
     onSubSuccess: onSubSuccess,
     onMessageArrived: onMessageArrived,
     onConnectionLost: onConnectionLost
   };
   var clientController = mqttController.getClient(connectOpts, cbs);
   /* ... define callback functions ... */
 }
 

The onConnect() function defines what happens when each device simulator completes the connection to AWS IoT. In this case, it then subscribes to topic, as defined in the Lambda event input (for example, test/topic). The onSubSuccess() function defines what happens when the subscription to a topic is successful. In this case, it starts the JavaScript interval, which will start publishing messages from your sample data.

 function onConnect() {
   clientController.subscribe();
 }
 function onMessageArrived(data) {
   // do nothing
 }
 function onSubSuccess() {
   var index = 0;
   var interval = setInterval(() => {
     var line = jsonData[index++][simOpts.index];
     clientController.publish(line);
   }, simOpts.interval);
   setTimeout(() => {
     clearInterval(interval);
     clientController.disconnect();
     setTimeout(() => { // set drain time to disconnect all connections
       context.succeed();
     }, 1000);
   }, simOpts.simTime);
 }
 function onConnectionLost() {
   console.log('Connection is lost');
 }
 

The simulator Lambda function is designed to simulate sensing devices; it does not define behavior for responding to incoming messages. If you’d like to define an actuator device or a bi-directional sensing/actuating device, we recommend that you add simulation logic to the onMessageArrived() function. By expanding the scope of topics subscribed and behaviors to take when new messages arrive, you can quickly define a device simulator that sends and receives messages.

Adding Hooks

The simplest way to start the device simulator is to send test events from the AWS Lambda console. If you’re wondering how to spin up a fleet of different devices or dynamically start simulators, we’ll describe ways to invoke Lambda functions.

One great way to define interfaces to Lambda is with Amazon API Gateway. With API Gateway, you can write an HTTP or HTTPS interface to invoke your Lambda functions. By wrapping your device simulator, you can write a web service to spin up device simulators either with a preconfigured simulator template or by passing HTTP parameters to dynamically define devices. This method makes it possible for web application and service developers to build on top of your simulator pattern. Now you can have a web hook, IFTTT, or Zapier plugin to start simulators!

Another pattern to start simulators is on a schedule with Amazon CloudWatch Events. You can specify a Cron expression that will set up a device fleet at certain times and days. Although you can add a CloudWatch event source in the AWS Lambda console, for this task the best practice is to add it from the CloudWatch console. It’s easier to configure the Lambda event input required to start your simulator. Alternatively, you can create additional defaults in your Lambda function to avoid passing in any parameters.

A third way to invoke your device simulators is in response to a message published to AWS IoT. You can set up a rule in AWS IoT that invokes your Lambda function to start up a simulated device fleet (with an AWS IoT Button perhaps?). Alternatively, you can model a fleet of devices based on a single prototype. Store all of a real device’s messages in an Amazon DynamoDB table with one rule, and then use any Lambda invocation method to replay your real device as a simulated fleet.

Finally, what better way to test your solution architecture than by running a fleet of simulated devices against it to see where it can be improved? Build a step into your workflow with Amazon Simple Workflow Service that starts up a simulation fleet as part of your flow. Or, as part of a code deployment in AWS CodeDeploy, you can send a deployment notification to an Amazon Simple Notification Service topic, which then invokes your Lambda simulator fleet to kick off a job that tests your new version of software.

Summary

This blog post provides you with a CloudFormation template for getting started with device simulation in AWS IoT and AWS Lambda. It steps you through running the template and starting the simulator. It also gives you ideas for editing, extending, and invoking the simulator. Feel free to leave comments here or share your stories about device simulation and let us know what you build in the AWS IoT forums!