Adding Nodes Automatically in AWS OpsWorks for Chef Automate
This topic describes how to add Amazon Elastic Compute Cloud (Amazon EC2) nodes to your Chef server automatically.
In Add Nodes for the Chef Server to Manage, you learned how to use the
knife
bootstrap
command to add one node at a time to your Chef server. The code
in this topic shows how to add nodes automatically using the unattended method. The
recommended method of unattended (or automatic) association of new nodes is to configure the
Chef Client
Cookbook. Before you run the chef-client
agent, upload the Chef Client
cookbook to your Chef server, and then install the chef-client
agent in service
mode with, for example, an HTTPD role, as shown in the following sample command.
chef-client -r "chef-client,role[httpd]"
To communicate with the Chef server, the chef-client
agent software must
have access to the public key of the client node. You can generate a public-private key pair
in Amazon EC2, and then pass the public key to the AWS OpsWorks associate-node
API call with
the node name. The script shown in this topic, included in the Starter Kit, gathers your
organization name, server name, and server endpoint for you. This ensures that the node is
associated with the Chef server, and the chef-client
agent software that runs on
the node can communicate with the server after matching the private key.
For information about how to disassociate a node, see Disassociate a Node from an AWS OpsWorks for Chef Automate Server in this guide,
and disassociate-node
in the AWS OpsWorks for Chef Automate API documentation.
Supported Operating Systems
For the current list of supported operating systems for nodes, see the Chef website.
Step 1: Create an IAM Role to Use as Your Instance Profile
Create an AWS Identity and Access Management (IAM) role to use as your EC2 instance profile, and attach the
following policy to the IAM role. This policy allows the AWS OpsWorks for Chef Automate
(opsworks-cm
) API to communicate with the EC2 instance during node
registration. For more information about instance profiles, see Using Instance Profiles in the Amazon EC2 documentation. For information about how
to create an IAM role, see Creating an IAM Role in the Console in the Amazon EC2 documentation.
{
"Version": "2012-10-17",
"Statement": [
{
"Action": [
"opsworks-cm:AssociateNode",
"opsworks-cm:DescribeNodeAssociationStatus"
],
"Effect": "Allow",
"Resource": [
"*"
]
}
]
}
Step 2: Install the Chef Client Cookbook
If you have not done so already, follow the steps in Use Berkshelf to Get Cookbooks from a Remote Source to ensure that your Berksfile references the Chef Client cookbook and installs the cookbook.
Step 3: Create Instances by Using an Unattended Association Script
To create EC2 instances, you can copy the following code to the
userdata
section of EC2 instance instructions, Auto Scaling group launch configurations, or an AWS CloudFormation template. You do not need to prefill your own values in this script. For more information about adding scripts to user data, see Running Commands on Your Linux Instance at Launch in the Amazon EC2 documentation.This script runs the
opsworks-cm
APIassociate-node
command to associate a new node with your Chef server. In this release, it also installs the current version of the AWS CLI on the node for you, in case it is not already running the most up-to-date version.By default, the name of the new registered node is the instance ID, but you can change the name by modifying the value of the
NODE_NAME
variable. Because changing the organization name in the Chef console UI is currently not possible, leaveCHEF_ORGANIZATION
set todefault
.#!/bin/bash # Required settings NODE_NAME="$(curl --silent --show-error --retry 3 http://169.254.169.254/latest/meta-data/instance-id)" # This uses the EC2 instance ID as the node name REGION="
region
" # Valid values are us-east-1, us-west-2, or eu-west-1 CHEF_SERVER_NAME="serverName
" # The name of your Chef server CHEF_SERVER_ENDPOINT="endpoint
" # Provide the FQDN or endpoint; it's the string after 'https://' # Optional settings CHEF_ORGANIZATION="default" # Leave as "default"; do not change. AWS OpsWorks for Chef Automate always creates the organization "default" NODE_ENVIRONMENT="" # e.g. development, staging, onebox ... CHEF_CLIENT_VERSION="12.16.42" # latest if empty # Recommended: upload the chef-client cookbook from the chef supermarket https://supermarket.chef.io/cookbooks/chef-client # Use this to apply sensible default settings for your chef-client configuration like logrotate, and running as a service. # You can add more cookbooks in the run list, based on your needs RUN_LIST="" # e.g. "recipe[chef-client],recipe[apache2]" # --------------------------- set -e -o pipefail AWS_CLI_TMP_FOLDER=$(mktemp --directory "/tmp/awscli_XXXX") CHEF_CA_PATH="/etc/chef/opsworks-cm-ca-2016-root.pem" install_aws_cli() { # see: http://docs.aws.amazon.com/cli/latest/userguide/installing.html#install-bundle-other-os cd "$AWS_CLI_TMP_FOLDER" curl --retry 3 -L -o "awscli-bundle.zip" "https://s3.amazonaws.com/opsworks-cm-${REGION}-prod-default-assets/misc/awscli-bundle-opsworks-cm.zip" unzip "awscli-bundle.zip" ./awscli-bundle/install -i "$PWD" } aws_cli() { "${AWS_CLI_TMP_FOLDER}/bin/aws" opsworks-cm --region "${REGION}" --output text "$@" --server-name "${CHEF_SERVER_NAME}" } associate_node() { client_key="/etc/chef/client.pem" mkdir /etc/chef ( umask 077; openssl genrsa -out "${client_key}" 2048 ) aws_cli associate-node \ --node-name "${NODE_NAME}" \ --engine-attributes \ "Name=CHEF_ORGANIZATION,Value=${CHEF_ORGANIZATION}" \ "Name=CHEF_NODE_PUBLIC_KEY,Value='$(openssl rsa -in "${client_key}" -pubout)'" } write_chef_config() { ( echo "chef_server_url 'https://${CHEF_SERVER_ENDPOINT}/organizations/${CHEF_ORGANIZATION}'" echo "node_name '${NODE_NAME}'" echo "ssl_ca_file '${CHEF_CA_PATH}'" ) >> /etc/chef/client.rb } install_chef_client() { # see: https://docs.chef.io/install_omnibus.html curl --silent --show-error --retry 3 --location https://omnitruck.chef.io/install.sh | bash -s -- -v "${CHEF_CLIENT_VERSION}" } install_trusted_certs() { curl --silent --show-error --retry 3 --location --output "${CHEF_CA_PATH}" \ "https://s3.amazonaws.com/opsworks-cm-${REGION}-prod-default-assets/misc/opsworks-cm-ca-2016-root.pem" } wait_node_associated() { aws_cli wait node-associated --node-association-status-token "$1" } install_aws_cli node_association_status_token="$(associate_node)" install_chef_client write_chef_config install_trusted_certs wait_node_associated "${node_association_status_token}" if [ -z "${NODE_ENVIRONMENT}" ]; then chef-client -r "${RUN_LIST}" else chef-client -r "${RUN_LIST}" -e "${NODE_ENVIRONMENT}" fi
Other Methods of Automating Repeated Runs of chef-client
Although more difficult to achieve, and not recommended, you can run the script in this
topic solely as part of standalone instance user data, use a AWS CloudFormation template to add it to
new instance user data, configure a cron
job to run the script regularly, or
run chef-client
within a service. However, we recommend the Chef Client
Cookbook method because of the following disadvantages of other automation
techniques:
The
chef-client
agent runs in the foreground of instance processes, and logs tostdout
. Because it is part of instance user data, the output is stored in/var/log/cloud-init-output.log
. This directory is not rotated—or purged—and eventually runs out of disk space.Automation of this script doesn't survive or resume automatically if the node is restarted, or the process is terminated in some other way (for example, if the Chef server is restarted).
Because
chef-client
runs processes in the foreground, its tasks in an instance's user data would never finish, and the instance could not signal other processes that it is finished booting. On Ubuntu nodes, for example, this can prevent other services from starting, and automatic updates from running.
Optionally, you can add one or more of the following parameters for
chef-client
in the last lines of the preceding script:
--runlist
RunlistItem1
,RunlistItem2
,...Adds run list items to the initial Chef run list.
--environment ENVIRONMENT
Sets the Chef environment on the node.
--json-attributes JSON_ATTRIBS
Loads attributes from a JSON file or URL.
For a complete list of parameters you can provide to chef-client
, see
the Chef
documentation.
The user data shown in this topic describes how to install chef-client
, and
perform authentication similarly to knife bootstrap
. To run specific recipes
on a new node, associate a role or environment with the node, and then pass the parameter
to the chef-client
command.