Menu
AWS OpsWorks
User Guide (API Version 2013-02-18)

Adding Nodes Automatically in AWS OpsWorks for Chef Automate

This topic describes how to add Amazon Elastic Compute Cloud (Amazon EC2) nodes to your Chef server automatically. In Add Nodes for the Chef Server to Manage, you learned how to use the knife bootstrap command to add one node at a time to your Chef server. The code in this topic shows how to add nodes automatically using the unattended method. The recommended method of unattended (or automatic) association of new nodes is to configure the Chef Client Cookbook. Before you run the chef-client agent, upload the Chef Client cookbook to your Chef server, and then install the chef-client agent in service mode with, for example, an HTTPD role, as shown in the following sample command.

chef-client -r "chef-client,role[httpd]"

To communicate with the Chef server, the chef-client agent software must have access to the public key of the client node. You can generate a public-private key pair in Amazon EC2, and then pass the public key to the AWS OpsWorks associate-node API call with the node name. The script shown in this topic, included in the Starter Kit, gathers your organization name, server name, and server endpoint for you. This ensures that the node is associated with the Chef server, and the chef-client agent software that runs on the node can communicate with the server after matching the private key.

For information about how to disassociate a node, see Disassociate a Node from an AWS OpsWorks for Chef Automate Server in this guide, and disassociate-node in the AWS OpsWorks for Chef Automate API documentation.

Supported Operating Systems

For the current list of supported operating systems for nodes, see the Chef website.

Step 1: Create an IAM Role to Use as Your Instance Profile

Create an AWS Identity and Access Management (IAM) role to use as your EC2 instance profile, and attach the following policy to the IAM role. This policy allows the AWS OpsWorks for Chef Automate (opsworks-cm) API to communicate with the EC2 instance during node registration. For more information about instance profiles, see Using Instance Profiles in the Amazon EC2 documentation. For information about how to create an IAM role, see Creating an IAM Role in the Console in the Amazon EC2 documentation.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": [
        "opsworks-cm:AssociateNode",
        "opsworks-cm:DescribeNodeAssociationStatus"
      ],
      "Effect": "Allow",
      "Resource": [
        "*"
      ]
    }
  ]
}

Step 2: Install the Chef Client Cookbook

If you have not done so already, follow the steps in Use Berkshelf to Get Cookbooks from a Remote Source to ensure that your Berksfile references the Chef Client cookbook and installs the cookbook.

Step 3: Create Instances by Using an Unattended Association Script

  • To create EC2 instances, you can copy the following code to the userdata section of EC2 instance instructions, Auto Scaling group launch configurations, or an AWS CloudFormation template. You do not need to prefill your own values in this script. For more information about adding scripts to user data, see Running Commands on Your Linux Instance at Launch in the Amazon EC2 documentation.

    This script runs the opsworks-cm API associate-node command to associate a new node with your Chef server. In this release, it also installs the current version of the AWS CLI on the node for you, in case it is not already running the most up-to-date version.

    By default, the name of the new registered node is the instance ID, but you can change the name by modifying the value of the NODE_NAME variable. Because changing the organization name in the Chef console UI is currently not possible, leave CHEF_ORGANIZATION set to default.

    #!/bin/bash
    
    # Required settings
    NODE_NAME="$(curl --silent --show-error --retry 3 http://169.254.169.254/latest/meta-data/instance-id)" # This uses the EC2 instance ID as the node name
    REGION="region" # Valid values are us-east-1, us-west-2, or eu-west-1
    CHEF_SERVER_NAME="serverName" # The name of your Chef server
    CHEF_SERVER_ENDPOINT="endpoint" # Provide the FQDN or endpoint; it's the string after 'https://'
    
    # Optional settings
    CHEF_ORGANIZATION="default"    # Leave as "default"; do not change. AWS OpsWorks for Chef Automate always creates the organization "default"
    NODE_ENVIRONMENT=""            # e.g. development, staging, onebox ...
    CHEF_CLIENT_VERSION="12.16.42" # latest if empty
    
    # Recommended: upload the chef-client cookbook from the chef supermarket  https://supermarket.chef.io/cookbooks/chef-client
    # Use this to apply sensible default settings for your chef-client configuration like logrotate, and running as a service.
    # You can add more cookbooks in the run list, based on your needs
    RUN_LIST="" # e.g. "recipe[chef-client],recipe[apache2]"
    
    # ---------------------------
    set -e -o pipefail
    
    AWS_CLI_TMP_FOLDER=$(mktemp --directory "/tmp/awscli_XXXX")
    CHEF_CA_PATH="/etc/chef/opsworks-cm-ca-2016-root.pem"
    
    install_aws_cli() {
      # see: http://docs.aws.amazon.com/cli/latest/userguide/installing.html#install-bundle-other-os
      cd "$AWS_CLI_TMP_FOLDER"
      curl --retry 3 -L -o "awscli-bundle.zip" "https://s3.amazonaws.com/opsworks-cm-${REGION}-prod-default-assets/misc/awscli-bundle-opsworks-cm.zip"
      unzip "awscli-bundle.zip"
      ./awscli-bundle/install -i "$PWD"
    }
    
    aws_cli() {
      "${AWS_CLI_TMP_FOLDER}/bin/aws" opsworks-cm --region "${REGION}" --output text "$@" --server-name "${CHEF_SERVER_NAME}"
    }
    
    associate_node() {
      client_key="/etc/chef/client.pem"
      mkdir /etc/chef
      ( umask 077; openssl genrsa -out "${client_key}" 2048 )
    
      aws_cli associate-node \
        --node-name "${NODE_NAME}" \
        --engine-attributes \
        "Name=CHEF_ORGANIZATION,Value=${CHEF_ORGANIZATION}" \
        "Name=CHEF_NODE_PUBLIC_KEY,Value='$(openssl rsa -in "${client_key}" -pubout)'"
    }
    
    write_chef_config() {
      (
        echo "chef_server_url   'https://${CHEF_SERVER_ENDPOINT}/organizations/${CHEF_ORGANIZATION}'"
        echo "node_name         '${NODE_NAME}'"
        echo "ssl_ca_file       '${CHEF_CA_PATH}'"
      ) >> /etc/chef/client.rb
    }
    
    install_chef_client() {
      # see: https://docs.chef.io/install_omnibus.html
      curl --silent --show-error --retry 3 --location https://omnitruck.chef.io/install.sh | bash -s -- -v "${CHEF_CLIENT_VERSION}"
    }
    
    install_trusted_certs() {
      curl --silent --show-error --retry 3 --location --output "${CHEF_CA_PATH}" \
        "https://s3.amazonaws.com/opsworks-cm-${REGION}-prod-default-assets/misc/opsworks-cm-ca-2016-root.pem"
    }
    
    wait_node_associated() {
      aws_cli wait node-associated --node-association-status-token "$1"
    }
    
    install_aws_cli
    node_association_status_token="$(associate_node)"
    install_chef_client
    write_chef_config
    install_trusted_certs
    wait_node_associated "${node_association_status_token}"
    
    if [ -z "${NODE_ENVIRONMENT}" ]; then
      chef-client -r "${RUN_LIST}"
    else
      chef-client -r "${RUN_LIST}" -e "${NODE_ENVIRONMENT}"
    fi

Other Methods of Automating Repeated Runs of chef-client

Although more difficult to achieve, and not recommended, you can run the script in this topic solely as part of standalone instance user data, use a AWS CloudFormation template to add it to new instance user data, configure a cron job to run the script regularly, or run chef-client within a service. However, we recommend the Chef Client Cookbook method because of the following disadvantages of other automation techniques:

  • The chef-client agent runs in the foreground of instance processes, and logs to stdout. Because it is part of instance user data, the output is stored in /var/log/cloud-init-output.log. This directory is not rotated—or purged—and eventually runs out of disk space.

  • Automation of this script doesn't survive or resume automatically if the node is restarted, or the process is terminated in some other way (for example, if the Chef server is restarted).

  • Because chef-client runs processes in the foreground, its tasks in an instance's user data would never finish, and the instance could not signal other processes that it is finished booting. On Ubuntu nodes, for example, this can prevent other services from starting, and automatic updates from running.

Optionally, you can add one or more of the following parameters for chef-client in the last lines of the preceding script:

  • --runlist RunlistItem1,RunlistItem2,...

    Adds run list items to the initial Chef run list.

  • --environment ENVIRONMENT

    Sets the Chef environment on the node.

  • --json-attributes JSON_ATTRIBS

    Loads attributes from a JSON file or URL.

For a complete list of parameters you can provide to chef-client, see the Chef documentation.

The user data shown in this topic describes how to install chef-client, and perform authentication similarly to knife bootstrap. To run specific recipes on a new node, associate a role or environment with the node, and then pass the parameter to the chef-client command.