Archive for category Cloud
The Evolving Role of DevOps in Emerging Technologies: Part 1/2
Posted by Gary A. Stafford in Cloud, DevOps, Technology Consulting on May 22, 2017
Growth of DevOps
The adoption of DevOps practices by global organizations has become mainstream, according to many recent industry studies. For instance, a late 2016 study, conducted by IDG Research for Unisys Corporation of global enterprise organizations, found 38 percent of respondents had already adopted DevOps, while another 29 percent were in the planning phase, and 17 percent in the evaluation stage. Adoption rates were even higher, 49 percent versus 38 percent, for larger organizations with 500 or more developers.
Another recent 2017 study by Red Gate Software, The State of Database DevOps, based on 1,000 global organizations, found 47 percent of the respondents had already adopted DevOps practices, with another 33 percent planning on adopting DevOps practices within the next 24 months. Similar to the Unisys study, prior adoption rates were considerably higher, 59 percent versus 47 percent, for larger organizations with over 1,000 employees.
Emerging Technologies
Although DevOps originated to meet the needs of Agile software development to release more frequently, DevOps is no longer just continuous integration and continuous delivery. As more organizations undergo a digital transformation and adopt disruptive technologies to drive business success, the role of DevOps continues to evolve and expand.
Emerging technology trends, such as Machine Learning, Artificial Intelligence (AI), and Internet of Things (IoT/IIoT), serve to both influence DevOps practices, as well as create the need for the application of DevOps practices to these emerging technologies. Let’s examine the impact of some of these emerging technology trends on DevOps in this brief, two-part post.
Mobile
Although mobile application development is certainly not new, DevOps practices around mobile continue to evolve as mobile becomes the primary application platform for many organizations. Mobile applications have unique development and operational requirements. Take for example UI functional testing. Whereas web application developers often test against a relatively small matrix of popular web browsers and operating systems (Desktop Browser Market Share – Net Application.com), mobile developers must test against a continuous outpouring of new mobile devices, both tablets and phones (Test on the right mobile devices – BroswerStack). The complexity of automating the testing of such a large number mobile devices has resulted in the growth of specialized cloud-based testing platforms, such as BrowserStack and SauceLabs.
Cloud
Similar to Mobile, the Cloud is certainly not new. However, as more firms move their IT operations to the Cloud, DevOps practices have had to adapt rapidly. The need to adjust is no more apparent than with Amazon Web Services. Currently, AWS lists no less than 18 categories of cloud offerings on their website, with each category containing several products and services. Categories include compute, storage, databases, networking, security, messaging, mobile, AI, IoT, and analytics.
In addition to products like compute, storage, and database, AWS now offers development, DevOps, and management tools, such as AWS OpsWorks and AWS CloudFormation. These products offer alternatives to traditional non-cloud CI/CD/RM workflows for deploying and managing complex application platforms on AWS. Learning the nuances of a growing list of AWS specific products and workflows, while simultaneously adapting your organization’s DevOps practices to them, has resulted in a whole new category of DevOps engineering specialization centered around AWS. Cloud-centric DevOps engineering specialization is also seen with other large cloud providers, such as Microsoft Azure and Google Cloud Platform.
Security
Call it DevSecOps, SecDevOps, SecOps, or Rugged DevOps, the intersection of DevOps and Security is bustling these days. As the complexity of modern application platforms grows, as well as the sophistication of threats from hackers and the requirements of government and industry compliance, security is no longer an afterthought or a process run in seeming isolation from software development and DevOps. In my recent experience, it is not uncommon to see IT security specialists actively participating on Agile development teams and embedded on DevOps and Platform teams.
Modern application platforms must be designed from day one to be bug-free, performant, compliant, and secure.
Security practices are now commonly part of the entire software development lifecycle, including enterprise architecture, software development, data governance, continuous testing, and infrastructure as code. Modern application platforms must be designed from day one to be bug-free, performant, compliant, and secure.
Take for example penetration (PEN) testing. Once a mostly manual process, done close to release time, evolving DevOps practices now allow testing for security vulnerabilities to applications and software-defined infrastructure to be done early and often in the software development lifecycle. Easily automatable and configurable cloud- and non-cloud-based tools like SonarQube, Veracode, Qualys, OWASP ZAP, and Chef Compliance, amongst others, are frequently incorporated into continuous integration workflows by development and DevOps teams. There is no longer an excuse for security vulnerabilities to be discovered just before release, or worse, in Production.
Modern Platforms
Along with the Cloud, modern application development trends, like the rise of the platform, microservices (or service-based architectures), containerization, NoSQL databases, and container orchestration, have likely provided the majority of fuel for the recent explosive growth of DevOps. Although innovative IT organizations have fostered these technologies for the past few years, their growth and relative maturity have risen sharply in the last 12 to 18 months.
No longer the stuff of Unicorns, platforms based on Evolutionary Architectures are being built and deployed by an increasing number of everyday organizations.
No longer the stuff of Unicorns, such as Amazon, Etsy, and Netflix, platforms based on Evolutionary Architectures are being built and deployed by an increasing number of everyday organizations. Although complexity continues to rise, the barrier to entry has been greatly reduced with technologies found across the SDLC, including Node, Spring Boot, Docker, Consul, Terraform, and Kubernetes, amongst others.
As modern platforms become more commonplace, the DevOps practices around them continue to mature and become specialized. Imagine, with potentially hundreds of moving parts, building, testing, deploying, and actively managing a large-scale microservice-based application on a container orchestration platform requires highly-specialized knowledge. The ability to ‘do DevOps at scale’ is critical.
Legacy Systems
Legacy systems as an emerging technology trend in DevOps? As the race to build the ‘next generation’ of application platforms accelerates to meet the demands of the business and their customers, there is a growing need to support ‘last generation’ systems. Many IT organizations support multiple legacy systems, ranging in age from as short as five years old to more than 25 years old. These monolithic legacy systems, which often contain a company’s secret sauce, such as complex business algorithms and decision engines, are built on out-moded technology stacks, often lack vendor support, and require separate processes to build, test, deploy, and manage. Worse, the knowledge to maintain these systems is frequently only known to a shrinking group of IT resources. Who wants to work on the old system with so many bright and shiny toys being built?
As a cost-effective means to maintain these legacy systems, organizations are turning to modern DevOps practices. Although not possible to the same degree, depending on the legacy technology, practices include the use source control, various types of automated testing, automated provisioning, deployment and configuration of system components, and infrastructure automation (DevOps for legacy systems – Infosys white paper).
Not specifically a DevOps practice, organizations are also implementing content collaboration systems, like Atlassian Confluence and Microsoft SharePoint, to document legacy system architectures and manual processes, before the resources and their knowledge is lost.
Part Two
In the second half of this post, we will continue to look at emerging technologies and their impact on DevOps, including:
- Big Data
- Internet of Things (IoT/IIoT)
- Artificial Intelligence (AI)
- Machine Learning
- COTS/SaaS
All opinions in this post are my own and not necessarily the views of my current employer or their clients.
Illustration Copyright: Andreus / 123RF Stock Photo
Provision and Deploy a Consul Cluster on AWS, using Terraform, Docker, and Jenkins
Posted by Gary A. Stafford in AWS, Build Automation, Cloud, Continuous Delivery, DevOps, Software Development on March 20, 2017
Introduction
Modern DevOps tools, such as HashiCorp’s Packer and Terraform, make it easier to provision and manage complex cloud architecture. Utilizing a CI/CD server, such as Jenkins, to securely automate the use of these DevOps tools, ensures quick and consistent results.
In a recent post, Distributed Service Configuration with Consul, Spring Cloud, and Docker, we built a Consul cluster using Docker swarm mode, to host distributed configurations for a Spring Boot application. The cluster was built locally with VirtualBox. This architecture is fine for development and testing, but not for use in Production.
In this post, we will deploy a highly available three-node Consul cluster to AWS. We will use Terraform to provision a set of EC2 instances and accompanying infrastructure. The instances will be built from a hybrid AMIs containing the new Docker Community Edition (CE). In a recent post, Baking AWS AMI with new Docker CE Using Packer, we provisioned an Ubuntu AMI with Docker CE, using Packer. We will deploy Docker containers to each EC2 host, containing an instance of Consul server.
All source code can be found on GitHub.
Jenkins
I have chosen Jenkins to automate all of the post’s build, provisioning, and deployment tasks. However, none of the code is written specific to Jenkins; you may run all of it from the command line.
For this post, I have built four projects in Jenkins, as follows:
- Provision Docker CE AMI: Builds Ubuntu AMI with Docker CE, using Packer
- Provision Consul Infra AWS: Provisions Consul infrastructure on AWS, using Terraform
- Deploy Consul Cluster AWS: Deploys Consul to AWS, using Docker
- Destroy Consul Infra AWS: Destroys Consul infrastructure on AWS, using Terraform
We will primarily be using the ‘Provision Consul Infra AWS’, ‘Deploy Consul Cluster AWS’, and ‘Destroy Consul Infra AWS’ Jenkins projects in this post. The fourth Jenkins project, ‘Provision Docker CE AMI’, automates the steps found in the recent post, Baking AWS AMI with new Docker CE Using Packer, to build the AMI used to provision the EC2 instances in this post.
Terraform
Using Terraform, we will provision EC2 instances in three different Availability Zones within the US East 1 (N. Virginia) Region. Using Terraform’s Amazon Web Services (AWS) provider, we will create the following AWS resources:
- (1) Virtual Private Cloud (VPC)
- (1) Internet Gateway
- (1) Key Pair
- (3) Elastic Cloud Compute (EC2) Instances
- (2) Security Groups
- (3) Subnets
- (1) Route
- (3) Route Tables
- (3) Route Table Associations
The final AWS architecture should resemble the following:
Production Ready AWS
Although we have provisioned a fairly complete VPC for this post, it is far from being ready for Production. I have created two security groups, limiting the ingress and egress to the cluster. However, to further productionize the environment would require additional security hardening. At a minimum, you should consider adding public/private subnets, NAT gateways, network access control list rules (network ACLs), and the use of HTTPS for secure communications.
In production, applications would communicate with Consul through local Consul clients. Consul clients would take part in the LAN gossip pool from different subnets, Availability Zones, Regions, or VPCs using VPC peering. Communications would be tightly controlled by IAM, VPC, subnet, IP address, and port.
Also, you would not have direct access to the Consul UI through a publicly exposed IP or DNS address. Access to the UI would be removed altogether or locked down to specific IP addresses, and accessed restricted to secure communication channels.
Consul
We will achieve high availability (HA) by clustering three Consul server nodes across the three Elastic Cloud Compute (EC2) instances. In this minimally sized, three-node cluster of Consul servers, we are protected from the loss of one Consul server node, one EC2 instance, or one Availability Zone(AZ). The cluster will still maintain a quorum of two nodes. An additional level of HA that Consul supports, multiple datacenters (multiple AWS Regions), is not demonstrated in this post.
Docker
Having Docker CE already installed on each EC2 instance allows us to execute remote Docker commands over SSH from Jenkins. These commands will deploy and configure a Consul server node, within a Docker container, on each EC2 instance. The containers are built from HashiCorp’s latest Consul Docker image pulled from Docker Hub.
Getting Started
Preliminary Steps
If you have built infrastructure on AWS with Terraform, these steps should be familiar to you:
- First, you will need an AMI with Docker. I suggest reading Baking AWS AMI with new Docker CE Using Packer.
- You will need an AWS IAM User with the proper access to create the required infrastructure. For this post, I created a separate Jenkins IAM User with PowerUser level access.
- You will need to have an RSA public-private key pair, which can be used to SSH into the EC2 instances and install Consul.
- Ensure you have your AWS credentials set. I usually source mine from a
.envfile, as environment variables. Jenkins can securely manage credentials, using secret text or files. - Fork and/or clone the Consul cluster project from GitHub.
- Change the
aws_key_nameandpublic_key_pathvariable values to your own RSA key, in thevariables.tffile - Change the
aws_amis_basevariable values to your own AMI ID (see step 1) - If you are do not want to use the US East 1 Region and its AZs, modify the
variables.tf,network.tf, andinstances.tffiles. - Disable Terraform’s remote state or modify the resource to match your remote state configuration, in the
main.tf file. I am using an Amazon S3 bucket to store my Terraform remote state.
Building an AMI with Docker
If you have not built an Amazon Machine Image (AMI) for use in this post already, you can do so using the scripts provided in the previous post’s GitHub repository. To automate the AMI build task, I built the ‘Provision Docker CE AMI’ Jenkins project. Identical to the other three Jenkins projects in this post, this project has three main tasks, which include: 1) SCM: clone the Packer AMI GitHub project, 2) Bindings: set up the AWS credentials, and 3) Build: run Packer.
The SCM and Bindings tasks are identical to the other projects (see below for details), except for the use of a different GitHub repository. The project’s Build step, which runs the packer_build_ami.sh script looks as follows:
The resulting AMI ID will need to be manually placed in Terraform’s variables.tf file, before provisioning the AWS infrastructure with Terraform. The new AMI ID will be displayed in Jenkin’s build output.
Provisioning with Terraform
Based on the modifications you made in the Preliminary Steps, execute the terraform validate command to confirm your changes. Then, run the terraform plan command to review the plan. Assuming are were no errors, finally, run the terraform apply command to provision the AWS infrastructure components.
In Jenkins, I have created the ‘Provision Consul Infra AWS’ project. This project has three tasks, which include: 1) SCM: clone the GitHub project, 2) Bindings: set up the AWS credentials, and 3) Build: run Terraform. Those tasks look as follows:
You will obviously need to use your modified GitHub project, incorporating the configuration changes detailed above, as the SCM source for Jenkins.
You will also need to configure your AWS credentials.
The provision_infra.sh script provisions the AWS infrastructure using Terraform. The script also updates Terraform’s remote state. Remember to update the remote state configuration in the script to match your personal settings.
The Jenkins build output should look similar to the following:
Although the build only takes about 90 seconds to complete, the EC2 instances could take a few extra minutes to complete their Status Checks and be completely ready. The final results in the AWS EC2 Management Console should look as follows:
Note each EC2 instance is running in a different US East 1 Availability Zone.
Installing Consul
Once the AWS infrastructure is running and the EC2 instances have completed their Status Checks successfully, we are ready to deploy Consul. In Jenkins, I have created the ‘Deploy Consul Cluster AWS’ project. This project has three tasks, which include: 1) SCM: clone the GitHub project, 2) Bindings: set up the AWS credentials, and 3) Build: run an SSH remote Docker command on each EC2 instance to deploy Consul. The SCM and Bindings tasks are identical to the project above. The project’s Build step looks as follows:
First, the delete_containers.sh script deletes any previous instances of Consul containers. This is helpful if you need to re-deploy Consul. Next, the deploy_consul.sh script executes a series of SSH remote Docker commands to install and configure Consul on each EC2 instance.
The entire Jenkins build process only takes about 30 seconds. Afterward, the output from a successful Jenkins build should show that all three Consul server instances are running, have formed a quorum, and have elected a Leader.
Persisting State
The Consul Docker image exposes VOLUME /consul/data, which is a path were Consul will place its persisted state. Using Terraform’s remote-exec provisioner, we create a directory on each EC2 instance, at /home/ubuntu/consul/config. The docker run command bind-mounts the container’s /consul/data path to the EC2 host’s /home/ubuntu/consul/config directory.
According to Consul, the Consul server container instance will ‘store the client information plus snapshots and data related to the consensus algorithm and other state, like Consul’s key/value store and catalog’ in the /consul/data directory. That container directory is now bind-mounted to the EC2 host, as demonstrated below.
Accessing Consul
Following a successful deployment, you should be able to use the public URL, displayed in the build output of the ‘Deploy Consul Cluster AWS’ project, to access the Consul UI. Clicking on the Nodes tab in the UI, you should see all three Consul server instances, one per EC2 instance, running and healthy.
Destroying Infrastructure
When you are finished with the post, you may want to remove the running infrastructure, so you don’t continue to get billed by Amazon. The ‘Destroy Consul Infra AWS’ project destroys all the AWS infrastructure, provisioned as part of this post, in about 60 seconds. The project’s SCM and Bindings tasks are identical to the both previous projects. The Build step calls the destroy_infra.sh script, which is included in the GitHub project. The script executes the terraform destroy -force command. It will delete all running infrastructure components associated with the post and update Terraform’s remote state.
Conclusion
This post has demonstrated how modern DevOps tooling, such as HashiCorp’s Packer and Terraform, make it easy to build, provision and manage complex cloud architecture. Using a CI/CD server, such as Jenkins, to securely automate the use of these tools, ensures quick and consistent results.
All opinions in this post are my own and not necessarily the views of my current employer or their clients.
Baking AWS AMI with new Docker CE Using Packer
Posted by Gary A. Stafford in AWS, Build Automation, Cloud, DevOps on March 6, 2017
Introduction
On March 2 (less than a week ago as of this post), Docker announced the release of Docker Enterprise Edition (EE), a new version of the Docker platform optimized for business-critical deployments. As part of the release, Docker also renamed the free Docker products to Docker Community Edition (CE). Both products are adopting a new time-based versioning scheme for both Docker EE and CE. The initial release of Docker CE and EE, the 17.03 release, is the first to use the new scheme.
Along with the release, Docker delivered excellent documentation on installing, configuring, and troubleshooting the new Docker EE and CE. In this post, I will demonstrate how to partially bake an existing Amazon Machine Image (Amazon AMI) with the new Docker CE, preparing it as a base for the creation of Amazon Elastic Compute Cloud (Amazon EC2) compute instances.
Adding Docker and similar tooling to an AMI is referred to as partially baking an AMI, often referred to as a hybrid AMI. According to AWS, ‘hybrid AMIs provide a subset of the software needed to produce a fully functional instance, falling in between the fully baked and JeOS (just enough operating system) options on the AMI design spectrum.’
Installing Docker CE on an AWS AMI should not be confused with Docker’s also recently announced Docker Community Edition (CE) for AWS. Docker for AWS offers multiple CloudFormation templates for Docker EE and CE. According to Docker, Docker for AWS ‘provides a Docker-native solution that avoids operational complexity and adding unneeded additional APIs to the Docker stack.’
Base AMI
Docker provides detailed directions for installing Docker CE and EE onto several major Linux distributions. For this post, we will choose a widely used Linux distro, Ubuntu. According to Docker, currently Docker CE and EE can be installed on three popular Ubuntu releases:
- Yakkety 16.10
- Xenial 16.04 (LTS)
- Trusty 14.04 (LTS)
To provision a small EC2 instance in Amazon’s US East (N. Virginia) Region, I will choose Ubuntu 16.04.2 LTS Xenial Xerus . According to Canonical’s Amazon EC2 AMI Locator website, a Xenial 16.04 LTS AMI is available, ami-09b3691f, for US East 1, as a t2.micro EC2 instance type.
Packer
HashiCorp Packer will be used to partially bake the base Ubuntu Xenial 16.04 AMI with Docker CE 17.03. HashiCorp describes Packer as ‘a tool for creating machine and container images for multiple platforms from a single source configuration.’ The JSON-format Packer file is as follows:
{
"variables": {
"aws_access_key": "{{env `AWS_ACCESS_KEY_ID`}}",
"aws_secret_key": "{{env `AWS_SECRET_ACCESS_KEY`}}",
"us_east_1_ami": "ami-09b3691f",
"name": "aws-docker-ce-base",
"us_east_1_name": "ubuntu-xenial-docker-ce-base",
"ssh_username": "ubuntu"
},
"builders": [
{
"name": "{{user `us_east_1_name`}}",
"type": "amazon-ebs",
"access_key": "{{user `aws_access_key`}}",
"secret_key": "{{user `aws_secret_key`}}",
"region": "us-east-1",
"vpc_id": "",
"subnet_id": "",
"source_ami": "{{user `us_east_1_ami`}}",
"instance_type": "t2.micro",
"ssh_username": "{{user `ssh_username`}}",
"ssh_timeout": "10m",
"ami_name": "{{user `us_east_1_name`}} {{timestamp}}",
"ami_description": "{{user `us_east_1_name`}} AMI",
"run_tags": {
"ami-create": "{{user `us_east_1_name`}}"
},
"tags": {
"ami": "{{user `us_east_1_name`}}"
},
"ssh_private_ip": false,
"associate_public_ip_address": true
}
],
"provisioners": [
{
"type": "file",
"source": "bootstrap_docker_ce.sh",
"destination": "/tmp/bootstrap_docker_ce.sh"
},
{
"type": "file",
"source": "cleanup.sh",
"destination": "/tmp/cleanup.sh"
},
{
"type": "shell",
"execute_command": "echo 'packer' | sudo -S sh -c '{{ .Vars }} {{ .Path }}'",
"inline": [
"whoami",
"cd /tmp",
"chmod +x bootstrap_docker_ce.sh",
"chmod +x cleanup.sh",
"ls -alh /tmp",
"./bootstrap_docker_ce.sh",
"sleep 10",
"./cleanup.sh"
]
}
]
}
The Packer file uses Packer’s amazon-ebs builder type. This builder is used to create Amazon AMIs backed by Amazon Elastic Block Store (EBS) volumes, for use in EC2.
Bootstrap Script
To install Docker CE on the AMI, the Packer file executes a bootstrap shell script. The bootstrap script and subsequent cleanup script are executed using Packer’s remote shell provisioner. The bootstrap is like the following:
#!/bin/sh
sudo apt-get remove docker docker-engine
sudo apt-get install \
apt-transport-https \
ca-certificates \
curl \
software-properties-common
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
sudo apt-key fingerprint 0EBFCD88
sudo add-apt-repository \
"deb [arch=amd64] https://download.docker.com/linux/ubuntu \
$(lsb_release -cs) \
stable"
sudo apt-get update
sudo apt-get -y upgrade
sudo apt-get install -y docker-ce
sudo groupadd docker
sudo usermod -aG docker ubuntu
sudo systemctl enable docker
This script closely follows directions provided by Docker, for installing Docker CE on Ubuntu. After removing any previous copies of Docker, the script installs Docker CE. To ensure sudo is not required to execute Docker commands on any EC2 instance provisioned from resulting AMI, the script adds the ubuntu user to the docker group.
The bootstrap script also uses systemd to start the Docker daemon. Starting with Ubuntu 15.04, Systemd System and Service Manager is used by default instead of the previous init system, Upstart. Systemd ensures Docker will start on boot.
Cleaning Up
It is best good practice to clean up your activities after baking an AMI. I have included a basic clean up script. The cleanup script is as follows:
#!/bin/sh set -e echo 'Cleaning up after bootstrapping...' sudo apt-get -y autoremove sudo apt-get -y clean sudo rm -rf /tmp/* cat /dev/null > ~/.bash_history history -c exit
Partially Baking
Before running Packer to build the Docker CE AMI, I set both my AWS access key and AWS secret access key. The Packer file expects the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables.
Running the packer build ubuntu_docker_ce_ami.json command builds the AMI. The abridged output should look similar to the following:
$ packer build docker_ami.json
ubuntu-xenial-docker-ce-base output will be in this color.
==> ubuntu-xenial-docker-ce-base: Prevalidating AMI Name...
ubuntu-xenial-docker-ce-base: Found Image ID: ami-09b3691f
==> ubuntu-xenial-docker-ce-base: Creating temporary keypair: packer_58bc7a49-9e66-7f76-ce8e-391a67d94987
==> ubuntu-xenial-docker-ce-base: Creating temporary security group for this instance...
==> ubuntu-xenial-docker-ce-base: Authorizing access to port 22 the temporary security group...
==> ubuntu-xenial-docker-ce-base: Launching a source AWS instance...
ubuntu-xenial-docker-ce-base: Instance ID: i-0ca883ecba0c28baf
==> ubuntu-xenial-docker-ce-base: Waiting for instance (i-0ca883ecba0c28baf) to become ready...
==> ubuntu-xenial-docker-ce-base: Adding tags to source instance
==> ubuntu-xenial-docker-ce-base: Waiting for SSH to become available...
==> ubuntu-xenial-docker-ce-base: Connected to SSH!
==> ubuntu-xenial-docker-ce-base: Uploading bootstrap_docker_ce.sh => /tmp/bootstrap_docker_ce.sh
==> ubuntu-xenial-docker-ce-base: Uploading cleanup.sh => /tmp/cleanup.sh
==> ubuntu-xenial-docker-ce-base: Provisioning with shell script: /var/folders/kf/637b0qns7xb0wh9p8c4q0r_40000gn/T/packer-shell189662158
...
ubuntu-xenial-docker-ce-base: Reading package lists...
ubuntu-xenial-docker-ce-base: Building dependency tree...
ubuntu-xenial-docker-ce-base: Reading state information...
ubuntu-xenial-docker-ce-base: E: Unable to locate package docker-engine
ubuntu-xenial-docker-ce-base: Reading package lists...
ubuntu-xenial-docker-ce-base: Building dependency tree...
ubuntu-xenial-docker-ce-base: Reading state information...
ubuntu-xenial-docker-ce-base: ca-certificates is already the newest version (20160104ubuntu1).
ubuntu-xenial-docker-ce-base: apt-transport-https is already the newest version (1.2.19).
ubuntu-xenial-docker-ce-base: curl is already the newest version (7.47.0-1ubuntu2.2).
ubuntu-xenial-docker-ce-base: software-properties-common is already the newest version (0.96.20.5).
ubuntu-xenial-docker-ce-base: 0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
ubuntu-xenial-docker-ce-base: OK
ubuntu-xenial-docker-ce-base: pub 4096R/0EBFCD88 2017-02-22
ubuntu-xenial-docker-ce-base: Key fingerprint = 9DC8 5822 9FC7 DD38 854A E2D8 8D81 803C 0EBF CD88
ubuntu-xenial-docker-ce-base: uid Docker Release (CE deb) <docker@docker.com>
ubuntu-xenial-docker-ce-base: sub 4096R/F273FCD8 2017-02-22
ubuntu-xenial-docker-ce-base:
ubuntu-xenial-docker-ce-base: Hit:1 http://us-east-1.ec2.archive.ubuntu.com/ubuntu xenial InRelease
ubuntu-xenial-docker-ce-base: Get:2 http://us-east-1.ec2.archive.ubuntu.com/ubuntu xenial-updates InRelease [102 kB]
...
ubuntu-xenial-docker-ce-base: Get:27 http://security.ubuntu.com/ubuntu xenial-security/universe amd64 Packages [89.5 kB]
ubuntu-xenial-docker-ce-base: Fetched 10.6 MB in 2s (4,065 kB/s)
ubuntu-xenial-docker-ce-base: Reading package lists...
ubuntu-xenial-docker-ce-base: Reading package lists...
ubuntu-xenial-docker-ce-base: Building dependency tree...
ubuntu-xenial-docker-ce-base: Reading state information...
ubuntu-xenial-docker-ce-base: Calculating upgrade...
ubuntu-xenial-docker-ce-base: 0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
ubuntu-xenial-docker-ce-base: Reading package lists...
ubuntu-xenial-docker-ce-base: Building dependency tree...
ubuntu-xenial-docker-ce-base: Reading state information...
ubuntu-xenial-docker-ce-base: The following additional packages will be installed:
ubuntu-xenial-docker-ce-base: aufs-tools cgroupfs-mount libltdl7
ubuntu-xenial-docker-ce-base: Suggested packages:
ubuntu-xenial-docker-ce-base: mountall
ubuntu-xenial-docker-ce-base: The following NEW packages will be installed:
ubuntu-xenial-docker-ce-base: aufs-tools cgroupfs-mount docker-ce libltdl7
ubuntu-xenial-docker-ce-base: 0 upgraded, 4 newly installed, 0 to remove and 0 not upgraded.
ubuntu-xenial-docker-ce-base: Need to get 19.4 MB of archives.
ubuntu-xenial-docker-ce-base: After this operation, 89.4 MB of additional disk space will be used.
ubuntu-xenial-docker-ce-base: Get:1 http://us-east-1.ec2.archive.ubuntu.com/ubuntu xenial/universe amd64 aufs-tools amd64 1:3.2+20130722-1.1ubuntu1 [92.9 kB]
...
ubuntu-xenial-docker-ce-base: Get:4 https://download.docker.com/linux/ubuntu xenial/stable amd64 docker-ce amd64 17.03.0~ce-0~ubuntu-xenial [19.3 MB]
ubuntu-xenial-docker-ce-base: debconf: unable to initialize frontend: Dialog
ubuntu-xenial-docker-ce-base: debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.)
ubuntu-xenial-docker-ce-base: debconf: falling back to frontend: Readline
ubuntu-xenial-docker-ce-base: debconf: unable to initialize frontend: Readline
ubuntu-xenial-docker-ce-base: debconf: (This frontend requires a controlling tty.)
ubuntu-xenial-docker-ce-base: debconf: falling back to frontend: Teletype
ubuntu-xenial-docker-ce-base: dpkg-preconfigure: unable to re-open stdin:
ubuntu-xenial-docker-ce-base: Fetched 19.4 MB in 1s (17.8 MB/s)
ubuntu-xenial-docker-ce-base: Selecting previously unselected package aufs-tools.
ubuntu-xenial-docker-ce-base: (Reading database ... 53844 files and directories currently installed.)
ubuntu-xenial-docker-ce-base: Preparing to unpack .../aufs-tools_1%3a3.2+20130722-1.1ubuntu1_amd64.deb ...
ubuntu-xenial-docker-ce-base: Unpacking aufs-tools (1:3.2+20130722-1.1ubuntu1) ...
...
ubuntu-xenial-docker-ce-base: Setting up docker-ce (17.03.0~ce-0~ubuntu-xenial) ...
ubuntu-xenial-docker-ce-base: Processing triggers for libc-bin (2.23-0ubuntu5) ...
ubuntu-xenial-docker-ce-base: Processing triggers for systemd (229-4ubuntu16) ...
ubuntu-xenial-docker-ce-base: Processing triggers for ureadahead (0.100.0-19) ...
ubuntu-xenial-docker-ce-base: groupadd: group 'docker' already exists
ubuntu-xenial-docker-ce-base: Synchronizing state of docker.service with SysV init with /lib/systemd/systemd-sysv-install...
ubuntu-xenial-docker-ce-base: Executing /lib/systemd/systemd-sysv-install enable docker
ubuntu-xenial-docker-ce-base: Cleanup...
ubuntu-xenial-docker-ce-base: Reading package lists...
ubuntu-xenial-docker-ce-base: Building dependency tree...
ubuntu-xenial-docker-ce-base: Reading state information...
ubuntu-xenial-docker-ce-base: 0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
==> ubuntu-xenial-docker-ce-base: Stopping the source instance...
==> ubuntu-xenial-docker-ce-base: Waiting for the instance to stop...
==> ubuntu-xenial-docker-ce-base: Creating the AMI: ubuntu-xenial-docker-ce-base 1288227081
ubuntu-xenial-docker-ce-base: AMI: ami-e9ca6eff
==> ubuntu-xenial-docker-ce-base: Waiting for AMI to become ready...
==> ubuntu-xenial-docker-ce-base: Modifying attributes on AMI (ami-e9ca6eff)...
ubuntu-xenial-docker-ce-base: Modifying: description
==> ubuntu-xenial-docker-ce-base: Modifying attributes on snapshot (snap-058a26c0250ee3217)...
==> ubuntu-xenial-docker-ce-base: Adding tags to AMI (ami-e9ca6eff)...
==> ubuntu-xenial-docker-ce-base: Tagging snapshot: snap-043a16c0154ee3217
==> ubuntu-xenial-docker-ce-base: Creating AMI tags
==> ubuntu-xenial-docker-ce-base: Creating snapshot tags
==> ubuntu-xenial-docker-ce-base: Terminating the source AWS instance...
==> ubuntu-xenial-docker-ce-base: Cleaning up any extra volumes...
==> ubuntu-xenial-docker-ce-base: No volumes to clean up, skipping
==> ubuntu-xenial-docker-ce-base: Deleting temporary security group...
==> ubuntu-xenial-docker-ce-base: Deleting temporary keypair...
Build 'ubuntu-xenial-docker-ce-base' finished.
==> Builds finished. The artifacts of successful builds are:
--> ubuntu-xenial-docker-ce-base: AMIs were created:
us-east-1: ami-e9ca6eff
Results
The result is an Ubuntu 16.04 AMI in US East 1 with Docker CE 17.03 installed. To confirm the new AMI is now available, I will use the AWS CLI to examine the resulting AMI:
aws ec2 describe-images \
--filters Name=tag-key,Values=ami Name=tag-value,Values=ubuntu-xenial-docker-ce-base \
--query 'Images[*].{ID:ImageId}'
Resulting output:
{
"Images": [
{
"VirtualizationType": "hvm",
"Name": "ubuntu-xenial-docker-ce-base 1488747081",
"Tags": [
{
"Value": "ubuntu-xenial-docker-ce-base",
"Key": "ami"
}
],
"Hypervisor": "xen",
"SriovNetSupport": "simple",
"ImageId": "ami-e9ca6eff",
"State": "available",
"BlockDeviceMappings": [
{
"DeviceName": "/dev/sda1",
"Ebs": {
"DeleteOnTermination": true,
"SnapshotId": "snap-048a16c0250ee3227",
"VolumeSize": 8,
"VolumeType": "gp2",
"Encrypted": false
}
},
{
"DeviceName": "/dev/sdb",
"VirtualName": "ephemeral0"
},
{
"DeviceName": "/dev/sdc",
"VirtualName": "ephemeral1"
}
],
"Architecture": "x86_64",
"ImageLocation": "931066906971/ubuntu-xenial-docker-ce-base 1488747081",
"RootDeviceType": "ebs",
"OwnerId": "931066906971",
"RootDeviceName": "/dev/sda1",
"CreationDate": "2017-03-05T20:53:41.000Z",
"Public": false,
"ImageType": "machine",
"Description": "ubuntu-xenial-docker-ce-base AMI"
}
]
}
Finally, here is the new AMI as seen in the AWS EC2 Management Console:
Terraform
To confirm Docker CE is installed and running, I can provision a new EC2 instance, using HashiCorp Terraform. This post is too short to detail all the Terraform code required to stand up a complete environment. I’ve included the complete code in the GitHub repo for this post. Not, the Terraform code is only used to testing. No security, including the use of a properly configured security groups, public/private subnets, and a NAT server, is configured.
Below is a greatly abridged version of the Terraform code I used to provision a new EC2 instance, using Terraform’s aws_instance resource. The resulting EC2 instance should have Docker CE available.
# test-docker-ce instance
resource "aws_instance" "test-docker-ce" {
connection {
user = "ubuntu"
private_key = "${file("~/.ssh/test-docker-ce")}"
timeout = "${connection_timeout}"
}
ami = "ami-e9ca6eff"
instance_type = "t2.nano"
availability_zone = "us-east-1a"
count = "1"
key_name = "${aws_key_pair.auth.id}"
vpc_security_group_ids = ["${aws_security_group.test-docker-ce.id}"]
subnet_id = "${aws_subnet.test-docker-ce.id}"
tags {
Owner = "Gary A. Stafford"
Terraform = true
Environment = "test-docker-ce"
Name = "tf-instance-test-docker-ce"
}
}
By using the AWS CLI, once again, we can confirm the new EC2 instance was built using the correct AMI:
aws ec2 describe-instances \ --filters Name='tag:Name,Values=tf-instance-test-docker-ce' \ --output text --query 'Reservations[*].Instances[*].ImageId'
Resulting output looks good:
ami-e9ca6eff
Finally, here is the new EC2 as seen in the AWS EC2 Management Console:
SSHing into the new EC2 instance, I should observe that the operating system is Ubuntu 16.04.2 LTS and that Docker version 17.03.0-ce is installed and running:
Welcome to Ubuntu 16.04.2 LTS (GNU/Linux 4.4.0-64-generic x86_64)
* Documentation: https://help.ubuntu.com
* Management: https://landscape.canonical.com
* Support: https://ubuntu.com/advantage
Get cloud support with Ubuntu Advantage Cloud Guest:
http://www.ubuntu.com/business/services/cloud
0 packages can be updated.
0 updates are security updates.
Last login: Sun Mar 5 22:06:01 2017 from <my_ip_address>
ubuntu@ip-<ec2_local_ip>:~$ docker --version
Docker version 17.03.0-ce, build 3a232c8
Conclusion
Docker EE and CE represent a significant step forward in expanding Docker’s enterprise-grade toolkit. Replacing or installing Docker EE or CE on your AWS AMIs is easy, using Docker’s guide along with HashiCorp Packer.
All source code for this post can be found on GitHub.
All opinions in this post are my own and not necessarily the views of my current employer or their clients.
-
You are currently browsing the archives for the Cloud category.
Gary Stafford
Accenture | Technology Architecture Delivery | Senior Manager | DevOps | Software Development | Docker | Microservices | JavaScript | Java | IaC | AWS
