In Part One of this post, we examined enterprise deployment tools and introduced two of Netflix’s open-source deployment tools, the Nebula Gradle plugins, and Spinnaker. In Part Two, we will deploy a production-ready Spring Boot application, the Election microservice, to multiple Amazon EC2 instances, behind an Elastic Load Balancer (ELB). We will use a fully automated DevOps workflow. The build, test, package, bake, deploy process will be handled by the Netflix Nebula Gradle Linux Packaging Plugin, Jenkins, and Spinnaker. The high-level process will involve the following steps:
- Configure Gradle to build a production-ready fully executable application for Unix systems (executable JAR)
- Using deb-s3 and GPG Suite, create a secure, signed APT (Debian) repository on Amazon S3
- Using Jenkins and the Netflix Nebula plugin, build a Debian package, containing the executable JAR and configuration files
- Using Jenkins and deb-s3, publish the package to the S3-based APT repository
- Using Spinnaker (HashiCorp Packer under the covers), bake an Ubuntu Amazon Machine Image (AMI), replete with the executable JAR installed from the Debian package
- Deploy an auto-scaling set of Amazon EC2 instances from the baked AMI, behind an ELB, running the Spring Boot application using both the Red/Black and Highlander deployment strategies
- Be able to repeat the entire automated build, test, package, bake, deploy process, triggered by a new code push to GitHub
The overall build, test, package, bake, deploy process will look as follows.
DevOps Architecture
Spinnaker’s modern architecture is comprised of several independent microservices. The codebase is written in Java and Groovy. It leverages the Spring Boot framework¹. Spinnaker’s configuration, startup, updates, and rollbacks are centrally managed by Halyard. Halyard provides a single point of contact for command line interaction with Spinnaker’s microservices.
Spinnaker can be installed on most private or public infrastructure, either containerized or virtualized. Spinnaker has links to a number of Quickstart installations on their website. For this demonstration, I deployed and configured Spinnaker on Azure, starting with one of the Azure Spinnaker quick-start ARM templates. The template provisions all the necessary Azure resources. For better performance, I chose upgraded the default VM to a larger Standard D4 v3, which contains 4 vCPUs and 16 GB of memory. I would recommend at least 2 vCPUs and 8 GB of memory at a minimum for Spinnaker.
Another Azure VM, in the same virtual network as the Spinnaker VM, already hosts Jenkins, SonarQube, and Nexus Repository OSS.
From Spinnaker on Azure, Debian Packages are uploaded to the APT package repository on AWS S3. Spinnaker also bakes Amazon Machine Images (AMI) on AWS. Spinnaker provisions the AWS resources, including EC2 instances, Load Balancers, Auto Scaling Groups, Launch Configurations, and Security Groups. The only resources you need on AWS to get started with Spinnaker are a VPC and Subnets. There are some minor, yet critical prerequisites for naming your VPC and Subnets.
Other external tools include GitHub for source control and Slack for notifications. I have built and managed everything from a Mac, however, all tools are platform agnostic. The Spring Boot application was developed in JetBrains IntelliJ.
Source Code
All source code for this post can be found on GitHub. The project’s README file contains a list of the Election service’s endpoints.
Code samples in this post are displayed as Gists, which may not display correctly on some mobile and social media browsers. Links to gists are also provided.
APT Repository
After setting up Spinnaker on Azure, I created an APT repository on Amazon S3, using the instructions provided by Netflix, in their Code Lab, An Introduction to Spinnaker: Hello Deployment. The setup involves creating an Amazon S3 bucket to serve as an APT (Debian) repository, creating a GPG key for signing, and using deb-s3 to manage the repository. The Code Lab also uses Aptly, a great tool, which I skipped for brevity.
GPG Key
On the Mac, I used GPG Suite to create a GPG (GNU Privacy Guard or GnuPG) automatic signing key for my APT repository. The key is required by Spinnaker to verify the Debian packages in the repository, before installation.
The Ruby Gem, deb-s3, makes management of the Debian packages easy and automatable with Jenkins. Jenkins uploads the Debian packages, using a deb-s3 command, such as the following (gist). In this post, Jenkins calls the command from the shell script, upload-deb-package.sh, which is included in the GitHub project.
| deb-s3 upload \ | |
| --bucket garystafford-spinnaker-repo \ | |
| --access-key-id=$AWS_ACCESS_KEY_ID \ | |
| --secret-access-key=$AWS_SECRET_ACCESS_KEY \ | |
| --arch=amd64 \ | |
| --codename=trusty \ | |
| --component=main \ | |
| --visibility=public \ | |
| --sign=$GPG_KEY_ID \ | |
| build/distributions/*.deb |
The Jenkins user requires access to the signing key, to build and upload the Debian packages. I created my GPG key on my Mac, securely copied the key to my Ubuntu-based Jenkins VM, and then imported the key for the Jenkins user. You could also create your key on Ubuntu, directly. Make sure you backup your private key in a secure location!
Nebula Packaging Plugin
Next, I set up a Gradle task in my build.gradle file to build my Debian packages using the Netflix Nebula Gradle Linux Packaging Plugin. Although Debian packaging tasks could become complex for larger application installations, this task for this post is pretty simple. I used many of the best-practices suggested by Spring for Production-grade deployments. The best-practices guide recommends file location, file modes, and file user and group ownership. I create the JAR as a fully executable JAR, meaning it is started like any other executable and does not have to be started with the standard java -jar command.
In the task, shown below (gist), the JAR and the external configuration file (optional) are copied to specific locations during the deployment and symlinked, as required. I used the older SysVInit system (init.d) to enable the application to automatically starts on boot. You should probably use systemctl for your services with Ubuntu 16.04.
| task packDeb(type: Deb) { | |
| description 'Creates .deb package.' | |
| into '/opt/' + project.name // root directory | |
| from(jar.outputs.files) { // copy *.jar | |
| into 'lib' | |
| fileMode 0500 | |
| user 'springapp' | |
| permissionGroup 'springapp' | |
| } | |
| from('build/resources/main/' + project.name + '.conf') { // copy .conf | |
| into 'conf' | |
| fileMode 0400 | |
| user 'root' | |
| permissionGroup 'root' | |
| } | |
| // symlinks jar to init.d | |
| link('/etc/init.d/election', | |
| '/opt/' + project.name + '/lib/' + jar.archiveName) | |
| // link init.d to rc2.d | |
| link('/opt/' + project.name + '/lib/' + project.name + '-' + project.version + '.conf', | |
| '/opt/' + project.name + '/conf/' + project.name + '.conf') | |
| // link conf to jar location | |
| link('/etc/rc2.d/S02election', '/etc/init.d/election') | |
| postInstall 'chattr +i ' + '/opt/' + project.name + '/lib/' + jar.archiveName | |
| } |
You can use the ar (archive) command (i.e., ar -x spring-postgresql-demo_4.5.0_all.deb), to extract and inspect the structure of a Debian package. The data.tar.gz file, displayed below in Atom, shows the final package structure.
Base AMI
Next, I baked a base AMI for Spinnaker to use. This base AMI is used by Spinnaker to bake (re-bake) the final AMI(s) used for provisioning the EC2 instances, containing the Spring Boot Application. The Spinnaker base AMI is built from another base AMI, the official Ubuntu 16.04 LTS image. I installed the OpenJDK 8 package on the AMI, which is required to run the Java-based Election service. Lastly and critically, I added information about the location of my S3-based APT Debian package repository to the list of configured APT data sources, and the GPG key required for package verification. This information and key will be used later by Spinnaker to bake AMIʼs, using this base AMI. The set-up script, base_ubuntu_ami_setup.sh, which is included in the GitHub project.
| #!/usr/bin/env sh | |
| # based on ami-6dfe5010 | |
| # Canonical, Ubuntu, 16.04 LTS, amd64 xenial image build on 2018-04-05 | |
| # References: | |
| # https://docs.spring.io/spring-boot/docs/current/reference/html/deployment-install.html | |
| # https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/building-shared-amis.html | |
| # https://gist.github.com/justindowning/5921369 | |
| set +x | |
| # update and install java | |
| sudo apt-get update -y \ | |
| && sudo apt-get upgrade -y \ | |
| && sudo apt-get install \ | |
| openjdk-8-jre-headless -y | |
| # a few optional ops tools | |
| sudo apt-get install \ | |
| tree htop glances -y | |
| # user:group application will run as | |
| sudo useradd springapp | |
| sudo usermod -a -G springapp springapp | |
| sudo usermod -s /usr/sbin/nologin springapp | |
| # add s3 deb repo | |
| echo "deb http://garystafford-spinnaker-repo.s3-website-us-east-1.amazonaws.com trusty main" | \ | |
| sudo tee -a /etc/apt/sources.list.d/gary-stafford.list | |
| curl -s https://s3.amazonaws.com/garystafford-spinnaker-repo/apt/doc/apt-key.gpg | \ | |
| sudo apt-key add - | |
| # clean up and secure | |
| sudo passwd -l root | |
| sudo shred -u /etc/ssh/*_key /etc/ssh/*_key.pub | |
| [ -f /home/ubuntu/.ssh/authorized_keys ] && rm /home/ubuntu/.ssh/authorized_keys | |
| sudo rm -rf /tmp/* | |
| cat /dev/null > ~/.bash_history | |
| shred -u ~/.*history | |
| history -c | |
| exit |
Jenkins
This post uses a single Jenkins CI/CD pipeline. Using a Webhook, the pipeline is automatically triggered by every git push to the GitHub project. The pipeline pulls the source code, builds the application, and performs unit-tests and static code analysis with SonarQube. If the build succeeds and the tests pass, the build artifact (JAR file) is bundled into a Debian package using the Nebula Packaging plugin, uploaded to the S3 APT repository using s3-deb, and archived locally for Spinnaker to reference. Once the pipeline is completed, on success or on failure, a Slack notification is sent. The Jenkinsfile, used for this post is available in the project on Github.
| #!/usr/bin/env groovy | |
| def ACCOUNT = "garystafford" | |
| def PROJECT_NAME = "spring-postgresql-demo" | |
| pipeline { | |
| agent any | |
| tools { | |
| gradle 'gradle' | |
| } | |
| stages { | |
| stage('Checkout GitHub') { | |
| steps { | |
| git changelog: true, poll: false, | |
| branch: 'master', | |
| url: "https://github.com/${ACCOUNT}/${PROJECT_NAME}" | |
| } | |
| } | |
| stage('Build') { | |
| steps { | |
| sh 'gradle wrapper' | |
| sh 'LOGGING_LEVEL_ROOT=INFO ./gradlew clean build -x test --info' | |
| } | |
| } | |
| stage('Unit Test') { // unit test against in-memory h2 | |
| steps { | |
| withEnv(['SPRING_DATASOURCE_URL=jdbc:h2:mem:elections']) { | |
| sh 'LOGGING_LEVEL_ROOT=INFO ./gradlew cleanTest test --info' | |
| } | |
| junit '**/build/test-results/test/*.xml' | |
| } | |
| } | |
| stage('SonarQube Analysis') { | |
| steps { | |
| withSonarQubeEnv('sonarqube') { | |
| sh "LOGGING_LEVEL_ROOT=INFO ./gradlew sonarqube -Dsonar.projectName=${PROJECT_NAME} --info" | |
| } | |
| } | |
| } | |
| stage('Build Debian Package') { | |
| steps { | |
| sh "LOGGING_LEVEL_ROOT=INFO ./gradlew packDeb --info" | |
| } | |
| } | |
| stage('Upload Debian Package') { | |
| steps { | |
| withCredentials([ | |
| string(credentialsId: 'GPG_KEY_ID', variable: 'GPG_KEY_ID'), | |
| string(credentialsId: 'AWS_ACCESS_KEY_ID', variable: 'AWS_ACCESS_KEY_ID'), | |
| string(credentialsId: 'AWS_SECRET_ACCESS_KEY', variable: 'AWS_SECRET_ACCESS_KEY')]) { | |
| sh "sh ./scripts/upload-deb-package.sh ${GPG_KEY_ID}" | |
| } | |
| } | |
| } | |
| stage('Archive Debian Package') { | |
| steps { | |
| archiveArtifacts 'build/distributions/*.deb' | |
| } | |
| } | |
| } | |
| post { | |
| success { | |
| slackSend(color: '#79B12B', | |
| message: "SUCCESS: Job '${env.JOB_NAME} [${env.BUILD_NUMBER}]' (${env.BUILD_URL})") | |
| } | |
| failure { | |
| slackSend(color: '#FF0000', | |
| message: "FAILURE: Job '${env.JOB_NAME} [${env.BUILD_NUMBER}]' (${env.BUILD_URL})") | |
| } | |
| } | |
| } |
Below is a traditional Jenkins view of the CI/CD pipeline, with links to unit test reports, SonarQube results, build artifacts, and GitHub source code.
Below is the same pipeline viewed using the Jenkins Blue Ocean plugin.
It is important to perform sufficient testing before building the Debian package. You donʼt want to bake an AMI and deploy EC2 instances, at a cost, before finding out the application has bugs.
Spinnaker Setup
First, I set up a new Spinnaker Slack channel and a custom bot user. Spinnaker details the Slack set up in their Notifications and Events Guide. You can configure what type of Spinnaker events trigger Slack notifications.
AWS Spinnaker User
Next, I added the required Spinnaker User, Policy, and Roles to AWS. Spinnaker uses this access to query and provision infrastructure on your behalf. The Spinnaker User requires Power User level access to perform all their necessary tasks. AWS IAM set up is detailed by Spinnaker in their Cloud Providers Setup for AWS. They also describe the setup of other cloud providers. You need to be reasonably familiar with AWS IAM, including the PassRole permission to set up this part. As part of the setup, you enable AWS for Spinnaker and add your AWS account using the Halyard interface.
Spinnaker Security Groups
Next, I set up two Spinnaker Security Groups, corresponding to two AWS Security Groups, one for the load balancer and one for the Election service. The load balancer security group exposes port 80, and the Election service security group exposes port 8080.
Spinnaker Load Balancer
Next, I created a Spinnaker Load Balancer, corresponding to an Amazon Classic Load Balancer. The Load Balancer will load-balance the Election service EC2 instances. Below you see a Load Balancer, balancing a pair of active EC2 instances, the result of a Red/Black deployment.
Spinnaker can currently create both AWS Classic Load Balancers as well as Application Load Balancers (ALB).
Spinnaker Pipeline
This post uses a single, basic Spinnaker Pipeline. The pipeline bakes a new AMI from the Debian package generated by the Jenkins pipeline. After a manual approval stage, Spinnaker deploys a set of EC2 instances, behind the Load Balancer, which contains the latest version of the Election service. Spinnaker finishes the pipeline by sending a Slack notification.
Jenkins Integration
The pipeline is triggered by the successful completion of the Jenkins pipeline. This is set in the Configuration stage of the pipeline. The integration with Jenkins is managed through Spinnaker’s Igor service.
Bake Stage
Next, in the Bake stage, Spinnaker bakes a new AMI, containing the Debian package generated by the Jenkins pipeline. The stageʼs configuration contains the package name to reference.
The stageʼs configuration also includes a reference to which Base AMI to use, to bake the new AMIs. Here I have used the AMI ID of the base Spinnaker AMI, I created previously.
Deploy Stage
Next, the Deploy stage deploys the Election service, running on EC2 instances, provisioned from the new AMI, which was baked in the last stage. To configure the Deploy stage, you define a Spinnaker Server Group. According to Spinnaker, the Server Group identifies the deployable artifact, VM image type, the number of instances, autoscaling policies, metadata, Load Balancer, and a Security Group.
The Server Group also defines the Deployment Strategy. Below, I chose the Red/Black Deployment Strategy (also referred to as Blue/Green). This strategy will disable, not terminate the active Server Group. If the new deployment fails, we can manually or automatically perform a Rollback to the previous, currently disabled Server Group.
Letʼs Start Baking!
With set up complete, letʼs kick off a git push, trigger and complete the Jenkins pipeline, and finally trigger the Spinnaker pipeline. Below we see the pipelineʼs Bake stage has been started. Spinnakerʼs UI lets us view the Bakery Details. The Bakery, provided by Spinnakerʼs Rosco service, bakes the AMIs. Rosco uses HashiCorp Packer to bake the AMIs, using standard Packer templates.
Below we see Spinnaker (Rosco/Packer) locating the Base Spinnaker AMI we configured in the Pipelineʼs Bake stage. Next, we see Spinnaker sshʼing into a new EC2 instance with a temporary keypair and Security Group and starting the Election service Debian package installation.
Continuing, we see the latest Debian package, derived from the Jenkins pipelineʼs archive, being pulled from the S3-based APT repo. The package is verified using the GPG key and then installed. Lastly, we see a new AMI is created, containing the deployed Election service, which was initially built and packaged by Jenkins. Note the AWS Resource Tags created by Spinnaker, as shown in the Bakery output.
The base Spinnaker AMI and the AMIs baked by Spinnaker are visible in the AWS Console. Note the naming conventions used by Spinnaker for the AMIs, the Source AMI used to build the new APIs, and the addition of the Tags, which we saw being applied in the Bakery output above. The use of Tags indirectly allows full traceability from the deployed EC2 instance all the way back to the original code commit to git by the Developer.
Red/Black Deployments
With the new AMI baked successfully, and a required manual approval, using a Manual Judgement type pipeline stage, we can now begin a Red/Black deployment to AWS.
Using the Server Group configuration in the Deploy stage, Spinnaker deploys two EC2 instances, behind the ELB.
Below, we see the successful results of the Red/Black deployment. The single Spinnaker Cluster contains two deployed Server Groups. One group, the previously active Server Group (RED), comprised of two EC2 instances, is disabled. The ‘RED’ EC2 instances are unregistered with the load balancer but still running. The new Server Group (BLACK), also comprised of two EC2 instances, is now active and registered with the Load Balancer. Spinnaker will spread EC2 instances evenly across all Availability Zones in the US East (N. Virginia) Region.
From the AWS Console, we can observe four running instances, though only two are registered with the load-balancer.
Here we see each deployed Server Group has a different Auto Scaling Group and Launch Configuration. Note the continued use of naming conventions by Spinnaker.
There can be only one, Highlander!
Now, in the Deploy stage of the pipeline, we will switch the Server Groupʼs Strategy to Highlander. The Highlander strategy will, as you probably guessed by the name, destroy all other Server Groups in the Cluster. This is more typically used for lower environments, like Development or Test, where you are only interested in the next version of the application for testing. The Red/Black strategy is more applicable to Production, where you want the opportunity to quickly rollback to the previous deployment, if necessary.
Following a successful deployment, below, we now see the first two Server Groups have been terminated, and a third Server Group in the Cluster is active.
In the AWS Console, we can confirm the four previous EC2 instances have been successfully terminated as a result of the Highlander deployment strategy, and two new instances are running.
As well, the previous Auto Scaling Groups and Launch Configurations have been deleted from AWS by Spinnaker.
As expected, the Classic Load Balancer only contains the two most recent EC2 instances from the last Server Group deployed.
Confirming the Deployment
Using the DNS address of the load balancer, we can hit the Election service endpoints, on either of the EC2 instances. All API endpoints are listed in the Projectʼs README file. Below, from a web browser, we see the candidates resource returning candidate information, retrieved from the Electionʼs PostgreSQL RDS database Test instance.
Similarly, from Postman, we can hit the load balancer and get back election information from the elections resource, using an HTTP GET.
I intentionally left out a discussion of the service’s RDS database and how configuration management was handled with Spring Profiles and Spring Cloud Config. Both topics were out of scope for this post.
Conclusion
Although this was a brief, whirlwind overview of deployment tools, it shows the power of delivery tools like Spinnaker, when seamlessly combined with other tools, like Jenkins and the Nebula plugins. Together, these tools are capable of efficiently, repeatably, and securely deploying large numbers of containerized and non-containerized applications to a variety of private, public, and hybrid cloud infrastructure.
All opinions expressed in this post are my own and not necessarily the views of my current or past employers or their clients.
Programmatic Ponderings 