Posts Tagged Continuous Integration
Preparing for Your Organization’s DevOps Journey
Posted by Gary A. Stafford in DevOps, Software Development, Technology Consulting on May 10, 2017
Introduction
Recently, I was asked two questions regarding DevOps. The first, ‘How do you get started implementing DevOps in an organization?’ A question I get asked, and answer, fairly frequently. The second was a bit more challenging to answer, ‘How do you prepare your organization to implement DevOps?’
Getting Started
The first question, ‘How do you get started implementing DevOps in an organization?’, is a popular question many companies ask. The answer varies depending on who you ask, but the process is fairly well practiced and documented by a number of well-known and respected industry pundits. A successful DevOps implementation is a combination of strategic planning and effective execution.
A successful DevOps implementation is a combination of strategic planning and effective execution.
Most commonly, an organization starts with some form of a DevOps maturity assessment. The concept of a DevOps maturity model was introduced by Jez Humble and David Farley, in their ground-breaking book, Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation (Addison-Wesley Signature Series), circa 2011.
Humble and Farley presented their ‘Maturity Model for Configuration and Release Management’ (page 419). This model, which encompassed much more than just CM and RM, was created as a means of evaluating and improving an organization’s DevOps practices.
Although there are several variations, maturity models ordinarily all provide some means of ranking the relative maturity of an organization’s DevOps practices. Less sophisticated models focus primarily on tooling and processes. More holistic models, such as Accenture’s DevOps Maturity Assessment, focus on tooling, processes, people, and culture.
Following the analysis, most industry experts recommend a strategic plan, followed an implementation plan. The plans set milestones for reaching higher levels of maturity, according to the model. Experts will identify key performance indicators, such as release frequency, defect rates, production downtime, and mean time to recovery from failures, which are often used to measure DevOps success.
Preparing for the Journey
As I said, the second question, ‘How do you prepare your organization to implement DevOps?’, is a bit more challenging to answer. And, as any good consultant would respond, it depends.
The exact answer depends on many factors. How engaged is management in wanting to transform their organization? How mature is the organization’s current IT practices? Are the other parts of the organization, such as sales, marketing, training, product documentation, and customer support, aligned with IT? Is IT aligned with them?
Even the basics matter, such as the organization’s size, both physical and financial, as well as the age of the organization? The industry? Are they in a highly regulated industry? Are they a global organization with distributed IT resources? Have they tried DevOps before and failed? Why did they fail?
As overwhelming as those questions might seem, I managed to break down my answer to the question, “How do you prepare your organization to implement DevOps?”, into five key areas. In my experience, each of these is critical for any DevOps transformation to succeed. Before the journey starts, these are five areas an organization needs to consider:
- Have an Agile Mindset
- Breakdown Silos
- Know Your Business
- Take the Long View
- Be Introspective
Have an Agile Mindset
It is commonly accepted that DevOps was born from the need of Agile software development to increase the frequency of releases. More releases required faster feedback loops, better quality control methods, and the increased use of automation, amongst other necessities. DevOps practices evolved to meet those challenges.
If an organization is considering DevOps, it should have already successfully embraced Agile, or be well along in their Agile transformation. An outgrowth of Agile software development, DevOps follow many Agile practices. Such Agile practices as cross-team collaboration, continuous and rapid feedback loops, continuous improvement, test-driven development, continuous integration, scheduling work in sprints, and breaking down business requirements into epics, stories, and tasks, are usually all part of a successful DevOps implementation.
If your organization cannot adopt Agile, it will likely fail to successfully embrace DevOps. Imagine a typical scenario in which DevOps enables an organization to release more frequently — monthly instead of quarterly, weekly instead of monthly. However, if the rest of the organization — sales, marketing, training, product documentation, and customer support, is still working in a non-Agile manner, they will not be able to match the improved cycle time DevOps would provide.
Breakdown Silos
Closely associated with an Agile mindset, is breaking down departmental silos. If your organization has already made an Agile transformation, then one should assume those ‘silos’, the physical or more often process-induced ‘walls’ between departments, have been torn down. Having embraced Agile, we assume that Development and Testing are working side-by-side as part of an Agile software development team.
Implementing DevOps requires closing the often wide gap between Development and Operations. If your organization cannot tear down the typically shorter wall between Development and Testing, then tearing down the larger walls between Development and Operations will be impossible.
Know Your Business
Before starting your DevOps journey, an organization needs to know thyself. Most organizations establish business metrics, such as sales quotas, profit targets, employee retention objectives, and client acquisition goals. However, many organizations have not formalized their IT-related Key Performance Indicators (KPIs) or Service Level Agreements (SLAs).
DevOps is all about measurements — application response time, incident volume, severity, and impact, defect density, Mean Time To Recovery (MTTR), downtime, uptime, and so forth. Established meaningful and measurable metrics is one of the best ways to evaluate the continuous improvements achieved by a maturing DevOps practice.
To successfully implement DevOps, an organization should first identify its business-critical performance metrics and service level expectations. Additionally, an organization must accurately and honestly measure itself against those metrics, before beginning the DevOps journey.
Take the Long View
Rome was not built in a day, organizations don’t transform overnight, and DevOps is a journey, not a time-boxed task in a team’s backlog. Before an organization sets out on their journey, they must be willing to take the long view on DevOps. There is a reason DevOps maturity models exist. Like most engineering practices, cultural and organizational transformation, and skill-building exercise, DevOps takes the time to become successfully entrenched in a company.
Rome was not built in a day, organizations don’t transform overnight, and DevOps is a journey, not a time-boxed task in a team’s backlog.
Organizations need to value quick, small wins, followed by more small wins. They should not expect a big bang with DevOps. Achieving high levels DevOps performance is similar to the Agile practice of delivering small pieces of valuable functionality, in an incremental fashion.
Getting the ‘Hello World’ application successfully through a simple continuous integration pipeline might seem small, but think of all the barriers that were overcome to achieve that task — source control, continuous integration server, unit testing, artifact repository, and so on. Your next win, deploy that ‘Hello World’ application to your Test environment, automatically, through a continuous deployment pipeline…
This practice reminds me of an adage. Would you prefer a dollar, every day for the next week, or seven dollars at the end of the week? Most people prefer the immediacy of a dollar each day (small wins), as well as the satisfaction of seeing the value build consistently, day after day. Exercise the same philosophy with DevOps.
Be Introspective
As stated earlier, generally, the first step in creating a strategic plan for implementing DevOps is analyzing your organization’s current level of IT maturity. Individual departments must be willing to be open, honest, and objective when assessing their current state.
The inability of organizations to be transparent about their practices, challenges, and performance, is a sign of an unhealthy corporate culture. Not only is an accurate perspective critical for a maturity analysis and strategic planning, but the existence of an unhealthy culture can also be fatal to most DevOps transformation. DevOps only thrives in an open, collaborative, and supportive culture.
Conclusion
As Alexander Graham Bell once famously said, ‘before anything else, preparation is the key to success.’ Although not a guarantee, properly preparing for a DevOps transformation by addressing these five key areas, should greatly improve an organization’s chances of success.
All opinions in this post are my own and not necessarily the views of my current employer or their clients.
Copyright: peshkova / 123RF Stock Photo
Software Delivery: Evaluating Risk within the Enterprise
Posted by Gary A. Stafford in Build Automation, DevOps, Enterprise Software Development, Software Development on November 9, 2014
Introduction
Many vendor whitepapers, industry publications, blog posts, podcasts, and e-books, extol the best practices in software development and delivery. Best practices include industry-standard concepts, such as Agile, DevOps, TTD, continuous integration, and continuous delivery. Generally, these best practices all strive to improve the process of delivering software enhancements and bug fixes to customers.
Rapidly, reliably and repeatedly push out enhancements and bug fixes to customers at low risk and with minimal manual overhead. – Wikipedia
Most learning resources present one of two idealized environments, ‘applications as islands’ and ‘utopian enterprise’. I am also often guilty of tailoring my own materials to one of these two idealized environments. Neither ‘applications as islands’ or ‘utopian enterprise’, best model the typical enterprise software environments in which many of us work.
Applications as Islands
The ‘applications as islands’ environment is one of completely isolated application stacks. These types of environments have multiple application stacks, consisting of web, mobile, and desktop components, services, data sources, utilities and scripts, messaging and reporting components, and so forth. Unrealistically, each application stack is completely isolated from the other application stacks within the same environment.
The Utopian Enterprise
The ‘utopian enterprise’ environments have multiple application stacks with multiple shared components. However, they are built, unrealistically, using consistent and modern architectural patterns and compatible technology stacks. They are designed from the ground up to be compartmentalized, scalable, and highly risk-tolerant to changes. They often avoid the challenges of monolithic legacy applications. The closest things in the real world are probably industry trendsetters, such as Facebook, Etsy, Amazon, and Twitter. We all probably wish we could evolve our own software environments into one of these Utopias.
Complexity and Risk
As an organization continues to evolve their software, they naturally increase the overall complexity, and thereby the challenge of effectively delivering reliable and performant software. In this post, I will explore the challenges of software delivery, as a software environment grows in complexity. Specifically, I will focus on how to evaluate the level of risk based on software changes made to various components within the software environment.
Sensitivity and Impact
As we examine the level of risk introduced by software changes within the environment, two aspects of risk are inescapable, sensitivity and impact. Sensitivity will be defined as the potential degree of which one component, such as an application, service, or data source, is affected by changes to other components within the same software environment. How sensitive is ‘Application A’ to changes made to other components within the same software environment, on which ‘Application A’ is directly or indirectly dependent?
The impact will be defined as the potential effect a component’s changes have on other components within the software environment. Teams tend to only evaluate the impact of changes to the immediate component or application stack. They do not sufficiently consider how those changes impact those components that are directly and indirectly dependent on them. What level of impact do changes to ‘Service B’ have on all other components within the software environment that are directly and indirectly dependent on ‘Service B’?
Notice I use the word potential. Any change has the potential to introduce risk. The level of risk varies, based on the type and volume of changes. A few simple changes should have a low potential for impact, as opposed to a high number of changes, or more complex changes. For example, changing an internal error message logged by a particular service operation should present a very low risk. This, as opposed to rewriting that operation’s complex algorithm for calculating a customer’s creditworthiness. The potential impact of those two types’ changes to dependent components varies significantly.
Measuring Risk
For both sensitivity to change and impact of change, I will use a color-coded scale to subjectively assign a level of potential risk to each component within a given software environment. The scale ranges from ‘Low’, to ‘Moderate’, to ‘High’, to ‘Very High’. Using the scale, it is possible to ‘heat map’ a software environment, based on the level of risk from changes.
Independent Aspects of Risk
Sensitivity and impact are two independent aspects of risk. Changes to one component may have a ‘Low’ potential impact on all other components within the environment. While at the same time, that same component may have a ‘High’ sensitivity to changes made to other components within the environment. Alternatively, a component may have a ‘Very High’ risk for potential impact on multiple components within the environment. At the same time, that same component may have a ‘Low’ potential sensitivity to changes made to other components. Sensitivity and risk do not parallel each other.
Growing Complexity
Let’s look at how sensitivity and impact change as we increase the software environment’s complexity. In the first example, we will look at one of the two environments I described earlier, isolated applications. Applications may have their own web and mobile components, SOAP or RESTful services, data sources, utilities, scheduled tasks, and so forth. However, the applications do not depend on each other or components outside their own immediate application stack; the applications are self-contained.
When making changes in this type of environment, the real potential impact is to the overall stability, security, and performance of the individual applications, themselves. As long as they are in isolation, the applications will have no impact on each other. Therefore, applications potential sensitivity to changes and their impact on other applications is ‘Low’.
Shared Components
A slightly more complex example is a software environment in which one or more applications have a dependency on a component outside their immediate application stack. For example, a healthcare provider develops a Windows-based application to track their employee’s work schedules (Application A). In addition, they develop a web application to track patient appointments (Application B). Lastly, they offer a client-facing mobile application for patients to track personal fitness and nutrition goals (Application C). Applications B and C share a common set of services and a database for managing patient data.
Software changes made to Applications A, B, and C, should have no effect on other components within the software environment. However, Applications B and C are potentially impacted by changes made to either the Services Layer or Data Layer. The Services Layer has ‘High’ potential impact to the software environment. Lastly, the Data Layer should not be directly impacted by changes made to the Services Layer or Applications. However, the Data Layer has the potential to directly affect the Services Layer, and indirectly affect Applications B and C. Therefore, the Data Layer’s potential impact on other dependent components within the environment is ‘Very High’.
Multiple Shared Components
An even more complex example is a software environment in which multiple applications have one or more dependencies on multiple components outside their immediate application stack (many-to-many).
Take, for example, a small financial institution. They have a ‘legacy’ COBOL-based application for managing their commercial mortgage business (Application A). They also have an older J2EE-based application, they acquired through a business merger, for managing their commercial banking relationships (Application B). Next, they have a relatively new Java EE-based investment banking application to manage their retail customers (Application C). Lastly, they have web-based, client-facing application for secure, online retail banking.
Since both Application A and B serve commercial clients, it is necessary to send financial data between the two application stacks. Since both applications are built on different, older technologies, the development team built a Custom Messaging Middleware component to connect the two applications. The Custom Messaging Middleware component receives, transforms, and delivers messages between the two applications.
Changes made to Applications C and D should have no impact on other components within the software environment. However, changes made to either Application A or B has the potential to indirectly affect the ability to successfully communicate with the other application, via the Custom Messaging Middleware. Changes to the Custom Messaging Middleware have the potential to affect both Applications A and B. The Custom Messaging Middleware has a ‘Moderate’ potential sensitivity to risk, versus ‘Low’, because one could argue that changes to either Application A or Application B’s messaging format could impact the Custom Messaging Middleware’s ability to properly process that application’s messages and successfully deliver them to opposite application.
Applications B, C, and D have a direct dependency on the Services Layer, and indirectly on the Data Layer. Therefore, the potential impact of changes to the Services Layer on other components is arguably higher than in the last example. The Services Layer’s potential impact on other components is ‘Very High’.
Since Application B has a direct dependency on both the Messaging Middleware and the Services Layer, it has a higher sensitivity to changes then the other three applications. Application B’s potential sensitivity to changes by other components is ‘Very High’.
Changes made to the Services Layer or the Applications will not affect the Data Layer. However, the Data Layer has the potential to directly affect the Services Layer, and indirectly affect Applications B, C, and D. Therefore, the Data Layer’s potential impact on the software environment is ‘Very High’.
Small Enterprise
The last example of increasing complexity is an environment in which even more applications are dependent on even more components. Additionally, there may be different types of components, such as a common UI and third-party APIs, which only increase the complexity of the dependencies. Although this example is nowhere near as complex as many enterprise software environments, it does begin to reflect their intricate, inner-dependent structure.
Let’s use an example of a large web-based retailer. The retailer has a standalone ERM application for managing their wholesale purchasing and product distribution (Application A). Next, they have their primary client-facing storefront (Application B). They also have a separate application to handle customer accounts (Application C). Lastly, they have an application that manages their online media retail business and media storage (Application D).
In addition to the Common Services Layer, Common Data Layer, and Custom Messaging Middleware, as seen in earlier examples, the retailer has two other components in their environment, a Common Web User Interface (UI) and a Web API. The Web UI provides the customer with a seamless branded experience, no matter which application they use – Application B, C, or D. The customer enters the Common Web UI and has all three application’s features seamlessly available to them.
The retailer also exposes a RESTful Web API for its marketing affiliates. Third parties can develop a variety of applications that drive sales to the retailer, in return for a sales commission.
In the earlier examples, individual applications had separate points of entry. However, in this example, the Common Web UI provides a single point of entry for users of Applications B, C, and D. Having a single point of entry also introduces a single point of failure for all three applications. Thus, the potential risk to the retailer and their customers is much greater. The Common Web UI’s potential impact on other components is ‘Very High’.
A single point of entry also introduces a single point of failure.
The potential sensitivity of the Common Web UI to changes comes from its direct dependency on the Services Layer, and indirectly on the Data Layer. Additionally, one could argue, since the Common Web UI displays the three Applications, it is also sensitive to changes made by those applications. If one of those applications becomes impaired due to a bad change, that application would seem to affect the Web UI’s functionality. The Common UI’s potential sensitivity to change is ‘High’.
The Web API is similar to the Common Web UI, in terms of potential sensitivity and impact. The potential impact of changes to the Web API is ‘Very High’, since a defect there could result in the potential impairment of the retailer’s affiliate applications. The potential sensitivity of the Web API to changes comes from its direct dependency on the Services Layer, and indirectly on the Data Layer. The Web API’s potential sensitivity to change is ‘High’. There is very little chance of potential impact to the Web API from the retailer’s affiliate applications.
Impact of Key Components
Lastly, as systems grow in complexity, certain components often become so key, they have the potential to impact the entire environment, a true single point of failure. Below, note the potential impact of changes to the Common Services Layer on all other components. As the software environment has grown in complexity, the Common Services Layer sits at the heart of the system. The Services Layer has multiple components directly dependent on it (i.e. Application C), as well as other components indirectly dependent on it (i.e. Third-Party Applications). It is also the only point of access to and from the Common Data Layer.
There are steps organizations can take to mitigate the potential risk caused by changes to key components, like the Services Layer. Areas organizations commonly focus on to reduce risk are higher code quality, increased test coverage, and improved performance, fault tolerance, system redundancy, and rollback capabilities. Additionally, management should more thoroughly scrutinize proposed software changes to key components, balancing new features with the need for stability, availability, and performance.
Management must balance the need for new features with need for stability, availability, and performance.
Specific to services, organizations often look to decouple larger services, creating smaller, more focused services. Better separation of concerns increases the likelihood that potential impairments caused by code defects are isolated to a smaller subset of functionality.
Conclusion
In this brief post, we examined a potential risk to delivering reliable software, the impact of software changes. There are many risks to delivering reliable software. Once all sources of risk are identified and quantified, the overall level of risk to delivering reliable software can be assessed, and steps taken to reduce the potential impact.
Cloud-based Continuous Integration and Deployment for .NET Development
Posted by Gary A. Stafford in .NET Development, Build Automation, Client-Side Development, DevOps, Enterprise Software Development, PowerShell Scripting, Software Development on May 25, 2014
Introduction
Whether you are part of a large enterprise development environment, or a member of a small start-up, you are likely working with remote team members. You may be remote, yourself. Developers, testers, web designers, and other team members, commonly work remotely on software projects. Distributed teams, comprised of full-time staff, contractors, and third-party vendors, often work in different buildings, different cities, and even different countries.
If software is no longer strictly developed in-house, why should our software development and integration tools be located in-house? We live in a quickly evolving world of Saas, PaaS, and IaaS. Popular SaaS development tools include Visual Studio Online, GitHub, BitBucket, Travis-CI, AppVeyor, CloudBees, JIRA, AWS, Microsoft Azure, Nodejitsu, and Heroku, to name just a few. With all these ‘cord-cutting’ tools, there is no longer a need for distributed development teams to be tethered to on-premise tooling, via VPN tunnels and Remote Desktop Connections.
There are many combinations of hosted software development and integration tools available, depending on your technology stack, team size, and budget. In this post, we will explore one such toolchain for .NET development. Using Git, GitHub, AppVeyor, and Microsoft Azure, we will continuously build, test, and deploy a multi-tier .NET solution, without ever leaving Visual Studio. This particular toolchain has strong integration between tools, and will scale to fit most development teams.
Git and GitHub
Git and GitHub are widely used in development today. Visual Studio 2013 has fully-integrated Git support and Visual Studio 2012 has supported Git via a plug-in since early last year. Git is fully compatible with Windows. Additionally, there are several third party tools available to manage Git and GitHub repositories on Windows. These include Git Bash (my favorite), Git GUI, and GitHub for Windows.
GitHub acts as a replacement for your in-house Git server. Developers commit code to their individual local Git project repositories. They then push, pull, and merge code to and from a hosted GitHub repository. For security, GitHub requires a registered username and password to push code. Data transfer between the local Git repository and GitHub is done using HTTPS with SSL certificates or SSH with public-key encryption. GitHub also offers two-factor authentication (2FA). Additionally, for those companies concerned about privacy and added security, GitHub offers private repositories. These plans range in price from $25 to $200 per month, currently.
GitHub View of Solution
AppVeyor
AppVeyor’s tagline is ‘Continuous Integration for busy developers’. AppVeyor automates building, testing and deployment of .NET applications. AppVeyor is similar to Jenkins and Hudson in terms of basic functionality, except AppVeyor is only provided as a SaaS. There are several hosted solutions in the continuous integration and delivery space similar to AppVeyor. They include CloudBees (hosted-Jenkins) and Travis-CI. While CloudBees and Travis CI works with several technology stacks, AppVeyor focuses specifically on .NET. Its closest competitor may be Microsoft’s new Visual Studio Online.
Identical to GitHub, AppVeyor also offers private repositories (spaces for building and testing code). Prices for private repositories currently range from $39 to $319 per month. Private repositories offer both added security and support. AppVeyor integrates nicely with several cloud-based code repositories, including GitHub, BitBucket, Visual Studio Online, and Fog Creek’s Kiln.
AppVeyor View of Latest Build of Solution
Azure
This post demonstrates continuous deployment from AppVeyor to a Microsoft Server 2012-based Azure VM. The VM has IIS 8.5, Web Deploy 3.5, IIS Web Management Service (WMSVC), and other components and configuration necessary to host the post’s sample Solution. AppVeyor would work just as well with Azure’s other hosting options, as well as other cloud-based hosting providers, such as AWS or Rackspace, which also supports the .NET stack.
New Microsoft Azure Portal View of VM
Sample Solution
The Visual Studio Solution used for this post was originally developed as part of an earlier post, Consuming Cross-Domain WCF REST Services with jQuery using JSONP. The original Solution, from 2011, demonstrated jQuery’s AJAX capabilities to communicate with a RESTful WCF service, cross-domains, using JSONP. I have since updated and modernized the Solution for this post. The revised Solution is on a new branch (‘rev2014’) on GitHub. Major changes to the Solution include an upgrade from VS2010 to VS2013, the use of Git DVCS, NuGet package management, Web Publish Profiles, Web Essentials for bundling JS and CSS, Twitter Bootstrap, unit testing, and a lot of code refactoring.
Revised Restaurant Menu Demo Viewed on Android Tablet
The updated VS Solution contains the following four Projects:
- Restaurant – C# Class Library
- RestaurantUnitTests – Unit Test Project
- RestaurantWcfService – C# WCF Service Application
- RestaurantDemoSite – Web Site (JS/HTML5)
VS 2013 View of Solution
The Visual Studio Solution Explorer tab, here, shows all projects contained in the Solution, and the primary files and directories they contain.
As explained in the earlier post, the ‘RestaurantDemoSite’ web site makes calls to the ‘RestaurantWcfService’ WCF service. The WCF service exposes two operations, one that returns the menu (‘GetCurrentMenu’), and the other that accepts an order (‘SendOrder’). For simplicity, orders are stored in the files system as JSON files. No database is required for the Solution. All business logic is contained in the ‘Restaurant’ class library, which is referenced by the WCF service. This architecture is illustrated in this Visual Studio Assembly Dependencies Diagram.
Installing and Configuring the Solution
The README.md file in the GitHub repository contains instructions for installing and configuring this Solution. In addition, a set of PowerShell scripts, part of the Solution’s repository, makes the installation and configuration process, quick and easy. The scripts handle creating the necessary file directories and environment variables, setting file access permissions, and configuring IIS websites. Make sure to change the values of the environment variables before running the script. For reference, below are the contents of several of the supplied scripts. You should use the supplied scripts.
# Create environment variables
[Environment]::SetEnvironmentVariable("AZURE_VM_HOSTNAME", `
"{YOUR HOSTNAME HERE}", "User")
[Environment]::SetEnvironmentVariable("AZURE_VM_USERNAME", `
"{YOUR USERNME HERE}", "User")
[Environment]::SetEnvironmentVariable("AZURE_VM_PASSWORD", `
"{YOUR PASSWORD HERE}", "User")
# Create new restaurant orders JSON file directory
$newDirectory = "c:\RestaurantOrders"
if (-not (Test-Path $newDirectory)){
New-Item -Type directory -Path $newDirectory
}
$acl = Get-Acl $newDirectory
$ar = New-Object System.Security.AccessControl.FileSystemAccessRule(`
"INTERACTIVE","Modify","ContainerInherit, ObjectInherit", "None", "Allow")
$acl.SetAccessRule($ar)
Set-Acl $newDirectory $acl
# Create new website directory
$newDirectory = "c:\RestaurantDemoSite"
if (-not (Test-Path $newDirectory)){
New-Item -Type directory -Path $newDirectory
}
$acl = Get-Acl $newDirectory
$ar = New-Object System.Security.AccessControl.FileSystemAccessRule(`
"IUSR","ReadAndExecute","ContainerInherit, ObjectInherit", "None", "Allow")
$acl.SetAccessRule($ar)
Set-Acl $newDirectory $acl
# Create new WCF service directory
$newDirectory = "c:\MenuWcfRestService"
if (-not (Test-Path $newDirectory)){
New-Item -Type directory -Path $newDirectory
}
$acl = Get-Acl $newDirectory
$ar = New-Object System.Security.AccessControl.FileSystemAccessRule(`
"IUSR","ReadAndExecute","ContainerInherit, ObjectInherit", "None", "Allow")
$acl.SetAccessRule($ar)
Set-Acl $newDirectory $acl
$ar = New-Object System.Security.AccessControl.FileSystemAccessRule(`
"IIS_IUSRS","ReadAndExecute","ContainerInherit, ObjectInherit", "None", "Allow")
$acl.SetAccessRule($ar)
Set-Acl $newDirectory $acl
# Create main website in IIS
$newSite = "MenuWcfRestService"
if (-not (Test-Path IIS:\Sites\$newSite)){
New-Website -Name $newSite -Port 9250 -PhysicalPath `
c:\$newSite -ApplicationPool "DefaultAppPool"
}
# Create WCF service website in IIS
$newSite = "RestaurantDemoSite"
if (-not (Test-Path IIS:\Sites\$newSite)){
New-Website -Name $newSite -Port 9255 -PhysicalPath `
c:\$newSite -ApplicationPool "DefaultAppPool"
}
Cloud-Based Continuous Integration and Delivery
Webhooks
The first point of integration in our hosted toolchain is between GitHub and AppVeyor. In order for AppVeyor to work with GitHub, we use a Webhook. Webhooks are widely used to communicate events between systems, over HTTP. According to GitHub, ‘every GitHub repository has the option to communicate with a web server whenever the repository is pushed to. These webhooks can be used to update an external issue tracker, trigger CI builds, update a backup mirror, or even deploy to your production server.‘ Basically, we give GitHub permission to tell AppVeyor every time code is pushed to the GitHub. GitHub sends a HTTP POST to a specific URL, provided by AppVeyor. AppVeyor responds to the POST by cloning the GitHub repository, and building, testing, and deploying the Projects. Below is an example of a webhook for AppVeyor, in GitHub.
GitHub’s AppVeyor Webhook Configuration
Unit Tests
To help illustrate the use of AppVeyor for automated unit testing, the updated Solution contains a Unit Test Project. Every time code is committed to GitHub, AppVeyor will clone and build the Solution, followed by running the set of unit tests shown below. The project’s unit tests test the Restaurant class library (‘restaurant.dll’). The unit tests provide 100% code coverage, as shown in the Visual Studio Code Coverage Results tab, below:
Code Coverage Results for Restaurant Class Library
AppVeyor runs the Solution’s automated unit tests using VSTest.Console.exe. VSTest.Console calls the unit test Project’s assembly (‘restaurantunittests.dll’). As shown below, the VSTest command (in light blue) runs all tests, and then displays individual test results, a results summary, and the total test execution time.
AppVeyor Running Automated Unit Tests Using VSTest.Console
VSTest.Console has several command line options similar to MSBuild. They can be adjusted to output various levels of feedback on test results. For larger projects, you can selectively choose which pre-defined test sets to run. Test sets needs are set-up in Solution, in advance.
Configuring Azure VM
Before we publish the Solution from AppVeyor to the Azure, we need to configure the VM. Again, we can use PowerShell to script most of the configuration. Most scripts are the same ones we used to configure our local environment. The README.md file in the GitHub repository contains instructions. The scripts handle creating the necessary file directories, setting file access permissions, configuring the IIS websites, creating the Web Deploy User account, and assigning it in IIS. For reference, below are the contents of several of the supplied scripts. You should use the supplied scripts.
# Create new restaurant orders JSON file directory
$newDirectory = "c:\RestaurantOrders"
if (-not (Test-Path $newDirectory)){
New-Item -Type directory -Path $newDirectory
}
$acl = Get-Acl $newDirectory
$ar = New-Object System.Security.AccessControl.FileSystemAccessRule(`
"INTERACTIVE","Modify","ContainerInherit, ObjectInherit", "None", "Allow")
$acl.SetAccessRule($ar)
Set-Acl $newDirectory $acl
# Create new website directory
$newDirectory = "c:\RestaurantDemoSite"
if (-not (Test-Path $newDirectory)){
New-Item -Type directory -Path $newDirectory
}
$acl = Get-Acl $newDirectory
$ar = New-Object System.Security.AccessControl.FileSystemAccessRule(`
"IUSR","ReadAndExecute","ContainerInherit, ObjectInherit", "None", "Allow")
$acl.SetAccessRule($ar)
Set-Acl $newDirectory $acl
# Create new WCF service directory
$newDirectory = "c:\MenuWcfRestService"
if (-not (Test-Path $newDirectory)){
New-Item -Type directory -Path $newDirectory
}
$acl = Get-Acl $newDirectory
$ar = New-Object System.Security.AccessControl.FileSystemAccessRule(`
"IUSR","ReadAndExecute","ContainerInherit, ObjectInherit", "None", "Allow")
$acl.SetAccessRule($ar)
Set-Acl $newDirectory $acl
$ar = New-Object System.Security.AccessControl.FileSystemAccessRule(`
"IIS_IUSRS","ReadAndExecute","ContainerInherit, ObjectInherit", "None", "Allow")
$acl.SetAccessRule($ar)
Set-Acl $newDirectory $acl
# Create main website in IIS
$newSite = "MenuWcfRestService"
if (-not (Test-Path IIS:\Sites\$newSite)){
New-Website -Name $newSite -Port 9250 -PhysicalPath `
c:\$newSite -ApplicationPool "DefaultAppPool"
}
# Create WCF service website in IIS
$newSite = "RestaurantDemoSite"
if (-not (Test-Path IIS:\Sites\$newSite)){
New-Website -Name $newSite -Port 9255 -PhysicalPath `
c:\$newSite -ApplicationPool "DefaultAppPool"
}
# Create new local non-admin User and Group for Web Deploy
# Main variables (Change these!)
[string]$userName = "USER_NAME_HERE" # mjones
[string]$fullName = "FULL USER NAME HERE" # Mike Jones
[string]$password = "USER_PASSWORD_HERE" # pa$$w0RD!
[string]$groupName = "GROUP_NAME_HERE" # Development
# Create new local user account
[ADSI]$server = "WinNT://$Env:COMPUTERNAME"
$newUser = $server.Create("User", $userName)
$newUser.SetPassword($password)
$newUser.Put("FullName", "$fullName")
$newUser.Put("Description", "$fullName User Account")
# Assign flags to user
[int]$ADS_UF_PASSWD_CANT_CHANGE = 64
[int]$ADS_UF_DONT_EXPIRE_PASSWD = 65536
[int]$COMBINED_FLAG_VALUE = 65600
$flags = $newUser.UserFlags.value -bor $COMBINED_FLAG_VALUE
$newUser.put("userFlags", $flags)
$newUser.SetInfo()
# Create new local group
$newGroup=$server.Create("Group", $groupName)
$newGroup.Put("Description","$groupName Group")
$newGroup.SetInfo()
# Assign user to group
[string]$serverPath = $server.Path
$group = [ADSI]"$serverPath/$groupName, group"
$group.Add("$serverPath/$userName, user")
# Assign local non-admin User in IIS for Web Deploy
[System.Reflection.Assembly]::LoadWithPartialName("Microsoft.Web.Management")
[Microsoft.Web.Management.Server.ManagementAuthorization]::Grant(`
$userName, "$Env:COMPUTERNAME\MenuWcfRestService", $FALSE)
[Microsoft.Web.Management.Server.ManagementAuthorization]::Grant(`
$userName, "$Env:COMPUTERNAME\RestaurantDemoSite", $FALSE)
Publish Profiles
The second point of integration in our toolchain is between AppVeyor and the Azure VM. We will be using Microsoft’s Web Deploy to deploy our Solution from AppVeyor to Azure. Web Deploy integrates with the IIS Web Management Service (WMSVC) for remote deployment by non-administrators. I have already configured Web Deploy and created a non-administrative user on the Azure VM. This user’s credentials will be used for deployments. These are the credentials in the username and password environment variables we created.
To continuously deploy to Azure, we will use Web Publish Profiles with Microsoft’s Web Deploy technology. Both the website and WCF service projects contain individual profiles for local development (‘LocalMachine’), as well as deployment to Azure (‘AzureVM’). The ‘AzureVM’ profiles contain all the configuration information AppVeyor needs to connect to the Azure VM and deploy the website and WCF service.
The easiest way to create a profile is by right-clicking on the project and selecting the ‘Publish…’ and ‘Publish Web Site’ menu items. Using the Publish Web wizard, you can quickly build and validate a profile.
Publish Web Profile Tab
Each profile in the above Profile drop-down, represents a ‘.pubxml’ file. The Publish Web wizard is merely a visual interface to many of the basic configurable options found in the Publish Profile’s ‘.pubxml’ file. The .pubxml profile files can be found in the Project Explorer. For the website, profiles are in the ‘App_Data’ directory (i.e. ‘Restaurant\RestaurantDemoSite\App_Data\PublishProfiles\AzureVM.pubxml’). For the WCF service, profiles are in the ‘Properties’ directory (i.e. ‘Restaurant\RestaurantWcfService\Properties\PublishProfiles\AzureVM.pubxml’).
As an example, below are the contents of the ‘LocalMachine’ profile for the WCF service (‘LocalMachine.pubxml’). This is about as simple as a profile gets. Note since we are deploying locally, the profile is configured to open the main page of the website in a browser, after deployment; a helpful time-saver during development.
<?xml version="1.0" encoding="utf-8"?>
<!--
This file is used by the publish/package process of your Web project.
You can customize the behavior of this process by editing this MSBuild file.
In order to learn more about this please visit http://go.microsoft.com/fwlink/?LinkID=208121.
-->
<Project ToolsVersion="4.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
<PropertyGroup>
<WebPublishMethod>FileSystem</WebPublishMethod>
<LastUsedBuildConfiguration>Debug</LastUsedBuildConfiguration>
<LastUsedPlatform>Any CPU</LastUsedPlatform>
<SiteUrlToLaunchAfterPublish>http://localhost:9250/RestaurantService.svc/help</SiteUrlToLaunchAfterPublish>
<LaunchSiteAfterPublish>True</LaunchSiteAfterPublish>
<ExcludeApp_Data>True</ExcludeApp_Data>
<publishUrl>C:\MenuWcfRestService</publishUrl>
<DeleteExistingFiles>True</DeleteExistingFiles>
</PropertyGroup>
</Project>
A key change we will make is to use environment variables in place of sensitive configuration values in the ‘AzureVM’ Publish Profiles. The Web Publish wizard does not allow this change. To do this, we must edit the ‘AzureVM.pubxml’ file for both the website and the WCF service. We will replace the hostname of the server where we will deploy the projects with a variable (i.e. AZURE_VM_HOSTNAME = ‘MyAzurePublicServer.net’). We will also replace the username and password used to access the deployment destination. This way, someone accessing the Solution’s source code, won’t be able to obtain any sensitive information, which would give them the ability to hack your site. Note the use of the ‘AZURE_VM_HOSTNAME’ and ‘AZURE_VM_USERNAME’ environment variables, show below.
<?xml version="1.0" encoding="utf-8"?>
<!--
This file is used by the publish/package process of your Web project.
You can customize the behavior of this process by editing this MSBuild file.
In order to learn more about this please visit http://go.microsoft.com/fwlink/?LinkID=208121.
-->
<Project ToolsVersion="4.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
<PropertyGroup>
<WebPublishMethod>MSDeploy</WebPublishMethod>
<LastUsedBuildConfiguration>AppVeyor</LastUsedBuildConfiguration>
<LastUsedPlatform>Any CPU</LastUsedPlatform>
<SiteUrlToLaunchAfterPublish />
<LaunchSiteAfterPublish>False</LaunchSiteAfterPublish>
<ExcludeApp_Data>True</ExcludeApp_Data>
<MSDeployServiceURL>https://$(AZURE_VM_HOSTNAME):8172/msdeploy.axd</MSDeployServiceURL>
<DeployIisAppPath>MenuWcfRestService</DeployIisAppPath>
<RemoteSitePhysicalPath />
<SkipExtraFilesOnServer>False</SkipExtraFilesOnServer>
<MSDeployPublishMethod>WMSVC</MSDeployPublishMethod>
<EnableMSDeployBackup>True</EnableMSDeployBackup>
<UserName>$(AZURE_VM_USERNAME)</UserName>
<_SavePWD>False</_SavePWD>
<_DestinationType>AzureVirtualMachine</_DestinationType>
</PropertyGroup>
</Project>
The downside of adding environment variables to the ‘AzureVM’ profiles, the Publish Profile wizard feature within Visual Studio will no longer allow us to deploy, using the ‘AzureVM’ profiles. As demonstrated below, after substituting variables for actual values, the ‘Server’ and ‘User name’ values will no longer display properly. We can confirm this by trying to validate the connection, which fails. This does not indicate your environment variable values are incorrect, only that Visual Studio can longer correctly parse the ‘AzureVM.pubxml’ file and display it properly in the IDE. No big deal…
Publish Web Connection Tab – Failed Validation
We can use the command line or PowerShell to deploy with the ‘AzureVM’ profiles. AppVeyor accepts both command line input, as well as PowerShell for most tasks. All examples in this post and in the GitHub repository use PowerShell.
To build and deploy (publish) to Azure from the command line or PowerShell, we will use MSBuild. Below are the MSBuild commands used by AppVeyor to build our Solution, and then deploy our Solution to Azure. The first two MSBuild commands build the WCF service and the website. The second two deploy them to Azure. There are several ways you could construct these commands to successfully build and deploy this Solution. I found these commands to be the most succinct. I have split the build and the deploy functions so that the AppVeyor can run the automated unit tests, in between. If the tests don’t pass, we don’t want to deploy the code.
# Build WCF service # (AppVeyor config ignores website Project in Solution) msbuild Restaurant\Restaurant.sln ` /p:Configuration=AppVeyor /verbosity:minimal /nologo # Build website msbuild Restaurant\RestaurantDemoSite\website.publishproj ` /p:Configuration=Release /verbosity:minimal /nologo Write-Host "*** Solution builds complete."
# Deploy WCF service # (AppVeyor config ignores website Project in Solution) msbuild Restaurant\Restaurant.sln ` /p:DeployOnBuild=true /p:PublishProfile=AzureVM /p:Configuration=AppVeyor ` /p:AllowUntrustedCertificate=true /p:Password=$env:AZURE_VM_PASSWORD ` /verbosity:minimal /nologo # Deploy website msbuild Restaurant\RestaurantDemoSite\website.publishproj ` /p:DeployOnBuild=true /p:PublishProfile=AzureVM /p:Configuration=Release ` /p:AllowUntrustedCertificate=true /p:Password=$env:AZURE_VM_PASSWORD ` /verbosity:minimal /nologo Write-Host "*** Solution deployments complete."
Below is the output from AppVeyor showing the WCF Service and website’s deployment to Azure. Deployment is the last step in the continuous delivery process. At this point, the Solution was already built and the automated unit tests completed, successfully.
AppVeyor Output from Deployments to Azure.
Below is the final view of the sample Solution’s WCF service and web site deployed to IIS 8.5 on the Azure VM.
Final View of IIS Sites Running on Azure VM
Links
- Introduction to Web Deploy (see ‘How does it work?’ diagram on non-admin deployments)
- ASP.NET Web Deployment using Visual Studio: Command Line Deployment
- IIS: Enable IIS remote management
- Sayed Ibrahim Hashimi’s Blog
- Continuous Integration and Continuous Delivery
- How to: Edit Deployment Settings in Publish Profile (.pubxml) Files
Building a Deployment Pipeline Using Git, Maven, Jenkins, and GlassFish (Part 2 of 2)
Posted by Gary A. Stafford in Build Automation, DevOps, Enterprise Software Development, Software Development on November 13, 2013
Build an automated deployment pipeline for your Java EE applications using leading open-source technologies, including NetBeans, Git, Maven, JUnit, Jenkins, and GlassFish. All source code for this post is available on GitHub.
Introduction
In part 1, Building a Deployment Pipeline Using Git, Maven, Jenkins, and GlassFish (Part 1 of 2), we built the first part of our basic deployment pipeline using leading open-source technologies. In part 2, we will use Jenkins CI Server and Oracle GlassFish Application Server to complete our deployment pipeline.
To review, the three main goals of our deployment pipeline are continuous integration, automated testing, and continuous deployment. Our objective is to automatically compile, test, assemble, and deploy our Java EE application to multiple environments, as the project progresses through the software development life cycle (SDLC).
Setting up Git Server
As I mentioned in part 1, as a part of a development team using Git, you would place your project on a remote Git Server. You and your team members would each clone the repository from the Git Server to your local development environments. You and your team would commit your code changes locally, then pull, merge, and push your changes back to the remote Git Server. Jenkins will pull the project’s source code from the Git Server.
In part 1 of this post, we just created a local Git repository. In part 2, we will properly set-up our project on a remote Git Server. First, we need to export our local repository into a new, bare repository on the Git Server. The Git term, ‘bare repository’, refers to a repository that does not contain a working directory. The repository has no working copies of your source files. You only use the bare repository to clone, pull from, and push to. The bare repository contains a .git extension (i.e. ssh://user@server:/git-repos/myproject.git).
From the root of your remote Git Server repository, execute the following command, substituting the path to your local project. If your Git Server is on a separate machine that your local project repository, you will need to copy the new bare repository to the remote Git Server. This involves a few simple steps, explained in this post, and at git-scm.com.
| git clone --bare {path-to-existing-local-repository}\{name-of-repository} {name-of-repository}.git |
Export Local Project to New Bare Repository
Once you have created the repository on the remote Git Server, I would recommend you clone the remote repository to your local machine and discard your original local repository from part 1 of the post. You don’t have to do this step, but cloning fresh from the server will make sure Git is working correctly. The screen grabs below illustrate an example of cloning a new repository to my local NetBeans Project folder.
Clone New Bare Server Repository – Screen 1
Clone New Bare Server Repository – Screen 2
Clone New Bare Server Repository – Screen 3
Configuring Jenkins
The diagram below illustrates the deployment pipeline from Git Server to Jenkins to GlassFish in finer detail. It begins with an initial commit to the local Git project repository and ends with the deployment of the project’s WAR file to the GlassFish domain. We will walk through it step-by-step.
Jenkins Plugins
Before we create our new Jenkins Jobs, we need to configure Jenkins properly. You will need a recent version of Jenkins installed, along with the following plugins:
- Build With Parameters Plugin
- Copy Artifact Plugin
- Jenkins GIT plugin (includes Jenkins GIT client plugin)
- Jenkins Parameterized Trigger plugin
- Maven Integration plugin
- Credentials Plugin (optional for use with Git Server if security is enabled)
- ThinBackup (optional to install supplied Jenkins jobs configuration files)
Global Security
Jenkins can be configured with or without Global Security. For this post, I have enabled Global Security, as it typical of most development environments. I chose to use ‘Jenkins’s own user database’ option for authentication. In larger development environments, authentication would normally be done against LDAP.
Configuring Global Security
The user I have set up, ‘jenkins’, will be the user that Git authenticates with when connecting to Jenkins (explained later). Set up your own user and note their API Token. Since Global Security has been enabled, we will need the token later to trigger the Jenkins build from Git. Your user’s unique api token will be different than in the example below.
Jenkins User API Token
Jenkins Jobs
We will set up two Jenkins ‘free-style software project’ jobs, ‘GitMavenGlassFish_Build’ and ‘GitMavenGlassFish_Deploy’. We won’t be using the obvious choice, a ‘maven2/3 project’. If you’re interested, here’s why. The first job, the build job, will be responsible for pulling the source code from the Git Server. The build job, with help from Maven, will compile, test, and assemble the application code. The second job, the deployment job, will pull the artifacts from the build job and deploy them to GlassFish. The build job will trigger the deployment job, once the build job completes successfully. This is explained in detail, to follow.
Why Two Jobs?
Following good modular design and Separation of Concerns (SoC) principles, separating the build from the deployment gains us several advantages, including:
- Modularity– Ability to change deployment methodology or deployment targets, without disrupting the build and test process. For example, we might move the application hosting from GlassFish to WebLogic, or decide to use Ant instead of Maven for deployment tasks. This can happen totally independent of the build and testing processes.
- Separation/Isolation – For any reason we are unable to deploy the artifacts as part of the deployment job, we won’t impact the continuous integration and automated testing processes, which are part of the separate build job.
- Support – Support is easier by having smaller pieces of functionality to troubleshoot and maintain.
In a larger enterprise environment, you would probably encounter further separation of concerns. Unit testing, performance testing, deployment validation, and documentation generation (javadocs) are often handled by separate jobs. Jenkins represents a smaller pipeline within our larger deployment pipeline.
I intentionally left out notification for brevity. At minimum, you would want to be notified when the build or deployment jobs failed. Additionally, with continuous deployment, the deployment would trigger a notification to the stakeholders of that environment, such as the Testers. This lets them know the new software is ready to be tested. Notifications often include a list of bug fixes and feature enhancements that need to be tested. This can easily be pulled from Git into Jenkins and out to the end user.
Both Jenkins jobs definitions are available as xml files on gist.github.com. Using Jenkins’ ThinBackup Plugin, you can save both gists locally, and then restore them to your Jenkins server. The build job gist is here and the deployment job gist is here. This may save you some configuration time.
Jenkins Build Job
Both the build job and the deployment jobs require an input parameter. This property represents the targeted environment (GlassFish domain) for deployment, such as ‘testing’.. How this parameter is passed to Jenkins is discussed later in the Git Hooks section, below.
Reviewing the below screen grab of the build job’s configuration, you will observe the following steps:
- Build Request – A build request is received by the job (explained later). The request contains an input parameter indicating the ‘environment’. The parameter must be one of the choices listed in ‘Choices’.
- Maven Dependencies – Based on the pom file, Maven retrieves all the required dependencies from the remote Maven repository, if the dependencies are not already contained in the workspace’s local repository. Note the setting ‘User private Maven repository. This creates a local repository for project dependencies within the project’s workspace.
- Pull from Git – Jenkins pulls the code from the Git Server using the supplied repository configuration information. Note my Git Server does not require authentication. If it did, we would set-up and use the proper credentials.
- Build – Jenkins builds the project using the Maven command ‘clean install -e’. The pom file contains the necessary configuration information.
- Unit Test – The above Maven ‘install’ command also calls JUnit to execute the unit tests. The results of these tests are published and displayed as part of the build job’s details.
- Assemble WAR – The above Maven ‘install’ command also assembles the project’s WAR file.
- Archive Artifacts – Based on the success of the build and unit tests, Jenkins archives specific artifacts needed by the deployment job. Jenkins uses the input parameter in #1 to define which properties file and password file to archive.
- Trigger Deployment Job – Based on the success of the build and unit tests, Jenkins triggers the ‘downstream’ deployment job, passing it the same environment parameter.
Jenkins Build Job Configuration
Jenkins Deployment Job
Reviewing the below screen grab of the deployment job’s configuration, you will observe the following steps:
- Build Request – A build request is received from the upstream build job. The request contains the input parameter indicating ‘environment’.
- Copy Artifacts – Jenkins copies the artifacts from the build job that called the deploy job.
- Read Properties – Maven executes the command ‘mvn properties:read-project-properties glassfish:redeploy -e’. The first half of this command instructs Maven to read the appropriate properties file, as indicated by the environment parameter, ‘glassfish.properties.file.argument=${environment}’.
- POM – Maven substitutes the key ‘glassfish.properties.file.argument’ in the pom file with the environment value. This tells Maven the name of the properties file, which supplies all the remaining property values to the pom file.
- Maven Dependencies – If the dependencies are not already contained in the workspace’s local repository, Maven retrieves all the required dependencies from the remote Maven repositories, based on the pom. Note the setting ‘User private Maven repository’ checked in the screen grab below. This option instructs Jenkins to creates a local repository for project dependencies within the project’s workspace.
- Deployment – The last half of the command in #3 deploys, or more accurately redeploys the application’s WAR file to GlassFish. The ‘glassfish:redeploy’ works only if the WAR file has already been initially deployed to the GlassFish domain using the ‘glassfish:deploy’ command. For this process, I am assuming the initial deployment was already done directly through the GlassFish Administration Console, NetBeans, or command line.
Jenkins Deploy Job Configuration
Git Hooks
To achieve continuous integration, we want to automatically build and test our job after each change to our code. We have a number of choices to make this happen. The obvious choice is letting Jenkins poll the Git Server. Although polling would simplify configuration, polling is frowned upon in many environments. Even the creator of Jenkins, Kohsuke Kawaguchi, frowns upon polling in his post, ‘Polling Must Die‘.
Why is polling bad? It adds unnecessary activity and delay. Let’s say Jenkins’ polling frequency is set to every 2 minutes, but you only have an average of 5 pushes to your remote Git Server project repository per day. Based on these stats, in just one day, Jenkins will poll Git 720 times to discover only 5 pushes. That’s 144 times per push. Also, based on the polling frequency, when you do push, you could wait up to 2 minutes for Jenkins to queue the build job. The longer you wait for feedback on your changes, the greater chance your defects could be pulled down by other developers. You should expect immediate and continuous feedback.
A vastly more efficient and configurable method of continuous integration between Git and Jenkins is Git Hooks. Git Hooks allow us to execute scripts based on specific Git actions. In our case, when a developer completes a successful push to the remote Git Server project repository, we want to call Jenkins to build, test, and deploy the modified project code. Using hooks means we only call Jenkins when a successful push is completed. Furthermore, we can be assured Jenkins will immediately queue our request to build and deploy the job when a push occurs.
Post-Receive Hook
There are several types of Git Hooks. They include ‘post-commit’, ‘pre-push’, ‘update’, ‘pre-rebase, and so forth. I recommend this post on kernel.org for a good explanation of the hook types and thier purposes. Git also includes sample hook files inside the ‘hooks’ subdirectory of each new repository .git folder.
For our pipeline, we will employ the ‘post-receive’ hook. Whenever a successful push is received by Git Server’s project repository, the ‘post-receive’ hook will be called. The script commands, contained in the post-receive hook file, will be executed. Hooks can language agnostic; they can be almost any scripting language, such as Perl, Shell, Bash, or Ruby.
To create the hook, create a new file, ‘post-receive’, in the hooks sub-directory of the Git Server’s project repository. Add the below code to the file. Change the command to match your local file path. Also, change the API Token to match your user’s token from Jenkins. Note the command requires cURL to be installed on the Git Server. If installing cURL is not an option, there are other options available to execute the http post call from the hook’s script.
| #!/bin/sh | |
| # Call Jenkins to start build and pass environment parameter | |
| # | |
| echo "executing post-receive hook" | |
| echo "environment=testing" | |
| echo "user=jenkins" | |
| # cURL POST request using jenkins user with API token | |
| curl -u jenkins:{your-api-token-here} \ | |
| --data "delay=0sec&environment=testing" \ | |
| "{your-jenkins-server-url:port}/job/GitMavenGlassFish_Build/buildWithParameters" |
NetBeans and Git Hooks
Now some slightly bad news. As with any integration, there is always trade-offs; that is the case with NetBeans and Git. Although NetBeans works well with Git, there are a few features that have not been implemented. Unfortunately, this lack of complete integration effects NetBeans’ ability to make use of Git Hooks. Only after three hours of troubleshooting and research on the Internet, did I realize this limitation. The hooks fire fine if a git push command is executed from a command prompt or from within a Git application like Git Gui or Git Bash. However, from NetBeans, the Team -> Remote -> Push… does not cause the hooks to be called.
Post-Receive Hook Working from a Windows Command Prompt
Git Hooks do not work with NetBeans because NetBeans does not use a command line client for Git. NetBeans uses a pure java implementation of the Git client, Java GIT, known as JGit. I understand that other IDE’s also share this limitation. There are several discussions on StackOverflow and on the NetBeans bug tracking site about the issue and workarounds.
So what does this mean? You can use NetBeans to perform all of your local tasks. However, when it comes time to push your code back to the remote Git Server repository, you must use a command prompt, Bash shell, or a command line based tool. I recommend Git Gui. Git ships with built-in GUI tools, including git-gui and gitk. It can be downloaded from git-scm.com.
Git Gui Graphical User Interface for Git
Push Files Using Git GUI Instead of NetBeans
Pushing changes to the remote Git Server using Git Gui instead of NetBeans may seem inconvenient at first. However, the more advanced your needs become with Git, the more you will find you need the additional functionality of Git Bash, Git Gui, and gitk. Tasks like resetting the branch to a previous revision, compressing the Git repository database, and visualizing repository history, can all be done with tools like Git Gui and gitk. I have Git Gui running when I am working in NetBeans or other IDEs; it becomes second nature.
Using Git Gui and gitk to Examine and Modify the Project Repository
Deploying to GlassFish
At this point we have configured the Git Server, created the Jenkins build and deploy jobs, and configured our Git hook. We are ready to test our deployment pipeline. First, make sure your GlassFish domains are running. Also, recall we are assuming that an initial deployment of the application has occurred. This might be directly through the GlassFish Administration Console, through NetBeans, or via the command line. Recall, Jenkins will be only be executing a re-deploy.
Check and Start GlassFish Domains
To test the system, make an innocuous change to the Project. Commit the change to your local Git repository. Following that, push the change back to the remote Git Server repository using Git Gui. If the hook fired, you will see output to the Git Gui terminal window, echoed from the post-receive hook as it executed its script.
Push with Git Gui Triggering Jenkins Build
The post-receive hook executes the cURL command, which posts an HTTP request to Jenkins via the Jenkins Remote API. You should observe is the Jenkins build job queued and running.
Jenkins Build Job Running
When the build completes, review the Parameters menu option in the left navigation menu. It shows that the environment parameter was passed from the post-receive hook to the build job. The build results window also provides test results, Git Build Data, and the changes pushed to Git that triggered the CI build.
Jenkins Build Job Results
The console output from the build provides a detailed view of the build process. Using the ‘-e’ for echo with the Maven command, increases the level of output detail. You see the details of Maven copying the required dependencies from the remote repository to the local workspace repository, prior to compilation. You see the unit tests being executed. Finally, you see the WAR file assembled and the required artifacts archived.
Regarding Maven Dependencies, you will only see the dependencies copied on the first build to an empty workspace. Maven does not re-pull dependencies if they already exist in the workspace’s local repository. To see the difference, empty your workspace and build the job, then immediately rebuild the job. Compare the console outputs of both jobs. You will see a significant difference in the Maven dependency activities.
Jenkins Build Job Console Results
Once the build job has completed successfully, you should notice the Jenkins deployment job running, triggered by the build job. When complete, note the detail that lists the exact build job that called the deployment job, and its build number. For example, the upstream build job #45 triggered the downstream deployment job #33. This linkage between upstream and downstream jobs is retained in the job’s history.
As before, review the Parameters menu option in the left navigation menu. It shows that the environment parameter was passed from the post-receive hook to the build job, and then on to the deployment job.
Jenkins Deployment Job Complete
A review of the console output will confirm that the artifacts were copied from the build job and the WAR file was deployed to the ‘testing’ GlassFish domain.
Jenkins Deployment Job Console Output
GlassFish
If the hook fired, and both the Jenkins build and deployment jobs ran successfully, you should observe that the project’s WAR files, containing your recent change, was deployed to the testing GlassFish domain.
Application Installed on GlassFish Server Testing Domain
You can verify this by calling the application’s RESTful ‘resources/helloWorld’ URI, from your browser. Repeat the process by changing the output string, commit the change, and push. See if you see your change deployed.
Application Running on GlassFish Server Testing Domain
Jenkins Workflows
Using our deployment pipeline, we have two distinct workflow options:
- Continuous– Use Git hooks to build, test, and deploy the WAR file to the domain(s) of choice when changes are pushed. Any time a change is pushed, a build, test, and deploy, should occur. This would be just for development at first. Once the project enters the testing phase of the SDLC, then it would include deployments to testing.
- Semi-Automated – Start the Jenkins build manually in the Jenkins browser-based Administration Console. This is more typical for a release to Production. Most teams are not comfortable extending the continuous deployment functionality into Production. Often, a deployment team will deploy the project artifacts in a controlled and staged approach. The Jenkins build and/or deployment jobs both allow this feature, along with the ability to provide the environment parameter both jobs needs.
Conclusion
In part 1, we learned how to create a simple Java EE web application project in NetBeans using Maven. We learned how to integrate JUnit for unit testing, and how use Git to manage our source code.
In part 2, we learned how to configure a remote Git Server, how to configure Jenkins CI Server to clone our project from the Git Server, build, test, and assemble it. If the build was successful, we learned how to configure Jenkins to deploy our project to a specific GlassFish domain, based on the project’s stage in the SDLC. We achieved our goals of continuous integration, automated testing, and continuous deployment.
Going Forward
To extend and enhance our deployment pipeline, you might consider adding the following features: 1) further separate the Jenkins jobs by function, 2) add build and deploy notifications, 3) add the ability to deploy to multiple environments simultaneously (i.e. development and testing), 4) add additional testing to confirm the deployment to GlassFish, 5) configure a versioning and naming scheme for the deployed artifacts, and 6) add error handling if a parameter is not received or is not one of the expected values.
Building a Deployment Pipeline Using Git, Maven, Jenkins, and GlassFish (Part 1 of 2)
Posted by Gary A. Stafford in DevOps, Enterprise Software Development, Java Development, Software Development on November 4, 2013
Build an automated deployment pipeline for your Java EE applications using leading open-source technologies, including NetBeans, Git, Maven, JUnit, Jenkins, and GlassFish. All source code for this post is available on GitHub.
Introduction
In my earlier post, Build a Continuous Deployment System with Maven, Hudson, WebLogic Server, and JUnit, I demonstrated a basic deployment pipeline using leading open-source technologies. In this post, we will demonstrate a similar pipeline, substituting Jenkins CI Server for Hudson, and Oracle’s GlassFish Application Server for WebLogic Server. We will use the same NetBeans Java EE ‘Hello World’ RESTful Web Service sample project.
The three main goals of our deployment pipeline will be continuous integration, automated testing, and continuous deployment. Our objective is to automatically compile, test, assemble, and deploy our Java EE application to multiple environments, as the project progresses through the software development life cycle (SDLC).
Building a reliable deployment pipeline is complex and time-consuming. To make it as easy as possible in this post, I chose NetBeans IDE for development, Git Distributed Version Control System (DVCS) for managing our source code, Jenkins Continuous Integration (CI) Server for build automation, JUnit for automated unit testing, GlassFish for application hosting, and Apache Maven to manage our project’s dependencies. Maven will also manage the build and deployment process to GlassFish, along with Jenkins. The beauty of NetBeans is its out-of-the-box, built-in integration with Git, Maven, JUnit, and GlassFish. Likewise, Jenkins has plugin-based integration with Git, Maven, JUnit, and GlassFish. Also, Maven has plugin-based integration with GlassFish.
Maven is a powerful tool for managing modern software development projects. This post will only draw upon a small part of Maven’s functionality and plug-in architecture extensibility. Specifically, we will use the Maven GlassFish Plugin. According to the Java.net website, which host’s the plug-in project, ‘the Maven GlassFish Plugin is a Maven2 plugin allowing management of GlassFish domains and component deployments from within the Maven build life cycle.’
Requirements
To follow along with this post, I will assume you have recent versions of the following software installed and configured on your Windows OS-based computer (the process is nearly identical for Linux):
- NetBeans IDE. Current version: 7.4
- JUnit. Current version: 4.11 (included with NetBeans 7.4)
- GlassFish Server. Current version: 4.0 (included with NetBeans 7.4)
- Jenkins CI Server. Current version: 1.538
- Apache Maven. Current version: 3.1.1
- cURL. Current version: 7.33.0
- Git with Git Gui and gitk. Current version: 1.8.4.3
- Necessary system environmental variables:
M2_HOME, M2, JAVA_HOME, GLASSFISH_HOME, and PATH
GlassFish Domains
To simulate a simple deployment pipeline, we will create three GlassFish domains, simulating three common software environments, Development, Testing, and Production. A typical software project is promoted through these environments as it moves from development, to testing, and finally release to production. Each environment has distinct stakeholders with specific roles to play in the software development life cycle, including developers, testers, deployment teams, and end-users. Larger-scale, enterprise software development often includes other environments, such as Performance and Staging.
Create the domains from the command line using ‘asadmin’ commands such as the ones below. Note I have a ‘GLASSFISH_HOME’ system environment variable set up. The ports are your choice, but make sure they don’t conflict with existing installations of other applications, such as Jenkins, Tomcat, IIS, WebLogic, and so forth.
| asadmin create-domain --domaindir "%GLASSFISH_HOME%\domains" --adminport 7070 --instanceport 7071 production | |
| asadmin create-domain --domaindir "%GLASSFISH_HOME%\domains" --adminport 6060 --instanceport 6061 testing | |
| asadmin create-domain --domaindir "%GLASSFISH_HOME%\domains" --adminport 5050 --instanceport 5051 development |
As part of the creation process, you’re prompted for an admin account and a new password. I kept the ‘admin’ username, but added a new password for each domain created. This password is the same as one used in the separate password files (explained below).
| C:\Users\gstaffor>asadmin create-domain --domaindir "%GLASSFISH_HOME%\domains" --adminport 7070 --instanceport 7071 production | |
| Enter admin user name [Enter to accept default "admin" / no password]>admin | |
| Enter the admin password [Enter to accept default of no password]> | |
| Enter the admin password again> | |
| Using port 7070 for Admin. | |
| Using port 7071 for HTTP Instance. | |
| Using default port 7676 for JMS. | |
| Using default port 3700 for IIOP. | |
| Using default port 8181 for HTTP_SSL. | |
| Using default port 3820 for IIOP_SSL. | |
| Using default port 3920 for IIOP_MUTUALAUTH. | |
| Using default port 8686 for JMX_ADMIN. | |
| Using default port 6666 for OSGI_SHELL. | |
| Using default port 9009 for JAVA_DEBUGGER. | |
| Distinguished Name of the self-signed X.509 Server Certificate is: | |
| [CN={my_computer_name},OU=GlassFish,O=Oracle Corporation,L=Santa Clara,ST=California,C=US] | |
| Distinguished Name of the self-signed X.509 Server Certificate is: | |
| [CN={my_computer_name}-instance,OU=GlassFish,O=Oracle Corporation,L=Santa Clara,ST=California,C | |
| =US] | |
| Domain production created. | |
| Domain production admin port is 7070. | |
| Domain production admin user is "admin". | |
| Command create-domain executed successfully. |
Add the GlassFish domains to NetBeans’ Services -> Server tab, and start them.
Create New GlassFish 4.0 Production Domain – Screen 1
Create New GlassFish 4.0 Production Domain – Screen 2
Create New GlassFish 4.0 Production Domain – Screen 3
Create New GlassFish 4.0 Production Domain – Screen 4
Setting Up the Project
To set up our NetBeans project, you can clone the repository on GitHub or build your own project from scratch and copy the files into the project. I will not spend a lot of time explaining the code since we have used it in earlier posts. This post is about the deployment pipeline system, not the project’s code.
If you choose to create a new project, first, create a new Maven ‘Project from Archetype’. Select the Archetype for a ‘web application using Java EE 7’ (webapp-javaee7).
New Maven Project – Screen 1
New Maven Project – Screen 2
I recommend you create the project inside of your local Git repository folder.
New Maven Project – Screen 3
Maven will execute a series of commands to create the default NetBeans project with dependencies.
Git
As a part of a development team using Git, you place your project on a remote Git Server. You and your team members each clone the repository on the Git Server to your local development environments. You and your team commit your code changes locally, then pull, merge, and push your changes back to the Git Server. Jenkins will pull the project’s source code from the remote Git Server.
In part 2, we will properly set-up our project on the Git Server, exporting our existing repository into a new, bare repository on the Git Server. However, for brevity in part 1 of this post, we will just create a local Git repository. To start, create a new Git repository for the project. In NetBeans, select Team -> Git -> Initialize Repository… Choose the new Maven project folder.
Initialize New Git Repository
The initial view of the Maven project should look like the below screen grabs. Note the icons and the green files show that the project is part of the Git repository.
Initial Projects Tab View of New Maven Project
Initial Files Tab View of New Maven Project
Perform an initial commit of the project to Git to make sure everything is working.
Initial Commit of New Maven Project to Git
Next, copy the supplied HelloWorldResource. java and NameStorageBean.java classes into the project. The package classpath will be refactored by NetBeans. Copy all the remaining files and folders, including the (3) files in the WEB-INF folder, properties folder with (3) properties files, and passwords folder with (3) password files.
JUnit
Next, right-click on the NameStorageBean.java class and select Tools -> Create Tests. Replace the contents of the new NameStorageBeanTest.java file’s NameStorageBeanTest class with the contents of the supplied NameStorageBeanTest.java file. These are two very simple unit tests that will show how JUnit provides automated testing capabilities.
Create JUnit Tests – Screen 1
Create JUnit Tests – Screen 2
Project Object Model (POM)
Copy the contents of the supplied pom file into the new pom file. There is a lot of configuration in the supplied pom. It will be easier to copy the supplied pom file’s contents into your project then trying to configure it from scratch.
Basically, beyond the normal boilerplate pom configuration, we have defined (3) properties, (3) dependencies, and (5) build plugins. The three dependencies are junit, jersey-servlet, and javaee-web-api. The five plugins are maven-compiler-plugin, maven-war-plugin, maven-dependency-plugin, properties-maven-plugin, and the maven-glassfish-plugin. Each plugin contains individual plug-in specific configuration. The name of the plugin should be sufficient to explain their primary purpose.
| <?xml version="1.0" encoding="UTF-8"?> | |
| <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> | |
| <modelVersion>4.0.0</modelVersion> | |
| <groupId>com.blogpost</groupId> | |
| <artifactId>HelloGlassFishMaven</artifactId> | |
| <version>1.0-SNAPSHOT</version> | |
| <packaging>war</packaging> | |
| <name>HelloGlassFishMaven</name> | |
| <properties> | |
| <!-- Input Parameter - GlassFish properties file --> | |
| <glassfish.properties.file.argument></glassfish.properties.file.argument> | |
| <endorsed.dir>${project.build.directory}/endorsed</endorsed.dir> | |
| <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding> | |
| </properties> | |
| <dependencies> | |
| <dependency> | |
| <groupId>junit</groupId> | |
| <artifactId>junit</artifactId> | |
| <version>4.11</version> | |
| </dependency> | |
| <dependency> | |
| <groupId>com.sun.jersey</groupId> | |
| <artifactId>jersey-servlet</artifactId> | |
| <version>1.13</version> | |
| </dependency> | |
| <dependency> | |
| <groupId>javax</groupId> | |
| <artifactId>javaee-web-api</artifactId> | |
| <version>7.0</version> | |
| <scope>provided</scope> | |
| </dependency> | |
| </dependencies> | |
| <build> | |
| <plugins> | |
| <plugin> | |
| <groupId>org.apache.maven.plugins</groupId> | |
| <artifactId>maven-compiler-plugin</artifactId> | |
| <version>3.1</version> | |
| <configuration> | |
| <source>1.7</source> | |
| <target>1.7</target> | |
| <compilerArguments> | |
| <endorseddirs>${endorsed.dir}</endorseddirs> | |
| </compilerArguments> | |
| </configuration> | |
| </plugin> | |
| <plugin> | |
| <groupId>org.apache.maven.plugins</groupId> | |
| <artifactId>maven-war-plugin</artifactId> | |
| <version>2.3</version> | |
| <configuration> | |
| <failOnMissingWebXml>false</failOnMissingWebXml> | |
| <filteringDeploymentDescriptors>true</filteringDeploymentDescriptors> | |
| <webresources> | |
| <resource> | |
| <directory>${basedir}/src/main/webapp/WEB-INF</directory> | |
| <filtering>true</filtering> | |
| <targetpath>WEB-INF</targetpath> | |
| <includes> | |
| <include>**/glassfish-web.xml</include> | |
| </includes> | |
| </resource> | |
| </webresources> | |
| </configuration> | |
| </plugin> | |
| <plugin> | |
| <groupId>org.apache.maven.plugins</groupId> | |
| <artifactId>maven-dependency-plugin</artifactId> | |
| <version>2.6</version> | |
| <executions> | |
| <execution> | |
| <phase>validate</phase> | |
| <goals> | |
| <goal>copy</goal> | |
| </goals> | |
| <configuration> | |
| <outputDirectory>${endorsed.dir}</outputDirectory> | |
| <silent>true</silent> | |
| <artifactItems> | |
| <artifactItem> | |
| <groupId>javax</groupId> | |
| <artifactId>javaee-endorsed-api</artifactId> | |
| <version>7.0</version> | |
| <type>jar</type> | |
| </artifactItem> | |
| </artifactItems> | |
| </configuration> | |
| </execution> | |
| </executions> | |
| </plugin> | |
| <plugin> | |
| <groupId>org.codehaus.mojo</groupId> | |
| <artifactId>properties-maven-plugin</artifactId> | |
| <version>1.0-alpha-2</version> | |
| <configuration> | |
| <files> | |
| <file>${basedir}/properties/${glassfish.properties.file.argument}.properties</file> | |
| </files> | |
| </configuration> | |
| </plugin> | |
| <plugin> | |
| <groupId>org.glassfish.maven.plugin</groupId> | |
| <artifactId>maven-glassfish-plugin</artifactId> | |
| <version>2.1</version> | |
| <configuration> | |
| <glassfishDirectory>${GLASSFISH_HOME}</glassfishDirectory> | |
| <user>${glassfish.user}</user> | |
| <passwordFile>${basedir}/passwords/${glassfish.pwdfile}</passwordFile> | |
| <echo>true</echo> | |
| <debug>true</debug> | |
| <terse>true</terse> | |
| <domain> | |
| <name>${glassfish.domain}</name> | |
| <host>${glassfish.host}</host> | |
| <adminPort>${glassfish.adminport}</adminPort> | |
| </domain> | |
| <components> | |
| <component> | |
| <name>${project.artifactId}</name> | |
| <artifact>${project.build.directory}/${project.build.finalName}.war</artifact> | |
| </component> | |
| </components> | |
| </configuration> | |
| </plugin> | |
| </plugins> | |
| </build> | |
| </project> |
When complete, right-click on the project and do a ‘Build with Dependencies…’. Make sure everything builds. The final view of the project, with all its Maven-managed dependencies should look like the two screen grabs shown below. Make sure to commit all your new code to Git.
Final Projects Tab View of Project
Final Files Tab View of Project
Maven and Properties Files
In part 2, will be deploying our project to multiple GlassFish domains. Each domain’s configuration is different. We will use Java properties files to store each of the GlassFish domain’s configuration properties. The ability to use Java properties files with Maven is possible using the Mojo Project’s Properties Maven Plugin. I introduced this plugin in an earlier post, Build a Continuous Deployment System with Maven, Hudson, WebLogic Server, and JUnit.
Each environment (Development, Testing, Production), represented by a GlassFish domain, has a separate properties file in the project (see the Files Tab view above). The properties files contain configuration values the Maven GlassFish Plugin will need to deploy the project’s WAR file to each GlassFish domain. Since the build and deployment configurations are required by the project, including them into our Git repository and automating their use based on the environment, are two best practices.
| # contents of all three files shown here | |
| # development domain properties file | |
| glassfish.domain=development | |
| glassfish.host=glassfish4-app-server | |
| glassfish.adminport=5050 | |
| glassfish.user=admin | |
| glassfish.pwdfile=pwdfile_development | |
| # testing domain properties file | |
| glassfish.domain=testing | |
| glassfish.host=glassfish4-app-server | |
| glassfish.adminport=6060 | |
| glassfish.user=admin | |
| glassfish.pwdfile=pwdfile_testing | |
| # production domain properties file | |
| glassfish.domain=production | |
| glassfish.host=glassfish4-app-server | |
| glassfish.adminport=7070 | |
| glassfish.user=admin | |
| glassfish.pwdfile=pwdfile_production |
In our project’s particular workflow, Maven accepts a single argument (‘glassfish.properties.file.argument’), which represents the environment we want to deploy to, such as ‘development’. The property value tells Maven which properties file to read, such as ‘development.properties’. Maven replaces the keys in the pom file with the values from the ‘development.properties’ file.
The properties file also tells Maven the full path to the separate password file, containing the admin user password, such as ‘pwdfile_development’. In an actual production environment, we would store encrypted password files on a secured file path. For simplicity in our example, we have included them unencrypted, within the project’s main directory.
There are other Maven capabilities that also would achieve our deployment goals. For example, you might consider the Maven Release Plugin, as well as look at using Maven Build Profiles.
Testing the Pipeline
Although we have not built the second half of our deployment pipeline yet, we can still test the system at this early stage. All the necessary foundational elements are in place. To test the our system, right-click on the Maven Project icon in the Projects tab and select Custom -> Goals… Enter the following Maven Goals: ‘properties:read-project-properties clean install glassfish:redeploy -e’. In the Properties text box, enter the following: ‘glassfish.properties.file.argument=testing’ (see screen grab below). This will execute a number of Maven Goals and associated commands, visible in the Output tab.
With this one simple command, we are asking Maven to 1) read in our Java properties file and password file, 2) clean the project, 3) pull down all our project’s dependencies, 4) compile the project’s code, 5) execute the unit tests with JUnit, 6) assemble the WAR file, and 7) deploy it to the ‘testing’ GlassFish domain using asadmin. The terse nature of the command really demonstrates the power of Maven to manage our project and the deployment pipeline!
Run Maven within NetBeans to Test Pipeline
If successful you should see a message in the Output tab, indicating as much. Reviewing the contents of the Output tab will give you complete insight into the Maven process under the NetBeans hood. We used the ‘-e’ (echo) argument with Maven and the ‘Show Debug Output’ to further provide information to us about the process. The output contains all calls to Maven and subsequently to asadmin (GlassFish). You can learn a lot about using Maven and asadmin (GlassFish) by studying the Debug Output.
Conclusion
In the first part of this post, we learned how to create a simple Java EE web application project in NetBeans, using Maven. We learned how to integrate JUnit for automated testing, and how use Git to manage our source code.
In the second half of this post, we will learn how to configure Jenkins CI Server to retrieve our project from the remote Git repository, build, test, and assemble it into a WAR file. If these steps are successful, Jenkins will deploy our project to a GlassFish domain or multiple domains, based on the project’s stage in the software development life cycle. We will demonstrate how to automate Jenkins to achieve true continuous integration and continuous deployment.
Convert VS 2010 Database Project to SSDT and Automate Publishing with Jenkins – Part 3/3
Posted by Gary A. Stafford in .NET Development, Software Development, SQL Server Development, Team Foundation Server (TFS) Development on August 8, 2012
Objectives of 3-Part Series:
Part I: Setting up the Example Database and Visual Studio Projects
- Setup and configure a new instance of SQL Server 2008 R2
- Setup and configure a copy of Microsoft’s Adventure Works database
- Create and configure both a Visual Studio 2010 server project and Visual Studio 2010 database project
- Test the project’s ability to deploy changes to the database
Part II: Converting the Visual Studio 2010 Database and Server Projects to SSDT
- Convert the Adventure Works Visual Studio 2010 database and server projects to SSDT projects
- Create a second Solution configuration and SSDT publish profile for an additional database environment
- Test the converted database project’s ability to publish changes to multiple database environments
Part III: Automate the Building and Publishing of the SSDT Database Project Using Jenkins
- Automate the build and delivery of a sql change script artifact, for any database environment, to a designated release location using a parameterized build.
- Automate the build and publishing of the SSDT project’s changes directly to any database environment using a parameterized build.
Part III: Automate the Building and Publishing of the SSDT Database Project Using Jenkins
In this last post we will use Jenkins to publishing of changes from the Adventure Works SSDT database project to the Adventure Works database. Jenkins, formally Hudson, is the industry-standard, java-based open-source continuous integration server.
Jenkins
If you are unfamiliar with Jenkins, I recommend an earlier post, Automated Deployment to GlassFish Using Jenkins and Ant. That post goes into detail on Jenkins and its associated plug-in architecture. Jenkins’ website provides excellent resources for installing and configuring Jenkins on Windows. For this post, I’ll assume that you have Jenkins installed and running as a Windows Service.
The latest available version of Jenkins, at the time of this post is 1.476. To follow along with the post, you will need to install and configure the following (4) plug-ins:
- Apache Ant Plug-in
- Jenkins MSBuild Plug-in
- Jenkins Artifact Deployer Plug-in
- Jenkins Email Extension Plug-in
User Authentication
In the first two posts, we connected to the Adventure Works database with the ‘aw_dev’ SQL Server user account, using SQL Authentication. This account was used to perform schema comparisons and publish changes from the Visual Studio project. Although SQL Authentication is an acceptable means of accessing SQL Server, Windows Authentication is more common in corporate and enterprise software environments, especially where Microsoft’s Active Directory is used. Windows Authentication with Active Directory (AD) provides an easier, centralized user account security model. It is considered more secure.
With Windows Authentication, we associate a SQL Server Login with an existing Windows user account. The account may be local to the SQL Server or part of an Active Directory domain. For this post, instead using SQL Authentication, passing the ‘aw_dev’ user’s credentials to SQL Server in database project’s connection strings, we will switch to Windows Authentication. Using Windows Authentication will allow Jenkins to connect directly to SQL Server.
Setting up the Jenkins Windows User Account
Let’s outline the process of creating a Jenkins Windows user account and using Windows Authentication with our Adventure Works project:
- Create a new ‘jenkins’ Windows user account.
- Change the Jenkins Windows service Log On account to the ‘jenkins’ Windows account.
- Create a new ‘jenkins’ SQL Server Login, associated with the ‘jenkins’ Windows user account, using Windows Authentication.
- Provide privileges in SQL Server to the ‘jenkins’ user identical to the ‘aw_dev’ user.
- Change the connection strings in the publishing profiles to use Windows Authentication.
First, create the ‘jenkins’ Windows user account on the computer where you have SQL Server and Jenkins installed. If they are on separate computers, then you will need to install the account on both computers, or use Active Directory. For this demonstration, I have both SQL Server and Jenkins installed on the same computer. I gave the ‘jenkins’ user administrative-level rights on my machine, by assigning it to the Administrators group.
Create New Jenkins User
Next, change the ‘Log On’ Windows user account for the Jenkins Windows service to the ‘jenkins’ Windows user account. Restart the Jenkins Windows service to apply the change. If the service fails to restart, it is likely you did not give enough rights to the user. I suggest adding the user to the Administrators group, to check if the problem you have encountering is permissions-related.
Jenkins Windows Service
Set Log On Account for Jenkins Windows Service
Log On Account for Jenkins Windows Service
Log On Account for Jenkins Windows Service Granted
Setting up the Jenkins SQL Server Login
Next, to use Windows Authentication with SQL Server, create a new ‘jenkins’ Login for the Production instance of SQL Server and it with the ‘jenkins’ Windows user account. Replicate the ‘aw_dev’ SQL user’s various permissions for the ‘jenkins’ user. The ‘jenkins’ account will be performing similar tasks to ‘aw_dev’, but this time initiated by Jenkins, not Visual Studio. Repeat this process for the Development instance of SQL Server.
Jenkins Login Added to Development Instance
Jenkins User Any Definition on Production Instance
Jenkins User View Definition on Production Instance Database
SSMS View of Jenkins User
Windows Authentication with the Publishing Profile
In Visual Studio, switch the connection strings in the Development and Production publishing profiles in both the server project and database projects to Windows Authentication with Integrated Security. They should look similar to the code below. Substitute your server name and SQL instance for each profile.
Data Source=[SERVER NAME]\[INSTANCE NAME];Integrated Security=True;Pooling=False
Important note here, once you switch the profile’s connection string to Windows Authentication, the Windows user account that you logged into your computer with, is the account that Visual Studio will now user to connect to the database. Make sure your Windows user account has at least the same level of permissions as the ‘aw_dev’ and ‘jenkins’ accounts. As a developer, you would likely have greater permissions than these two accounts.
Configuring Jenkins for Delivery of Script to Release
In many production environments, delivering or ‘turning over’ release-ready code to another team for deployment, as opposed to deploying the code directly, is common practice. A developer starts or ‘kicks off a build’ of the job in Jenkins, which generates artifact(s). Artifacts are usually logical collections of deployable code and other associated components and files, constituting the application being built. Artifacts are often separated by type, such as database, web, Windows services, web services, configuration files, and so forth. Each type may be deployed by a different team or to a different location. Our project will only have one artifact to deliver, the sql change script.
This first Jenkins job we create will just generate the change script, which will then be delivered to a specific remote location for later release. We start by creating what Jenkins refers to as a parameterized build job. It allows us to pass parameters to each build of our job. We pass the name of the configuration (same as our environment name) we want our build to target. With this single parameter, ‘TARGET_ENVIRONMENT’, we can use a single Jenkins job to target any environment we have configured by simply passing its name to the build; a very powerful, time-saving feature of Jenkins.
Step 1 – Parameterized Build Parameter
Let’s outline the steps we will configure our Jenkins job with, to deliver a change script for release:
- Copy the Solution from its current location to the Jenkins job’s workspace.
- Accept the target environment as a parameterized build parameter (ex. ‘Production’ or ‘Development’).
- Build the database project and its dependencies based on the environment parameter.
- Generate the sql change script based on the environment parameter.
- Compress and name the sql change script based on the environment parameter and build id.
- Deliver the compressed script artifact to a designated release location for deployment.
- Notify release team that the artifact is ready for release.
- Archive the build’s artifact(s).
Copy the Solution to Jenkins
I am not using a revision control system, such as TFS or Subversion, for our example. The Adventure Works Solution resides in a file directory, on my development machine. To copy the entire Solution from its current location into job’s workspace, we add a step in the Jenkins job to execute a simple xcopy command. With source control, you would replace the xcopy step with a similar step to retrieve the project from a specific branch)within the revision control system, using one of many Jenkins’ revision control plug-ins.
Step 2 – Copy Solution to Jenkins Workspace
echo 'Copying Adventure Works Solution to Jenkins workspace...' xcopy "[Path to your Project]\AdventureWorks2008" "%WORKSPACE%" /S /E /H /Y /R /EXCLUDE:[Path to exclude file]\[name of exclude file].txt echo 'Deleting artifacts from previous builds...' del "%WORKSPACE%\*_publish.zip" /F /Q
Excluding Solution files from Jenkins job’s workspace that are unnecessary for the job to succeed is good practice. Excluding files saves time during the xcopy and can make troubleshooting build problems easier. To exclude unneeded Solution files, use the xcopy command’s ‘exclude’ parameter. To use exclude, we must first create an exclude text file, listing the directories we don’t need copied, and call it using with the exclude parameter with the xcopy command. Make sure to change the path shown above to reflect the location and name of your exclude file. Here is a list of the directories I chose to exclude. They are either unused by the build, or created as part of the build, for example the sql directories and there subdirectories.
\AdventureWorks2008\sql\ \AdventureWorks2008\Sandbox\ \AdventureWorks2008\_ConversionReport_Files\ \Development\sql\ \Development\Sandbox\ \Development\_ConversionReport_Files\
Build the Solution with Jenkins
Once the Solution’s files are copied into the Jenkins job’s workspace, we perform a build of the database project with an MSBuild build step, using the Jenkins MSBuild Plug-in. Jenkins executes the same MSBuild command Visual Studio would execute to build the project. Jenkins calls MSBuild, which in turn calls the MSBuild ‘Build’ target with parameters that specify the Solution configuration and platform to target.
Generate the Script with Jenkins
After Building the database project, in the same step as the build, we perform a publish of the database project. MSBuild calls the new SSDT’s ‘Publish’ target with parameters that specify the Solution configuration, target platform, publishing profile to use, and whether to only generate a sql change script, or publish the project’s changes directly to the database. In this first example, we are only generating a script. Note the use of the build parameter (%TARGET_ENVIRONMENT%) and environmental variables (%WORKSPACE%) in the MSBuild command. Again, a very powerful feature of Jenkins.
Step 3 – Build and Publish Project
"%WORKSPACE%\AdventureWorks2008\AdventureWorks2008.sqlproj" /p:Configuration=%TARGET_ENVIRONMENT% /p:Platform=AnyCPU /t:Build;Publish /p:SqlPublishProfilePath="%WORKSPACE%\AdventureWorks2008\%TARGET_ENVIRONMENT%.publish.xml" /p:UpdateDatabase=False
Compressing Artifacts with Apache Ant
To streamline the delivery, we will add a step to compress the change script using Jenkins Apache Ant Plug-in. Many consider Ant strictly a build tool for Java development. To the contrary, there are many tasks that can be automated for .NET developers with Ant. One particularly nice feature of Ant is its built-in support for zip compression.
Step 4 – Invoke Ant to Compress Artifact
configuration=$TARGET_ENVIRONMENT buildNo=$BUILD_NUMBER
The Ant plug-in calls Ant, which in turn calls an Ant buildfile, passing it the properties we give. First, create an Ant buildfile with a single task to zip the change script. To avoid confusion during release, Ant will also append the configuration name and unique Jenkins job build number to the filename. For example, ‘AdventureWorks.publish.sql’ becomes ‘AdventureWorks_Production_123_publish.zip’. This is accomplished by passing the configuration name (Jenkins parameterized build parameter) and the build number (Jenkins environmental variable), as parameters to the buildfile (shown above). The parameters, in the form of key-value-pairs, are treated as properties within the buildfile. Using Ant to zip and name the script literally took us one line of Ant code. The contents of the build.xml buildfile is shown below.
<?xml version="1.0" encoding="utf-8"?>
<project name="AdventureWorks2008" basedir="." default="default">
<description>SSDT Database Project Type ZIP Example</description>
<!-- Example configuration ant call with parameter:
ant -Dconfiguration=Development -DbuildNo=123 -->
<target name="default" description="ZIP sql deployment script">
<echo>$${basedir}=${basedir}</echo>
<echo>$${configuration}=${configuration}</echo>
<echo>$${buildNo}=${buildNo}</echo>
<zip basedir="AdventureWorks2008/sql/${configuration}"
destfile="AdventureWorks_${configuration}_${buildNo}_publish.zip"
includes="*.publish.sql" />
</target>
</project>
Delivery of Artifacts
Lastly, we add a step to deliver the zipped script artifact to a ‘release’ location. Ideally, another team would retrieve and execute the change script against the database. Delivering the artifact to a remote location is easily accomplished using the Jenkins Artifact Deployer Plug-in. First, if it doesn’t already exist, create the location where you will deliver the scripts. Then, ensure Jenkins has permission to manage the location’s contents. In this example, the ‘release’ location is a shared folder I created. In order for Jenkins to access the ‘release’ location, give the ‘jenkins’ Windows user Read/Write (Change) permissions to the shared folder. With the deployment plug-in, you also have the option to delete the previous artifact(s) each time there is a new deployment, or leave them to accumulate.
Sharing Folder for Released Artifacts
Jenkins User Permissions for Shared Folder
Permissions for Shared Folder
Step 5 – Deploy Artifact to Release Location
Multiple Zipped Artifacts in Release Folder
Email Notification
Lastly, we want to alert the right team that artifacts have been turned-over for release. There are many plug-ins Jenkins to communicate with end-users or other system. We will use the Jenkins Email Extension Plug-in to email the release team. Configuring dynamic messages to include the parameterized build parameters and Jenkins’ environmental variables is easy with this plug-in. My sample message includes several variables in the body of the message, including target environment, target database, artifact name, and Jenkins build URL.
I had some trouble passing the Jenkins’ parameterized build parameter (‘TARGET_ENVIRONMENT’) to the email plug-in, until I found this post. The format required by the plug-in for the type of variable is a bit obscure as compared to Ant, MSBuild, or other plug-ins.
Artifact: AdventureWorks_${ENV,var="TARGET_ENVIRONMENT"}_${BUILD_NUMBER}_publish.zip
Environment: ${ENV,var="TARGET_ENVIRONMENT"}
Database: AdventureWorks
Jenkins Build URL: ${BUILD_URL}
Please contact Development for questions or problems regarding this release.
Release Request Notification Email Message
Publishing Directly to the Database
As the last demonstration in this series of posts, we will publish the project changes directly to the database. Good news, we have done 95% of the work already. We merely need to copy the Jenkins job we already created, change one step, remove three others steps, and we’re publishing! Start by creating a new Jenkins job by copying the existing script delivery job. Next, drop the Invoke Ant, Artifact Deployer, and Archive Artifacts steps from the job’s configuration. Lastly, set the last parameter of the MSBuild task, ‘UpdateDatabase’, to True from False. That’s it! Instead of creating the script, compressing it, and sending it to a location to be executed later, the changes are generated and applied to the database in a single step.
Hybrid Solution
If you are not comfortable with the direct approach, there is a middle ground between only generating a script and publishing directly to the database. You can keep a record of the changes made to the database as part of publishing. To do so, change the ‘UpdateDatabase’ parameter to True, and only drop the Artifact Deployer step; leave the Invoke Ant and Archive Artifacts steps. The resulting job generates the change script, publishes the changes to the database, and compresses and archives the script. You now have a record of the changes made to the database.
Conclusion
In this last of three posts we demonstrated the use of Jenkins and its plug-ins to created three jobs, representing three possible SSDT publishing workflows. Using the parameterized build feature of Jenkins, each job capable of being executed against any database environment that we have a configuration and publishing profile defined for. Hopefully, one of these three workflows may fit your particular release methodology.
Jenkins SSDT Jobs View
Automated Deployment to GlassFish Using Jenkins CI Server and Apache Ant
Posted by Gary A. Stafford in DevOps, Enterprise Software Development, Java Development, Software Development on May 19, 2012
Use Jenkins and Apace Ant to compile, assemble, test, and deploy a RESTful web service to GlassFish. All source code for this post is available on GitHub. Note GitHub repo reflects updates to project on 10/31/2013.
Jenkins, formally Hudson, is the industry-standard, java-based open-source continuous integration server. According to their website, Jenkins provides over 400 plug-ins to support building and testing almost any type of project. According to Apache, Ant is a Java library and command-line tool whose mission is to drive processes described in build files as targets. This post demonstrates the use of Jenkins and Apache Ant to compile, assemble, unit test, and deploy a Java EE 6 RESTful web service to Oracle’s GlassFish open-source application server.
For the sake of brevity, I have chosen to use the HelloWorld RESTful web service example included with NetBeans. I will use NetBeans to create the project, write the unit tests, and produce an Ant target in the build file. I will not delve deeply into the inner workings of the web service itself since the focus of this post is automation.
System Configuration
This post assumes that you have current versions of NetBeans, JUnit, Jenkins, GlassFish, Ant and Java installed and configured on your Microsoft Windows-based computer. A full installation of NetBeans comes with JUnit, Ant, and GlassFish. At the time of the original post, I was using NetBeans 7.1.2, GlassFish 3.1.2, Jenkins 1.4.6.3, Ant 1.8.3, and JDK 1.7.0_02.
For simplicity, I am using a single development machine for this demonstration, on which all applications are installed. In a true production environment you would most likely have a distributed configuration with GlassFish installed on an application server, Jenkins on a build server, and NetBeans on your development machine. Also, for this post, I am also not using a source-code management (SCM) system, also called a version control system (VCS), such as Subversion or Mercurial, to house the project’s source code. Again, in a production environment, your source-code would be placed on SCM/VCS server.
Both GlassFish and Jenkins are configured by default to run on server port 8080. Since I have both applications installed on the same machine, I have changed Jenkins’ default port to another unused port, 9090. Changing Jenkins’ port is easy to do. If you don’t know how, consult this post or similar.
NetBeans
First, create a new project in NetBeans, by selecting the New Project -> Samples -> Java Web Services, REST: Hello World (Java EE 6), as shown below. Rename the project to HelloGlassFish. When complete, the project, in the Projects tab, should look like the screen-grab, below.
New Project View in NetBeans
JUnit
Next create a unit-test using JUnit, the open-source unit-testing framework. Jenkins will eventually run this test each time the project is built. Creating unit-tests is easy in NetBeans. Select the ‘NameStorageBean.java’ class object, right-click, and select Tools -> Create JUnit Tests… This will create a default ‘NameStorageBeanTest.java’ class object in a new, ‘Test Packages’ directory. Overwrite NameStorageBeanTest.java contents with the follows code. This will create a single unit test we can use to demonstrate JUnit’s integration with Jenkins. You will also notice new test objects in the Project tab.
| package helloworld; | |
| import javax.ejb.embeddable.EJBContainer; | |
| import javax.naming.NamingException; | |
| import org.junit.*; | |
| import static org.junit.Assert.assertEquals; | |
| /** | |
| * | |
| * @author Gary A. Stafford | |
| */ | |
| public class NameStorageBeanTest { | |
| private NameStorageBean instance = null; | |
| private EJBContainer container = null; | |
| public NameStorageBeanTest() { | |
| } | |
| @BeforeClass | |
| public static void setUpClass() throws Exception { | |
| } | |
| @AfterClass | |
| public static void tearDownClass() throws Exception { | |
| } | |
| @Before | |
| public void setUp() throws NamingException { | |
| container = javax.ejb.embeddable.EJBContainer.createEJBContainer(); | |
| instance = (NameStorageBean) container.getContext(). | |
| lookup("java:global/classes/NameStorageBean"); | |
| } | |
| @After | |
| public void tearDown() { | |
| } | |
| /** | |
| * Test of getName method, of class NameStorageBean. | |
| */ | |
| @Test | |
| public void testGetName() throws Exception { | |
| System.out.println("getName"); | |
| String expResult = "Test"; | |
| instance.setName(expResult); | |
| String result = instance.getName(); | |
| assertEquals(expResult, result); | |
| container.close(); | |
| } | |
| } |
Build the project and run the ‘testGetName’ unit-test to make sure it works correctly and the test passes.
Apache Ant
Next, change to the Files tab. Open the ‘build.xml’ file, as shown below. Also, for later reference, note the contents of the ‘HelloGlassFish.war’ and the location of the ‘pwdfile_domain1’ password file.
Place the following Ant target, entitled ‘jenkins-glassfish-deploy’, into the build.xml file, between the end of the commented section and the closing <project/> tag, as shown below.
| <!-- Older style. Not what is in repo on GitHub. --> | |
| <target name="jenkins-glassfish-deploy" | |
| description="Clean, build, test and deploy application to GlassFish"> | |
| <antcall target="clean"> | |
| <antcall target="default"> | |
| <antcall target="test"> | |
| <exec failonerror="true" executable="cmd" description="Deploy to GlassFish"> | |
| <arg value="/c" /> | |
| <arg value="asadmin --echo=true --host=localhost --port=4848 --user=admin | |
| --passwordfile=pwdfile_domain1 --secure=false | |
| deploy --force=true --name=HelloGlassFish --contextroot=/HelloGlassFish dist\HelloGlassFish.war" /> | |
| </exec> | |
| </target> |
This is the Ant target Jenkins will use to build, test, and deploy the project. The primary ‘jenkins-glassfish-deploy’ target calls three Ant targets using the
antcall element. They include clean, default, and test. Each of these Ant targets has dependencies on other Ant targets, which in turn depend on yet other targets – a dependency tree. For example, default depends on dist and javadoc. The test target depends on other targets to build the .war file. If you are not using test to execute unit tests, you can call the test target to build the .war file.
The last part of the ‘jenkins-glassfish-deploy’ target is a little different. It’s an exec (execute) element, which calls asadmin to deploy the project to GlassFish with a series of GlassFish domain-specific parameters. These parameters include the GlassFish domain’s URL and port, the domain’s administrative user and password account info (found in a password file), the location of the .war file to deploy, and destination of the .war within GlassFish. Calling asadmin deploy gives you fine control over the details of how the project is deployed to GlassFish.
The password file, referenced in the target is a simple text file, which stores the password for the user account used to execute the asadmin deploy call. The contents of the file look like:
AS_ADMIN_PASSWORD=Your_Password_Here
This target could be simplified with the depends attribute. Instead of the three antcall elements, you could simply add depends="clean, default, test" to the target element:
| <!-- Older style. Not what is in repo on GitHub. --> | |
| <target name="jenkins-glassfish-deploy-updated" depends="clean, default, test" | |
| description="Clean, build, test and deploy application to GlassFish"> | |
| <exec failonerror="true" executable="cmd" description="Deploy to GlassFish"> | |
| <arg value="/c" /> | |
| <arg value="asadmin --echo=true --host=localhost --port=4848 --user=admin | |
| --passwordfile=pwdfile_domain1 --secure=false | |
| deploy --force=true --name=HelloGlassFish --contextroot=/HelloGlassFish dist\HelloGlassFish.war" /> | |
| </exec> | |
| </target> |
According to Oracle, the asadmin utility is used to perform any administrative tasks for GlassFish from the command line. You can use this
asadmin utility in place of using the GlassFish Administrator interface. I am able to call asadmin directly because I have added the path to asadmin.bat to the Windows’ environmental variable, PATH. The asadmin.bat file is in the GlassFish bin directory, similar to ‘C:\Program Files\glassfish-3.1.2\glassfish\bin\’.
Jenkins
Switching to Jenkins, create a new Job named HelloGlassFish. In the HelloGlassFish configuration, we need to add two Build steps and one post-build Action. For the first Build step, since we are not using SCM, we will copy the files from the project in the NetBeans workspace to the Jenkins workspace. To do this, add an ‘Execute Windows batch command’ action with code similar to code snippet below, but substituting your own project’s file path. Note, you can substitute the %WORKSPACE% environmental variable for the xcopy destination (see call-out 1 in the below screen-grab). This variable represents the absolute path of the directory assigned to the build as a workspace, according to Jenkins. Jenkins offers many useful variables, accessible to Windows batch scripts.
xcopy "C:\Users\gstaffor\Documents\NetBeansProjects\HelloGlassFish\HelloGlassFish" "%WORKSPACE%" /s /e /h /y
Next, add the second Build task, ‘Invoke Ant’. I assume you already have Ant configured for Jenkins. In the ‘Target’s text box, enter the Ant target we created in NetBeans build.xml file, entitled ‘jenkins-glassfish-deploy’ (see call-out 2 in the below screen-grab). If the name of your build file is anything other than the default ‘build.xml’, you will need to enter the Ant file name.
Lastly, add the single Post-build Action, ‘Publish JUnit test result report’. This will show us a visual representation of the results of our project’s unit-tests. Input the relative path to your reports from the workspace root. The path should be similar to call-out 3 in the screen-grab, below.
When complete, the HelloGlassFish Job’s configuration should resemble the screen-grab, below.
Jenkins HelloGlassFish Project Configuration
Save and close the configuration. Build the HelloGlassFish Job in Jenkins and make sure it succeeds with error.
GlassFish
Open GlassFish’s browser-based Domain Admin Console, usually on server port 4848, by default. On the left-hand side of the main window, under ‘Common Tasks’, tip the ‘Applications’ node. You should see the HelloGlassFish application is now deployed to GlassFish. You don’t have to do anything in GlassFish, Jenkins and Ant has taken care of everything.
GlassFish’s browser-based Domain Admin Console
To view the HelloGlassFish application, open a new browser window and direct it to ‘http://localhost:8080/HelloGlassFish/resources/helloWorld’. You should see a ‘Hello World!’ message displayed in your browser’s window. Note, since we only changed the name of the default HelloWorld NetBeans sample project to HelloGlassFish, not the web service’s URI, ‘helloWorld’ is still a required part of the URL path.
Redeploying the Project
Lastly, let’s demonstrate how easily changes to our project can be re-complied, re-tested, and re-deployed to GlassFish by Jenkins and Ant. Return to the HelloGlassFish project in NetBeans and open the NameStorageBean.java class. Change the value of the ‘name’ field from ‘World’ to ‘GlassFish’ and save the changes. Don’t build or do anything else in NetBeans. Instead, return to Jenkins and build the HelloGlassFish Job, again.
Change the NameStorageBean name Field
When the Job has finished building, re-direct your browser back to ‘http://localhost:8080/HelloGlassFish/resources/helloWorld’. You should now see a ‘Hello GlassFish!’ message displayed in your browser’s window instead of the earlier message, ‘Hello World!’. Jenkins has called the Ant target, which in turn re-compiled, re-tested, and re-deployed the modified HelloGlassFish application to GlassFish.
HelloGlassFish RESTful Web Service Demo
Helpful Links
- Apache Ant: ant.apache.org
- Apache Ant Tutorial: www.vogella.com/articles/ApacheAnt/article.html
- GlassFish: glassfish.java.net
- GlassFish asadmin deploy sub-command: docs.oracle.com/cd/E26576_01/doc.312/e24938/deploy.htm#deploy-1
- Jenkins: jenkins-ci.org
- NetBeans: netbeans.org
- Windows command line xcopy command: commandwindows.com/xcopy.htm
- Modifying PATH Environment Variable: www.java.com/en/download/help/path.xml
Gary Stafford
AWS Area Principal Solutions Architect | 10x AWS Certified Pro | DevOps | Data/ML | Serverless | Polyglot Developer | Former ThoughtWorks and Accenture
Programmatic Ponderings 