Posts Tagged Deploy

Build Automation – Calling GlassFish’s asadmin and Apache Ant Directly

Automating deployment of applications from NetBeans to GlassFish is easy using Apache Ant and GlassFish’s asadmin utility. Calling these two applications directly, without requiring the complete file path, can be a real time-savings. With Ubuntu (Linux), like with Windows OS, this can be done by adding their file paths to the $PATH environment variable.

Below is an example of adding both asadmin and Ant to the .bashrc file in your home directory. To open the .bashrc file, open the Terminal and enter ‘sudo gedit ~/.bashrc‘. You will be prompted for your password. When the .bashrc file opens, enter the following text at the end of the .bashrc file. Make sure you change the file paths to match your local system if they are different.

export ANT_HOME=./netbeans-7.2/java/ant
export ASADMIN_HOME=./glassfish-3.1.2.2/glassfish
export PATH=$PATH:$ASADMIN_HOME/bin:$ANT_HOME/bin

Close the .bashrc file and type ‘asadmin’ at the Terminal window prompt. You should see the response below. Type ‘exit’ to get out of asadmin. Next, type ‘ant’. Again, you should see the response below. This means both applications are now available directly, on any file path or from within any application, like Jenkins or Hudson.

Adding GlassFish's asadmin and Apache Ant to $Path Environmental Variable

Adding GlassFish’s asadmin and Apache Ant to $Path Environmental Variable

You can also add these variables in other ways. Here are links to other posts, which go into much more detail, and show methods to add these for all users, in addition to just yourself:

, , , , , , , ,

Leave a comment

Discover All Properties Available to an Apache Ant Target

Ever waste time searching for a certain property you need to build an Ant target? Here’s a quick tip to save you some time – echoproperties. According to The Apache Ant Project website, the echoproperties task ” displays all the current properties (or a subset of them specified by a nested <propertyset>) in the project. The output can be sent to a file if desired. This task can be used as a somewhat contrived means of returning data from an <ant> invocation, but is really for debugging build files.”

Recently, I was working on a new Java Web Application Project in NetBeans IDE 7.2.1. I wanted to build an Ant target to automate the deployment of the project’s .war file to GlassFish. To do so, I needed to identify properties that could return 1) the project’s name, 2) the path to the project’s .war file, and 3) the path to GlassFish’s asadmin utility. Calling the echoproperties task from within the Ant target below, from within my open project, returned a list of over 90 property key/value pairs.

<target name="list-all-properties">
    <echoproperties />
</target>

Although the results were enlightening, I couldn’t find the properties I was hoping to reference in the new target. Next however, I ran the Ant target again, adding the two dependency targets my GlassFish deployment target was going to need, clean and dist.

<target name="list-all-properties" depends="clean, dist">
    <echoproperties />
</target>

Running the revised target returned almost 450 properties, all available to Ant. The new properties were a result of the clean and dist targets running before the call to echoproperties. Those target’s properties were now also available. Here is a snippet of the results:

...
ant.project.invoked-targets=list-all-properties
ant.project.name=MySqlEntityWebDemo
ant.version=Apache Ant(TM) version 1.8.3 compiled on February 26 2012
ap.cmd.line.internal=
ap.proc.none.internal=
ap.processors.internal=
ap.supported.internal=true
application.args.param=
awt.toolkit=sun.awt.X11.XToolkit
basedir=/home/gstaffor/NetBeansProjects/MySqlEntityWebDemo
build.classes.dir=build/web/WEB-INF/classes
build.classes.excludes=**/*.java,**/*.form
build.compiler.emacs=true
build.dir=build
build.dir.to.clean=build/web
build.generated.dir=build/generated
build.generated.sources.dir=build/generated-sources
build.meta.inf.dir=build/web/META-INF
build.test.classes.dir=build/test/classes
build.test.results.dir=build/test/results
build.web.dir=build/web
build.web.excludes=**/*.java,**/*.form
client.urlPart=
compile.jsps=false
conf.dir=src/conf
debug-args-line=-Xdebug
debug-transport=dt_socket
debug-transport-by-os=dt_socket
debug.classpath=build/web/WEB-INF/classes\:/home/gstaffor/JavaFiles/eclipselink_2_4_1/jlib/eclipselink.jar...
debug.test.classpath=/home/gstaffor/JavaFiles/eclipselink_2_4_1/jlib/eclipselink.jar...
default.javac.source=1.7
default.javac.target=1.7
deploy.ant.properties.file=/home/gstaffor/.netbeans/7.2/gfv3-430621021.properties
display.browser=true
dist.dir=dist
dist.ear.war=dist/MySqlEntityWebDemo.war
dist.jar.dir=/home/gstaffor/NetBeansProjects/MySqlEntityWebDemo/dist
dist.javadoc.dir=dist/javadoc
dist.war=dist/MySqlEntityWebDemo.war
...
j2ee.compile.on.save=true
j2ee.copy.static.files.on.save=true
j2ee.deploy.on.save=true
j2ee.platform=1.6-web
j2ee.platform.classpath=/home/gstaffor/glassfish-3.1.2.2/glassfish/modules/bean-validator.jar...
j2ee.platform.embeddableejb.classpath=/home/gstaffor/glassfish-3.1.2.2/glassfish/lib/embedded/glassfish-embedded-static-shell.jar
j2ee.platform.is.jsr109=true
j2ee.platform.wscompile.classpath=/home/gstaffor/glassfish-3.1.2.2/glassfish/modules/webservices-osgi.jar...
j2ee.platform.wsit.classpath=
j2ee.server.domain=/home/gstaffor/glassfish-3.1.2.2/glassfish/domains/domain1
j2ee.server.home=/home/gstaffor/glassfish-3.1.2.2/glassfish
j2ee.server.instance=[/home/gstaffor/glassfish-3.1.2.2/glassfish...
j2ee.server.middleware=/home/gstaffor/glassfish-3.1.2.2
j2ee.server.type=gfv3ee6
jar.compress=false
...
war.content.additional=
war.ear.name=MySqlEntityWebDemo.war
war.name=MySqlEntityWebDemo.war
web.docbase.dir=web
webinf.dir=web/WEB-INF

Reviewing the results, I was able to find all the properties I needed to build the target, below.

<target name="glassfish-deploy" depends="clean, dist"
        description="Build distribution (WAR) and deploy to GlassFish">               
    <exec failonerror="true" vmlauncher="false" 
          executable="${j2ee.server.home}/bin/asadmin" >
        <arg line="--host=localhost --port=4848 
            --user=admin --passwordfile=pwdfile --secure=false
            deploy --force=true --name='${ant.project.name}' 
            --contextroot='/${ant.project.name}' '${dist.war}'" />
    </exec>
</target>

Almost any properties you need to develop an Ant Target is probably available if you know where, or how to look.

, , , , , , , , , , , , ,

Leave a comment

Connecting Java EE RESTful Web Services to Microsoft SQL Server Using NetBeans and GlassFish

Connecting Java EE RESTful web services, hosted on GlassFish, to Microsoft SQL Server – a high level overview. Demonstrate the creation of a Web Application project in NetBeans, including a SQL Server data source, entity classes from a SQL database, and RESTful web services. Show how to test and deploy the project to GlassFish.

Introduction

In a previous post, Connecting Java EE to SQL Server with Microsoft’s JDBC Driver 4.0, I demonstrated how Microsoft’s JDBC Driver 4.0 can connect Java-based RESTful web services to Microsoft SQL Server. In a more recent post, Calling Microsoft SQL Server Stored Procedures from a Java Application Using JDBC, I demonstrated the use of JDBC to call stored procedures from a Java application. In this post, I am going to offer a high-level, end-to-end overview on how to create and connect Java EE RESTful web services, hosted on GlassFish, to SQL Server. The goals of post are:

  • Demonstrate the use of Microsoft’s JDBC Driver 4.0 to connect a Java-based application to SQL Server 2008 R2;
  • Demonstrate the use of NetBeans to:
    • Create a SQL Server Data Source;
    • Create entity classes from the SQL Server database using the SQL Server Data Source;
    • Create RESTful web services using JAX-RS, which communicate with database, through the entity classes;
    • Compile and deploy the data source, entities, and services to GlassFish;
  • Test the RESTful Web Services locally in NetBeans, and once deployed, in GlassFish.

Setting up the Post’s Example

To demonstrate the example in this post, I have the follow applications installed, configured, and running in my development environment:

If you have any questions about installing and configuring the Adventure Works database, please refer to my post, Convert VS 2010 Database Project to SSDT and Automate Publishing with Jenkins – Part 1/3. The post takes you through creating a SQL Server 2008 R2 instance (entitled ‘Development’), installing the Adventure Works database, and creating a database user (‘aw-dev’). Also, refer to my earlier post, Connecting Java EE to SQL Server with Microsoft’s JDBC Driver 4.0, for directions on installing the Microsoft JDBC driver in the lib directory of GlassFish. This is necessary before following along with this demonstration.

Note, since I have NetBeans, SQL Server, and GlassFish all installed on a single computer, the URLs in several of the screen-grabs switch between ‘localhost’ and my computer’s actual name. Both references are interchangeable.

The Demonstration

Here are the high-level steps I will walk-through in this post:

  1. Confirm the SQL Server instance, database, and user are functioning properly;
  2. Create a new Web Application project in NetBeans;
  3. Create the SQL Server data source in the project;
  4. Create entity classes from the SQL Server database;
  5. Create RESTful web services using the entity classes;
  6. Test the web services locally in NetBeans;
  7. Build and deploy the project to GlassFish;
  8. Test the web services on GlassFish.

SQL Server

Using Microsoft’s SQL Server Management Studio, Quest’s Toad for SQL, or similar IDE, confirm the ‘Development’ instance (or whatever you have named your instance) of SQL Server 2008 R2 is running. Confirm the Adventure Works database is installed in that instance. Lastly, confirm the ‘aw_dev’ user can connect to the Adventure Works database and view and interact with all the database objects. Confirming these items will elevate many problems you might otherwise encounter when creating the data source, next.

SQL Server 2008 R2 AdventureWorks Database

View of the SQL Server Instance from SSMS. Note the : 1) Server, Instance, User, 2) Database, and 3) Schema

Create New Web Application Project

Create a new Java Web Application project in NetBeans. Name the project whatever you would like; I named mine ‘JdbcSqlWebSrvTest’. Select the correct GlassFish server instance and GlassFish domain where the project will be deployed. I am deploying my project to the default ‘domain1’ domain.

Creating a New Java Web Application Project in NetBeans - 02

Choose the Web Application Project-Type in NetBeans

Creating a New Java Web Application Project in NetBeans - 03

Provide a Project Name and Location

Creating a New Java Web Application Project in NetBeans - 04

Select the Target GlassFish Server and Domain Where the Project Will be Installed

Creating a New Java Web Application Project in NetBeans - 05

Include any Frameworks You Will Use with the Project

Creating a New Java Web Application Project in NetBeans - 06

View of the New Web Application Project in NetBeans

Create SQL Server Data Source and Entity Classes from the Database

Right-click on the project again and select ‘New’ -> ‘Other…’. From the list of Categories, select ‘Persistence’. From the list of Persistence choices, choose ‘Entity Classes from Database’. Click Next.

Creating New Entity Classes from a Database - 01

Choose ‘Entity Classes from Database’

Before we can choose which database tables we want from the Adventure Works database to create entity classes, we must create a connection to the database – a SQL Server Data Source. Click on the Data Source drop down and select ‘New Data Source…’. Give a Java Naming and Directory Interface (JNDI) name for the data source. I called mine ‘AdventureWorks’. Click on the ‘Database Connection’ dropdown, select ‘New Database Connection…’.

Creating New Entity Classes from a Database - 02

Database Tables View Before Data Source is Created

Creating a New Datasource for Entity Classes from a Database - 01

Name the Data Source

This starts the ‘New Connection Wizard’. The first screen, ‘Locate Driver’, is where we point the will instruct NetBeans to use the Microsoft JDBC Driver 4.0 for SQL Server Driver. Locate the sqljdbc4.jar file.

Creating a New Datasource for Entity Classes from a Database - 02

Locate the Microsoft JDBC Driver 4.0 for SQL Server Driver .jar File

On the next screen, ‘Customize the Connection’, input the required SQL Server information. The host is the machine your instance of SQL Server is installed on, such as ‘localhost’. The instance is the name of the SQL Server instance in which the Adventure Works database is installed, such as ‘Development’. Once you complete the form, click ‘Test Connection’. If it doesn’t succeed, check your settings, again.

Creating a New Datasource for Entity Classes from a Database - 03

Provide the SQL Server Adventure Works Database Connection Information

As I mentioned in an earlier post, the SQL Server Data Source forces you to select a single database schema. On the ‘Choose Database Schema’ screen, select the ‘HumanResources’ schema. The database tables you will be able to reference from you entity classes are limited to just this schema, when using this data source. To reference other schemas, you will need to create more data sources.

Creating a New Datasource for Entity Classes from a Database - 04

Select the ‘HumanResources’ Database Schema

Creating a New Datasource for Entity Classes from a Database - 05

New Data Source Complete with JNDI Name and Database Connection Created

Back in the ‘New Entity Classes from Database’ window, you will now have the ‘AdventureWorks’ data source selected as the Data Source. After a few seconds of processing, all ‘Available Tables’ within the ‘HumanResources’ schema are displayed. Choose the four tables shown in the screen-grab, below. Actually, two are database tables and two are virtual tables, called database ‘views’. We will need to do an extra step later on, to use the two entity classes built from the database views.

Creating New Entity Classes from a Database - 03A

Retrieving all the ‘HumanResources’ Schema Database Tables and Views

Creating New Entity Classes from a Database - 03B

All the ‘HumanResources’ Schema Database Tables and Views Available

Creating New Entity Classes from a Database - 04

Selecting the Database Views Throws a Warning Regarding no Entity IDs (Primary Keys)

On the next screen, ‘Entity Classes’, in the ‘New Entity Classes from Database’ window, give a package name to place the individual entity classes into. I chose to call mine ‘entityclasses’.

Creating New Entity Classes from a Database - 05

Provide the Name of the Package Where the Entity Classes Will be Created

On the next screen, ‘Mapping Options’, choose ‘Fully Qualified Database Table Names’. Without this option selected, I have had problems trying to make the RESTful web services function properly. This is also the reason I chose to create the entity classes first, and then create the RESTful web services, separately. NetBeans has an option that combines these two tasks into a single step, by choosing ‘RESTful Web Services from Database’. However, the ‘Fully Qualified Database Table Names’ option is not available on the equivalent screen, using that process (at least in my version of NetBeans 7.2). I prefer the two-step approach.

Creating New Entity Classes from a Database - 06

Select the ‘Fully Qualified Database Table Names’ Option

Click finished. You have successfully created the SQL Server data source and entity classes.

The data source we created, which will be deployed to GlassFish, is referred to as a JDBC Resource and a JDBC Connection Pool. This JDBC information is stored in the ‘glassfish-resources.xml’ file by NetBeans.

New Glassfish JDBC Connection Pool and Resource - Source (glassfish-resources.xml)

New GlassFish JDBC Connection Pool and JDBC Resource Contained in the glassfish-resources.xml File

The JPA Persistence Unit is found in the ‘persistence.xml’ file in the ‘Configuration Files’ folder. This file describes the Persistence Unit (PU). The PU serves to register the project’s four persistable entity classes, which are referred to by JPA as managed classes.

JPA Configuration - Design View (persistence.xml)

persistence.xml Design View

JPA Configuration - Source (persistence.xml)

persistence.xml Source View

RESTful Web Services from Entity Classes

Now that have a SQL Server Data Source and our entity classes, we will create the RESTful web services. Right-click on the project and select ‘New’ -> ‘Other…’ -> ‘Persistence’ -> ‘RESTful Web Services from ‘Entity Classes’. You will see a list of four entity classes we just created, from which to choose. Add all four entity classes.

Creating New RESTful Web Service from Entity Classes - 01

Creating the New RESTful Web Service from Four Entity Classes

Creating New RESTful Web Service from Entity Classes - 02

Add the Four Entity Classes

On the next screen, give a name for Resource Package to store the service classes in; I called mine ‘service’. That’s it; you now have four RESTful web services and the corresponding Enterprise Beans and Façade service classes. The service class sits between the RESTful web service and the entity class.

Creating New RESTful Web Service from Entity Classes - 03

Provide a Package Location to Place the RESTful Web Service Classes Into

Click finished. You have successfully created the RESTful web services.

Web Application Project with Entities and Services Added - 01

View of Web Application Project with Entity Classes and RESTful Web Services Added

Web Application Project with Entities and Services Added - 02

View of Web Application Project with Entity Classes and RESTful Web Services Added

Adding a Primary Key to Entity Classes

If you recall, I mentioned a problem with the two entity classes we created from the database views. To avoid an error when you build and deploy your project to GlassFish, we need to make a small change to the VEmployee.java and VEmployeeDepartment.java entity classes. Entity classes need a unique identifier, a primary key (or, Entity ID) identified. Since these two entity classes are built from database views, as opposed to database tables, they lack a primary key. To fix this, annotate the ‘businessEntityID’ field with ‘@Id’ in each class. This indicates that ‘businessEntityID’ is the primary key (Entity ID) for this class. The field, ‘businessEntityID’, must contain unique values, for this to work properly. NetBeans will make the suggested correction for you, if you allow it.

Fix Entity Id Error in View Entity Classes - 01

NetBeans Highlights the Entity Id Error in Two View-Based Entity Classes

Fix Entity Id Error in View Entity Classes - 02

Select the ‘businessEntityID’ Field as the Primary Key

Fix Entity Id Error in View Entity Classes - 03

View of Entity Class with ‘@Id’ Annotation Added

Test RESTful Web Services Locally in NetBeans

NetBeans provides an easy way to test the RESTful web services, locally. Right-click on the ‘RESTful Web Services’ project folder within the main project, and select ‘Test RESTful Web Services’. Select the first option, ‘Locally Generated Test Client’, in the ‘Configure REST Test Client’ pop-up window.

Test RESTful Web Service in NetBeans - 01

Choose the Local Test Client Option

NetBeans will open a web browser window and displays the RESTful URI (Universal Resource Identifier) for the services in a tree structure. There are four primary URIs, corresponding to the four services. Each primary URI has additional child URIs grouped under them. Are they URIs or URLs? I found this excellent post that does a very good job explaining the difference between the URL (how to get there) and the URI (the resource), which is part of the URL.

Test RESTful Web Service in NetBeans - 02

You May Get an ActiveX Warning When Using IE to Test the RESTful Web Services

Test RESTful Web Service in NetBeans - 03

View of all the RESTful Web Services Universal Resource Identifiers (URIs)

Click on the ‘entityclasses.employee’ URI. Choose the HTTP ‘GET()’ request method from the drop-down and click the ‘Test’ button. The service should return a status of ‘200 (OK)’, along with xml output containing information on all the Adventure Works employees. Congratulation, the RESTful web services have just returned data to your browser from the SQL Server Adventure Works database, using the entity classes and data source you created.

Test RESTful Web Service in NetBeans - 04

All Employees Being Successfully Retrieved from the Adventure Works Database

Click on the other URIs to familiarize yourself with the various default resources. Test the employee ‘from/to’ URI by inputting two parameters, test the ‘count’ URI, and try changing the MIME type where applicable from XML to JSON and observe the results.

Test RESTful Web Service in NetBeans - 05

A Single Employee Being Successfully Retrieved from the Adventure Works Database Using Input Parameter

Test RESTful Web Service in NetBeans - 06

Count of All Employees Being Successfully Retrieved from the Adventure Works Database

WADL

Note the link in the upper right corner of the above screens, labeled WADL: ‘http://[your_server_path]/JdbcSqlWebSrvTest/webresources/application.wadl’

The WADL (Web Application Description Language) file is the machine-readable XML description of the RESTful web service(s). The WADL file is to RESTful web services, as the WSDL (Web Service Definition Language) file is to non-RESTful, SOA-/SOAP-oriented web services. The WADL provides all the information you need to understand to the various RESTful web service’s resources, and how to call them using their URIs. According to Wikipedia, in the WADL file, ‘the service is described using a set of resource elements. Each resource has param elements to describe the inputs, and method elements which describe the request and response of a resource. The request element specifies how to represent the input, what types are required and any specific HTTP headers that are required. The response describes the representation of the service’s response, as well as any fault information, to deal with errors.’ You can download the WADL file (application.wadl), and review it in an XML-friendly viewer such as Notepad++.

View of WADL - application.wadl

View of the RESTful Web Services’ WADL File – application.wadl

Deploy Project to GlassFish

Now that the RESTful web services are working properly from within NetBeans, we can deploy them to GlassFish. To deploy the project to GlassFish, right-click on the main project icon in the Projects tab and select ‘Clean and Build’. Once the project builds successfully, right-click again and select ‘Deploy’. This will instruct Apache Ant to deploy the project as a .war file to GlassFish, using the project’s default Ant deploy task. The SQL Server data source will also be installed into GlassFish.

Once the deployment is complete, switch to GlassFish and refresh the home page if necessary. Under the ‘Applications’ item on the left-hand navigation menu, you should see a new application with the same name as your project, ‘JdbcSqlWebSrvTest’.

GlassFish 01 - Prior to Deploying New Application

Default View of GlassFish Domain Prior to Deploying the New Web Application

GlassFish 02 - New Application Deployed to GlassFish

New Web Application and JDBC Resource and Pool Deployed Successfully to GlassFish

Also, under the ‘JDBC’ -> ‘JDBC Resources’ item, you should see a resource with the same name as the data source you created in NetBeans, ‘AdventureWorks’. Under the ‘JDBC’ -> ‘JDBC Connection Pools’, you should see a pool entitled ‘microsoft_sql_AdventureWorks_aw_devPool’. The JDBC Resource, ‘AdventureWorks’, is linked to this pool. The pool is a ‘javax.sql.DataSource’ resource type, which references the ‘com.microsoft.sqlserver.jdbc.SQLServerDataSource’. This data source is identical to the data source you built in NetBeans.

GlassFish 03 - New Datasource (Resource) Deployed to GlassFish

New JDBC Resource Successfully Deployed to GlassFish

GlassFish 04 - New JDBC Connection Pool

New JDBC Connection Pool Successfully Deployed to GlassFish

Test Web Services on GlassFish

To test the RESTful web services from GlassFish, begin by clicking on the ‘JdbcSqlWebSrvTest’ application, under ‘Applications’ menu item. On the Applications page, click on the ‘Launch’ action link. GlassFish open a new web browser window, and presents you with two ‘Web Application Links’, one link is HTTP and the other, HTTPS. Click on the HTTP link. This should display the default index.jsp page’s ‘Hello World!’ message.

Test RESTful Web Service in GlassFish - 01

Default Response from Application

To call the service, append the current URL to match the resource URIs you used when testing the services in NetBeans. For example, to display all the employees again like you did in NetBeans, append the current URL, http://%5Byour_server_name%5D:%5Bport%5D/JdbcSqlWebSrvTest/, to include the following:

http://%5Byour_server_name%5D:%5Bport%5D/JdbcSqlWebSrvTest/webresources/entityclasses.employee

This URI should return the same xml content you observed when testing this same URI locally in NetBeans.

Test RESTful Web Service in GlassFish - 02

All Employees Being Successfully Retrieved from the Adventure Works Database

As another test, append the URI to also include the Id of a single employee, as follows:

http://%5Byour_server_name%5D:%5Bport%5D/JdbcSqlWebSrvTest/webresources/entityclasses.employee/2

This should cut the amount of data returned from the Adventure Works database to a single employee record.

Test RESTful Web Service in GlassFish - 03

A Single Employee Being Successfully Retrieved from the Adventure Works Database Using a Parameter

One last test, remove the number two from the URI and add the word ‘count’, as follows:

http://%5Byour_server_name%5D:%5Bport%5D/JdbcSqlWebSrvTest/webresources/entityclasses.employee/count

This time, you should see a single integer returned to the browser, representing the count of all employees in the database’s employee table.

Test RESTful Web Service in GlassFish - 04

Count of All Employees Being Successfully Retrieved from the Adventure Works Database

Conclusion

Congratulations, the Java EE RESTful web services have been successfully deployed to GlassFish. The services are connecting to Adventure Works SQL Server database, through the entity classes and data source, and returning data to your web browser! Next step is to create a RESTful web services client application, to display the data returned by the services and/or to perform CRUD operations on the database.

, , , , , , , , , , , , , , , , , , , ,

15 Comments

Convert VS 2010 Database Project to SSDT and Automate Publishing with Jenkins – Part 3/3

Objectives of 3-Part Series:

Part I: Setting up the Example Database and Visual Studio Projects

  • Setup and configure a new instance of SQL Server 2008 R2
  • Setup and configure a copy of Microsoft’s Adventure Works database
  • Create and configure both a Visual Studio 2010 server project and Visual Studio 2010 database project
  • Test the project’s ability to deploy changes to the database

Part II: Converting the Visual Studio 2010 Database and Server Projects to SSDT

  • Convert the Adventure Works Visual Studio 2010 database and server projects to SSDT projects
  • Create a second Solution configuration and SSDT publish profile for an additional database environment
  • Test the converted database project’s ability to publish changes to multiple database environments

Part III: Automate the Building and Publishing of the SSDT Database Project Using Jenkins

  • Automate the build and delivery of a sql change script artifact, for any database environment, to a designated release location using a parameterized build.
  • Automate the build and publishing of the SSDT project’s changes directly to any database environment using a parameterized build.

Part III: Automate the Building and Publishing of the SSDT Database Project Using Jenkins

In this last post we will use Jenkins to publishing of changes from the Adventure Works SSDT database project to the Adventure Works database. Jenkins, formally Hudson, is the industry-standard, java-based open-source continuous integration server.

Jenkins

If you are unfamiliar with Jenkins, I recommend an earlier post, Automated Deployment to GlassFish Using Jenkins and Ant. That post goes into detail on Jenkins and its associated plug-in architecture. Jenkins’ website provides excellent resources for installing and configuring Jenkins on Windows. For this post, I’ll assume that you have Jenkins installed and running as a Windows Service.

The latest available version of Jenkins, at the time of this post is 1.476. To follow along with the post, you will need to install and configure the following (4) plug-ins:

User Authentication

In the first two posts, we connected to the Adventure Works database with the ‘aw_dev’ SQL Server user account, using SQL Authentication. This account was used to perform schema comparisons and publish changes from the Visual Studio project. Although SQL Authentication is an acceptable means of accessing SQL Server, Windows Authentication is more common in corporate and enterprise software environments, especially where Microsoft’s Active Directory is used. Windows Authentication with Active Directory (AD) provides an easier, centralized user account security model. It is considered more secure.

With Windows Authentication, we associate a SQL Server Login with an existing Windows user account. The account may be local to the SQL Server or part of an Active Directory domain. For this post, instead using SQL Authentication, passing the ‘aw_dev’ user’s credentials to SQL Server in database project’s connection strings, we will switch to Windows Authentication. Using Windows Authentication will allow Jenkins to connect directly to SQL Server.

Setting up the Jenkins Windows User Account

Let’s outline the process of creating a Jenkins Windows user account and using Windows Authentication with our Adventure Works project:

  1. Create a new ‘jenkins’ Windows user account.
  2. Change the Jenkins Windows service Log On account to the ‘jenkins’ Windows account.
  3. Create a new ‘jenkins’ SQL Server Login, associated with the ‘jenkins’ Windows user account, using Windows Authentication.
  4. Provide privileges in SQL Server to the ‘jenkins’ user identical to the ‘aw_dev’ user.
  5. Change the connection strings in the publishing profiles to use Windows Authentication.

First, create the ‘jenkins’ Windows user account on the computer where you have SQL Server and Jenkins installed. If they are on separate computers, then you will need to install the account on both computers, or use Active Directory. For this demonstration, I have both SQL Server and Jenkins installed on the same computer. I gave the ‘jenkins’ user administrative-level rights on my machine, by assigning it to the Administrators group.

Create New Jenkins User

Create New Jenkins User

Next, change the ‘Log On’ Windows user account for the Jenkins Windows service to the ‘jenkins’ Windows user account. Restart the Jenkins Windows service to apply the change. If the service fails to restart, it is likely you did not give enough rights to the user. I suggest adding the user to the Administrators group, to check if the problem you have encountering is permissions-related.

Jenkins Windows Service

Jenkins Windows Service

Set Log On Account for Jenkins Windows Service

Set Log On Account for Jenkins Windows Service

Log On Account for Jenkins Windows Service

Log On Account for Jenkins Windows Service

Log On Account for Jenkins Windows Service Granted

Log On Account for Jenkins Windows Service Granted

Setting up the Jenkins SQL Server Login

Next, to use Windows Authentication with SQL Server, create a new ‘jenkins’ Login for the Production instance of SQL Server and it with the ‘jenkins’ Windows user account. Replicate the ‘aw_dev’ SQL user’s various permissions for the ‘jenkins’ user. The ‘jenkins’ account will be performing similar tasks to ‘aw_dev’, but this time initiated by Jenkins, not Visual Studio. Repeat this process for the Development instance of SQL Server.

Jenkins Login Added to Development Instance

Jenkins Login Added to Development Instance

Jenkins User Any Definition on Production Instance

Jenkins User Any Definition on Production Instance

Jenkins User View Definition on Production Instance Database

Jenkins User View Definition on Production Instance Database

SSMS View of Jenkins User

SSMS View of Jenkins User

Windows Authentication with the Publishing Profile

In Visual Studio, switch the connection strings in the Development and Production publishing profiles in both the server project and database projects to Windows Authentication with Integrated Security. They should look similar to the code below. Substitute your server name and SQL instance for each profile.

Data Source=[SERVER NAME]\[INSTANCE NAME];Integrated Security=True;Pooling=False

Important note here, once you switch the profile’s connection string to Windows Authentication, the Windows user account that you logged into your computer with, is the account that Visual Studio will now user to connect to the database. Make sure your Windows user account has at least the same level of permissions as the ‘aw_dev’ and ‘jenkins’ accounts. As a developer, you would likely have greater permissions than these two accounts.

Configuring Jenkins for Delivery of Script to Release

In many production environments, delivering or ‘turning over’ release-ready code to another team for deployment, as opposed to deploying the code directly, is common practice. A developer starts or ‘kicks off a build’ of the job in Jenkins, which generates artifact(s). Artifacts are usually logical collections of deployable code and other associated components and files, constituting the application being built. Artifacts are often separated by type, such as database, web, Windows services, web services, configuration files, and so forth. Each type may be deployed by a different team or to a different location. Our project will only have one artifact to deliver, the sql change script.

This first Jenkins job we create will just generate the change script, which will then be delivered to a specific remote location for later release. We start by creating what Jenkins refers to as a parameterized build job. It allows us to pass parameters to each build of our job. We pass the name of the configuration (same as our environment name) we want our build to target. With this single parameter, ‘TARGET_ENVIRONMENT’, we can use a single Jenkins job to target any environment we have configured by simply passing its name to the build; a very powerful, time-saving feature of Jenkins.

Step 1 - Parameterized Build Parameter

Step 1 – Parameterized Build Parameter

Let’s outline the steps we will configure our Jenkins job with, to deliver a change script for release:

  1. Copy the Solution from its current location to the Jenkins job’s workspace.
  2. Accept the target environment as a parameterized build parameter (ex. ‘Production’ or ‘Development’).
  3. Build the database project and its dependencies based on the environment parameter.
  4. Generate the sql change script based on the environment parameter.
  5. Compress and name the sql change script based on the environment parameter and build id.
  6. Deliver the compressed script artifact to a designated release location for deployment.
  7. Notify release team that the artifact is ready for release.
  8. Archive the build’s artifact(s).

Copy the Solution to Jenkins

I am not using a revision control system, such as TFS or Subversion, for our example. The Adventure Works Solution resides in a file directory, on my development machine. To copy the entire Solution from its current location into job’s workspace, we add a step in the Jenkins job to execute a simple xcopy command. With source control, you would replace the xcopy step with a similar step to retrieve the project from a specific branch)within the revision control system, using one of many Jenkins’ revision control plug-ins.

Step 2 - Copy Solution to Jenkins Workspace

Step 2 – Copy Solution to Jenkins Workspace

echo 'Copying Adventure Works Solution to Jenkins workspace...'
xcopy "[Path to your Project]\AdventureWorks2008" "%WORKSPACE%" /S /E /H /Y /R /EXCLUDE:[Path to exclude file]\[name of exclude file].txt

echo 'Deleting artifacts from previous builds...'
del "%WORKSPACE%\*_publish.zip" /F /Q

Excluding Solution files from Jenkins job’s workspace that are unnecessary for the job to succeed is good practice. Excluding files saves time during the xcopy and can make troubleshooting build problems easier. To exclude unneeded Solution files, use the xcopy command’s ‘exclude’ parameter. To use exclude, we must first create an exclude text file, listing the directories we don’t need copied, and call it using with the exclude parameter with the xcopy command. Make sure to change the path shown above to reflect the location and name of your exclude file. Here is a list of the directories I chose to exclude. They are either unused by the build, or created as part of the build, for example the sql directories and there subdirectories.

\AdventureWorks2008\sql\
\AdventureWorks2008\Sandbox\
\AdventureWorks2008\_ConversionReport_Files\
\Development\sql\
\Development\Sandbox\
\Development\_ConversionReport_Files\

Build the Solution with Jenkins

Once the Solution’s files are copied into the Jenkins job’s workspace, we perform a build of the database project with an MSBuild build step, using the Jenkins MSBuild Plug-in. Jenkins executes the same MSBuild command Visual Studio would execute to build the project. Jenkins calls MSBuild, which in turn calls the MSBuild ‘Build’ target with parameters that specify the Solution configuration and platform to target.

Generate the Script with Jenkins

After Building the database project, in the same step as the build, we perform a publish of the database project. MSBuild calls the new SSDT’s ‘Publish’ target with parameters that specify the Solution configuration, target platform, publishing profile to use, and whether to only generate a sql change script, or publish the project’s changes directly to the database. In this first example, we are only generating a script. Note the use of the build parameter (%TARGET_ENVIRONMENT%) and environmental variables (%WORKSPACE%) in the MSBuild command. Again, a very powerful feature of Jenkins.

Step 3 - Build and Publish Project

Step 3 – Build and Publish Project

"%WORKSPACE%\AdventureWorks2008\AdventureWorks2008.sqlproj"
/p:Configuration=%TARGET_ENVIRONMENT%
/p:Platform=AnyCPU
/t:Build;Publish
/p:SqlPublishProfilePath="%WORKSPACE%\AdventureWorks2008\%TARGET_ENVIRONMENT%.publish.xml"
/p:UpdateDatabase=False

Compressing Artifacts with Apache Ant

To streamline the delivery, we will add a step to compress the change script using Jenkins Apache Ant Plug-in. Many consider Ant strictly a build tool for Java development. To the contrary, there are many tasks that can be automated for .NET developers with Ant. One particularly nice feature of Ant is its built-in support for zip compression.

Step 4 - Invoke Ant to Compress Artifact

Step 4 – Invoke Ant to Compress Artifact

configuration=$TARGET_ENVIRONMENT
buildNo=$BUILD_NUMBER

The Ant plug-in calls Ant, which in turn calls an Ant buildfile, passing it the properties we give. First, create an Ant buildfile with a single task to zip the change script. To avoid confusion during release, Ant will also append the configuration name and unique Jenkins job build number to the filename. For example, ‘AdventureWorks.publish.sql’  becomes ‘AdventureWorks_Production_123_publish.zip’. This is accomplished by passing the configuration name (Jenkins parameterized build parameter) and the build number (Jenkins environmental variable), as parameters to the buildfile (shown above). The parameters, in the form of key-value-pairs, are treated as properties within the buildfile. Using Ant to zip and name the script literally took us one line of Ant code. The contents of the build.xml buildfile is shown below.

<?xml version="1.0" encoding="utf-8"?>
<project name="AdventureWorks2008" basedir="." default="default">
<description>SSDT Database Project Type ZIP Example</description>
<!-- Example configuration ant call with parameter:
ant -Dconfiguration=Development -DbuildNo=123 -->
<target name="default" description="ZIP sql deployment script">
<echo>$${basedir}=${basedir}</echo>
<echo>$${configuration}=${configuration}</echo>
<echo>$${buildNo}=${buildNo}</echo>
<zip basedir="AdventureWorks2008/sql/${configuration}"
destfile="AdventureWorks_${configuration}_${buildNo}_publish.zip"
includes="*.publish.sql" />
</target>
</project>

Delivery of Artifacts

Lastly, we add a step to deliver the zipped script artifact to a ‘release’ location. Ideally, another team would retrieve and execute the change script against the database. Delivering the artifact to a remote location is easily accomplished using the Jenkins Artifact Deployer Plug-in. First, if it doesn’t already exist, create the location where you will deliver the scripts. Then, ensure Jenkins has permission to manage the location’s contents. In this example, the ‘release’ location is a shared folder I created. In order for Jenkins to access the ‘release’ location, give the ‘jenkins’ Windows user Read/Write (Change) permissions to the shared folder. With the deployment plug-in, you also have the option to delete the previous artifact(s) each time there is a new deployment, or leave them to accumulate.

Sharing Folder for Released Artifacts

Sharing Folder for Released Artifacts

Jenkins User Permissions for Shared Folder

Jenkins User Permissions for Shared Folder

Permissions for Shared Folder

Permissions for Shared Folder

Step 5 - Deploy Artifact to Release Location

Step 5 – Deploy Artifact to Release Location

Multiple Zipped Artifacts in Release Folder

Multiple Zipped Artifacts in Release Folder

Email Notification

Lastly, we want to alert the right team that artifacts have been turned-over for release. There are many plug-ins Jenkins to communicate with end-users or other system. We will use the Jenkins Email Extension Plug-in to email the release team. Configuring dynamic messages to include the parameterized build parameters and Jenkins’ environmental variables is easy with this plug-in. My sample message includes several variables in the body of the message, including target environment, target database, artifact name, and Jenkins build URL.

I had some trouble passing the Jenkins’ parameterized build parameter (‘TARGET_ENVIRONMENT’) to the email plug-in, until I found this post. The format required by the plug-in for the type of variable is a bit obscure as compared to Ant, MSBuild, or other plug-ins.

Step 6 - Email Notification

Artifact: AdventureWorks_${ENV,var="TARGET_ENVIRONMENT"}_${BUILD_NUMBER}_publish.zip
Environment: ${ENV,var="TARGET_ENVIRONMENT"}
Database: AdventureWorks
Jenkins Build URL: ${BUILD_URL}
Please contact Development for questions or problems regarding this release.
Release Request Notification Email Message

Release Request Notification Email Message

Publishing Directly to the Database

As the last demonstration in this series of posts, we will publish the project changes directly to the database. Good news, we have done 95% of the work already. We merely need to copy the Jenkins job we already created, change one step, remove three others steps, and we’re publishing! Start by creating a new Jenkins job by copying the existing script delivery job. Next, drop the Invoke Ant, Artifact Deployer, and Archive Artifacts steps from the job’s configuration. Lastly, set the last parameter of the MSBuild task, ‘UpdateDatabase’, to True from False. That’s it! Instead of creating the script, compressing it, and sending it to a location to be executed later, the changes are generated and applied to the database in a single step.

Hybrid Solution

If you are not comfortable with the direct approach, there is a middle ground between only generating a script and publishing directly to the database. You can keep a record of the changes made to the database as part of publishing. To do so, change the ‘UpdateDatabase’ parameter to True, and only drop the Artifact Deployer step; leave the Invoke Ant and Archive Artifacts steps. The resulting job generates the change script, publishes the changes to the database, and compresses and archives the script. You now have a record of the changes made to the database.

Conclusion

In this last of three posts we demonstrated the use of Jenkins and its plug-ins to created three jobs, representing three possible SSDT publishing workflows. Using the parameterized build feature of Jenkins, each job capable of being executed against any database environment that we have a configuration and publishing profile defined for. Hopefully, one of these three workflows may fit your particular release methodology.

Jenkins SSDT Jobs View

Jenkins SSDT Jobs View

, , , , , , , , , , , , , , , , , , , ,

7 Comments

Convert VS 2010 Database Project to SSDT and Automate Publishing with Jenkins – Part 2/3

Objectives of 3-Part Series:

Part I: Setting up the Example Database and Visual Studio Projects

  • Setup and configure a new instance of SQL Server 2008 R2
  • Setup and configure a copy of Microsoft’s Adventure Works database
  • Create and configure both a Visual Studio 2010 server project and Visual Studio 2010 database project
  • Test the project’s ability to deploy changes to the database

Part II: Converting the Visual Studio 2010 Database and Server Projects to SSDT

  • Convert the Adventure Works Visual Studio 2010 database and server projects to SSDT projects
  • Create a second Solution configuration and SSDT publish profile for an additional database environment
  • Test the converted database project’s ability to publish changes to multiple database environments

Part III: Automate the Building and Publishing of the SSDT Database Project Using Jenkins

  • Automate the build and delivery of a sql change script artifact, for any database environment, to a designated release location using a parameterized build.
  • Automate the build and publishing of the SSDT project’s changes directly to any database environment using a parameterized build.

Part II: Converting the Visual Studio 2010 Database and Server Projects to SSDT

Picking up where part one of this three-part series left off, we are ready to convert the Adventure Works VS 2010 database project and associated server project to the new SSDT project type. Once converted, we will create an additional Solution configuration for Production. Finally, we will publish (vs. deploy) changes to database project’s schema to both the Development and Production environments. Note that Microsoft refers to the new format as either SSDT project type or SQL Server Database Project. I chose the prior in this post, it seemed clearer.

Convert the Projects to SSDT

Microsoft could not have made the conversion to the new SSDT project-type any simpler. Right-click on the Development server project and select ‘Convert to SQL Server Database project’. Select ‘Yes’, select ‘Backup originals for converted files’, and click ‘OK’. The conversion process should take only a minute or two. Following that, you are presented with a Conversion Report when the process is complete. The report should show the successful conversion to the SSDT project type.

Convert Project to SQL Server Database Project

Convert Project to SQL Server Database Project

Server Project Conversion Report

Server Project Conversion Report

Repeat this process for the AdventureWorks2008 database project. Again, you see a Conversion Report when complete. It should also not contain any errors, nor files marked as ‘Not converted’.

Database Project Conversion Report

Database Project Conversion Report

The New Project File Format

Reviewing the Conversion Report for the databae project, note the change to the primary project file. This is the first key difference between the VS 2010 project types and the new SSDT project types. The project file was converted from ‘AdventureWorks2008.dbproj’ to ‘AdventureWorks2008.sqlproj’ (see Conversion Report screen grab, above). Although the earlier project file with the ‘.dbproj’ file suffix is still in the project’s file directory, the Visual Studio Solution is now associated with the new ‘.sqlproj’ project file. This is the same for the server project. The ‘.dbproj’ files are no longer needed. You can drop then from the project’s file directory or from your source control system. This will prevent any confusion going forward.

Publishing Profiles

The second change you will note after the conversion is in the Solution Explorer tab. Each project has three items with the file suffix ‘.publish.xml’. These are publishing profiles. There are profiles for the each Solution configuration – Debug, Release, and Development. A publishing profile has all the settings necessary to publish changes made to the SSDT database project to a specific database environment. As part of the conversion to SSDT, all existing project settings migrated into the current project. Portions of the configuration-specific settings stay in the converted Solution configurations, while publish-specific settings are in the publishing profiles. Publishing profiles, like pre- and post-deployment scripts, are not part of the build. Select a profile in the Project Explorer tab. Note the ‘Build Action’ property in the Properties tab is set to ‘None’.

Solution Explorer View of Converted Projects

Solution Explorer View of Converted Projects

Additional Project and Profile Settings

There are also new settings in the converted projects. They support newer technologies like SSDT, SQL 2012, and Azure. As part of our first major conversion to SSDT, took the opportunity to review all project and publish settings with our database developers and DBAs. We stove to understand all the setting’s purpose and make sure they were correctly configured and documented for each of our many database environments.

Development Publishing Profile Advanced Settings

Development Publishing Profile Advanced Settings

Testing the Converted Projects

To test the successful conversion of the both project to the SSDT project types, select the Development Solution configuration and perform a Rebuild on the Solution. In the Build section of the Output tab, you should see both projects built successfully.

Initial Build Results of SSDT Projects

Initial Build Results of SSDT Projects

Development Publishing Profile

Right-click on the ‘Development.publish.xml’ file in the AdventureWorks2008 project and select ‘Publish…’ Wait for the project to build. Selecting Publish or opening a publishing profile causes the project to build. Afterwards, you should see the ‘Publish Database’ window appear. Here is where you change the settings of Development profile. When converting the Adventure Works project to SSDT, I’ve found the database connection information does not migrate to the profile. Setup the ‘Target database connection’ information in the ‘Connection Properties’ pop-up window by clicking ‘Edit…’. When finished, click ‘OK’ to return to the ‘Publish Database’ window. Finally, save the revised Development publishing profile by clicking ‘Save Profile As…’. I will not cover the specific profile settings, accessed by clicking ‘Advanced…’. Many of these settings will be specific to your environments and workflows. They can be left as default for this demonstration.

Development Publishing Profile

Development Publishing Profile

Development Publishing Profile Connection Properties

Development Publishing Profile Connection Properties

Saving Development Publishing Profile Settings for Server

Saving Development Publishing Profile Settings for Server

Generate Script for Adventure Works Database

Without leaving the ‘Publish Database’ window, click ‘Generate Script’. As in the first post, this action will initiate a schema comparison resulting the generation of a script that aggregates all the schema changes to the project, not already reflected in the database. The script represents the schema ‘delta’ (the difference) between the project and the database. The script will automatically open in Visual Studio’s main window after being created. In the ‘Data Tools Operations’ tab you should see messages indicating generation of the script was successful.

Generate Script Results

Generate Script Results

Also included in the script, along with the schema changes, are any pre- and post-deployment scripts. You should see the single post-deployment script that we created in part one of this series. Pre- and post-deployment scripts are always included in the script, whether or not they have already been executed. This is why it is imperative that pre- and post-deployment scripts are re-runnable. Scripts must have the ability to be executed more than once without producing unintended changes to the database.

Publishing to the Development Database

Next, right-click on the ‘Development.publish.xml’ file, and select ‘Publish…’. This will return you to the same window you were just in to generate the script. Click ‘Publish’. Again, in the ‘Data Tools Operations’ tab you should see messages that the publish operation completed successfully.

Successful Script Generation and Deployment to Development

Successful Script Generation and Deployment to Development

Congratulations, you have completed and tested conversion of the Adventure Works database project to SSDT project type.

Note with SSDT, the term ‘Deploy’, which refers to a specific MSBuild target, is replaced with ‘Publish’, a SSDT-specific build target. Instead of deploying changes to the database, like we did in the first post, we will publish changes with SSDT. To understand how MSBuild is able to call the new SSDT Publish target, open the AdventureWorks2008.sqlproj file by right-clicking on the project and selecting ‘Edit Project File’. In the project file’s xml you will find an ‘Import’ tag that imports the SSDT targets into the project, making them accessible to MSBuild.


<!--Import the settings-->
MSBuildExtensionsPath)\Microsoft\VisualStudio\v$(VisualStudioVersion)\SSDT\Microsoft.Data.Tools.Schema.SqlTasks.targets" />

New Production Environment

Between posts, I installed another instance of SQL Server 2008 R2, named ‘Production’. Into this instance I installed another copy of the Adventure Works database. I added the same ‘aw_dev’ user that we used in Development, with the same permissions. This SQL Server instance and Adventure Works database simulates a second database environment, Production. Normally, this instance would be installed on to separate server, but for simplicity sake I installed the Production instance on the same physical server as the Development instance. It makes no difference for the purposes of this post.

If you wish to follow all examples presented in the next two posts, you will need to install and configure the Production instance of SQL Server. Otherwise, you can disregard the portions of the two posts on publishing to Production, and just stick with the single Development environment. The conversion to SSDT doesn’t require the added Production environment.

New Production Configuration and Publish Profile

Next, we will create a new configuration in the SSDT project’s Solution and configure the resulting publishing profile, targeting the Production environment. We will use this to publish changes from the project to the Production environment. Using the Solution’s Configuration Manager, create a new Solution configuration. This process is unchanged from the VS 2010 database project-type.

New Production Solution Configuration

New Production Solution Configuration

Solution Configuration Manager

Solution Configuration Manager

New Production Configuration Build Settings for Database Project

New Production Configuration Build Settings for Database Project

Right click on the AdventureWorks2008 project and select ‘Publish…’. This will return us to the ‘Publish Database’ window. Like before, with the Development publishing profile, complete the connection string information, this time targeting the Production instance. Change the ‘Publish script name’ setting to ‘Production.sql’. Click ‘Save Profile As…’, and save this profile configuration into the project file path as ‘Production.publish.xml’. Repeat this process for the Development SSDT server project.

Database Project

Production Publishing Profile Settings for Database

Production Publishing Profile Settings for Database

Production Publishing Profile Connection Properties for Database

Production Publishing Profile Connection Properties for Database

Saving Production Publishing Profile Settings for Database

Saving Production Publishing Profile Settings for Database

Server Project

Production Publishing Profile Settings for Server

Production Publishing Profile Settings for Server

Saving Development Publishing Profile Settings for Server

Saving Development Publishing Profile Settings for Server

We now have a new Production Solution configuration and corresponding publishing profiles in each of our two projects.

Solution Explorer View of Production Publishing Profiles

Solution Explorer View of Production Publishing Profiles

We can now target two different database environments from our AdventureWorks2008 project, Development and Production. In a typical production workflow, as a developer, you would make changes to the database project directly, or copy using a source control system like TFS. After testing your changes locally, you execute the publish task to send your schema changes and/or pre- and post-deployment scripts to the Development database instance. This process is be repeated by other developers in your department.

After successfully testing your application(s) against the Development database, you are ready to release the database changes to Testing, or in this example, directly to Production. You execute the Publish task again, this time choosing the Production Solution configuration and Production publishing profile. The schema changes and any pre- and post-deployment scripts are now executed against the Production database. You would follow the same process for other environments, such as Testing or Staging.

Making Schema Changes to Multiple Environments

For this test, we will make schema changes to the ‘Employee’ table, part of the ‘HumanResources’ schema. In Visual Studio, open the Employee table and add two new columns to the end of the table, as shown below. If you have not worked with the SSDT project type before, the view of the table will look very different to you. Microsoft has changed the earlier table view to include a friendlier design view as seen in SSMS, versus the earlier sql create statement only view. There is also a window which details all the key, indexes, and triggers associated with the table. I consider this light years better in term of usability from the developer’s standpoint. Save the changes to the table object and close it.

New SSDT Table Object View

New SSDT Table Object View

Select the Development Solution configuration. Right-click on the Development profile in the AdventureWorks2008 project and click ‘Publish…’ Wait for the project to build. When the ‘Publish Database’ window appears, click ‘Publish’. You have just deployed the Employee table schema changes to the Development instance of the database.

Schema Changes to Employee Table in Script

Schema Changes to Employee Table in Script

Repeat this same process for Production. Don’t forget to switch to the Production Solution configuration and select the Production publish profile. You have now applied the same schema changes to the Production environment. Your customer will be happy they can now track the drug testing of their employees.

Successful Script Generation and Deployment to Development

Successful Script Generation and Deployment to Development

There are other methods available with SSDT to deploy changes to the database. Using a script is the method I have chosen to show in this post.

Conclusion

In this post we converted the Adventure Works database project and Development server project to SSDT project-types. We created a new Solution Configuration and publishing profiles, targeting Production. We made schema changes to the SSDT database project. Finally, we deployed those changes to both the Development and Production database environments.

In Part III of this series, I will show how to use Jenkins CI Server to automate building, testing, delivering scripts, and publishing to a database from the SSDT database project.

, , , , , , , , , , , , , , , , , ,

7 Comments

Convert VS 2010 Database Project to SSDT and Automate Publishing with Jenkins – Part 1/3

Objectives of 3-Part Series:

Part I: Setting up the Example Database and Visual Studio Projects

  • Setup and configure a new instance of SQL Server 2008 R2
  • Setup and configure a copy of Microsoft’s Adventure Works database
  • Create and configure both a Visual Studio 2010 server project and Visual Studio 2010 database project
  • Test the project’s ability to deploy changes to the database

Part II: Converting the Visual Studio 2010 Database and Server Projects to SSDT

  • Convert the Adventure Works Visual Studio 2010 database and server projects to SSDT projects
  • Create a second Solution configuration and SSDT publish profile for an additional database environment
  • Test the converted database project’s ability to publish changes to multiple database environments

Part III: Automate the Building and Publishing of the SSDT Database Project Using Jenkins

  • Automate the build and delivery of a sql change script artifact, for any database environment, to a designated release location using a parameterized build.
  • Automate the build and publishing of the SSDT project’s changes directly to any database environment using a parameterized build.

Background

Microsoft’s Visual Studio 2010 (VS 2010) IDE has been available to developers since April, 2010. Microsoft’s SQL Server 2008 R2 (SQL 2008 R2) has also been available since April, 2010. If you are a modern software development shop or in-house corporate development group, using the .NET technology stack, you probably use VS 2010 and SQL 2008 R2. Moreover, odds are pretty good that you’ve implemented a Visual Studio 2010 Database Project SQL Server Project to support what Microsoft terms a Database Development Life Cycle (DDLC).

Now, along comes SSDT. Recently, along with the release of SQL Server 2012, Microsoft released SQL Server Data Tools (SSDT). Microsoft refers to SSDT as “an evolution of the existing Visual Studio Database project type.” According to Microsoft, SSDT offers an integrated environment within Visual Studio 2010 Professional SP1 or higher for developers to carry out all their database design work for any SQL Server platform (both on and off premise). The term ‘off premises’ refers to SSDT ‘s ability to handle development in the Cloud – SQL Azure. SSDT supports all current versions of SQL Server, including SQL Server 2005, SQL Server 2008, SQL Server 2008 R2, SQL Server 2012 and SQL Azure.

SSDT offers many advantages over the VS 2010 database project type. SSDT provides a more database-centric user-experience, similar to SQL Server Management Studio (SSMS). Anyone who has used VS 2010 database project knows the Visual Studio experience offered a less sophisticated user-interface than SSMS or similar database IDE’s, like Toad for SQL. Installing Microsoft’s free SSDT component, after making sure you have SP1 installed for VS 2010 Professional or higher, you can easily convert you VS 2010 database projects to the new SSDT project type. The conversion offers a better development and deployment experience, and prepare you for an eventual move to SQL Server 2012.

Part I: Setting up the Example Database and Visual Studio Projects

Setting up the Example

To avoid learning SSDT with a copy of your client’s or company’s database, I suggest taking the same route I’ve taken in this post. To demonstrate how to convert from a VS 2010 database project to SSDT, I am using a copy of Microsoft’s Adventure Works 2008 database. Installed, the database only takes up about 180 MBs of space, but is well designed and has enough data to use as a good training tool it, as Microsoft intended. There are several version of the AdventureWorks2008 database available for download depending on your interests – OLTP, SSRS, Analysis Services, Azure, or SQL 2012. I chose to download the full database backup of the AdventureWorks2008R2 without filestream for this post.

Creating the SQL Server 2008 R2 Instance and Database

Before installing the database, I used the SQL Server Installation Center to install a new instance of SQL Server 2008 R2, which I named ‘Development’. This represents a development group’s database environment. Other environments in the software release life-cycle commonly include Testing, Staging, and Production. Each environment usually has its own servers with their own instances of SQL Server, with its own copy of the databases. Environments can also include web servers, application servers, firewalls, routers, and so forth.

After installing the Development instance, I logged into it as an Administrator and created a new empty database, which I named ‘AdventureWorks’. I then restored the downloaded backup copy of Adventure Works 2008 R2, which I downloaded, to the newly created Adventure Works database.

Creating the Database 01

Creating the New Database

Creating the Database 02

Creating the New Database

Creating the Database 03

Creating the Database: Restoring the Full Backup

Creating the Database 04

Creating the Database: Restoring the Full Backup

Creating the Database 05

Creating the Database: Restoring the Full Backup

You may note some differences between the configuration settings displayed below in the screen grabs and the default configuration of the Adventure Works database. This post is not intended to recommend configuration settings for your SQL Server databases or database projects. Every company is different in the way it configures its databases. The key is to make sure that the configuration settings in your database align with the equivalent configuration settings in the database project in Visual Studio 2010. If not, when you initially publish changes to the database from the database project, the project will script the differences and change the database to align to the project.

Creating the Database 06

Creating the Database: Database Properties

Creating the Server Login and Database User

Lastly in SSMS, I added a new login account to the Development SQL Server instance and a user to the Adventure Works database, named ‘aw_dev’. This user represents a developer who will interact with the database through the SSDT database project in VS 2010. For simplicity, I used SQL authentication for this user versus Windows authentication. I gave the user the minimal permissions necessary for this example. Depending on the types of interactions you have with the database, you may need to extend the rights of the user.

New Database Login 01

New Database Login

New Database Login 02

New Database Login

Two key, explicit permissions must be assigned for the user for SSDT to work properly. First is the ‘view any definition’ permission on the Development instance. Second is the ‘view definition’ permission on the Adventure Works database. These enable the SSDT project to perform a schema comparison to the Adventure Works database, explained later in the post. Lack of the view definition permission is one of the most common errors I’ve seen during deployments. They usually occur after adding a new database environment, database, database user, or continuous integration server.

New Database Login: ‘View any definition’ Permission

New Database Login: ‘View any definition’ Permission

New Database Login 03

New Database Login: ‘View definition’ Permission

Setting up Visual Studio Database Project

In VS 2010, I created a new SQL Server 2008 database project, named ‘AdventureWorks2008’. In the same Visual Studio Solution, I also created a new SQL Server 2008 server project, named ‘Development’. The database projects mirrors the schema of the Adventure Works database, and the server project mirrors the instance of SQL Server 2008 R2 on which the database is housed. The exact details of creating and configuring these two projects are too long for this post, but I have a set of screen grabs, hyperlinked below, to aid in creating these two projects. For the database project, I only showed the screens that are signficantly different then the server project screens to save some space.

Server Project

VS 2010 Server Project 01

VS 2010 Server Project: New Project SQL Server 2008 Wizard

VS 2010 Server Project 02

VS 2010 Server Project: New Project SQL Server 2008 Wizard

VS 2010 Server Project 03

VS 2010 Server Project: Project Properties

VS 2010 Server Project 04

VS 2010 Server Project: Set Database Options

VS 2010 Server Project 06

VS 2010 Server Project: Import Database Schema

VS 2010 Server Project 08

VS 2010 Server Project: Database Connection Properties

VS 2010 Server Project 09

VS 2010 Server Project: Configure Build and Deploy

Database Project

VS 2010 Database Project 01

VS 2010 Database Project: New Project SQL Server 2008 Wizard

VS 2010 Database Project 02

VS 2010 Database Project: Project Properties

VS 2010 Database Project 03

VS 2010 Database Project: Summary

Reference from Database Project to Server Project

After creating both projects, I created a reference (dependency) from the Adventure Works 2008 database project to the Development server project. The database project reference to the server project creates the same parent-child relationship that exists between the Development SQL Server instance and the Adventure Works database. Once both projects are created and the reference made, your Solution should look like the second screen grab, below.

Adding a Reference to the Database Project

Adding a Reference to the Database Project

Database Project Dependencies

Database Project Dependencies

Final View of VS 2010 Solution with both Projects and Reference

Final View of VS 2010 Solution with both Projects and Reference

Creating the Development Solution Configuration

Next, I created a new ‘Development’ Solution configuration. This configuration defines the build and deployment parameters for the database project when it targets the Development environment. Again, in a normal production environment, you would have several configurations, each targeting a specific database environment. In the context of this post, a database environment refers to a unique combination of  servers, server instances, databases, data, and users. For this first example we are only setting up one database environment, Development.

New Development Solution Configuration 01

New Development Solution Configuration

New Development Solution Configuration 02

New Development Solution Configuration

The configuration specifies the parameters specific to the Adventure Works database in the Development environment. The connection string containing the server, instance, and database names, user account, and connection parameters, are all specific to the Development environment. They are different in the other environments – Testing, Staging, and Production.

Testing the Development Configuration

Once the Development configuration was completed, I ran the ‘Rebuild’ command on the Solution, using the Development configuration, to make sure there are no errors or warnings. Next, with the Development configuration set to only create the deployment script, not to create and deploy the changes to the database, I ran the ‘Deploy’ command. This created a deployment script, entitled ‘AdventureWorks2008.sql’, in the ‘sql\Development’ folder of the AdventureWorks2008 database project.

Create a Deployment Script Only

Database Project Configuration Settings to Create a Deployment Script Only

Deployment Script Location in Project

Deployment Script Location in Project

Since I just created both the Adventure Works database and the database project, based on the database, there are no schema changes in the deployment script. You will see ‘filler’ code for error checking and so forth, but no real executable schema changes to the database are present at this point. If you do see initial changes included in the script, usually database configuration changes, I suggest modifying the settings of the database project and/or the database to align to one another. For example, you may see code in the script to change the database’s default cursor from global to local, or vice-versa. Or, you may also see code in the script to the databases recovery model from full to simple, or vice-versa. You should decide whether the project or the database is correct, and change the other one to match. If necessary, re-run the ‘Deploy’ command and re-check the deployment script. Optionally, you can always execute the script with the changes, thus changing the database to match the project, if the project settings are correct.

Testing Deployment

After successfully testing the development configuration and the deployment script, making any configuration changes necessary to the project and/or the database, I then tested the project’s ability to successfully execute a Deploy command against the database. I changed the Development configuration’s deploy action from ‘create a deployment script (.sql)’ to from ‘create a deployment script (.sql) and deploy to the database’. I then ran the ‘Deploy’ command again, and this time the script is created and executed against the database. Since I still had not made any changes to the project, there were no schema changes made to the database. I just tested the project’s ability to create and deploy to the database at this point. Most errors at this stage are insufficient database user permissions (see example, below).

Test Deployment to the Database

Test Deployment to the Database

View Any Definition Permission Error

View Any Definition Permission Error

Testing Changes to the Project

Finally, I tested the project’s ability to make changes to the database as part of the deployment. To do so, I created a simple post-deployment script that changes the first name of a single, existing employee. After adding the post-deployment script to the database project and adding the script’s path to the post-deployment script file, I again ran the ‘Deploy’ command again, still using the Development configuration. This time the deployment script contained the post-deployment script’s contents. When deployed, one record was affected, as indicated in VS 2010 Output tab. I verified the change was successful in the Adventure Works database table, using SSMS.

Test Changes Deployed to Database 02

Simple Post-Deployment Script

Test Changes Deployed to Database 01

Simple Post-Deployment Script

Test Changes Deployed to Database 03

Post-Deployment Script Location in Project

Test Changes Deployed to Database 04

One Row Affected by Deployment this Time

Before Post-Deployment Script was Ran

Before Post-Deployment Script was Deployed

After Post-Deployment Script was Ran

After Post-Deployment Script was Deployed

Conclusion

We now have a SQL Server 2008 R2 database instance representing a Development environment, and a copy of the Adventure Works database, being served from that instance. We have corresponding VS 2010 database and server projects. We also have a new Development Solution configuration, targeting the Development environment. Lastly, we tested the database project’s capability to successfully build and deploy a change to the database.

In Part II of this series, I will show how to convert the VS 2010 database and server projects to SSDT.

, , , , , , , , , , , , , , , , , , ,

5 Comments

Automated Deployment to GlassFish Using Jenkins CI Server and Apache Ant

Use Jenkins and Apace Ant to compile, assemble, test, and deploy a RESTful web service to GlassFish. All source code for this post is available on GitHub. Note GitHub repo reflects updates to project on 10/31/2013.

Jenkins, formally Hudson, is the industry-standard, java-based open-source continuous integration server. According to their website, Jenkins provides over 400 plug-ins to support building and testing almost any type of project. According to Apache, Ant is a Java library and command-line tool whose mission is to drive processes described in build files as targets. This post demonstrates the use of Jenkins and Apache Ant to compile, assemble, unit test, and deploy a Java EE 6 RESTful web service to Oracle’s GlassFish open-source application server.

For the sake of brevity, I have chosen to use the HelloWorld RESTful web service example included with NetBeans. I will use NetBeans to create the project, write the unit tests, and produce an Ant target in the build file. I will not delve deeply into the inner workings of the web service itself since the focus of this post is automation.

System Configuration

This post assumes that you have current versions of NetBeans, JUnit, Jenkins, GlassFish, Ant and Java installed and configured on your Microsoft Windows-based computer. A full installation of NetBeans comes with JUnit, Ant, and GlassFish. At the time of the original post, I was using NetBeans 7.1.2, GlassFish 3.1.2, Jenkins 1.4.6.3, Ant 1.8.3, and JDK 1.7.0_02.

For simplicity, I am using a single development machine for this demonstration, on which all applications are installed. In a true production environment you would most likely have a distributed configuration with GlassFish installed on an application server, Jenkins on a build server, and NetBeans on your development machine. Also, for this post, I am also not using a source-code management (SCM) system, also called a version control system (VCS), such as Subversion or Mercurial, to house the project’s source code. Again, in a production environment, your source-code would be placed on SCM/VCS server.

Both GlassFish and Jenkins are configured by default to run on server port 8080. Since I have both applications installed on the same machine, I have changed Jenkins’ default port to another unused port, 9090. Changing Jenkins’ port is easy to do. If you don’t know how, consult this post or similar.

NetBeans

First, create a new project in NetBeans, by selecting the New Project -> Samples -> Java Web Services, REST: Hello World (Java EE 6), as shown below. Rename the project to HelloGlassFish. When complete, the project, in the Projects tab, should look like the screen-grab, below.

New Project View in NetBeans

New Project View in NetBeans

JUnit

Next create a unit-test using JUnit, the open-source unit-testing framework. Jenkins will eventually run this test each time the project is built. Creating unit-tests is easy in NetBeans. Select the ‘NameStorageBean.java’ class object, right-click, and select Tools -> Create JUnit Tests… This will create a default ‘NameStorageBeanTest.java’ class object in a new, ‘Test Packages’ directory. Overwrite NameStorageBeanTest.java contents with the follows code. This will create a single unit test we can use to demonstrate JUnit’s integration with Jenkins. You will also notice new test objects in the Project tab.

Build the project and run the ‘testGetName’ unit-test to make sure it works correctly and the test passes.

Apache Ant

Next, change to the Files tab. Open the ‘build.xml’ file, as shown below. Also, for later reference, note the contents of the ‘HelloGlassFish.war’ and the location of the ‘pwdfile_domain1’ password file.

New Project File View in NetBeans

Place the following Ant target, entitled ‘jenkins-glassfish-deploy’, into the build.xml file, between the end of the commented section and the closing <project/> tag, as shown below.

This is the Ant target Jenkins will use to build, test, and deploy the project. The primary ‘jenkins-glassfish-deploy’ target calls three Ant targets using the antcall element. They include clean, default, and test. Each of these Ant targets has dependencies on other Ant targets, which in turn depend on yet other targets – a dependency tree. For example, default depends on dist and javadoc. The test target depends on other targets to build the .war file. If you are not using test to execute unit tests, you can call the test target to build the .war file.

The last part of the ‘jenkins-glassfish-deploy’ target is a little different. It’s an exec (execute) element, which calls asadmin to deploy the project to GlassFish with a series of GlassFish domain-specific parameters. These parameters include the GlassFish domain’s URL and port, the domain’s administrative user and password account info (found in a password file), the location of the .war file to deploy, and destination of the .war within GlassFish. Calling asadmin deploy gives you fine control over the details of how the project is deployed to GlassFish.

The password file, referenced in the target is a simple text file, which stores the password for the user account used to execute the asadmin deploy call. The contents of the file look like:

AS_ADMIN_PASSWORD=Your_Password_Here

This target could be simplified with the depends attribute. Instead of the three antcall elements, you could simply add depends="clean, default, test" to the target element:

According to Oracle, the asadmin utility is used to perform any administrative tasks for GlassFish from the command line. You can use this asadmin utility in place of using the GlassFish Administrator interface. I am able to call asadmin directly because I have added the path to asadmin.bat to the Windows’ environmental variable, PATH. The asadmin.bat file is in the GlassFish bin directory, similar to ‘C:\Program Files\glassfish-3.1.2\glassfish\bin\’.

Jenkins

Switching to Jenkins, create a new Job named HelloGlassFish. In the HelloGlassFish configuration, we need to add two Build steps and one post-build Action. For the first Build step, since we are not using SCM, we will copy the files from the project in the NetBeans workspace to the Jenkins workspace. To do this, add an ‘Execute Windows batch command’ action with code similar to code snippet below, but substituting your own project’s file path. Note, you can substitute the %WORKSPACE% environmental variable for the xcopy destination (see call-out 1 in the below screen-grab). This variable represents the absolute path of the directory assigned to the build as a workspace, according to Jenkins. Jenkins offers many useful variables, accessible to Windows batch scripts.

xcopy "C:\Users\gstaffor\Documents\NetBeansProjects\HelloGlassFish\HelloGlassFish" "%WORKSPACE%" /s /e /h /y

Next, add the second Build task, ‘Invoke Ant’. I assume you already have Ant configured for Jenkins. In the ‘Target’s text box, enter the Ant target we created in NetBeans build.xml file, entitled ‘jenkins-glassfish-deploy’ (see call-out 2 in the below screen-grab). If the name of your build file is anything other than the default ‘build.xml’, you will need to enter the Ant file name.

Lastly, add the single Post-build Action, ‘Publish JUnit test result report’. This will show us a visual representation of the results of our project’s unit-tests. Input the relative path to your reports from the workspace root. The path should be similar to call-out 3 in the screen-grab, below.

When complete, the HelloGlassFish Job’s configuration should resemble the screen-grab, below.

Jenkins HelloGlassFish Project Configuration

Jenkins HelloGlassFish Project Configuration

Save and close the configuration. Build the HelloGlassFish Job in Jenkins and make sure it succeeds with error.

GlassFish

Open GlassFish’s browser-based Domain Admin Console, usually on server port 4848, by default. On the left-hand side of the main window, under ‘Common Tasks’, tip the ‘Applications’ node. You should see the HelloGlassFish application is now deployed to GlassFish. You don’t have to do anything in GlassFish, Jenkins and Ant has taken care of everything.

GlassFish’s browser-based Domain Admin Console

GlassFish’s browser-based Domain Admin Console

To view the HelloGlassFish application, open a new browser window and direct it to ‘http://localhost:8080/HelloGlassFish/resources/helloWorld’. You should see a ‘Hello World!’ message displayed in your browser’s window. Note, since we only changed the name of the default HelloWorld NetBeans sample project to HelloGlassFish, not the web service’s URI, ‘helloWorld’ is still a required part of the URL path.

Redeploying the Project

Lastly, let’s demonstrate how easily changes to our project can be re-complied, re-tested, and re-deployed to GlassFish by Jenkins and Ant. Return to the HelloGlassFish project in NetBeans and open the NameStorageBean.java class. Change the value of the ‘name’ field from ‘World’ to ‘GlassFish’ and save the changes. Don’t build or do anything else in NetBeans. Instead, return to Jenkins and build the HelloGlassFish Job, again.

Change the NameStorageBean name Field

Change the NameStorageBean name Field

When the Job has finished building, re-direct your browser back to ‘http://localhost:8080/HelloGlassFish/resources/helloWorld’. You should now see a ‘Hello GlassFish!’ message displayed in your browser’s window instead of the earlier message, ‘Hello World!’. Jenkins has called the Ant target, which in turn re-compiled, re-tested, and re-deployed the modified HelloGlassFish application to GlassFish.

HelloGlassFish RESTful Web Service Demo

HelloGlassFish RESTful Web Service Demo

Helpful Links

, , , , , , , , , , , , , , , , ,

5 Comments