Posts Tagged Virtual Machine
Create and manage ‘multi-machine’ environments with Vagrant, using JSON configuration files. Allow increased portability across hosts, environments, and organizations.
As their website says, Vagrant has made it very easy to ‘create and configure lightweight, reproducible, and portable development environments.’ Based on Ruby, the elegantly simple open-source programming language, Vagrant requires a minimal learning curve to get up and running.
In this post, we will create what Vagrant refers to as a ‘multi-machine’ environment. We will provision three virtual machines (VMs). The VMs will mirror a typical three-tier architected environment, with separate web, application, and database servers.
We will move all the VM-specific information from the Vagrantfile to a separate JSON format configuration file. There are a few advantages to moving the configuration information to separate file. First, we can configure any number VMs, while keeping the Vagrantfile exactly the same. Secondly and more importantly, we can re-use the same Vagrantfile to build different VMs on another host machine.
Although certainly not required, I am also using Chef in this example. More specifically, I am using Hosted Chef to further configure the VMs. Like the VM-specific information above, I have also moved the Chef-specific information to a separate JSON configuration file. We can now use the same Vagrantfile within another Chef Environment, or even within another Chef Organization, using an alternate configuration files. If you are not a Chef user, you can disregard that part of the configuration code. Alternately, you can substitute the Chef configuration code for Puppet, if that is your configuration automation tool of choice.
The only items we will not remove from the Vagrantfile are the Vagrant Box and synced folder configurations. These items could also be moved to a separate configuration file, making the Vagrantfile even more generic and portable.
Below is the VM-specific JSON configuration file, containing all the individual configuration information necessary for Vagrant to build the three VMs: ‘apps’, dbs’, and ‘web’. Each child ‘node’ in the parent ‘nodes’ object contains key/value pairs for VM names, IP addresses, forwarding ports, host names, and memory settings. To add another VM, you would simply add another ‘node’ object.
Next, is the Chef-specific JSON configuration file, containing Chef configuration information common to all the VMs.
Lastly, the Vagrantfile, which loads both configuration files. The Vagrantfile instructs Vagrant to loop through all nodes in the nodes.json file, provisioning VMs for each node. Vagrant then uses the chef.json file to further configure the VMs.
The environment and node configuration items in the chef.json reference an actual Chef Environment and Chef Nodes. They are both part of a Chef Organization, which is configured within a Hosted Chef account.
Each VM has a varying number of ports it needs to configue and forward. To accomplish this, the Vagrantfile not only loops through the each node, it also loops through each port configuration object it finds within the node object. Shown below is the Database Server VM within VirtualBox, containing three forwarding ports.
Running the ‘vagrant up’ command will provision all three individually configured VMs. Once created and running in VirtualBox, Chef further configures the VMs with the necessary settings and applications specific to each server’s purposes. You can just as easily create 10, 100, or 1,000 VMs using this same process.
According to Canonical, ‘Ubuntu Cloud Images are pre-installed disk images that have been customized by Ubuntu engineering to run on cloud-platforms such as Amazon EC2, Openstack, Windows and LXC’. Ubuntu also disk images, or ‘boxes’, built specifically for Vagrant and VirtualBox. Boxes, according to Vagrant, ‘are the skeleton from which Vagrant machines are constructed. They are portable files which can be used by others on any platform that runs Vagrant to bring up a working environment‘. Ubuntu’s images are very popular with Vagrant users to build their VMs.
Assuming you have VirtualBox and Vagrant installed on your Windows, Mac OS X, or Linux system, with a few simple commands, ‘vagrant add box…’, ‘vagrant init…’, and ‘vagrant up’, you can provision a VM from one of these boxes.
Dynamically Allocated Storage
The Ubuntu Cloud Images (boxes), are Virtual Machine Disk (VMDK) format files. These VMDK files are configured for dynamically allocated storage, with a virtual size of 40 GB. That means the VMDK format file should grow to an actual size of 40 GB, as files are added. According to VirtualBox, the VM ‘will initially be very small and not occupy any space for unused virtual disk sectors, but will grow every time a disk sector is written to for the first time, until the drive reaches the maximum capacity chosen when the drive was created’.
To illustrate dynamically allocated storage, below are three freshly provisioned VirtualBox virtual machines (VM), on three different hosts, all with different operating systems. One VM is hosted on Windows 7 Enterprise, another on Ubuntu 13.10 Desktop Edition, and the last on Mac OS X 10.6.8. The VMs were all created with Vagrant from the official Ubuntu Server 13.10 (Saucy Salamander) cloud images. The Windows and Ubuntu hosts used the 64-bit version. The Mac OS X host used the 32-bit version. According to VirtualBox Manager, on all three host platforms, the virtual size of the VMs is 40 GB and the actual size is about 1 GB.
So What’s the Problem?
After a significant amount of troubleshooting Chef recipe problems on two different Ubuntu-hosted VMs, the issue with the cloud images became painfully clear. Other than a single (seemingly charmed) Windows host, none of the VMs I tested on Windows-, Ubuntu-, and Mac OS X-hosts would expand beyond 4 GB. Below is the file system disk space usage report from four host’s VMs. All four were created with the most current version of Vagrant (1.4.1), and managed with the most current version of VirtualBox (4.3.6.x).
Windows-hosted 64-bit Cloud Image VM #1:
vagrant@vagrant-ubuntu-saucy-64:/tmp$ df -hT Filesystem Type Size Used Avail Use% Mounted on /dev/sda1 ext4 40G 1.1G 37G 3% / none tmpfs 4.0K 0 4.0K 0% /sys/fs/cgroup udev devtmpfs 241M 12K 241M 1% /dev tmpfs tmpfs 50M 336K 49M 1% /run none tmpfs 5.0M 0 5.0M 0% /run/lock none tmpfs 246M 0 246M 0% /run/shm none tmpfs 100M 0 100M 0% /run/user /vagrant vboxsf 233G 196G 38G 85% /vagrant
Windows-hosted 32-bit Cloud Image VM #2:
vagrant@vagrant-ubuntu-saucy-32:~$ df -hT Filesystem Type Size Used Avail Use% Mounted on /dev/sda1 ext4 4.0G 1012M 2.8G 27% / none tmpfs 4.0K 0 4.0K 0% /sys/fs/cgroup udev devtmpfs 245M 8.0K 245M 1% /dev tmpfs tmpfs 50M 336K 50M 1% /run none tmpfs 5.0M 4.0K 5.0M 1% /run/lock none tmpfs 248M 0 248M 0% /run/shm none tmpfs 100M 0 100M 0% /run/user /vagrant vboxsf 932G 209G 724G 23% /vagrant
Ubuntu-hosted 64-bit Cloud Image VM:
vagrant@vagrant-ubuntu-saucy-64:~$ df -hT Filesystem Type Size Used Avail Use% Mounted on /dev/sda1 ext4 4.0G 1.1G 2.7G 28% / none tmpfs 4.0K 0 4.0K 0% /sys/fs/cgroup udev devtmpfs 241M 8.0K 241M 1% /dev tmpfs tmpfs 50M 336K 49M 1% /run none tmpfs 5.0M 0 5.0M 0% /run/lock none tmpfs 246M 0 246M 0% /run/shm none tmpfs 100M 0 100M 0% /run/user /vagrant vboxsf 74G 65G 9.1G 88% /vagrant
Mac OS X-hosted 32-bit Cloud Image VM:
vagrant@vagrant-ubuntu-saucy-32:~$ df -hT Filesystem Type Size Used Avail Use% Mounted on /dev/sda1 ext4 4.0G 1012M 2.8G 27% / none tmpfs 4.0K 0 4.0K 0% /sys/fs/cgroup udev devtmpfs 245M 12K 245M 1% /dev tmpfs tmpfs 50M 336K 50M 1% /run none tmpfs 5.0M 0 5.0M 0% /run/lock none tmpfs 248M 0 248M 0% /run/shm none tmpfs 100M 0 100M 0% /run/user /vagrant vboxsf 149G 71G 79G 48% /vagrant
On the first Windows-hosted VM (the only host that actually worked), the virtual SCSI disk device (sda1), formatted ‘ext4‘, had a capacity of 40 GB. But, on the other three hosts, the same virtual device only had a capacity of 4 GB. I tested the various 32- and 64-bit Ubuntu Server 12.10 (Quantal Quetzal), 13.04 (Raring Ringtail), and 13.10 (Saucy Salamander) cloud images. They all exhibited the same issue. However, the Ubuntu 12.04.3 LTS (Precise Pangolin) worked fine on all three host OS systems.
To prove the issue was specifically with Ubuntu’s cloud images, I also tested boxes from Vagrant’s own repository, as well as other third-party providers. They all worked as expected, with no storage discrepancies. This was suggested in the only post I found on this issue, from StackExchange.
To confirm the Ubuntu-hosted VM will not expand beyond 4 GB, I also created a few multi-gigabyte files on each VM, totally 4 GB. The VMs virtual drive would not expand beyond 4 GB limit to accommodate the new files, as demonstrated below on a Ubuntu-hosted VM:
vagrant@vagrant-ubuntu-saucy-64:~$ dd if=/dev/zero of=/tmp/big_file2.bin bs=1M count=2000 dd: writing '/tmp/big_file2.bin': No space left on device 742+0 records in 741+0 records out 777560064 bytes (778 MB) copied, 1.81098 s, 429 MB/s vagrant@vagrant-ubuntu-saucy-64:~$ df -hT Filesystem Type Size Used Avail Use% Mounted on /dev/sda1 ext4 4.0G 3.7G 196K 100% /
The exact cause eludes me, but I tend to think the cloud images are the issue. I know they are capable of working, since the Ubuntu 12.04.3 cloud images expand to 40 GB, but the three most recent releases are limited to 4 GB. Whatever the cause, it’s a significant problem. Imagine you’ve provisioned a 100 or a 1,000 server nodes on your network from any of these cloud images, expecting them to grow to 40 GB, but really only having 10% of that potential. Worse, they have live production data on them, and suddenly run out of space.
Below is the complete shell sessions from three hosts.
Windows-hosted 64-bit Cloud Image VM #1:
Ubuntu-hosted 64-bit Cloud Image VM:
Mac OS X-hosted 32-bit Cloud Image VM:
Build an automated testing, continuous integration, and continuous deployment system, using Maven, Hudson, WebLogic Server, JUnit, and NetBeans. Developed with Oracle’s Pre-Built Enterprise Java Development VM. Download the complete source code from Dropbox and on GitHub.
In this post, we will build a basic automated testing, continuous integration, and continuous deployment system, using Oracle’s Pre-Built Enterprise Java Development VM. The primary goal of the system is to automatically compile, test, and deploy a simple Java EE web application to a test environment. As this post will demonstrate, the key to a successful system is not a single application, but the effective integration of all the system’s applications into a well-coordinated and consistent workflow.
Building system such as this can be complex and time-consuming. However, Oracle’s Pre-Built Enterprise Java Development VM already has all the components we need. The Oracle VM includes NetBeans IDE for development, Apache Subversion for version control, Hudson Continuous Integration (CI) Server for build automation, JUnit and Hudson for unit test automation, and WebLogic Server for application hosting.
In addition, we will use Apache Maven, also included on the Oracle VM, to help manage our project’s dependencies, as well as the build and deployment process. Overlapping with some of Apache Ant’s build task functionality, Maven is a powerful cross-cutting tool for managing the modern software development projects. This post will only draw upon a small part of Maven’s functionality.
To save some time, we will use the same WebLogic Server (WLS) domain we built-in the last post, Deploying Applications to WebLogic Server on Oracle’s Pre-Built Development VM. We will also use code from the sample Hello World Java EE web project from that post. If you haven’t already done so, work through the last post’s example, first.
Here is a quick list of requirements for this demonstration:
- Oracle VM
- Oracle’s Pre-Built Enterprise Java Development VM running on current version of Oracle VM VirtualBox (mine: 4.2.12)
- Oracle VM’s has the latest system updates installed (see earlier post for directions)
- WLS domain from last post created and running in Oracle VM
- Credentials supplied with Oracle VM for Hudson (username and password)
- Window’s Development Machine
- Current version of Apache Maven installed and configured (mine: 3.0.5)
- Current version of NetBeans IDE installed and configured (mine: 7.3)
- Optional: Current version of WebLogic Server installed and configured
- All environmental variables properly configured for Maven, Java, WLS, etc. (MW_HOME, M2, etc.)
The steps involved in this post’s demonstration are as follows:
- Install the WebLogic Maven Plugin into the Oracle VM’s Maven Repositories, as well as the Development machine
- Create a new Maven Web Application Project in NetBeans
- Copy the classes from the Hello World project in the last post to new project
- Create a properties file to store Maven configuration values for the project
- Add the Maven Properties Plugin to the Project’s POM file
- Add the WebLogic Maven Plugin to project’s POM file
- Add JUnit tests and JUnit dependencies to project
- Add a WebLogic Descriptor to the project
- Enable Tunneling on the new WLS domain from the last post
- Build, test, and deploy the project locally in NetBeans
- Add project to Subversion
- Optional: Upgrade existing Hudson 2.2.0 and plugins on the Oracle VM latest 3.x version
- Create and configure new Hudson CI job for the project
- Build the Hudson job to compile, test, and deploy project to WLS
WebLogic Maven Plugin
First, we need to install the WebLogic Maven Plugin (‘weblogic-maven-plugin’) onto both the Development machine’s local Maven Repository and the Oracle VM’s Maven Repository. Installing the plugin will allow us to deploy our sample application from NetBeans and Hudson, using Maven. The weblogic-maven-plugin, a JAR file, is not part of the Maven repository by default. According to Oracle, ‘WebLogic Server provides support for Maven through the provisioning of plug-ins that enable you to perform various operations on WebLogic Server from within a Maven environment. As of this release, there are two separate plug-ins available.’ In this post, we will use the weblogic-maven-plugin, as opposed to the wls-maven-plugin. Again, according to Oracle, the weblogic-maven-plugin “delivered in WebLogic Server 11g Release 1, provides support for deployment operations.”
The best way to understand the plugin install process is by reading the Using the WebLogic Development Maven Plug-In section of the Oracle Fusion Middleware documentation on Developing Applications for Oracle WebLogic Server. It goes into detail on how to install and configure the plugin.
In a nutshell, below is a list of the commands I executed to install the weblogic-maven-plugin version 18.104.22.168 on both my Windows development machine and on my Oracle VM. If you do not have WebLogic Server installed on your development machine, and therefore no access to the plugin, install it into the Maven Repository on the Oracle VM first, then copy the jar file to the development machine and follow the normal install process from that point forward.
On Windows Development Machine:
On the Oracle VM:
To test the success of your plugin installation, you can run the following maven command on Windows or Linux:
mvn help:describe -Dplugin=com.oracle.weblogic:weblogic-maven-plugin
Sample Maven Web Application
Using NetBeans on your development machine, create a new Maven Web Application. For those of you familiar with Maven, the NetBeans’ Maven Web Application project is based on the ‘webapp-javaee6:1.5’ Archetype. NetBeans creates the project by executing a ‘archetype:generate’ Maven Goal. This is seen in the ‘Output’ tab after the project is created.
By default you may have Tomcat and GlassFish as installed options on your system. Unfortunately, NetBeans currently does not have the ability to configure a remote connection to the WLS instance running on the Oracle VM, as I understand. You do not need an instance of WLS installed on your development machine since we are going to use the copy on the Oracle VM. We will use Maven to deploy the project to WLS on the Oracle VM, later in the post.
Next, copy the two java class files from the previous blog post’s Hello World project to the new project’s source package. Alternately, download a zipped copy this post’s complete sample code from Dropbox or on GitHub.
Because we are copying a RESTful web service to our new project, NetBeans will prompt us for some REST resource configuration options. To keep this new example simple, choose the first option and uncheck the Jersey option.
Next, create a set of JUnit tests for each class by right-clicking on both classes and selecting ‘Tools’ -> ‘Create Tests’.
We will use the test classes and dependencies NetBeans just added to the project. However, we will not use the actual JUnit tests themselves that NetBeans created. To properly set-up the default JUnit tests to work with an embedded version of WLS is well beyond the scope of this post.
Overwrite the contents of the class file with the code provided from Dropbox. I have replaced the default JUnit tests with simpler versions for this demonstration. Build the file to make sure all the JUnit tests all pass.
Next, add a new Properties file to the project, entitled ‘maven.properties’.
Add the following key/value pairs to the properties file. These key/value pairs are referenced will be referenced the POM.xml by the weblogic-maven-plugin, added in the next step. Placing the configuration values into a Properties file is not necessary for this post. However, if you wish to deploy to multiple environments, moving environmentally-specific configurations into separate properties files, using Maven Build Profiles, and/or using frameworks such as Spring, are all best practices.
Java Properties File (maven.properties):
Maven Plugins and the POM File
Next, add the WLS Maven Plugin (‘weblogic-maven-plugin’) and the Maven Properties Plugin (‘properties-maven-plugin’) to the end of the project’s Maven POM.xml file. The Maven Properties Plugin, part of the Mojo Project, allows us to substitute configuration values in the Maven POM file from a properties file. According to codehaus,org, who hosts the Mojo Project, ‘It’s main use-case is loading properties from files instead of declaring them in pom.xml, something that comes in handy when dealing with different environments.’
Project Object Model File (pom.xml):
WebLogic Deployment Descriptor
A WebLogic Deployment Descriptor file is the last item we need to add to the new Maven Web Application project. NetBeans has descriptors for multiple servers, including Tomcat (context.xml), GlassFish (application.xml), and WebLogic (weblogic.xml). They provide a convenient location to store specific server properties, used during the deployment of the project.
Add the ‘context-root’ tag. The value will be the name of our project, ‘HelloWorldMaven’, as shown below. According to Oracle, “the context-root element defines the context root of this standalone Web application.” The context-root of the application will form part of the URL we enter to display our application, later.
Make sure to the WebLogic descriptor file (‘weblogic.xml’) is placed in the WEB-INF folder. If not, the descriptor’s properties will not be read. If the descriptor is not read, the context-root of the deployed application will default to the project’s WAR file’s name. Instead of ‘HelloWorldMaven’ as the context-root, you would see ‘HelloWorldMaven-1.0-SNAPSHOT’.
Before we compile, test, and deploy our project, we need to make a small change to WLS. In order to deploy our project remotely to the Oracle VM’s WLS, using the WebLogic Maven Plugin, we must enable tunneling on our WLS domain. According to Oracle, the ‘Enable Tunneling’ option “Specifies whether tunneling for the T3, T3S, HTTP, HTTPS, IIOP, and IIOPS protocols should be enabled for this server.” To enable tunneling, from the WLS Administration Console, select the ‘AdminServer’ Server, ‘Protocols’ tab, ‘General’ sub-tab.
Build and Test the Project
Right-click and select ‘Build’, ‘Clean and Build’, or ‘Build with Dependencies’. NetBeans executes a ‘mvn install’ command. This command initiates a series of Maven Goals. The goals, visible NetBean’s Output window, include ‘dependency:copy’, ‘properties:read-project-properties’, ‘compiler:compile’, ‘surefire:test’, and so forth. They move the project’s code through the Maven Build Lifecycle. Most goals are self-explanatory by their title.
The last Maven Goal to execute, if the other goals have succeeded, is the ‘weblogic:deploy’ goal. This goal deploys the project to the Oracle VM’s WLS domain we configured in our project. Recall in the POM file, we configured the weblogic-maven-plugin to call the ‘deploy’ goal whenever ‘install’, referred to as Execution Phase by Maven, is executed. If all goals complete without error, you have just compiled, tested, and deployed your first Maven web application to a remote WLS domain. Later, we will have Hudson do it for us, automatically.
Executing Maven Goals in NetBeans
A small aside, if you wish to run alternate Maven goals in NetBeans, right-click on the project and select ‘Custom’ -> ‘Goals…’. Alternately, click on the lighter green arrows (‘Re-run with different parameters’), adjacent to the ‘Output’ tab.
For example, in the ‘Run Maven’ pop-up, replace ‘install’ with ‘surefire:test’ or simply ‘test’. This will compile the project and run the JUnit tests. There are many Maven goals that can be ran this way. Use the Control key and Space Bar key combination in the Maven Goals text box to display a pop-up list of available goals.
Now that our project is complete and tested, we will commit the project to Subversion (SVN). We will commit a copy of our source code to SVN, installed on the Oracle VM, for safe-keeping. Having our source code in SVN also allows Hudson to retrieve a copy. Hudson will then compile, test, and deploy the project to WLS.
The Repository URL, User, and Password are all supplied in the Oracle VM information, along with the other URLs and credentials.
When you import you project for the first time, you will see more files than are displayed below. I had already imported part of the project earlier while creating this post. Therefore most of my files were already managed by Subversion.
Upgrading Hudson CI Server
The Oracle VM comes with Hudson pre-installed in it’s own WLS domain, ‘hudson-ci_dev’, running on port 5001. Start the domain from within the VM by double-clicking the ‘WLS 12c – Hudson CI 5001’ icon on the desktop, or by executing the domain’s WLS start-up script from a terminal window:
Once started, the WLS Administration Console 12c is accessible at the following URL. User your VM’s IP address or ‘localhost’ if you are within the VM.
The Oracle VM comes loaded with Hudson version 2.2.0. I strongly suggest is updating Hudson to the latest version (3.0.1 at the time of this post). To upgrade, download, deploy, and started a new 3.0.1 version in the same domain on the same ‘AdminServer’ Server. I was able to do this remotely, from my development machine, using the browser-based Hudson Dashboard and WLS Administration Console. There is no need to do any of the installation from within the VM, itself.
When the upgrade is complete, stop the 2.2.0 deployment currently running in the WLS domain.
The new version of Hudson is accessible from the following URL (adjust the URL your exact version of Hudson):
It’s also important to update all the Hudson plugins. Hudson makes this easy with the Hudson Plugin Manager, accessible via the Manage Hudson’ option.
Note on the top of the Manage Hudson page, there is a warning about the server’s container not using UTF-8 to decode URLs. You can follow this post, if you want to resolve the issue by configuring Hudson differently. I did not worry about it for this post.
Building a Hudson Job
We are ready to configure Hudson to build, test, and deploy our Maven Web Application project. Return to the ‘Hudson Dashboard’, select ‘New Job’, and then ‘Build a new free-style software job’. This will open the ‘Job Configurations’ for the new job.
Start by configuring the ‘Source Code Management’ section. The Subversion Repository URL is the same as the one you used in NetBeans to commit the code. To avoid the access error seen below, you must provide the Subversion credentials to Hudson, just as you did in NetBeans.
Next, configure the Maven 3 Goals. I chose the ‘clean’ and ‘install’ goals. Using ‘clean’ insures the project is compiled each time by deleting the output of the build directory.
Optionally, you can configure Hudson to publish the JUnit test results as shown below. Be sure to save your configuration.
Start a build of the new Hudson Job, by clicking ‘Build Now’. If your Hudson job’s configurations are correct, and the new WLS domain is running, you should have a clean build. This means the project compiled without error, all tests passed, and the web application’s WAR file was deployed successfully to the new WLS domain within the Oracle VM.
To view the newly deployed Maven Web Application, log into the WebLogic Server Administration Console for the new domain. In my case, the new domain was running on port 7031, so the URL would be:
You should see the deployment, in an ‘Active’ state, as shown below.
To test the deployment, open a new browser tab and go to the URL of the Servlet. In this case the URL would be:
You should see the original phrase from the previous project displayed, ‘Hello WebLogic Server!’.
To further test the system, make a simple change to the project in NetBeans. I changed the name variable’s default value from ‘WebLogic Server’ to ‘Hudson, Maven, and WLS’. Commit the change to SVN.
Return to Hudson and run a new build of the job.
After the build completes, refresh the sample Web Application’s browser window. You should see the new text string displayed. Your code change was just re-compiled, re-tested, and re-deployed by Hudson.
True Continuous Deployment
Although Hudson is now doing a lot of the work for us, the system still is not fully automated. We are still manually building our Hudson Job, in order to deploy our application. If you want true continuous integration and deployment, you need to trust the system to automatically deploy the project, based on certain criteria.
SCM polling with Hudson is one way to demonstrate continuous deployment. In ‘Job Configurations’, turn on ‘Poll SCM’ and enter Unix cron-like value(s) in the ‘Schedule’ text box. In the example below, I have indicated a polling frequency every hour (‘@hourly’). Every hour, Hudson will look for committed changes to the project in Subversion. If changes are found, Hudson w retrieves the source code, compiles, and tests. If the project compiles and passes all tests, it is deployed to WLS.
There are less resource-intense methods to react to changes than SCM polling. Push-notifications from the repository is alternate, more preferable method.
Additionally, you should configure messaging in Hudson to notify team members of new deployments and the changes they contain. You should also implement a good deployment versioning strategy, for tracking purposes. Knowing the version of deployed artifacts is critical for accurate change management and defect tracking.
Create a new WebLogic Server domain on Oracle’s Pre-built Development VM. Remotely deploy a sample web application to the domain from a remote machine.
In my last two posts, Using Oracle’s Pre-Built Enterprise Java VM for Development Testing and Resizing Oracle’s Pre-Built Development Virtual Machines, I introduced Oracle’s Pre-Built Enterprise Java Development VM, aka a ‘virtual appliance’. Oracle has provided ready-made VMs that would take a team of IT professionals days to assemble. The Oracle Linux 5 OS-based VM has almost everything that comprises basic enterprise test and production environment based on the Oracle/Java technology stack. The VM includes Java JDK 1.6+, WebLogic Server, Coherence, TopLink, Subversion, Hudson, Maven, NetBeans, Enterprise Pack for Eclipse, and so forth.
One of the first things you will probably want to do, once your Oracle’s Pre-Built Enterprise Java Development VM is up and running, is deploy an application to WebLogic Server. According to Oracle, WebLogic Server is ‘a scalable, enterprise-ready Java Platform, Enterprise Edition (Java EE) application server.’ Even if you haven’t used WebLogic Server before, don’t worry, Oracle has designed it to be easy to get started.
In this post I will cover creating a new WebLogic Server (WLS) domain, and the deployment a simple application to WLS from a remote development machine. The major steps in the process presented in this post are as follows:
- Create a new WLS domain
- Create and build a sample application
- Deploy the sample application to the new WLS domain
- Access deployed application via a web browser
First, let me review how I have my VM configured for networking, so you will understand my deployment methodology, discussed later. The way you configure your Oracle VM VirtualBox appliance will depend on your network topology. Again, keeping it simple for this post, I have given the Oracle VM a static IP address (192.168.1.88). The machine on which I am hosting VirtualBox uses DHCP to obtain an IP address on the same local wireless network.
For the VM’s VirtualBox networking mode, I have chosen the ‘Bridged Adapter‘ mode. Using this mode, any machine on the network can access the VM through the host machine, via the VM’s IP address. One of the best posts I have read on VM networking is on Oracle’s The Fat Bloke Sings blog, here.
Creating New WLS Domain
A domain, according Oracle, is ‘the basic administrative unit of WebLogic Server. It consists of one or more WebLogic Server instances, and logically related resources and services that are managed, collectively, as one unit.’ Although the Oracle Development VM comes with pre-existing domains, we will create our own for this post.
To create the new domain, we will use the Oracle’s Fusion Middleware Configuration Wizard. The Wizard will take you through a step-by-step process to configure your new domain. To start the wizard, from within the Oracle VM, open a terminal window, and use the following command to switch to the Wizard’s home directory and start the application.
There are a lot of configuration options available, using the Wizard. I have selected some basic settings, shown below, to configure the new domain. Feel free to change the settings as you step through the Wizard, to meet your own needs. Make sure to use the ‘Development Mode’ Start Mode Option for this post. Also, make sure to note the admin port of the domain, the domain’s location, and the username and password you choose.
Starting the Domain
To start the new domain, open a terminal window in the VM and run the following command to change to the root directory of the new domain and start the WLS domain instance. Your domain path and domain name may be different. The start script command will bring up a new terminal window, showing you the domain starting.
WLS Administration Console
Once the domain starts, test it by opening a web browser from the host machine and entering the URL of the WLS Administration Console. If your networking is set-up correctly, the host machine will able to connect to the VM and open the domain, running on the port you indicated when creating the domain, on the static IP address of the VM. If your IP address and port are different, make sure to change the URL. To log into WLS Administration Console, use the username and password you chose when you created the domain.
Before we start looking around the new domain however, let’s install an application into it.
Sample Java Application
If you have an existing application you want to install, you can skip this part. If you don’t, we will quickly create a simple Java EE Hello World web application, using a pre-existing sample project in NetBeans – no coding required. From your development machine, create a new Samples -> Web Services -> REST: Hello World (Java EE 6) Project. You now have a web project containing a simple RESTful web service, Servlet, and Java Server Page (.jsp). Build the project in NetBeans. We will upload the resulting .war file manually, in the next step.
In a previous post, Automated Deployment to GlassFish Using Jenkins CI Server and Apache Ant, we used the same sample web application to demonstrate automated deployments to Oracle’s GlassFish application server.
Deploying the Application
There are several methods to deploy applications to WLS, depending on your development workflow. For this post, we will keep it simple. We will manually deploy our web application’s .war file to WLS using the browser-based WLS Administration Console. In a future post, we will use Hudson, also included on the VM, to build and deploy an application, but for now we will do it ourselves.
To deploy the application, switch back to the WLS Administration Console. Following the screen grabs below, you will select the .war file, built from the above web application, and upload it to the Oracle VM’s new WLS domain. The .war file has all the necessary files, including the RESTful web service, Servlet, and the .jsp page. Make sure to deploy it as an ‘application’ as opposed to a ‘library’ (see ‘target style’ configuration screen, below).
Accessing the Application
Now that we have deployed the Hello World application, we will access it from our browser. From any machine on the same network, point a browser to the following URL. Adjust your URL if your VM’s IP address and domain’s port is different.
The Hello World RESTful web service’s Web Application Description Language (WADL) description can be viewed at:
Since the Oracle VM is accessible from anywhere on the network, the deployed application is also accessible from any device on the network, as demonstrated below.
This was a simple demonstration of deploying an application to WebLogic Server on Oracle’s Pre-Built Enterprise Java Development VM. WebLogic Server is a powerful, feature-rich Java application server. Once you understand how to configure and administer WLS, you can deploy more complex applications. In future posts we will show a more common, slightly more complex example of automated deployment from Hudson. In addition, we will show how to create a datasource in WLS and access it from the deployed application, to talk to a relational database.
Expand the size of Oracle’s Pre-Built Development VM’s (virtual appliances) to accommodate software package updates and additional software installations.
In my last post, Using Oracle’s Pre-Built Enterprise Java VM for Development Testing, I discussed issues with the small footprint of the VM. Oracle’s Pre-Built Enterprise Java Development VM, aka virtual appliance, is comprised of (2) 8 GB Virtual Machine Disks (VMDK). The small size of the VM made it impossible to update the system software, or install new applications. After much trial and error, and a few late nights reading posts by others who had run into this problem, I found a fairly easy series of steps to increase the size of the VM. In the following example, I’ll demonstrate how to resize the VM’s virtual disks to 15 GB each; you can resize the disks to whatever size you need.
Before we start, a few quick terms. It’s not critical to have a complete understanding of Linux’s Logical Volume Manager (LVM). However, without some basic knowledge of how Linux manages virtual disks and physical storage, the following process might seem a bit confusing. I pulled the definitions directly from Wikipedia and LVM How-To.
- Logical Volume Manager (LVM) – LVM is a logical volume manager for the Linux kernel; it manages disk drivers and similar mass-storage devices.
- Volume Group (VG) – Highest level abstraction within the LVM. Gathers Logical Volumes (LV) and Physical Volumes (PV) into one administrative unit.
- Physical Volume (PV) – Typically a hard disk, or any device that ‘looks’ like a hard disk (eg. a software raid device).
- Logical Volume (LV) – Equivalent of disk partition in a non-LVM system. Visible as a standard block device; as such the LV can contain a file system (eg. /home).
- Logical Extents (LE) – Each LV is split into chunks of data, known as logical extents. The extent size is the same for all logical volumes in the volume group. The system pools LEs into a VG. The pooled LEs can then be concatenated together into virtual disk partitions (LV).
- Virtual Machine Disk (VMDK) – Container for virtual hard disk drives used in virtual machines. Developed by VMware for its virtual appliance products, but is now an open format.
- Virtual Disk Image (VDI) – VirtualBox-specific container format for storing files on the host operating system.
Ten Simple Steps
After having downloaded, assembled, and imported the original virtual appliance into VirtualBox Manager on my Windows computer, I followed the these simple steps to enlarge the size of the VM’s (2) 8 GB VMDK files:
- Locate the VM’s virtual disk images;
- Clone VMDK files to VDI files;
- Resize VDI files;
- Replace VM’s original VMDK files;
- Boot VM using GParted Live;
- Resize VDI partitions;
- Resize Logical Volume (LV);
- Resize file system;
- Restart the VM;
- Delete the original VMDK files.
1. Locate the VM’s virtual disk images
- From the Windows command line, run the following command to find information about virtual disk images currently in use by VirtualBox.
- Locate the (2) VMDK images associated with the VM you are going to resize.
cd C:\Program Files\Oracle\VirtualBox VBoxManage list hdds
2. Clone the (2) VMDK files to VDI files (substitute your own file paths and VMDK file names)
VBoxManage clonehd "C:\Users\your_username\VirtualBox VMs\VDD_WLS_labs_2012\VDD_WLS_labs_2012-disk1.vmdk" "C:\Users\your_username\VirtualBox VMs\VDD_WLS_labs_2012\VDD_WLS_labs_2012-disk1_ext.vdi" --format vdi VBoxManage clonehd "C:\Users\your_username\VirtualBox VMs\VDD_WLS_labs_2012\VDD_WLS_labs_2012-disk2.vmdk" "C:\Users\your_username\VirtualBox VMs\VDD_WLS_labs_2012\VDD_WLS_labs_2012-disk2_ext.vdi" --format vdi
3. Resize the (2) VDI files
To calculate the new size, multiple the number of gigabytes you want to end up with by 1,024 (ie. 15 x 1,024 = 15,360)
VBoxManage modifyhd "C:\Users\your_username\VirtualBox VMs\VDD_WLS_labs_2012\VDD_WLS_labs_2012-disk1_ext.vdi" --resize 15360 VBoxManage modifyhd "C:\Users\your_username\VirtualBox VMs\VDD_WLS_labs_2012\VDD_WLS_labs_2012-disk2_ext.vdi" --resize 15360
4. Replace the VM’s original VMDK files with the VDI files
- Using VirtualBox Manger -> Settings -> Storage, replace the (2) original VMDK files with the new VDI files.
- Keep the Attributes -> Hard Disk set to the same SATA Port 0 and SATA Port 1 of the original VMDK files.
5. Boot VM using the GParted Live ISO image
- Download GParted Live ISO image to the host system (Windows).
- Mount the GParted ISO (Devices -> CD/DVD Devices -> Chose a virtual CD/DVD disk file…).
- Start the VM, hit f12 as VM boots up, and select CD/DVD (option c).
- Answer a few questions for GParted to boot up properly.
6. Using GParted, resize the (2) VDI partitions
- Expand the ‘/dev/sda2’ and ‘/dev/sdb1’ partitions to use all unallocated space (in gray below).
7. Resize Logical Volume (LV)
- Using the Terminal, resize the Logical Volume to the new size of the VDI (‘/dev/sda’).
lvresize -L +7GB /dev/VolGroup00/LogVol00
8. Resize file system
- Using the Terminal, resize the file system to match Logical Volume size.
resize2fs -p /dev/mapper/VolGroup00-LogVol00
9. Restart the VM
- Restart the VM, hit f12 as it’s booting up, and select ‘1) Hard disk’ as the boot device.
- Open the Terminal and enter ‘df -hT’ to confirm your results.
- You can now safely install YUM Server configuration to update software packages, and install new applications.
10. Delete the original VMDK files
- Once you sure everything worked, don’t forget to delete the original VMDK files. They are taking up 16 GB of disk space.
Here are links to the two posts where I found most of the above information:
Install and configure Oracle’s Pre-Built Enterprise Java Development VM, with Oracle Linux 5, to create quick, full-featured development test environments.
Virtual Machines for Software Developers
As software engineers, we spend a great deal of time configuring our development machines to simulate test and production environments in which our code will eventually run. With the Microsoft/.NET technology stack, that most often means installing and configuring .NET, IIS, and SQL Server. With the Oracle/Java technology stack – Java, WebLogic or GlassFish Application Server, and Oracle 11g.
Within the last few years, the growth of virtual machines (VMs) within IT/IS organizations has exploded. According to Wikipedia, a virtual machine (VM), is ‘a software implementation of a machine (i.e. a computer) that executes programs like a physical machine.’ Rapid and inexpensive virtualization of business infrastructure using VMs has led to the exponential growth of private and public cloud platforms.
Instead of attempting to configure development machines to simulate the test and production environments, which are simultaneously running development applications and often personal programs, software engineers can leverage VMs in the same way as IT/IS organizations. Free, open-source virtualization software products from Oracle and VMware offer developers the ability to easily ‘spin-up’ fresh environments to compile, deploy, and test code. Code is tested in a pristine environment, closely configured to match production, without the overhead and baggage of day-to-day development. When testing is complete, the VM is simply deleted and a new copy re-deployed for the next project.
Oracle Pre-Built Virtual Appliances
I’ve worked with a number of virtualization products, based on various Windows and Linux operating systems. Not matter the product or OS, the VM still needs to be set up just like any other new computer system, with software and configuration. However, recently I began using Oracle’s pre-built developer VMs. Oracle offers a number of pre-built VMs for various purposes, including database development, enterprise Java development, business intelligence, application hosting, SOA development, and even PHP development. The VMs, called virtual appliances, are Open Virtualization Format Archive files, built to work with Oracle VM VirtualBox. Simply import the appliance into VirtualBox and start it up.
Oracle has provided ready-made VMs that would take even the most experienced team of IT professionals days to download, install, configure, and integrate. All the configuration details, user accounts information, instructions for use, and even pre-loaded tutorials, are provided. Oracle notes on their site that these VMs are not intended for use in production. However, the VMs are more than robust enough to use as a development test environment.
Because of its similarity to my production environment, I the installed the Enterprise Java Development VM on a Windows 7 Enterprise-based development computer. The Oracle Linux 5 OS-based VM has almost everything that comprises basic enterprise test and production environment based on the Oracle/Java technology stack. The VM includes an application server, source control server, build automation server, Java SDK, two popular IDE’s, and related components. The VM includes Java JDK 1.6+, WebLogic Server, Coherence, TopLink, Subversion, Hudson, Maven, NetBeans, Enterprise Pack for Eclipse, and so forth.
Aside from a database server, the environment has everything most developers might need to develop, build, store, and host their code. If you need a database, as most of us do, you can install it into the VM, or better yet, implement the Database App Development VM, in parallel. The Database VM contains Oracle’s 11g Release 2 enterprise-level relational database, along with several related database development and management tools. Using a persistence layer (data access layer), built with the included EclipseLink, you can connect the Enterprise appliance to the database appliance.
I followed the following steps to setup my VM:
- Update (or download and install) Oracle VM VirtualBox to the latest release.
- Download (6) Open Virtualization Format Archive (OVF/.ova) files.
- Download script to combine the .ova files.
- Execute script to assemble (6) .ova files into single. ova file.
- Import the appliance (combined .ova file) into VirtualBox.
- Optional: Clone and resize the appliance’s (2) virtual machines disks (see note below).
- Optional: Add the Yum Server configuration to the VM to enable normal software updates (see instructions below).
- Change any necessary settings within VM: date/time, timezone, etc.
- Install and/or update system software and development applications within VM: Java 1.7, etc.
Issue with Small Footprint of VM
The small size of the of pre-built VM is major issue I ran into almost immediately. Note in the screen grab above of VirtualBox, the Oracle VM only has (2) 8 GB virtual machine disks (.vmdk). Although Oracle designed the VMs to have a small footprint, it was so small that I quickly filled up its primary partition. At that point, the VM was too full to apply the even the normal system updates. I switched the cache location for yum to a different partition, but then ran out of space again when yum tried to apply the updates it had downloaded to the primary partition.
Lack of disk space was a complete show-stopper for me until I researched a fix. Unfortunately, VirtualBox’s ‘VBoxManage modifyhd –resize’ command is not yet compatible with the virtual machine disk (.vmdk) format. After much trial and error, and a few late nights reading posts by others who had run into this problem, I found a fairly easy series of steps to enlarge the size of the VM. It doesn’t require you to be a complete ‘Linux Geek’, and only takes about 30 minutes of copying and restarting the VM a few times. I will included the instructions in this separate, upcoming post.
Issue with Package Updater
While solving the VM’s disk space issue, I also discoverer the VM’s Enterprise Linux System was setup with something called the Unbreakable Linux Network (ULN) Update Agent. From what I understood, without a service agreement with Oracle, I could not update the VM’s software using the standard Package Updater. However, a few quick commands I found on the Yum Server site, overcame that limitation and allowed me to update the VM’s software. Just follow the simple instructions here, for Oracle Linux 5. There are several hundred updates that will be applied, including an upgrade of Oracle Linux from 5.5 to 5.9.
Issue with Java Updates
Along with the software updates, I ran into an issue installing the latest version of Java. I attempted to install the standard Oracle package that contained the latest Java JDK, JRE, and NetBeans for Linux. Upon starting the install script, I immediately received a ‘SELinux AVC denial’ message. This security measure halted my installation with the following error: ‘The java application attempted to load /labs/java/jre1.7.0_21/lib/i386/client/libjvm.so which requires text relocation. This is a potential security problem.‘
To get around the SELinux AVC denial issue, I installed the JRE, JDK, and NetBeans separately. Although this took longer and required a number of steps, it allowed me to get around the security and install the latest version of Java.
Note I later discovered that I could have simply changed the SELinux Security Level to ‘Permissive’ in the SELinux Security and Firewall, part of the Administrative Preferences. This would have allowed the original Oracle package containing the JDK, JRE, and NetBeans, to run.