Posts Tagged JSON
Building a Microservices-based REST API with RestExpress, Java EE, and MongoDB: Part 3
Posted by Gary A. Stafford in Enterprise Software Development, Java Development, Software Development on June 5, 2015
Develop a well-architected and well-documented REST API, built on a tightly integrated collection of Java EE-based microservices.
Note: All code available on GitHub. For the version of the code that matches the details in this blog post, check out the master branch, v1.0.0 tag (after running git clone …, run a git checkout tags/v1.0.0
command).
Previous Posts
In Part One of this series, we introduced the microservices-based Virtual-Vehicles REST API example. The vehicle-themed Virtual-Vehicles microservices offers a comprehensive set of functionality, through a REST API, to application developers. In Part Two, we installed a copy of the Virtual-Vehicles project from GitHub. In Part Two, we also gained a basic understanding of how RestExpress works. Finally, we discovered how to get the Virtual-Vehicles microservices up and running.
Part Three
In part three of this series, we will take the Virtual-Vehicles for a test drive (get it? maybe it was funnier the first time…). There are several tools we can use to test the Virtual-Vehicles API. One of my favorite tools is Postman. We will explore how to use Postman, along with the Virtual-Vehicles API documentation, to test the Virtual-Vehicles microservice’s endpoints, which compose the Virtual-Vehicles API.
Testing the API
There are three categories of tools available to test RESTful APIs, which are GUI-based applications, command line tools, and testing frameworks. Postman, Advanced REST Client, REST Console, and SmartBear’s SoapUI and SoapUI NG Pro, are examples of GUI-based applications, designed specifically to test RESTful APIs. cURL and GNU Wget are two examples of command line tools, which among other capabilities, can test APIs. Lastly, JUnit is an example of a testing framework that can be used to test a RESTful API. Surprisingly, JUnit is not only designed to manage unit tests. Each category of testing tools has their pros and cons, depending on your testing needs. We will explore all of these categories in this post as we test the Virtual-Vehicles REST API.
JUnit
JUnit is probably the best known of all Java unit testing frameworks. JUnit’s website describes JUnit as ‘a simple, open source framework to write and run repeatable tests. It is an instance of the xUnit architecture for unit testing frameworks.’ Most Java developers turn to JUnit for unit testing. However, JUnit is capable of other forms of testing, including integration testing. In his post, ‘Unit Testing with JUnit – Tutorial’, Lars Vogel states ‘an integration test has the target to test the behavior of a component or the integration between a set of components. The term functional test is sometimes used as a synonym for integration test. This kind of tests allow you to translate your user stories into a test suite, i.e., the test would resemble an expected user interaction with the application.’
Testing the Virtual-Vehicles RESTful API’s operations with JUnit would be considered integration (functional) testing. At a minimum, to complete requests, we call one microservice, which in turn authenticates the JWT by calling another microservice. If authenticated, the first microservice makes a request to its MongoDB database. As Vogel stated, whereas a unit test targets a small unit of code, such as a method, the request/response operation is integration between a set of components. testing an API call requires several dependencies.
The simplest example of testing the Virtual-Vehicles API with JUnit, would be to test an HTTP GET request to return a single instance of a vehicle. The code below demonstrates how this might be done. Notice the request depends on helper methods (not included, for brevity). To request the vehicle, assuming we already have a registered client, we need a valid JWT. We also need a valid vehicle ObjectId
. To obtain these two pieces of data, we call helper methods, which in turn makes the necessary request to retrieve a JWT and vehicle ObjectId
.
/** | |
* Test of HTTP GET to read a single vehicle. | |
*/ | |
@Test | |
public void testVehicleRead() { | |
String responseBody = ""; | |
String output; | |
Boolean result = true; | |
Boolean expResult = true; | |
try { | |
URL url = new URL(getBaseUrlAndPort() + "/vehicles/" + getVehicleObjectId()); | |
HttpURLConnection conn = (HttpURLConnection) url.openConnection(); | |
conn.setRequestMethod("GET"); | |
conn.setRequestProperty("Authorization", "Bearer " + getJwt()); | |
conn.setRequestProperty("Accept", "application/json"); | |
if (conn.getResponseCode() != 200) { | |
// if not 200 response code then fail test | |
result = false; | |
} | |
BufferedReader br = new BufferedReader(new InputStreamReader( | |
(conn.getInputStream()))); | |
while ((output = br.readLine()) != null) { | |
responseBody = output; | |
} | |
if (responseBody.length() < 1) { | |
// if response body is empty then fail test | |
result = false; | |
} | |
conn.disconnect(); | |
} catch (IOException e) { | |
// if MalformedURLException, ConnectException, etc. then fail test | |
result = false; | |
} | |
assertEquals(expResult, result); | |
} |
Below are the results of the above test, run in NetBeans IDE, using the built-in support for JUnit.
JUnit can also be run from the command line using the Maven goal, surefire:test
:
mvn -q -Dtest=com.example.vehicle.objectid.VehicleControllerIT surefire:test |
cURL
One of the best-known command line tools for calling for all types of operations centered around calling a URL is cURL. According to their website, ‘curl is a command line tool and library for transferring data with URL syntax, supporting…HTTP, HTTPS…curl supports SSL certificates, HTTP POST, HTTP PUT, FTP uploading, HTTP form based upload, proxies, HTTP/2, cookies, user+password authentication (Basic, Plain, Digest, CRAM-MD5, NTLM, Negotiate, and Kerberos), file transfer resume, proxy tunneling and more.’ I prefer the website’s briefer description, cURL ‘groks those URLs’.
Using cURL, we could make an HTTP PUT request to the Vehicle microservice’s /vehicles/{oid}.{format}
endpoint. With cURL, we have the ability to add the JWT-based Authorization header and the raw request body, containing the modified vehicle object. Below is an example of that cURL command, which can be run from a terminal prompt.
curl --url 'http://virtual-vehicles.com:8581/vehicles/557310cfec7291b25cd3a1c2' \ | |
-X PUT \ | |
-H 'Pragma: no-cache' \ | |
-H 'Cache-Control: no-cache' \ | |
-H 'Accept: application/json; charset=UTF-8' \ | |
-H 'Authorization: Bearer eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpc3MiOiJ2aXJ0dWFsLXZlaGljbGVzLmNvbSIsImFwaUtleSI6IlJncjg0YzF6VkdtMFd1N25kWjd5UGNURSIsImV4cCI6MTQzMzY2ODEwNywiYWl0IjoxNDMzNjMyMTA3fQ.xglaKWufcj4TZtMXW3DLa9uy5JB_FJHxxtk_iF1WT6U' \ | |
--data-binary $'{ "year": 2015, "make": "Chevrolet", "model": "Corvette Stingray", "color": "White", "type": "Coupe", "mileage": 902, "createdAt": "2015-05-09T22:36:04.808Z" }' \ | |
--compressed |
The response body contains the expected modified vehicle object in JSON-format, along with a 201 Created
response status.
The cURL commands may be incorporated into many types of automated testing processes. These might be as simple as a bash script. The script could a series of automated tests, including the following: register an API client, use the API key to create a JWT, use the JWT to create a new vehicle, use the new vehicle’s ObjectId
to modify that same vehicle, delete that vehicle, confirm the vehicle is removed using the count operation and returns a test results report to the user.
cURL Commands from Chrome
Quick tip, instead of hand-coding complex cURL commands, containing form data, URL parameters, and Headers, use Chrome. First, open the Chrome Developer Tools (f12). Next, using the Postman – REST Client for Chrome, available in the Chrome App Store, execute your HTTP request. Finally, in the ‘Network’ tab of the Developers tools, find and right-click on the request and select ‘Copy as cURL’. You have a complete cURL command equivalent of your Postman request, which you can paste directly into the command line or insert into a script. Below is an example of using the Postman – REST Client for Chrome to generate a cURL command.
curl --url 'http://virtual-vehicles.com:8581/vehicles/554e8bd4c830093007d8b949' \ | |
-X PUT \ | |
-H 'Pragma: no-cache' \ | |
-H 'Origin: chrome-extension://fdmmgilgnpjigdojojpjoooidkmcomcm' \ | |
-H 'Accept-Encoding: gzip, deflate, sdch' \ | |
-H 'Accept-Language: en-US,en;q=0.8' \ | |
-H 'CSP: active' \ | |
-H 'Authorization: Bearer eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpc3MiOiJ2aXJ0dWFsLXZlaGljbGVzLmNvbSIsImFwaUtleSI6IlBUMklPSWRaRzZoU0VEZGR1c2h6U04xRyIsImV4cCI6MTQzMzU2MDg5NiwiYWl0IjoxNDMzNTI0ODk2fQ.4q6EMuxE0vS43zILjE6e1tYrb5ulCe69-1QTFLYGbFU' \ | |
-H 'Content-Type: text/plain;charset=UTF-8' \ | |
-H 'Accept: */*' \ | |
-H 'User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.81 Safari/537.36' \ | |
-H 'Cache-Control: no-cache' \ | |
-H 'Connection: keep-alive' \ | |
-H 'X-FirePHP-Version: 0.0.6' \ | |
--data-binary $'{ "year": 2015, "make": "Chevrolet", "model": "Corvette Stingray", "color": "White", "type": "Coupe", "mileage": 902, "createdAt": "2015-05-09T22:36:04.808Z" }' \ | |
--compressed |
The generated command is a bit verbose. Compare this command to the cURL command, earlier.
Wget
Similar to cURL, GNU Wget provides the ability to call the Virtual-Vehicles API’s endpoints. According to their website, ‘GNU Wget is a free software package for retrieving files using HTTP, HTTPS and FTP, the most widely-used Internet protocols. It is a non-interactive command line tool, so it may easily be called from scripts, cron jobs, terminals without X-Windows support, etc.’ Again, like cURL, we can run Wget commands from the command line or incorporate them into scripted testing processes. The Wget website contains excellent documentation.
Using Wget, we could make the same HTTP PUT request to the Vehicle microservice /vehicles/{oid}.{format}
endpoint. Like cURL, we have the ability to add the JWT-based Authorization header and the raw request body, containing the modified vehicle object.
wget -O - 'http://virtual-vehicles.com:8581/vehicles/557310cfec7291b25cd3a1c2' \ | |
--method=PUT \ | |
--header='Authorization: Bearer eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpc3MiOiJ2aXJ0dWFsLXZlaGljbGVzLmNvbSIsImFwaUtleSI6IlJncjg0YzF6VkdtMFd1N25kWjd5UGNURSIsImV4cCI6MTQzMzY2ODEwNywiYWl0IjoxNDMzNjMyMTA3fQ.xglaKWufcj4TZtMXW3DLa9uy5JB_FJHxxtk_iF1WT6U' \ | |
--header='Content-Type: text/plain;charset=UTF-8' \ | |
--header='Accept: application/json' \ | |
--body-data=$'{ "year": 2015, "make": "Chevrolet", "model": "Corvette Stingray", "color": "White", "type": "Coupe", "mileage": 902, "createdAt": "2015-05-09T22:36:04.808Z" }' |
The response body contains the expected modified vehicle object in JSON-format, along with a 201 Created
response status.
cURL Bash Testing
We can combine cURL and Wget with several of the tools bash provides, to develop fairly complex integration tests. The bash-based script below just scratches the surface as a complete set of integration tests. However, the tests demonstrate an efficient multi-stage test approach to handling the complex nature of RESTful service request requirements. The tests build upon each other.
After setting up some variables and doing a quick health check on one service, the tests register a new API client by calling the Authentication service. Next, they use the new client’s API key to obtain a JWT. The tests then use the JWT to authenticate themselves and create a new vehicle. Finally, they use the new vehicle’s id and the JWT to verify the existence for the new vehicle.
Although some may consider using bash to test somewhat primitive, the following script demonstrates the effectiveness of bash’s curl
, grep
, sed
, awk
, along with regular expressions, to test our RESTful services. Note how we grep certain values from the response, such as the new client’s API key, and then use that value as a parameter for the following test request, such as to obtain a JWT.
#!/bin/sh | |
######################################################################## | |
# | |
# title: Virtual-Vehicles Project Integration Tests | |
# author: Gary A. Stafford (https://programmaticponderings.com) | |
# url: https://github.com/garystafford/virtual-vehicles-docker | |
# description: Performs integration tests on the Virtual-Vehicles | |
# microservices | |
# to run: sh tests_color.sh -v | |
# | |
######################################################################## | |
echo --- Integration Tests --- | |
### VARIABLES ### | |
hostname="localhost" | |
application="Test API Client $(date +%s)" # randomized | |
secret="$(date +%s | sha256sum | base64 | head -c 15)" # randomized | |
echo hostname: ${hostname} | |
echo application: ${application} | |
echo secret: ${secret} | |
### TESTS ### | |
echo "TEST: GET request should return 'true' in the response body" | |
url="http://${hostname}:8581/vehicles/utils/ping.json" | |
echo ${url} | |
curl -X GET -H 'Accept: application/json; charset=UTF-8' \ | |
--url "${url}" \ | |
| grep true > /dev/null | |
[ "$?" -ne 0 ] && echo "RESULT: fail" && exit 1 | |
echo "RESULT: pass" | |
echo "TEST: POST request should return a new client in the response body with an 'id'" | |
url="http://${hostname}:8587/clients" | |
echo ${url} | |
curl -X POST -H "Cache-Control: no-cache" -d "{ | |
\"application\": \"${application}\", | |
\"secret\": \"${secret}\" | |
}" --url "${url}" \ | |
| grep '"id":"[a-zA-Z0-9]\{24\}"' > /dev/null | |
[ "$?" -ne 0 ] && echo "RESULT: fail" && exit 1 | |
echo "RESULT: pass" | |
echo "SETUP: Get the new client's apiKey for next test" | |
url="http://${hostname}:8587/clients" | |
echo ${url} | |
apiKey=$(curl -X POST -H "Cache-Control: no-cache" -d "{ | |
\"application\": \"${application}\", | |
\"secret\": \"${secret}\" | |
}" --url "${url}" \ | |
| grep -o '"apiKey":"[a-zA-Z0-9]\{24\}"' \ | |
| grep -o '[a-zA-Z0-9]\{24\}' \ | |
| sed -e 's/^"//' -e 's/"$//') | |
echo apiKey: ${apiKey} | |
echo | |
echo "TEST: GET request should return a new jwt in the response body" | |
url="http://${hostname}:8587/jwts?apiKey=${apiKey}&secret=${secret}" | |
echo ${url} | |
curl -X GET -H "Cache-Control: no-cache" \ | |
--url "${url}" \ | |
| grep '[a-zA-Z0-9_-]\{1,\}\.[a-zA-Z0-9_-]\{1,\}\.[a-zA-Z0-9_-]\{1,\}' > /dev/null | |
[ "$?" -ne 0 ] && echo "RESULT: fail" && exit 1 | |
echo "RESULT: pass" | |
echo "SETUP: Get a new jwt using the new client for the next test" | |
url="http://${hostname}:8587/jwts?apiKey=${apiKey}&secret=${secret}" | |
echo ${url} | |
jwt=$(curl -X GET -H "Cache-Control: no-cache" \ | |
--url "${url}" \ | |
| grep '[a-zA-Z0-9_-]\{1,\}\.[a-zA-Z0-9_-]\{1,\}\.[a-zA-Z0-9_-]\{1,\}' \ | |
| sed -e 's/^"//' -e 's/"$//') | |
echo jwt: ${jwt} | |
echo "TEST: POST request should return a new vehicle in the response body with an 'id'" | |
url="http://${hostname}:8581/vehicles" | |
echo ${url} | |
curl -X POST -H "Cache-Control: no-cache" \ | |
-H "Authorization: Bearer ${jwt}" \ | |
-d '{ | |
"year": 2015, | |
"make": "Test", | |
"model": "Foo", | |
"color": "White", | |
"type": "Sedan", | |
"mileage": 250 | |
}' --url "${url}" \ | |
| grep '"id":"[a-zA-Z0-9]\{24\}"' > /dev/null | |
[ "$?" -ne 0 ] && echo "RESULT: fail" && exit 1 | |
echo "RESULT: pass" | |
echo "SETUP: Get id from new vehicle for the next test" | |
url="http://${hostname}:8581/vehicles?filter=make::Test|model::Foo&limit=1" | |
echo ${url} | |
id=$(curl -X GET -H "Cache-Control: no-cache" \ | |
-H "Authorization: Bearer ${jwt}" \ | |
--url "${url}" \ | |
| grep '"id":"[a-zA-Z0-9]\{24\}"' \ | |
| grep -o '[a-zA-Z0-9]\{24\}' \ | |
| tail -1 \ | |
| sed -e 's/^"//' -e 's/"$//') | |
echo vehicle id: ${id} | |
echo "TEST: GET request should return a vehicle in the response body with the requested 'id'" | |
url="http://${hostname}:8581/vehicles/${id}" | |
echo ${url} | |
curl -X GET -H "Cache-Control: no-cache" \ | |
-H "Authorization: Bearer ${jwt}" \ | |
--url "${url}" \ | |
| grep '"id":"[a-zA-Z0-9]\{24\}"' > /dev/null | |
[ "$?" -ne 0 ] && echo "RESULT: fail" && exit 1 | |
echo "RESULT: pass" |
Since these tests are just a bash script, they can from the command line, or easily called from a continuous integration tool, Such as Jenkins CI or Hudson.
Postman
Postman, like several similar tools, is an application designed specifically for test API endpoints. The Postman website describes Postman as tool that allows you to ‘build, test, and document your APIs faster.’ There are two versions of Postman in the Chrome Web Store. They are Postman – REST Client, the in-browser extension, which we mentioned above, and Postman, the standalone application. There is also Postman Interceptor, which helps you send requests that use browser cookies through the Postman application.
Postman and similar applications, have add-ons and extensions to extend their features. In particular, Postman, which is free, offers the Jetpacks paid extension. Jetpacks add the ability to ‘write and run tests inside Postman, extract data from responses, chain requests together and test requests with thousands of variations’. Jetpacks allow you to move beyond basic one-off API request-based testing, to automated regression and performance testing.
Using Postman
Let’s use the same HTTP PUT example we used with cURL and Wget, and see how we would perform the same task with Postman. In the first screen grab below, you can see all elements of the HTTP request, including the RESTful API’s URL, URI including the vehicle’s ObjectId
(/vehicles/{ObjectId}.{format}
), HTTP method (PUT), Authorization Header with JWT (Bearer), and the raw request body. The raw request body contains a JSON representation of the vehicle we want to update. Note how Postman saves the request in history so we can easily replay it later.
In the next screen-grab, we see the response to the HTTP PUT request. Note the response body, response status, timing, and response headers.
Looking at the response body in Postman, you easily see the how RestExpress demonstrates the RESTful principle we discussed in Part Two of the series, HATEOAS (Hypermedia as the Engine of Application State). Note the link to this vehicle’s ‘self’ href) and the entire vehicles collection (‘up’ href).
Postman Collections
A great feature of Postman with Jetpacks is Collections. Collections are sets of requests, which can be saved, recalled, and shared. The Collection Runner runs requests in a collection, in the order in which you set them. Ordered collections are ideal for the Virtual-Vehicles API. The screen grab below shows a collection of requests, arranged in the order we would execute them to test the Virtual-Vehicles API, as it applies to specifically to vehicle CRUD operations:
- Execute HTTP POST request to register the new API client, passing the application name and a shared secret in the request
Receive the new client’s API key in response - Execute HTTP GET to request, passing the new client’s API key and the shared secret in the request
Receive the new JWT in response - Execute HTTP POST request to create a new vehicle, passing the JWT in the header for authentication (used for all following requests)
Receive the new vehicle object in response - Execute HTTP PUT request to modify the new vehicle, using the vehicle’s
ObjectId
Receive the modified vehicle object in response - Execute HTTP GET to request the modified vehicle, to confirm it exists in the expected state
Receive the vehicle object in response - Execute HTTP DELETE request to delete the new vehicle, using the vehicle’s
ObjectId
- Execute HTTP GET to request the new vehicle and to confirm it has been removed
Receive a 404 Not Found status response, as expected
Using saved collections for testing the Virtual-Vehicles API is a real-time saving. However, the collections cannot easily be re-run without hand-editing or some advanced scripting. In the simple example above, we hard-coded a JWT and vehicle ObjectId
in the requests. Unfortunately, the JWT has an expiration of only 10 hours by default. More immediately, the ObjectId
is unique. The earlier collection test run created, then deleted, the vehicle with that ObjectId
.
Negative Testing
You may also perform negative testing with Postman. For example, do you receive the expected response when you don’t include the Authorization Header with JWT in a request (401 Unauthorized status)? When you include a JWT, which has expired (401 Unauthorized status)? When you request a vehicle, whose ObjectId
is incorrect or is not found in the database (400 Bad Request status)? Do you receive the expected response when you call an actual service, but an endpoint that doesn’t exist (405 Method Not Allowed)?
Postman Test Automation
In addition to manually viewing the HTTP response, to verify the results of a request, Postman allows you to write and run automated tests for each request. According to their website, a ‘Postman test is essentially JavaScript code which sets values for the special tests object. You can set a descriptive key for an element in the object and then say if it’s true or false’. This allows you to write a set of response validation tests for each request.
Below is a quick example of testing the same HTTP POST request, used to create the new API client, above. In this example, we:
- Test that the
Content-Type
response header is present - Test that the HTTP POST successfully returned a 201 status code
- Test that the new client’s API key was returned in the response body
- Test that the response time was less than 200ms
Reviewing Postman’s ‘Tests’ tab, above, observe the four tests have run successfully. Using the Postman’s testing feature, you can create even more advanced tests, eliminating the need to manually validate responses.
This post demonstrates a small subset of the features Postman and other similar applications provide for testing RESTful API. The tools and processes you use to test your RESTful API will depend on the stage of development and testing you are in, as well as the existing technology stacks you build, and on which you host your services.
Building a Microservices-based REST API with RestExpress, Java EE, and MongoDB: Part 2
Posted by Gary A. Stafford in Enterprise Software Development, Java Development, Software Development on May 31, 2015
Develop a well-architected and well-documented REST API, built on a tightly integrated collection of Java EE-based microservices.
Note: All code available on GitHub. For the version of the code that matches the details in this blog post, check out the master branch, v1.0.0 tag (after running git clone …, run a ‘git checkout tags/v1.0.0’ command).
Previous Post
In Part One of this series, we introduced the microservices-based Virtual-Vehicles REST API example. The vehicle-themed Virtual-Vehicles microservices offers a comprehensive set of functionality, through a REST API, to application developers. The developers, in turn, will use the Virtual-Vehicles REST API’s functionality to build applications and games for their end-users.
In Part One, we also decided on the proper amount and responsibility of each microservice. We also determined the functionality of each microservice to meet the hypothetical functional and nonfunctional requirements of Virtual-Vehicles. To review, the four microservices we are building, are as follows:
Virtual-Vehicles REST API Resources |
||
Microservice | Purpose (Business Capability) | Functions |
Authentication |
Manage API clients and JWT authentication |
|
Vehicle |
Manage virtual vehicles |
|
Maintenance |
Manage maintenance on vehicles |
|
Valet Parking |
Manage a valet service to park for vehicles |
|
To review, the first five functions for each service are all basic CRUD operations: create
(POST), read
(GET), readAll
(GET), update
(PUT), delete
(DELETE). The readAll
function also has find
, count
, and pagination
functionality using query parameters. Unfortunately, RestExpress does not support PATCH
for updates. However, I have updated RestExpress’ PUT HTTP methods to return the modified object in the response body instead of the nothing (status of 201 Created vs. 200 OK). See StackOverflow for an explanation.
All services also have an internal authenticateJwt
function, to authenticate the JWT, passed in the HTTP request header, before performing any operation. Additionally, all services have a basic health-check function, ping
(GET). There are only a few other functions required for our Virtual-Vehicles example, such as for creating JWTs.
Part Two Introduction
In Part Two, we will build our four Virtual-Vehicles microservices. Recall from our first post, we will be using RestExpress. RestExpress composes best-of-breed open-source tools to enable quickly creating RESTful microservices that embrace industry best practices. Those best-of-breed tools include Java EE, Maven, MongoDB, and Netty, among others.
In this post, we will accomplish the following:
- Create a default microservice project in NetBeans using RestExpress MongoDB Maven Archetype
- Understand the basic structure of a default RestExpress microservice project
- Review the changes made to the default RestExpress microservice project to create the Virtual-Vehicles example
- Compile and run the individual microservices directly from NetBeans
I used NetBeans IDE 8.0.2 on Linux Ubuntu 14.10 to build the microservices. You may also follow along in other IDE’s, such as Eclipse or IntelliJ, on Mac or Windows. We won’t cover installing MongoDB, Maven, and Java. I’ll assume if your building enterprise applications, you have the basics covered.
Using the RestExpress MongoDB Maven Archetype
All the code for this project is available on GitHub. However, to understand RestExpress, you should go through the exercise of scaffolding a new microservice using the RestExpress MongoDB Maven Archetype. You will also be able to use this default microservice project to compare and contrast to the modified versions, used in the Virtual-Vehicles example. The screen grabs below demonstrate how to create a new microservice project using the RestExpress MongoDB Maven Archetype. At the time of this post, the archetype version was restexpress-mongodb version 1.15.
Default Project Architecture
Reviewing the two screen grabs below (Project tab), note the key components of the RestExpress MongoDB Maven project, which we just created:
- Base Package (com.example.vehicle)
- Configuration class reads in environment properties (see Files tab) and instantiates controllers
- Constants class contains project constants
- Relationships class defines linking resource which aids service discoverability (HATEOAS)
- Main executable class
- Routes class defines the routes (endpoints) exposed by the service and the corresponding controller class
- Model/Controllers Packages (com.example.vehicle.objectid and .uuid)
- Entity class defines the data entity – a Vehicle in this case
- Controller class contains the methods executed when the route (endpoint) is called
- Repository class defines the connection details to MongoDB
- Service class contains the calls to the persistence layer, validation, and business logic
- Serialization Package (com.example.vehicle.serialization)
- XML and JSON serialization classes
- Object ID and UUID serialization and deserialization classes
Again, I strongly recommend reviewing each of these package’s classes. To understand the core functionality of RestExpress, you must understand the relationships between RestExpress microservice’s Route, Controller, Service, Repository, Relationships, and Entity classes. In addition to reviewing the default Maven project, there are limited materials available on the Internet. I would recommend the RestExpress Website on GitHub, RestExpress Google Group Forum, and the YouTube 3-part video series, Instant REST Services with RESTExpress.
Unit Tests?
Disappointingly, the current RestExpress MongoDB Maven Archetype sample project does not come with sample JUnit unit tests. I am tempted to start writing my own unit tests if I decided to continue to use the RestExpress microservices framework for future projects.
Properties Files
Also included in the default RestExpress MongoDB Maven project is a Java properties file (environment.properties
). This properties file is displayed in the Files tab, as shown below. The default properties file is located in the ‘dev’ environment config folder. Later, we will create an additional properties file for our production environment.
Ports
Within the ‘dev’ environment, each microservice is configured to start on separate ports (i.e. port = 8581
). Feel free to change the service’s port mappings if they conflict with previously configured components running on your system. Be careful when changing the Authentication service’s port, 8587, since this port is also mapped in all other microservices using the authentication.port
property (authentication.port = 8587
). Make sure you change both properties, if you change Authentication service’s port mapping.
Base URL
Also, in the properties files is the base.url
property. This property defines the URL the microservice’s endpoints will be expecting calls from, and making internal calls between services. In our post’s example, this property in the ‘dev’ environment is set to localhost (base.url = http://localhost
). You could map an alternate hostname from your hosts file (/etc/hosts
). We will do this in a later post, in our ‘prod’ environment, mapping the base.url
property to Virtual-Vehicles (base.url = http://virtual-vehicles.com
). In the ‘dev’ environment properties file, MongoDB is also mapped to localhost
(i.e. mongodb.uri = mongodb://virtual-vehicles.com:27017/virtual_vehicle
).
Metrics Plugin and Graphite
RestExpress also uses the properties file to hold configuration properties for Metrics Plugin and Graphite. The Metrics Plugin and Graphite are both first class citizens of RestExpress. Below is the copy of the Vehicles service environment.properties
file for the ‘dev’ environment. Note, the Metrics Plugin and Graphite are both disabled in the ‘dev’ environment.
# Default is 8081 | |
port = 8581 | |
# Port used to call Authentication service endpoints | |
authentication.port = 8587 | |
# The size of the executor thread pool | |
# (that can handle blocking back-end processing). | |
executor.threadPool.size = 20 | |
# A MongoDB URI/Connection string | |
# see: http://docs.mongodb.org/manual/reference/connection-string/ | |
mongodb.uri = mongodb://localhost:27017/virtual_vehicle | |
# The base URL, used as a prefix for links returned in data | |
base.url = http://localhost | |
#Configuration for the MetricsPlugin/Graphite | |
metrics.isEnabled = false | |
#metrics.machineName = | |
metrics.prefix = web1.example.com | |
metrics.graphite.isEnabled = false | |
metrics.graphite.host = graphite.example.com | |
metrics.graphite.port = 2003 | |
metrics.graphite.publishSeconds = 60 |
Choosing a Data Identification Method
RestExpress offers two identification models for managing data, the MongoDB ObjectId and a Universally Unique Identifier (UUID). MongoDB uses an ObjectId to identify uniquely a document within a collection. The ObjectId is a special 12-byte BSON type that guarantees uniqueness of the document within the collection. Alternately, you can use the UUID identification model. The UUID identification model in RestExpress uses a UUID, instead of a MongoDB ObjectId. The UUID also contains createdAt
and updatedAt
properties that are automatically maintained by the persistence layer. You may stick with ObjectId, as we will in the Virtual-Vehicles example, or choose the UUID. If you use multiple database engines for your projects, using UUID will give you a universal identification method.
Project Modifications
Many small code changes differentiate our Virtual-Vehicles microservices from the default RestExpress Maven Archetype project. Most changes are superficial; nothing changed about how RestExpress functions. Changes between the screen grabs above, showing the default project, and the screen grabs below, showing the final Virtual-Vehicles microservices, include:
- Remove all packages, classes, and code references to the UUID identification methods (example uses ObjectId)
- Rename several classes for convenience (dropped use of word ‘Entity’)
- Add the Utilities (com.example.utilities) and Authentication (com.example.authenticate) packages
MongoDB
Following a key principle of microservices mentioned in the first post, Decentralized Data Management, each microservice will have its own instance of a MongoDB database associated with it. The below diagram shows each service and its corresponding database, collection, and fields.
From the MongoDB shell, we can observe the individual instances of the four microservice’s databases.
In the case of the Vehicle microservice, the associated MongoDB database is virtual_vehicle
. This database contains a single collection, vehicles
. While the properties file defines the database name, the Vehicles entity class defines the collection name, using the org.mongodb.morphia.annotations
classes annotation functionality.
@Entity("vehicles") | |
public class Vehicle | |
extends AbstractMongodbEntity | |
implements Linkable { | |
private int year; | |
private String make; | |
private String model; | |
private String color; | |
private String type; | |
private int mileage; | |
// abridged... |
Looking at the virtual_vehicle
database in the MongoDB shell, we see that the sample document’s fields correspond to the Vehicle entity classes properties.
Each of the microservice’s MongoDB databases are configured in the environments.properties
file, using the mongodb.uri
property. In the ‘dev’ environment we use localhost
as our host URL (i.e. mongodb.uri = mongodb://localhost:27017/virtual_vehicle
).
Authentication and JSON Web Tokens
The three microservices, Vehicle, Valet, and Maintenance, are almost identical. However, the Authentication microservice is unique. This service is called by each of the other three services, as well as also being called directly. The Authentication service provides a very basic level of authentication using JSON Web Tokens (JWT), pronounced ‘jot‘.
Why do we want authentication? We want to confirm that the requester using the Virtual-Vehicles REST API is the actual registered API client they are who they claim to be. JWT allows us to achieve this requirement with minimal effort.
According to jwt.io, ‘a JSON Web Token is a compact URL-safe means of representing claims to be transferred between two parties. The claims in a JWT are encoded as a JSON object that is digitally signed using JSON Web Signature (JWS).‘ I recommend reviewing the JWT draft standard to fully understand the structure, creation, and use of JWTs.
Virtual-Vehicles Authentication Process
There are different approaches to implementing JWT. In our Virtual-Vehicles REST API example, we use the following process for JWT authentication:
- Register the new API client by supplying the application name and a shared secret (one time only)
- Receive an API key in response (one time only)
- Obtain a JWT using the API key and the shared secret (each user session or renew when the previous JWT expires)
- Include the JWT in each API call
In our example, we are passing four JSON fields in our set of claims. Those fields are the issuer (‘iss’), API key, expiration (‘exp’), and the time the JWT was issued (‘ait’). Both the ‘iss’ and the ‘exp’ claims are defined in the Authentication service’s environment.properties
file (jwt.expire.length
and jwt.issuer
).
Expiration and Issued date/time use the JWT standard recommended “Seconds Since the Epoch“. The default expiration for a Virtual-Vehicles JWT is set to an arbitrary 10 hours from the time the JWT was issued (jwt.expire.length = 36000
). That amount, 36,000, is equivalent to 10 hours x 60 minutes/hour x 60 seconds/minute.
Decoding a JWT
Using the jwt.io site’s JT Debugger tool, I have decoded a sample JWT issued by the Virtual-Vehicles REST API, generated by the Authentication service. Observe the three parts of the JWT, the JOSE Header, Claims Set, and the JSON Web Signature (JWS).
The JWT’s header indicates that our JWT is a JWS that is MACed using the HMAC SHA-256 algorithm. The shared secret, passed by the API client, represents the HMAC secret cryptographic key. The secret is used in combination with the cryptographic hash function to calculate the message authentication code (MAC). In the example below, note how the API client’s shared secret is used to validate our JWT’s JWS.
Sequence Diagrams of Authentication Process
Below are three sequence diagrams, which detail the following processes: API client registration process, obtaining a new JWT, and a REST call being authenticated using the JWT. The end-user of the API self-registers their application using the Authentication service and receives back an API key. The API key is unique to that client.
The end-user application then uses the API key and the shared secret to receive a JWT from the Authentication service.
After receiving the JWT, the end-user application passes the JWT in the header of each API request. The JWT is validated by calling the Authentication service. If the JWT is valid, the request is fulfilled. If not valid, a ‘401 Unauthorized’ status is returned.
JWT Validation
The JWT draft standard recommends how to validate a JWT. Our Virtual-Vehicles Authentication microservice uses many of those criteria to validate the JWT, which include:
- API Key – Retrieve API client’s shared secret from MongoDB, using API key contained in JWT’s claims set (secret is returned; therefore API key is valid)
- Algorithm – confirm the algorithm (‘alg’), found in the JWT Header, which used to encode JWT, was ‘HS256’ (HMAC SHA-256)
- Valid JWS – Use the client’s shared secret from #1 above, decode HMAC SHA-256 encrypted JWS
- Expiration – confirm JWT is not expired (‘exp’)
Inter-Service Communications
By default, the RestExpress Archetype project does not offer an example of communications between microservices. Service-to-service communications for microservices is most often done using the same HTTP-based REST calls used to by our Virtual-Vehicles REST API. Alternately, a message broker, such as RabbitMQ, Kafka, ActiveMQ, or Kestrel, is often used. Both methods of communicating between microservices, referred to as ‘inter-service communication mechanisms’ by InfoQ, have their pros and cons. The InfoQ website has an excellent microservices post, which discusses the topic of service-to-service communication.
For the Virtual-Vehicles microservices example, we will use HTTP-based REST calls for inter-service communications. The primary service-to-service communications in our example, is between the three microservices, Vehicle, Valet, and Maintenance, and the Authentication microservice. The three services validate the JWT passed in each request to a CRUD operation, by calling the Authentication service and passing the JWT, as shown in the sequence diagram, above. Validation is done using an HTTP GET request to the Authentication service’s .../jwts/{jwt}
endpoint. The Authentication service’s method, called by this endpoint, minus some logging and error handling, looks like the following:
public boolean authenticateJwt(Request request, String baseUrlAndAuthPort) { | |
String jwt, output, valid = ""; | |
try { | |
jwt = (request.getHeader("Authorization").split(" "))[1]; | |
} catch (NullPointerException | ArrayIndexOutOfBoundsException e) { | |
return false; | |
} | |
try { | |
URL url = new URL(baseUrlAndAuthPort + "/jwts/" + jwt); | |
HttpURLConnection conn = (HttpURLConnection) url.openConnection(); | |
conn.setRequestMethod("GET"); | |
conn.setRequestProperty("Accept", "application/json"); | |
if (conn.getResponseCode() != 200) { | |
return false; | |
} | |
BufferedReader br = new BufferedReader(new InputStreamReader( | |
(conn.getInputStream()))); | |
while ((output = br.readLine()) != null) { | |
valid = output; | |
} | |
conn.disconnect(); | |
} catch (IOException e) { | |
return false; | |
} | |
return Boolean.parseBoolean(valid); | |
} |
Primarily, we are using the java.net
and java.io
packages, along with the org.restexpress.Request
class to build and send our HTTP request to the Authentication service. Alternately, you could use just the org.restexpress
package to construct request and handle the response. This same basic method structure shown above can be used to create unit tests for your service’s endpoints.
Health Ping
Each of the Virtual-Vehicles microservices contain a DiagnosticController
in the .utilities
package. In our example, we have created a ping()
method. This simple method, called through the .../utils/ping
endpoint, should return a 200 OK
status and a boolean value of ‘true’, indicating the microservice is running and reachable. This route’s associated method could not be simpler:
public void ping(Request request, Response response) { | |
response.setResponseStatus(HttpResponseStatus.OK); | |
response.setResponseCode(200); | |
response.setBody(true); | |
} |
The ping health check can even be accessed with a simple curl command, curl localhost:8581/vehicles/utils/ping
.
In a real-world application, we would add additional health checks to all services, providing additional insight into the health and performance of each microservice, as well as the service’s dependencies.
API Documentation
A well written RESTful API will not require extensive documentation to understand the API’s operations. Endpoints will be discoverable through linking (see Response Body links section in below example). API documentation should provide HTTP method, required headers and URL parameters, expected response body and response status, and so forth.
An API should be documented before any code is written, similar to TDD. The API documentation is the result of a clear understanding of requirements. The API documentation should make the coding even easier since the documentation serves as a further refinement of the requirements. The requirements are an architectural plan for the microservice’s code structure.
Sample Documentation
Below, is a sample of the Virtual-Vehicles REST API documentation. It details the function responsible for creating a new API client. The documentation provides a brief description of the function, the operation’s endpoint (URI), HTTP method, request parameters, expected response body, expected response status, and even a view of the MongoDB collection’s document for a new API client.
You can download a PDF version of the Virtual-Vehicles RESTful API documentation on GitHub or review the source document on Google Docs. It contains general notes about the API, and details about every one of the API’s operations.
Running the Individual Microservices
For development and testing purposes, the easiest way to start the microservices is in NetBeans using the Run
command. The Run
command executes the Maven exec
goal. Based on the DEFAULT_ENVIRONMENT
constant in the org.restexpress.util
Environment class, RestExpress will use the ‘dev’ environment’s environment.properties
file, in the project’s /config/dev
directory.
Alternately, you can use the RestExpress project’s recommended command from a terminal prompt to start each microservice from its root directory (mvn exec:java -Dexec.mainClass=test.Main -Dexec.args="dev"
). You can also use this command to switch from the ‘dev’ to ‘prod’ environment properties (-Dexec.args="prod"
).
You may use a variety of commands to confirm all the microservices are running. I prefer something basic like sudo netstat -tulpn | grep 858[0-9]
. This will find all the ports within the ‘dev’ port range. For more in-depth info, you can use a command like ps -aux | grep com.example | grep -v grep
Part Three: Testing our Services
We now have a copy of the Virtual-Vehicles project pulled from GitHub, a basic understanding of how RestExpress works, and our four microservices running on different ports. In Part Three of this series, we will take them for a drive (get it?). There are many ways to test our service’s endpoints. One of my favorite tools is Postman. we will explore how to use several tools, including Postman, and our API documentation, to test our microservice’s endpoints.
Building a Microservices-based REST API with RestExpress, Java EE, and MongoDB: Part 1
Posted by Gary A. Stafford in Enterprise Software Development, Java Development, Software Development on May 18, 2015
Develop a well-architected and well-documented REST API, built on a tightly integrated collection of Java EE-based microservices.
Microservices
Microservices are a popular and growing trend in software development. According to Wikipedia, microservices are “a software architecture style, in which complex applications are composed of small, independent processes communicating with each other using language-agnostic APIs. These services are small, highly decoupled and focus on doing a small task.”
Martin Fowler and James Lewis (ThoughtWorks) have done an exemplary job capturing the essence of microservice architecture in their March 2014 post, microservices. Fowler has also discussed these principles in several presentations, including the January 2015 goto; Conference, Keynote: Microservices by Martin Fowler.
Additionally, noted technical consultant and speaker, Adrian Cockcroft (Battery Ventures), has made significant contributions to the definition of microservices, such as in his December 2014 dockercon14 | eu presentation, State of the Art in Microservices.
Lastly, Zhamak Dehghani (ThoughtWorks), delivered an in-depth discussion of microservices, including customer perspectives, in her October 2014 presentation, Real-World Microservices: Lessons from the Frontline.
Some of the major characteristics of microservices and REST cited by these experts, include:
- Organized Around Business Capabilities
- Single Responsibility
- Loose Coupling / High Cohesion
- Smart Endpoints and Dumb Pipes
- Decentralized Data Management
- Hypermedia as the Engine of Application State (HATEOAS)
As we develop this post’s example, I will demonstrate how all of the above characteristics are implemented.
REST API
A REST API is the mash-up of two common software concepts, Representational State Transfer (REST) and an application programming interface (API). Although even Wikipedia doesn’t have an exact definition of a REST API, they come close to their discussion of REST. According to Wikipedia, “Web service APIs that adhere to the REST architectural constraints are called RESTful APIs. HTTP-based RESTful APIs are defined with these aspects: base URI, an Internet media type for the data, standard HTTP methods, hypertext links to a reference state, and hypertext links to reference related resources.”
An important nuance and differentiator from SOA-based APIs, RESTful APIs do not require XML-based Web service protocols (SOAP and WSDL) to support their interfaces (Wikipedia).
The author of the WebConcepts channel does an excellent job capturing the essence of REST APIs in REST API concepts and examples. Two additional presentations I strongly recommend are REST+JSON API Design – Best Practices for Developers and Designing a Beautiful REST+JSON API, both by Les Hazlewood, CTO of Stormpath. Stormpath is a leader in the commercial REST API space.
Microservices-Based REST API
A microservices-based REST API is a REST API, whose HTTP requests call an orchestrated collection or collections of language-agnostic and platform-agnostic microservices. The combination of these two trends, microservices, and a REST API, offers a simple, reliable, and scalable solution for providing flexible functionality to an end-user, in a technology-agnostic manner.
REST API Example
There is a fast-growing volume of reference materials describing the characteristics, benefits, and general architecture of microservices and REST APIs. However, in researching these topics, I have found a shortage of practical examples or tutorials on building microservices-based REST API solutions.
Undoubtedly, the complexity of even the simplest microservices-based solution limits the number of available cases. A minimally viable solution require planning, coding, testing, and documentation. The addition of cross-cutting features such as security, logging, monitoring, and orchestration, creates an enormous task to build a practical microservices-based example.
In the following series of posts, we will use many of the characteristics of a modern microservice architecture as described by Fowler, Lewis, Cockcroft, and Dehghani. We will combine these microservice characteristics with the best practices of good REST API design, as described by Hazlewood and WebConcepts, to build a minimally viable microservices-based REST API.
In a future post, we will create an application, which leverages the microservices-based solution, through the REST API. Additionally, we will demonstrate how to ensure high-availability of the individual microservices and data sources.
Vehicle for Learning
In a similar vein to the publicly available Twitter, Facebook, and Google REST APIs, we will build the Virtual-Vehicles REST API. The Virtual-Vehicles REST API will constitute a collection of vehicle-themed microservices. Collectively, the microservices will offer a comprehensive set of functionality to the end-user, an application developer. They, in turn, will use the functionality of the Virtual-Vehicles REST API to build applications and games for their end-users.
Technology Choices
There are a seemingly infinite number of technology choices for building microservices and REST APIs. Your choice of development languages, databases, application servers, third-party libraries, API gateway, logging, monitoring, automated testing, ORM or ODM, and even the IDE, all define your technology stack.
For the Virtual-Vehicles solution, we will use the following key technologies:
- RestExpress-Create performant, stand-alone microservices-based REST APIs
- Java EE – Primary development language
- Netty – Async, event-driven Java network application framework
- Apache Maven – Dependency management and more
- MongoDB – Data source
- JSON Web Tokens (JWT) – Claims-based authentication
- Apache Log4j 2 – Logging
- JUnit – Unit testing
- HAProxy – Service gateway and load-balancing
- NetBeans – IDE
- Git and GitHub – SCM
- Postman – REST API testing
What is RestExpress?
According to their website, RestExpress, composes best-of-breed open-source tools to enable quickly creating RESTful microservices that embrace industry best practices. Built from the ground-up for container-less, microservice architectures, RestExpress is the easiest way to create RESTful APIs in Java. RestExpress is an extremely lightweight, fast, REST engine and API for Java. RestExpress is a thin wrapper on Netty IO HTTP handling. RestExpress lets you create performant, stand-alone REST APIs rapidly. RestExpress provides several Maven archetypes, which we will use as a basis for our microservices.
RestExpress will also drive our technology decisions to use Java EE, Maven, MongoDB, and Netty.
Virtual-Vehicles Microservices
Adhering to the first few microservice architectural principles listed above, organized around business capabilities, single responsibility, and high cohesion, we first must determine the proper number of microservices, and their individual responsibilities. In the case of our solution, we will break down Virtual-Vehicles’ business capabilities into the following microservices:
Virtual-Vehicles Services |
|
Microservice | Purpose (Business Capability) |
Authentication Service | Manage API clients and JWT authentication |
Vehicle Service | Manage virtual vehicles |
Maintenance Service | Manage maintenance on vehicles |
Valet Parking Service | Manage a valet service to park for vehicles |
Sales Service | Manage the buying and selling of vehicles |
Registration Service | Manage registration of vehicles to owners |
Auction Service | Manage a virtual car auction |
Car Show Service | Manage a virtual car show |
Interaction Service | Manage interaction of users with vehicles |
For simplicity in this post’s example, we will only be exploring the (4) services shown above in bold.
This segmentation of service functionality is unlike what we might encounter in traditional monolithic, n-tier applications, and SOA-based architecture. Traditional applications were built around application-centric functionality or business’ organizational structure. Microservices, however, are client-centric and built around business capabilities.
REST API Functionality
The next decision we need to make is required functionality. What are the operational requirements of each business segment, represented by the microservices? Additionally, what are the nonfunctional requirements, such as monitoring, logging, and authentication. Requirements are translated into functionality, which is translated into the available resources exposed via the service’s RESTful endpoints.
For the Virtual-Vehicles microservices solution, based on a hypothetical set of business and non-functional requirements, we will expose the following resources. Collectively, they will compose the REST API:
Virtual-Vehicles REST API Resources |
||
Microservice | Purpose (Business Capability) | Functions |
Authentication Service | Manage API clients and JWT authentication |
|
Vehicle Service | Manage virtual vehicles |
|
Maintenance Service | Manage maintenance on vehicles |
|
Valet Parking Service | Manage a valet service to park for vehicles |
|
Reviewing the table above, note the first five functions for each service are all basic CRUD operations: create
(POST), read
(GET), readAll
(GET), update
(PUT), delete
(DELETE). The readAll
function also has find
, count
, and pagination
functionality using query parameters.
All services also have an internal authenticateJwt
function, to authenticate the JWT, passed in the HTTP request header, before performing any operation. Additionally, all services have a basic health-check function, ping
(GET). There are only a few other functions required for our Virtual-Vehicles example, such as for creating JWTs.
I’ve labeled each function as to suggested user scope. Scopes include public, admin, and internal. As a consumer of the REST API, you may only want to expose certain functionality to your general end-user (public). Additional functionality may be reserved for an administrative user (admin) or only yourself as a developer (internal). Creating a new vehicle might be a common end-user feature. However, the ability to permanently delete one or more vehicles may be reserved for an admin-level user, or not exposed at all.
REST API Patterns
We will not spend a lot of time discussing patterns for building REST APIs. There are many useful materials available on the Internet regarding industry-standard patterns for REST API resource URI construction. The two presentations I recommend above by Les Hazlewood, CTO of Stormpath, are excellent. Also, Microservices.io, RestApiTutorial.com, swagger.io, and raml.org websites offer solid overviews of REST patterns and RESTful standards.
A common RESTful anti-pattern, which is hard to avoid as a OOP developer, is the temptation to use verbs versus nouns and method-like names, in resource URIs. Remember, we are not designing an end-user application. We are building an API, used by API consumers (application developers), to build a variety of platform and language-agnostic applications. Functions like paintCar
, changeOil
, or parkVehicle
are not something the API should define. The Vehicle microservice exposes the update operation, which allows an application developer to change the car’s paint color in their paintCar
method. Similarly, the valet service exposes the create operation, which allows the application developer to create a function to park the vehicle (or car, or truck, in a garage, or parking lot, etc.). A good REST API allows for maximum end-user flexibility.
Part Two
In Part Two, we will install a copy of the Virtual-Vehicles project from GitHub. In Part Two, we will gain a basic understanding of how RestExpress works. Finally, we will discover how to get the Virtual-Vehicles microservices up and running.
Retrieving and Displaying Data with AngularJS and the MEAN Stack: Part II
Posted by Gary A. Stafford in Client-Side Development, DevOps, Enterprise Software Development, Mobile HTML Development, Software Development on March 24, 2014
Explore various methods of retrieving and displaying data using AngularJS and the MEAN Stack.
Introduction
In this two-part post, we are exploring methods of retrieving and displaying data using AngularJS and the MEAN Stack. The post’s corresponding GitHub project, ‘meanstack-data-samples‘, is based on William Lepinski’s ‘generator-meanstack‘, which is in turn is based on Yeoman’s ‘generator-angular‘. As a bonus, since both projects are based on ‘generator-angular’, all the code generators work. Generators can save a lot of time and aggravation when building AngularJS components.
In part one of this post, we installed and configured the ‘meanstack-data-samples’ project from GitHub. In part two, we will we will look at five examples of retrieving and displaying data using AngularJS:
- Function within AngularJS Controller returns array of strings.
- AngularJS Service returns an array of simple object literals to the controller.
- AngularJS Factory returns the contents of JSON file to the controller.
- AngularJS Factory returns the contents of JSON file to the controller using a resource object
(In GitHub project, but not discussed in this post). - AngularJS Factory returns a collection of documents from MongoDB Database to the controller.
- AngularJS Factory returns results from Google’s RESTful Web Search API to the controller.
Project Structure
For brevity, I have tried to limit the number of files in the project. There are two main views, both driven by a single controller. The primary files, specific to data retrieval and display, are as follows:
- Default site file (./)
- index.html – loads all CSS and JavaScript files, and views
- App and Routes (./app/scripts/)
- app.js – instantiates app and defines routes (route/view/controller relationship)
- Views (./app/views/)
- data-bootstrap.html – uses Twitter Bootstrap
- data-no-bootstrap.html – basically the same page, without Twitter Bootstrap
- Controllers (./app/scripts/controllers/)
- DataController.js (DataController) – single controller used by both views
- Services and Factories (./app/scripts/services/)
- meanService.js (meanService) – service returns array of object literals to DataController
- jsonFactory.js (jsonFactory) – factory returns contents of JSON file
- jsonFactoryResource.js (jsonFactoryResource) – factory returns contents of JSON file using resource object (new)
- mongoFactory.js (mongoFactory) – factory returns MongoDB collection of documents
- googleFactory.js (googleFactory) – factory call Google Web Search API
- Models (./app/models/)
- Components.js – mongoose constructor for the Component schema definition
- Routes (./app/)
- routes.js – mongoose RESTful routes
- Data (./app/data/)
- otherStuff.json – static JSON file loaded by jsonFactory
- Environment Configuration (./config/environments/)
- index.js – defines all environment configurations
- test.js – Configuration specific to the current ‘test’ environment
- Unit Tests (./test/spec/…)
- Various files – all controller and services/factories unit test files are in here…
There are many more files, critical to the project’s functionality, include app.js, Gruntfile.js, bower.json, package.json, server.js, karma.conf.js, and so forth. You should understand each of these file’s purposes.
Function Returns Array
In the first example, we have the yeomanStuff()
method, a member of the $scope
object, within the DataController
. The yeomanStuff()
method return an array object containing three strings. In JavaScript, a method is a function associated with an object.
$scope.yeomanStuff = function () { return [ 'yo', 'Grunt', 'Bower' ]; };
The yeomanStuff()
method is called from within the view by Angular’s ng-repeat
directive. The directive, ng-repeat
, allows us to loop through the array of strings and add them to an unordered list. We will use ng-repeat
for all the examples in this post.
<ul class="list-group"> <li class="list-group-item" ng-repeat="stuff in yeomanStuff()"> {{stuff}} </li> <ul>
Although this first example is easy to implement, it is somewhat impractical. Generally, you would not embed static data into your code. This limits your ability to change the data, independent of a application’s code. In addition, the function is tightly coupled to the controller, limiting its reuse.
Service Returns Array
In the second example, we also use data embedded in our code. However, this time we have improved the architecture slightly by moving the data to an Angular Service. The meanService
contains the getMeanStuff()
function, which returns an array containing four object literals. Using a service, we can call the getMeanStuff()
function from anywhere in our project.
angular.module('generatorMeanstackApp') .service('meanService', function () { this.getMeanStuff = function () { return ([ { component: 'MongoDB', url: 'http://www.mongodb.org' }, { component: 'Express', url: 'http://expressjs.com' }, { component: 'AngularJS', url: 'http://angularjs.org' }, { component: 'Node.js', url: 'http://nodejs.org' } ]) }; });
Within the DataController
, we assign the array object, returned from the meanService.getMeanStuff()
function, to the meanStuff
object property of the $scope
object.
$scope.meanStuff = {}; try { $scope.meanStuff = meanService.getMeanStuff(); } catch (error) { console.error(error); }
The meanStuff
object property is accessed from within the view, using ng-repeat
. Each object in the array contains two properties, component
and url
. We display the property values on the page using Angular’s double curly brace expression notation (i.e. ‘{{stuff.component}}
‘).
<ul class="nav nav-pills nav-stacked"> <li ng-repeat="stuff in meanStuff"> url}}" target="_blank">{{stuff.component}} </li> <ul>
Promises, Promises…
The remaining methods implement an asynchronous (non-blocking) programming model, using the $http
and $q
services of Angular’s ng
module. The services implements the asynchronous Promise and Deferred APIs. According to Chris Webb, in his excellent two-part post, Promise & Deferred objects in JavaScript: Theory and Semantics, a promise represents a value that is not yet known and a deferred represents work that is not yet finished. I strongly recommend reading Chris’ post, before continuing. I also highly recommend watching RED Ape EDU’s YouTube video, Deferred and Promise objects in Angular js. This video really clarified the promise and deferred concepts for me.
Factory Loads JSON File
In the third example, we will read data from a JSON file (‘./app/data/otherStuff.json
‘) using an AngularJS Factory. The differences between a service and a factory can be confusing, and are beyond the scope of this post. Here is two great links on the differences, one on Angular’s site and one on StackOverflow.
{ "components": [ { "component": "jQuery", "url": "http://jquery.com" }, { "component": "Jade", "url": "http://jade-lang.com" }, { "component": "JSHint", "url": "http://www.jshint.com" }, { "component": "Karma", "url": "http://karma-runner.github.io" }, ... ] }
The jsonFactory
contains the getOtherStuff()
function. This function uses $http.get()
to read the JSON file and returns a promise of the response object. According to Angular’s site, “since the returned value of calling the $http function is a promise, you can also use the then method to register callbacks, and these callbacks will receive a single argument – an object representing the response. A response status code between 200 and 299 is considered a success status and will result in the success callback being called. ” As I mentioned, a complete explanation of the deferreds and promises, is too complex for this short post.
angular.module('generatorMeanstackApp') .factory('jsonFactory', function ($q, $http) { return { getOtherStuff: function () { var deferred = $q.defer(), httpPromise = $http.get('data/otherStuff.json'); httpPromise.then(function (response) { deferred.resolve(response); }, function (error) { console.error(error); }); return deferred.promise; } }; });
The response object contains the data
property. Angular defines the response object’s data
property as a string or object, containing the response body transformed with the transform functions. One of the properties of the data
property is the components
array containing the seven objects. Within the DataController
, if the promise is resolved successfully, the callback function assigns the contents of the components
array to the otherStuff
property of the $scope
object.
$scope.otherStuff = {}; jsonFactory.getOtherStuff() .then(function (response) { $scope.otherStuff = response.data.components; }, function (error) { console.error(error); });
The otherStuff
property is accessed from the view, using ng-repeat
, which displays individual values, exactly like the previous methods.
<ul class="nav nav-pills nav-stacked"> <li ng-repeat="stuff in otherStuff"> <a href="{{stuff.url}}" target="_blank">{{stuff.component}}</a> </li> </ul>
This method of reading a JSON file is often used for configuration files. Static configuration data is stored in a JSON file, external to the actual code. This way, the configuration can be modified without requiring the main code to be recompiled and deployed. It is a technique used by the components within this very project. Take for example the bower.json
files and the package.json
files. Both contain configuration data, stored as JSON, used by Bower and npm to perform package management.
Factory Retrieves Data from MongoDB
In the fourth example, we will read data from a MongoDB database. There are a few more moving parts in this example than in the previous examples. Below are the documents in the components
collection of the meanstack-test
MongoDB database, which we will retrieve and display with this method. The meanstack-test
database is defined in the test.js
environments file (discussed in part one).
To connect to the MongoDB, we will use Mongoose. According to their website, “Mongoose provides a straight-forward, schema-based solution to modeling your application data and includes built-in type casting, validation, query building, business logic hooks and more, out of the box.” But wait, MongoDB is schemaless? It is. However, Mongoose provides a schema-based API for us to work within. Again, according to Mongoose’s website, “Everything in Mongoose starts with a Schema. Each schema maps to a MongoDB collection and defines the shape of the documents within that collection.”
In our example, we create the componentSchema
schema, and pass it to the Component
model (the ‘M’ in MVC). The componentSchema
maps to the database’s components
collection.
var mongoose = require('mongoose'); var Schema = mongoose.Schema; var componentSchema = new Schema({ component: String, url: String }); module.exports = mongoose.model('Component', componentSchema);
The routes.js
file associates routes (Request URIs) and HTTP methods to Mongoose actions. These actions are usually CRUD operations. In our simple example, we have a single route, ‘/api/components
‘, associated with an HTTP GET
method. When an HTTP GET
request is made to the ‘/api/components
‘ request URI, Mongoose calls the Model.find()
function, ‘Component.find()
‘, with a callback function parameter. The Component.find()
function returns all documents in the components
collection.
var Component = require('./models/component'); module.exports = function (app) { app.get('/api/components', function (req, res) { Component.find(function (err, components) { if (err) res.send(err); res.json(components); }); }); };
You can test these routes, directly. Below, is the results of calling the ‘/api/components
‘ route in Chrome.
The mongoFactory
contains the getMongoStuff()
function. This function uses $http.get()
to call the ‘/api/components
‘ route. The route is resolved by the routes.js
file, which in turn executes the Component.find()
command. The promise of an array of objects is returned by the getMongoStuff()
function. Each object represents a document in the components
collection.
angular.module('generatorMeanstackApp') .factory('mongoFactory', function ($q, $http) { return { getMongoStuff: function () { var deferred = $q.defer(), httpPromise = $http.get('/api/components'); httpPromise.success(function (components) { deferred.resolve(components); }) .error(function (error) { console.error('Error: ' + error); }); return deferred.promise; } }; });
Within the DataController
, if the promise is resolved successfully, the callback function assigns the array of objects, representing the documents in the collection, to the mongoStuff
property of the $scope object.
$scope.mongoStuff = {}; mongoFactory.getMongoStuff() .then(function (components) { $scope.mongoStuff = components; }, function (error) { console.error(error); });
The mongoStuff
property is accessed from the view, using ng-repeat
, which displays individual values using Angular expressions, exactly like the previous methods.
<ul class="list-group"> <li class="list-group-item" ng-repeat="stuff in mongoStuff"> <b>{{stuff.component}}</b> <div class="text-muted">{{stuff.description}}</div> </li> </ul>
Factory Calls Google Search
Post Update: the Google Web Search API is no longer available as of September 29, 2014. The post’s example post will no longer return a resultset. Please migrate to the Google Custom Search API (https://developers.google.com/custom-search/). Please read ‘Calling Third-Party HTTP-based RESTful APIs from the MEAN Stack‘ post for more information on using Google’s Custom Search API.
In the last example, we will call the Google Web Search API from an AngularJS Factory. The Google Web Search API exposes a simple RESTful interface. According to Google, “in all cases, the method supported is GET and the response format is a JSON encoded result set with embedded status codes.” Google describes this method of using RESTful access to the API, as “for Flash developers, and those developers that have a need to access the Web Search API from other Non-JavaScript environment.” However, we will access it in our JavaScript-based MEAN stack application, due to the API’s ease of implementation.
Note according to Google’s site, “the Google Web Search API has been officially deprecated…it will continue to work…but the number of requests…will be limited. Therefore, we encourage you to move to Custom Search, which provides an alternative solution.” Google Search, or more specifically, the Custom Search JSON/Atom API, is a newer API, but the Web Search API is easier to demonstrate in this brief post than Custom Search JSON/Atom API, which requires the use of an API key.
The googleFactory
contains the getSearchResults()
function. This function uses $http.jsonp()
to call the Google Web Search API RESTful interface and return the promise of the JSONP-formatted (‘JSON with padding’) response. JSONP provides cross-domain access to a JSON payload, by wrapping the payload in a JavaScript function call (callback).
angular.module('generatorMeanstackApp') .factory('googleFactory', function ($q, $http) { return { getSearchResults: function () { var deferred = $q.defer(), host = 'https://ajax.googleapis.com/ajax/services/search/web', args = { 'version': '1.0', 'searchTerm': 'mean%20stack', 'results': '8', 'callback': 'JSON_CALLBACK' }, params = ('?v=' + args.version + '&q=' + args.searchTerm + '&rsz=' + args.results + '&callback=' + args.callback), httpPromise = $http.jsonp(host + params); httpPromise.then(function (response) { deferred.resolve(response); }, function (error) { console.error(error); }); return deferred.promise; } }; });
The getSearchResults()
function uses the HTTP GET method to make an HTTP request the following RESTful URI:
https://ajax.googleapis.com/ajax/services/search/web?v=1.0&q=mean%20stack&rsz=8&callback=angular.callbacks._0
Using Google Chrome’s Developer tools, we can preview the Google Web Search JSONP-format HTTP response (abridged). Note the callback function that wraps the JSON payload.
Within the DataController
, if the promise is resolved successfully, our callback function returns the response object. The response object contains a lot of information. We are able to limit that amount of information sent to the view by only assigning the actual search results, an array of eight objects contained in the response object, to the googleStuff
property of the $scope
object.
$scope.googleStuff = {}; googleFactory.getSearchResults() .then(function (response) { $scope.googleStuff = response.data.responseData.results; }, function (error) { console.error(error); });
Below is the full response returned by the The googleFactory
. Note the path to the data we are interested in: ‘response.data.responseData.results
‘.
Below is the filtered results assigned to the googleStuff
property:
The googleStuff
property is accessed from the view, using ng-repeat
, which displays individual values using Angular expressions, exactly like the previous methods.
<ul class="list-group"> <li class="list-group-item" ng-repeat="stuff in googleStuff"> <a href="{{unescapedUrl.url}}" target="_blank"><b>{{stuff.visibleUrl}}</b></a> <div class="text-muted">{{stuff.titleNoFormatting}}</div> </li> </ul>
Links
- Promise & Deferred objects in JavaScript Pt.1: Theory and Semantics
- Promise & Deferred objects in JavaScript Pt.2: Theory and Semantics
- Deferred and Promise objects in Angular js (YouTube video – excellent!)
- Deferred Promise Workshop (goes with above video)
- Angular.js Promise and Deferred Api – Implementation Explained
- Creating a Single Page Todo App with Node and Angular
- Build an Angular Todo App with a Node Backend
- Consuming REST APIs
Retrieving and Displaying Data with AngularJS and the MEAN Stack: Part I
Posted by Gary A. Stafford in Client-Side Development, DevOps, Mobile HTML Development, Software Development on March 22, 2014
Explore various methods of retrieving and displaying data using AngularJS and the MEAN Stack.
Introduction
In the following two-part post, we will explore several methods of retrieving and displaying data using AngularJS and the MEAN Stack. The post’s corresponding GitHub project, ‘meanstack-data-samples‘, is based on William Lepinski’s ‘generator-meanstack‘, which is in turn is based on Yeoman’s ‘generator-angular‘. As a bonus, since both projects are based on ‘generator-angular’, all the code generators work. Generators can save a lot of time and aggravation when building AngularJS components.
In part one of this post, we will install and configure the ‘meanstack-data-samples’ project from GitHub, which corresponds to this post. In part two, we will we will look at several methods for retrieving and displaying data using AngularJS:
- Function within AngularJS Controller returns array of strings.
- AngularJS Service returns an array of simple object literals to the controller.
- AngularJS Factory returns the contents of JSON file to the controller.
- AngularJS Factory returns the contents of JSON file to the controller using a resource object
(In GitHub project, but not discussed in this post). - AngularJS Factory returns a collection of documents from MongoDB Database to the controller.
- AngularJS Factory returns results from Google’s RESTful Web Search API to the controller.
Preparation
If you need help setting up your development machine to work with the MEAN stack, refer to my last post, Installing and Configuring the MEAN Stack, Yeoman, and Associated Tooling on Windows. You will need to install all the MEAN and Yeoman components.
For this post, I am using JetBrains’ new WebStorm 8RC to build and demonstrate the project. There are several good IDE’s for building modern web applications; WebStorm is one of the current favorites of developers.
Complexity of Modern Web Applications
Building modern web applications using the MEAN stack or comparable technologies is complex. The ‘meanstack-data-samples’ project, and the projects it is based on, ‘generator-meanstack’ and ‘generator-angular’, have dozens of moving parts. In this simple project, we have MongoDB, ExpressJS, AngularJS, Node.js, yo, Grunt, Bower, Git, jQuery, Twitter Bootstrap, Karma, JSHint, jQuery, Mongoose, and hundreds of other components, all working together. There are almost fifty Node packages and hundreds of their dependencies loaded by npm, in addition to another dozen loaded by Bower.
Installing, configuring, and managing all the parts of a modern web application requires a basic working knowledge of these technologies. Understanding how Bower and npm install and manage packages, how Grunt builds, tests, and serves the application with ExpressJS, how Yo scaffolds applications, how Karma and Jasmine run unit tests, or how Mongoose and MongoDB work together, are all essential. This brief post will primarily focus on retrieving and displaying data, not necessarily how the components all work, or work together.
Installing and Configuring the Project
Environment Variables
To start, we need to create (3) environment variables. The NODE_ENV
environment variable is used to determine the environment our application is operating within. The NODE_ENV
variable determines which configuration file in the project is read by the application when it starts. The configuration files contain variables, specific to that environment. There are (4) configuration files included in the project. They are ‘development’, ‘test’, ‘production’, and ‘travis’ (travis-ci.org). The NODE_ENV
variable is referenced extensively throughout the project. If the NODE_ENV
variable is not set, the application will default to ‘development
‘.
For this post, set the NODE_ENV
variable to ‘test
‘. The value, ‘test
‘, corresponds to the ‘test
‘ configuration file (‘meanstack-data-samples\config\environments\test.js
‘), shown below.
// set up ===================================== var express = require('express'); var bodyParser = require('body-parser'); var errorHandler = require('errorhandler'); var favicon = require('serve-favicon'); var logger = require('morgan'); var cookieParser = require('cookie-parser'); var methodOverride = require('method-override'); var session = require('express-session'); var path = require('path'); var env = process.env.NODE_ENV || 'development'; module.exports = function (app) { if ('test' == env) { console.log('environment = test'); app.use(function staticsPlaceholder(req, res, next) { return next(); }); app.set('db', 'mongodb://localhost/meanstack-test'); app.set('port', process.env.PORT || 3000); app.set('views', path.join(app.directory, '/app')); app.engine('html', require('ejs').renderFile); app.set('view engine', 'html'); app.use(favicon('./app/favicon.ico')); app.use(logger('dev')); app.use(bodyParser()); app.use(methodOverride()); app.use(cookieParser('your secret here')); app.use(session()); app.use(function middlewarePlaceholder(req, res, next) { return next(); }); app.use(errorHandler()); } };
The second environment variable is PORT
. The application starts on the port indicated by the PORT
variable, for example, ‘localhost:3000’. If the the PORT
variable is not set, the application will default to port ‘3000
‘, as specified in the each of the environment configuration files and the ‘Gruntfile.js’ Grunt configuration file.
Lastly, the CHROME_BIN
environment variable is used Karma, the test runner for JavaScript, to determine the correct path to browser’s binary file. Details of this variable are discussed in detail on Karma’s site. In my case, the value for the CHROME_BIN
is ‘C:\Program Files (x86)\Google\Chrome\Application\chrome.exe'
. This variable is only necessary if you will be configuring Karma to use Chrome to run the tests. The browser can be changes to any browser, including PhantomJS. See the discussion at the end of this post regarding browser choice for Karma.
You can easily set all the environment variables on Windows from a command prompt, with the following commands. Remember to exit and re-open your interactive shell or command prompt window after adding the variables so they can be used.
REM cofirm the path to Chrome, change value if necessary | |
setx /m NODE_ENV "test" | |
setx /m PORT "3000" | |
setx /m CHROME_BIN "C:\Program Files (x86)\Google\Chrome\Application\chrome.exe" |
Install and Configure the Project
To install and configure the project, we start by cloning the ‘meanstack-data-samples‘ project from GitHub. We then use npm and bower to install the project’s dependencies. Once installed, we create and populate the Mongo database. We then use Grunt and Karma to unit test the project. Finally, we will use Grunt to start the Express Server and run the application. This is all accomplished with only a few individual commands. Please note, the ‘npm install’ command could take several minutes to complete, depending on your network speed; the project has many direct and indirect Node.js dependencies.
# install meanstack-data-samples project | |
git clone https://github.com/garystafford/meanstack-data-samples.git | |
cd meanstack-data-samples | |
npm install | |
bower install | |
mongoimport --db meanstack-$NODE_ENV --collection components components.json --drop # Unix | |
# mongoimport --db meanstack-%NODE_ENV% --collection components components.json --drop # Windows | |
grunt test | |
grunt server |
If everything was installed correctly, running the ‘grunt test’ command should result in output similar to below:
If everything was installed correctly, running the ‘grunt server’ command should result in output similar to below:
Running the ‘grunt server’ command should start the application and open your browser to the default view, as shown below:
Karma’s Browser Choice for Unit Tests
The GitHub project is currently configured to use Chrome for running Karma’s unit tests in the ‘development’ and ‘test’ environments. For the ‘travis’ environment, it uses PhantomJS. If you do not have Chrome installed on your machine, the ‘grunt test’ task will fail during the ‘karma:unit’ task portion. To change Karma’s browser preference, simply change the ‘testBrowser’ variable in the ‘./karma.conf.js’ file, as shown below.
// Karma configuration | |
module.exports = function (config) { | |
// Determines Karma's browser choice based on environment | |
var testBrowser = 'Chrome'; // Default browser | |
if (process.env.NODE_ENV === 'travis') { | |
testBrowser = 'PhantomJS'; // Must use for headless CI (Travis-CI) | |
} | |
console.log("Karma browser: " + testBrowser); | |
... | |
// Start these browsers, currently available: | |
// Chrome, ChromeCanary, Firefox, Opera, | |
// Safari (only Mac), PhantomJS, IE (only Windows) | |
browsers: [testBrowser], |
I recommend installing and using PhantomJS headless WebKit, locally. Since PhantomJS is headless, Karma runs the unit tests without having to open and close browser windows. To run this project on continuous integration servers, like Jenkins or Travis-CI, you must PhantomJS. If you decide to use PhantomJS on Windows, don’t forget add the PhantomJS executable directory path to your ‘PATH’ environment variable to, after downloading and installing the application.
Code Generator
As I mentioned at the start of this post, this project was based on William Lepinski’s ‘generator-meanstack‘, which is in turn is based on Yeoman’s ‘generator-angular‘. Optionally, to install the ‘generator-meanstack’ npm package, globally, on our system use the following command The ‘generator-meanstack’ code generator will allow us to generate additional AngularJS components automatically, within the project, if we choose. The ‘generator-meanstack’ is not required for this post.
npm install -g generator-meanstack
Part II
In part two of this post, we will explore each methods of retrieving and displaying data using AngularJS, in detail.
Links
Installing and Configuring the MEAN Stack, Yeoman, and Associated Tooling on Windows
Posted by Gary A. Stafford in Bash Scripting, Build Automation, Client-Side Development, DevOps, Software Development on March 8, 2014
Configure your Windows environment for developing modern web applications using the popular MEAN Stack and Yeoman suite of utilities.
Introduction
It’s an exciting time to be involved in web development. There are dozens of popular open-source JavaScript frameworks, libraries, code-generators, and associated tools, exploding on to the development scene. It is now possible to use a variety of popular technology mashups to provide a complete, full-stack JavaScript application platform.
MEAN Stack
One of the JavaScript mashups gaining a lot of traction recently is the MEAN Stack. If you’re reading this post, you probably already know MEAN is an acronym for four leading technologies: MongoDB, ExpressJS, AngularJS, and Node.js. The MEAN stack provides an end-to-end JavaScript application solution. MEAN provides a NoSQL document database (Mongo), a server-side solution for JavaScript (Node/Express), and a magic client-side MV* framework (Angular).
Depending in which MEAN stack generator or pre-build project you start with, in addition to the main four technologies, you will pull down several other smaller libraries and frameworks. These most commonly include jQuery, Twitter Bootstrap, Karma (test runner), Jade (template engine), JSHint, Underscore.js (utility-belt library), Mongoose (MongoDB object modeling tool), Passport (authentication), RequireJS (file and module loader), BreezeJS (data entity management), and so forth.
Common Tooling
If you are involved in these modern web development trends, then you are aware there is also a fairly common set of tools used by a majority of these developers, including source control, IDE, OS, and other helper-utilities. For SCM/VCS, Git is the clear winner. For an IDE, WebStorm, Sublime Text, and Kompozer, are heavy favorites. The platform of choice for most developers most often appears to be either Mac or Linux. It’s far less common to see a demonstration of these technologies, or tutorials built on the Microsoft Windows platform.
Yeoman
Another area of commonality is help-utilities, used to make the development, building, dependency management, and deployment of modern JavaScript applications, easier. Two popular ones are Brunch and Yeoman. Yeoman is also an acronym for a set of popular tools: yo, Grunt, and Bower. The first, yo, is best described as a scaffolding tool. Grunt is the build tool. Bower is a tool for client-side package and dependency management. We will install Yeoman, along with the MEAN Stack, in this post.
Windows
There is no reason Windows cannot serve as your development and hosting platform for modern web development, without specifically using Microsoft’s .NET stack. In fact, with minimal set-up, you would barely know you were using Windows as opposed to Linux or Mac. In this post, I will demonstrate how to configure your Windows machine for developing these modern web applications using the MEAN Stack and Yeoman.
Here is a list of the components we will discuss:
Installations
Git
The use of Git for source control is obvious. Git is the overwhelming choice of modern developers. Git has been integrated into most major IDEs and hosting platforms. There are hooks into Git available for most leading development tools. However, there are more benefits to using Git than just SCM. Being a Linux/Mac user, I prefer to use a Unix-like shell on Windows, versus the native Windows Command Prompt. For this reason, I use Git for Windows, available from msysGit. This package includes Git SCM, Git Bash, and Git GUI. I use the Git Bash interactive shell almost exclusively for my daily interactions requiring a command prompt. I will be using the Git Bash interactive shell for this post. OpenHatch has great post and training materials available on using Git Bash with Windows.
Git for Windows provides a downloadable Windows executable file for installation. Follow the installation file’s instructions.
To test your installation of Git for Windows, call the Git binary with the ‘–version’ flag. This flag can be used to test all the components we are installing in this post. If the command returns a value, then it’s a good indication that the component is installed properly and can be called from the command prompt:
gstafford: ~/Documents/git_repos $ git --version git version 1.9.0.msysgit.0
You can also verify Git using the ‘where’ and ‘which’ commands. The ‘where’ command will display the location of files that match the search pattern and are in the paths specified by the PATH environment variable. The ‘which’ command tells you which file gets executed when you run a command. These commands will work for most components we will install:
gstafford: ~/Documents/git_repos $ where git C:\Program Files (x86)\Git\bin\git.exe C:\Program Files (x86)\Git\cmd\git.cmd C:\Program Files (x86)\Git\cmd\git.exe gstafford: ~/Documents/git_repos $ which git /bin/git
Ruby
The reasons for Git are obvious, but why Ruby? Yeoman, specifically yo, requires Ruby. Installing Ruby on Windows is easy. Ruby recommends using RubyInstaller for Windows. RubyInstaller downloads an executable file, making install easy. I am using Ruby 1.9.3. I had previously installed the latest 2.0.0, but had to roll-back after some 64-bit compatibility issues with other applications.
To test the Ruby installation, use the ‘–version’ flag again:
gstafford: ~/Documents/git_repos $ ruby --version ruby 1.9.3p484 (2013-11-22) [i386-mingw32]
RubyGems
Optionally, you might also to install RubyGems. RubyGems allow you to add functionality to the Ruby platform, in the form of ‘Gems’. A common Gem used with the MEAN stack is Compass, the Sass-based stylesheet framework creation and maintenance of CSS. According to their website, Ruby 1.9 and newer ships with RubyGems built-in but you may need to upgrade for bug fixes or new features.
On Windows, installation of RubyGems is as simple as downloading the .zip file from the RubyGems download site. To install, RubyGems, unzip the downloaded file. From the root of the unzipped directory, run the following Ruby command:
ruby setup.rb
To confirm your installation:
gstafford: ~/Documents/git_repos $ gem --version 2.2.2
If you already have RubyGems installed, it’s recommended you update RubyGems before continuing. Use the first command, with the ‘–system’ flag, will update to the latest RubyGems. Use the second command, without the tag, if you want to update each of your individually installed Ruby Gems:
gem update --system gem update
MongoDB
MongoDB provides a great set of installation and configuration instructions for Windows users. To install MongoDB, download the MongoDB package. Create a ‘mongodb’ folder. Mongo recommends at the root of your system drive. Unzip the MongoDB package to ‘c:\mongodb’ folder. That’s really it, there is no installer file.
Next, make a default Data Directory location, use the following two commands:
mkdir c://data && mkdir c://data/db
Unlike most other components, to call Mongo from the command prompt, I had to manually add the path to the Mongo binaries to my PATH environment variable. You can get access to your Windows environment variables using the Windows and Pause keys. Add the path ‘c:\mongodb\bin’ to end of the PATH environment variable value.
To test the MongoDB installation, and that the PATH variable is set correctly, close any current interactive shells or command prompt windows. Open a new shell and use the same ‘–version’ flag for Mongo’s three core components:
gstafford: ~/Documents/git_repos $ mongo --version; mongod --version; mongos --version MongoDB shell version: 2.4.9 db version v2.4.9 Sun Mar 09 16:26:48.730 git version: 52fe0d21959e32a5bdbecdc62057db386e4e029c MongoS version 2.4.9 starting: pid=15436 port=27017 64-bit host=localhost (--help for usage) git version: 52fe0d21959e32a5bdbecdc62057db386e4e029c build sys info: windows sys.getwindowsversion(major=6, minor=1, build=7601, platform=2, service_pack='Service Pack 1') BOOST_LIB_VERSION=1_49
To start MongoDB, use the ‘mongod’ or ‘start mongod’ commands. Adding ‘start’ opens a new command prompt window, versus tying up your current shell. If you are not using the default MongoDB Data Directory (‘c://data/db’) you created in the previous step, use the ‘–dbpath’ flag, for example ‘start mongod –dbpath ‘c://alternate/path’.
.
Node.js
To install Node.js, download and run the Node’s .msi installer for Windows. Along with Node.js, you will get npm (Node Package Manager). You will use npm to install all your server-side components, such as Express, yo, Grunt, and Bower.
gstafford: ~/Documents/git_repos $ node --version && npm --version v0.10.26 1.4.3
Express
To install Express, the web application framework for node, use npm:
npm install -g express
The ‘-g’ flag (or, ‘–global’ flag) should be used. According to Stack Overflow, ‘if you’re installing something that you want to use in your shell, on the command line or something, install it globally, so that its binaries end up in your PATH environment variable:
gstafford: ~/Documents/git_repos $ express --version 3.5.0
Yeoman – yo, Grunt, and Bower
You will also use npm to install yo, Grunt, and Bower. Actually, we will use npm to install the Grunt Command Line Interface (CLI). The Grunt task runner will be installed in your MEAN stack project, locally, later. Read these instructions on the Grunt website for a better explanation. Use the same basic command as with Express:
npm install -g yo grunt-cli bower
To test the installs, run the same command as before:
gstafford: ~/Documents/git_repos $ yo --version; grunt --version; bower --version 1.1.2 grunt-cli v0.1.13 1.2.8
If you already had Yeoman installed, confirm you have the latest versions with the ‘npm update’ command:
gstafford: ~/Documents/git_repos/gen-angular-sample $ npm update -g yo grunt-cli bower npm http GET https://registry.npmjs.org/grunt-cli npm http GET https://registry.npmjs.org/bower npm http GET https://registry.npmjs.org/yo npm http 304 https://registry.npmjs.org/bower npm http 304 https://registry.npmjs.org/yo npm http 200 https://registry.npmjs.org/grunt-cli
All of the npm installs, including Express, are installed and called from a common location on Windows:
gstafford: ~/Documents/git_repos/gen-angular-sample $ where express yo grunt bower c:\Users\gstaffor\AppData\Roaming\npm\yo c:\Users\gstaffor\AppData\Roaming\npm\yo.cmd c:\Users\gstaffor\AppData\Roaming\npm\grunt c:\Users\gstaffor\AppData\Roaming\npm\grunt.cmd c:\Users\gstaffor\AppData\Roaming\npm\bower c:\Users\gstaffor\AppData\Roaming\npm\bower.cmd gstafford: ~/Documents/git_repos $ which express; which yo; which grunt; which bower ~/AppData/Roaming/npm/express ~/AppData/Roaming/npm/yo ~/AppData/Roaming/npm/grunt ~/AppData/Roaming/npm/bower
Use the command, ‘npm list –global | less’ (or, ‘npm ls -g | less’) to view all npm packages installed globally, in a tree-view. After you have generated your project (see below), check the project-specific server-side packages with the ‘npm ls’ command from within the project’s root directory. For the client-side packages, use the ‘bower ls’ command from within the project’s root directory.
If your in a hurry, or have more Windows boxes to configure you can use one npm command for all four components, above:
npm install -g express yo grunt-cli bower
MEAN Boilerplate Generators and Projects
That’s it, you’ve installed most of the core components you need to get started with the MEAN stack on Windows. Next, you will want to download one of the many MEAN boilerplate projects, or use a MEAN code generator with npm and yo. I recommend trying one or all of the following projects. They are each slightly different architecturally, but fairly stable:
- Linnovate’s MEAN.io project on GitHub (boilerplate MEAN project)
- James Cryer’s npm generator-mean, a Yeoman generator for MEAN stack, inspired by mean.io
- William Lepinski’s MEAN stack generator for Yeoman, generator-meanstack, based on the generator-angular.
.
Links
- Jim Lavin’s Introduction to the MEAN Stack on YouTube
- Google Developer’s Getting MEAN on YouTube
- Introduction to the MEAN Stack, Part One: Setting Up Your Tools
- Full-Stack JavaScript With MEAN And Yeoman
- The MEAN Stack: MongoDB, ExpressJS, AngularJS and Node.js
Create Multi-VM Environments Using Vagrant, Chef, and JSON
Posted by Gary A. Stafford in Build Automation, DevOps, Enterprise Software Development, Software Development on February 27, 2014
Create and manage ‘multi-machine’ environments with Vagrant, using JSON configuration files. Allow increased portability across hosts, environments, and organizations.
Introduction
As their website says, Vagrant has made it very easy to ‘create and configure lightweight, reproducible, and portable development environments.’ Based on Ruby, the elegantly simple open-source programming language, Vagrant requires a minimal learning curve to get up and running.
In this post, we will create what Vagrant refers to as a ‘multi-machine’ environment. We will provision three virtual machines (VMs). The VMs will mirror a typical three-tier architected environment, with separate web, application, and database servers.
We will move all the VM-specific information from the Vagrantfile to a separate JSON format configuration file. There are a few advantages to moving the configuration information to separate file. First, we can configure any number VMs, while keeping the Vagrantfile exactly the same. Secondly and more importantly, we can re-use the same Vagrantfile to build different VMs on another host machine.
Although certainly not required, I am also using Chef in this example. More specifically, I am using Hosted Chef to further configure the VMs. Like the VM-specific information above, I have also moved the Chef-specific information to a separate JSON configuration file. We can now use the same Vagrantfile within another Chef Environment, or even within another Chef Organization, using an alternate configuration files. If you are not a Chef user, you can disregard that part of the configuration code. Alternately, you can substitute the Chef configuration code for Puppet, if that is your configuration automation tool of choice.
The only items we will not remove from the Vagrantfile are the Vagrant Box and synced folder configurations. These items could also be moved to a separate configuration file, making the Vagrantfile even more generic and portable.
The Code
Below is the VM-specific JSON configuration file, containing all the individual configuration information necessary for Vagrant to build the three VMs: ‘apps’, dbs’, and ‘web’. Each child ‘node’ in the parent ‘nodes’ object contains key/value pairs for VM names, IP addresses, forwarding ports, host names, and memory settings. To add another VM, you would simply add another ‘node’ object.
{ | |
"nodes": { | |
"apps": { | |
":node": "ApplicationServer-201", | |
":ip": "192.168.33.21", | |
":host": "apps.server-201", | |
"ports": [ | |
{ | |
":host": 2201, | |
":guest": 22, | |
":id": "ssh" | |
}, | |
{ | |
":host": 7709, | |
":guest": 7709, | |
":id": "wls-listen" | |
} | |
], | |
":memory": 2048 | |
}, | |
"dbs": { | |
":node": "DatabaseServer-301", | |
":ip": "192.168.33.31", | |
":host": "dbs.server-301", | |
"ports": [ | |
{ | |
":host": 2202, | |
":guest": 22, | |
":id": "ssh" | |
}, | |
{ | |
":host": 1529, | |
":guest": 1529, | |
":id": "xe-db" | |
}, | |
{ | |
":host": 8380, | |
":guest": 8380, | |
":id": "xe-listen" | |
} | |
], | |
":memory": 2048 | |
}, | |
"web": { | |
":node": "WebServer-401", | |
":ip": "192.168.33.41", | |
":host": "web.server-401", | |
"ports": [ | |
{ | |
":host": 2203, | |
":guest": 22, | |
":id": "ssh" | |
}, | |
{ | |
":host": 4756, | |
":guest": 4756, | |
":id": "apache" | |
} | |
], | |
":memory": 1024 | |
} | |
} | |
} |
Next, is the Chef-specific JSON configuration file, containing Chef configuration information common to all the VMs.
{ | |
"chef": { | |
":chef_server_url": "https://api.opscode.com/organizations/my-organization", | |
":client_key_path": "/etc/chef/my-client.pem", | |
":environment": "my-environment", | |
":provisioning_path": "/etc/chef", | |
":validation_client_name": "my-client", | |
":validation_key_path": "~/.chef/my-client.pem" | |
} | |
} |
Lastly, the Vagrantfile, which loads both configuration files. The Vagrantfile instructs Vagrant to loop through all nodes in the nodes.json file, provisioning VMs for each node. Vagrant then uses the chef.json file to further configure the VMs.
The environment and node configuration items in the chef.json reference an actual Chef Environment and Chef Nodes. They are both part of a Chef Organization, which is configured within a Hosted Chef account.
# -*- mode: ruby -*- | |
# vi: set ft=ruby : | |
# Multi-VM Configuration: Builds Web, Application, and Database Servers using JSON config file | |
# Configures VMs based on Hosted Chef Server defined Environment and Node (vs. Roles) | |
# Author: Gary A. Stafford | |
# read vm and chef configurations from JSON files | |
nodes_config = (JSON.parse(File.read("nodes.json")))['nodes'] | |
chef_config = (JSON.parse(File.read("chef.json")))['chef'] | |
VAGRANTFILE_API_VERSION = "2" | |
Vagrant.require_plugin "vagrant-omnibus" | |
Vagrant.configure(VAGRANTFILE_API_VERSION) do |config| | |
config.vm.box = "vagrant-oracle-vm-saucy64" | |
config.vm.box_url = "http://cloud-images.ubuntu.com/vagrant/saucy/current/saucy-server-cloudimg-amd64-vagrant-disk1.box" | |
config.omnibus.chef_version = :latest | |
nodes_config.each do |node| | |
node_name = node[0] # name of node | |
node_values = node[1] # content of node | |
config.vm.define node_name do |config| | |
# configures all forwarding ports in JSON array | |
ports = node_values['ports'] | |
ports.each do |port| | |
config.vm.network :forwarded_port, | |
host: port[':host'], | |
guest: port[':guest'], | |
id: port[':id'] | |
end | |
config.vm.hostname = node_values[':node'] | |
config.vm.network :private_network, ip: node_values[':ip'] | |
# syncs local repository of large third-party installer files (quicker than downloading each time) | |
config.vm.synced_folder "#{ENV['HOME']}/Documents/git_repos/chef-artifacts", "/vagrant" | |
config.vm.provider :virtualbox do |vb| | |
vb.customize ["modifyvm", :id, "--memory", node_values[':memory']] | |
vb.customize ["modifyvm", :id, "--name", node_values[':node']] | |
end | |
# chef configuration section | |
config.vm.provision :chef_client do |chef| | |
chef.environment = chef_config[':environment'] | |
chef.provisioning_path = chef_config[':provisioning_path'] | |
chef.chef_server_url = chef_config[':chef_server_url'] | |
chef.validation_key_path = chef_config[':validation_key_path'] | |
chef.node_name = node_values[':node'] | |
chef.validation_client_name = chef_config[':validation_client_name'] | |
chef.client_key_path = chef_config[':client_key_path'] | |
end | |
end | |
end | |
end |
Each VM has a varying number of ports it needs to configue and forward. To accomplish this, the Vagrantfile not only loops through the each node, it also loops through each port configuration object it finds within the node object. Shown below is the Database Server VM within VirtualBox, containing three forwarding ports.
In addition to the gists above, this repository on GitHub contains a complete copy of all the code used in the post.
The Results
Running the ‘vagrant up’ command will provision all three individually configured VMs. Once created and running in VirtualBox, Chef further configures the VMs with the necessary settings and applications specific to each server’s purposes. You can just as easily create 10, 100, or 1,000 VMs using this same process.
.
Helpful Links
Java RESTful Web Services Using MySQL Server, EclipseLink, and Jersey
Posted by Gary A. Stafford in Client-Side Development, Java Development on April 12, 2013

Introduction
When implementing a Relational Database Management System (RDBMS), many enterprise software developers tend to favor Oracle 11g or Microsoft SQL Server relational databases, depending on their technology stack. However, there are several excellent alternative relational databases, including MySQL. In fact, MySQL is the world’s most popular open source database software, according to Oracle.
MySQL is available on over 20 platforms and operating systems including Linux, Unix, Mac and Windows, according to the MySQL website. Like Oracle and Microsoft’s flagship RDBMS, MySQL Server comes in at least four flavors, ranging from the free Community Edition, demonstrated here, to a full-featured, enterprise-level Cluster Carrier Grade Edition. Support for MySQL, like Oracle and Microsoft, extends beyond just technical support. MySQL provides JDBC, ODBC, .NET drivers for Java and .NET development, as well as other languages. MySQL is supported by many popular IDE’s, including MySQL’s own RDBMS IDE, MySQL Workbench. Lastly, like Oracle and Microsoft, MySQL provides extensive documentation, tutorials, and even sample databases, built using recommended architectural patterns.
In this post, we will use JDBC to map JPA entity classes to tables and views within a MySQL database. We will then build RESTful web services, EJB classes, which communicate with MySQL through the entities. We will separate the JPA entities into a Java Class Library. The class library will be referenced by the RESTful web services. The RESTful web services, part of a Java Web Application, will be deployed to GlassFish, where they are accessed with HTTP methods and tested.
Installation and Configuration
If you’ve worked with Microsoft SQL Server or particularly Oracle 11g, you’ll have a minimal learning curve with MySQL. Basic installation, configuration, and integration within your Java applications is like Oracle and Microsoft. Start by downloading and installing the latest versions of MySQL Server, MySQL Workbench, MySQL JDBC Connector/J Driver, and MySQL Sakila sample database. If on Linux, you could use the command line, or a native application management application, like Synaptic Package Manager, to perform most of the installations. To get the latest software and installation and configuration recommendations, I prefer to download and install them myself from the MySQL web site. All links are included at the end of this post.
For reference when following this post, I have installed MySQL Server 5.5.x on 64-bit Ubuntu 12.10 LTS, running within a Windows version of Oracle VM VirtualBox. I will be using the latest Linux version of NetBeans IDE 7.3 to develop the demonstration project. I will host the project on Oracle’s GlassFish Open Source Application Server 3.1.2.2, running on Ubuntu. Lastly, I will be referring to the latest JDK 1.7, in NetBeans, for the project.
MySQL Demo User Account
Once MySQL is installed and running, I suggest adding a new MySQL demo user account, to the Sakila database for this demonstration, using MySQL Workbench. For security, you should limit the user account to just those permissions necessary for this demonstration, as detailed in the following screen-grabs. You can also add the user from the command line, if you are familiar with administering MySQL in that way.
New MySQL Database Connection
To begin development in NetBeans, first create a new JDBC database connection to the MySQL Sakila database. In the Services tab, right-click on the Databases item and select New Connection… Use the new demo user account for the connection.
Note in the first screen-grab below, that instead of using the default NetBeans JDBC MySQL Connector/J driver version, I have downloaded and replaced it with the most current version, 5.1.24. This is not necessary, but I like to use the latest drivers to avoid problems.
Make sure to test your connection before finishing, using the ‘Test’ button. It’s frustrating to track down database connection issues once you start coding and testing.
New Java Class Library
Similar to an earlier post, create new Java Class Library project in NetBeans. Select New Project -> Java -> Java Class Library. This library will eventually contain the JPA entity classes, mapped to tables and views in the MySQL Sakila database. Following standard n-tier design principles, I prefer separate the data access layer (DAL) from the service layer. You can then reuse the data access layer for other types of data-consumers, such as SOAP-based services.
Entity Classes from Database
Next, we will add entity classes to our project, mapped to several of the MySQL Sakila database’s tables and views. Right-click on the project and select New -> Entity Classes from Database… In the next window, choose the database connection we made before. NetBeans will then load all the available tables and views from the Sakila database. Next, select ‘actor_info(view)’, ‘film_actor’, and ‘film_list(view)’. Three related tables will also be added automatically by NetBeans. Not the warning at the bottom of the window about the need to specify Entity IDs. We will address this next.
When selecting ‘Entity Classes from Database…’, NetBeans adds the ‘EclipseLink (JPA 2.0)’ global library to the project. This library contains three jars, including EclipseLink 2.3.x, Java Persistence API (JPA) 2.0.x, and state model API for JPQL queries. There is a newer EclipseLink 2.4.x library available from their web site. The 2.4.x version has many new features. You can download and replace NetBeans’ EclipseLink (JPA 2.0) library by creating a new EclipseLink 2.4.x library, if you want to give its new features, like JPA-RS, a try. It is not necessary for this demonstration, however.
Adding Entity IDs to Views
To eliminate warnings displayed when we built the entities, Entity ID’s must be designated for the two database views we selected, ‘actor_info(view)’ and ‘film_list(view)’. Database views (virtual tables), do not have a primary key defined, which NetBeans requires for the entity classes. NetBeans will guide you through adding the ID, if you click on the error icon shown below.
ActorInfo.java Entity Class contents:
package com.mysql.entities; | |
import java.io.Serializable; | |
import javax.persistence.Basic; | |
import javax.persistence.Column; | |
import javax.persistence.Entity; | |
import javax.persistence.Id; | |
import javax.persistence.Lob; | |
import javax.persistence.NamedQueries; | |
import javax.persistence.NamedQuery; | |
import javax.persistence.Table; | |
import javax.xml.bind.annotation.XmlRootElement; | |
@Entity | |
@Table(name = "actor_info") | |
@XmlRootElement | |
@NamedQueries({ | |
@NamedQuery(name = "ActorInfo.findAll", query = "SELECT a FROM ActorInfo a"), | |
@NamedQuery(name = "ActorInfo.findByActorId", query = "SELECT a FROM ActorInfo a WHERE a.actorId = :actorId"), | |
@NamedQuery(name = "ActorInfo.findByFirstName", query = "SELECT a FROM ActorInfo a WHERE a.firstName = :firstName"), | |
@NamedQuery(name = "ActorInfo.findByLastName", query = "SELECT a FROM ActorInfo a WHERE a.lastName = :lastName")}) | |
public class ActorInfo implements Serializable { | |
private static final long serialVersionUID = 1L; | |
@Basic(optional = false) | |
@Column(name = "actor_id") | |
@Id | |
private short actorId; | |
@Basic(optional = false) | |
@Column(name = "first_name") | |
private String firstName; | |
@Basic(optional = false) | |
@Column(name = "last_name") | |
private String lastName; | |
@Lob | |
@Column(name = "film_info") | |
private String filmInfo; | |
public ActorInfo() { | |
} | |
public short getActorId() { | |
return actorId; | |
} | |
public void setActorId(short actorId) { | |
this.actorId = actorId; | |
} | |
public String getFirstName() { | |
return firstName; | |
} | |
public void setFirstName(String firstName) { | |
this.firstName = firstName; | |
} | |
public String getLastName() { | |
return lastName; | |
} | |
public void setLastName(String lastName) { | |
this.lastName = lastName; | |
} | |
public String getFilmInfo() { | |
return filmInfo; | |
} | |
public void setFilmInfo(String filmInfo) { | |
this.filmInfo = filmInfo; | |
} | |
} |
New Java Web Application
Next, we will create the RESTful Web Services. Each service will be mapped to one of the corresponding JPA entity we just created in the Java class library project. Select New Project -> Java Web -> Web Application.
RESTful Web Services from Entity Classes
Before we will build the RESTful web services, we need to add a reference to the previous Java class library project, containing the JPA entity classes. In the Java web application’s properties dialog window, under Categories -> Libraries -> Compile, add a link to the Java class library project’s .jar file.
Next, right-click on the project and select New -> RESTful Web Services from Entity Classes…
In the preceding dialogue window, add all the ‘Available Entity Classes’ to the ‘Selected Entity Classes’ column.
After clicking next, you will prompted to configure the Persistence Unit and the Persistence Unit’s Data Source. Please refer to my earlier post for more information on the Persistence Unit. This data source will also be used by GlassFish, once the project is deployed, to connect to the Sakila MySQL database. The Persistence Unit will use the JNDI name to reference the data source.
Persistence Unit (persistence.xml) contents:
<?xml version="1.0" encoding="UTF-8"?> | |
<persistence version="2.0" xmlns="http://java.sun.com/xml/ns/persistence" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://java.sun.com/xml/ns/persistence http://java.sun.com/xml/ns/persistence/persistence_2_0.xsd"> | |
<persistence-unit name="MySQLDemoServicePU" transaction-type="JTA"> | |
<jta-data-source>jdbc/mysql_sakila</jta-data-source> | |
<class>com.mysql.entities.Actor</class> | |
<class>com.mysql.entities.ActorInfo</class> | |
<class>com.mysql.entities.Film</class> | |
<class>com.mysql.entities.FilmActor</class> | |
<class>com.mysql.entities.Language</class> | |
<exclude-unlisted-classes>true</exclude-unlisted-classes> | |
<properties/> | |
</persistence-unit> | |
</persistence> |
As part of constructing the RESTful Web Services, notice NetBeans has added several Jersey (JAX-RS) libraries to the project. These libraries also reference Jackson (JSON Processor), Jettison (JSON StAX), MOXy (JAXB), and Grizzly (NIO) APIs.
Creating RESTful Web Services Test
Finally, we will test the RESTful Web Services, and indirectly the underlying entity classes mapped to the MySQL Sakila database. NetBeans makes this easy. To begin, right-click on the ‘RESTful Web Services’ folder in the Java web application project and select ‘Test RESTful Web Services’. NetBeans will automatically generate all the necessary files and links to test each of the RESTful web services’ operations.
As part of creating the tests, NetBeans will deploy the web application to GlassFish. When configuring the tests in the ‘Configure REST Test Client’ dialog window, make sure to use the second option, ‘Web Test Client in Project’. The first option only works with Microsoft’s Internet Explorer, an odd choice for a Java-based application running on Linux.
Highlighted below in red are the components NetBeans will install on the GlassFish application server. They include the RESTful web services application, a .war file. Each of the RESTful web service are Stateless Session Beans, installed as part of the application. In deployment also includes a JDBC Resource and a JDBC Connection Pool, which connects the application to the MySQL Sakila database. The Resource is automatically associated with the Connection Pool.
After creating the necessary files and deploying the application, NetBeans will open a web browser. allowing you can test the services. Each of the RESTful web services is available to test by clicking on the links in the left-hand navigation menu. NetBeans has generated a few default operations, including ‘{id}’, ‘{from/to}’, and ‘count’, each mapped to separate methods in the service classes. Also notice you can choose to display the results of the service calls in multiple formats, including XML, JSON, and plain text.
We can also test the RESTful Web Services by calling the service URLs, directly. Below, is the results of a my call to the Actor service’s URL, from a separate Windows client machine.
You can also use applications like Fiddler, cURL, Firefox with Firebug, and Google Chrome’s Advanced REST Client and REST Console to test the services. Below, I used Fiddler to call the Actor service, again. Note the response contains a JSON payload, not XML. With Jersey, you can request and receive JSON from the services without additional programming.
Conclusion
Using these services, you can build any number of server-side and client-side data-driven applications. The service layer is platform agnostic, accessible from any web-browser, mobile device, or native desktop application, on Windows, Linux, and Apple.
Links
MySQL Server: http://www.mysql.com/downloads/mysql
MySQL Connector/J JDBC driver for MySQL: http://dev.mysql.com/downloads/connector/j
MySQL Workbench: http://www.mysql.com/downloads/workbench
MySQL Sakila Sample Database: http://dev.mysql.com/doc/sakila/en/sakila-installation.html
NetBeans IDE: http://www.netbeans.org
EclipseLink: http://projects.eclipse.org/projects/rt.eclipselink
RESTful Mobile: Consuming Java EE RESTful Web Services Using jQuery Mobile
Posted by Gary A. Stafford in Java Development, Mobile HTML Development, Software Development on October 18, 2012
Use jQuery Mobile to build a mobile HTML website, capable of calling Jersey-specific Java EE RESTful web services and displaying JSONP in a mobile web browser.
Both NetBeans projects used in this post are available on DropBox. If you like DropBox, please use this link to sign up for a free 2 GB account. It will help me post more files to DropBox for future posts.
Background
In the previous two-part series, Returning JSONP from Java EE RESTful Web Services Using jQuery, Jersey, and GlassFish, we created a Jersey-specific RESTful web service from a database using EclipseLink (JPA 2.0 Reference Implementation), Jersey (JAX-RS Reference Implementation), JAXB, and Jackson Java JSON-processor. The service and associated entity class mapped to a copy of Microsoft SQL Server’s Adventure Works database. An HTML and jQuery-based client called the service, which returned a JSONP response payload. The JSON data it contained was formatted and displayed in a simple HTML table, in a web-browser.
Objectives
In this post, we will extend the previous example to the mobile platform. Using jQuery and jQuery Mobile JavaScript libraries, we will call two RESTful web services and display the resulting JSONP data using the common list/detail UX design pattern. We will display a list of Adventure Works employees. When the end-user clicks on an employee in the web-browser, a new page will display detailed demographic information about that employee.
Similar to the previous post, when the client website is accessed by the end-user in a mobile web browser, the client site’s HTML, CSS, and JavaScript files are downloaded and cached on the end-users machine. The JavaScript file, using jQuery and Ajax, makes a call to the RESTful web service, which returns JSON (or, JSONP in this case). This simulates a typical cross-domain situation where a client needs to consume RESTful web services from a remote source. This is not allowed by the same origin policy, but overcome by returning JSONP to the client, which wraps the JSON payload in a function call.
We will extend both the ‘JerseyRESTfulServices’ and ‘JerseyRESTfulClient’ projects we built in the last series of posts. Here are the high-level steps we will walk-through in this post:
- Create a second view (virtual table) in the Adventure Works database;
- Create a second entity class that maps to the new database view;
- Modify the existing entity class, adding JAXB and Jackson JSON annotations;
- Create a second Jersey-specific RESTful web service from the new entity using Jersey and Jackson;
- Modify the existing Jersey-specific RESTful web service, adding one new methods;
- Modify the web.xml file to allow us to use natural JSON notation;
- Implement a JAXBContext resolver to serialize the JSON using natural JSON notation;
- Create a simple list/detail two-page mobile HTML5 website using jQuery Mobile;
- Use jQuery, Ajax, and CSS to call, parse, and display the JSONP returned by the service.
RESTful Web Services Project
When we are done, the final RESTful web services projects will look like the screen-grab, below. It will contain (2) entity classes, (2) RESTful web service classes, (1) JAXBContext resolver class, and the web.xml configuration file:
1: Create the Second Database View
Create a new database view, vEmployeeNames
, in the Adventure Works database:
USE [AdventureWorks] GO SET ANSI_NULLS ON GO SET QUOTED_IDENTIFIER ON GO CREATE VIEW [HumanResources].[vEmployeeNames] AS SELECT TOP (100) PERCENT BusinessEntityID, REPLACE(RTRIM(LastName + COALESCE (' ' + Suffix + '', N'') + COALESCE (', ' + FirstName + ' ', N'') + COALESCE (MiddleName + ' ', N'')), ' ', ' ') AS FullName FROM Person.Person WHERE (PersonType = 'EM') ORDER BY FullName GO
2: Create the Second Entity
Add the new VEmployeeNames.java
entity class, mapped to the vEmployeeNames
database view, using NetBeans’ ‘Entity Classes from Database…’ wizard. Then, modify the class to match the code below.
package entities; import java.io.Serializable; import javax.persistence.Basic; import javax.persistence.Column; import javax.persistence.Entity; import javax.persistence.Id; import javax.persistence.NamedQueries; import javax.persistence.NamedQuery; import javax.persistence.Table; import javax.validation.constraints.NotNull; import javax.validation.constraints.Size; import javax.xml.bind.annotation.XmlRootElement; import javax.xml.bind.annotation.XmlType; @Entity @Table(name = "vEmployeeNames", catalog = "AdventureWorks", schema = "HumanResources") @XmlRootElement(name = "vEmployeeNames") @NamedQueries({ @NamedQuery(name = "VEmployeeNames.findAll", query = "SELECT v FROM VEmployeeNames v"), @NamedQuery(name = "VEmployeeNames.findByBusinessEntityID", query = "SELECT v FROM VEmployeeNames v WHERE v.businessEntityID = :businessEntityID"), @NamedQuery(name = "VEmployeeNames.findByFullName", query = "SELECT v FROM VEmployeeNames v WHERE v.fullName = :fullName")}) public class VEmployeeNames implements Serializable { private static final long serialVersionUID = 1L; @Basic(optional = false) @NotNull @Id @Column(name = "BusinessEntityID") private int businessEntityID; @Basic(optional = false) @NotNull @Size(min = 1, max = 102) @Column(name = "FullName") private String fullName; public VEmployeeNames() { } public int getBusinessEntityID() { return businessEntityID; } public void setBusinessEntityID(int businessEntityID) { this.businessEntityID = businessEntityID; } public String getFullName() { return fullName; } public void setFullName(String fullName) { this.fullName = fullName; } }
3: Modify the Existing Entity
Modify the existing VEmployee.java
entity class to use JAXB and Jackson JSON Annotations as shown below (class code abridged). Note the addition of the @XmlType(propOrder = { "businessEntityID"... })
to the class, the @JsonProperty(value = ...)
tags to each member variable, and the @Id
tag to the businessEntityID
, which serves as the entity’s primary key. We will see the advantages of the first two annotations later in the post when we return the JSON to the client.
package entities; import java.io.Serializable; import javax.persistence.Basic; import javax.persistence.Entity; import javax.persistence.Id; import javax.persistence.NamedQueries; import javax.persistence.NamedQuery; import javax.persistence.Table; import javax.validation.constraints.NotNull; import javax.validation.constraints.Size; import javax.xml.bind.annotation.XmlRootElement; import javax.xml.bind.annotation.XmlType; import org.codehaus.jackson.annotate.JsonProperty; @Entity @Table(name = "vEmployee", catalog = "AdventureWorks", schema = "HumanResources") @XmlRootElement @NamedQueries({ @NamedQuery(name = "VEmployee.findAll", query = "SELECT v FROM VEmployee v"), ...}) @XmlType(propOrder = { "businessEntityID", "title", "firstName", "middleName", "lastName", "suffix", "jobTitle", "phoneNumberType", "phoneNumber", "emailAddress", "emailPromotion", "addressLine1", "addressLine2", "city", "stateProvinceName", "postalCode", "countryRegionName", "additionalContactInfo" }) public class VEmployee implements Serializable { private static final long serialVersionUID = 1L; @Basic(optional = false) @NotNull @Id @JsonProperty(value = "Employee ID") private int businessEntityID; @Size(max = 8) @JsonProperty(value = "Title") private String title; @Basic(optional = false) @NotNull @Size(min = 1, max = 50) @JsonProperty(value = "First Name") private String firstName; @Size(max = 50) @JsonProperty(value = "Middle Name") private String middleName; @Basic(optional = false) @NotNull @Size(min = 1, max = 50) @JsonProperty(value = "Last Name") private String lastName; @Size(max = 10) @JsonProperty(value = "Suffix") private String suffix; @Basic(optional = false) @NotNull @Size(min = 1, max = 50) @JsonProperty(value = "Job Title") private String jobTitle; @Size(max = 25) @JsonProperty(value = "Phone Number") private String phoneNumber; @Size(max = 50) @JsonProperty(value = "Phone Number Type") private String phoneNumberType; @Size(max = 50) @JsonProperty(value = "Email Address") private String emailAddress; @Basic(optional = false) @NotNull @JsonProperty(value = "Email Promotion") private int emailPromotion; @Basic(optional = false) @NotNull @Size(min = 1, max = 60) @JsonProperty(value = "Address Line 1") private String addressLine1; @Size(max = 60) @JsonProperty(value = "Address Line 2") private String addressLine2; @Basic(optional = false) @NotNull @Size(min = 1, max = 30) @JsonProperty(value = "City") private String city; @Basic(optional = false) @NotNull @Size(min = 1, max = 50) @JsonProperty(value = "State or Province Name") private String stateProvinceName; @Basic(optional = false) @NotNull @Size(min = 1, max = 15) @JsonProperty(value = "Postal Code") private String postalCode; @Basic(optional = false) @NotNull @Size(min = 1, max = 50) @JsonProperty(value = "Country or Region Name") private String countryRegionName; @Size(max = 2147483647) @JsonProperty(value = "Additional Contact Info") private String additionalContactInfo; public VEmployee() { } ... }
4: Create the New RESTful Web Service
Add the new VEmployeeNamesFacadeREST.java
RESTful web service class using NetBean’s ‘RESTful Web Services from Entity Classes…’ wizard. Then, modify the new class, adding the new findAllJSONP()
method shown below (class code abridged). This method call the same super.findAll()
method from the parent AbstractFacade.java
class as the default findAll({id})
method. However, the findAllJSONP()
method returns JSONP instead of XML or JSON, as findAll({id})
does. This is done by passing the results of super.findAll()
to a new instance of Jersey’s JSONWithPadding()
class (com.sun.jersey.api.json.JSONWithPadding
).
package service; import com.sun.jersey.api.json.JSONWithPadding; import entities.VEmployeeNames; import java.util.ArrayList; import java.util.Collection; import java.util.List; import javax.ejb.Stateless; import javax.persistence.EntityManager; import javax.persistence.PersistenceContext; import javax.persistence.criteria.CriteriaBuilder; import javax.persistence.criteria.CriteriaQuery; import javax.persistence.criteria.Root; import javax.ws.rs.Consumes; import javax.ws.rs.DELETE; import javax.ws.rs.GET; import javax.ws.rs.POST; import javax.ws.rs.PUT; import javax.ws.rs.Path; import javax.ws.rs.PathParam; import javax.ws.rs.Produces; import javax.ws.rs.QueryParam; import javax.ws.rs.core.GenericEntity; @Stateless @Path("entities.vemployeenames") public class VEmployeeNamesFacadeREST extends AbstractFacade<VEmployeeNames> { ... @GET @Path("jsonp") @Produces({"application/javascript"}) public JSONWithPadding findAllJSONP(@QueryParam("callback") String callback) { CriteriaBuilder cb = getEntityManager().getCriteriaBuilder(); CriteriaQuery cq = cb.createQuery(); Root empRoot = cq.from(VEmployeeNames.class); cq.select(empRoot); cq.orderBy(cb.asc(empRoot.get("fullName"))); javax.persistence.Query q = getEntityManager().createQuery(cq); List<VEmployeeNames> employees = q.getResultList(); return new JSONWithPadding( new GenericEntity<Collection<VEmployeeNames>>(employees) { }, callback); } ... }
5: Modify the Existing Service
Modify the existing VEmployeeFacadeREST.java
RESTful web service class, adding the findJSONP()
method shown below (class code abridged). This method calls the same super.find({id})
in the AbstractFacade.java
parent class as the default find({id})
method, but returns JSONP instead of XML or JSON. As with the previous service class above, this is done by passing the results to a new instance of Jersey’s JSONWithPadding()
class (com.sun.jersey.api.json.JSONWithPadding
). There are no changes required to the default AbstractFacade.java
class.
package service; import com.sun.jersey.api.json.JSONWithPadding; import entities.VEmployee; import java.util.ArrayList; import java.util.Collection; import java.util.List; import javax.ejb.Stateless; import javax.persistence.EntityManager; import javax.persistence.PersistenceContext; import javax.persistence.criteria.CriteriaBuilder; import javax.persistence.criteria.CriteriaQuery; import javax.persistence.criteria.Root; import javax.ws.rs.Consumes; import javax.ws.rs.DELETE; import javax.ws.rs.GET; import javax.ws.rs.POST; import javax.ws.rs.PUT; import javax.ws.rs.Path; import javax.ws.rs.PathParam; import javax.ws.rs.Produces; import javax.ws.rs.QueryParam; import javax.ws.rs.core.GenericEntity; @Stateless @Path("entities.vemployee") public class VEmployeeFacadeREST extends AbstractFacade<VEmployee> { ... @GET @Path("{id}/jsonp") @Produces({"application/javascript"}) public JSONWithPadding findJSONP(@PathParam("id") Integer id, @QueryParam("callback") String callback) { List<VEmployee> employees = new ArrayList<VEmployee>(); employees.add(super.find(id)); return new JSONWithPadding( new GenericEntity<Collection<VEmployee>>(employees) { }, callback); } ... }
6: Allow POJO JSON Support
Add the JSONConfiguration.FEATURE_POJO_MAPPING
servlet init parameter to web.xml, as shown below (xml abridged). According to the Jersey website, this will allow us to use POJO support, the easiest way to convert our Java Objects to JSON. It is based on the Jackson library.
<?xml version="1.0" encoding="UTF-8"?> <web-app version="3.0" xmlns="http://java.sun.com/xml/ns/javaee" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://java.sun.com/xml/ns/javaee http://java.sun.com/xml/ns/javaee/web-app_3_0.xsd"> <servlet> <servlet-name>ServletAdaptor</servlet-name> <servlet-class>com.sun.jersey.spi.container.servlet.ServletContainer</servlet-class> <init-param> <description>Multiple packages, separated by semicolon(;), can be specified in param-value</description> <param-name>com.sun.jersey.config.property.packages</param-name> <param-value>service</param-value> </init-param> <init-param> <param-name>com.sun.jersey.api.json.POJOMappingFeature</param-name> <param-value>true</param-value> </init-param> ...
7: Implement a JAXBContext Resolver
Create the VEmployeeFacadeREST.java
JAXBContext resolver class, shown below. This allows us to serialize the JSON using natural JSON notation. A good explanation of the use of a JAXBContext resolver can be found on the Jersey website.
package config; import com.sun.jersey.api.json.JSONConfiguration; import com.sun.jersey.api.json.JSONJAXBContext; import javax.ws.rs.ext.ContextResolver; import javax.ws.rs.ext.Provider; import javax.xml.bind.JAXBContext; @Provider public class JAXBContextResolver implements ContextResolver<JAXBContext> { JAXBContext jaxbContext; private Class[] types = {entities.VEmployee.class, entities.VEmployeeNames.class}; public JAXBContextResolver() throws Exception { this.jaxbContext = new JSONJAXBContext(JSONConfiguration.natural().build(), types); } @Override public JAXBContext getContext(Class<?> objectType) { for (Class type : types) { if (type == objectType) { return jaxbContext; } } return null; } }
What is Natural JSON Notation?
According to the Jersey website, “with natural notation, Jersey will automatically figure out how individual items need to be processed, so that you do not need to do any kind of manual configuration. Java arrays and lists are mapped into JSON arrays, even for single-element cases. Java numbers and booleans are correctly mapped into JSON numbers and booleans, and you do not need to bother with XML attributes, as in JSON, they keep the original names.”
What does that mean? Better yet, what does that look like? Here is an example of an employee record, first as plain old JAXB JSON in a JSONP wrapper:
callback({"vEmployee":{"businessEntityID":"211","firstName":"Hazem","middleName":"E","lastName":"Abolrous","jobTitle":"Quality Assurance Manager","phoneNumberType":"Work","phoneNumber":"869-555-0125","emailAddress":"hazem0@adventure-works.com","emailPromotion":"0","addressLine1":"5050 Mt. Wilson Way","city":"Kenmore","stateProvinceName":"Washington","postalCode":"98028","countryRegionName":"United States"}})
And second, JSON wrapped in JSONP, using Jersey’s natural notation. Note the differences in the way the parent vEmployee node, numbers, and nulls are handled in natural JSON notation.
callback([{"Employee ID":211,"Title":null,"First Name":"Hazem","Middle Name":"E","Last Name":"Abolrous","Suffix":null,"Job Title":"Quality Assurance Manager","Phone Number Type":"Work","Phone Number":"869-555-0125","Email Address":"hazem0@adventure-works.com","Email Promotion":0,"Address Line 1":"5050 Mt. Wilson Way","Address Line 2":null,"City":"Kenmore","State or Province Name":"Washington","Postal Code":"98028","Country or Region Name":"United States","Additional Contact Info":null}])
Mobile Client Project
When we are done with the mobile client, the final RESTful web services mobile client NetBeans projects should look like the screen-grab, below. Note the inclusion of jQuery Mobile 1.2.0. You will need to download the library and associated components, and install them in the project. I chose to keep them in a separate folder since there were several files included with the library. This example requires a few new features introduced in jQuery Mobile 1.2.0. Make sure to get this version or later.
8: Create a List/Detail Mobile HTML Site
The process to display the data from the Adventure Works database in the mobile web browser is identical to the process used in the last series of posts. We are still using jQuery with Ajax, calling the same services, but with a few new methods. The biggest change is the use of jQuery Mobile to display the employee data. The jQuery Mobile library, especially with the release of 1.2.0, makes displaying data, quick and elegant. The library does all the hard work under the covers, with the features such as the listview control. We simply need to use jQuery and Ajax to retrieve the data and pass it to the control.
We will create three new files. They include the HTML, CSS, and JavaScript files. We add a ‘.m’ to the file names to differentiate them from the normal web browser files from the last post. As with the previous post, the HTML page and CSS file are minimal. The HTML page uses the jQuery Mobile multi-page template available on the jQuery Mobile website. Although it appears as two different web pages to the end-user, it is actually a single-page site.
Source code for employee.m.html:
<!DOCTYPE html> <html> <head> <title>Employee List</title> <meta name="viewport" content="width=device-width, initial-scale=1"> <meta http-equiv="Content-Type" content="text/html; charset=UTF-8"> <link rel="stylesheet" href="jquery.mobile-1.2.0/jquery.mobile-1.2.0.min.css" /> <link type="text/css" rel="stylesheet" href="employees.m.css" /> <script src="jquery-1.8.2.min.js" type="text/javascript"></script> <script src="jquery.mobile-1.2.0/jquery.mobile-1.2.0.min.js" type="text/javascript"></script> <script src="employees.m.js" type="text/javascript"></script> </head> <body> <!-- Start of first page: #one --> <div data-role="page" id="one" data-theme="b"> <div data-role="header" data-theme="b"> <h1>Employee List</h1> </div><!-- /header --> <div data-role="content"> <div id="errorMessage"></div> <div class="ui-grid-solo"> <form> <ul data-role="listview" data-filter="true" id="employeeList" data-theme="c" data-autodividers="true"> </ul> </form> </div> </div><!-- /content --> <div data-role="footer" data-theme="b"> <h4>Programmatic Ponderings, 2012</h4> </div><!-- /footer --> </div><!-- /page --> <!-- Start of second page: #two --> <div data-role="page" id="two" data-theme="c"> <div data-role="header" data-theme="b"> <a href="#one" data-icon="back">Return</a> <h1>Employee Detail</h1> </div><!-- /header --> <div data-role="content" data-theme="c"> <div id="employeeDetail"></div> </div><!-- /content --> <div data-role="footer" data-theme="b"> <h4>Programmatic Ponderings, 2012</h4> </div><!-- /footer --> </div><!-- /page two --> </body> </html>
Source code for employee.m.css:
#employeeList { clear:both; } #employeeDetail div { padding-top: 2px; white-space: nowrap; } .field { margin-bottom: 0px; font-size: smaller; color: #707070; } .value { font-weight: bolder; padding-bottom: 12px; border-bottom: 1px #d0d0d0 solid; } .ui-block-a{ padding-left: 6px; padding-right: 6px; } .ui-grid-a{ padding-bottom: 12px; padding-top: -6px; }
8: Retrieve, Parse, and Display the Data
The mobile JavaScript file below is identical in many ways to the JavaScript file used in the last series of posts for a non-mobile browser. One useful change we have made is the addition of two arguments to the function that calls jQuery.Ajax()
. The address of the service (URI) that the jQuery.Ajax()
method requests, and the function that Ajax calls after successful completion, are both passed into the callService(Uri, successFunction)
function as arguments. This allows us to reuse the Ajax method for different purposes. In this case, we call the function once to populate the Employee List with the full names of the employees. We call it again to populate the Employee Detail page with demographic information of a single employee chosen from the Employee List. Both calls are to different URIs representing the two different RESTful web services, which in turn are associated with the two different entities, which in turn are mapped to the two different database views.
callService = function (uri, successFunction) { $.ajax({ cache: true, url: uri, data: "{}", type: "GET", contentType: "application/javascript", dataType: "jsonp", error: ajaxCallFailed, failure: ajaxCallFailed, success: successFunction }); };
The rest of the functions are self-explanatory. There are two calls to the jQuery Ajax method to return data from the service, two functions to parse and format the JSONP for display in the browser, and one jQuery method that adds click events to the Employee List. We perform a bit of string manipulation to imbed the employee id into the id
property of each list item (li
element. Later, when the end-user clicks on the employee name in the list, the employee id is extracted from the id property of the selected list item and passed back to the service to retrieve the employee detail. The HTML snippet below shows how a single employee row in the jQuery listview. Note the id
property of the li
element, id="empId_121"
, for employee id 121.
<li id="empId_121" class="ui-btn ui-btn-icon-right ui-li-has-arrow ui-li ui-btn-up-c" data-corners="false" data-shadow="false" data-iconshadow="true" data-wrapperels="div" data-icon="arrow-r" data-iconpos="right" data-theme="c"> <div class="ui-btn-inner ui-li"> <div class="ui-btn-text"> <a class="ui-link-inherit" href="#">Ackerman, Pilar G</a> </div> <span class="ui-icon ui-icon-arrow-r ui-icon-shadow"> </span> </div> </li>
To make this example work, you need to change the restfulWebServiceBaseUri
variable to the server and port of the GlassFish domain running your RESTful web services. If you are testing the client locally on your mobile device, I suggest using the IP address for the GlassFish server versus a domain name, which your phone will be able to connect to in your local wireless environment. At least on the iPhone, there is no easy way to change the hosts file to provide local domain name resolution.
Source code for employee.m.js:
// =========================================================================== // // Author: Gary A. Stafford // Website: http://www.programmaticponderings.com // Description: Call RESTful Web Services from mobile HTML pages // using jQuery mobile, Jersey, Jackson, and EclipseLink // // =========================================================================== // Immediate function (function () { "use strict"; var restfulWebServiceBaseUri, employeeListFindAllUri, employeeByIdUri, callService, ajaxCallFailed, getEmployeeById, displayEmployeeList, displayEmployeeDetail; // Base URI of RESTful web service restfulWebServiceBaseUri = "http://your_server_name_or_ip:8080/JerseyRESTfulServices/webresources/"; // URI maps to service.VEmployeeNamesFacadeREST.findAllJSONP employeeListFindAllUri = restfulWebServiceBaseUri + "entities.vemployeenames/jsonp"; // URI maps to service.VEmployeeFacadeREST.findJSONP employeeByIdUri = restfulWebServiceBaseUri + "entities.vemployee/{id}/jsonp"; // Execute after the page one dom is fully loaded $(".one").ready(function () { // Retrieve employee list callService(employeeListFindAllUri, displayEmployeeList); // Attach onclick event to each row of employee list on page one $("#employeeList").on("click", "li", function(event){ getEmployeeById($(this).attr("id").split("empId_").pop()); }); }); // Call a service URI and return JSONP to a function callService = function (Uri, successFunction) { $.ajax({ cache: true, url: Uri, data: "{}", type: "GET", contentType: "application/javascript", dataType: "jsonp", error: ajaxCallFailed, failure: ajaxCallFailed, success: successFunction }); }; // Called if ajax call fails ajaxCallFailed = function (jqXHR, textStatus) { console.log("Error: " + textStatus); console.log(jqXHR); $("form").css("visibility", "hidden"); $("#errorMessage").empty(). append("Sorry, there was an error."). css("color", "red"); }; // Display employee list on page one displayEmployeeList = function (employee) { var employeeList = ""; $.each(employee, function(index, employee) { employeeList = employeeList.concat( "<li id=empId_" + employee.businessEntityID.toString() + ">" + "<a href='#'>" + employee.fullName.toString() + "</a></li>"); }); $('#employeeList').empty(); $('#employeeList').append(employeeList).listview("refresh", true); }; // Display employee detail on page two displayEmployeeDetail = function(employee) { $.mobile.loading( 'show', { text: '', textVisible: false, theme: 'a', html: "" }); window.location = "#two"; var employeeDetail = ""; $.each(employee, function(key, value) { $.each(value, function(key, value) { if(!value) { value = " "; } employeeDetail = employeeDetail.concat( "<div class='detail'>" + "<div class='field'>" + key + "</div>" + "<div class='value'>" + value + "</div>" + "</div>"); }); }); $("#employeeDetail").empty().append(employeeDetail); }; // Retrieve employee detail based on employee id getEmployeeById = function (employeeID) { callService(employeeByIdUri.replace("{id}", employeeID), displayEmployeeDetail); }; } ());
The Final Result
Viewed in Google’s Chrome for Mobile web browser on iOS 6, the previous project’s Employee List looks pretty bland and un-mobile like:
However, with a little jQuery Mobile magic you get a simple yet effective and highly functional mobile web presentation. Seen below on page one, the Employee List is displayed in Safari on an iPhone 4 with iOS 6. It features some of the new capabilities of jQuery Mobile 1.2.0’s improved listview, including autodividers.
Here again is the Employee List using the jQuery Mobile 1.2.0’s improved listview search filter bar:
Here is the Employee Detail on page 2. Note the order and names of the fields. Remember previously when we annotated the VEmployeeNames.java
entity with the @XmlType(propOrder = {"businessEntityID", ...})
to the class and the @JsonProperty(value = ...)
tags to each member variable. This is the results of those efforts; our JSON is delivered pre-sorted and titled the way we want. No need to handle those functions on the client-side. This allows the client to be loosely-coupled to the data. The client simply displays whichever key/value pairs are delivered in the JSONP response payload.
Returning JSONP from Java EE RESTful Web Services Using jQuery, Jersey, and GlassFish – Part 2 of 2
Posted by Gary A. Stafford in Java Development, Software Development on October 4, 2012
Create a Jersey-specific Java EE RESTful web service, and an HTML-based client to call the service and display JSONP. Test and deploy the service and the client to different remote instances of GlassFish.
Background
In part 1 of this series, we created a Jersey-specific RESTful web service from a database using NetBeans. The service returns JSONP in addition to JSON and XML. The service was deployed to a GlassFish domain, running on a Windows box. On this same box is the SQL Server instance, running the Adventure Works database, from which the service obtains data, via the entity class.
Objectives
In part two of this series, we will create a simple web client to consume and display the JSONP returned by the RESTful web service. There are many options available for creating a service consumer (client) depending on your development platform and project requirements. We will keep it simple, no complex, complied code, just HTML and JavaScript with jQuery, the well-known JavaScript library.
We will host the client on a separate GlassFish domain, running on an Ubuntu Linux VM using Oracle’s VM VirtualBox. This is a different machine than the service was installed on. When opened by the end-user in a web browser, the client files, including the JavaScript file that calls the service, are downloaded to the end-users machine. This will simulate a typical cross-domain situation where a client application needs to consume RESTful web services from a remote source. This is not allowed by the same origin policy, but overcome by returning JSONP to the client, which wraps the JSON payload in a function call.
Here are the high-level steps we will walk-through in part two:
- Create a simple HTML client using jQuery and ajax to call the RESTful web service;
- Add jQuery functionality to parse and display the JSONP returned by the service;
- Deploy the client to a separate remote instance of GlassFish using Apache Ant;
- Test the client’s ability to call the service across domains and display JSONP.
Creating the RESTful Web Service Client
New NetBeans Web Application Project
Create a new Java Web Application project in NetBeans. Name the project ‘JerseyRESTfulClient’. The choice of GlassFish server and domain where the project will be deployed is unimportant. We will use Apache Ant to deploy the client when we finish the building the project. By default, I chose my local instance of GlassFish, for testing purposes.
Adding Files to Project
The final client project will contains four new files:
- employees.html – HTML web page that displays a list of employees;
- employees.css – CSS information used to by employees.html;
- employees.js – JavaScript code used to by employees.html;
- jquery-1.8.2.min.js – jQuery 1.8.2 JavaScript library, minified.
First, we need to download and install jQuery. At the time of this post, jQuery 1.8.2 was the latest version. I installed the minified version (jquery-1.8.2.min.js
) to save space.
Next, we will create the three new files (employees.html
, employees.css
, and employees.js
), using the code below. When finished, we need to place all four files into the ‘Web Pages’ folder. The final project should look like:
HTML
The HTML file is the smallest of the three files. The HTML page references the CSS file, the JavaScript file, and the jQuery library file. The CSS file provides the presentation (look and feel) and JavaScript file, using jQuery, dynamically provides much of the content that the HTML page normally would contain.
<!DOCTYPE html> <html> <head> <title>Employee List</title> <meta http-equiv="Content-Type" content="text/html; charset=UTF-8"> <link type="text/css" rel="stylesheet" href="employees.css" /> <script src="jquery-1.8.2.min.js" type="text/javascript"></script> <script src="employees.js" type="text/javascript"></script> </head> <body> <div id="pageTitle">Employee List</div> <div id="employeeList"></div> </body> </html>
Cascading Style Sheets (CSS)
The CSS file is also pretty straight-forward. The pageTitle
and employeeList
id selectors and type selectors are used directly by the HTML page. The class selectors are all applied to the page by jQuery, in the JavaScript file.
body { font-family: sans-serif; font-size: small; padding-left: 6px; } span { padding: 6px; display: inline-block; } div { border-bottom: lightgray solid 1px; } #pageTitle { font-size: medium; font-weight: bold; padding: 12px 0px 12px 0px; border: none; } #employeeList { float: left; border: gray solid 1px; } .empId { width: 50px; text-align: center; border-right: lightgray solid 1px; } .name { width: 200px; border-right: lightgray solid 1px; } .jobTitle { width: 250px; } .header { font-weight: bold; border-bottom: gray solid 1px; } .even{ background-color: rgba(0, 255, 128, 0.09); } .odd { background-color: rgba(0, 255, 128, 0.05); } .last { border-bottom: none; }
jQuery and JavaScript
The JavaScript file is where all the magic happens. There are two primary functions. First, getEmployees
, which calls the jQuery.ajax()
method. According jQuery’s website, the jQuery Ajax method performs an asynchronous HTTP (Ajax) request. In this case, it calls our RESTful web service and returns JSONP. The jQuery Ajax method uses an HTTP GET method to request the following service resource (URI):
http://[your-service's-glassfish-server-name]:[your-service's-glassfish-domain-port]/JerseyRESTfulService/webresources/entities.vemployee/{from}/{to}/jsonp?callback={callback}
.
The base (root) URI of the service in the URI above is as follows:
http://[server]:[port]/JerseyRESTfulService/webresources/entities.vemployee/
This is followed by a series of elements (nodes), {from}/{to}/jsonp
, which together form a reference to a specific method in our service. As explained in the first post of this series, we include the /jsonp
element to indicate we want to call the new findRangeJsonP
method to return JSONP, as opposed to findRange
method that returns JSON or XML. We pass the {from}
path parameter a value of ‘0’ and the {to}
path parameter a value of ‘10’.
Lastly, the method specifies the callback function name for the JSONP request, parseResponse
, using the jsonpCallback
setting. This value will be used instead of the random name automatically generated by jQuery. The callback function name is appended to the end of the URI as a query parameter. The final URL is as follows:
http://[server]:[port]/JerseyRESTfulService/webresources/entities.vemployee/0/10/jsonp?callback=parseResponse
.
Note the use of the jsonpCallback
setting is not required, or necessarily recommended by jQuery. Without it, jQuery generate a unique name as it will make it easier to manage the requests and provide callbacks and error handling. This example will work fine if you exclude the jsonpCallback: "parseResponse"
setting.
getEmployees = function () { $.ajax({ cache: true, url: restfulWebServiceURI, data: "{}", type: "GET", jsonpCallback: "parseResponse", contentType: "application/javascript", dataType: "jsonp", error: ajaxCallFailed, failure: ajaxCallFailed, success: parseResponse }); };
Once we have successfully returned the JSONP, the jQuery Ajax method calls the parseResponse(data)
function, passing the JSON to the data
argument. The parseResponse
function iterates through the employee objects using the jQuery.each()
method. Each field of data is surrounding with span
and div
tags, and concatenated to the employeeList
string variable. The string is appended to the div tag with the id of ‘employeeList’, using jQuery’s .append()
method. The result is an HTML table-like grid of employee names, ids, and job title, displayed on the employees.html
page.
Lastly, we call the colorRows()
function. This function uses jQuery’s .addClass(className)
to assign CSS classes to objects in the DOM. The classes are added to stylize the grid with alternating row colors and other formatting.
parseResponse = function (data) { var employee = data.vEmployee; var employeeList = ""; employeeList = employeeList.concat("<div class='header'>" + "<span class='empId'>Id</span>" + "<span class='name'>Employee Name</span>" + "<span class='jobTitle'>Job Title</span>" + "</div>"); $.each(employee, function(index, employee) { employeeList = employeeList.concat("<div class='employee'>" + "<span class='empId'>" + employee.businessEntityID + "</span><span class='name'>" + employee.firstName + " " + employee.lastName + "</span><span class='jobTitle'>" + employee.jobTitle + "</span></div>"); }); $("#employeeList").empty(); $("#employeeList").append(employeeList); colorRows(); };
Here are the complete JavaScript file contents:
// Immediate function (function () { "use strict"; var restfulWebServiceURI, getEmployees, ajaxCallFailed, colorRows, parseResponse; restfulWebServiceURI = "http://[your-service's-server-name]:[your-service's-port]/JerseyRESTfulService/webresources/entities.vemployee/0/10/jsonp"; // Execute after the DOM is fully loaded $(document).ready(function () { getEmployees(); }); // Retrieve Employee List as JSONP getEmployees = function () { $.ajax({ cache: true, url: restfulWebServiceURI, data: "{}", type: "GET", jsonpCallback: "parseResponse", contentType: "application/javascript", dataType: "jsonp", error: ajaxCallFailed, failure: ajaxCallFailed, success: parseResponse }); }; // Called if ajax call fails ajaxCallFailed = function (jqXHR, textStatus) { console.log("Error: " + textStatus); console.log(jqXHR); $("#employeeList").empty(); $("#employeeList").append("Error: " + textStatus); }; // Called if ajax call is successful parseResponse = function (data) { var employee = data.vEmployee; var employeeList = ""; employeeList = employeeList.concat("<div class='header'>" + "<span class='empId'>Id</span>" + "<span class='name'>Employee Name</span>" + "<span class='jobTitle'>Job Title</span>" + "</div>"); $.each(employee, function(index, employee) { employeeList = employeeList.concat("<div class='employee'>" + "<span class='empId'>" + employee.businessEntityID + "</span><span class='name'>" + employee.firstName + " " + employee.lastName + "</span><span class='jobTitle'>" + employee.jobTitle + "</span></div>"); }); $("#employeeList").empty(); $("#employeeList").append(employeeList); colorRows(); }; // Styles the Employee List colorRows = function(){ $("#employeeList .employee:odd").addClass("odd"); $("#employeeList .employee:even").addClass("even"); $("#employeeList .employee:last").addClass("last"); }; } ());
Deployment to GlassFish
To deploy the RESTful web service client to GlassFish, run the following Apache Ant target. The target first calls the clean
and dist
targets to build the .war file, Then, the target calls GlassFish’s asadmin
deploy
command. It specifies the remote GlassFish server, admin port, admin user, admin password (in the password file), secure or insecure connection, the name of the container, and the name of the .war file to be deployed. Note that the server is different for the client than it was for the service in part 1 of the series.
<target name="glassfish-deploy-remote" depends="clean, dist" description="Build distribution (WAR) and deploy to GlassFish"> <exec failonerror="true" executable="cmd" description="asadmin deploy"> <arg value="/c" /> <arg value="asadmin --host=[your-client's-glassfish-server-name] --port=[your-client's-glassfish-domain-admin-port] --user=admin --passwordfile=pwdfile --secure=false deploy --force=true --name=JerseyRESTfulClient --contextroot=/JerseyRESTfulClient dist\JerseyRESTfulClient.war" /> </exec> </target>
Although the client application does not require any Java code, JSP pages, or Servlets, I chose to use NetBeans’ Web Application project template to create the client and chose to create a .war file to make deployment to GlassFish easier. You could just install the four client files (jQuery, HTML, CSS, and JavaScript) on Apache, IIS, or any other web server as a simple HTML site.
Once the application is deployed to GlassFish, you should see the ‘JerseyRESTfulClient’ listed under the Applications tab within the remote server domain.
We will call the client application from our browser. The client application, whose files are downloaded and are now local on our machine, will in turn will call the service. The URL to call the client is: http://[your-client's-glassfish-server-name]:[your-client's-glassfish-domain-port]/JerseyRESTfulClient/employees.html
(see call-out 1, in the screen-grab, below).
Using Firefox with Firebug, we can observe a few important items once the results are displayed (see the screen-grab, below):
- The four client files (jQuery, HTML, CSS, and JavaScript) are cached after the first time the client URL loads, but the jQuery Ajax service call is never cached (call-out 2);
- All the client application files are loaded from one domain, while the service is called from another domain (call-out 3);
- The ‘parseRequest’ callback function in the JSONP response payload, wraps the JSON data (call-out 4).
The JSONP returned by the service to the client (abridged for length):
parseResponse({"vEmployee":[{"addressLine1":"4350 Minute Dr.","businessEntityID":"1","city":"Newport Hills","countryRegionName":"United States","emailAddress":"ken0@adventure-works.com","emailPromotion":"0","firstName":"Ken","jobTitle":"Chief Executive Officer","lastName":"Sánchez","middleName":"J","phoneNumber":"697-555-0142","phoneNumberType":"Cell","postalCode":"98006","stateProvinceName":"Washington"},{"addressLine1":"7559 Worth Ct.","businessEntityID":"2","city":"Renton","countryRegionName":"United States","emailAddress":"terri0@adventure-works.com","emailPromotion":"1","firstName":"Terri","jobTitle":"Vice President of Engineering","lastName":"Duffy","middleName":"Lee","phoneNumber":"819-555-0175","phoneNumberType":"Work","postalCode":"98055","stateProvinceName":"Washington"},{...}]})
The JSON passed to the parseResponse(data)
function’s data
argument (abridged for length):
{"vEmployee":[{"addressLine1":"4350 Minute Dr.","businessEntityID":"1","city":"Newport Hills","countryRegionName":"United States","emailAddress":"ken0@adventure-works.com","emailPromotion":"0","firstName":"Ken","jobTitle":"Chief Executive Officer","lastName":"Sánchez","middleName":"J","phoneNumber":"697-555-0142","phoneNumberType":"Cell","postalCode":"98006","stateProvinceName":"Washington"},{"addressLine1":"7559 Worth Ct.","businessEntityID":"2","city":"Renton","countryRegionName":"United States","emailAddress":"terri0@adventure-works.com","emailPromotion":"1","firstName":"Terri","jobTitle":"Vice President of Engineering","lastName":"Duffy","middleName":"Lee","phoneNumber":"819-555-0175","phoneNumberType":"Work","postalCode":"98055","stateProvinceName":"Washington"},{...}]}
Firebug also allows us to view the JSON in a more structured and object-oriented view:
Conclusion
We have successfully built and deployed a RESTful web service to one GlassFish domain, capable of returning JSONP. We have also built and deployed an HTML client to another GlassFish domain, capable of calling the service and displaying the JSONP. The service and client in this example have very minimal functionality. However, the service can easily be scaled to include multiple entities and RESTful services. The client’s capability can be expanded to perform a full array of CRUD operations on the database, through the RESTful web service(s).