Posts Tagged JSON

Building a Microservices-based REST API with RestExpress, Java EE, and MongoDB: Part 3

Develop a well-architected and well-documented REST API, built on a tightly integrated collection of Java EE-based microservices.

Virtual-Vehicles Architecture

Note: All code available on GitHub. For the version of the code that matches the details in this blog post, check out the master branch, v1.0.0 tag (after running git clone …, run a git checkout tags/v1.0.0 command).

Previous Posts

In Part One of this series, we introduced the microservices-based Virtual-Vehicles REST API example. The vehicle-themed Virtual-Vehicles microservices offers a comprehensive set of functionality, through a REST API, to application developers. In Part Two, we installed a copy of the Virtual-Vehicles project from GitHub. In Part Two, we also gained a basic understanding of how RestExpress works. Finally, we discovered how to get the Virtual-Vehicles microservices up and running.

Part Three

In part three of this series, we will take the Virtual-Vehicles for a test drive (get it? maybe it was funnier the first time…). There are several tools we can use to test the Virtual-Vehicles API. One of my favorite tools is Postman.  We will explore how to use Postman, along with the Virtual-Vehicles API documentation, to test the Virtual-Vehicles microservice’s endpoints, which compose the Virtual-Vehicles API.

Testing the API

There are three categories of tools available to test RESTful APIs, which are GUI-based applications, command line tools, and testing frameworks. Postman, Advanced REST ClientREST Console, and SmartBear’s SoapUI and SoapUI NG Pro, are examples of GUI-based applications, designed specifically to test RESTful APIs. cURL and GNU Wget are two examples of command line tools, which among other capabilities, can test APIs. Lastly, JUnit is an example of a testing framework that can be used to test a RESTful API. Surprisingly, JUnit is not only designed to manage unit tests. Each category of testing tools has their pros and cons, depending on your testing needs. We will explore all of these categories in this post as we test the Virtual-Vehicles REST API.

JUnit

JUnit is probably the best known of all Java unit testing frameworks. JUnit’s website describes JUnit as ‘a simple, open source framework to write and run repeatable tests. It is an instance of the xUnit architecture for unit testing frameworks.’ Most Java developers turn to JUnit for unit testing. However, JUnit is capable of other forms of testing, including integration testing. In his post, ‘Unit Testing with JUnit – Tutorial’, Lars Vogel states ‘an integration test has the target to test the behavior of a component or the integration between a set of components. The term functional test is sometimes used as a synonym for integration test. This kind of tests allow you to translate your user stories into a test suite, i.e., the test would resemble an expected user interaction with the application.’

Testing the Virtual-Vehicles RESTful API’s operations with JUnit would be considered integration (functional) testing. At a minimum, to complete requests, we call one microservice, which in turn authenticates the JWT by calling another microservice. If authenticated, the first microservice makes a request to its MongoDB database. As Vogel stated, whereas a unit test targets a small unit of code, such as a method, the request/response operation is integration between a set of components. testing an API call requires several dependencies.

The simplest example of testing the Virtual-Vehicles API with JUnit, would be to test an HTTP GET request to return a single instance of a vehicle. The code below demonstrates how this might be done. Notice the request depends on helper methods (not included, for brevity). To request the vehicle, assuming we already have a registered client, we need a valid JWT. We also need a valid vehicle ObjectId. To obtain these two pieces of data, we call helper methods, which in turn makes the necessary request to retrieve a JWT and vehicle ObjectId.

Below are the results of the above test, run in NetBeans IDE, using the built-in support for JUnit.

JUnit Test Results

JUnit can also be run from the command line using the Maven goal, surefire:test:

Running JUnit from Command Line

cURL

One of the best-known command line tools for calling for all types of operations centered around calling a URL is cURL. According to their website, ‘curl is a command line tool and library for transferring data with URL syntax, supporting…HTTP, HTTPS…curl supports SSL certificates, HTTP POST, HTTP PUT, FTP uploading, HTTP form based upload, proxies, HTTP/2, cookies, user+password authentication (Basic, Plain, Digest, CRAM-MD5, NTLM, Negotiate, and Kerberos), file transfer resume, proxy tunneling and more.’ I prefer the website’s  briefer description, cURL ‘groks those URLs’.

Using cURL, we could make an HTTP PUT request to the Vehicle microservice’s /vehicles/{oid}.{format} endpoint. With cURL, we have the ability to add the JWT-based Authorization header and the raw request body, containing the modified vehicle object. Below is an example of that cURL command, which can be run from a terminal prompt.

The response body contains the expected modified vehicle object in JSON-format, along with a 201 Created response status.

cURL  HTTP PUT Vehicle

The cURL commands may be incorporated into many types of automated testing processes. These might be as simple as a bash script. The script could a series of automated tests, including the following: register an API client, use the API key to create a JWT, use the JWT to create a new vehicle, use the new vehicle’s ObjectId to modify that same vehicle, delete that vehicle, confirm the vehicle is removed using the count operation and returns a test results report to the user.

cURL Commands from Chrome
Quick tip, instead of hand-coding complex cURL commands, containing form data, URL parameters, and Headers, use Chrome. First, open the Chrome Developer Tools (f12). Next, using the Postman – REST Client for Chrome, available in the Chrome App Store, execute your HTTP request. Finally, in the ‘Network’ tab of the Developers tools, find and right-click on the request and select ‘Copy as cURL’. You have a complete cURL command equivalent of your Postman request, which you can paste directly into the command line or insert into a script. Below is an example of using the Postman – REST Client for Chrome to generate a cURL command.

Using Postman in Chrome to get cURL

The generated command is a bit verbose. Compare this command to the cURL command, earlier.

Wget

Similar to cURL, GNU Wget provides the ability to call the Virtual-Vehicles API’s endpoints. According to their website, ‘GNU Wget is a free software package for retrieving files using HTTP, HTTPS and FTP, the most widely-used Internet protocols. It is a non-interactive command line tool, so it may easily be called from scripts, cron jobs, terminals without X-Windows support, etc.’ Again, like cURL, we can run Wget commands from the command line or incorporate them into scripted testing processes. The Wget website contains excellent documentation.

Using Wget, we could make the same HTTP PUT request to the Vehicle microservice /vehicles/{oid}.{format} endpoint. Like cURL, we have the ability to add the JWT-based Authorization header and the raw request body, containing the modified vehicle object.

The response body contains the expected modified vehicle object in JSON-format, along with a 201 Created response status.

Wget HTTP PUT Vehicle

cURL Bash Testing

We can combine cURL and Wget with several of the tools bash provides, to develop fairly complex integration tests. The bash-based script below just scratches the surface as a complete set of integration tests. However, the tests demonstrate an efficient multi-stage test approach to handling the complex nature of RESTful service request requirements. The tests build upon each other.

After setting up some variables and doing a quick health check on one service, the tests register a new API client by calling the Authentication service. Next, they use the new client’s API key to obtain a JWT. The tests then use the JWT to authenticate themselves and create a new vehicle. Finally, they use the new vehicle’s id and the JWT to verify the existence for the new vehicle.

Although some may consider using bash to test somewhat primitive, the following script demonstrates the effectiveness of bash’s  curl, grep, sed, awk, along with regular expressions, to test our RESTful services. Note how we grep certain values from the response, such as the new client’s API key, and then use that value as a parameter for the following test request, such as to obtain a JWT.

Since these tests are just a bash script, they can from the command line, or easily called from a continuous integration tool, Such as Jenkins CI or Hudson.

Running Integration Tests

Postman

Postman, like several similar tools, is an application designed specifically for test API endpoints. The Postman website describes Postman as tool that allows you to ‘build, test, and document your APIs faster.’  There are two versions of Postman in the Chrome Web Store. They are Postman – REST Client, the in-browser extension, which we mentioned above, and Postman, the standalone application. There is also Postman Interceptor, which helps you send requests that use browser cookies through the Postman application.

Postman and similar applications, have add-ons and extensions to extend their features. In particular, Postman, which is free, offers the Jetpacks paid extension. Jetpacks add the ability to ‘write and run tests inside Postman, extract data from responses, chain requests together and test requests with thousands of variations’. Jetpacks allow you to move beyond basic one-off API request-based testing, to automated regression and performance testing.

Using Postman
Let’s use the same HTTP PUT example we used with cURL and Wget, and see how we would perform the same task with Postman. In the first screen grab below, you can see all elements of the HTTP request, including the RESTful API’s URL, URI including the vehicle’s ObjectId (/vehicles/{ObjectId}.{format}), HTTP method (PUT), Authorization Header with JWT (Bearer), and the raw request body. The raw request body contains a JSON representation of the vehicle we want to update. Note how Postman saves the request in history so we can easily replay it later.

Postman HTTP PUT of Vehicle

In the next screen-grab, we see the response to the HTTP PUT request. Note the response body, response status, timing, and response headers.

Postman HTTP POST of Vehicle Response

Looking at the response body in Postman, you easily see the how RestExpress demonstrates the RESTful principle we discussed in Part Two of the series, HATEOAS (Hypermedia as the Engine of Application State). Note the link to this vehicle’s ‘self’ href) and the entire vehicles collection (‘up’ href).

Postman Collections
A great feature of Postman with Jetpacks is Collections. Collections are sets of requests, which can be saved, recalled, and shared. The Collection Runner runs requests in a collection, in the order in which you set them. Ordered collections are ideal for the Virtual-Vehicles API. The screen grab below shows a collection of requests, arranged in the order we would execute them to test the Virtual-Vehicles API, as it applies to specifically to vehicle CRUD operations:

  1. Execute HTTP POST request to register the new API client, passing the application name and a shared secret in the request
    Receive the new client’s API key in response
  2. Execute HTTP GET to request, passing the new client’s API key and the shared secret in the request
    Receive the new JWT in response
  3. Execute HTTP POST request to create a new vehicle, passing the JWT in the header for authentication (used for all following requests)
    Receive the new vehicle object in response
  4. Execute HTTP PUT request to modify the new vehicle, using the vehicle’s ObjectId
    Receive the modified vehicle object in response
  5. Execute HTTP GET to request the modified vehicle, to confirm it exists in the expected state
    Receive the vehicle object in response
  6. Execute HTTP DELETE request to delete the new vehicle, using the vehicle’s ObjectId
  7. Execute HTTP GET to request the new vehicle and to confirm it has been removed
    Receive a 404 Not Found status response, as expected

Postman Ordered Series of REST Calls

Using saved collections for testing the Virtual-Vehicles API is a real-time saving. However, the collections cannot easily be re-run without hand-editing or some advanced scripting. In the simple example above, we hard-coded a JWT and vehicle ObjectId in the requests. Unfortunately, the JWT has an expiration of only 10 hours by default. More immediately, the ObjectId is unique. The earlier collection test run created, then deleted, the vehicle with that ObjectId.

Negative Testing
You may also perform negative testing with Postman. For example, do you receive the expected response when you don’t include the Authorization Header with JWT in a request (401 Unauthorized status)? When you include a JWT, which has expired (401 Unauthorized status)? When you request a vehicle, whose ObjectId is incorrect or is not found in the database (400 Bad Request status)? Do you receive the expected response when you call an actual service, but an endpoint that doesn’t exist (405 Method Not Allowed)?

Negative Testing in Postman

Postman Test Automation

In addition to manually viewing the HTTP response, to verify the results of a request, Postman allows you to write and run automated tests for each request. According to their website, a ‘Postman test is essentially JavaScript code which sets values for the special tests object. You can set a descriptive key for an element in the object and then say if it’s true or false’. This allows you to write a set of response validation tests for each request.

Below is a quick example of testing the same HTTP POST request, used to create the new API client, above. In this example, we:

  1. Test that the Content-Type response header is present
  2. Test that the HTTP POST successfully returned a 201 status code
  3. Test that the new client’s API key was returned in the response body
  4. Test that the response time was less than 200ms

Postman Test Editor Example

Reviewing Postman’s ‘Tests’ tab, above, observe the four tests have run successfully. Using the Postman’s testing feature, you can create even more advanced tests, eliminating the need to manually validate responses.

This post demonstrates a small subset of the features Postman and other similar applications provide for testing RESTful API. The tools and processes you use to test your RESTful API will depend on the stage of development and testing you are in, as well as the existing technology stacks you build, and on which you host your services.

, , , , , , , , , , , , ,

1 Comment

Building a Microservices-based REST API with RestExpress, Java EE, and MongoDB: Part 2

Develop a well-architected and well-documented REST API, built on a tightly integrated collection of Java EE-based microservices.

Virtual-Vehicles Architecture

Note: All code available on GitHub. For the version of the code that matches the details in this blog post, check out the master branch, v1.0.0 tag (after running git clone …, run a ‘git checkout tags/v1.0.0’ command).

Previous Post

In Part One of this series, we introduced the microservices-based Virtual-Vehicles REST API example. The vehicle-themed Virtual-Vehicles microservices offers a comprehensive set of functionality, through a REST API, to application developers. The developers, in turn, will use the Virtual-Vehicles REST API’s functionality to build applications and games for their end-users.

In Part One, we also decided on the proper amount and responsibility of each microservice. We also determined the functionality of each microservice to meet the hypothetical functional and nonfunctional requirements of Virtual-Vehicles. To review, the four microservices we are building, are as follows:

Virtual-Vehicles REST API Resources

Microservice Purpose (Business Capability) Functions
Authentication
Manage API clients and
JWT authentication
  • Create a new API client (public)
  • Read, filter, sort, count, paginate API clients (admin)
  • Read a single API client (public)
  • Update an existing API client (public)
  • Delete an existing API client (admin)
  • Create new JWT (public)
  • Validate a JWT (internal)
  • Service health ping (admin)
Vehicle
Manage virtual vehicles
  • Create a new vehicle (public)
  • Read, filter, sort, count, paginate vehicles (admin)
  • Read a single vehicle (public)
  • Update an existing vehicle (public)
  • Delete an existing vehicle (admin)
  • Validate a JWT (internal)
  • Service health ping (admin)
Maintenance
Manage maintenance on vehicles
  • Create a new maintenance record (public)
  • Read, filter, sort, count, paginate maintenance records (admin)
  • Read a single maintenance record (public)
  • Update an existing maintenance record (public)
  • Delete an existing maintenance record (admin)
  • Validate a JWT (internal)
  • Service health ping (admin)
Valet Parking
Manage a valet service to park for vehicles
  • Create a new valet parking transaction (public)
  • Read, filter, sort, count, paginate valet parking transactions (admin)
  • Read a single valet parking transaction (public)
  • Update an existing valet parking transaction (public)
  • Delete an existing valet parking transaction (admin)
  • Validate a JWT (internal)
  • Service health ping (admin)

To review, the first five functions for each service are all basic CRUD operations: create (POST), read (GET), readAll (GET), update (PUT), delete (DELETE). The readAll function also has find, count, and pagination functionality using query parameters. Unfortunately, RestExpress does not support PATCH for updates. However, I have updated RestExpress’ PUT HTTP methods to return the modified object in the response body instead of the nothing (status of 201 Created vs. 200 OK). See StackOverflow for an explanation.

All services also have an internal authenticateJwt function, to authenticate the JWT, passed in the HTTP request header, before performing any operation. Additionally, all services have a basic health-check function, ping (GET). There are only a few other functions required for our Virtual-Vehicles example, such as for creating JWTs.

 Part Two Introduction

In Part Two, we will build our four Virtual-Vehicles microservices. Recall from our first post, we will be using RestExpress. RestExpress composes best-of-breed open-source tools to enable quickly creating RESTful microservices that embrace industry best practices. Those best-of-breed tools include Java EE, Maven, MongoDB, and Netty, among others.

In this post, we will accomplish the following:

  • Create a default microservice project in NetBeans using RestExpress MongoDB Maven Archetype
  • Understand the basic structure of a default RestExpress microservice project
  • Review the changes made to the default RestExpress microservice project to create the Virtual-Vehicles example
  • Compile and run the individual microservices directly from NetBeans

I used NetBeans IDE 8.0.2 on Linux Ubuntu 14.10 to build the microservices. You may also follow along in other IDE’s, such as Eclipse or IntelliJ, on Mac or Windows. We won’t cover installing MongoDB, Maven, and Java. I’ll assume if your building enterprise applications, you have the basics covered.

Using the RestExpress MongoDB Maven Archetype

All the code for this project is available on GitHub. However, to understand RestExpress, you should go through the exercise of scaffolding a new microservice using the RestExpress MongoDB Maven Archetype. You will also be able to use this default microservice project to compare and contrast to the modified versions, used in the Virtual-Vehicles example. The screen grabs below demonstrate how to create a new microservice project using the RestExpress MongoDB Maven Archetype. At the time of this post, the archetype version was restexpress-mongodb version 1.15.

New Maven Project from Archetype Step 1

New Maven Project from Archetype Step 2

New Maven Project from Archetype Step 2

Default Project Architecture

Reviewing the two screen grabs below (Project tab), note the key components of the RestExpress MongoDB Maven project, which we just created:

  • Base Package (com.example.vehicle)
    • Configuration class reads in environment properties (see Files tab) and instantiates controllers
    • Constants class contains project constants
    • Relationships class defines linking resource which aids service discoverability (HATEOAS)
    • Main executable class
    • Routes class defines the routes (endpoints) exposed by the service and the corresponding controller class
  • Model/Controllers Packages (com.example.vehicle.objectid and .uuid)
    • Entity class defines the data entity – a Vehicle in this case
    • Controller class contains the methods executed when the route (endpoint) is called
    • Repository class defines the connection details to MongoDB
    • Service class contains the calls to the persistence layer, validation, and business logic
  • Serialization Package (com.example.vehicle.serialization)
    • XML and JSON serialization classes
    • Object ID and UUID serialization and deserialization classes

New Maven Project from Archetype Project View

New Maven Project from Archetype Projects View

Again, I strongly recommend reviewing each of these package’s classes. To understand the core functionality of RestExpress, you must understand the relationships between RestExpress microservice’s Route, Controller, Service, Repository, Relationships, and Entity classes. In addition to reviewing the default Maven project, there are limited materials available on the Internet. I would recommend the RestExpress Website on GitHub, RestExpress Google Group Forum, and the YouTube 3-part video series, Instant REST Services with RESTExpress.

Unit Tests?
Disappointingly, the current RestExpress MongoDB Maven Archetype sample project does not come with sample JUnit unit tests. I am tempted to start writing my own unit tests if I decided to continue to use the RestExpress microservices framework for future projects.

Properties Files

Also included in the default RestExpress MongoDB Maven project is a Java properties file (environment.properties). This properties file is displayed in the Files tab, as shown below. The default properties file is located in the ‘dev’ environment config folder. Later, we will create an additional properties file for our production environment.

Default Properties File

Ports
Within the ‘dev’ environment, each microservice is configured to start on separate ports  (i.e. port = 8581). Feel free to change the service’s port mappings if they conflict with previously configured components running on your system. Be careful when changing the Authentication service’s port, 8587, since this port is also mapped in all other microservices using the authentication.port property (authentication.port = 8587). Make sure you change both properties, if you change Authentication service’s port mapping.

Base URL
Also, in the properties files is the base.url property. This property defines the URL the microservice’s endpoints will be expecting calls from, and making internal calls between services. In our post’s example, this property in the ‘dev’ environment is set to localhost (base.url = http://localhost). You could map an alternate hostname from your hosts file (/etc/hosts). We will do this in a later post, in our ‘prod’ environment, mapping the base.url property to Virtual-Vehicles (base.url = http://virtual-vehicles.com). In the ‘dev’ environment properties file, MongoDB is also mapped to localhost (i.e. mongodb.uri = mongodb://virtual-vehicles.com:27017/virtual_vehicle).

Metrics Plugin and Graphite
RestExpress also uses the properties file to hold configuration properties for Metrics Plugin and Graphite. The Metrics Plugin and Graphite are both first class citizens of RestExpress. Below is the copy of the Vehicles service environment.properties file for the ‘dev’ environment. Note, the Metrics Plugin and Graphite are both disabled in the ‘dev’ environment.

Choosing a Data Identification Method

RestExpress offers two identification models for managing data, the MongoDB ObjectId and a Universally Unique Identifier (UUID). MongoDB uses an ObjectId to identify uniquely a document within a collection. The ObjectId is a special 12-byte BSON type that guarantees uniqueness of the document within the collection. Alternately, you can use the UUID identification model. The UUID identification model in RestExpress uses a UUID, instead of a MongoDB ObjectId. The UUID also contains createdAt and updatedAt properties that are automatically maintained by the persistence layer. You may stick with ObjectId, as we will in the Virtual-Vehicles example, or choose the UUID. If you use multiple database engines for your projects, using UUID will give you a universal identification method.

Project Modifications

Many small code changes differentiate our Virtual-Vehicles microservices from the default RestExpress Maven Archetype project. Most changes are superficial; nothing changed about how RestExpress functions. Changes between the screen grabs above, showing the default project, and the screen grabs below, showing the final Virtual-Vehicles microservices, include:

  • Remove all packages, classes, and code references to the UUID identification methods (example uses ObjectId)
  • Rename several classes for convenience (dropped use of word ‘Entity’)
  • Add the Utilities (com.example.utilities) and Authentication (com.example.authenticate) packages

Final Virtual-Vehicles Microservices Projects

Final Virtual-Vehicles Microservices Projects

MongoDB

Following a key principle of microservices mentioned in the first post, Decentralized Data Management, each microservice will have its own instance of a MongoDB database associated with it. The below diagram shows each service and its corresponding database, collection, and fields.

Virtual-Vehicles Database Diagram
From the MongoDB shell, we can observe the individual instances of the four microservice’s databases.

MongoDB Virtual-Vehicles Databases

In the case of the Vehicle microservice, the associated MongoDB database is virtual_vehicle. This database contains a single collection, vehicles. While the properties file defines the database name, the Vehicles entity class defines the collection name, using the org.mongodb.morphia.annotations classes annotation functionality.

Looking at the virtual_vehicle database in the MongoDB shell, we see that the sample document’s fields correspond to the Vehicle entity classes properties.

MongoDB vehicles Document
Each of the microservice’s MongoDB databases are configured in the environments.properties file, using the mongodb.uri property. In the ‘dev’ environment we use localhost as our host URL (i.e. mongodb.uri = mongodb://localhost:27017/virtual_vehicle).

Authentication and JSON Web Tokens

The three microservices, Vehicle, Valet, and Maintenance, are almost identical. However, the Authentication microservice is unique. This service is called by each of the other three services, as well as also being called directly. The Authentication service provides a very basic level of authentication using JSON Web Tokens (JWT), pronounced ‘jot‘.

Why do we want authentication? We want to confirm that the requester using the Virtual-Vehicles REST API is the actual registered API client they are who they claim to be. JWT allows us to achieve this requirement with minimal effort.

According to jwt.io, ‘a JSON Web Token is a compact URL-safe means of representing claims to be transferred between two parties. The claims in a JWT are encoded as a JSON object that is digitally signed using JSON Web Signature (JWS).‘ I recommend reviewing the JWT draft standard to fully understand the structure, creation, and use of JWTs.

Virtual-Vehicles Authentication Process
There are different approaches to implementing JWT. In our Virtual-Vehicles REST API example, we use the following process for JWT authentication:

  1. Register the new API client by supplying the application name and a shared secret (one time only)
  2. Receive an API key in response (one time only)
  3. Obtain a JWT using the API key and the shared secret (each user session or renew when the previous JWT expires)
  4. Include the JWT in each API call

In our example, we are passing four JSON fields in our set of claims. Those fields are the issuer (‘iss’), API key, expiration (‘exp’), and the time the JWT was issued (‘ait’). Both the ‘iss’ and the ‘exp’ claims are defined in the Authentication service’s environment.properties file (jwt.expire.length and jwt.issuer).

Expiration and Issued date/time use the JWT standard recommended “Seconds Since the Epoch“.  The default expiration for a Virtual-Vehicles JWT is set to an arbitrary 10 hours from the time the JWT was issued (jwt.expire.length = 36000). That amount, 36,000, is equivalent to 10 hours x 60 minutes/hour x 60 seconds/minute.

Decoding a JWT
Using the jwt.io site’s JT Debugger tool, I have decoded a sample JWT issued by the Virtual-Vehicles REST API, generated by the Authentication service. Observe the three parts of the JWT, the JOSE Header, Claims Set, and the JSON Web Signature (JWS).

Decoded JWT

The JWT’s header indicates that our JWT is a JWS that is MACed using the HMAC SHA-256 algorithm. The shared secret, passed by the API client, represents the HMAC secret cryptographic key. The secret is used in combination with the cryptographic hash function to calculate the message authentication code (MAC). In the example below, note how the API client’s shared secret is used to validate our JWT’s JWS.

Sequence Diagrams of Authentication Process
Below are three sequence diagrams, which detail the following processes: API client registration process, obtaining a new JWT, and a REST call being authenticated using the JWT. The end-user of the API self-registers their application using the Authentication service and receives back an API key. The API key is unique to that client.

Register API Client (Authentication Service)

The end-user application then uses the API key and the shared secret to receive a JWT from the Authentication service.

Requesting a JWT (Authentication Service)After receiving the JWT, the end-user application passes the JWT in the header of each API request. The JWT is validated by calling the Authentication service. If the JWT is valid, the request is fulfilled. If not valid, a ‘401 Unauthorized’ status is returned.

API Call with JWT Passed in Header (Vehicle Service)JWT Validation
The JWT draft standard recommends how to validate a JWT. Our Virtual-Vehicles Authentication microservice uses many of those criteria to validate the JWT, which include:

  1. API Key – Retrieve API client’s shared secret from MongoDB, using API key contained in JWT’s claims set (secret is returned; therefore API key is valid)
  2. Algorithm – confirm the algorithm (‘alg’), found in the JWT Header, which used to encode JWT, was ‘HS256’ (HMAC SHA-256)
  3. Valid JWS – Use the client’s shared secret from #1 above, decode HMAC SHA-256 encrypted JWS
  4. Expiration – confirm JWT is not expired (‘exp’)

Inter-Service Communications

By default, the RestExpress Archetype project does not offer an example of communications between microservices. Service-to-service communications for microservices is most often done using the same HTTP-based REST calls used to by our Virtual-Vehicles REST API. Alternately, a message broker, such as RabbitMQ, Kafka, ActiveMQ, or Kestrel, is often used. Both methods of communicating between microservices, referred to as ‘inter-service communication mechanisms’ by InfoQ, have their pros and cons. The InfoQ website has an excellent microservices post, which discusses the topic of service-to-service communication.

For the Virtual-Vehicles microservices example, we will use HTTP-based REST calls for inter-service communications. The primary service-to-service communications in our example, is between the three microservices, Vehicle, Valet, and Maintenance, and the Authentication microservice. The three services validate the JWT passed in each request to a CRUD operation, by calling the Authentication service and passing the JWT, as shown in the sequence diagram, above. Validation is done using an HTTP GET request to the Authentication service’s .../jwts/{jwt} endpoint. The Authentication service’s method, called by this endpoint, minus some logging and error handling, looks like the following:

Primarily, we are using the java.net and java.io packages, along with the org.restexpress.Request class to build and send our HTTP request to the Authentication service. Alternately, you could use just the org.restexpress package to construct request and handle the response. This same basic method structure shown above can be used to create unit tests for your service’s endpoints.

Health Ping

Each of the Virtual-Vehicles microservices contain a DiagnosticController in the .utilities package. In our example, we have created a ping() method. This simple method, called through the .../utils/ping endpoint, should return a 200 OK status and a boolean value of ‘true’, indicating the microservice is running and reachable. This route’s associated method could not be simpler:

The ping health check can even be accessed with a simple curl command, curl localhost:8581/vehicles/utils/ping.

In a real-world application, we would add additional health checks to all services, providing additional insight into the health and performance of each microservice, as well as the service’s dependencies.

API Documentation

A well written RESTful API will not require extensive documentation to understand the API’s operations. Endpoints will be discoverable through linking (see Response Body links section in below example). API documentation should provide HTTP method, required headers and URL parameters, expected response body and response status, and so forth.

An API should be documented before any code is written, similar to TDD. The API documentation is the result of a clear understanding of requirements. The API documentation should make the coding even easier since the documentation serves as a further refinement of the requirements. The requirements are an architectural plan for the microservice’s code structure.

Sample Documentation
Below, is a sample of the Virtual-Vehicles REST API documentation. It details the function responsible for creating a new API client. The documentation provides a brief description of the function, the operation’s endpoint (URI), HTTP method, request parameters, expected response body, expected response status, and even a view of the MongoDB collection’s document for a new API client.

Virtual-Vehicles API Guide

You can download a PDF version of the Virtual-Vehicles RESTful API documentation on GitHub or review the source document on Google Docs. It contains general notes about the API, and details about every one of the API’s operations.

Running the Individual Microservices

For development and testing purposes, the easiest way to start the microservices is in NetBeans using the Run command. The Run command executes the Maven exec goal. Based on the DEFAULT_ENVIRONMENT constant in the org.restexpress.util Environment class, RestExpress will use the ‘dev’ environment’s environment.properties file, in the project’s /config/dev directory.

Alternately, you can use the RestExpress project’s recommended command from a terminal prompt to start each microservice from its root directory (mvn exec:java -Dexec.mainClass=test.Main -Dexec.args="dev"). You can also use this command to switch from the ‘dev’ to ‘prod’ environment properties (-Dexec.args="prod").

You may use a variety of commands to confirm all the microservices are running. I prefer something basic like sudo netstat -tulpn | grep 858[0-9]. This will find all the ports within the ‘dev’ port range. For more in-depth info, you can use a command like ps -aux | grep com.example | grep -v grep

Microservices Running

Part Three: Testing our Services

We now have a copy of the Virtual-Vehicles project pulled from GitHub, a basic understanding of how RestExpress works, and our four microservices running on different ports. In Part Three of this series, we will take them for a drive (get it?). There are many ways to test our service’s endpoints. One of my favorite tools is Postman. we will explore how to use several tools, including Postman, and our API documentation, to test our microservice’s endpoints.

Postman Preview

, , , , , , , , , , , , ,

5 Comments

Building a Microservices-based REST API with RestExpress, Java EE, and MongoDB: Part 1

Develop a well-architected and well-documented REST API, built on a tightly integrated collection of Java EE-based microservices.

Generic API Architecture

Microservices

Microservices are a popular and growing trend in software development. According to Wikipedia, microservices are “a software architecture style, in which complex applications are composed of small, independent processes communicating with each other using language-agnostic APIs. These services are small, highly decoupled and focus on doing a small task.

Martin Fowler and James Lewis (ThoughtWorks) have done an exemplary job capturing the essence of microservice architecture in their March 2014 post, microservices. Fowler has also discussed these principles in several presentations, including the January 2015 goto; Conference, Keynote: Microservices by Martin Fowler.

Additionally, noted technical consultant and speaker, Adrian Cockcroft (Battery Ventures), has made significant contributions to the definition of microservices, such as in his December 2014 dockercon14 | eu presentation, State of the Art in Microservices.

Lastly, Zhamak Dehghani (ThoughtWorks), delivered an in-depth discussion of microservices, including customer perspectives, in her October 2014 presentation, Real-World Microservices: Lessons from the Frontline.

Some of the major characteristics of microservices and REST cited by these experts, include:

  • Organized Around Business Capabilities
  • Single Responsibility
  • Loose Coupling / High Cohesion
  • Smart Endpoints and Dumb Pipes
  • Decentralized Data Management
  • Hypermedia as the Engine of Application State (HATEOAS)

As we develop this post’s example, I will demonstrate how all of the above characteristics are implemented.

REST API

A REST API is the mash-up of two common software concepts, Representational State Transfer (REST) and an application programming interface (API). Although even Wikipedia doesn’t have an exact definition of a REST API, they come close to their discussion of REST. According to Wikipedia“Web service APIs that adhere to the REST architectural constraints are called RESTful APIs. HTTP-based RESTful APIs are defined with these aspects: base URI, an Internet media type for the data, standard HTTP methods, hypertext links to a reference state, and hypertext links to reference related resources.

An important nuance and differentiator from SOA-based APIs, RESTful APIs do not require XML-based Web service protocols (SOAP and WSDL) to support their interfaces (Wikipedia).

The author of the WebConcepts channel does an excellent job capturing the essence of REST APIs in REST API concepts and examples. Two additional presentations I strongly recommend are REST+JSON API Design – Best Practices for Developers and Designing a Beautiful REST+JSON API, both by Les Hazlewood, CTO of Stormpath. Stormpath is a leader in the commercial REST API space.

Microservices-Based REST API

A microservices-based REST API is a REST API, whose HTTP requests call an orchestrated collection or collections of language-agnostic and platform-agnostic microservices. The combination of these two trends, microservices, and a REST API, offers a simple, reliable, and scalable solution for providing flexible functionality to an end-user, in a technology-agnostic manner.

REST API Example

There is a fast-growing volume of reference materials describing the characteristics, benefits, and general architecture of microservices and REST APIs. However, in researching these topics, I have found a shortage of practical examples or tutorials on building microservices-based REST API solutions.

Undoubtedly, the complexity of even the simplest microservices-based solution limits the number of available cases. A minimally viable solution require planning, coding, testing, and documentation. The addition of cross-cutting features such as security, logging, monitoring, and orchestration, creates an enormous task to build a practical microservices-based example.

In the following series of posts, we will use many of the characteristics of a modern microservice architecture as described by Fowler, Lewis, Cockcroft, and Dehghani. We will combine these microservice characteristics with the best practices of good REST API design, as described by Hazlewood and WebConcepts, to build a minimally viable microservices-based REST API.

In a future post, we will create an application, which leverages the microservices-based solution, through the REST API. Additionally, we will demonstrate how to ensure high-availability of the individual microservices and data sources.

Vehicle for Learning

In a similar vein to the publicly available Twitter, Facebook, and Google REST APIs, we will build the Virtual-Vehicles REST API. The Virtual-Vehicles REST API will constitute a collection of vehicle-themed microservices. Collectively, the microservices will offer a comprehensive set of functionality to the end-user, an application developer. They, in turn, will use the functionality of the Virtual-Vehicles REST API to build applications and games for their end-users.

Technology Choices

There are a seemingly infinite number of technology choices for building microservices and REST APIs. Your choice of development languages, databases, application servers, third-party libraries, API gateway, logging, monitoring, automated testing, ORM or ODM, and even the IDE, all define your technology stack.

For the Virtual-Vehicles solution, we will use the following key technologies:

What is RestExpress?

According to their website, RestExpress, composes best-of-breed open-source tools to enable quickly creating RESTful microservices that embrace industry best practices. Built from the ground-up for container-less, microservice architectures, RestExpress is the easiest way to create RESTful APIs in Java. RestExpress is an extremely lightweight, fast, REST engine and API for Java. RestExpress is a thin wrapper on Netty IO HTTP handling. RestExpress lets you create performant, stand-alone REST APIs rapidly. RestExpress provides several Maven archetypes, which we will use as a basis for our microservices.

RestExpress will also drive our technology decisions to use Java EE, Maven, MongoDB, and Netty.

Virtual-Vehicles Microservices

Adhering to the first few microservice architectural principles listed above, organized around business capabilities, single responsibility, and high cohesion, we first must determine the proper number of microservices, and their individual responsibilities. In the case of our solution, we will break down Virtual-Vehicles’ business capabilities into the following microservices:

Virtual-Vehicles Services

Microservice Purpose (Business Capability)
Authentication Service Manage API clients and JWT authentication
Vehicle Service Manage virtual vehicles
Maintenance Service Manage maintenance on vehicles
Valet Parking Service Manage a valet service to park for vehicles
Sales Service Manage the buying and selling of vehicles
Registration Service Manage registration of vehicles to owners
Auction Service Manage a virtual car auction
Car Show Service Manage a virtual car show
Interaction Service Manage interaction of users with vehicles

For simplicity in this post’s example, we will only be exploring the (4) services shown above in bold.

This segmentation of service functionality is unlike what we might encounter in traditional monolithic, n-tier applications, and SOA-based architecture. Traditional applications were built around application-centric functionality or business’ organizational structure. Microservices, however, are client-centric and built around business capabilities.

REST API Functionality

The next decision we need to make is required functionality. What are the operational requirements of each business segment, represented by the microservices? Additionally, what are the nonfunctional requirements, such as monitoring, logging, and authentication. Requirements are translated into functionality, which is translated into the available resources exposed via the service’s RESTful endpoints.

For the Virtual-Vehicles microservices solution, based on a hypothetical set of business and non-functional requirements, we will expose the following resources. Collectively, they will compose the REST API:

Virtual-Vehicles REST API Resources

Microservice Purpose (Business Capability) Functions
Authentication Service Manage API clients and
JWT authentication
  • Create a new API client (public)
  • Read, filter, sort, count, paginate API clients (admin)
  • Read a single API client (public)
  • Update an existing API client (public)
  • Delete an existing API client (admin)
  • Create new JWT (public)
  • Validate a JWT (internal)
  • Service health ping (admin)
Vehicle Service Manage virtual vehicles
  • Create a new vehicle (public)
  • Read, filter, sort, count, paginate vehicles (admin)
  • Read a single vehicle (public)
  • Update an existing vehicle (public)
  • Delete an existing vehicle (admin)
  • Validate a JWT (internal)
  • Service health ping (admin)
Maintenance Service Manage maintenance on vehicles
  • Create a new maintenance record (public)
  • Read, filter, sort, count, paginate maintenance records (admin)
  • Read a single maintenance record (public)
  • Update an existing maintenance record (public)
  • Delete an existing maintenance record (admin)
  • Validate a JWT (internal)
  • Service health ping (admin)
Valet Parking Service Manage a valet service to park for vehicles
  • Create a new valet parking transaction (public)
  • Read, filter, sort, count, paginate valet parking transactions (admin)
  • Read a single valet parking transaction (public)
  • Update an existing valet parking transaction (public)
  • Delete an existing valet parking transaction (admin)
  • Validate a JWT (internal)
  • Service health ping (admin)

Reviewing the table above, note the first five functions for each service are all basic CRUD operations: create (POST), read (GET), readAll (GET), update (PUT), delete (DELETE). The readAll function also has find, count, and pagination functionality using query parameters.

All services also have an internal authenticateJwt function, to authenticate the JWT, passed in the HTTP request header, before performing any operation. Additionally, all services have a basic health-check function, ping (GET). There are only a few other functions required for our Virtual-Vehicles example, such as for creating JWTs.

I’ve labeled each function as to suggested user scope. Scopes include public, admin, and internal. As a consumer of the REST API, you may only want to expose certain functionality to your general end-user (public). Additional functionality may be reserved for an administrative user (admin) or only yourself as a developer (internal). Creating a new vehicle might be a common end-user feature. However, the ability to permanently delete one or more vehicles may be reserved for an admin-level user, or not exposed at all.

REST API Patterns

We will not spend a lot of time discussing patterns for building REST APIs. There are many useful materials available on the Internet regarding industry-standard patterns for REST API resource URI construction. The two presentations I recommend above by Les Hazlewood, CTO of Stormpath, are excellent. Also, Microservices.ioRestApiTutorial.com, swagger.io, and raml.org websites offer solid overviews of REST patterns and RESTful standards.

A common RESTful anti-pattern, which is hard to avoid as a OOP developer, is the temptation to use verbs versus nouns and method-like names, in resource URIs. Remember, we are not designing an end-user application. We are building an API, used by API consumers (application developers), to build a variety of platform and language-agnostic applications. Functions like paintCar, changeOil, or parkVehicle are not something the API should define. The Vehicle microservice exposes the update operation, which allows an application developer to change the car’s paint color in their paintCar method. Similarly, the valet service exposes the create operation, which allows the application developer to create a function to park the vehicle (or car, or truck, in a garage, or parking lot, etc.). A good REST API allows for maximum end-user flexibility.

Part Two

In Part Two, we will install a copy of the Virtual-Vehicles project from GitHub. In Part Two, we will gain a basic understanding of how RestExpress works. Finally, we will discover how to get the Virtual-Vehicles microservices up and running.

Virtual-Vehicles Architecture

, , , , , , , , , , , , ,

1 Comment

Retrieving and Displaying Data with AngularJS and the MEAN Stack: Part II

Explore various methods of retrieving and displaying data using AngularJS and the MEAN Stack.

Mobile View on Android Smartphone

Mobile View on Android Smartphone

Introduction

In this two-part post, we are exploring methods of retrieving and displaying data using AngularJS and the MEAN Stack. The post’s corresponding GitHub project, ‘meanstack-data-samples‘, is based on William Lepinski’s ‘generator-meanstack‘, which is in turn is based on Yeoman’s ‘generator-angular‘. As a bonus, since both projects are based on ‘generator-angular’, all the code generators work. Generators can save a lot of time and aggravation when building AngularJS components.

In part one of this post, we installed and configured the ‘meanstack-data-samples’ project from GitHub. In part two, we will we will look at five examples of retrieving and displaying data using AngularJS:

  • Function within AngularJS Controller returns array of strings.
  • AngularJS Service returns an array of simple object literals to the controller.
  • AngularJS Factory returns the contents of JSON file to the controller.
  • AngularJS Factory returns the contents of JSON file to the controller using a resource object
    (In GitHub project, but not discussed in this post).
  • AngularJS Factory returns a collection of documents from MongoDB Database to the controller.
  • AngularJS Factory returns results from Google’s RESTful Web Search API to the controller.

Project Structure

For brevity, I have tried to limit the number of files in the project. There are two main views, both driven by a single controller. The primary files, specific to data retrieval and display, are as follows:

  • Default site file (./)
    • index.html – loads all CSS and JavaScript files, and views
  • App and Routes (./app/scripts/)
    • app.js – instantiates app and defines routes (route/view/controller relationship)
  • Views (./app/views/)
    • data-bootstrap.html – uses Twitter Bootstrap
    • data-no-bootstrap.html – basically the same page, without Twitter Bootstrap
  • Controllers (./app/scripts/controllers/)
    • DataController.js (DataController) – single controller used by both views
  • Services and Factories (./app/scripts/services/)
    • meanService.js (meanService) – service returns array of object literals to DataController
    • jsonFactory.js (jsonFactory) – factory returns contents of JSON file
    • jsonFactoryResource.js (jsonFactoryResource) – factory returns contents of JSON file using resource object (new)
    • mongoFactory.js (mongoFactory) – factory returns MongoDB collection of documents
    • googleFactory.js (googleFactory) – factory call Google Web Search API
  • Models (./app/models/)
    • Components.js – mongoose constructor for the Component schema definition
  • Routes (./app/)
    • routes.js – mongoose RESTful routes
  • Data (./app/data/)
    • otherStuff.json – static JSON file loaded by jsonFactory
  • Environment Configuration (./config/environments/)
    • index.js – defines all environment configurations
    • test.js – Configuration specific to the current ‘test’ environment
  • Unit Tests (./test/spec/…)
    • Various files – all controller and services/factories unit test files are in here…
Project in JetBrains WebStorm 8.0

Project in JetBrains WebStorm 8.0

There are many more files, critical to the project’s functionality, include app.js, Gruntfile.js, bower.json, package.json, server.js, karma.conf.js, and so forth. You should understand each of these file’s purposes.

Function Returns Array

In the first example, we have the yeomanStuff() method, a member of the $scope object, within the DataController.  The yeomanStuff() method return an array object containing three strings. In JavaScript, a method is a function associated with an object.

$scope.yeomanStuff = function () {
  return [
    'yo',
    'Grunt',
    'Bower'
  ];
};
'yeomanStuff' Method of the '$scope' Object

‘yeomanStuff’ Method of the ‘$scope’ Object

The yeomanStuff() method is called from within the view by Angular’s ng-repeat directive. The directive, ng-repeat, allows us to loop through the array of strings and add them to an unordered list. We will use ng-repeat for all the examples in this post.

<ul class="list-group">
  <li class="list-group-item"
	  ng-repeat="stuff in yeomanStuff()">
	{{stuff}}
  </li>
<ul>

Method1

Although this first example is easy to implement, it is somewhat impractical. Generally, you would not embed static data into your code. This limits your ability to change the data, independent of a application’s code. In addition, the function is tightly coupled to the controller, limiting its reuse.

Service Returns Array

In the second example, we also use data embedded in our code. However, this time we have improved the architecture slightly by moving the data to an Angular Service. The meanService contains the getMeanStuff() function, which returns an array containing four object literals. Using a service, we can call the getMeanStuff() function from anywhere in our project.

angular.module('generatorMeanstackApp')
  .service('meanService', function () {
    this.getMeanStuff = function () {
      return ([
        {
          component: 'MongoDB',
          url: 'http://www.mongodb.org'
        },
        {
          component: 'Express',
          url: 'http://expressjs.com'
        },
        {
          component: 'AngularJS',
          url: 'http://angularjs.org'
        },
        {
          component: 'Node.js',
          url: 'http://nodejs.org'
        }
      ])
    };
  });

Within the DataController, we assign the array object, returned from the meanService.getMeanStuff() function, to the meanStuff object property of the  $scope object.

$scope.meanStuff = {};
try {
  $scope.meanStuff = meanService.getMeanStuff();
} catch (error) {
  console.error(error);
}
'meanStuff' Property of the '$scope' Object

‘meanStuff’ Property of the ‘$scope’ Object

The meanStuff object property is accessed from within the view, using ng-repeat. Each object in the array contains two properties, component and url. We display the property values on the page using Angular’s double curly brace expression notation (i.e. ‘{{stuff.component}}‘).

<ul class="nav nav-pills nav-stacked">
  <li ng-repeat="stuff in meanStuff">
    url}}"
       target="_blank">{{stuff.component}}
  </li>
<ul>

Method2

Promises, Promises…

The remaining methods implement an asynchronous (non-blocking) programming model, using the $http and $q services of Angular’s ng module. The services implements the asynchronous Promise and Deferred APIs. According to Chris Webb, in his excellent two-part post, Promise & Deferred objects in JavaScript: Theory and Semantics, a promise represents a value that is not yet known and a deferred represents work that is not yet finished. I strongly recommend reading Chris’ post, before continuing. I also highly recommend watching RED Ape EDU’s YouTube video, Deferred and Promise objects in Angular js. This video really clarified the promise and deferred concepts for me.

Factory Loads JSON File

In the third example, we will read data from a JSON file (‘./app/data/otherStuff.json‘) using an AngularJS Factory. The differences between a service and a factory can be confusing, and are beyond the scope of this post. Here is two great links on the differences, one on Angular’s site and one on StackOverflow.

{
  "components": [
    {
      "component": "jQuery",
      "url": "http://jquery.com"
    },
    {
      "component": "Jade",
      "url": "http://jade-lang.com"
    },
    {
      "component": "JSHint",
      "url": "http://www.jshint.com"
    },
    {
      "component": "Karma",
      "url": "http://karma-runner.github.io"
    },
    ...
  ]
}

The jsonFactory contains the getOtherStuff() function. This function uses $http.get() to read the JSON file and returns a promise of the response object. According to Angular’s site, “since the returned value of calling the $http function is a promise, you can also use the then method to register callbacks, and these callbacks will receive a single argument – an object representing the response. A response status code between 200 and 299 is considered a success status and will result in the success callback being called. ” As I mentioned, a complete explanation of the deferreds and promises, is too complex for this short post.

angular.module('generatorMeanstackApp')
  .factory('jsonFactory', function ($q, $http) {
    return {
      getOtherStuff: function () {
        var deferred = $q.defer(),
          httpPromise = $http.get('data/otherStuff.json');

        httpPromise.then(function (response) {
          deferred.resolve(response);
        }, function (error) {
          console.error(error);
        });

        return deferred.promise;
      }
    };
  });

The response object contains the data property. Angular defines the response object’s data property as a string or object, containing the response body transformed with the transform functions. One of the properties of the data property is the components array containing the seven objects. Within the DataController, if the promise is resolved successfully, the callback function assigns the contents of the components array to the otherStuff property of the $scope object.

$scope.otherStuff = {};
jsonFactory.getOtherStuff()
  .then(function (response) {
    $scope.otherStuff = response.data.components;
  }, function (error) {
    console.error(error);
  });
'otherStuff' Property of the '$scope' Object

‘otherStuff’ Property of the ‘$scope’ Object

The otherStuff property is accessed from the view, using ng-repeat, which displays individual values, exactly like the previous methods.

<ul class="nav nav-pills nav-stacked">
  <li ng-repeat="stuff in otherStuff">
    <a href="{{stuff.url}}"
       target="_blank">{{stuff.component}}</a>
  </li>
</ul>

Method3

This method of reading a JSON file is often used for configuration files. Static configuration data is stored in a JSON file, external to the actual code. This way, the configuration can be modified without requiring the main code to be recompiled and deployed. It is a technique used by the components within this very project. Take for example the bower.json files and the package.json files. Both contain configuration data, stored as JSON, used by Bower and npm to perform package management.

Factory Retrieves Data from MongoDB

In the fourth example, we will read data from a MongoDB database. There are a few more moving parts in this example than in the previous examples. Below are the documents in the components collection of the meanstack-test MongoDB database, which we will retrieve and display with this method.  The meanstack-test database is defined in the test.js environments file (discussed in part one).

'meanstack-test' Database's 'components' Collection Documents

‘meanstack-test’ Database’s ‘components’ Collection Documents

To connect to the MongoDB, we will use Mongoose. According to their website, “Mongoose provides a straight-forward, schema-based solution to modeling your application data and includes built-in type casting, validation, query building, business logic hooks and more, out of the box.” But wait, MongoDB is schemaless? It is. However, Mongoose provides a schema-based API for us to work within. Again, according to Mongoose’s website, “Everything in Mongoose starts with a Schema. Each schema maps to a MongoDB collection and defines the shape of the documents within that collection.

In our example, we create the componentSchema schema, and pass it to the Component model (the ‘M’ in MVC). The componentSchema maps to the database’s components collection.

var mongoose = require('mongoose');
var Schema = mongoose.Schema;

var componentSchema = new Schema({
  component: String,
  url: String
});

module.exports = mongoose.model('Component', componentSchema);

The routes.js file associates routes (Request URIs) and HTTP methods to Mongoose actions. These actions are usually CRUD operations. In our simple example, we have a single route, ‘/api/components‘, associated with an HTTP GET method. When an HTTP GET request is made to the ‘/api/components‘ request URI, Mongoose calls the Model.find() function, ‘Component.find()‘, with a callback function parameter. The Component.find() function returns all documents in the components collection.

var Component = require('./models/component');

module.exports = function (app) {
  app.get('/api/components', function (req, res) {
    Component.find(function (err, components) {
      if (err)
        res.send(err);

      res.json(components);
    });
  });
};

You can test these routes, directly. Below, is the results of calling the ‘/api/components‘ route in Chrome.

Response from MongoDB Using Mongoose

Response from MongoDB Using Mongoose

The mongoFactory contains the getMongoStuff() function. This function uses $http.get() to call  the ‘/api/components‘ route. The route is resolved by the routes.js file, which in turn executes the Component.find() command. The promise of an array of objects is returned by the getMongoStuff() function. Each object represents a document in the components collection.

angular.module('generatorMeanstackApp')
  .factory('mongoFactory', function ($q, $http) {
    return {
      getMongoStuff: function () {
        var deferred = $q.defer(),
          httpPromise = $http.get('/api/components');

        httpPromise.success(function (components) {
          deferred.resolve(components);
        })
          .error(function (error) {
            console.error('Error: ' + error);
          });

        return deferred.promise;
      }
    };
  });

Within the DataController, if the promise is resolved successfully, the callback function assigns the array of objects, representing the documents in the collection, to the mongoStuff property of the $scope object.

$scope.mongoStuff = {};
mongoFactory.getMongoStuff()
  .then(function (components) {
    $scope.mongoStuff = components;
  }, function (error) {
    console.error(error);
  });
'mongoStuff' Property of the '$scope' Object

‘mongoStuff’ Property of the ‘$scope’ Object

The mongoStuff property is accessed from the view, using ng-repeat, which displays individual values using Angular expressions, exactly like the previous methods.

<ul class="list-group">
  <li class="list-group-item" ng-repeat="stuff in mongoStuff">
    <b>{{stuff.component}}</b>
    <div class="text-muted">{{stuff.description}}</div>
  </li>
</ul>

Method4

Factory Calls Google Search

Post Update: the Google Web Search API is no longer available as of September 29, 2014. The post’s example post will no longer return a resultset. Please migrate to the Google Custom Search API (https://developers.google.com/custom-search/). Please read ‘Calling Third-Party HTTP-based RESTful APIs from the MEAN Stack‘ post for more information on using Google’s Custom Search API.

In the last example, we will call the Google Web Search API from an AngularJS Factory. The Google Web Search API exposes a simple RESTful interface. According to Google, “in all cases, the method supported is GET and the response format is a JSON encoded result set with embedded status codes.” Google describes this method of using RESTful access to the API, as “for Flash developers, and those developers that have a need to access the Web Search API from other Non-JavaScript environment.” However, we will access it in our JavaScript-based MEAN stack application, due to the API’s ease of implementation.

Note according to Google’s site, “the Google Web Search API has been officially deprecated…it will continue to work…but the number of requests…will be limited. Therefore, we encourage you to move to Custom Search, which provides an alternative solution.Google Search, or more specifically, the Custom Search JSON/Atom API, is a newer API, but the Web Search API is easier to demonstrate in this brief post than Custom Search JSON/Atom API, which requires the use of an API key.

The googleFactory contains the getSearchResults() function. This function uses $http.jsonp() to call the Google Web Search API RESTful interface and return the promise of the JSONP-formatted (‘JSON with padding’) response. JSONP provides cross-domain access to a JSON payload, by wrapping the payload in a JavaScript function call (callback).

angular.module('generatorMeanstackApp')
  .factory('googleFactory', function ($q, $http) {
    return {
      getSearchResults: function () {
        var deferred = $q.defer(),
          host = 'https://ajax.googleapis.com/ajax/services/search/web',
          args = {
            'version': '1.0',
            'searchTerm': 'mean%20stack',
            'results': '8',
            'callback': 'JSON_CALLBACK'
          },
          params = ('?v=' + args.version + '&q=' + args.searchTerm + '&rsz=' +
            args.results + '&callback=' + args.callback),
          httpPromise = $http.jsonp(host + params);

        httpPromise.then(function (response) {
          deferred.resolve(response);
        }, function (error) {
          console.error(error);
        });

        return deferred.promise;
      }
    };
  });

The getSearchResults() function uses the HTTP GET method to make an HTTP request the following RESTful URI:
https://ajax.googleapis.com/ajax/services/search/web?v=1.0&q=mean%20stack&rsz=8&callback=angular.callbacks._0

Using Google Chrome’s Developer tools, we can preview the Google Web Search JSONP-format HTTP response (abridged). Note the callback function that wraps the JSON payload.

Google Web Search Results in Chrome Browser

Google Web Search Results in Chrome Browser

Within the DataController, if the promise is resolved successfully, our callback function returns the response object. The response object contains a lot of information. We are able to limit that amount of information sent to the view by only assigning the actual search results, an array of eight objects contained in the response object, to the googleStuff property of the $scope object.

$scope.googleStuff = {};
googleFactory.getSearchResults()
  .then(function (response) {
    $scope.googleStuff = response.data.responseData.results;
  }, function (error) {
    console.error(error);
  });

Below is the full response returned by the The googleFactory. Note the path to the data we are interested in: ‘response.data.responseData.results‘.

Google Search Response Object

Google Search Response Object

Below is the filtered results assigned to the googleStuff property:

'googleStuff' Property of the '$scope' Object

‘googleStuff’ Property of the ‘$scope’ Object

The googleStuff property is accessed from the view, using ng-repeat, which displays individual values using Angular expressions, exactly like the previous methods.

<ul class="list-group">
  <li class="list-group-item"
      ng-repeat="stuff in googleStuff">
    <a href="{{unescapedUrl.url}}"
       target="_blank"><b>{{stuff.visibleUrl}}</b></a>

    <div class="text-muted">{{stuff.titleNoFormatting}}</div>
  </li>
</ul>

Method5

Links

, , , , , , , , , , , , , , , , , , , , ,

6 Comments

Retrieving and Displaying Data with AngularJS and the MEAN Stack: Part I

Explore various methods of retrieving and displaying data using AngularJS and the MEAN Stack.

Mobile View of Application on Android Smartphone

Mobile View of Application on Android Smartphone

Introduction

In the following two-part post, we will explore several methods of retrieving and displaying data using AngularJS and the MEAN Stack. The post’s corresponding GitHub project, ‘meanstack-data-samples‘, is based on William Lepinski’s ‘generator-meanstack‘, which is in turn is based on Yeoman’s ‘generator-angular‘. As a bonus, since both projects are based on ‘generator-angular’, all the code generators work. Generators can save a lot of time and aggravation when building AngularJS components.

In part one of this post, we will install and configure the ‘meanstack-data-samples’ project from GitHub, which corresponds to this post. In part two, we will we will look at several methods for retrieving and displaying data using AngularJS:

  • Function within AngularJS Controller returns array of strings.
  • AngularJS Service returns an array of simple object literals to the controller.
  • AngularJS Factory returns the contents of JSON file to the controller.
  • AngularJS Factory returns the contents of JSON file to the controller using a resource object
    (In GitHub project, but not discussed in this post).
  • AngularJS Factory returns a collection of documents from MongoDB Database to the controller.
  • AngularJS Factory returns results from Google’s RESTful Web Search API to the controller.

Preparation

If you need help setting up your development machine to work with the MEAN stack, refer to my last post, Installing and Configuring the MEAN Stack, Yeoman, and Associated Tooling on Windows. You will need to install all the MEAN and Yeoman components.

For this post, I am using JetBrains’ new WebStorm 8RC to build and demonstrate the project. There are several good IDE’s for building modern web applications; WebStorm is one of the current favorites of developers.

Complexity of Modern Web Applications

Building modern web applications using the MEAN stack or comparable technologies is complex. The ‘meanstack-data-samples’ project, and the projects it is based on, ‘generator-meanstack’ and ‘generator-angular’, have dozens of moving parts. In this simple project, we have MongoDBExpressJSAngularJS, Node.js, yoGrunt, BowerGitjQueryTwitter BootstrapKarmaJSHint, jQueryMongoose, and hundreds of other components, all working together. There are almost fifty Node packages and hundreds of their dependencies loaded by npm, in addition to another dozen loaded by Bower.

Installing, configuring, and managing all the parts of a modern web application requires a basic working knowledge of these technologies. Understanding how Bower and npm install and manage packages, how Grunt builds, tests, and serves the application with ExpressJS, how Yo scaffolds applications, how Karma and Jasmine run unit tests, or how Mongoose and MongoDB work together, are all essential. This brief post will primarily focus on retrieving and displaying data, not necessarily how the components all work, or work together.

Installing and Configuring the Project

Environment Variables

To start, we need to create (3) environment variables. The NODE_ENV environment variable is used to determine the environment our application is operating within. The NODE_ENV variable determines which configuration file in the project is read by the application when it starts. The configuration files contain variables, specific to that environment. There are (4) configuration files included in the project. They are ‘development’, ‘test’, ‘production’, and ‘travis’ (travis-ci.org). The NODE_ENV variable is referenced extensively throughout the project. If the NODE_ENV variable is not set, the application will default to ‘development‘.

For this post, set the NODE_ENV variable to ‘test‘. The value, ‘test‘, corresponds to the ‘test‘ configuration file (‘meanstack-data-samples\config\environments\test.js‘), shown below.

// set up =====================================
var express          = require('express');
var bodyParser       = require('body-parser');
var errorHandler     = require('errorhandler');
var favicon          = require('serve-favicon');
var logger           = require('morgan');
var cookieParser     = require('cookie-parser');
var methodOverride   = require('method-override');
var session          = require('express-session');
var path             = require('path');
var env              = process.env.NODE_ENV || 'development';

module.exports = function (app) {
    if ('test' == env) {
        console.log('environment = test');
        app.use(function staticsPlaceholder(req, res, next) {
            return next();
        });
        app.set('db', 'mongodb://localhost/meanstack-test');
        app.set('port', process.env.PORT || 3000);
        app.set('views', path.join(app.directory, '/app'));
        app.engine('html', require('ejs').renderFile);
        app.set('view engine', 'html');
        app.use(favicon('./app/favicon.ico'));
        app.use(logger('dev'));
        app.use(bodyParser());
        app.use(methodOverride());
        app.use(cookieParser('your secret here'));
        app.use(session());

        app.use(function middlewarePlaceholder(req, res, next) {
            return next();
        });

        app.use(errorHandler());
    }
};

The second environment variable is PORT. The application starts on the port indicated by the PORT variable, for example, ‘localhost:3000’. If the the PORT variable is not set, the application will default to port ‘3000‘, as specified in the each of the environment configuration files and the ‘Gruntfile.js’ Grunt configuration file.

Lastly, the CHROME_BIN environment variable is used Karma, the test runner for JavaScript, to determine the correct path to browser’s binary file. Details of this variable are discussed in detail on Karma’s site. In my case, the value for the CHROME_BIN is ‘C:\Program Files (x86)\Google\Chrome\Application\chrome.exe'. This variable is only necessary if you will be configuring Karma to use Chrome to run the tests. The browser can be changes to any browser, including PhantomJS. See the discussion at the end of this post regarding browser choice for Karma.

You can easily set all the environment variables on Windows from a command prompt, with the following commands. Remember to exit and re-open your interactive shell or command prompt window after adding the variables so they can be used.

Install and Configure the Project

To install and configure the project, we start by cloning the ‘meanstack-data-samples‘ project from GitHub. We then use npm and bower to install the project’s dependencies. Once installed, we create and populate the Mongo database. We then use Grunt and Karma to unit test the project. Finally, we will use Grunt to start the Express Server and run the application. This is all accomplished with only a few individual commands. Please note, the ‘npm install’ command could take several minutes to complete, depending on your network speed; the project has many direct and indirect Node.js dependencies.

If everything was installed correctly, running the ‘grunt test’ command should result in output similar to below:

Results of Running 'grunt test' with Chrome

Results of Running ‘grunt test’ with Chrome

If everything was installed correctly, running the ‘grunt server’ command should result in output similar to below:

Results of Running 'grunt server' to Start Application

Results of Running ‘grunt server’ to Start Application

Running the ‘grunt server’ command should start the application and open your browser to the default view, as shown below:

Displaying the Application's Google Search Results on Desktop Browser

Displaying the Application’s Google Search Results on Desktop Browser

Karma’s Browser Choice for Unit Tests

The GitHub project is currently configured to use Chrome for running Karma’s unit tests in the ‘development’ and ‘test’ environments. For the ‘travis’ environment, it uses PhantomJS. If you do not have Chrome installed on your machine, the ‘grunt test’ task will fail during the ‘karma:unit’ task portion. To change Karma’s browser preference, simply change the ‘testBrowser’ variable in the ‘./karma.conf.js’ file, as shown below.

I recommend installing and using  PhantomJS headless WebKit, locally. Since PhantomJS is headless, Karma runs the unit tests without having to open and close browser windows. To run this project on continuous integration servers, like Jenkins or Travis-CI, you must PhantomJS. If you decide to use PhantomJS on Windows, don’t forget add the PhantomJS executable directory path to your ‘PATH’ environment variable to, after downloading and installing the application.

 

Code Generator

As I mentioned at the start of this post, this project was based on William Lepinski’s ‘generator-meanstack‘, which is in turn is based on Yeoman’s ‘generator-angular‘. Optionally, to install the ‘generator-meanstack’ npm package, globally, on our system use the following command The  ‘generator-meanstack’ code generator will allow us to generate additional AngularJS components automatically, within the project, if we choose. The ‘generator-meanstack’ is not required for this post.

npm install -g generator-meanstack

 

Part II

In part two of this post, we will explore each methods of retrieving and displaying data using AngularJS, in detail.

Links

, , , , , , , , , , , , , , , , ,

2 Comments

Installing and Configuring the MEAN Stack, Yeoman, and Associated Tooling on Windows

Configure your Windows environment for developing modern web applications using the popular MEAN Stack and Yeoman suite of utilities.

MEAN Stack Using the MEAN.io Project, Running on Windows

MEAN Stack Using the MEAN.io Project Running on Windows

Introduction

It’s an exciting time to be involved in web development. There are dozens of popular open-source JavaScript frameworks, libraries, code-generators, and associated tools, exploding on to the development scene. It is now possible to use a variety of popular technology mashups to provide a complete, full-stack JavaScript application platform.

MEAN Stack

One of the JavaScript mashups gaining a lot of traction recently is the MEAN Stack. If you’re reading this post, you probably already know MEAN is an acronym for four leading technologies: MongoDB, ExpressJS, AngularJS, and Node.js. The MEAN stack provides an end-to-end JavaScript application solution. MEAN provides a NoSQL document database (Mongo), a server-side solution for JavaScript (Node/Express), and a magic client-side MV* framework (Angular).

Depending in which MEAN stack generator or pre-build project you start with, in addition to the main four technologies, you will pull down several other smaller libraries and frameworks.  These most commonly include jQuery, Twitter Bootstrap, Karma (test runner), Jade (template engine), JSHint, Underscore.js (utility-belt library), Mongoose (MongoDB object modeling tool), Passport (authentication), RequireJS (file and module loader), BreezeJS (data entity management), and so forth.

Common Tooling

If you are involved in these modern web development trends, then you are aware there is also a fairly common set of tools used by a majority of these developers, including source control, IDE, OS, and other helper-utilities. For SCM/VCS, Git is the clear winner. For an IDE, WebStormSublime Text, and Kompozer, are heavy favorites. The platform of choice for most developers most often appears to be either Mac or Linux. It’s far less common to see a demonstration of these technologies, or tutorials built on the Microsoft Windows platform.

Yeoman

Another area of commonality is help-utilities, used to make the development, building, dependency management, and deployment of modern JavaScript applications, easier. Two popular ones are Brunch and YeomanYeoman is also an acronym for a set of popular tools: yo, Grunt, and Bower. The first, yo, is best described as a scaffolding tool. Grunt is the build tool. Bower is a tool for client-side package and dependency management. We will install Yeoman, along with the MEAN Stack,  in this post.

Windows

There is no reason Windows cannot serve as your development and hosting platform for modern web development, without specifically using Microsoft’s .NET stack. In fact, with minimal set-up, you would barely know you were using Windows as opposed to Linux or Mac. In this post, I will demonstrate how to configure your Windows machine for developing these modern web applications using the MEAN Stack and Yeoman.

Here is a list of the components we will discuss:

Installations

Git

The use of Git for source control is obvious. Git is the overwhelming choice of modern developers. Git has been integrated into most major IDEs and hosting platforms. There are hooks into Git available for most leading development tools. However, there are more benefits to using Git than just SCM. Being a Linux/Mac user, I prefer to use a Unix-like shell on Windows, versus the native Windows Command Prompt. For this reason, I use Git for Windows, available from msysGit. This package includes Git SCM, Git Bash, and Git GUI. I use the Git Bash interactive shell almost exclusively for my daily interactions requiring a command prompt. I will be using the Git Bash interactive shell for this post. OpenHatch has great post and training materials available on using Git Bash with Windows.

Using Git Bash on Windows for a Unix-like Experience

Using Git Bash on Windows for a Unix-like Experience

Git for Windows provides a downloadable Windows executable file for installation. Follow the installation file’s instructions.

Git Installation Process

To test your installation of Git for Windows, call the Git binary with the ‘–version’ flag. This flag can be used to test all the components we are installing in this post. If the command returns a value, then it’s a good indication that the component is installed properly and can be called from the command prompt:

gstafford: ~/Documents/git_repos $ git --version
git version 1.9.0.msysgit.0

You can also verify Git using the ‘where’ and ‘which’ commands. The ‘where’ command will display the location of files that match the search pattern and are in the paths specified by the PATH environment variable. The ‘which’ command tells you which file gets executed when you run a command. These commands will work for most components we will install:

gstafford: ~/Documents/git_repos $ where git
C:\Program Files (x86)\Git\bin\git.exe
C:\Program Files (x86)\Git\cmd\git.cmd
C:\Program Files (x86)\Git\cmd\git.exe

gstafford: ~/Documents/git_repos $ which git
/bin/git

Ruby

The reasons for Git are obvious, but why Ruby? Yeoman, specifically yo, requires Ruby. Installing Ruby on Windows is easy. Ruby recommends using RubyInstaller for Windows. RubyInstaller downloads an executable file, making install easy. I am using Ruby 1.9.3. I had previously installed the latest 2.0.0, but had to roll-back after some 64-bit compatibility issues with other applications.

Ruby Installation Process

To test the Ruby installation, use the ‘–version’ flag again:

gstafford: ~/Documents/git_repos $ ruby --version
ruby 1.9.3p484 (2013-11-22) [i386-mingw32]

RubyGems

Optionally, you might also to install RubyGems. RubyGems allow you to add functionality to the Ruby platform, in the form of ‘Gems’. A common Gem used with the MEAN stack is Compass, the Sass-based stylesheet framework creation and maintenance of CSS. According to their website, Ruby 1.9 and newer ships with RubyGems built-in but you may need to upgrade for bug fixes or new features.

On Windows, installation of RubyGems is as simple as downloading the .zip file from the RubyGems download site. To install, RubyGems, unzip the downloaded file. From the root of the unzipped directory, run the following Ruby command:

ruby setup.rb

To confirm your installation:

gstafford: ~/Documents/git_repos $ gem --version
2.2.2

If you already have RubyGems installed, it’s recommended you update RubyGems before continuing. Use the first command, with the ‘–system’ flag, will update to the latest RubyGems. Use the second command, without the tag, if you want to update each of your individually installed Ruby Gems:

gem update --system
gem update

MongoDB

MongoDB provides a great set of installation and configuration instructions for Windows users. To install MongoDB, download the MongoDB package. Create a ‘mongodb’ folder. Mongo recommends at the root of your system drive. Unzip the MongoDB package to ‘c:\mongodb’ folder. That’s really it, there is no installer file.

Next, make a default Data Directory location, use the following two commands:

mkdir c://data && mkdir c://data/db

Unlike most other components, to call Mongo from the command prompt, I had to manually add the path to the Mongo binaries to my PATH environment variable. You can get access to your Windows environment variables using the Windows and Pause keys. Add the path ‘c:\mongodb\bin’ to end of the PATH environment variable value.

Adding Mongo to the PATH Environment Variable

Adding Mongo to the PATH Environment Variable

To test the MongoDB installation, and that the PATH variable is set correctly, close any current interactive shells or command prompt windows. Open a new shell and use the same ‘–version’ flag for Mongo’s three core components:

gstafford: ~/Documents/git_repos
$ mongo --version; mongod --version; mongos --version
MongoDB shell version: 2.4.9

db version v2.4.9
Sun Mar 09 16:26:48.730 git version: 52fe0d21959e32a5bdbecdc62057db386e4e029c

MongoS version 2.4.9 starting: pid=15436 port=27017 64-bit 
host=localhost (--help for usage)
git version: 52fe0d21959e32a5bdbecdc62057db386e4e029c
build sys info: windows sys.getwindowsversion(major=6, minor=1, build=7601, 
platform=2, service_pack='Service Pack 1') 
BOOST_LIB_VERSION=1_49

To start MongoDB, use the ‘mongod’ or ‘start mongod’ commands. Adding ‘start’ opens a new command prompt window, versus tying up your current shell. If you are not using the default MongoDB Data Directory (‘c://data/db’) you created in the previous step, use the ‘–dbpath’ flag, for example ‘start mongod –dbpath ‘c://alternate/path’.

MongoDB Running on Windows

MongoDB Running on Windows

.

MongoDB Default Data Directory Containing New Databases

MongoDB Default Data Directory Containing New MEAN Database

Node.js

To install Node.js, download and run the Node’s .msi installer for Windows. Along with Node.js, you will get npm (Node Package Manager). You will use npm to install all your server-side components, such as Express, yo, Grunt, and Bower.

Node Installation Process

gstafford: ~/Documents/git_repos 
$ node --version && npm --version
v0.10.26
1.4.3

Express

To install Express, the web application framework for node, use npm:

npm install -g express

The ‘-g’ flag (or, ‘–global’ flag) should be used. According to Stack Overflow, ‘if you’re installing something that you want to use in your shell, on the command line or something, install it globally, so that its binaries end up in your PATH environment variable:

gstafford: ~/Documents/git_repos $ express --version
3.5.0

Yeoman – yo, Grunt, and Bower

You will also use npm to install yoGrunt, and Bower. Actually, we will use npm to install the Grunt Command Line Interface (CLI). The Grunt task runner will be installed in your MEAN stack project, locally, later. Read these instructions on the Grunt website for a better explanation. Use the same basic command as with Express:

npm install -g yo grunt-cli bower

To test the installs, run the same command as before:

gstafford: ~/Documents/git_repos 
$ yo --version; grunt --version; bower --version
1.1.2
grunt-cli v0.1.13
1.2.8

If you already had Yeoman installed, confirm you have the latest versions with the ‘npm update’ command:

gstafford: ~/Documents/git_repos/gen-angular-sample 
$ npm update -g yo grunt-cli bower
npm http GET https://registry.npmjs.org/grunt-cli
npm http GET https://registry.npmjs.org/bower
npm http GET https://registry.npmjs.org/yo
npm http 304 https://registry.npmjs.org/bower
npm http 304 https://registry.npmjs.org/yo
npm http 200 https://registry.npmjs.org/grunt-cli

All of the npm installs, including Express, are installed and called from a common location on Windows:

gstafford: ~/Documents/git_repos/gen-angular-sample 
$ where express yo grunt bower
c:\Users\gstaffor\AppData\Roaming\npm\yo
c:\Users\gstaffor\AppData\Roaming\npm\yo.cmd
c:\Users\gstaffor\AppData\Roaming\npm\grunt
c:\Users\gstaffor\AppData\Roaming\npm\grunt.cmd
c:\Users\gstaffor\AppData\Roaming\npm\bower
c:\Users\gstaffor\AppData\Roaming\npm\bower.cmd

gstafford: ~/Documents/git_repos 
$ which express; which yo; which grunt; which bower
~/AppData/Roaming/npm/express
~/AppData/Roaming/npm/yo
~/AppData/Roaming/npm/grunt
~/AppData/Roaming/npm/bower

Use the command, ‘npm list –global | less’ (or, ‘npm ls -g | less’) to view all npm packages installed globally, in a tree-view. After you have generated your project (see below), check the project-specific server-side packages with the ‘npm ls’ command from within the project’s root directory. For the client-side packages, use the ‘bower ls’ command from within the project’s root directory.

If your in a hurry, or have more Windows boxes to configure you can use one npm command for all four components, above:

npm install -g express yo grunt-cli bower

MEAN Boilerplate Generators and Projects

That’s it, you’ve installed most of the core components you need to get started with the MEAN stack on Windows. Next, you will want to download one of the many MEAN boilerplate projects, or use a MEAN code generator with npm and yo. I recommend trying one or all of the following projects. They are each slightly different architecturally, but fairly stable:

Yeoman Running MEAN Generator

Yeoman Running MEAN Generator

.

MEAN Stack Using James Cryer's 'generator-mean'

MEAN Stack Using James Cryer’s ‘generator-mean’

Links

, , , , , , , , , , , , , , , , , , ,

20 Comments

Create Multi-VM Environments Using Vagrant, Chef, and JSON

Create and manage ‘multi-machine’ environments with Vagrant, using JSON configuration files. Allow increased portability across hosts, environments, and organizations. 

Diagram of VM Architecture3

Introduction

As their website says, Vagrant has made it very easy to ‘create and configure lightweight, reproducible, and portable development environments.’ Based on Ruby, the elegantly simple open-source programming language, Vagrant requires a minimal learning curve to get up and running.

In this post, we will create what Vagrant refers to as a ‘multi-machine’ environment. We will provision three virtual machines (VMs). The VMs will mirror a typical three-tier architected environment, with separate web, application, and database servers.

We will move all the VM-specific information from the Vagrantfile to a separate JSON format configuration file. There are a few advantages to moving the configuration information to separate file. First, we can configure any number VMs, while keeping the Vagrantfile exactly the same. Secondly and more importantly, we can re-use the same Vagrantfile to build different VMs on another host machine.

Although certainly not required, I am also using Chef in this example. More specifically, I am using Hosted Chef to further configure the VMs. Like the VM-specific information above, I have also moved the Chef-specific information to a separate JSON configuration file. We can now use the same Vagrantfile within another Chef Environment, or even within another Chef Organization, using an alternate configuration files. If you are not a Chef user, you can disregard that part of the configuration code. Alternately, you can substitute the Chef configuration code for Puppet, if that is your configuration automation tool of choice.

The only items we will not remove from the Vagrantfile are the Vagrant Box and synced folder configurations. These items could also be moved to a separate configuration file, making the Vagrantfile even more generic and portable.

The Code

Below is the VM-specific JSON configuration file, containing all the individual configuration information necessary for Vagrant to build the three VMs: ‘apps’, dbs’, and ‘web’. Each child ‘node’ in the parent ‘nodes’ object contains key/value pairs for VM names, IP addresses, forwarding ports, host names, and memory settings. To add another VM, you would simply add another ‘node’ object.

Next, is the Chef-specific JSON configuration file, containing Chef configuration information common to all the VMs.

Lastly, the Vagrantfile, which loads both configuration files. The Vagrantfile instructs Vagrant to loop through all nodes in the nodes.json file, provisioning VMs for each node. Vagrant then uses the chef.json file to further configure the VMs.

The environment and node configuration items in the chef.json reference an actual Chef Environment and Chef Nodes. They are both part of a Chef Organization, which is configured within a Hosted Chef account.

Each VM has a varying number of ports it needs to configue and forward. To accomplish this, the Vagrantfile not only loops through the each node, it also loops through each port configuration object it finds within the node object. Shown below is the Database Server VM within VirtualBox, containing three forwarding ports.

VirtualBox Port Forwarding Rules

VirtualBox Port Forwarding Rules

In addition to the gists above, this repository on GitHub contains a complete copy of all the code used in the post.

The Results

Running the ‘vagrant up’ command will provision all three individually configured VMs. Once created and running in VirtualBox, Chef further configures the VMs with the necessary settings and applications specific to each server’s purposes. You can just as easily create 10, 100, or 1,000 VMs using this same process.

VirtualBox View of Multiple Virtual Machines

VirtualBox View of Multiple Virtual Machines

.

Virtual Media Manager View of VMs

Virtual Media Manager View of VMs

Helpful Links

  • Dustin Collins’ ‘Multi-VM Vagrant the DRY way’ Blog Post (link)
  • Red Badger’s ‘Automating your Infrastructure with Vagrant & Chef – From Development to the Cloud’ Blog Post (link)
  • David Lutz’s Multi-Machine Vagrantfile GitHub Gist (link)
  • Kevin Jackson’s Multi-Machine Vagrantfile GitHub Gist (link)

, , , , , , , , , ,

4 Comments