Posts Tagged Mongoose
Scaffold a RESTful API with Yeoman, Node, Restify, and MongoDB
Posted by Gary A. Stafford in Continuous Delivery, Enterprise Software Development, Software Development on June 22, 2016
Using Yeoman, scaffold a basic RESTful CRUD API service, based on Node, Restify, and MongoDB.
Introduction
NOTE: Generator updated on 11-13-2016 to v0.2.1.
Yeoman generators reduce the repetitive coding of boilerplate functionality and ensure consistency between full-stack JavaScript projects. For several recent Node.js projects, I created the generator-node-restify-mongodb Yeoman generator. This Yeoman generator scaffolds a basic RESTful CRUD API service, a Node application, based on Node.js, Restify, and MongoDB.
According to their website, Restify, used most notably by Netflix, borrows heavily from Express. However, while Express is targeted at browser applications, with templating and rendering, Restify is keenly focused on building API services that are maintainable and observable.
Along with Node, Restify, and MongoDB, theNode application’s scaffolded by the Node-Restify-MongoDB Generator, also implements Bunyan, which includes DTrace, Jasmine, using jasmine-node, Mongoose, and Grunt.
Portions of the scaffolded Node application’s file structure and code are derived from what I consider the best parts of several different projects, including generator-express, generator-restify-mongo, and generator-restify.
Installation
To begin, install Yeoman and the generator-node-restify-mongodb using npm. The generator assumes you have pre-installed Node and MongoDB.
npm install -g yo npm install -g generator-node-restify-mongodb
Then, generate the new project.
mkdir node-restify-mongodb cd $_ yo node-restify-mongodb
Yeoman scaffolds the application, creating the directory structure, copying required files, and running ‘npm install’ to load the npm package dependencies.
Using the Generated Application
Next, import the supplied set of sample widget documents into the local development instance of MongoDB from the supplied ‘data/widgets.json’ file.
NODE_ENV=development grunt mongoimport --verbose
Similar to Yeoman’s Express Generator, this application contains configuration for three typical environments: ‘Development’ (default), ‘Test’, and ‘Production’. If you want to import the sample widget documents into your Test or Production instances of MongoDB, first change the ‘NODE_ENV’ environment variable value.
NODE_ENV=production grunt mongoimport --verbose
To start the application in a new terminal window, use the following command.
npm start
The output should be similar to the example, below.
To test the application, using jshint and the jasmine-node module, the sample documents must be imported into MongoDB and the application must be running (see above). To test the application, open a separate terminal window, and use the following command.
npm test
The project contains a set of jasmine-node tests, split between the ‘/widgets’ and the ‘/utils’ endpoints. If the application is running correctly, you should see the following output from the tests.
Similarly, the following command displays a code coverage report, using the grunt, mocha, istanbul, and grunt-mocha-istanbul node modules.
grunt coverage
Grunt uses the grunt-mocha-istanbul module to execute the same set of jasmine-node tests as shown above. Based on those tests, the application’s code coverage (statement, line, function, and branch coverage) is displayed.
You may test the running application, directly, by cURLing the ‘/widgets’ endpoints.
curl -X GET -H "Accept: application/json" "http://localhost:3000/widgets"
For more legible output, try prettyjson.
npm install -g prettyjson curl -X GET -H "Accept: application/json" "http://localhost:3000/widgets" --silent | prettyjson curl -X GET -H "Accept: application/json" "http://localhost:3000/widgets/SVHXPAWEOD" --silent | prettyjson
The JSON-formatted response body from the HTTP GET requests should look similar to the output, below.
A much better RESTful API testing solution is Postman. Postman provides the ability to individually configure each environment and abstract that environment-specific configuration, such as host and port, from the actual HTTP requests.
Continuous Integration
As part of being published to both the npmjs and Yeoman registries, the generator-node-restify-mongodb generator is continuously integrated on Travis CI. This should provide an addition level of confidence to the generator’s end-users. Currently, Travis CI tests the generator against Node.js v4, v5, and v6, as well as IO.js. Older versions of Node.js may have compatibility issues with the application.
Additionally, Travis CI feeds test results to Coveralls, which displays the generator’s code coverage. Note the code coverage, shown below, is reported for the yeoman generator, not the generator’s scaffolded application. The scaffolded application’s coverage is shown above.
Application Details
API Endpoints
The scaffolded application includes the following endpoints.
# widget resources var PATH = '/widgets'; server.get({path: PATH, version: VERSION}, findDocuments); server.get({path: PATH + '/:product_id', version: VERSION}, findOneDocument); server.post({path: PATH, version: VERSION}, createDocument); server.put({path: PATH, version: VERSION}, updateDocument); server.del({path: PATH + '/:product_id', version: VERSION}, deleteDocument); # utility resources var PATH = '/utils'; server.get({path: PATH + '/ping', version: VERSION}, ping); server.get({path: PATH + '/health', version: VERSION}, health); server.get({path: PATH + '/info', version: VERSION}, information); server.get({path: PATH + '/config', version: VERSION}, configuraton); server.get({path: PATH + '/env', version: VERSION}, environment);
The Widget
The Widget is the basic document object used throughout the application. It is used, primarily, to demonstrate Mongoose’s Model and Schema. The Widget object contains the following fields, as shown in the sample widget, below.
{ "product_id": "4OZNPBMIDR", "name": "Fapster", "color": "Orange", "size": "Medium", "price": "29.99", "inventory": 5 }
MongoDB
Use the mongo shell to access the application’s MongoDB instance and display the imported sample documents.
mongo > show dbs > use node-restify-mongodb-development > show tables > db.widgets.find()
The imported sample documents should be displayed, as shown below.
Environmental Variables
The scaffolded application relies on several environment variables to determine its environment-specific runtime configuration. If these environment variables are present, the application defaults to using the Development environment values, as shown below, in the application’s ‘config/config.js’ file.
var NODE_ENV = process.env.NODE_ENV || 'development'; var NODE_HOST = process.env.NODE_HOST || '127.0.0.1'; var NODE_PORT = process.env.NODE_PORT || 3000; var MONGO_HOST = process.env.MONGO_HOST || '127.0.0.1'; var MONGO_PORT = process.env.MONGO_PORT || 27017; var LOG_LEVEL = process.env.LOG_LEVEL || 'info'; var APP_NAME = 'node-restify-mongodb-';
Future Project TODOs
Future project enhancements include the following:
- Add filtering, sorting, field selection and paging
- Add basic HATEOAS-based response features
- Add authentication and authorization to production MongoDB instance
- Convert from out-dated jasmine-node to Jasmine?
Preventing Race Conditions Between Containers in ‘Dockerized’ MEAN Applications
Posted by Gary A. Stafford in Bash Scripting, Build Automation, Client-Side Development, DevOps, Enterprise Software Development, Software Development on November 30, 2014
Introduction
The MEAN stack is a has gained enormous popularity as a reliable and scalable full-stack JavaScript solution. MEAN web application’s have four main components, MongoDB, Express, AngularJS, and Node.js. MEAN web-applications often includes other components, such as Mongoose, Passport, Twitter Bootstrap, Yoeman, Grunt or Gulp, and Bower. The two most popular ready-made MEAN application templates are MEAN.io from Linnovate, and MEAN.JS. Both of these offer a ready-made application framework for building MEAN applications.
Docker has also gained enormous popularity. According to Docker, Docker is an open platform, which enables developers and sysadmins apps to be quickly assembled from components. ‘Dockerized’ apps are completely portable and can run anywhere.
Docker is an ideal solution for MEAN applications. Being a full-stack JavaScript solution, MEAN applications are based on a multi-tier architecture. The MEAN application’s data tier contains the MongoDB noSQL database. The application tier (logic tier) contains Node.js and Express. The application tier can also contain other components, such as Mongoose, a Node.js Object Document Mapper (ODM) for MongoDB, and Passport, an authentication middleware for Node.js. Lastly, the presentation tier (front end) has client-side tools, such as AngularJS and Twitter Bootstrap.
Using Docker, we can ‘Dockerize’ or containerize each tier of a MEAN application, mirroring the physical architecture we would deploy a MEAN application to, in a Production environment. Just as we would always run a separate database server or servers for MongoDB, we can isolate MongoDB into a Docker container. Likewise, we can isolate the Node.js web server, along with the rest of the components (Mongoose, Express, Passport) on the application and presentation tiers, into a Docker container. We can easily add more containers, for more functionality, such as load-balancing and reverse-proxies (nginx), and caching (Redis and Memcached).
The MEAN.JS project has been very progressive in implementing Docker, to offer a more realistic environment for development and testing. An additional tool that the MEAN.JS project has implemented, to automate the creation of multiple Docker containers, is Fig. The tool, Fig, provides quick, automated creation of multiple, linked Docker containers.
Using Docker and Fig, a Developer can pull down ready-made base containers from Docker Hub, configure the containers as part of a multi-tier application environment, deploy our MEAN application components to the containers, and start the applications, all with a short list of commands.
Note, I said development and test, not production. To extend Docker and Fig to production, you can use tools such as Flocker. Flocker, by ClusterHQ, can scale the single-host Fig environment to multiple containers on multiple machines (hosts).
Race Conditions
Docker containers have a very fast start-up time, compared to other technologies, such as VMs (virtual machines). However, based on their contents, containers take varying amounts of time to fully start-up. In most multi-tier applications, there is a required start-up sequence for components (tiers, servers, applications). For example, in a database-driven application, like a MEAN application, you should make sure the MongoDB database server is up and running, before starting the application. Although this is obvious, it becomes harder to guarantee the order in which components will start-up, when you leverage an asynchronous, automated, continuous delivery solution like Docker with Fig.
When component dependencies are not met because another container is not fully started, we can refer to this as race condition. I have found with most multi-container MEAN application, the slower starting MongoDB data container prevents the quicker-starting Node.js web-application container from properly starting the MEAN application. In other words, the application crashes.
Fixing Race Conditions with MEAN.JS Applications
In order to eliminate race conditions, we need to script our start-up sequence to guarantee the order in which components will start, ensuring the overall application starts correctly. Specifically in this post, we will eliminate the potential race condition between the MongoDB data container (db_1) and the Node.js web-application container (web_1). At the same time, we will fix a small error with the existing MEAN.JS project, that prevents proper start-up of the ‘dockerized’ container MEAN.JS application.
Download and Build MEAN.JS App
Clone the meanjs/mean repository, and install npm and bower packages.
git clone https://github.com/meanjs/mean.git cd mean npm install bower install
Modify MEAN.JS App
- Add
fig_start.sh
start-up script to root of mean project. - Modify the Dockerfile, replace
CMD["grunt"]
withCMD /bin/sh /home/mean/wait_mongo_start.sh
- Optional, add
wait_mongo_start.sh
clean-up script to root of mean project.
Fix Existing Issue with MEAN.JS App When Using Docker and Fig
The existing MEAN.JS application references localhost
in the development configuration (config/env/development.js
). The development
configuration is the one used by the MEAN.JS application, at start-up. The MongoDB data container (db_1) is not running on localhost
, it is running on a IP address, assigned my Docker. To discover the IP address, we must reference an environment variable (DB_1_PORT_27017_TCP_ADDR
), created by Docker, within the Node.js web-application container (web_1).
- Modify the config/env/development.js file, add
var DB_HOST = process.env.DB_1_PORT_27017_TCP_ADDR || 'localhost';
- Modify the config/env/development.js file, change
db: 'mongodb://localhost/mean-dev',
todb: 'mongodb://' + DB_HOST + '/mean-dev',
Start the Application
Start the application using Fig commands or using the clean-up/start-up script (sh fig_start.sh
).
- Run
fig build && fig up
- Alternately, run
sh fig_start.sh
The Details…
The CMD
command is the last step in the Dockerfile
.The CMD
command sets the wait_mongo_start.sh
script to execute in the Node.js web-application container (web_1) when the container starts. This script prevents the grunt
command from running, until nc
(or netcat) succeeds at connecting to the IP address and port of mongod
, the primary daemon process for the MongoDB system, on the MongoDB data container (db_1). The script uses a 3-second polling interval, which can be modified if necessary.
#!/bin/sh polling_interval=3 # optional, view db_1 container-related env vars #env | grep DB_1 | sort echo "wait for mongo to start first..." # wait until mongo is running in db_1 container until nc -z $DB_1_PORT_27017_TCP_ADDR $DB_1_PORT_27017_TCP_PORT do echo "waiting for $polling_interval seconds..." sleep $polling_interval done # start node app grunt
The environment variables referenced in the script are created in the Node.js web-application container (web_1), automatically, by Docker. They are shown in the screen grab, below. You can discover these variables by uncommenting the env | grep DB_1 | sort
line, above.
The Dockerfile
modification is highlighted below.
#FROM dockerfile/nodejs MAINTAINER Matthias Luebken, matthias@catalyst-zero.com WORKDIR /home/mean # Install Mean.JS Prerequisites RUN npm install -g grunt-cli RUN npm install -g bower # Install Mean.JS packages ADD package.json /home/mean/package.json RUN npm install # Manually trigger bower. Why doesn't this work via npm install? ADD .bowerrc /home/mean/.bowerrc ADD bower.json /home/mean/bower.json RUN bower install --config.interactive=false --allow-root # Make everything available for start ADD . /home/mean # Currently only works for development ENV NODE_ENV development # Port 3000 for server # Port 35729 for livereload EXPOSE 3000 35729 CMD /bin/sh /home/mean/wait_mongo_start.sh
The config/env/development.js
modifications are highlighted below (abridged code).
'use strict'; // used when building application using fig and Docker var DB_HOST = process.env.DB_1_PORT_27017_TCP_ADDR || 'localhost'; module.exports = { db: 'mongodb://' + DB_HOST + '/mean-dev', log: { // Can specify one of 'combined', 'common', 'dev', 'short', 'tiny' format: 'dev', // Stream defaults to process.stdout // Uncomment to enable logging to a log on the file system options: { //stream: 'access.log' } }, ...
The fig_start.sh
file is optional and not part of the solution for the race condition. Instead of repeating multiple commands, I prefer running a single script, which can execute the commands, consistently. Note, commands in this script remove ALL ‘Exited’ containers and untagged (<none>) images.
#!/bin/sh # remove all exited containers echo "Removing all 'Exited' containers..." docker rm -f $(docker ps --filter 'status=Exited' -a) > /dev/null 2>&1 # remove all images echo "Removing all untagged images..." docker rmi $(docker images | grep "^" | awk "{print $3}") > /dev/null 2>&1 # build and start containers with fig fig build && fig up
MEAN Application Start-Up Screen Grabs
Below are screen grabs showing the MEAN.JS application starting up, both before and after the changes were implemented.