Calling Third-Party HTTP-based RESTful APIs from the MEAN Stack

Example of calling Google’s Custom Search http-based RESTful API, using Node.js with Express and Request, from a MEAN.io-generated MEAN stack application. CustomSearchExample

Introduction

Most MEAN stack articles and tutorials demonstrate how AngularJS, on the client-side, calls Node.js with Express on the server-side, via a http-based RESTful API. In turn, on the server-side, Node.js with Express, and often a ODM like Mongoose, calls MongoDB. Below is a simple, high-level sequence diagram of a typical MEAN stack request/response data flow from the client to the server to the database, and back.

Typical MEAN Stack Request/Response Data Flow

Typical MEAN Stack Request/Response Data Flow

However in many situations, applications don’t only call into their own application stack. Applications often call third-party http-based RESTful APIs, including social networks, cloud providers, e-commerce, and news aggregators. Twitter’s REST API and Facebook Graph API are two popular social network examples. Within larger enterprise environments, applications call multiple internal applications. For example, an online retailer’s storefront application accesses their own inventory control system via RESTful URIs. This is the same RESTful API the retailer’s authorized resellers use to interact with the retailer’s own inventory control system.

Calling APIs from the MEAN Stack

From the Client-Side
There are two ways to call third-party http-based APIs from a MEAN stack application. The first approach is calling directly from the client-side. AngularJS calls the third-party API, directly. All logic is on the client-side, instead of on the server-side. Node.js and Express are not involved in the process. The approach requires less moving parts than the next approach, but is less secure and places more demand on the client to handle the application’s business logic. Below is a simple, high-level sequence diagram demonstrating a request/response data flow from AngularJS on the client-side to a third-party API, and back.

Example AngularJS/Third-Party API Request/Response Data Flow

Example AngularJS/Third-Party API Request/Response Data Flow

From the Server-Side
The second approach, using Node.js and Express, on the servers-side, is slightly more complex. However, this approach is also more architecturally sound, scalable, secure, and performant. AngularJS, on the client side, calls Node.js with Express, on the server-side. Node.js with Express then calls the service and pass the response back to the client-side, to AngularJS. Below is a simple, high-level sequence diagram demonstrating a request/response data flow from the client-side to the server-side, to a third-party API, and back.

Example Node.js/Third-Party API Request/Response Data Flow

Example Node.js/Third-Party API Request/Response Data Flow

Example

MEAN.io
Using the MEAN.io ‘FullStack JS Development’ framework, I have created a basic example of calling Google’s Custom Search http-based RESTful API, from Node.js with Express and Request. MEAN.io provides an ready-made MEAN stack boilerplate framework/generator, saving a lot of coding time. Irregardless of the generator or framework you choose, you would architect this example the same.

Google Custom Search API
Google provides the Custom Search API as part of their Custom Search, one of many API’s, available through the Google Developers portal. According to Google, “the JSON/Atom Custom Search API lets you develop websites and applications to retrieve and display search results from Google Custom Search programmatically. With this API, you can use RESTful requests to get either web search or image search results in JSON or Atom format.

Google APIs Explorer - Exploring Custom Search API

Google APIs Explorer – Exploring Custom Search API

In order to use the Custom Search API, you will need to first create a Google account, API project, API keyCustom Search Engine (CSE), and CSE ID, through Google’s Developers Console. If you have previously worked with Google, FaceBook, or Twitter APIs, creating an API project, CSE, API key, and CSE ID, if very similar.

Google Custom Search - Your Search Engine ID

Google Custom Search – Your Search Engine ID

Like most of Google’s APIs, the Custom Search API pricing and quotas depend on the engine’s edition. You have a choice of two engines. According to Google, the free Custom Search Engine provides 100 search queries per day for free. If you need more, you may sign up for billing in the Developers Console. Additional requests cost $5 per 1000 queries, up to 10k queries per day. The limit of 100 is more than enough for this demonstration.

Installing and Configuring the Project

All the code for this project is available on GitHub at /meanio-custom-search. Before continuing, make sure you have the prerequisite software installed – GitNode.js with npm, and MongoDB. To install the GitHub project, follow these commands:

git clone https://github.com/garystafford/meanio-custom-search.git
cd meanio-custom-search
npm install

Alternatively, if you want to code the project yourself, these are the commands I used to set up the base MEAN.io framework, and create ‘search‘ package:

sudo npm install -g mean-cli
mean init meanio-custom-search
cd meanio-custom-search
npm install
mean package search

After creating your own CSE ID and API key, create two environmental variables, GOOGLE_CSE_ID and GOOGLE_API_KEY, to hold the values.

echo "export GOOGLE_API_KEY=<YOUR_API_KEY_HERE>" >> ~/.bashrc
echo "export GOOGLE_CSE_ID=<YOUR_CSE_ID_HERE>"   >> ~/.bashrc

The code is run from a terminal prompt with the grunt command. Then, in the browser, go to http://localhost:3000. Once on the main home page, you can navigate to the ‘Search Example’ page, and input a search term, such as ‘MEAN Stack’. All the instructions on the MEAN.io Github site, apply to this project.

The Project’s Architecture

According to MEAN.io, everything in mean.io is a ‘package’. When extending mean with custom functionality, you create a new ‘package’. In this case, I have created a ‘search’ package, with the command above, ‘mean package search‘. Below is the basic file structure of the ‘search‘ package, within the overall MEAN.io project framework. The ‘public‘ folder contains all the client-side, AngularJS code. The ‘server‘ folder contains all the server-side, Node.js/Express/Request code. Note that each ‘package’ also has its own ‘package.json‘ npm file and ‘bower.json‘ Bower file.

Folder Structure of Search Package with Callouts

The simple, high-level sequence diagram below shows the flow of the custom search request from the ‘Search Example’ view to the Google Custom Search API. The diagram also shows the response from the Google Custom Search API all the way back up the MEAN stack to the client-side view.

High-Level Custom Search API Request/Response Data Flow

High-Level Custom Search API Request/Response Data Flow

Client-Side Request/Response
If you view the network traffic in your web browser, you will see a RESTful URI call is made between AngularJS’ service factory, on the client-side, and Node.js with Express, on the server-side. The RESTful endpoint, called with $http.jsonp(), will be similar to: http://localhost:3000/customsearch/MEAN.io/10?callback=angular.callbacks._0. In actuality, the callback parameter name, the AngularJS service factory, is ‘JSON_CALLBACK‘. This is replaced by AngularJS with an incremented ‘angular.callbacks._X‘ parameter name, making the response callback name incremental and unique.

The response returned to AngularJS from Node.js is a sub-set of full response from Google’s Custom Search API. Only the search results items and a ‘200’ status code are returned to AngularJS as JavaScript, JSONP wrapped in a callback. Below is a sample response, truncated to just a single search result. I have highlighted the four fields that are displayed in the ‘Search Example’ view, using AngularJS’ ng-repeat directive.

/**/
typeof angular.callbacks._0 === 'function' && angular.callbacks._0({
    "statusCode": 200,
    "items"     : [{
        "kind"            : "customsearch#result",
        "title"           : "MEAN.IO - MongoDB, Express, Angularjs Node.js powered fullstack ...",
        "htmlTitle"       : "<b>MEAN</b>.<b>IO</b> - MongoDB, Express, Angularjs Node.js powered fullstack <b>...</b>",
        "link"            : "http://mean.io/",
        "displayLink"     : "mean.io",
        "snippet"         : "MEAN - MongoDB, ExpressJS, AngularJS, NodeJS. based fullstack js framework.",
        "htmlSnippet"     : "<b>MEAN</b> - MongoDB, ExpressJS, AngularJS, NodeJS. based fullstack js framework.",
        "cacheId"         : "_CZQNNP6VMEJ",
        "formattedUrl"    : "mean.io/",
        "htmlFormattedUrl": "<b>mean</b>.<b>io</b>/",
        "pagemap"         : {
            "cse_image"    : [{"src": "http://i.ytimg.com/vi/oUtWtSF_VNY/hqdefault.jpg"}],
            "cse_thumbnail": [{
                "width" : "259",
                "height": "194",
                "src"   : "https://encrypted-tbn2.gstatic.com/images?q=tbn:ANd9GcSIVwPo7OcW9u_b3P3DGxv8M7rKifGZITi1Bhmpy10_I2tlUqjRUVVUBKNG"
            }],
            "metatags"     : [{
                "viewport"      : "width=1024",
                "fb:app_id"     : "APP_ID",
                "og:title"      : "MEAN.IO - MongoDB, Express, Angularjs Node.js powered fullstack web framework - MEAN.IO - MongoDB, Express, Angularjs Node.js powered fullstack web framework",
                "og:description": "MEAN  MongoDB, ExpressJS, AngularJS, NodeJS.",
                "og:type"       : "website",
                "og:url"        : "APP_URL",
                "og:image"      : "APP_LOGO",
                "og:site_name"  : "MEAN.IO",
                "fb:admins"     : "APP_ADMIN"
            }]
        }
    }]
});

Server-Side Request/Response
On the server-side, Node.js with Express and Request, calls the Google Custom Search API via a RESTful URI. The RESTful URI, called with request.get(), will be similar to: https://www.googleapis.com/customsearch/v1?cx=ed026i714398r53510g2ja1ru6741h:73&q=MEAN.io&num=10&key=jtHeNjIAtSa1NaWJzmVvBC7qoubrRSyIAmVJjpQu. Note the URI contains both the your CSE ID and API key (not my real ones, of course). The JSON response from Google’s Custom Search API has other data, which is not necessary to display the results.

Shown below is a sample response with a single search result. Like the URI above, the response from Google has your Custom Search Engine ID. Your CSE ID and API key should both be considered confidential and not visible to the client. The CSE ID could be easily intercepted in both the URI and the response object, and used without your authorization. Google has a page that suggests methods to keep your keys secure.

{
  kind: "customsearch#search",
  url: {
    type: "application/json",
    template: "https://www.googleapis.com/customsearch/v1?q={searchTerms}&num={count?}&start={startIndex?}&lr={language?}&safe={safe?}&cx={cx?}&cref={cref?}&sort={sort?}&filter={filter?}&gl={gl?}&cr={cr?}&googlehost={googleHost?}&c2coff={disableCnTwTranslation?}&hq={hq?}&hl={hl?}&siteSearch={siteSearch?}&siteSearchFilter={siteSearchFilter?}&exactTerms={exactTerms?}&excludeTerms={excludeTerms?}&linkSite={linkSite?}&orTerms={orTerms?}&relatedSite={relatedSite?}&dateRestrict={dateRestrict?}&lowRange={lowRange?}&highRange={highRange?}&searchType={searchType}&fileType={fileType?}&rights={rights?}&imgSize={imgSize?}&imgType={imgType?}&imgColorType={imgColorType?}&imgDominantColor={imgDominantColor?}&alt=json"
  },
  queries: {
    nextPage: [
      {
        title: "Google Custom Search - MEAN.io",
        totalResults: "12100000",
        searchTerms: "MEAN.io",
        count: 10,
        startIndex: 11,
        inputEncoding: "utf8",
        outputEncoding: "utf8",
        safe: "off",
        cx: "ed026i714398r53510g2ja1ru6741h:73"
      }
    ],
    request: [
      {
        title: "Google Custom Search - MEAN.io",
        totalResults: "12100000",
        searchTerms: "MEAN.io",
        count: 10,
        startIndex: 1,
        inputEncoding: "utf8",
        outputEncoding: "utf8",
        safe: "off",
        cx: "ed026i714398r53510g2ja1ru6741h:73"
      }
    ]
  },
  context: {
    title: "my_search_engine"
  },
  searchInformation: {
    searchTime: 0.237431,
    formattedSearchTime: "0.24",
    totalResults: "12100000",
    formattedTotalResults: "12,100,000"
  },
  items: [
    {
      kind: "customsearch#result",
      title: "MEAN.IO - MongoDB, Express, Angularjs Node.js powered fullstack ...",
      htmlTitle: "<b>MEAN</b>.<b>IO</b> - MongoDB, Express, Angularjs Node.js powered fullstack <b>...</b>",
      link: "http://mean.io/",
      displayLink: "mean.io",
      snippet: "MEAN - MongoDB, ExpressJS, AngularJS, NodeJS. based fullstack js framework.",
      htmlSnippet: "<b>MEAN</b> - MongoDB, ExpressJS, AngularJS, NodeJS. based fullstack js framework.",
      cacheId: "_CZQNNP6VMEJ",
      formattedUrl: "mean.io/",
      htmlFormattedUrl: "<b>mean</b>.<b>io</b>/",
      pagemap: {
        cse_image: [
          {
            src: "http://i.ytimg.com/vi/oUtWtSF_VNY/mqdefault.jpg"
          }
        ],
        cse_thumbnail: [
          {
            width: "256",
            height: "144",
            src: "https://encrypted-tbn1.gstatic.com/images?q=tbn:ANd9GcTXm3rYwGdWs9Cx3s5VvooATKlgtrVZoP83hxfAOjGvsRMqLpMKuycVl_sF"
          }
        ],
        metatags: [
          {
            viewport: "width=1024",
            fb:app_id: "APP_ID",
            og:title: "MEAN.IO - MongoDB, Express, Angularjs Node.js powered fullstack web framework - MEAN.IO - MongoDB, Express, Angularjs Node.js powered fullstack web framework",
            og:description: "MEAN  MongoDB, ExpressJS, AngularJS, NodeJS.",
            og:type: "website",
            og:url: "APP_URL",
            og:image: "APP_LOGO",
            og:site_name: "MEAN.IO",
            fb:admins: "APP_ADMIN"
          }
        ]
      }
    }
  ]
}

The best way to understand the project’s sample code is to clone the GitHub repo, and explore the files directly associated with the search, starting in the ‘packages/custom/search‘ subdirectory.

Helpful Links

Learn REST: A RESTful Tutorial
Using an AngularJS Factory to Interact with a RESTful Service
Google APIs Client Library for JavaScript (Beta)
REST-ful URI design
Creating a CRUD App in Minutes with Angular’s $resource

, , , , , , , , , , , , , ,

  1. #1 by Shruts on February 22, 2015 - 10:54 pm

    good tutorial, elaborately explained.

  2. #2 by Kyle Rebstock on April 29, 2015 - 10:11 pm

    Great tutorial Gary Stafford!

    This tutorial had me regularly hip thrusting the air every few moments with excitation at how great of a tutorial this was and painless it was pretty painless! Very well documented and you went into amazing detail! everything up and running here with just a few updates to the bower for mongo updates and what not.

    I run a search and get a null returned on grunt and “Failed to load resource: the server responded with a status of 401 (Unauthorized)” on the console followed by “failed to load resource: net::ERR_EMPTY_RESPONSE”. I assume there is a tad bit more custom work to be done here with the likes of the search engine or keys, but I’m not entirely sure where to start as I am very new to mean/express/backend and all that. Any ideas or pointers would be appreciated to the max!

    Adoringly,
    Kyle Rebstock

  3. #3 by Kyle Rebstock on April 29, 2015 - 10:23 pm

    Great tutorial Gary Stafford!

    This tutorial had me regularly hip thrusting the air every few moments with excitation at how great of a tutorial this was and painless it was pretty painless! Very well documented and you went into amazing detail! everything up and running here with just a few updates to the bower for mongo updates and what not.

    I run a search and get a null returned on grunt and “Failed to load resource: the server responded with a status of 401 (Unauthorized)” on the console followed by “failed to load resource: net::ERR_EMPTY_RESPONSE”. I assume there is a tad bit more custom work to be done here with the likes of the search engine or keys, but I’m not entirely sure where to start as I am very new to mean/express/backend and all that. Any ideas or pointers would be appreciated to the max!

    Adoringly,
    Kyle Rebstock

    • #4 by Gary A. Stafford on April 29, 2015 - 11:26 pm

      Thank you! In order to use the Custom Search API, you will need to first create a Google account, API project, API key, Custom Search Engine (CSE), and CSE ID, through Google’s Developers Console. I would start by making sure all the those items are working, before writing any code. Use Postman or similar RESTful test tool. Then incorporate into code as shown in post.

      • #5 by Kyle Rebstock on April 30, 2015 - 1:45 am

        I have a google acc, API project, API key, custom search engine CSE, and cse id through the google dev. console and I can easily use their script posted on the html page to do custom searches so I think everything on that end is working fine. I however am really wanting the mean framework to handle the call using your fantastic app so that I can learn the proper dealings in the backend. I am hitting the api on the google developers console as it says that I have a max of 100 request per day and I have used 30 of them, but there was a client error(4xx) on every request. I am told from another developer friend that this doesn’t necessarily mean it is in fact the client side. So I am kinda lost. I’ll continue to tinker and search, but if you have any further ideas I am happy to hear.

      • #6 by Kyle Rebstock on April 30, 2015 - 2:51 am

        I remembered something from an internship where some research brought about some info like this http://stackoverflow.com/questions/1217926/jsonp-callback-doesnt-execute-when-running-at-localhost/1218004#1218004. To summarize it says that it is possible that jsonp may not work with a url of localhost because 1.’Localhost’ is one of few DNS hostnames that are dynamic in meaning depending on when and where it’s queried, making the potential targets vulnerable. and he recommends changing the hostname.

        Also!
        I have a new and exciting piece to the puzzle. I have managed to call directly in Advanced Rest Client without the use of headers(should i be using headers?) this url
        https://www.googleapis.com/customsearch/v1?cx=AIzaSyDYPylFKYXm25ehQNEt8ViC30e74U6kbzE&q=asdf&num=10&key=AIzaSyA5lOVg_veudp8iUMc2GkTMnMXPJrjFNrk

        and I received this object back!

        {
        error: {
        errors: [1]
        0: {
        domain: “usageLimits”
        reason: “ipRefererBlocked”
        message: “There is a per-IP or per-Referer restriction configured on your API key and the request does not match these restrictions. Please use the Google Developers Console to update your API key configuration if request from this IP or referer should be allowed.”
        extendedHelp: “https://console.developers.google.com”
        }-

        code: 403
        message: “There is a per-IP or per-Referer restriction configured on your API key and the request does not match these restrictions. Please use the Google Developers Console to update your API key configuration if request from this IP or referer should be allowed.”
        }-
        }

        Is there a specific way to include the uris and urls that I’ll be working with?

        Thanks again for all of this help! You’re of a very magnanimous man for sure and I sincerely appreciate the quick responses! You’re blog has to be one of the best I’ve come across so far!

  4. #7 by Gary A. Stafford on April 30, 2015 - 10:54 pm

    Just to clarify, you get an error calling the RESTful endpoint at Google ( i.e. https://www.googleapis.com/customsearch/v1?…), as node would do from the server-side? Or, you do get an error if you call the RESTful endpoint of the node server (i.e.http://localhost:3000/customsearch/MEAN.io/10?callback=angular.callbacks._0), like the client would do?

    If it is calling Google, what does the error say. For example, if I call Google with the wrong credentials in my request, like from Chrome, I see ‘keyInvalid’

    {
    error: {
    errors: [
    {
    domain: “usageLimits”,
    reason: “keyInvalid”,
    message: “Bad Request”
    }
    ],
    code: 400,
    message: “Bad Request”
    }
    }

  5. #8 by Karu Price on September 10, 2016 - 4:00 pm

    Why had nobody explained it like this before??? Mr. Stafford, thank you!

  6. #9 by Arnab Saha on November 9, 2016 - 1:36 pm

    Hey !

    Been trying to setup this project. While, cloning the repo directly had an issue of the bson C++ module not found as mentioned in the other issue. I managed to update the mean.io boilerplate project and setup a new package as mentioned in the description.

    But, I still have an issue, ” Refused to execute script from ‘http://127.0.0.1:3000/customsearch/hello/10?callback=angular.callbacks._0’ because its MIME type (‘text/html’) is not executable, and strict MIME type checking is enabled ”

    when I hit submit with the keyword on the route: localhost:3000/search

    Any pointers? Appreciate any help!

    Thanks,
    Arnab

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: