Posts Tagged search engine id
Calling Third-Party HTTP-based RESTful APIs from the MEAN Stack
Posted by Gary A. Stafford in Client-Side Development, Software Development on January 6, 2015
Example of calling Google’s Custom Search http-based RESTful API, using Node.js with Express and Request, from a MEAN.io-generated MEAN stack application.
Introduction
Most MEAN stack articles and tutorials demonstrate how AngularJS, on the client-side, calls Node.js with Express on the server-side, via a http-based RESTful API. In turn, on the server-side, Node.js with Express, and often a ODM like Mongoose, calls MongoDB. Below is a simple, high-level sequence diagram of a typical MEAN stack request/response data flow from the client to the server to the database, and back.
However in many situations, applications don’t only call into their own application stack. Applications often call third-party http-based RESTful APIs, including social networks, cloud providers, e-commerce, and news aggregators. Twitter’s REST API and Facebook Graph API are two popular social network examples. Within larger enterprise environments, applications call multiple internal applications. For example, an online retailer’s storefront application accesses their own inventory control system via RESTful URIs. This is the same RESTful API the retailer’s authorized resellers use to interact with the retailer’s own inventory control system.
Calling APIs from the MEAN Stack
From the Client-Side
There are two ways to call third-party http-based APIs from a MEAN stack application. The first approach is calling directly from the client-side. AngularJS calls the third-party API, directly. All logic is on the client-side, instead of on the server-side. Node.js and Express are not involved in the process. The approach requires less moving parts than the next approach, but is less secure and places more demand on the client to handle the application’s business logic. Below is a simple, high-level sequence diagram demonstrating a request/response data flow from AngularJS on the client-side to a third-party API, and back.
From the Server-Side
The second approach, using Node.js and Express, on the servers-side, is slightly more complex. However, this approach is also more architecturally sound, scalable, secure, and performant. AngularJS, on the client side, calls Node.js with Express, on the server-side. Node.js with Express then calls the service and pass the response back to the client-side, to AngularJS. Below is a simple, high-level sequence diagram demonstrating a request/response data flow from the client-side to the server-side, to a third-party API, and back.
Example
MEAN.io
Using the MEAN.io ‘FullStack JS Development’ framework, I have created a basic example of calling Google’s Custom Search http-based RESTful API, from Node.js with Express and Request. MEAN.io provides an ready-made MEAN stack boilerplate framework/generator, saving a lot of coding time. Irregardless of the generator or framework you choose, you would architect this example the same.
Google Custom Search API
Google provides the Custom Search API as part of their Custom Search, one of many API’s, available through the Google Developers portal. According to Google, “the JSON/Atom Custom Search API lets you develop websites and applications to retrieve and display search results from Google Custom Search programmatically. With this API, you can use RESTful requests to get either web search or image search results in JSON or Atom format.”
In order to use the Custom Search API, you will need to first create a Google account, API project, API key, Custom Search Engine (CSE), and CSE ID, through Google’s Developers Console. If you have previously worked with Google, FaceBook, or Twitter APIs, creating an API project, CSE, API key, and CSE ID, if very similar.
Like most of Google’s APIs, the Custom Search API pricing and quotas depend on the engine’s edition. You have a choice of two engines. According to Google, the free Custom Search Engine provides 100 search queries per day for free. If you need more, you may sign up for billing in the Developers Console. Additional requests cost $5 per 1000 queries, up to 10k queries per day. The limit of 100 is more than enough for this demonstration.
Installing and Configuring the Project
All the code for this project is available on GitHub at /meanio-custom-search. Before continuing, make sure you have the prerequisite software installed – Git, Node.js with npm, and MongoDB. To install the GitHub project, follow these commands:
git clone https://github.com/garystafford/meanio-custom-search.git cd meanio-custom-search npm install
Alternatively, if you want to code the project yourself, these are the commands I used to set up the base MEAN.io framework, and create ‘search
‘ package:
sudo npm install -g mean-cli mean init meanio-custom-search cd meanio-custom-search npm install mean package search
After creating your own CSE ID and API key, create two environmental variables, GOOGLE_CSE_ID
and GOOGLE_API_KEY
, to hold the values.
echo "export GOOGLE_API_KEY=<YOUR_API_KEY_HERE>" >> ~/.bashrc echo "export GOOGLE_CSE_ID=<YOUR_CSE_ID_HERE>" >> ~/.bashrc
The code is run from a terminal prompt with the grunt
command. Then, in the browser, go to http://localhost:3000
. Once on the main home page, you can navigate to the ‘Search Example’ page, and input a search term, such as ‘MEAN Stack’. All the instructions on the MEAN.io Github site, apply to this project.
The Project’s Architecture
According to MEAN.io, everything in mean.io is a ‘package’. When extending mean with custom functionality, you create a new ‘package’. In this case, I have created a ‘search’ package, with the command above, ‘mean package search
‘. Below is the basic file structure of the ‘search
‘ package, within the overall MEAN.io project framework. The ‘public
‘ folder contains all the client-side, AngularJS code. The ‘server
‘ folder contains all the server-side, Node.js/Express/Request code. Note that each ‘package’ also has its own ‘package.json
‘ npm file and ‘bower.json
‘ Bower file.
The simple, high-level sequence diagram below shows the flow of the custom search request from the ‘Search Example’ view to the Google Custom Search API. The diagram also shows the response from the Google Custom Search API all the way back up the MEAN stack to the client-side view.
Client-Side Request/Response
If you view the network traffic in your web browser, you will see a RESTful URI call is made between AngularJS’ service factory, on the client-side, and Node.js with Express, on the server-side. The RESTful endpoint, called with $http.jsonp()
, will be similar to: http://localhost:3000/customsearch/MEAN.io/10?callback=angular.callbacks._0
. In actuality, the callback parameter name, the AngularJS service factory, is ‘JSON_CALLBACK
‘. This is replaced by AngularJS with an incremented ‘angular.callbacks._X
‘ parameter name, making the response callback name incremental and unique.
The response returned to AngularJS from Node.js is a sub-set of full response from Google’s Custom Search API. Only the search results items and a ‘200’ status code are returned to AngularJS as JavaScript, JSONP wrapped in a callback. Below is a sample response, truncated to just a single search result. I have highlighted the four fields that are displayed in the ‘Search Example’ view, using AngularJS’ ng-repeat
directive.
/**/ typeof angular.callbacks._0 === 'function' && angular.callbacks._0({ "statusCode": 200, "items" : [{ "kind" : "customsearch#result", "title" : "MEAN.IO - MongoDB, Express, Angularjs Node.js powered fullstack ...", "htmlTitle" : "<b>MEAN</b>.<b>IO</b> - MongoDB, Express, Angularjs Node.js powered fullstack <b>...</b>", "link" : "http://mean.io/", "displayLink" : "mean.io", "snippet" : "MEAN - MongoDB, ExpressJS, AngularJS, NodeJS. based fullstack js framework.", "htmlSnippet" : "<b>MEAN</b> - MongoDB, ExpressJS, AngularJS, NodeJS. based fullstack js framework.", "cacheId" : "_CZQNNP6VMEJ", "formattedUrl" : "mean.io/", "htmlFormattedUrl": "<b>mean</b>.<b>io</b>/", "pagemap" : { "cse_image" : [{"src": "http://i.ytimg.com/vi/oUtWtSF_VNY/hqdefault.jpg"}], "cse_thumbnail": [{ "width" : "259", "height": "194", "src" : "https://encrypted-tbn2.gstatic.com/images?q=tbn:ANd9GcSIVwPo7OcW9u_b3P3DGxv8M7rKifGZITi1Bhmpy10_I2tlUqjRUVVUBKNG" }], "metatags" : [{ "viewport" : "width=1024", "fb:app_id" : "APP_ID", "og:title" : "MEAN.IO - MongoDB, Express, Angularjs Node.js powered fullstack web framework - MEAN.IO - MongoDB, Express, Angularjs Node.js powered fullstack web framework", "og:description": "MEAN MongoDB, ExpressJS, AngularJS, NodeJS.", "og:type" : "website", "og:url" : "APP_URL", "og:image" : "APP_LOGO", "og:site_name" : "MEAN.IO", "fb:admins" : "APP_ADMIN" }] } }] });
Server-Side Request/Response
On the server-side, Node.js with Express and Request, calls the Google Custom Search API via a RESTful URI. The RESTful URI, called with request.get()
, will be similar to: https://www.googleapis.com/customsearch/v1?cx=ed026i714398r53510g2ja1ru6741h:73&q=MEAN.io&num=10&key=jtHeNjIAtSa1NaWJzmVvBC7qoubrRSyIAmVJjpQu
. Note the URI contains both the your CSE ID and API key (not my real ones, of course). The JSON response from Google’s Custom Search API has other data, which is not necessary to display the results.
Shown below is a sample response with a single search result. Like the URI above, the response from Google has your Custom Search Engine ID. Your CSE ID and API key should both be considered confidential and not visible to the client. The CSE ID could be easily intercepted in both the URI and the response object, and used without your authorization. Google has a page that suggests methods to keep your keys secure.
{ kind: "customsearch#search", url: { type: "application/json", template: "https://www.googleapis.com/customsearch/v1?q={searchTerms}&num={count?}&start={startIndex?}&lr={language?}&safe={safe?}&cx={cx?}&cref={cref?}&sort={sort?}&filter={filter?}&gl={gl?}&cr={cr?}&googlehost={googleHost?}&c2coff={disableCnTwTranslation?}&hq={hq?}&hl={hl?}&siteSearch={siteSearch?}&siteSearchFilter={siteSearchFilter?}&exactTerms={exactTerms?}&excludeTerms={excludeTerms?}&linkSite={linkSite?}&orTerms={orTerms?}&relatedSite={relatedSite?}&dateRestrict={dateRestrict?}&lowRange={lowRange?}&highRange={highRange?}&searchType={searchType}&fileType={fileType?}&rights={rights?}&imgSize={imgSize?}&imgType={imgType?}&imgColorType={imgColorType?}&imgDominantColor={imgDominantColor?}&alt=json" }, queries: { nextPage: [ { title: "Google Custom Search - MEAN.io", totalResults: "12100000", searchTerms: "MEAN.io", count: 10, startIndex: 11, inputEncoding: "utf8", outputEncoding: "utf8", safe: "off", cx: "ed026i714398r53510g2ja1ru6741h:73" } ], request: [ { title: "Google Custom Search - MEAN.io", totalResults: "12100000", searchTerms: "MEAN.io", count: 10, startIndex: 1, inputEncoding: "utf8", outputEncoding: "utf8", safe: "off", cx: "ed026i714398r53510g2ja1ru6741h:73" } ] }, context: { title: "my_search_engine" }, searchInformation: { searchTime: 0.237431, formattedSearchTime: "0.24", totalResults: "12100000", formattedTotalResults: "12,100,000" }, items: [ { kind: "customsearch#result", title: "MEAN.IO - MongoDB, Express, Angularjs Node.js powered fullstack ...", htmlTitle: "<b>MEAN</b>.<b>IO</b> - MongoDB, Express, Angularjs Node.js powered fullstack <b>...</b>", link: "http://mean.io/", displayLink: "mean.io", snippet: "MEAN - MongoDB, ExpressJS, AngularJS, NodeJS. based fullstack js framework.", htmlSnippet: "<b>MEAN</b> - MongoDB, ExpressJS, AngularJS, NodeJS. based fullstack js framework.", cacheId: "_CZQNNP6VMEJ", formattedUrl: "mean.io/", htmlFormattedUrl: "<b>mean</b>.<b>io</b>/", pagemap: { cse_image: [ { src: "http://i.ytimg.com/vi/oUtWtSF_VNY/mqdefault.jpg" } ], cse_thumbnail: [ { width: "256", height: "144", src: "https://encrypted-tbn1.gstatic.com/images?q=tbn:ANd9GcTXm3rYwGdWs9Cx3s5VvooATKlgtrVZoP83hxfAOjGvsRMqLpMKuycVl_sF" } ], metatags: [ { viewport: "width=1024", fb:app_id: "APP_ID", og:title: "MEAN.IO - MongoDB, Express, Angularjs Node.js powered fullstack web framework - MEAN.IO - MongoDB, Express, Angularjs Node.js powered fullstack web framework", og:description: "MEAN MongoDB, ExpressJS, AngularJS, NodeJS.", og:type: "website", og:url: "APP_URL", og:image: "APP_LOGO", og:site_name: "MEAN.IO", fb:admins: "APP_ADMIN" } ] } } ] }
The best way to understand the project’s sample code is to clone the GitHub repo, and explore the files directly associated with the search, starting in the ‘packages/custom/search
‘ subdirectory.
Helpful Links
Learn REST: A RESTful Tutorial
Using an AngularJS Factory to Interact with a RESTful Service
Google APIs Client Library for JavaScript (Beta)
REST-ful URI design
Creating a CRUD App in Minutes with Angular’s $resource