Using page tokens to retrieve extended results from Google APIs

Typically, Google APIs return between 20 and 100 results from API calls. This is done to minimize the amount of unnecessary data is sent to API clients which improves client performance. However, sometimes you want to get additional results. What do you do if you want to get additional results?

The quick answer is that to get additional results, you must pass a special variable called a page token. The page token represents the position in the result set you are looking at, like a bookmark in a book, and is returned when additional results are available.

In this blog post, I will give you a very basic reference for how to use page tokens to seek deeper into query results.

A basic sample – Url to ID

To demonstrate getting additional results, I have created a demo app, URL to ID that, given a Google+ URL, will try and determine the post associated with the URL. You can see the demo Google+ URL to Post ID app here.

The app works as follows:

  • Parse the URL parts to determine the Google+ use associated with the post
  • Search the user’s post until you find the post URL
  • Return the post URL when the post is found

Because the post you are searching for could be further back in a user’s history than 20-100 posts, you must continue searching through additional pages of results until you find the matching URL.

Using the JavaScript Google API client library

The easiest and most reliable way to use Google APIs from JavaScript is to use the Google API client library. To initialize the client library, you must include the sources from Google:

<script src="https://apis.google.com/js/client.js?onload=handleClientLoad"></script>

Because I’m specifying handleClientLoad in the onload parameter, the client library will call the handleClientLoad function when the script has been initialized. The following function, called after the script initializes, constructs the client library from the JSON discovery document for the Google+ API and sets the simple API key internally within the client library.

  /**
   * Handles loading the API clients from endpoints.
   */
  function handleClientLoad(){
    console.log('Loading plus api...');
    gapi.client.setApiKey(key);
    gapi.client.load('plus', 'v1', function(){console.log('...done loading plus.')});
  }

Something very important to note on the callback function triggered by the client.js script: it must be defined before the script loads. Always put the script include after you have defined your callback function. Now that you have loaded the client library and initialized it with the Google+ discovery document, you will be able to access Google+ APIs through the gapi.client.plus.* APIs. Worth noting is that you can load ANY of Google’s APIs in this manner. For example, you could load the Google Drive v2 API as follows:

gapi.client.load('drive', 'v2');

At this point, we’re ready to query Google+ for the relevant social activities.

Querying Google APIs – the basics

At a basic level, the relevant Google API is called as follows:

    gapi.client.plus.activities.list({userId: userId}).execute(function(resp){
          // Perform operations here...
        });

In  the API call, the user ID, parsed from the input URL, is passed to the backend server via the Google RESTful endpoints on Google servers in the API method activities.list. You can explore the API call and resulting data using the Google API explorer and searching for the relevant Google+ API. The execute method accepts a function that is called when the API query finishes. When exploring the API and testing results from my own apps, I do the following:

    gapi.client.plus.activities.list({userId: userId}).execute(function(resp){
          console.log(resp);
        });

This way, I can see the data returned from the API call without requiring the reference every time. What’s even more convenient is using the developer console to trigger API calls and checking results within the console while you develop.

Traversing results – the actual implementation

Now that you’ve seen the basics of using the Google APIs from the JavaScript client library, let’s take a look at the API call in the code itself. The following code is how I’m querying the Google API:

  /**
   * Performs the API queries for searching,
   *
   * @param {String} searchUrl The Url containing the user ID for searching.
   * @param {int} queryCount The number of API calls made.
   * @param {String} nextPageToken The next page token for paged api calls.
   */
  function searchForUrl(searchUrl, queryCount, nextPageToken){
    var userId = searchUrl.split('/')[3];

    gapi.client.plus.activities.list({userId: userId,
        pageToken: nextPageToken}).execute( function(resp){
          handleActivities(resp.items, searchUrl, queryCount, resp.nextPageToken);
        });
  }

In the above code, an additional parameter to those passed in the preceding examples, the page token, is passed to the Google API endpoint. This page token, when null, returns the first set of data. When the page token is present, the specified page of data is returned. Let’s look at the code for handling the result set:

  /**
   * Parses and stores activities from the XMLHttpRequest
   *
   * @param {Object} activities The response activity objects as an array.
   * @param {String} postUrl The URL of the post.
   * @param {int} queryCount The number of API calls.
   * @param {string} nextPageToken The next page token.
   */
  function handleActivities(activities, postUrl, queryCount, nextPageToken){
    for (var activity in activities){
      activity = activities[activity];
      if (activity['url'] == postUrl){
        targetActivity = activity;
        document.getElementById('result').value = 'ID is: ' +
            activity.id + '\n' +
            'PlusCount is: ' + activity.object.plusoners.totalItems + '\n' +
            'Replies are: ' + activity.object.replies.totalItems + '\n' +
            'Reshares are: ' + activity.object.resharers.totalItems + '\n';
        isFound = true;
      }else{
        console.log(activity);
      }
    }

    if (queryCount < maxQueryCount && !isFound){
      queryCount++;

      // throttle calls using timer to avoid reaching query limit
      console.log('retrying with ' + nextPageToken);
      setTimeout(searchForUrl(postUrl, queryCount, nextPageToken), 100);
    }
  }

In the above example function, the activities are passed as a JavaScript array of objects. Of note, objects returned from Google API calls are typically returned in the items parameter which was passed to this function in the API callback. Details of the activities object are described in the Google+ REST API documentation. To traverse the result set, the page token is passed back to the same function called before in order to keep searching for the social activity.

Closing Thoughts

When you need to get more results than the initial window of results, use page tokens to retrieve additional results.