Using JSON Mapping to Work with APIs of Various Image Services

In one of my projects, I use the API of various services that provide images.

My apps work with them quite well, but only until the API developers decide to change something in their API specification. In this case, I have to modify my entire app.

I decided it was a good idea to separate the specifications of each service into a separate file so that when any of the APIs are changed, the rework of my app only consists of editing this configuration file.

Any set of images can be represented in the form of a certain universal structure that can be used in many projects.

Let's write an example of this in JSON format:

dataset =
{
  "name": "My animals",
  "desc": "My animals dataset descrtiption",
  "items": [
    {"id": 0,
    "title": "My dog image",
    "thumbnail": "https://loremflickr.com/150/150/dog",
    "src": "https://loremflickr.com/1680/1050/dog"
    },
    {"id": 1,
      "title": "My cat image",
      "thumbnail": "https://loremflickr.com/150/150/cat",
      "src": "https://loremflickr.com/1680/1050/cat"
    },
    {"id": 2,
      "title": "My parrot image",
      "thumbnail": "https://loremflickr.com/150/150/parrot",
      "src": "https://loremflickr.com/1680/1050/parrot"
    }
  ]
}

There is a kind of description of the dataset, and an array containing image elements with their descriptions and links to the sources of images themselves. If necessary, it will be possible to make changes to this structure, but these changes will usually be associated with some changes in our project. When we retrieve data from an image search service using the appropriate API, the results for each service will be different.

For example, when using the Flickr service, the flickr.photos.search method will return results in the following format:

response =
{
    "photos": {
        "page": 1,
        "pages": 89071,
        "perpage": 25,
        "total": 2226767,
        "photo": [
            {
                "id": "12312362",
                "title": "Cat",
                "description": {
                    "_content": "My favorite cat..."
                },
                "datetaken": "2015-06-05 19:23:19",
                "ownername": "User512341234",
                "latitude": "57.142993",
                "longitude": "33.699757",
                "place_id": "uQBm12323UrxpE99h",
                "url_q": "https://live.staticflickr.com/1957/45543214321_abcabcabac_q.jpg",
                "url_l": "https://live.staticflickr.com/1957/45543214321_abcabcabac_b.jpg",
            },
			...
		]
    },
    "stat": "ok"
  }
}

Here the array of image elements is in the response.photos.photo element.

The image url is in response.photos.photo [index].url_l.

The thumbnail URL (image preview) is in response.photos.photo [index].url_q.

And the description will be contained in response.photos.photo [index].description._content.

Using the Pixabay API, the image search response will be as follows:

response =
{
    "total": 23858,
    "totalHits": 500,
    "hits": [
        {
            "id": 736877,
            "pageURL": "https://pixabay.com/photos/tree-cat-736877/",
            "type": "photo",
            "tags": "tree, cat, silhouette",
            "previewURL": "https://cdn.pixabay.com/photo/2015/.../tree-736877_150.jpg",
            "largeImageURL": "https://pixabay.com/get/g07531348f4443b076ebc0567_1280.jpg",
        },
		...
    ]
}

The array of image elements should be taken from the response.hits element.

The image url is in response.hits[index].largeImageURL

The thumbnail URL is in response.hits[index].previewURL

The description will have to be taken from some other elements.

In the examples above, I have omitted some of the fields to make it easier to understand their structure.

To process responses from different services in the same way, we need to convert them to the format of our dataset. First, let's describe the classes that represent our image element and a set of images.

For the image:

class Item {
  constructor(object, schema) {
    this.id = getProperty(object, schema.item_id);
    this.title = getProperty(object, schema.item_title);
    this.desc = getProperty(object, schema.item_desc);
    this.thumbnail = getProperty(object, schema.item_thumbnail);
    this.src = getProperty(object, schema.item_src);
  }
}

For the dataset:

class Dataset {
  constructor(object, schema) {
    this.name = "";
    this.desc = "";
    schema.ds_name.forEach(e => { this.name += getProperty(object, e); });
    schema.ds_desc.forEach(e => { this.desc += getProperty(object, e); });
    this.items = getProperty(object, schema.items).map(
      i => new Item(i, schema)
    );
  }
}

The constructors of these classes create objects whose structure is similar to that of our dataset. In the dataset constructor, we also call the constructor to create the items.

The getProperty function allows you to get the value of an element from any object by specifying the path to it (for example, photos.photo). You can see its code later.

The key point here is that we pass 2 parameters to the constructors: object and schema.

object is the original formatted dataset or image element that the API service returned to us.

schema - an object containing the mapping of object elements to elements of our dataset.

We can put the above code in datasets.js file as a module.

Let's describe the schemas for three image services (Flickr, Pixabay, Pexels):

export const FLICKR_SCHEMA = {
  ds_name: ["My Flickr dataset. Page#: ", "photos.page", ", total pages: ", "photos.total"],
  ds_desc: ["Photos per page: ", "photos.perpage"],
  items: "photos.photo",
  item_id: "id",
  item_title: "title",
  item_desc: "description._content",
  item_thumbnail: "url_q",
  item_src: "url_l"
};

export const PIXABAY_SCHEMA = {
  ds_name: ["My Pixabay dataset. Total: ", "total"],
  ds_desc: ["Total hits: ", "totalHits"],
  items: "hits",
  item_id: "id",
  item_title: "title",
  item_desc: "tags",
  item_thumbnail: "previewURL",
  item_src: "largeImageURL"
};

export const PEXELS_SCHEMA = {
  ds_name: ["My Pexels dataset. Page: ", "page"],
  ds_desc: ["Per page: ", "per_page", " Total results: ", "total_results"],
  items: "photos",
  item_id: "id",
  item_title: "photographer",
  item_desc: "url",
  item_thumbnail: "src.tiny",
  item_src: "src.original"
};

The names of the keys in the schemas correspond to the format of our dataset, and the values tell us from which elements to take for the corresponding keys. As you can see, we refer to different elements depending on the service.

For ds_name and ds_desc, I used arrays, which will be concatenated into a string using forEach in the dataset constructor. Let's save our schemas in the dataschemas.js file.

To test this idea, we will develop a simple web application based on the following logic:

1. Enter search string
2. Select an image search service
3. Click the "Search" button
4. Call the API of the corresponding service and get the result
5. Call the Dataset constructor, into which we transfer the received response in JSON format, as well as the corresponding API schema
6. The constructor returns an object of the format of our dataset
7. Use a function to draw the resulting images into the DOM

Let's sketch out the HTML structure:

<!DOCTYPE html>
<html>
<head>
  <meta charset='utf-8'>
  <meta http-equiv='X-UA-Compatible' content='IE=edge'>
  <title>Image APIs Schemas</title>
  <meta name='viewport' content='width=device-width, initial-scale=1'>
  <link rel='stylesheet' type='text/css' media='screen' href='main.css'>
  <script type="module" src='main.js'></script>
</head>
<body>
  <header>
    <input id="searchText" type="text" placeholder="Input search text" value="skyline">
    <select id="sourceType">
      <option value="flickr">Flickr</option>
      <option value="pixabay">Pixabay</option>
      <option value="pexels">Pexels</option>
    </select>
    <button id="searchImages">Search Images</button></header>
  <main></main>
</body>
</html>

In the HTML file, we will include our main.js script as a module.

The main.css file contains the minimum style settings for the correct display of elements.

Image services use personal API keys, let's put them in a separate file, api_keys.js:

export const FLICKR_API_KEY = "abcdef123456789";
export const PIXABAY_API_KEY = "abcdef123456789";
export const PEXELS_API_KEY = "abcdef123456789";

Anyone can get the keys for free by registering on the corresponding service.

Let's include our JS files in the main.js script and write some code to implement our logic:

import { Dataset } from "./datasets.js";
import * as Schemas from "./dataschemas.js";
import * as Keys from "./api_keys.js";

let myDataset;
let flickrSearchURL = "https://api.flickr.com/services/rest/?method=flickr.photos.search&format=json&nojsoncallback=1&sort=interestingness-desc&per_page=5&page=1&extras=o_dims,url_sq,url_t,url_s,url_q,url_m,url_n,url_z,url_c,url_l,url_o,description,geo,date_upload,date_taken,owner_name&per_page=25";
let pixabaySearchURL = "https://pixabay.com/api/?image_type=photo&per_page=25";
let pexelsSearchURL = "https://api.pexels.com/v1/search?per_page=25";

window.onload = () => {

  document.querySelector("#searchImages").onclick = () => {
    let queryString = document.querySelector("#searchText").value;
    let sourceType = document.querySelector("#sourceType").value;
    let searchUrl, options, schema;

    switch ( sourceType ) {
      case 'flickr':
        searchUrl = `${flickrSearchURL}&api_key=${Keys.FLICKR_API_KEY}&text=${queryString}`;
        schema = Schemas.FLICKR_SCHEMA;
        options = {mode: "cors"};
        break;
      case 'pixabay':
        searchUrl = `${pixabaySearchURL}&key=${Keys.PIXABAY_API_KEY}&q=${queryString}`;
        schema = Schemas.PIXABAY_SCHEMA;
        options = {mode: "cors"};
        break;
      case 'pexels':
        searchUrl = `${pexelsSearchURL}&query=${queryString}`;
        schema = Schemas.PEXELS_SCHEMA;
        options = {headers: {Authorization: `${Keys.PEXELS_API_KEY}` }, mode: "cors"};
        break;
    }

    fetch(searchUrl, options)
    .then( response => response.json())
    .then( json => {
      console.log(json);
      myDataset = new Dataset(json, schema);
      myDataset.draw(document.querySelector("main"));
      console.log(myDataset);
    });
  }
  
}

In the code, we have one onclick event handler on the button, which implements the algorithm described above. Depending on the selected service, we form the corresponding search URL string, call the API, and create a dataset, converted from the result using the appropriate schema.

Now it's time to test our application. Let's install a web server:

npm install http-server –g

To start the web server, we will use the command (you can specify any other port):

http-server -p 3000

This is how the page of our application looks like:

Let's search on Flickr:

By adding the output of the response received from the Flickr service and the resulting dataset to the developer console, we can see how the results were converted to the desired format:

The same for the Pixabay service:

And for the Pexels service:

Thus, to add a data transformation for an additional service, you just need to add a description of the schema of this service to the dataschemas.js file and use this schema when calling the dataset constructor.

If the API specification changes, simply correcting the schema description will be enough.

I would like to note that this approach is applicable not only to image search services but also to any other services that are similar in type of returned results.

I hope that this idea will be useful to someone developing similar applications.

The source code of the project can be taken here:

https://github.com/yuridanil/apimapping