At e.GO Mobile we need to access a lot of data that is stored on external systems.

For example, this can be translations from SQL databases or external API services.

Since these data are frequently and each time only in small quantities requested, most overhead occurs and the source systems are heavily used as a result.

To save costs while still ensuring a reliable access to external data, I would like to present the “Data Fetcher” concept that I had integrated into our self-developed Redis module some time ago.

About the module

Our Redis module is a simplification of the official redis module at NPM and uses it under the hood.

It provides methods for the most common CRUD operations:

import RedisCache from "@egomobile/redis";

// the credentials to connect to the
// server come from REDIS_* environment
// variables
const cache = new RedisCache();

await cache.flush(); // remove all entries

// non existing values
await cache.get("foo"); // (undefined)
await cache.get("foo", "<custom-default-value>"); // "<custom-default-value>"

await cache.set("foo", "bar"); // set "bar" value to "foo" key
await cache.get("foo", "<custom-default-value>"); // "bar"

await cache.set("foo", null); // remove value
// alternative: await cache.set("foo", undefined)
await cache.get("foo", "<custom-default-value>"); // "<custom-default-value>"

Data Fetchers

Beside that, you can define higher ordered functions, which we call “Data Fetchers”:

// ...
import axios from "axios";  // npm i axios

// [3] loadRandomUsers: () => Record<string, any>
const loadRandomUsers = cache.createFetcher(
  // [2] the name of the key where to store in
  // underlying `RedisCache` instance in `cache`
  "randomUsersKey",

  // [1] the function to wrap
  async () => {
    const response = await axios.get<Record<string, any>>(
        'https://randomuser.me/api/'
    );

    return response.data;
  }
);

const data1 = await loadRandomUsers("foobar1");

// ...

What happens is that the createFetcher() method builds a wrapped version of the function in [1] with the same structure (from TypeScript perspective of course).

If the very first call is successful, it saves the result of the function as randomUsersKey ([2]) in the underlying Redis server. This result is then saved for a specific time.

While this time loadRandomUsers() ([3]) will return this value again and again without executing the function defined in [1].

Only after the cached value in [2] gets invalid, the fetcher tries to reinvoke the original function until it has an up-to-date value which it can save it to randomUsersKey. While the fetcher attempts to obtain a current value, errors are ignored, so that the application does not crash and the previously cached value remains the current value until the process of application has been closed.

Conclusion

Hereby it becomes apparent that you can prevent unnecessary redundant servers from being set up with relatively simple tools.

This is particularly significant for many small requests, such as translation texts made of many small pieces that must be frequently requested.

Have fun while trying it out! 🎉