Things you might want to consider when adding caching to your code

Eviction strategies should be fine-tuned considering your needs and environment.

You should not “just” cache values.

The issue with concurrency

To explain you this, we will just use a basic example written in javascript.

We will call every 250ms a function that use a basic cache strategy and takes about 1s to complete. We will stop at 5 iterations.

/*To simulate a process needing a few seconds to complete */function sleep(ms) {  return new Promise(resolve => setTimeout(resolve, ms));}// basic cache with no key, only a valueconst basicCache = {    current: undefined}async function processingFunction(i) {    if (basicCache.current) {        console.log(`${i}: Cache HIT!`)        return basicCache.current;    }    console.log(`${i}: Cache MISS!`)    // 1s    await sleep(1000)    basicCache.current = "Hello World";    console.log(`${i}: Cache SET!`)    return basicCache.current;}for (let i = 0; i < 5; i++) {    // we do not await here to demonstrate concurrent operations    processingFunction(i);    await sleep(250);}

Let’s see the result.

0: Cache MISS!1: Cache MISS!2: Cache MISS!3: Cache MISS!0: Cache SET!4: Cache HIT!1: Cache SET!2: Cache SET!3: Cache SET!

Of the 5 calls, only one had a cache hit. The first four calls all completed the entire process before setting the value in the cache.

It is because this example does not prevent the processing from being performed multiple times at the same time! The cache will always be missing until the result of the first call is known!

Illustration by the author: value-based caching does not prevent the same processing from being made multiple times.

Fixing concurrency issues

In JavaScript a simple way to fix it is to store a promise instead of a value in the cache.

/*To simulate a process needing a few seconds to complete */function sleep(ms) {  return new Promise(resolve => setTimeout(resolve, ms));}// basic cache with no key, only a valueconst basicCache = {    current: undefined}async function processingFunction(i) {    if (basicCache.current) {        console.log(`${i}: Cache HIT!`)        return basicCache.current;    }    console.log(`${i}: Cache MISS!`)    basicCache.current = new Promise((resolve) => {        // will resolve after 1s        setTimeout(() => {            resolve(`Hello World ${i}`);        }, 1000)    });    return basicCache.current;}for (let i = 0; i < 5; i++) {    // we do not await here to demonstrate concurrent operations    processingFunction(i).then(value => console.log(`> ${i}: ${value}`));    await sleep(250);}

And the result:

0: Cache MISS!1: Cache HIT!2: Cache HIT!3: Cache HIT!> 0: Hello World 0> 1: Hello World 0> 2: Hello World 0> 3: Hello World 04: Cache HIT!> 4: Hello World 0

This time only the first call miss the cache:

• the four first call are resolved at the same time,
• the fifth one resolve directly.

This is because creating a promise does not take times, and so all five calls are using the same promise in the end: the processing was only performed once!

Illustration by the author: promise-based caching took less time to handle the fifth calls and the processing was done only once.

---

You can also use a mutex/locking strategy where you:

• start by checking if the value is in the cache
• if not, acquire a lock (to prevent a piece of code from being called multiple times at the same time)
• when you have the lock, check a second time if the value was not added in the cache in the meantime
• yes? => you release the lock and return the value
• no? => you do your processing, store the value in the cache and then release the lock

It is often better not to implement caching directly in your processing function/methods.

For this part we will write a class with a single “process” method.

We will then create a second class with the same interface that will be responsible for handling the cache and calling the first class if it is a miss.

/*To simulate a process needing a few seconds to complete */function sleep(ms) {  return new Promise(resolve => setTimeout(resolve, ms));}/** * Basic in-memory cache implementation, * with no eviction strategy included */class BasicCache {    constructor() {        this._content = {};    }    has(key) {        return this._content.hasOwnProperty(key);    }    get(key) {        return this._content[key];    }    set(key, value) {        this._content[key] = value;    }}class ProcessingClass {    async process(key, i) {        await sleep(1000);        return `Hello ${key} ${i}`;    }}class ProcessingCacheClass {    constructor(cache, proxy) {        this._proxy = proxy;        this._cache = cache;    }    // the same signature as in ProcessingClass    // key is a "real" parameter while i is given only for the demonstration    process(key, i) {        // takes the value from the cache if present ...        if (this._cache.has(key)) {            console.log(`${i}: Cache HIT!`)            return this._cache.get(key);        }        // else launch the process and store it in the cache        console.log(`${i}: Cache MISS!`)        // do not await !!!!        const processPromise = this._proxy.process(key, i);        this._cache.set(key, processPromise);        // return the promise (not the value!!)        return processPromise;    }}const processingInstance = new ProcessingCacheClass(    new BasicCache(),    new ProcessingClass());for (let i = 0; i < 5; i++) {    // we do not await here to demonstrate concurrent operations    processingInstance.process('World', i).then(value => console.log(`> ${i}: ${value}`));    await sleep(250);}

The result:

0: Cache MISS!1: Cache HIT!2: Cache HIT!3: Cache HIT!> 0: Hello World 0> 1: Hello World 0> 2: Hello World 0> 3: Hello World 04: Cache HIT!> 4: Hello World 0

Using this kind of approach you can easily:

• change your caching strategy (just pass your caching mechanism in the constructor of the caching implementation),
• chain multiple caching strategies together, ie:
• 1. first level: in memory,
• 2. second level: in a database
• …

---

That’s all folks!