A possible strategy for cache purging

[FSaldanha]

It's public knowledge that Next.js does not have a native feature for invalidating cache of static, SSG, ISG or ISR pages on demand, which could be used with an API for triggering content update without having to rebuild the application, although maintainers had already demonstrated interest in implementing it (see here and here).

Many people are asking for a solution or workaround for this (e.g. #21903 and #16488).

Until an official implementation, I'd like to suggest setting a custom server to ask Next.js to delete prerendering files and purging LRU Cache for a specific URL, by just appending the parameter purge=1 to it. This appears to preserve Automatic Static Optimization, even though documentation states the opposite.

There's a serious caveat, though: it cannot be deployed to Vercel.

The code is below and also published in this repo. I'd appreciate any contributions or ideas about this proposal.

// server.js

const { createServer } = require('http');
const next = require('next');
const { promises } = require('fs');

const dev = process.env.NODE_ENV !== 'production';
const app = next({ dev });
const handle = app.getRequestHandler();

const purgeData = async (pathname) => {
  const fullPathname = `.next/server/pages${pathname}`;
  const fullPathHTML = `${fullPathname}.html`;
  const fullPathJSON = `${fullPathname}.json`;
  try {
    await promises.unlink(fullPathHTML);
    await promises.unlink(fullPathJSON);
    const cachedData = await app.incrementalCache.get(pathname);
    const staleTime = new Date().getTime() - 1000;
    app.incrementalCache.set(pathname, {
      ...cachedData,
      revalidateAfter: staleTime
    }, 1);
    console.log(`Cache of ${fullPathname} was successfully purged`);
  } catch (err) {
    console.error(`Could not purge cache of ${fullPathname} - ${err}`);
  }
}

app.prepare().then(() => {
  createServer((req, res) => {
    const url = new URL(req.url, "http://localhost:3000/");
    if (url.searchParams.get('purge') == '1') {
      purgeData(url.pathname);
    }
    handle(req, res);
  }).listen(3000, (err) => {
    if (err) throw err;
    console.log(`> Ready on http://localhost:3000/`);
  })
})

[samuelgoldenbaum]

Have you tested the code above and it works?

[FSaldanha]

Yes. I tested it locally. You may need to adjust fullPathname if your configurations are not default.

[samuelgoldenbaum]

I cloned your repo and seems to be working. I created a TS project that does not use a custom server and tested having an API route handle the purge.. However, the incrementalCache.cache is always empty when accessing it like this. The ISG works as expected, yet the entry never seems to be in the cache after getting the data and rendering the page.

Any ideas?

[FSaldanha]

Very interesting! I saw you're using the next-connect package, but I have no experience with API middlewares in Next.js. Maybe it is starting a new instance of server on this line and that is why cache is always empty?

[samuelgoldenbaum]

You 100% correct - this approach is creating a new server so obviously will not work. I need to get access to the next instance via the API routes. Need to see how to do this.

[hmadej]

@samuelgoldenbaum have you had any luck accessing the next instance via API routes? I am attempting to do the same.

[lucaspbordignon]

Thanks a lot for the code snippet @FSaldanha!

This feature is definitively something that would help.

Also, another alternative is to delete the entry from the LRUCache. Inspired by #22699:

try {
    await promises.unlink(fullPathHTML);
    await promises.unlink(fullPathJSON);
    const cachedData = await app.incrementalCache.get(pathname);
    const staleTime = new Date().getTime() - 1000;
    
    /* Delete the entry in cache */
    app.incrementalCache.cache.del(pathname);
    
    console.log(`Cache of ${fullPathname} was successfully purged`);
  } catch (err) {
    console.error(`Could not purge cache of ${fullPathname} - ${err}`);
  }

[FSaldanha]

Great call! I've tried app.incrementalCache.del before and it didn't worked, since Next.js incremental cache does not provide a native function to delete cache data (although it does provide get and set functions). I missed this "extra" cache, which would allow me to access LRUCache directly.

On the other hand, maybe forcefully making data stale, but not actually deleting them, could improve UX in case of database unavailability right at the time of purging, for example.

[overdev-l]

This doesn't seem to work in next12, do you have any suggestions?

[lucaspbordignon]

Hey @overdev-l

Don't really recall what used to be available in Next@12, but maybe related to this comment?

[overdev-l]

In next12 server is already a private member, thanks a lot for the reply!

[nachos5]

something like this is really needed when deploying on Vercel

[mattiasw]

In Next.js 10.2 app.incrementalCache has moved to app.server.incrementalCache.

[ialapont451]

I tested it locally with express and works for me.

In this way, it updates the page on the next visit.

server.js

const express = require('express')
const next = require('next')

const port = parseInt(process.env.PORT, 10) || 3000
const dev = process.env.NODE_ENV !== 'production'
const app = next({ dev })
const handle = app.getRequestHandler()
const { promises } = require('fs');

app.prepare().then(() => {
  const server = express()

  server.all('*', (req, res) => {
    const url = new URL(req.url, "http://localhost:3000/");
    if (url.pathname.includes("purge-data")) {
      purgeData(url.pathname.replace("purge-data/", ""), res);
      res.sendStatus(200);
      return res.end();
    }
    return handle(req, res)
  })

  server.listen(port, (err) => {
    if (err) throw err
    console.log(`> Ready on http://localhost:${port}`)
  })

})


const purgeData = async (pathname, res) => {
  const fullPathname = `.next/server/pages${pathname}`;
  const fullPathHTML = `${fullPathname}.html`;
  const fullPathJSON = `${fullPathname}.json`;
  try {
    const cachedData = await app.server.incrementalCache.get(pathname);
    const staleTime = new Date().getTime() - 1000;
    app.server.incrementalCache.del(pathname); //<--- add method 'del' in incrementalCache
    await promises.unlink(fullPathHTML);
    await promises.unlink(fullPathJSON);

    console.log(`Cache of ${fullPathname} was successfully purged`);


  } catch (err) {
    console.error(`Could not purge cache of ${fullPathname} - ${err}`);
  }
}

in node_modules\next\dist\next-server\server\incremental-cache.js

 async del(pathname, data, revalidateSeconds) {
        if (this.incrementalOptions.dev) return;
        if (typeof revalidateSeconds !== "undefined") {
            // TODO: Update this to not mutate the manifest from the
            // build.
            this.prerenderManifest.routes[pathname] = {
                dataRoute: _path.default.posix.join("/_next/data", `${(0, _normalizePagePath.normalizePagePath)(pathname)}.json`),
                srcRoute: null, // FIXME: provide actual source route, however, when dynamically appending it doesn't really matter
                initialRevalidateSeconds: revalidateSeconds,
            };
        }
        pathname = (0, _normalizePagePath.normalizePagePath)(pathname);
        this.cache.del(pathname); // TODO: This option needs to cease to exist unless it stops mutating the
    }
```
`

[ratnabh-classic]

@FSaldanha one thing to ask, the solution you provided has a path to the ./next folder in your nextjs project, what i currently have is my entire backend in different space (let's say hosted on AWS), so is there a way i could run this solution in main file of backend (which is already hosted on AWS) and provide a path to ./next folder anyhow? or i should add this particular solution to server.js file inside next project ??
P.S - already saw many people saying to create an entirely diff. space for backend so this was the reason for the seperation of my backend from the nextjs project

[FSaldanha]

I'm not used to AWS so I'm afraid I can't help you with the most accurate answer. That said, I think as long as you are not using a serverless solution and you have not set a custom build directory, this code will work as is once it is pasted into your custom server file.

(Edit: for the sake of clarity, if your project is serverless, don't bother with files inside the .next folder, because Next.js won't be able to mess with them anyway. Instead, go ahead with cache purge only.)

[alexandrosk]

I believe you can setup a nodejs server to run on top of the nextjs app
Check this I will try also and let you know
https://nextjs.org/docs/advanced-features/custom-server

[ianhogersltn]

When using the above code, I get the following error:

Cache of .next/server/pages/blog/page-slug was successfully purged
(node:4063) UnhandledPromiseRejectionWarning: TypeError: Cannot read property 'length' of undefined
    at LRUCache.length (/Users/.../project/node_modules/next/dist/server/incremental-cache.js:50:39)
    at LRUCache.set (/Users/.../project/node_modules/next/dist/compiled/lru-cache/index.js:1:2155)
    at IncrementalCache.set (/Users/.../project/node_modules/next/dist/server/incremental-cache.js:135:24)
    at purgeData (/Users/.../project/server.js:19:33)
(node:4063) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 4)

I'm having some trouble debugging it. for me it's also not rebuilding the file but just serving the cached file instead.

[FSaldanha]

Not sure what's going on... Could you put your code on Gitpod or similar and share with us?

[ianhogersltn]

HI @FSaldanha Unfortunately not. I'll see If i can reproduce it on a clean build.

[ianhogersltn]

Definitely something to do with my build. works perfectly on a clean build. Any suggestions what to look for?

[ianhogersltn]

I've found the following:
when IncrementalCache is getting constructed it's trying to return value.html.length but can't find html on value for some reason, right after purging the value object structure becomes: value.value.html...

I managed to solve it by adding .value to ...cachedData just on a hunch.

so final file for me was:

// server.js
const { createServer } = require('http');
const next = require('next');
const { promises } = require('fs');

const dev = process.env.NODE_ENV !== 'production';
const app = next({ dev });
const handle = app.getRequestHandler();

const purgeData = async (pathname) => {
  const fullPathname = `.next/server/pages${pathname}`;
  const fullPathHTML = `${fullPathname}.html`;
  const fullPathJSON = `${fullPathname}.json`;
  try {
    await promises.unlink(fullPathHTML);
    await promises.unlink(fullPathJSON);
    const cachedData = await app.server.incrementalCache.get(pathname);
    const staleTime = new Date().getTime() - 1000;
    app.server.incrementalCache.set(
      pathname,
      {
        ...cachedData.value,
        revalidateAfter: staleTime,
      },
      1
    );
    console.log(`Cache of ${fullPathname} was successfully purged`);
  } catch (err) {
    console.error(`Could not purge cache of ${fullPathname} - ${err}`);
  }
};

app.prepare().then(() => {
  createServer((req, res) => {
    const url = new URL(req.url, 'http://localhost:3000/');
    if (url.searchParams.get('purge') === '1') {
      purgeData(url.pathname);
    }
    handle(req, res);
  }).listen(3000, (err) => {
    if (err) throw err;
    console.log('> Ready on http://localhost:3000/');
  });
});

[robd91]

I also agree with @FSaldanha that there is a need for this feature. The possibility to clear a cache with the path (e.g. /api/data/:id/) manually would give you a more granular control to invalidate caches.

for example let´s say you have a resource that gives you data e.g. GET /api/data/:id and returns some sort of JSON object, and you mutate changes to this resource with separate actions like e.g. POST /api/data/:id/change-name. Once this action is called and you have mutated the data, it would come in handy to invalidate the cache for /api/data/:id only in this case (after the POST was completed). This way you could utilize long cache times, e.g. 1 day, 1 month, ... and only invalidate it, when something has really changed. Also you could present new data faster to the user this way, instead of waiting for the cache to clear every X time interval.

[eric-burel]

Hi, I've thought of a possible alternative, based on route params:

• add a timestamp route params to your route
• add a middleware, that get the latest timestamp for your page (for instance, latest update of your blog article)
• rewrite requests with the right timestamp

Timestamp changes will retrigger a render, as if it was a new article in the example of a blog. Which makes sense, as it is a new version of the article.

Pros:

• the timestamp param is not visible, so you have the same UX but now you can invalidate pages
• no custom server

Downsides:

• it costs a call to the db for each request, you need to make it fast, eg introducing a Redis cache into your architecture
• it bloats memory a bit more, because the old version of the article still exists until it's purged based on it's time to live (not sure how Next handles this)
• I haven't tested it :)

[styxlab]

This is a great idea and should fix the problem of many unnecessary revalidations (and associated costs). It could also be extended to a time-machine where you could go back to previous versions of the page. I will definitely try it out.

However, one additional ISR problem is not solved by this strategy: every re-deploy of your site automatically purges all pages which means they all have to be re-build.

[eric-burel]

Your second point seems like a different issue: you want to warn start your ISR pages based on a past render, as far as I understand. That would definitely be worth a feature request! I think it might be done by somehow preloading certain cache file from .next in your new build, based on your previous build, for pages that uses ISR.

[meotimdihia]

My website has millions of pages. And I dunno how to clear cache when my pages are changed.
And It looks like Next.js developers are not subscribed to this discussion.

[PsyGik]

Without any details about implementation or reproducible code, it's very difficult for people to help, dear Internet stranger! 🙃

[meotimdihia]

Without any details about implementation or reproducible code, it's very difficult for people to help, dear Internet stranger! 🙃

What do you mean by reproducible code?
A way to clear the cache of pages is not new, it has on all PHP frameworks from my experience. 😶
Edit: Sorry, if this topic is for the custom server only then I was mistaken.

[eric-burel]

See #11552 (comment)

[meotimdihia]

See #11552 (comment)

Thanks, we still have to wait for this feature then. 😅

[leerob]

I'm incredibly excited to share that Next.js 12.1 was just released, which includes on-demand Incremental Static Regeneration! We know many of you have been looking forward to this for a while, including myself, and I'm so happy it is here.

• Learn about Next.js 12.1
• Watch a demo of on-demand ISR
• View another demo and give us your feedback
• Read the documentation

[pranavmalvawala]

Did you find a way @gijsbotje?

[gijsbotje]

@pranavmalvawala unfortunately no. No response on this issue means I hsve not found or got a reply with an answer. We just use a low max-age but stretch the revalidate setting reaaly long.

[umashankar-pardhi-amla]

Is there a way to revalidate the specific pages, considering that @leerob and @steven-tey are facing the same issue?

[leerob]

Update: We have some new improvements here, including revalidateTag and revalidatePath!

https://nextjs.org/docs/app/building-your-application/data-fetching/revalidating

[Yf102]

@leerob I have been struggling for days now to revalidate imported translation.json files but with absolute no success. Would really appreciate if you could give me some advice. What we are using for nextjs 14 for client translations is the following client.ts

import { getOptions, languages } from '@/app/_i18n/settings.tsx'
import i18next, { KeyPrefix, TranslationsArr, TranslationsType } from 'i18next'
import resourcesToBackend from 'i18next-resources-to-backend'
import { useEffect, useState } from 'react'
import {
  FallbackNs,
  UseTranslationOptions,
  UseTranslationResponse,
  initReactI18next,
  useTranslation as useTranslationOrg,
} from 'react-i18next'

const runsOnServerSide: boolean = typeof window === 'undefined'

//
i18next
  .use(initReactI18next)
  .use(
    resourcesToBackend(
      (language: string, namespace: string) =>
        import(`./locales/${language}/${namespace}.json`),
    ),
  )
  .init({
    ...getOptions(),
    lng: undefined, // let detect the language on client side
    detection: {
      order: ['path', 'htmlTag', 'cookie', 'navigator'],
    },
    preload: runsOnServerSide ? languages : [],
    interpolation: {
      escapeValue: false, // react already safes from xss
    },
  })

export function useTranslation<
  Ns extends TranslationsArr | TranslationsType,
  KPrefix extends KeyPrefix<FallbackNs<Ns>> = undefined,
>(
  ns?: Ns,
  options?: UseTranslationOptions<KPrefix>,
): UseTranslationResponse<FallbackNs<Ns>, KPrefix> {
  const lng = options?.lng || 'en-CA'
  const ret = useTranslationOrg(ns, options)
  const { i18n } = ret
  if (runsOnServerSide && lng && i18n.resolvedLanguage !== lng) {
    i18n.changeLanguage(lng)
  } else {
    // eslint-disable-next-line react-hooks/rules-of-hooks
    const [activeLng, setActiveLng] = useState<string>(
      i18n.resolvedLanguage || 'en-CA',
    )
    // eslint-disable-next-line react-hooks/rules-of-hooks
    useEffect(() => {
      if (activeLng === i18n.resolvedLanguage) return
      setActiveLng(i18n.resolvedLanguage || 'en-CA')
    }, [activeLng, i18n.resolvedLanguage])
    // eslint-disable-next-line react-hooks/rules-of-hooks
    useEffect(() => {
      if (!lng || i18n.resolvedLanguage === lng) return
      i18n.changeLanguage(lng)
    }, [lng, i18n])
  }
  return ret
}

During build of course it is great, but during run time we have a revalidate api call that updates the translations with custom js script and revalidate page with revalidatePath it seems that the import(./locales/${language}/${namespace}.json) is always getting the cached version of the json files and never gets the new information.

Thank you in advance,

[thedv91]

Is there any update on this issue? I don't really wanna have to validate multiple pages. I only want to delete those pages and validate them when the client sends a request.

[dev-ronak-patel]

Anyone? i have folder size of .next/server/pages in GBs.