Skip to main content
Izrotech Logo

Server-Rendered HTML for Search and AI Crawlers: What a Loading Screen Hid

A loading boundary hid our pages from crawlers that run no JavaScript. See how to spot it in the raw HTML, fix it in Next.js and test every route for it.

Izrotech

Our pages looked complete in every browser we tried. Then we fetched one with curl and read what came back. Where the article should have been there was a single word, "Loading", and the real content sat further down the document inside a hidden element. A browser repairs that in milliseconds. A program that reads HTML without running scripts never does. This is what caused it on this site and on the eBay Fees Calculator, how we fixed it, and the test that now keeps it fixed.

What the raw HTML looked like

Both sites are built with the Next.js App Router. The initial HTML of every page contained three things in this order:

  • a placeholder where the page content belongs, showing the loading text;
  • the actual content, complete and server-rendered, inside <div hidden id="S:0">;
  • a short inline script that moves the hidden content into the placeholder's position.

The structured data was in the hidden block as well. So was the page's only <h1>.

Why it happened

A file named loading.tsx at the root of the app tells the framework to wrap every page in a Suspense boundary and show that file's output as the fallback. With streaming server rendering, the server may send the fallback first and the finished content afterwards, in the same response, as a hidden block plus the script that swaps it in.

That design is right for a page that waits on something slow: the visitor sees a skeleton immediately instead of a blank screen. Our pages wait on nothing slow. The loading file was scaffolding from the start of the project, and nobody had asked what it did to the HTML.

Who reads HTML without running it

Googlebot renders JavaScript, but rendering is a separate step that can happen later than the first crawl. Many other fetchers never render at all:

  • Link preview bots from messaging and social apps read the head of the raw HTML and stop.
  • An analysis of crawler traffic published by Vercel and MERJ in late 2024 found that none of the major AI crawlers it observed executed JavaScript.
  • Simple tools, feed readers, uptime monitors and many SEO checkers work the same way.

For all of them, our pages said "Loading" and nothing else. Whether or not a given crawler copes, there is no reason to make the content depend on it.

The fix

We deleted the root loading.tsx. Without it there is no boundary around the page, so the server finishes rendering before it sends the body and the content arrives where it belongs. Suspense is still the right tool around a part of a page that waits on slow data; the rule is to wrap that part, not the whole page.

Three related things were corrected in the same pass.

One heading, inside the main content. Each page has exactly one <h1>, at the top of <main>, followed by a sentence or two that says what the page is. The logo in the header is a link to the home page, not a heading.

Metadata in the head for everyone. For pages rendered on each request, Next.js can stream the title and meta tags into the body after the first bytes, and holds them in the head only for user agents on a list of bots that cannot run scripts. The htmlLimitedBots setting controls that list. We set it to match every user agent, so the title, description, canonical link and Open Graph tags are always in the head of the first chunk.

Status codes decided before the body starts. Once a response has begun streaming, its status line has been sent. A page that discovers halfway through that it does not exist can render a "not found" message, but the status is already 200, and a search engine files it as a real page with thin content. A missing page has to be recognised before any HTML is written, and its 404 response should carry noindex and no canonical link.

How to check your own site

Fetch a page the way a crawler does and look for the marker of a streamed boundary:

curl -s -A "OAI-SearchBot" https://www.example.com/some-page | grep -c 'hidden id="S:'

A count of zero is what you want. Then check, still in the raw HTML and not in the browser's inspector:

  • the <h1> appears once, and inside <main>;
  • a distinctive sentence from the middle of the page is present as text;
  • each application/ld+json block is present and parses;
  • <title>, the meta description and the canonical link are before </head>;
  • a made-up URL returns status 404, not 200.

The inspector shows the document after scripts have run, which is exactly the view that hides this problem. Use View Source or curl.

Make it a test

A manual check is forgotten by the next refactor. Both sites now run an automated test that requests every URL in the sitemap with a crawler's user agent and asserts the list above for each one, plus a unique title and description per page. It takes a few seconds and fails the moment a loading boundary, a second <h1> or a streamed title comes back.

While you are in the head: the share image

The same pass replaced one site-wide share image with one per page. A detail worth knowing: a search result thumbnail is square, cut from the middle of the wide 1200 by 630 image. Text that runs edge to edge is cropped mid-word. We draw everything legible inside the centre 560 by 560 square and leave only background at the sides, and a test checks the pixels of every image so a long title cannot spill out of it.

Checklist

  • Read your raw HTML with curl, not the inspector.
  • Remove page-wide loading boundaries from pages that do not wait on slow data.
  • Keep one <h1> inside <main>, and structured data in the initial HTML.
  • Keep metadata in the head for every user agent.
  • Decide 404 before streaming begins.
  • Test every sitemap URL automatically.

The performance side of the same habit, rendering as much as possible ahead of time and shipping less script, is covered in fast web tools for e-commerce sellers.