Technical GEO: Rendering, JavaScript and Accessibility for AI Crawlers

There is a technical GEO trap that affects more than 60% of modern websites: content that depends on JavaScript to render. AI crawlers — GPTBot, PerplexityBot, ClaudeBot — behave differently from a web browser: most do not execute JavaScript or do so in a limited way. If your key content is in a React component that loads dynamically, if your blog lives behind an API call, or if your most important information requires interaction to display, AI crawlers simply don't see it.

How AI Crawlers Read Your Site

AI crawlers work similarly to search crawlers from 10 years ago: they make an HTTP request to your URL and process the HTML they receive in the response. What is not in that initial HTML — what loads afterward with JavaScript, what appears after scrolling, what requires authentication, what's in iframes — simply doesn't exist for them.

This has direct implications for your content's citability:

• If your blog uses pure client-side rendering (CSR), the crawler receives an almost empty HTML with just the app skeleton.
• If your most valuable content is in tabs or accordions that open with JavaScript, it is not indexed.
• If your Schema.org structured data is dynamically injected by JS, it may not be read.
• If you have infinite scroll or lazy-load pagination, crawlers only see the first block of content.

SSR and SSG: Solutions for Technical GEO

The two technical strategies that guarantee AI crawlers access your complete content are:

• Server-Side Rendering (SSR): The server generates the complete HTML before sending it to the client. The crawler receives all content in the first request. In Next.js, this is achieved with getServerSideProps or dynamic pages in the App Router.
• Static Site Generation (SSG): The complete HTML is generated at build time and served directly. It's the highest-performance option and the most "crawler-friendly." In Next.js, achieved with generateStaticParams() and static pages — which is exactly how this blog is built.

ISR (Incremental Static Regeneration) is also a good option for content that updates frequently but doesn't need to be dynamic in real time.

Technical Accessibility Checklist for AI Crawlers

• Verify that the initial HTML contains your content: Disable JavaScript in your browser (DevTools → Settings → Disable JavaScript) and browse your site. What you see is what AI crawlers see. If you see a blank or contentless page, you have an SSR problem.
• Schema.org in static HTML: Your Schema.org JSON-LD must be in the initial HTML, not injected by JavaScript afterward. In Next.js, use the Script component with beforeInteractive strategy or include it directly in the JSX.
• Server response time: AI crawlers have short timeouts. If your server takes more than 5–10 seconds to respond, the crawler may abort the request. Optimize your Time to First Byte (TTFB).
• Content not behind login or paywall: Content that requires authentication is not accessible to crawlers. If you want your premium content to be cited by LLMs, it needs to have a publicly accessible version.
• Correct canonical URLs: Ensure canonical URLs are the ones you want crawlers to index and there are no chain redirects that could confuse or exhaust the crawler.

Speed and Core Web Vitals for GEO

Although AI crawlers don't execute JavaScript, server response time does matter. A high TTFB (over 800ms) can result in the AI crawler timing out or indexing an incomplete version of your page. Optimize:

• Aggressive CDN caching: For static content, serve from the CDN closest to the crawler. Vercel, Cloudflare and AWS CloudFront have global distribution that drastically reduces TTFB.
• Optimized images: Although crawlers don't see images, page weight can slow server response. Use WebP and proper compression.
• Minification and Gzip/Brotli compression: Reduce the size of the HTML the crawler downloads.

How to Verify That AI Crawlers Access Correctly

• Use Google Search Console's URL inspection tool with "Test as Googlebot" — an approximate indicator of what crawlers see.
• Install the "Web Developer" extension in Chrome and disable JS to see your site as a crawler would.
• Verify in your server logs that AI crawler user agents (GPTBot, PerplexityBot, etc.) appear with 200 status codes, not 403, 429 or 5xx.

The Technical Foundation of a Complete GEO Strategy

Without correct technical accessibility, everything else in GEO is useless. The best answer-ready content, the best link building and the best structured data have no impact if the AI crawler cannot read your site. Audit technical accessibility before any other GEO optimization.

Esbuenisimo Links includes technical GEO auditing in its advanced services, verifying that each client's site is correctly accessible to all major AI crawlers before launching any Digital PR or link building campaign.