Being found — SEO as an engineering problem, and the app you don't want indexed

The site is live. It is fast, it looks right, the copy took three weeks. You search for the thing it does — the exact thing, in the words a customer would use — and you are not on the first page. You are not on the fourth. You search for your own company name and find your LinkedIn, a directory listing you never made, and eventually, somewhere below a competitor's comparison post, yourself.

The advice you find next is mostly folklore: keyword density, meta keywords, posting frequency, a plugin that gives you a green light for putting a word in a heading. Some of it was true in 2011. Almost none of it explains the situation, because the situation is usually not that you rank badly. It is that you are not in the running.

That distinction is the whole of this issue. Underneath the folklore there is a system with about five moving parts, and four of them are decided by engineers — what the server sends, how fast it sends it, what the URLs mean, and which pages are allowed to exist at all. So: the parts, the starter checklist that covers most of the damage, the rendering decision in Next.js written as one table, how to check any of it without guessing, and the question every team eventually hits — what to do when the landing page and the product are the same codebase.

Ranking is the fifth gate, not the first

A search engine is not a scoring function you can appeal to. It is a pipeline, and a page has to survive every stage of it before a ranking is even a meaningful concept.

Discover — the engine has to learn the URL exists at all, from a link on a page it already knows, from your sitemap, or from a redirect. A page nobody links to and no sitemap mentions is, to a crawler, a rumour.

Crawl — it fetches the URL, on a budget. Every site has a rough allowance of requests per day based on how big and how important it seems, and that budget is spent on whatever you let it spend it on: a thousand filtered search-result URLs, five hundred redirect hops, or your actual pages.

Render — Google puts the fetched page in a queue to run its JavaScript later, in a headless Chrome. Later can mean seconds; it can mean days when the site is unimportant or the render is expensive. Bing does much less of this, and most other crawlers do none.

Index — the result is stored, deduplicated against near-identical pages, and assigned a canonical. This is where you find the phrase that most confuses people in Search Console: crawled — currently not indexed, which means the engine fetched your page, read it, and decided not to keep it. That is a content and duplication verdict, not a keyword one.

Rank — only now do the hundreds of signals apply, and this is the only stage you cannot directly set. Everything before it, you can.

So the working method is: debug left to right. Before asking why a page ranks fifteenth, confirm it is indexed. Before asking why it is not indexed, confirm the crawler got usable HTML. It sounds obvious written down, and it is the step almost everyone skips, because the fifth gate is the one all the articles are about.

The starters — six things, none of them clever

If you do nothing else in this issue, do these. They are the difference between a site that can compete and a site that is invisible for reasons that have nothing to do with how good it is.

The title is the blue line someone clicks. Write it for a person scanning ten results at speed: the distinguishing word first, about sixty characters, and different on every page. Home | Acme on forty pages is forty pages that look identical to a machine trying to tell them apart.

The meta description is not a ranking factor and has not been one for years. It is a click-through factor — the grey sentence under the title — and Google rewrites it perhaps half the time when it thinks a snippet from your page fits the query better. Write it anyway; a good one earns clicks that a truncated first paragraph does not.

rel=canonical names the one true URL for this content. It matters more than it sounds, because sites generate duplicates without meaning to: /pricing and /pricing/, the http and https versions, the same page with a tracking parameter, a print view, a paginated variant. Left alone, the engine picks a canonical itself and may well pick the wrong one, and your signals are split between two addresses that are competing with each other.

meta robots is the tag that removes pages. noindex asks that a page not be listed; nofollow asks that its links not be counted. Both are requests, honoured by the majors and ignored by plenty of others, and both are how a staging site ends up ranking above production — or, more painfully, how a production page disappears because a directive written for staging shipped with it.

Then the heading tree: one h1 that says what the page is, h2 for its sections, h3 below those, in order and without skipping levels for visual reasons. This is the same discipline that makes a page usable with a screen reader, which is not a coincidence — both readers are trying to build an outline from your markup.

In Next.js all of the head is one API, and the important habit is metadataBase: set it once and every relative URL you write resolves to an absolute one, which is the single most common thing to get wrong by hand.

import type { Metadata } from "next";

export const metadata: Metadata = {
  // Set this and every relative URL below resolves to an absolute one.
  // Without it, canonicals and og:image quietly ship as "/og/x.png",
  // which is meaningless to a crawler that is not a browser.
  metadataBase: new URL("https://site.com"),

  title: {
    // Pages set only their own part; this supplies the suffix.
    template: "%s · Acme",
    default: "Acme — invoicing that closes itself",
  },
  description:
    "Send an invoice, get paid, and never chase a client again. Built for freelancers who would rather be working.",

  // Sensible global defaults. Individual routes override.
  robots: { index: true, follow: true },
  alternates: { canonical: "/" },
};

import type { Metadata } from "next";

export const metadata: Metadata = {
  // Becomes "Pricing — simple, per-seat, no annual lock-in · Acme".
  // Distinguishing words first: someone scanning results reads about
  // four of them before deciding.
  title: "Pricing — simple, per-seat, no annual lock-in",

  // Not a ranking factor. It is the sentence that earns the click,
  // and Google will replace it whenever it disagrees with you.
  description:
    "Flat $12 per seat per month, everything included, cancel any time. Compare plans and see what a five-person team actually pays.",

  // Absolute after metadataBase resolves it. Match the trailing slash
  // your site really serves, or you have declared a URL that 301s.
  alternates: { canonical: "/pricing" },
};

export default function Page() {
  // Exactly one h1, and it says the same thing the title does.
  return <h1>Pricing</h1>;
}

The last two starters are the ones that make discovery deliberate instead of lucky. A sitemap is a list of the URLs you consider canonical and want crawled; robots.txt is the note at the door about where not to bother going. Neither improves a ranking. Both decide whether a page is in the game.

import type { MetadataRoute } from "next";
import { POSTS } from "./data";

const BASE = "https://site.com";

// A sitemap built by hand is a sitemap that goes stale. Generate it
// from the same source the pages render from and it cannot disagree
// with what actually exists.
export default function sitemap(): MetadataRoute.Sitemap {
  const pages = ["", "/pricing", "/about", "/writing"].map((p) => ({
    url: `${BASE}${p}`,
    lastModified: new Date(),
  }));

  const posts = POSTS.map((p) => ({
    url: `${BASE}/writing/${p.slug}`,
    // Honest dates only. lastModified: new Date() on every URL, every
    // build, is a signal that everything changed - so nothing did.
    lastModified: p.updatedAt ?? p.publishedAt,
  }));

  // Only canonical, indexable, 200-returning URLs belong here.
  // No /app/*, no redirects, no noindexed pages.
  return [...pages, ...posts];
}

// ---- app/robots.ts ----------------------------------------------
export function robots(): MetadataRoute.Robots {
  return {
    rules: [
      {
        userAgent: "*",
        allow: "/",
        // Keep the crawl budget on pages that matter. These are not
        // secrets - robots.txt is public and is the first file an
        // attacker reads. It is a traffic instruction, nothing more.
        disallow: ["/app/", "/api/", "/*?sort=", "/*?filter="],
      },
    ],
    sitemap: `${BASE}/sitemap.xml`,
  };
}

The rendering question, in one table

This is where framework choices become SEO choices, and the whole decision collapses to one question: what is in the first response? Not what the page looks like after hydration — what arrives in the bytes of the initial HTML.

Static (SSG) is the default in the Next.js App Router and the right answer for almost everything public. The page is rendered once at build and served as a file from a CDN: the crawler gets complete HTML on the first byte, from an edge node, with no server involved. Marketing pages, documentation, blog posts, changelogs, comparison pages — all of it.

ISR is static with an expiry. Set revalidate and the page is served from cache while a fresh copy is regenerated in the background. This is the answer for content that changes on its own schedule but not per visitor — a product catalogue, a pricing page fed by a CMS, a listings site with fifty thousand pages you would rather not rebuild all at once.

SSR renders per request. The crawler still gets complete HTML, so it is perfectly indexable — you are simply paying for a server on every hit, including every crawler hit, and adding your database's latency to your time to first byte. Use it when the content genuinely differs per request: prices in the visitor's currency, live inventory, anything personalised.

PPR — partial prerendering — is the newer shape, and it resolves an old tension. A static shell is served instantly while dynamic holes stream in behind Suspense boundaries. The crawler gets the shell immediately, which is usually where all the indexable content lives, and the visitor gets their name in the header a moment later.

Client rendering sends an empty container and a script bundle. For a dashboard this is correct and always has been — nobody needs to index a page that requires a login. For a landing page it is the single most expensive mistake in this issue, and the reason is the next section.

// ---- app/(marketing)/pricing/page.tsx --------------------------
// Nothing here. No config, no dynamic APIs, no "use client".
// This is a static page, and silence is how you ask for one.

// ---- app/(marketing)/blog/[slug]/page.tsx ----------------------
// A dynamic segment becomes a set of files the moment you can list
// the params at build time. If a page CAN be static, it should be.
export async function generateStaticParams() {
  const posts = await getPosts();
  return posts.map((p) => ({ slug: p.slug }));
}
// Anything not in the list above 404s instead of rendering on demand.
export const dynamicParams = false;

// ---- app/(marketing)/changelog/page.tsx ------------------------
// ISR: serve from cache, refresh in the background every hour.
export const revalidate = 3600;

// ---- app/(marketing)/deals/page.tsx ----------------------------
// Genuinely per-request: live stock, per-region prices. Still fully
// server-rendered, so still fully indexable - just not free.
export const dynamic = "force-dynamic";

// ---- app/(app)/dashboard/page.tsx ------------------------------
// Behind a login. Client-heavy, per-user, and deliberately invisible
// to search - see the (app) layout later in this issue.
export const dynamic = "force-dynamic";

Why "Google runs JavaScript" is not the reassurance it sounds like

It does. Googlebot renders pages in a current headless Chrome, and a well-built client-rendered site can absolutely be indexed. The catch is in the word when.

Rendering happens on a second pass. The crawler fetches your HTML, sees a shell, and puts the URL in a render queue. That queue is drained on the engine's schedule, weighted by how important your site appears to be — which for a new site is not very. The gap between fetch and render is often seconds and occasionally days. Everything the first pass decides — whether to follow your links, what your title is, whether this page is worth queueing at all — it decides from the shell.

Then there is everyone else. Bing renders far less. LinkedIn, WhatsApp, Slack, and X run no JavaScript at all when building a link preview. And the crawlers that increasingly matter — the ones feeding AI answers and assistants — mostly fetch raw HTML and read it as text, exactly like a 1998 search engine with better comprehension.

Speed, as a measurable input

Page experience is a real, modest ranking input, and a very large conversion input. It is measured by three field metrics, collected from actual Chrome users on your actual site, and scored at the 75th percentile — meaning three out of four visits must be good, not the median one, and certainly not yours.

LCP is when the largest thing above the fold finishes painting — usually a hero image or a headline. The fixes are unglamorous and reliable: serve the hero as a modern format at the right size, give it priority so it is not lazy-loaded, preconnect to whatever host it comes from, and stop blocking the paint behind a font that has not downloaded. In Next.js, next/image with priority and next/font covers most of it in two lines.

INP is the delay between a tap and something visibly happening, across the whole visit. It is a JavaScript problem almost by definition: the main thread is busy hydrating, or running an analytics bundle, or re-rendering a tree that did not need to re-render. The cheapest fix on a marketing page is to ship less — server components by default, use client only where interactivity actually lives.

CLS is how much the page jumps while loading, and it is nearly always the same cause: something arrived and pushed everything down. An image without dimensions, an ad slot, a cookie banner, a font swap that changes line height. Reserve the space and the number goes to zero.

One caution about tooling. Lighthouse and PageSpeed Insights give you a lab score from a simulated device — useful for finding causes, but it is not the number in the ranking system. The field data is, and it comes from the Chrome UX Report, aggregated over twenty-eight days. That lag is worth knowing about before you conclude your fix did not work.

The half that is not code

Everything above gets you into the running. It does not win anything, because so does everyone else's framework. What decides the rest is whether your page is the best answer to a specific question — and the engineering part of that is mostly about structure.

One page per intent. People search with an intent behind the words: to learn something, to compare options, to buy, or to reach a specific site. A page trying to serve two of those serves neither, and — worse — two pages of yours competing for the same query is one page's worth of signals divided in half. Pick the intent, name the page for it, and link the others to it.

Answer near the top. The reader who bounces in nine seconds bounced because the answer was in the eighth paragraph. This has always been good writing advice; it is now also how you get quoted, because the systems summarising pages into answers read the same way an impatient person does.

Be a plausible source. Google's guidelines describe this as experience, expertise, authoritativeness, and trust, which sounds like an abstraction and cashes out as concrete markup: a real author with a real page, a visible date, citations that point outward, an about page, a contact route. It matters most in areas where being wrong is expensive — health, money, law.

Say it in a machine-readable way too. Structured data does not raise your ranking directly. It changes what your result looks like — stars, prices, an FAQ accordion, a breadcrumb trail instead of a raw URL — and a result that occupies more space and answers more of the question gets clicked more. It is also, increasingly, how an answer engine decides what your page is claiming.

// Server component: this ends up in the first response, which is the
// only place a crawler is guaranteed to look. Injecting JSON-LD from
// a useEffect is a common and completely wasted pattern.
export function ArticleJsonLd({ post }: { post: Post }) {
  const data = {
    "@context": "https://schema.org",
    "@type": "Article",
    headline: post.title,
    datePublished: post.publishedAt,
    dateModified: post.updatedAt ?? post.publishedAt,
    author: {
      "@type": "Person",
      name: "Subhra Jyoti Baroi",
      // Point at a real page about a real person. An author with no
      // URL is a string, not an attribution.
      url: "https://site.com/about",
    },
    // Every claim here must be visibly true on the page. Marking up
    // an FAQ that no visitor can see is the definition of the spam
    // policy on structured data, and it is enforced.
    description: post.dek,
    image: [`https://site.com/og/${post.slug}.png`],
    mainEntityOfPage: `https://site.com/writing/${post.slug}`,
  };

  return (
    <script
      type="application/ld+json"
      // The content is ours and is serialised from typed data, not
      // user input. Escape "<" so a string in the data can never
      // close this tag early.
      dangerouslySetInnerHTML={{
        __html: JSON.stringify(data).replace(/</g, "\\u003c"),
      }}
    />
  );
}

And then the part that engineers control more than they realise: what links to what. A crawler discovers pages by following links, and it treats a link from an established page as a small vote of confidence that flows onward. Which means the shape of your site is a distribution mechanism.

Two practical rules come out of that picture. Keep important pages within about three clicks of the home page — depth is a decent proxy for how often a crawler returns. And notice that the pages which earn links are rarely the pages you want to rank: the essay gets shared, the pricing page pays the bills. A hub page that links deliberately between them is how the credit moves.

Write the anchor text like a human describing the destination — how we price per seat, not click here. It is a description of the target page written by someone else's page, which is exactly the signal search engines have trusted the longest.

The landing page and the app

Here is the situation nearly every product hits. There is a marketing site — home, pricing, blog, a few comparison pages — which needs to be found. And there is the product, which is a client-heavy application behind a login, which needs to be fast, stateful, and completely uninterested in Google. They have different requirements, often different frameworks, sometimes different teams. Where do you put the seam?

The decision that matters is not the repository. It is the hostname, because reputation in search accumulates per hostname, and that is the one asset you cannot buy back quickly.

One host, split by path — site.com/pricing and site.com/app — is the default worth defending. Every link anyone ever gives you strengthens the same hostname. There is one analytics property, one cookie domain, one certificate, one place a customer types. When the blog earns a mention, the pricing page benefits.

A subdomain — app.site.com — is usually treated as a related but separate site. Some signals carry across; many do not, and you now have two entities to maintain. This is a fine choice when there is a real reason: a wildcard for per-customer subdomains, a cookie isolation requirement, an infrastructure boundary you cannot move. It is a poor choice when the reason is that routing two deploys onto one domain seemed like a hassle for an afternoon.

A second domain is a second reputation, from zero, forever. There are legitimate reasons — a rebrand, an acquisition, a genuinely separate product — and there is no way to make it cheap. If you are choosing today, choose one domain.

Which leaves the practical question: what if they are already two codebases, two repos, two deploy pipelines? That is not a problem, and it is not a reason to accept a subdomain. Put a rewrite in front of them. The browser and the crawler see one origin; behind it, two deployments answer for different paths.

/** @type {import('next').NextConfig} */
export default {
  async rewrites() {
    return [
      // Everything under /app is served by the product deployment.
      // A rewrite, not a redirect: the URL in the address bar stays
      // site.com/app/..., so the browser, the crawler, and every
      // link that ever gets shared all see one hostname.
      {
        source: "/app/:path*",
        destination: "https://acme-product.vercel.app/app/:path*",
      },
      // Docs might be a third system entirely - a static site
      // generator, a hosted help centre. Same trick.
      { source: "/docs/:path*", destination: "https://acme-docs.pages.dev/:path*" },
    ];
  },

  async redirects() {
    return [
      // If app.site.com already exists and has been shared, do not
      // just delete it. A permanent redirect moves most of what it
      // accumulated onto the path-based URL. This is a migration,
      // not a config change: expect weeks, and watch Search Console.
      {
        source: "/:path*",
        has: [{ type: "host", value: "app.site.com" }],
        destination: "https://site.com/app/:path*",
        permanent: true,
      },
    ];
  },
};

Two cautions on the rewrite. Whatever sits behind it must generate its own URLs as /app/... rather than assuming it owns the root, or the first click sends someone to a path that does not exist. And it should not emit its own canonicals or sitemap for the public hostname; the marketing side owns the head, the app side owns nothing search-facing at all — which is the next section.

When it is one codebase

This is the case people expect to be hardest and it is actually the easiest, because everything is one deploy and one design system. The mistake is not having them in one repo. The mistake is letting them share defaults, when their defaults should be opposites.

Route groups are the tool. A directory in parentheses organises files without contributing a URL segment, so app/(marketing)/pricing/page.tsx still serves /pricing. Each group gets its own layout — which means its own metadata, its own providers, its own bundle — and the two can be as different as two projects while shipping as one.

app/
├─ layout.tsx                 // as small as possible: <html>, fonts, reset
│                             // NOT the auth provider, NOT the app's store
├─ (marketing)/
│  ├─ layout.tsx              // marketing nav + footer. Server components.
│  ├─ page.tsx                // /            static
│  ├─ pricing/page.tsx        // /pricing     static
│  └─ blog/[slug]/page.tsx    // /blog/:slug  static via generateStaticParams
│
├─ (app)/
│  ├─ layout.tsx              // auth gate, providers, "use client" island
│  ├─ dashboard/page.tsx      // /dashboard   dynamic, noindex
│  └─ settings/page.tsx       // /settings    dynamic, noindex
│
├─ sitemap.ts                 // marketing URLs only
└─ robots.ts                  // Disallow: /dashboard, /settings

The important line is the one about the root layout. It is shared by both groups, so anything you put in it is paid for by every marketing page — and a root layout that mounts the app's auth provider, state store, and analytics stack quietly turns a static landing page into a page that ships two hundred kilobytes of JavaScript to hydrate a headline. Keep the root to html, body, fonts, and the reset. Everything else belongs to a group.

import type { Metadata } from "next";
import { redirect } from "next/navigation";
import { getSession } from "@/lib/auth";

// Set once for the group, inherited by every route inside it. No
// remembering to add a tag to each new page - which is exactly the
// kind of remembering that fails on a Friday.
export const metadata: Metadata = {
  robots: {
    index: false,
    follow: false,
    // Also drop cached copies and text snippets, for the URLs that
    // were indexed before anyone thought about this.
    noarchive: true,
    nosnippet: true,
  },
};

export default async function AppLayout({
  children,
}: {
  children: React.ReactNode;
}) {
  // This is the real control. noindex is a polite request that most
  // crawlers honour and anyone can ignore; the redirect is what
  // actually makes the page unavailable. Never rely on a meta tag
  // to protect anything.
  const session = await getSession();
  if (!session) redirect("/login");

  return <AppProviders>{children}</AppProviders>;
}

Checking it, without guessing

SEO tools are largely a market in confident numbers. There are exactly two sources of truth: what the search engine tells you it did, and what your server actually sends. Everything else is inference.

Search Console is not optional and takes ten minutes to set up. Three parts of it do most of the work. URL Inspection tells you, for one URL, whether it is indexed, when it was last crawled, what canonical Google chose (which may not be the one you declared), and what the rendered HTML looked like. Pages groups every URL it knows by why it is or is not indexed. And Performance shows the queries you actually appear for, with impressions and clicks — which is regularly the most surprising screen, because the queries bringing people in are rarely the ones anyone targeted.

The other side is what you serve, and it takes one command. Being the crawler is more informative than any audit score.

URL="https://site.com/pricing"

# 1. The head, as delivered - no JavaScript, no cookies. If a tag is
#    missing here it does not exist, however good devtools looks.
curl -sL "$URL" \
  | grep -Eio '<title>[^<]*|<meta name="(description|robots)"[^>]*|<link rel="canonical"[^>]*'

# 2. Status and redirects. Every hop costs crawl budget, and a chain
#    of three is a chain some crawlers abandon.
curl -sIL "$URL" | grep -E '^(HTTP/|location:)'

# 3. Is the content in the first response, or only after hydration?
#    If this prints 0, the page is client-rendered as far as the
#    first pass is concerned.
curl -sL "$URL" | grep -c "per-seat"

# 4. What a crawler that does not run JavaScript sees, as text.
curl -sL -A 'Mozilla/5.0 (compatible; Googlebot/2.1)' "$URL" \
  | sed -e 's/<[^>]*>//g' | tr -s '[:space:]' ' ' | head -c 600

# 5. Discovery files exist and are honest.
curl -s https://site.com/robots.txt
curl -s https://site.com/sitemap.xml | grep -c "<loc>"

# 6. What is already known - run in a search box, not a shell:
#    site:site.com                 everything indexed
#    site:site.com/blog            one section
#    site:site.com inurl:dashboard the leak you are looking for

That last one deserves emphasis, because it is free and it is the fastest way to find the problem this issue keeps circling. Search site:yourdomain.com and read the list. If your app's internal pages are in it, you have a leak. If a page you care about is absent, you have a gate-four problem to take to URL Inspection. If half the results are staging or a preview deployment, you have found this afternoon's work.

Then a note on expectations, because it is where most efforts quietly die. A deploy is not a result. The crawler has to come back, the page has to be re-indexed, and any ranking change settles over weeks. Change one thing at a time, write down what and when, and resist reading the graph on Tuesday. Requesting indexing in Search Console nudges a single URL to the front of the queue; it does not skip the queue.

One more reader

A growing share of the traffic that used to arrive as ten blue links now arrives as a summary — an AI overview, an assistant answering in a chat window, a model citing three sources. The optimisation for this is not a new discipline, which is the good news: those systems crawl the same HTML, and they reward the same things good documentation always rewarded.

Concretely: serve real HTML, because most of these crawlers render less JavaScript than Google does. Answer the question in a self-contained paragraph near the top, because that paragraph is what gets extracted. Use headings that read like questions people ask. Keep facts near the claims they support — a number in a sentence is quotable, a number in an image is invisible. And decide deliberately whether you want to be in the training and answering sets at all; robots.txt takes named user-agents, and that is a business decision rather than a technical one.

The short version

None of this is mysterious. It is the same work as making a site legible to a screen reader, or to a colleague reading it in a text browser, or to a person on a bad connection in a hurry — done for a reader who cannot ask you a follow-up question and who leaves without telling you why. Get the HTML honest and the URLs meaningful, keep the product out of the index and the marketing out of the app bundle, and the rest is writing something worth finding.