[GH-ISSUE #1010] Support non-standard directives in `robots.ts` via `other` field #224

New issue

Closed

opened 2026-05-06 12:38:20 +02:00 by BreizhHardware · 0 comments

BreizhHardware commented

2026-05-06 12:38:20 +02:00

Owner

Originally created by @github-actions[bot] on GitHub (May 2, 2026).
Original GitHub issue: https://github.com/cloudflare/vinext/issues/1010

Upstream change

Next.js extended MetadataRoute.Robots so each rule can carry an other map of non-standard per-user-agent directives (e.g. Seznam Request-Rate, Yandex Clean-param) alongside the existing allow/disallow/crawlDelay fields.

// app/robots.ts
export default function robots(): MetadataRoute.Robots {
  return {
    rules: [
      { userAgent: '*', allow: '/' },
      {
        userAgent: 'SeznamBot',
        allow: '/',
        other: { 'Request-Rate': '10/1m' },
      },
    ],
  }
}

emits:

User-Agent: *
Allow: /

User-Agent: SeznamBot
Allow: /
Request-Rate: 10/1m

Array values expand into repeated lines and null/undefined entries are skipped. The new directives are scoped to their User-Agent block and emitted between Crawl-delay and the trailing blank line.

Commit: github.com/vercel/next.js@65340b2772
PR: https://github.com/vercel/next.js/pull/93206
Upstream issue: https://github.com/vercel/next.js/issues/89521

Serializer change (packages/next/src/build/webpack/loaders/metadata/resolve-route-data.ts):

if (rule.other) {
  for (const key of Object.keys(rule.other)) {
    const value = rule.other[key]
    if (value == null) continue
    const values = Array.isArray(value) ? value : [value]
    for (const v of values) {
      content += `${key}: ${v}\n`
    }
  }
}

Why this matters for vinext

vinext implements app/robots.{ts,js} resolution as part of its metadata route handling. To match Next.js parity we should:

Accept rule.other?: Record<string, string | number | Array<string | number>> on each rule.
Emit it after Crawl-delay and before the blank line separating user-agent blocks, with array values expanded into repeated lines and null/undefined skipped.
Update the public MetadataRoute.Robots type re-exported from our next/* shims so users get type-safe access to the new field.

Suggested work

Mirror the type change from packages/next/src/lib/metadata/types/metadata-interface.ts (factor out a RobotsRuleBase containing the optional fields plus other).
Update the vinext robots serializer to emit other directives in the correct position.
Port the new resolveRobots test cases (from resolve-route-data.test.ts) covering Request-Rate, array-valued Clean-param, and null/undefined handling.

Originally created by @github-actions[bot] on GitHub (May 2, 2026). Original GitHub issue: https://github.com/cloudflare/vinext/issues/1010 ## Upstream change Next.js extended `MetadataRoute.Robots` so each rule can carry an `other` map of non-standard per-user-agent directives (e.g. Seznam `Request-Rate`, Yandex `Clean-param`) alongside the existing `allow`/`disallow`/`crawlDelay` fields. ```ts // app/robots.ts export default function robots(): MetadataRoute.Robots { return { rules: [ { userAgent: '*', allow: '/' }, { userAgent: 'SeznamBot', allow: '/', other: { 'Request-Rate': '10/1m' }, }, ], } } ``` emits: ``` User-Agent: * Allow: / User-Agent: SeznamBot Allow: / Request-Rate: 10/1m ``` Array values expand into repeated lines and `null`/`undefined` entries are skipped. The new directives are scoped to their `User-Agent` block and emitted between `Crawl-delay` and the trailing blank line. - Commit: https://github.com/vercel/next.js/commit/65340b27724dc2878e4fcef4b8608ad06e7b4cc8 - PR: https://github.com/vercel/next.js/pull/93206 - Upstream issue: https://github.com/vercel/next.js/issues/89521 Serializer change (`packages/next/src/build/webpack/loaders/metadata/resolve-route-data.ts`): ```ts if (rule.other) { for (const key of Object.keys(rule.other)) { const value = rule.other[key] if (value == null) continue const values = Array.isArray(value) ? value : [value] for (const v of values) { content += `${key}: ${v}\n` } } } ``` ## Why this matters for vinext vinext implements `app/robots.{ts,js}` resolution as part of its metadata route handling. To match Next.js parity we should: - Accept `rule.other?: Record<string, string | number | Array<string | number>>` on each rule. - Emit it after `Crawl-delay` and before the blank line separating user-agent blocks, with array values expanded into repeated lines and `null`/`undefined` skipped. - Update the public `MetadataRoute.Robots` type re-exported from our `next/*` shims so users get type-safe access to the new field. ## Suggested work - Mirror the type change from `packages/next/src/lib/metadata/types/metadata-interface.ts` (factor out a `RobotsRuleBase` containing the optional fields plus `other`). - Update the vinext robots serializer to emit `other` directives in the correct position. - Port the new `resolveRobots` test cases (from `resolve-route-data.test.ts`) covering `Request-Rate`, array-valued `Clean-param`, and `null`/`undefined` handling.