Web detection - Aphrody Bun

Offline technology and anti-bot challenge detection for fetched responses, and an SSRF address deny-set

Bun.WebDetect analyzes a response you already fetched and reports the technologies it uses and any anti-bot challenge it serves. It performs no I/O. It also exposes the SSRF address deny-set used by Bun's browser transports.

const url = "https://example.com/";
const response = await fetch(url);
const report = Bun.WebDetect.detect({
  url,
  status: response.status,
  headers: Object.fromEntries(response.headers),
  body: await response.text(),
});

if (report.challenge.provider !== "none") {
  // e.g. "cloudflare-turnstile", recommendation "max": retry with Bun.WebView
}
console.log(report.findings.map(f => `${f.bucket}: ${f.name}`));

Input

Field	Type	Description
`url`	`string`	The requested URL. Only `http:` and `https:`
`finalUrl`	`string`	The URL after redirects (defaults to `url`)
`status`	`number`	The HTTP status (default `200`)
`headers`	`Record<string, string \| string[]>`	Response headers; repeat `set-cookie` as an array
`body`	`string`	The response body
`resolvedIps`	`string[]`	IP addresses the hostname resolved to
`cnameChain`	`string[]`	CNAME records of the hostname
`nsRecords`	`string[]`	NS records of the domain

detect() throws a TypeError when the input is not an object, a field has the wrong type, or the URL is not http: or https:.

Report

The report repeats url, finalUrl, httpStatus, hostname, resolvedIps, cnameChain and nsRecords, and adds:

findings: detected technologies, each with name, bucket (frontend, backend, cdn, dns, hosting, server, language, cms, analytics, tag-manager, framework, library or other), evidence, source, a numeric confidence and, when known, a version.
challenge: the anti-bot provider (none, cloudflare-block, cloudflare-challenge, cloudflare-turnstile, akamai-bot-manager, perimeter-x, data-dome, kasada, incapsula, aws-waf, sucuri, js-required, soft-block or rate-limit), its confidence (high, medium or low), a recommendation (static, fast, stealth, max, wait or abort) and the matching evidence strings.
strategy: a suggested crawl strategy with profile (static, http or render), waitFor, blockResources, reDetectAfterHydration, isSpa, hasAntiBot, the site shape and a rationale.

SSRF deny-set

Bun.WebDetect.isForbiddenAddress(ip) returns true for addresses that an outbound request from untrusted input should never reach: loopback, RFC 1918, link-local and cloud metadata (169.254.169.254, 100.100.100.200), CGNAT, benchmarking, documentation, multicast, reserved and unspecified addresses, and IPv6 forms embedding any of them (mapped, compatible, NAT64, 6to4, Teredo). Brackets around IPv6 literals are accepted; any other string that is not an IP address throws a TypeError. Check every resolved address, not only the hostname, before connecting.

const url = "https://example.com/";
const addresses = await Bun.dns.lookup(new URL(url).hostname, { family: 0 });
if (addresses.some(a => Bun.WebDetect.isForbiddenAddress(a.address))) throw new Error("blocked");

Bun.WebDetect cannot be constructed; call its static methods.