Fetching a robot-protected site with curl

Robot-verification pages often sit between a browser and a public webpage, leaving command-line users with a small challenge form instead of the content they expected. The block may come from a content delivery network, web application firewall, anti-bot service or ordinary rate limiter, so the useful first move is to identify the response rather than guess at curl options.

Using a command-line tool like curl to fetch a site without a robot check is best treated as a legitimate-access problem. A site may provide an API, feed or approved client route that does not use human verification; curl can then request that route normally. Trying to defeat a CAPTCHA, forge clearance tokens or evade an explicit access rule is a different activity with legal and security consequences.

What the robot check is doing

Verification can be issued before an edge server passes traffic to the origin, which explains why even a page that appears in search results fails when requested with curl. A concise account of why robot checks appear separates several common layers and the assets each one protects. The key distinction is whether the server is asking for proof of a human or simply applying a rule to a particular request.

A robot check is also not the same thing as robots.txt. The latter is a crawler policy file that states paths to crawl, paths to avoid and sometimes a crawl delay; it does not authenticate a client or guarantee access. Conversely, a successful human challenge can issue a cookie or token, but that does not cancel the site’s terms, rate limits or privacy obligations.

When a check is clearly designed to gate an account, member area or protected form, do not attempt to bypass it by rotating IP addresses, solving CAPTCHAs through a paid service or copying another person’s session cookie. Instead, ask the owner for an API key, service account, allowlisted address or dedicated export. That route is easier to audit and gives Australian administrators a defensible reason for the request.

Inspecting the HTTP response

Set SITE_URL to the exact address you are authorised to request, then make one ordinary GET and save both the response metadata and body. Avoid a custom browser user agent at first because it hides the real behaviour of the client you intend to use.

curl --compressed -sS -L -D /tmp/headers.txt -o /tmp/body.html \
  -w 'status=%{http_code} type=%{content_type}\n' "$SITE_URL"

Read the status line, redirect chain, Location, Set-Cookie, Retry-After, Content-Type and vendor headers, then search the body for challenge markers.

sed -n '1,40p' /tmp/headers.txt
grep -Eio 'captcha|challenge|verify|robot|too many requests' \
  /tmp/body.html | sort -u

A 200 response can still contain a verification page rather than the requested article. A 403 may indicate an access rule, a 429 usually calls for slower requests, and a redirect to a challenge path identifies a browser-oriented gate. Status codes are clues, not verdicts, so preserve the saved response instead of repeatedly hitting the endpoint.

If the failure is a TLS, HTTP/2 or proxy compatibility problem, test the same request with --http1.1. This can repair transport negotiation without changing the request’s identity or defeating the challenge.

curl --http1.1 --compressed -sS -L -D /tmp/headers.txt \
  -o /tmp/body.html "$SITE_URL"

Use sensible backoff after 429 or 503; repeated attempts can extend the block and make diagnosis harder.

Using a permitted route to the content

Many sites expose a JSON API, RSS or Atom feed, sitemap, media library or downloadable asset designed for automated retrieval. If the API documentation names a supported token, keep it in an environment variable or secret manager and pass it in the request header rather than placing it in a script or shell history.

curl -sS \
  -H "Authorization: Bearer $API_TOKEN" \
  -H 'Accept: application/json' "$API_URL"

This is a normal fetch of an authorised endpoint, not an attempt to impersonate a person.

A public API is not always present. Some publishers provide a feed or export file, while membership sites may document session tokens for legitimate integrations. Follow the exact renewal and scope rules, store credentials securely and respect deletion or revocation requirements. If you are retrieving your own data, a browser session may be appropriate, but sharing cookies or exporting clearance cookies from someone else’s browser creates avoidable privacy and account-takeover risks.

When protection depends on JavaScript, curl cannot execute that code by itself. Browser automation such as Playwright can render the page and complete a normal verification where the site’s terms permit it, but it should not be configured to conceal automation or defeat an explicit denial. A better outcome is an owner-supported integration key or server-side feed that removes the need for browser emulation.

Australian conditions and local triage

Australian sites commonly use .com.au addresses, and local retailers often run WooCommerce, Shopify or Adobe Commerce alongside separate content or membership systems. Australian shoppers expect prices in Australian dollars, but edge protection on the catalogue does not necessarily apply to account or checkout routes. A useful overview of security signals helps distinguish a basic challenge from stronger controls, while local traffic and terms still determine which path is appropriate.

Local operational details can also explain apparently inconsistent results. Australian dates are commonly written as DD/MM/YYYY, while Sydney and Melbourne move between UTC+10 and UTC+11 under daylight saving and Perth remains on UTC+8; HTTP headers usually use UTC, so log both forms when comparing incidents. Privacy and records duties under Australian law, along with ACSC and OAIC guidance, matter when fetched pages contain personal information. In everyday Australian speech, a report that something failed “this arvo” may only mean that an afternoon run broke, not that the service was unavailable all day.

Read the first-pass result against these indicators:

• A 200 response containing challenge text or a verification form usually means the page was accepted but the content was withheld.
• A 403 suggests a policy denial, although it can also come from an origin rule rather than the anti-bot layer.
• A 429 or Retry-After header points to throttling; pause before trying another client.
• A redirect ending at a challenge or security path confirms that browser-style verification is part of the current route.

Then select a route that the operator supports:

• Use a documented API, feed, sitemap or export endpoint for recurring collection.
• Request a service account or scoped token when the content is intended for authenticated access.
• Ask the host to allowlist a stable integration address or approve a particular crawler user agent.
• Use a normal rendered browser only where the terms allow it, and avoid stealth or CAPTCHA-solving services.

A reliable workflow for owners and operators

If you operate the site, keep interactive verification on browser routes and issue a separate credential for server-to-server collection. Allowlisting an integration, restricting the API key to required scopes, rotating it regularly and logging status codes gives security staff a clear audit trail. Update robots.txt when crawlers change, but do not treat that file as a substitute for access control or API permissions.

A public-facing business site such as MP3CD Software can keep catalogue and marketing content easy to index while applying stricter controls to forms, accounts and administrative endpoints. The exact implementation may differ, but this separation keeps legitimate monitoring apart from privileged actions and reduces the incentive to bypass a check.

Your next step is to set SITE_URL to the exact Australian site path you are authorised to fetch, run the first GET command once, and save the status, headers and body so the block can be classified as a redirect, JavaScript challenge, rate limit or access rule.