Bot spoofing and how to detect it with Arcjet

The User-Agent header is the name badge for web requests. Although it's been deprecated by some browsers, it's still sent by well behaving clients and is commonly used to identify automated clients. It's what robots.txt is based on.

But just like a name badge, clients can write whatever they like in the User-Agent header. This is a problem if it's the only thing you use to set up rules for managing bots, and is one reason why Arcjet uses other fingerprinting techniques like IP address analysis as part of our bot detection features.

Now we're adding more detailed verification options to developers where every request will be checked behind the scenes using published IP and reverse DNS data for common bots.

Bot detection is never perfect, but this improvement helps protect against spoofed bots where clients pretend to be someone else. For example, we can detect if a client is really Googlebot by checking if the request IP is within Google’s published IP ranges.

The analysis happens automatically for all Arcjet Pro plan users. If we detect a spoofed bot (or successfully verify a bot), additional metadata will be added to the response decision so you can decide how to handle it.

For example, to check for spoofed bots:

if (decision.reason.isBot() && decision.reason.isSpoofed()) {
  console.log("Detected spoofed bot", decision.reason.spoofed);
  // Return a 403 or similar response
}

And to confirm whether a bot has been verified:

if (decision.reason.isBot() && decision.reason.isVerified()) {
  console.log("Verified bot", decision.reason.verified);
  // Allow the request
}

Right now we support verification for Google, Bing, ChatGPT, and Datadog. Our bot list is open source and we'll be adding more over time.

So if you're having trouble with bot traffic, try out verified bot detection in Arcjet by signing up for free today. When you're ready to go to production, reach out to upgrade to Pro (pricing).