How to implement functional testing of security rules

Testing! It’s not a new concept in software development, but it’s mostly been missing from security tools.

How can you be confident that your rate limiting, bot detection, or validation rules work as expected as your code evolves? The traditional challenge has been that security tools often exist separately from the application, making testing cumbersome and risky.

If your security rules are implemented outside of your application, you need to build an entirely separate system for testing them. If they’re on the network layer, you have to send real traffic to trigger them, which often means you have to attack production infrastructure.

The developer security tooling we're building at Arcjet changes the game by unifying security and application logic. It runs consistently across development, CI/CD, and production environments, allowing you to integrate security rule testing directly into your functional test suite.

Why testing security rules matters

I’m sure many of you have flipped that security switch on a new tool, WAF, firewall, or other security system only to brick your production deployment! Not fun.

Untested security tools and rules can lead to:

• Broken application responses: returning an HTML error page when an API client is rate limited.
• Incorrect rules: scenarios don’t trigger rules as expected.
• Accidentally breaking production: rules that trigger in unexpected ways because they only execute in production.

Regular testing ensures your security rules remain effective and aligned with your application's behavior.

Arcjet is the same wherever it runs

Arcjet is designed to run in every environment. Install the SDK into your application then use middleware or call protect() in each route. The request is passed through our WebAssembly module for local analysis and our real-time decision API is used when we need to track state. Then within a few milliseconds you get a result that can be integrated into your application logic.

It’s the same process wherever your application runs. On a developer laptop. In CI/CD. In production. This means you can now include security rules as part of your functional test suite.

How to test security rules

Testing security rules just means sending the right traffic to your Arcjet-protected application. How to test each rule depends on the rule type:

• Shield is the Arcjet request analysis component. It’s like a Web Application Firewall, but designed to run passively in the background to avoid false positives. It has a special test flag so sending 5 requests with the x-arcjet-suspicious: true header will cause the 6th request to be denied. This simulates analysis of a client over multiple requests before it hits the suspicion threshold.
• Rate limiting can be tested by sending more requests than the configured limit. Quite straightforward.
• Bot protection involves multiple signals to decide whether a request should be considered an automated client or not. There’s deeper categorization and bot types, but in a test scenario you’re more interested in what the deny response will be. In that case, the easiest way to test a bot detection rule is to send a request from a client that is always considered automated - curl. Send a request with the User-Agent header set to curl and you should see a DENY response.
• Email validation checks not just the string syntax, but also whether the email address is routable. Is the domain valid? Does it have MX records? So to test this rule you need to specify an invalid email address or one with a non-existent or non-email domain

Automating security testing

There are many tools that can send requests to web applications. The likes of k6 and Artillery can be used, but they are more designed for load testing. Instead, we recommend using Newman. Newman is a CLI that uses the popular Postman Collections format to define request templates, but can also be included as a Node.js library which makes it ideal for scripting test scenarios. 

And if you don’t use Postman, it’s not required. Newman can be used independently.

Testing a rate limit

This example shows the configuration of a fixed window rate limit with a maximum limit of 50 requests over a very short 3 second window:

import arcjet, { fixedWindow } from "@arcjet/next";

const aj = arcjet({
  key: process.env.ARCJET_KEY,
  rules: [
    fixedWindow({
      mode: "LIVE", // will block requests. Use "DRY_RUN" to log only
      window: "3s", // 3 second fixed window
      max: 50, // allow a maximum of 50 requests
    }),
  ],
});

We could then test it with this Postman collection template JSON:

{
  "variable": [{ "key": "baseUrl", "value": "http://localhost:8080" }],
  "item": [
    {
      "name": "/api/high-rate-limit",
      "item": [
        {
          "name": "Test high rate limit",
          "request": {
            "url": "{{baseUrl}}/api/high-rate-limit",
            "header": [
              {
                "key": "Accept",
                "value": "application/json"
              }
            ],
            "method": "GET",
            "body": {},
            "auth": null
          },
          "event": [
            {
              "listen": "test",
              "script": {
                "type": "text/javascript",
                "exec": [
                  "pm.test('returns proper status based on iteration', () => pm.response.to.have.status(iteration < 50 ? 200 : 429))"
                ]
              }
            }
          ]
        }
      ]
    }
  ],
  "event": []
}

Using the Newman CLI to execute the test, we would expect to be under the limit with a 200 response status for 50 iterations. The 51st iteration should then return a 429 response.

npx newman run tests/high-rate-limit.json -n 51

This works nicely from the terminal, but can also be converted into a test script using the standard Node.js test runner.

import { after, before, describe, test } from "node:test";
import assert from "node:assert";
import { fileURLToPath } from "node:url";
import { promisify } from "node:util";

import { run } from "newman";

// Promisify the `newman.run` API as `newmanRun` in the tests
const newmanRun = promisify(run);

describe("API Tests", async () => {
  test("/api/high-rate-limit", async () => {
    const summary = await newmanRun({
      collection: fileURLToPath(
        new URL("./high-rate-limit.json", import.meta.url),
      ),
      iterationCount: 51, // 50 are allowed, so 51 trigger the rate limit
    });

    // The `summary` contains a lot of information that might be useful
    // console.log(summary);

    assert.strictEqual(
      summary.run.failures.length,
      0,
      "expected suite to run without error",
    );
  });
});

Place this file in tests/api.test.js alongside the high-rate-limit.json file, then execute it with: node --test

This snippet assumes the application server is already running. You can see a full example on GitHub that includes starting the API server as part of the test suite.

Conclusion

By weaving security rule testing into your functional testing process, developers can deliver software that's not just functional, but also more secure. And avoids breaking production!