How we run Arcjet like an open source project

Arcjet is a remote-first company. We're developing an SDK that streamlines bot detection, attack prevention, and spam protection for developers, so building a startup culture that matches the users of the product is an advantage.

Even though our core service is not open source our SDK and docs are, so we've adopted many of the workflows that are used in open source. We're a remote team and as several of our team members have a long history of open source contributions, our practices have evolved from how many open source projects operate. We're not quite requiring contributions go through git email, but remote working requires systems and tools to make it work well.

For example, we track feature requests, ideas, bugs, and technical debates in GitHub issues. Code comments reference issue IDs (e.g., // TODO(#123): something something), and we use draft PRs to solicit feedback before formal reviews.

Over the last 16 years I’ve led three startups remotely - Server Density (acquired 2018), Console.dev (a devtools newsletter) and now Arcjet. In this blog post I'll talk through some of the processes we're using to make remote engineering work at a devtools startup.

Real-time sometimes, asynchronous most of the time

Slack aimed to replace email as a single workspace for all communication, but in practice it fragments attention and search is unreliable.

Following the 37signals Guide to Internal Communication recommendation that unrestricted group chat can overload teams, we set clear boundaries.

• Slack for socializing, quick clarifications or discussions, and link-sharing.
• No expectation of immediate replies.
• If it will matter after today, all lasting ideas, decisions, and conclusions migrate to GitHub issues/PRs or Notion pages.
• We auto-delete Slack history after 7 days to enforce permanent records elsewhere.

Over the past two years, we shifted from GitHub Discussions to GitHub Issues due to notification limitations. Engineering questions, bugs, and feature ideas are all tracked via an issue. For code-in-progress, Pull Requests are used for context-specific discussions and PRs are often opened in draft first for early feedback. The principle here is that everything has a permanent link we can reference in future. This has worked well in general, but big PRs and extensive comments easily get lost in the GitHub web UI.

Notion is for design notes, policies, research, and general information that doesn't belong in a PR. Notion is like a powerful wiki, however version history and comment threading can be cumbersome, causing duplication between Notion pages and GitHub discussions (e.g., architecture feedback appearing in both).

We’ve recently started capturing more structured Architectural Decision Records as a way to improve this. The advantage of committing ADRs to the codebase is the ability to use PRs and commit history for tracking. However, discovery is more of a challenge and we've not yet solved the problem of information existing in multiple places e.g. design notes in Notion and decision records in GitHub.

I am optimistic about using Notion’s AI integration with GitHub, Google Docs, and Slack, to answer questions and summarize discussions. There’s a lot of information embedded across several tools, so it seems the perfect AI use case.

Stripe's internal email groups

A decade ago, Stripe wrote about how they use internal email lists to improve transparency. They provided more detail on their methodology with a post a year later and even built a custom tool for managing Google Groups.

Stripe was trying to solve the problem where email is directed to individuals. Using a group meant that you could search all past history and get a link to any specific thread. However, their tool was achieved in 2019 and I couldn't find anything recent about whether the approach still works. I know they use Slack, but has that replaced email?

We continue using email externally and use group addresses for finance, operations, hiring, and marketing. Almost all external emails are CC’d to the internal group to maintain transparency. This has proven useful as we hire people into roles that I was previously involved with - the handover is much easier when context is contained within a searchable group.

Internal weekly update emails

We also introduced a weekly email to an internal-only updates mailing list. Every Friday everyone sends an email to the list answering three questions:

1. What did you ship? Briefly explain the main thing you shipped this week. “Shipped” = merged to main and deployed to production. Dependency updates don’t count unless there was major migration work.
2. What did you work on? A couple of bullet points of what else you worked on, not including what you shipped.
3. What did you learn or find interesting? A couple of sentences explaining one or two things you found interesting, with links. Must be related to Arcjet in some way.

This keeps everyone up to date and you can read the updates at your leisure. I particularly enjoy reading everyone's responses to the final question, which I adapted from Jensen Huang’s Top 5 Things (T5T) emails. Sometimes an update gets no replies, but there's often discussion with people replying to ask about or add to these points.

So far Google Groups has worked well (it’s a shame GitHub Discussions wasn’t better).  Each person can subscribe how they like (every email, summaries, digests), they’re searchable, and there’s a link to each message. However, the web interface is outdated, threading doesn’t work perfectly, and external spam protection seems poor compared to regular Gmail. WordPress P2 might be an option, but it feels too much like it’s a hacked together blog.

Remote-first, in-person regularly

The remote-only purists are wrong. So are those who say everyone must be in the office all the time. There’s always been a balance to strike and there are plenty of examples of both styles working well. It comes down to who is setting principles and defining the culture.

One thing I’ve learned is that you can create a remote-first culture and then add an in-person or in-office element later, but not the other way around. The most important thing is to have the right tools and systems in place.

If the default assumption is that everyone is in the office then these systems don’t get fully developed and important information gets lost. The result is the remote team being left out, the whole system failing, and then you get “return to office” mandates. The solution is to always assume that everyone is remote and record every discussion and decision.

But working in-person works well for coming up with ideas and iterating quickly. I’ve always found a big difference before and after people meet for the first time. Once you’ve met in-person, it’s a lot easier to work together remotely.

Arcjet is set up as a remote-first, distributed company (US and Western EU timezones to ease collaboration). However, we aim to do in-person meetups 2-3 times a year. It’s not cheap to organize, but remote working has never been about saving costs - the money you save on an office goes into travel. We’ve run these multiple times in London, New York, and Las Vegas (for Defcon) and are planning other varied locations for the next ones. The main challenge is finding somewhere to work from together - comfortable chairs and good wifi are hard to find!

Key principles

Open-source-inspired workflows suit our developer-focused security product by improving transparency and feedback loops. Our team has more context to help make good decisions because we can search through past work to understand the current state of things.

To make this work we follow a few key principles:

1. Document everything in linkable, permanent repositories.
2. Favor async workflows; use real-time sparingly.
3. Invest in periodic in-person meetups to build rapport.

The tools have never been better, and I expect AI to make linking systems more effective. It'll be interesting to look back in another 10 years and see what else has changed.