Blog

A thing I didn’t expect from putting a price on individual articles: it changed what I wrote.

Not immediately, and not in the direction I assumed. I thought I’d start writing for machines — denser, more structured, easier to extract. What actually happened is that I got much more suspicious of posts that existed to fill a slot.

When everything is free and monetised by attention, the marginal listicle is close to costless. It might catch something. It costs you nothing but an afternoon. So you write it, and it sits there, and over a few years a third of your archive is stuff you’d be mildly embarrassed to have quoted back at you.

Attach a price to each piece and that calculus inverts, because now every article makes a claim about itself. Half a cent is not much money. It is a surprisingly effective filter on self-delusion. I found myself looking at drafts and asking whether I’d charge anyone for this, even trivially, even a machine — and the answer for a lot of them was no, and those drafts didn’t get published.

The second effect took longer to notice. Original material — the stuff where I’d actually done the work, run the thing, collected the numbers — started getting fetched at a completely different rate than commentary. In hindsight this is obvious. If an agent is answering a question, a post summarising three other posts is worth nothing to it; it can reach the sources directly. What it can’t get anywhere else is the thing only you did.

Which is the most useful signal I’ve gotten from any analytics in years, and it arrived as a side effect of billing rather than as a metric anyone designed. Page views told me what got shared. This tells me what was needed.

I’m wary of over-reading a small sample. My archive isn’t big and my traffic isn’t representative of anything. But the direction is stark enough that I’ve stopped treating it as noise: the pieces where I did primary work earn, and the pieces where I had opinions about other people’s primary work don’t, and no amount of the second kind adds up to the first.

There’s a version of this that turns into a lecture about “just write better content”, which I want to avoid, because it’s smug and also not actionable. The point is narrower and more mechanical. A pricing signal is feedback, and we’ve spent twenty years running publishing on a feedback loop that measured attention instead of usefulness. Those aren’t the same thing and everyone has always known they aren’t, but nobody had the other number.

Now there’s another number. It’s small and noisy and I don’t fully trust it yet. It’s still the first one in a while that told me something I didn’t already know.

robots.txt is a sign, not a fence. Everyone technical knows this and quite a lot of people writing about AI licensing seem to have forgotten it.

The file has no enforcement. It is a request, served publicly, that well-behaved clients honour because honouring it is cheaper than the reputational cost of not doing so. That worked for search engines because search engines had something to lose: a brand, a business relationship with publishers, and a strong incentive to stay welcome. The convention held for thirty years on nothing but mutual interest.

Take away the mutual interest and you’re left with a text file.

This is why the current wave of “add these lines to block AI crawlers” advice sits badly with me. The lines work on exactly the actors who would have asked politely anyway. The ones you’re worried about either ignore the file, or rename their agent, or fetch through a residential proxy pool that looks like a person in Ohio. You have not blocked anyone. You have documented your preferences for the benefit of people already inclined to respect them, which is fine, but it is not a control.

There’s a deeper problem with the block-first framing, which is that it assumes the interaction is inherently adversarial. Sometimes it is. Often it isn’t. Someone’s assistant fetching your post to answer their question is a reader arriving by an unusual route. The thing you actually object to is not being read — it’s being read for free at scale by something that will never come back, never subscribe, and never see an ad. That’s a payment problem wearing a permissions costume.

Permissions and payment are different mechanisms with different failure modes. A permission says yes or no and has to be enforced by someone. A price says how much and enforces itself, because the thing on the other end either pays or doesn’t get the bytes. You don’t need to identify the caller, you don’t need to trust their user agent, and you don’t need a lawyer. You need to be able to answer a request with terms instead of content.

I’d go further: the identity question is a trap. Every scheme I’ve seen that starts with “first, reliably identify the agent” ends up either trivially spoofable or requiring a registry that someone has to run and everyone has to trust. Whereas a request that arrives with a valid payment has proven the only thing you actually needed to know. It doesn’t matter who they are. They paid.

None of which means robots.txt is useless. It’s a good sign. Keep it, keep it accurate, and let it say what you’d like to happen. Just don’t confuse the sign with a lock, and definitely don’t build a business model on the assumption that everyone reads signs.

The uncomfortable version of this: if your content is reachable over HTTP without payment, it is free, and no amount of policy text changes that. The only question is whether that’s a decision you made or one that was made for you.

Check your access logs for the last week. Not the analytics dashboard — the raw logs. Count how many requests came from something that never rendered a page.

For most sites I’ve looked at, the answer surprises the owner. Not because the number is enormous, though sometimes it is, but because none of that traffic appears anywhere they normally look. Analytics runs on JavaScript. Agents don’t run JavaScript. So an entire category of consumption happens in a blind spot, and the blind spot is growing faster than the part you can see.

The instinct is to call this a crawler problem, and it isn’t, quite. Crawlers have been around forever and we made our peace with them: Googlebot takes your page and sends you readers, and that trade was good enough that nobody thought about it much. What’s new isn’t crawling. It’s that a growing share of the fetches are made on behalf of a specific person, right now, and the answer goes back to that person without them ever arriving.

Those two things look identical in your logs. They are completely different economically. The first is a loan against future traffic. The second is a sale you didn’t get paid for.

I keep meeting people who want to solve this by blocking. It’s an understandable reflex and I think it’s mostly wrong, for a boring reason: the traffic you’d block is genuine demand. Someone wanted your analysis badly enough that their assistant went and got it. Turning that away is not a strategy, it’s a sulk. And the practical result of a blanket block is that you disappear from the surface where people are increasingly asking their questions, which is a strange victory to celebrate.

The other reflex is to do nothing and hope the licensing deals arrive. Some of them will. If you’re a national newspaper, a large lab will eventually send a lawyer and a number. If you write a good technical blog with four thousand readers, nobody is sending you anything, ever. The deal-making tier is real and it is very short.

So the useful question is what a small publisher can do unilaterally, without a lawyer, without a negotiation, and without punishing the humans who show up. I think the answer has to have three properties. It has to be per-request, because that’s the granularity the consumption actually happens at. It has to be automatic, because nobody is going to manually approve fetches. And it has to leave people entirely alone, because the moment a reader hits a wall you’ve traded a real relationship for a rounding error.

Which lands you, whether you like it or not, at something like a toll: machine asks, machine pays, person reads free, and the whole thing settles without either party filling in a form.

The part I’m least sure about is price. Everyone’s first instinct is to price like a subscription divided by expected reads, which produces numbers that are either insultingly small or high enough that agents route around you. My current guess is that the right price is closer to “annoying to ignore” than “meaningful revenue” for a single read, and the revenue only becomes interesting in aggregate. But I’ve been wrong about pricing before and I’d rather see the data.

What I’m confident about is the accounting. Right now that traffic is worth zero to you by construction — not because it has no value, but because you have no way to charge for it. That’s a tooling gap, not a market verdict.

HTTP has had a status code for money since 1997. Nobody used it.

402 Payment Required shows up in RFC 2068 with a single line of explanation — reserved for future use — and it stayed that way through RFC 2616 and again through RFC 7231. Twenty-odd years of every web server on earth shipping a code that meant “pay me” and no agreed way to say what, how much, or to whom. The plumbing existed. The protocol on top of it never did.

The reason is not mysterious. A status code is only useful if the thing receiving it can act on it, and for most of the web’s life the thing receiving it was a person looking at a browser. People do not respond well to a machine-readable payment challenge. They respond to a checkout page, which is why we got checkout pages, and why 402 sat in the registry like a spare key to a house nobody had built yet.

What changed is the client. A meaningful and growing share of requests to any site with useful writing on it are now made by software acting for someone else — a research agent, an assistant answering a question, a crawler feeding a model. That software does not need a checkout page. It needs a price, a payee, and a way to prove it paid. Which is precisely the shape of a 402 with structured terms in the header.

I want to be careful about the framing here, because there is a version of this argument that is really just paywalls with extra steps. That version is uninteresting and mostly bad. The interesting version is narrower: a machine asked for something, a machine can be charged for it, and the person reading your site stays free forever. Those are separable, and the separation is the whole product. If your toll can’t tell a person from an agent, you don’t have a toll, you have a paywall that occasionally embarrasses you.

The classification problem is real and it is not solved by user-agent sniffing, whatever anyone tells you. User agents are a courtesy. They are trivially spoofed and routinely wrong, and the failure modes are asymmetric in a way that should determine your whole design: charging a bot that claimed to be a browser costs you a fraction of a cent in missed revenue, and charging a person who happened to look like a bot costs you that reader permanently. Bias hard toward free. Be wrong in the cheap direction.

What makes this workable now, and did not before, is that the payment leg finally has an answer that isn’t “sign up for an account”. Stablecoin transfers settle in seconds for fractions of a cent, which means a per-article price of half a cent is not absurd overhead — it’s a real transaction. Micropayments failed in 2001 because moving a nickel cost a dime. That arithmetic has changed, and most of the objections to the idea are still arguing with the old numbers.

I don’t think this replaces advertising or subscriptions. It covers a case neither of them handles: a machine reads your work once, cites it, and leaves. No page view to sell, no relationship to convert, no return visit to bank on. Under the old models that read is worth exactly nothing to you, which is a strange thing to accept about the fastest-growing category of traffic on your site.

402 was reserved for future use. It’s mildly funny that the future turned out to be the part of the web that isn’t human.