So I had an agent doing work on my VPS on Tuesday evening, and about ten minutes after it finished, the gas sponsor for the arcade games stopped working. My first thought was not a generous one. It was more along the lines of right, that is what I get for letting an agent near production, what has it broken this time.
It had broken nothing. The timing was a coincidence, and a really unhelpful one. The agent had been rebuilding a snapshot exporter container in a completely different part of the estate, and the thing that actually went down was Sui's testnet fullnode dropping gRPC-Web. I spent the first ten minutes of that investigation looking in exactly the wrong place, because the timeline pointed straight at me.
I think that is worth writing up on its own, before any of the technical bit. When you work with agents on live infrastructure, the correlation between "an agent touched something" and "something broke" is going to fire a lot of false positives. The container had been up for 28 hours. I could have checked that in about four seconds and saved myself the detour, but I did not, because the story in my head was already written.
Three things broke at once
What I actually had was three separate failures with one cause, and I only spotted them one at a time, which made it feel worse than it was.
The gas sponsor went first. That is the little service that pays the Sui gas so people can play the arcade games without needing testnet SUI in their wallet. It went from working to /health returning a 500, and the error was unexpected response content type: application/json. Last good sponsorship was 14:44 UTC, first failure at 19:03:58 UTC, so it had been sitting there quietly broken for a while before anyone tried to play anything.
Then the Grafana dashboard showed treasury reconciliation as no data, which I had never seen before. Same error in the logs. That is a separate container doing a completely different job, so at that point it stopped looking like one broken service and started looking like something underneath both of them.
The third one was the worst, and I only found it because I went and tried to start a ranked run myself. The browser could not even build a transaction. Not sponsored, not unsponsored, nothing. The console was full of CORS errors pointing at fullnode.testnet.sui.io. That also explained why the fallback to player-paid gas had not saved anyone: the failure happens when the transaction is being assembled, which is before the sponsor is ever contacted, so there was nothing to fall back to. Both routes needed the same dead connection.
The evidence that fooled me
Here is the bit I got wrong, and I would rather write it down than quietly not mention it.
Sui have been removing JSON-RPC from their public fullnodes. That is real, it has been announced for months, and the dates were roughly now. When gRPC-Web started failing, the thing it returned was literally the JSON-RPC deprecation message. Not a 502, not a timeout, not a connection reset. A tidy JSON body saying JSON-RPC on public fullnodes has been deprecated, please migrate.
So the picture in front of me was: a scheduled deprecation is happening, and the endpoint I depend on is now answering with the deprecation notice. I also checked mainnet, and mainnet was still serving gRPC-Web fine. Testnet and devnet were down, mainnet was up. That fitted a staged per-network rollout exactly, because that is how they had done the JSON-RPC removal.
Every one of those observations was correct. The conclusion I built on top of them was not. gRPC-Web was not being deprecated. What had most likely happened is that the gRPC-Web routing fell through to the JSON-RPC handler while that handler was being taken out, so the deprecation body was collateral rather than a message aimed at me. It was an outage wearing a deprecation notice.
The honest failure here is not that I read it wrong. Given what was on screen I think that was a reasonable read, and I will come back to why in a minute. The failure is that I wrote it down as fact. It went into commit messages, into a decision log, and worst of all into a public repo's changelog telling other operators to expect the same thing on mainnet and upgrade before it lands. That advice was wrong and it was addressed to people who are not me.
What I built while being wrong
The fix split three ways, because the three broken things have genuinely different constraints.
The two backend services, the gas sponsor and the reconciliation collector, are Node processes on my Hetzner VPS. Node can speak native gRPC over HTTP/2, so those just moved off gRPC-Web entirely. That is the change I am happiest with, and it would have been worth doing regardless of the outage, because it drops a translation layer and moves onto the transport Sui treat as primary. Two small releases, and both services have been fine since.
The browser could not do that. Browsers have no access to HTTP/2 trailers, which is the whole reason gRPC-Web exists as a separate thing. So the fix there was an Envoy proxy on the same VPS: it accepts gRPC-Web from the page, speaks native gRPC to Sui's own fullnode, and hands the answer back. Sui still serve all the data. No third party sits in the middle of anyone's transactions, which mattered to me, because the alternative was pointing everyone's browser at some other RPC provider and I did not fancy that for a service that signs things.
Two traps in that, if you ever build one. Envoy forwards its own hostname as the authority by default, and Sui's edge routes on Host, so every single call came back as grpc-status 12 with the message "fault filter abort", which tells you absolutely nothing. It needs an explicit host rewrite. The other one is that the proxy sits behind an existing Node service, and that service read upstream responses as text. gRPC-Web frames are binary with the trailers packed into the body, so reading them as text quietly corrupts them. That needed a proper binary passthrough.
There was also a security thing I got wrong and had to go back and fix, which is a bit embarrassing given I had already told myself it was done. I put an origin allowlist in Envoy, tested it, watched an unknown origin get refused, and moved on. Cloudflare's edge then appends its own permissive header on the way out, so the allowlist was not actually enforced. Withholding a response header is not a control when something downstream can add it back. It is enforced properly now by refusing the request outright, which is the only version that survives the edge.
Sui Discord, and someone else building the same thing
I was on Discord voice while all this was going on, and it turned out a few of us were hitting it at the same time. Nobody really knew what was happening for a while. That is genuinely useful information when you are three hours into something and starting to wonder whether it is you.
After maybe an hour and a half, the word on the Sui Discord was that it was an outage and it was back up. Which it was. gRPC-Web is working on testnet, devnet and mainnet right now. If I had gone and made a cup of tea instead, the problem would have solved itself.
The bit I found most reassuring came later. Another builder, Hecate, hit the same outage at the same time and independently landed on exactly the same solution, an Envoy proxy on his own VPS. He has kept his running too. I take some comfort from that. When two people looking at the same evidence separately build the same thing, the read was probably not stupid, even if the conclusion drawn from it turned out to be wrong.
Was any of it a waste
Some of it, honestly, yes. If I had waited ninety minutes I would have written no code at all.
But not most of it, and this is the part I keep coming back to. The backend services moving to native gRPC is a straight improvement that I would not have got round to otherwise. And the browser now has two independent routes to Sui, my proxy first and Sui's own endpoint as a fallback, which are two different edges that demonstrably failed independently on the day. That is real redundancy rather than the diagram kind.
I did have to be careful not to talk myself into more of that than actually exists. The obvious next thought was to point the backend services at the proxy too, for consistency. That would have been false comfort, because the proxy's only upstream is the same Sui fullnode. If Sui go down, both paths go down together. It would have added a hop and a dependency on my own VPS in exchange for nothing. I went looking for a genuine second provider that serves native gRPC and could not find one that works without a paid account, so the backend services are still single-route and I have written that down as a known gap rather than pretending otherwise.
The other thing that came out of it is that Intelligence purchases were broken the whole time too and I had not noticed, because I had been looking at the arcade. Same client, same cause. Worth knowing that a transport-level break is a pattern and not an incident. The right move is to go and grep for every place that constructs one of these clients, rather than fixing the service that is currently shouting at you. I fixed the sponsor, declared it done, and then found the collector hours later because the operator, which in this case was also me, kept reporting symptoms.
What I would do differently
Label an inference as an inference. "The gRPC-Web endpoint is returning the JSON-RPC deprecation body" was something I observed. "gRPC-Web is being deprecated" was something I concluded. Those two sentences went into the write-up looking identical, and only one of them was true. I have gone back and corrected the public changelog with the actual position, because leaving wrong advice out there for other people to act on is worse than the original mistake.
Check the boring thing first. The container had been up 28 hours. That single fact ruled out my agent theory immediately and I did not look at it for ten minutes because I was already sure.
And I would probably still build the proxy. Not because gRPC-Web is going anywhere, it plainly is not, but because a browser that can only reach one edge is one bad afternoon from having no arcade and no purchases, and I have now watched that happen once. The previous time I wrote up an outage here it was a token rotation that I caused myself, so this makes a change, even if I did manage to add my own mistake on top of someone else's.
Anyway, everything is back and has been stable since. If you hit anything odd with the arcade or with Intelligence, let me know and I will take a look.