Field · Fleet Operations

Revoked, still running.

The Cloud Hub era began on October 1. In the eleven days through October 11 the fleet shipped ten desktop releases and twenty-plus Android releases; a ¥125 radio box was found with a WiFi chip slowly wedged by firmware from 2020; a Yaesu FT-891 became the eleventh supported radio; and a Cloud Hub instance that had been revoked was still running, squatting a tunnel port the registry had already given to another callsign. Two of those stories are told in full here. The rest is the ledger.

10 desktop releases · Oct 1–11 (v1.24.2 → v1.25.5) Android 20+ releases in 8 days (v1.0.0 → v1.1.31) W103D ¥125 box · 2021 N9 firmware vendored in 1 revoked instance squatting another callsign's port

By BG1SB  ·   ·  ~14 min read

On October 11 the fleet produced two quiet facts inside one day. The ¥125 W103D radio box — its image re-issued that day with six first-boot fixes — turned out to have a WiFi chip progressively wedged by firmware from 2020: connections dropped after roughly thirty seconds, wireless scans wedged entirely, one sixteen-minute MCU hang. The cure was the manufacturer's own 2021 factory firmware, vendored into the build behind a SHA-256 check and dpkg-divert protection — effective from the next image; after the swap, a 200-packet run measured 0.5% loss and a 6.3 ms average, with zero drops. Hours later the Cloud Hub produced the complement: an instance revoked days earlier was still running, and it was squatting a tunnel port the registry had already reallocated to another callsign. One machine was healed by replacing what the base image had always shipped. One machine was never stopped, because revoking a row is not stopping a process. Between those two facts sit eleven days, ten desktop releases, twenty-plus Android releases, and one new radio.

TL;DR — six findings

  • FactThe ¥125 box's WiFi was a firmware case, not a hardware case. The base image's 2020 N9 firmware progressively wedges the MT7663S: drops after ~30 s, scans that wedge, a ~16 min MCU hang. The factory 2021 N9 firmware is now vendored into the build (SHA-256 verified, dpkg-divert protected) and ships in the next image. After the swap: 200 packets, 0.5% loss, 6.3 ms average, zero drops.
  • Fact撤销≠吊销: a revoked Cloud Hub instance kept running. It squatted a tunnel port that had been reallocated to another callsign. The forensics doc carries the incident's name in its title; three boxes were repaired.
  • FactThe cadence held: ten in eleven, twenty-plus in eight. MRRC Modern shipped ten releases in eleven days (v1.24.2 → v1.25.5). The Android app went from its first public release v1.0.0 on Oct 3 to v1.1.31 by Oct 10. v1.1.29 was built from a dirty worktree; its features actually shipped in v1.1.30.
  • FactSpectrum tiers negotiate full/mid/low per socket: 151 → 38 → 19 kbps. Clients that declare no tier receive byte-identical streams to what they always got — old clients neither break nor change.
  • ThesisIn fleet operations, revocation must stop the thing. A revocation that only deletes a row is a wish until something the process can feel checks it. The registry and the process disagreed; the registry lost.
  • ExcludedNo physical FT-891 sat on the bench in this window. Its tables were sourced offline from Hamlib; the unverified-model TX gate refuses keying by default, and field verification of the new backend remains open.

One day carried both quiet facts

October 11, in order: a box re-issued, a firmware case closed, an overhaul shipped, an incident surfaced.

This article is the second half of the fortnight. The first half — three model sessions, three repositories, twenty-six releases across October 3–8 — is told in Reins Side by Side, which is prior reading and stands on its own. This one starts where that story ends and reaches October 11; it also stands alone.

ItemWhat happened on 2026-10-11
W103D image Image v1.25.5 re-issued with six first-boot fixes (first published 10-07)
WiFi firmware "Firmware day": MT7663S forensics closed — the base image's 2020 N9 firmware out, the factory 2021 N9 firmware vendored into the build (next image)
Admin portal Twenty-task UI overhaul shipped: design system, CSRF everywhere, two-stage confirmations, a no-JS fallback, audit filtering
Revocation A revoked instance found still running, squatting a tunnel port reallocated to another callsign; forensics doc written, three boxes repaired

The same-day pile-up is the point. The fleet's surface — box images, desktop releases, an admin portal, a tunnel registry — grew faster than its stop mechanisms. Two of these rows are centerpiece stories: the firmware forensics, because a year-old factory file was the cure; the revocation incident, because every other row on this table assumed that "revoked" means "stopped". The remaining sections are the cadence and the engineering that carried them.

Fact · every row is dated in the record

Each item in this table traces to a dated entry in the repositories' git history or changelogs: release tags, image build logs, the admin portal's task list, the incident's forensics document. Nothing here is recalled from memory.

The 2020 firmware was wedging the chip

A ¥125 box, an MT7663S, and a cure that had been sitting in the factory's own 2021 image all along.

The W103D is a ¥125 radio box. Its image v1.25.5 was published on October 7 and re-issued on October 11 with six first-boot fixes; that same October 11 became "firmware day" for its WiFi. The MT7663S was failing in a way that looks like dying hardware: a connection would drop after roughly thirty seconds, wireless scans would wedge entirely, and once the MCU hung for about sixteen minutes.

The forensics did not blame the hardware. The base image ships a 2020 N9 firmware for this chip, and that firmware progressively wedges it. The manufacturer's factory image carries a 2021 N9 firmware for the same silicon — so the fix was vendored straight from the factory: the 2021 N9 file went into the image, with two protections welded on. The build verifies the file's SHA-256 at image build time, and dpkg-divert guards it so a routine package upgrade cannot silently put the 2020 file back.

After the swap, the measured run: 200 packets, 0.5% loss, 6.3 ms average, zero drops. The chip was never bad; it was being administered a poison it had no driver-level defence against.

Two boundaries the firmware could not move

The hotspot does not exist in the air

On this board, AP mode sends no beacons. The onboarding hotspot that the software story assumes is literally not on the air, no matter what the configuration says. The first configuration therefore goes over wired Ethernet or HDMI. This is a hardware boundary, recorded as one — no firmware day moves it.

The "open" hotspot was never open

Even where a hotspot could exist, NetworkManager 1.42's passwordless hotspot command silently generates a random WPA password. The "open" hotspot was never actually open: clients were being asked for a password that nobody had been given. The two findings retire wireless onboarding for this board — written into the documentation instead of being discovered by the user.

Inference · a progressive failure teaches a testing lesson

The wedge was progressive: early checks green, late checks red. Any check that watches only the first moments — association succeeded, scan returned — reports health while the chip is already dying. "Boots fine" is not a connectivity verdict for a failure that needs thirty seconds to arrive.

Fact · what was measured, and what was not

The evidence for "fixed" is the 200-packet run: 0.5% loss, 6.3 ms average, zero drops. Nothing longer is claimed here — no overnight soak, no throughput ceiling, no distance test. Those would be worth running; they have not been run at the time of writing.

Revoked is not stopped

撤销≠吊销. The registry deleted a row; the process kept the port.

The Cloud Hub (mrrc_hub) went live on October 1 at portal.mrrc.vlsc.net: callsign self-registration, verified against Club Log's callsign database of 270k+ entries. On October 5 it grew admin authentication and frps tunnel metrics. On October 11 the admin portal's UI overhaul shipped — twenty tasks: a design system, CSRF everywhere, two-stage confirmations, a no-JS fallback, audit filtering.

The same day produced the incident that names this article. An instance that had been revoked days earlier was still running, and it was squatting a tunnel port the registry had since reallocated to another callsign. The forensics document carries the finding in its title: 撤销≠吊销 — revoking the record is not revoking the process. Three boxes were repaired.

What the registry believedWhat the process was doing
Instance revoked; its port free Tunnel still up; port still held
Port reallocated to another callsign The revoked instance's process still squatting on it
Thesis · in fleet operations, revocation must stop the thing

A revocation that only deletes a row is a wish until something the process can feel checks it. Here the registry and the process disagreed, and the registry lost — the row was gone, the port was not. The stop button for a fleet is not a database flag; it is a verification that the thing is actually down.

Inference · "registered is not serving", sign flipped

An earlier retrospective recorded that an frpc registration never meant serving. This incident is the same shape with the sign reversed: a revocation never meant stopping. Both are one class of defect — a database row treated as a live guarantee. The class now has two named cases.

Ten in eleven; twenty-plus in eight

The release trains ran on schedule; the table lists the dated stops.

MRRC Modern shipped ten releases in the eleven days of this window, from v1.24.2 through v1.25.5. The dated ones:

DateReleaseWhat it carried
10-01 v1.24.2 / v1.24.3 The Cloud Hub era begins: edge by default, one entry per callsign — the :9988 port and the /mrrc_modern/<callsign>/ path suffix are gone
10-02 v1.24.4 Whole-fleet migration from the Korea VPS to Hong Kong; single-host layout — one machine, one address
10-03 v1.25.0 Approved Cloud Hub instances connect themselves, with a 30-second self-check; the first-run serial probe can no longer hang
10-04 v1.25.1 Ships its own TLS trust store — CERTIFICATE_VERIFY_FAILED on macOS fixed; the FTDI path fix put the spectrum back
10-06 v1.25.4 macOS orphaned-frpc cleanup: the guard that should have been killing stale tunnels had silently disabled itself, and six orphaned tunnels were found after six launches; shutdown now stops the tunnel child
10-09 v1.25.5 Quick-row buttons answer "where am I now" — MODE/BAND/FILTER/ATT/PRE show current values; spectrum bandwidth tiers AD-025 (§05); the env file collapsed to a single writer/reader (env_store)

The Android app had its first public release, v1.0.0, on October 3; by October 10 it stood at v1.1.31 — twenty-plus releases in eight days. It grew a Cloud Hub onboarding wizard, one-touch CQ, an FT-710-style S-meter, an incremental waterfall, and the same spectrum bandwidth tiers as the desktop.

Fact · one release was not what it claimed

v1.1.29 was built from a dirty worktree; the features recorded under its number actually shipped in v1.1.30. It is named here because the record should say so — a version number that means "whatever was on disk" is a version number that means nothing.

151 → 38 → 19 kilobits

Spectrum tiers AD-025, negotiated per socket; undeclared old clients are byte-identical.

v1.25.5 also answered the operator's first question — "where am I now?" — in the quick row: the MODE/BAND/FILTER/ATT/PRE buttons now show the radio's current values instead of demanding that the user remember them.

The engineering win of the release is spectrum bandwidth tiers, AD-025. A socket negotiates one of three tiers — full, mid, low — and the waterfall payload compresses accordingly: 151 → 38 → 19 kbps. A phone on a metered or weak link trades waterfall richness for smoothness, per connection, without touching the radio path. The same release collapsed the environment file to a single writer/reader pair, env_store: one writer, one reader, no more configuration drift from multiple hands.

Fact · compatibility by negotiation, not by version check

A client that declares no tier receives a stream byte-identical to what it always received. Old clients neither break nor change; the negotiation, not the release notes, carries the compatibility.

Inference · per-socket beats global, because the constraint is per-socket

A global bandwidth setting forces the worst-connected client to dictate for everyone on the same server. Per-socket negotiation lets the metered phone and the desktop on a LAN reach the same instance with different budgets; the socket knows its own constraint.

An eleventh backend, a default refusal

Yaesu FT-891, supported on 10-06: tables offline from Hamlib, transmit gated until verified.

On October 6 the radio registry gained the Yaesu FT-891 — the eleventh backend overall, the fifth from Yaesu. All of its tables were sourced offline from Hamlib. And the unverified-model TX gate refuses keying by default: a radio model whose tables have not been verified cannot transmit until a human says it can.

Excluded · no physical FT-891 in this window

No FT-891 sat on the bench during these eleven days. Its CAT and audio tables are Hamlib-sourced and table-verified, not radio-verified; field verification of the new backend remains open. The TX gate is the other half of the same honesty: where verification has not happened, the default is refusal, not permission.

Two items on the bench

One image, one token scheme.

  • The firmware-fixed image build. The October 11 re-issue of the W103D image already carries the vendored 2021 N9 firmware; the next image build makes that fix the default everyone receives.
  • Hub per-instance tokens. Cloud Hub moves to per-instance tokens, so each instance authenticates as itself rather than as a shared credential.
Thesis · the fortnight's second half closes like its first

A field fact becomes a gate. The box's gate is a hash-checked firmware file that package upgrades cannot overwrite; the hub's gate is still being built — a revocation that stops the thing, not just the record of the thing. The lesson of these eleven days is one sentence: 撤销≠吊销. In a fleet, nothing is stopped until something checks.