routerd architecture overview

Web Console Connections summarizes local IPv4 SNAT entries by translated address before applying entry display limits. DS-Lite rows use the resource's observed innerLocalIPv4; shared addresses represent combined state, not unique tunnel attribution. Missing observations remain unknown, never an assumed zero.

This document is an introduction to routerd for operators and contributors. It covers the design intent and the major moving parts. For day-to-day usage, start with the tutorials and the how-to guides. For resource definitions, see the API reference.

Configuration replacement, runtime activation, generation persistence and response delivery are separate outcomes. Their failure and cancellation contracts are described in Apply and render.

---

1. Where routerd fits

routerd is a declarative router framework. Its goal is to let you build a home router, a SOHO router, or a small data-center edge router from the same set of primitives.

The three deployment targets we design for:

Target	Scope	Required tier
Home-router replacement	One host, one or two uplinks, one to three LAN VLANs	H
Hypervisor SDN router	VXLAN / EVPN / underlay routing inside a cluster	C
Kubernetes cluster edge	Advertise Pod CIDR / LoadBalancer IP via BGP, terminate ingress	S → C

All three are expressible with the same declarative primitives. The applicable feature set scales with the deployment.

1.1 Capability tiers

Tier	Use case	Headline features
H (Home)	Home or small office	WAN acquire (PD/RA/PPPoE/DHCPv4/DS-Lite), LAN service (RA/DHCPv6/dnsmasq), NAT44, firewall, EgressRoutePolicy
S (SOHO/branch)	Several sites with VPN	+ WireGuard / IPsec, VRF, dynamic routing across VPN, commit-confirmed
C (Campus / small DC)	Tens of nodes	+ EVPN-VXLAN, iBGP RR, BFD, RouteMap DSL, richer routing policy
E (Enterprise / SP)	Hundreds of nodes	+ Full BGP, MP-BGP L3VPN, segment routing, HA leader election

The primitives are the same from H to E. Higher tiers add more routing and policy controllers on top of the same model.

---

2. Runtime environment

2.1 Deployment shape

routerd targets virtual machines. Embedded appliances are out of scope for now.

Requirements for the host environment:

• virtio NICs (vmxnet, ne2k, etc. are out of scope)
• No dependency on privileged kernel modules (DPDK / XDP optional, host passthrough not required)
• Console + SSH for operations
• For lab work, snapshots and clones are encouraged

2.2 OS strategy

routerd is designed to be cross-OS. The same binary and the same configuration target multiple operating systems.

OS	Strengths	Role
Linux (Ubuntu / Debian)	systemd standard, easy to obtain, recent kernels	Primary platform for development and production
FreeBSD	Stable base, small footprint, jail isolation	Long-running and low-resource deployments

OS-specific differences are absorbed in the pkg/platform layer. Mappings such as nftables ↔ pf, systemd-networkd ↔ rc.conf, and systemd unit ↔ rc.d script are owned by per-OS renderers.

Versioning policy: routerd uses date-and-time-based release versions in vYYYYMMDD.HHmm format; the previous 0.x.y and yyyymmdd.N pre-release numbering is discontinued.

---

3. End-to-end picture

┌─────────────────────────────────────────────────────────────────┐
│ User                                                              │
│   /etc/routerd/*.yaml  +  routerctl CLI                          │
└─────────┬─────────────────────────────────────────┬───────────────┘
          │ inotify                          HTTP+JSON
          │ (notify only)                    (explicit apply)
          ▼                                         ▼
┌─────────────────────────────────────────────────────────────────┐
│ routerd (1 binary, multi-OS)                                      │
│                                                                   │
│   ConfigWatcher ──notify only──▶ Bus                              │
│   ConfigLoader ◀──explicit trigger───── routerctl apply           │
│                                                                   │
│   ┌──────────────────────────────────────────────────────────┐   │
│   │ Bus (in-process channel + persistent SQLite event log)    │   │
│   │  topics: routerd.<area>.<subject>.<verb>                  │   │
│   │  cursor: events.id (autoincrement)                        │   │
│   │  fanout: subscribe pattern match → controller channel     │   │
│   └─────┬─────────────────────────────────────────────────────┘   │
│         │                                                         │
│         ▼ Controllers (in-process reactors)                       │
│   PrefixDelegationCtrl / LANAddressCtrl / RAAnnouncerCtrl         │
│   DNSAnswerCtrl / DNSResolverCtrl / FirewallCtrl / RouteCtrl      │
│   EgressRouteCtrl / ServiceLifecycleCtrl / ConfigLoaderCtrl       │
│   EventRuleEngine / DerivedEventEngine                            │
│         │                                                         │
│         ▼ SQLite state DB (objects/events/artifacts/generations)  │
└─────────┬─────────────────────────────────────────────────────────┘
          │ Unix socket HTTP+JSON                fsnotify (lease/snapshot)
          ▼                                            ▲
┌─────────────────────────────────────────────────────────────────┐
│ Layer 1 source daemons (one process each)                         │
│   routerd-dhcpv6-client / routerd-dhcpv4-client                   │
│   routerd-pppoe-client / routerd-dns-resolver                     │
│   routerd-healthcheck@<resource> / routerd-firewall-logger        │
└─────────────────────────────────────────────────────────────────┘

---

4. Resource model

routerd configuration is a set of resources. The shape is similar to Kubernetes but the apiVersion hierarchy and the controller plumbing are simpler.

- apiVersion: net.routerd.net/v1alpha1
  kind: DSLiteTunnel
  metadata:
    name: ds-lite-primary
  spec:
    aftrFQDN: gw.transix.jp

4.1 Major apiVersions

apiVersion	Responsibility
net.routerd.net/v1alpha1	Networking (Interface, IPv4Static, DSLite, PPPoE, EgressRoute, HealthCheck, etc.)
dns.routerd.net/v1alpha1	DNS (DNSZone, DNSResolver, DHCPv4Reservation, etc.)
firewall.routerd.net/v1alpha1	Firewall (FirewallZone, FirewallPolicy, FirewallRule, NAT44Rule, etc.)
system.routerd.net/v1alpha1	OS bootstrap intent and overrides (Package, SysctlProfile, WebConsole, etc.); host runtime artifacts are derived from resources
control.routerd.net/v1alpha1	controller chain and routerctl control surface

The full list is in the API reference.

4.2 Cross-resource references

When one resource refers to the status of another, use a typed *From field instead of a literal value.

- apiVersion: net.routerd.net/v1alpha1
  kind: WebConsole
  spec:
    listenAddressFrom:
      resource: Interface/mgmt
      field: ipv4Addresses
    port: 8080

addressFrom, ipv4From, ipv6From, prefixFrom, rdnssFrom, gatewayFrom, and upstreamFrom follow the same shape. Dependencies (dependsOn) use the same mechanism.

A *From reference whose target has not published a value yet is a normal bootstrap condition, not an error: the consuming controller reports the resource as Pending (with a reason naming the unresolved reference) and re-reconciles when the referenced status changes — no explicit dependsOn is required. For example a DNSResolver forward source whose upstreamFrom points at a DHCPv6Information server stays Pending until that server learns its DNS servers, then becomes Applied on the next reconcile. A source that declares no upstream at all (neither upstreams nor upstreamFrom) is a real misconfiguration and is rejected by validation instead.

For details, see resource model and state and ownership.

DNS forwarding separates ordinary per-upstream queryTimeout from background health-check deadlines. The optional positive duration inherits healthcheck.timeout when absent (default 3s). Only explicitly listed candidates in the same matching forwarder are tried; valid NXDOMAIN is preserved. Exhaustion or DNSSEC validation failure returns uncached SERVFAIL, never a fabricated negative answer or a fallthrough into a public source. Sequential candidate deadlines add up, so an outer resolver needs enough time for the inner chain. The NextDNS example uses a one-second deadline and an explicit Cloudflare DoH alternative, which does not inherit NextDNS filtering/logging policy.

---

5. Event bus and controller chain

routerd combines an in-process event bus with a set of controllers to converge to the desired state declared in configuration.

5.1 Event bus

• in-process channels backed by a SQLite event log for persistence
• topics use the pattern routerd.<area>.<subject>.<verb> (for example routerd.dhcpv6.bind.changed)
• subscribers receive events via pattern match
• every event has an events.id cursor so re-evaluation is possible after a restart

5.2 Controller chain

Every controller follows the common framework.FuncController shape:

• Subscriptions: topics this controller cares about
• Bootstrap: one-shot initialisation at startup
• PeriodicFunc: idempotent periodic re-evaluation
• ReconcileFunc: state convergence on event arrival

The eventedStore wrapper persists current status and emits routerd.resource.status.changed for meaningful resource changes, which downstream controllers consume to resolve cross-resource dependencies. Routine observation timestamps and counters are excluded from the event comparison where the resource defines them as diagnostics.

For ARPObserver, packet/probe/scan counters and observation timestamps remain available in status. Refreshing observedClients[].seenAt for the same client does not create another status event. Client additions/removals, IP/MAC/source changes, observer configuration changes, and health/error transitions still emit events. This projection affects event comparison only: it preserves the full stored diagnostics and the separate ARP observation/request/probe events used for discovery and lease freshness. Matching EventRule history uses the same comparison, so routine status refreshes do not inflate event counts.

observed.lastCommandProbe is JSON text describing only the last successful probe-target command: sequence, IPv4 target, RFC3339 startedAt and completedAt, and packetsSent. The sequence equals commandProbeCount in the same atomic status snapshot. The record is absent before the first success and resets with the observer process. Cooldown suppression, rejected commands, partial/failed sends, and autonomous probes do not replace it. Later successful commands overwrite it; it is bounded diagnostics, not a command history or a federated request identifier. Collect it with process identity and actual packet captures when correlating a fresh request. Its updates are excluded from status events and EventRule history while remaining available in stored status.

For federated on-demand ARP requests, a ready observer whose source address is the requested target completes selection without sending a probe command. This keeps the request's deduplication claim instead of repeatedly submitting a self-target command that the daemon will reject. Other eligible observers are still probed; their transport failures and the absence of a ready matching observer remain retryable. A refreshed observation can be processed again. The daemon independently retains its source-address and prefix safety checks.

The Linux SAM controller reconciles an empty forwarding intent by removing stale owned rules and per-interface accept_local state. An absent routerd_sam_forward chain is a read-only no-op. iptables-nft 1.8.7 can report an absent chain as incompatible, so that diagnostic requires a successful whole-table listing proving the exact chain is absent. Existing incompatible chains and failed reads remain errors; the controller does not flush them.

Kubernetes edge resources use this status flow directly. IngressService health checks choose an active backend and the NAT renderer uses that status on the next reconcile. BGPRouter / BGPPeer status is observed from the long-lived routerd-bgp daemon with typed ListPeer / ListPath API calls and can lower VirtualAddress VRRP priority through track. BGP config changes are applied to that daemon with GoBGP API objects instead of rendering FRR-style text config or shelling out to reload tools. VirtualAddress and IngressService hostnames feed DNSResolver-served zones as derived A/AAAA records, and BGP/VRRP/Ingress status is also surfaced through routerctl get BGPRouter, routerctl get VirtualAddress, and routerctl get IngressService views plus low-cardinality OTel metrics for transitions and backend health.

5.3 Dynamic SAM topology keeps a safe fallback

Dynamic SAM enrollment is a functional-core boundary: an RR admits a client-authored leaf claim (authenticated by joinTokenFrom when configured), projects a typed SAMRRSet, and a leaf's existing transport and BGP controllers apply that snapshot. A policy may bundle an optional direct leaf SAMPeerGroup in the same dynamic-config part. This does not replace the RR topology. The leaf retains its RR peers, and direct imported routes have a higher local preference only while their direct BGP session is established. Missing, incompatible, or unreachable direct peers therefore fall back to the RR path without a special recovery controller or a destructive topology update.

For direct topology, the leaf sends a digest of its current claim and requires every configured RR to echo that same accepted-claim digest and the same peer group. It also sends an identity digest that covers only client-authored claim material, not RR-owned expiry or revocation fields. A joinTokenFrom policy authenticates that material with its join HMAC; a token-less policy is trusted control-plane input, not cryptographic admission. On a direct-refresh GET, an RR that lost its volatile admission row can explicitly report that this client identity is absent. The leaf re-submits only when every other RR either still attests that identity or gives the same identity-aware absence response. A revoked identity, a different active identity, an unreachable or ambiguous legacy RR, or any mismatched topology leaves the RR-only fallback in place. Revoked client identities are retained in a durable identity ledger, so replacing an expired dynamic row cannot make an old identity replayable. A stale or older RR can still refresh the RRSet, but cannot authorize the higher-preference direct path.

5.4 Daemon contract

Long-running OS processes (DHCPv6 client, DNS resolver, healthcheck, etc.) live as daemons rather than as controllers. Each daemon talks to the controller chain over a Unix domain socket using JSON, and persists its own state under files such as lease.json.

For details, see reconcile loop behaviour.

---

6. Operating the configuration file

The routerd configuration file (default /usr/local/etc/routerd/router.yaml) is rolled out as follows.

edit → routerctl validate → routerctl apply
                              │
                              └─ observe host state
                                 → plan
                                 → render host artifacts
                                 → record state and exit

routerd serve
  → consumes state/events
  → starts / enables / reloads managed daemons
  → updates OS state (nftables / netlink / systemd) continuously

We strongly recommend keeping the configuration in git. Apply changes to production via routerd; do not run ad hoc commands such as nft add rule, ip route add, or sysctl -w directly on the host. Ad hoc changes are either reverted by the next reconcile or, worse, create drift between the routerd state DB and what the kernel actually has.

The right response to drift is to express the new desired state in configuration and apply it again. apply must return quickly and hand daemon lifecycle to the controller runtime; the long-running serve process keeps the configuration ↔ state DB ↔ OS state triangle aligned.

---

7. Observability and debugging

routerd exposes its operating state through several surfaces.

• routerctl get status — phase per resource
• routerctl describe <kind>/<name> — spec, status, and recent events for one resource
• routerctl get events --topic <pattern> --resource <kind>/<name> — tail the bus
• routerctl plan — preview the running daemon’s plan for the canonical configuration; use routerctl plan -f candidate.yaml --replace for a complete candidate file
• Web Console (default http://<mgmt-ip>:8080/) — summary, events, connections, clients, firewall, configuration in a browser
• journalctl -u routerd.service -f | grep "routerd event" — bus events through the systemd journal

Logs are persisted across four databases by purpose: events.db (controller-driven), dns-queries.db (DNS resolver), traffic-flows.db (conntrack/pf), and firewall-logs.db (NFLOG/pflog). For details, see log storage.

OpenTelemetry export is configured by the Telemetry resource in observability.routerd.net/v1alpha1. routerd does not bundle an OTLP collector. When an endpoint is declared, generated systemd, NixOS, and FreeBSD rc.d units receive the matching OTEL_* environment variables and the existing SDK path sends logs, metrics, and traces to that endpoint.

Managed client supervision belongs to the serve lifetime, not to one controller generation. An unchanged client survives runtime reload; a completed supervisor must be replaced even when its desired spec is unchanged. VIP readiness gates initial admission. Retention during degradation requires saved Ready/advertised MASTER publication for the same address and interface; role demotion withdraws it. vipPublicationConfirmed retains that history independently of observation errors. Demotion, observed VIP absence, and failed publication invalidate it. Installers preserve current generated routerd units and replace obsolete controller-chain units explicitly.

---

8. Related documents

• What is routerd
• Resource model
• Design philosophy
• Apply and render
• State and ownership
• Reconcile loop
• State database operations
• API reference v1alpha1
• Plugin protocol
• Supported platforms