Selective Address Mobility

This is not full L2 extension. routerd CloudEdge does not stretch an Ethernet segment into public cloud. Public cloud fabrics do not expose an operator-controlled broadcast domain, and each provider has its own route and address ownership model.

Selective Address Mobility captures selected /32 IPv4 addresses at one side and delivers packets for those addresses to the owning side over a routerd-to-routerd overlay. TCP/IP source and destination addresses are preserved by the abstraction. Firewall and NAT are separate routerd layers, not fields on mobility resources.

Resource Model

For the CloudEdge Mobility control plane, SAMNodeSet is the shared operator-authored identity/topology intent and MobilityPool is the local address/capture intent. The pool declares the logical IPv4 pool, EventGroup, BGP delivery, local capture policy, and provider trap details. The node set is the one shared record of every participant's identity, site, role, placement, and transport endpoint; a pool does not repeat remote members.

The north-star config shape is:

• declare the full shared identity/topology and placement once in a SAMNodeSet, then import it with MobilityPool.spec.membersFrom;
• declare only the self site as a local MobilityPool overlay, including its capture and provider discovery details;
• keep reusable local cloud capture details in profiles.cloudCaptures;
• keep non-secret node-local values in spec.values, then project them with capture.targetFrom and ownershipDiscovery.subnetRefFrom.

SAMNodeSet is the write-once aggregation point for a fabric. It supplies the shared MobilityPool membership fields (nodeRef, site, role, and optional placement/maintenance) as well as Event Federation, WireGuard, and SAM transport identity. Provider references, capture details, discovery selectors, profiles, and static owned addresses remain a local MobilityPool overlay. SAMTransportProfile topology derivation from a node set uses addressingMode: pair-stable so adding a node does not renumber existing tunnel /31 assignments.

SAMNodeSet.spec.nodes[].macAddresses can list static member MAC addresses for the same fabric. On-prem ARP observers use the union of those MAC addresses as an ignore set so ARP frames from routerd members do not become ownership signals for mobile /32 addresses. Editing macAddresses is declarative intent: routerd derives the observer ignore set and reconciles it through the observer socket without requiring an observer or routerd restart. Observer status reports the active ignore set and ignored observation count so convergence is visible.

apiVersion: mobility.routerd.net/v1alpha1
kind: SAMNodeSet
metadata: { name: svnet1-nodes }
spec:
  nodes:
    - nodeRef: pve-rt01
      site: pve01
      role: onprem
      eventEndpoint: http://10.99.0.11:9443
      samEndpoint: 10.99.0.11
      wireGuard:
        publicKey: "${PVE_RT01_WG_PUBLIC_KEY}"
        allowedIPs: [10.99.0.11/32]
    - nodeRef: rr01
      site: backbone
      role: cloud
      routeReflector: true
      eventEndpoint: http://10.99.0.1:9443
      samEndpoint: 10.99.0.1
      wireGuard:
        publicKey: "${RR01_WG_PUBLIC_KEY}"
        endpoint: rr01.example.net:51820
        allowedIPs: [10.99.0.1/32]

A pool can import one or more node sets. Imported members are added first and local spec.members entries are overlaid by nodeRef, so a leaf can keep only its self member with capture/discovery details while still learning the shared topology from the node registry.

When an overlay matches a sourced node, it may contain nodeRef plus only its local profileRef, capture, static-owned-address, and discovery fields. site, role, placement, maintenance, and secondary-IP capacity belong exclusively to SAMNodeSet; an overlay cannot change them.

apiVersion: mobility.routerd.net/v1alpha1
kind: MobilityPool
metadata: { name: svnet1 }
spec:
  prefix: 10.88.60.0/24
  groupRef: svnet1
  membersFrom:
    - resource: SAMNodeSet/svnet1-nodes
  members:
    - nodeRef: pve-rt01
      capture:
        type: proxy-arp
        interface: vmbr0
      ownershipDiscovery:
        mode: onprem-l2
        sources:
          - type: pve-svnet
            bridge: vmbr0

If a required membersFrom SAMNodeSet is not present, the pool reports Pending. Mark the source optional: true only when a partial local member list is acceptable during bootstrap. Node-set membership is declarative router configuration, not a dynamic RR cache, so MobilityPool planning never depends on an HTTP membership-sync endpoint.

WireGuard interfaces can import peers from the same node registry:

apiVersion: net.routerd.net/v1alpha1
kind: WireGuardInterface
metadata: { name: wg-svnet1 }
spec:
  selfNodeRef: pve-rt01
  privateKeyFile: /usr/local/etc/routerd/secrets/wg-svnet1.key
  listenPort: 51820
  peersFrom:
    - resource: SAMNodeSet/svnet1-nodes

WireGuardInterface.spec.peersFrom reads SAMNodeSet.spec.nodes[].wireGuard and generates ordinary WireGuard peer entries from publicKey, endpoint, allowedIPs, and persistentKeepalive. The node whose nodeRef matches selfNodeRef is skipped; when selfNodeRef is omitted, Router.metadata.name is used. Imported peers are added first, then static WireGuardPeer resources are overlaid by metadata.name, so a hand-authored peer named like the remote nodeRef remains a bootstrap or emergency override. If a required source is missing, the interface reports Pending and routerd leaves the current WireGuard runtime config untouched.

When privateKeyFile is set and the file is absent, a non-dry-run apply creates the parent directory with restrictive permissions and writes a new WireGuard private key with mode 0600. Existing non-empty key files are never overwritten. Dry-run and plan paths remain non-mutating. The interface status publishes the derived publicKey when it can observe WireGuard runtime state or derive it from configured key material.

For hub/spoke deployments, peersFrom removes repeated peer authoring after a trusted node registry exists, but it does not by itself solve first-contact registration. ADR 0015 describes a proposed WireGuard peer enrollment flow where leaf routers submit their WireGuard identity to a fixed RR/spine endpoint over a non-WireGuard bootstrap path, and the RR/spine approves the registration before it becomes generated peer input.

For example, on an AWS router:

apiVersion: mobility.routerd.net/v1alpha1
kind: SAMNodeSet
metadata: { name: lab-same-subnet-nodes }
spec:
  nodes:
    - nodeRef: onprem-router
      site: onprem
      role: onprem
    - nodeRef: cloud-router
      site: aws
      role: cloud
      placement:
        group: aws-edge
        priority: 10
      maintenance:
        drain: false
    - nodeRef: azure-router
      site: azure
      role: cloud
      placement:
        group: azure-edge
        priority: 10
    - nodeRef: oci-router
      site: oci
      role: cloud
      placement:
        group: oci-edge
        priority: 10
---
apiVersion: mobility.routerd.net/v1alpha1
kind: MobilityPool
metadata: { name: lab-same-subnet }
spec:
  prefix: 10.0.0.0/24
  groupRef: cloudedge
  membersFrom:
    - resource: SAMNodeSet/lab-same-subnet-nodes
  values:
    self.region: ap-northeast-1
    self.subnetRef: subnet-0123456789abcdef0
  profiles:
    cloudCaptures:
      aws-self:
        capture:
          type: provider-secondary-ip
          providerRef: aws-lab
          nicRef: eni-0123456789abcdef0
          targetFrom:
            region: self.region
        ownershipDiscovery:
          mode: provider-private-ip
          scanInterval: 60s
          subnetRefFrom: self.subnetRef
          # Empty/hold keeps a stopped instance as an ownership fact; release
          # expires it instead.
          stoppedInstancePolicy: hold
  members:
    - nodeRef: cloud-router
      profileRef: aws-self

On the on-prem node, the self overlay normally carries staticOwnedAddresses and a proxy-arp capture with an explicit activeWhen.type. Use single-router when one local router owns capture for the site, or vrrp-master when an HA pair gates capture by VRRP master state. activeWhen is supported only for this on-prem proxy-arp capture; a cloud provider-secondary-ip capture is rejected if it sets the field, so a local VRRP transition can never create a provider assignment without its matching BGP and local-dataplane plan. For discovery of dynamic on-prem clients beyond this bootstrap owner, add ownershipDiscovery with mode: onprem-l2 and at least one source (for example type: arp-observer on ens21). The complete cloud and on-prem topology remains in SAMNodeSet; the local router owns only its local implementation details.

routerd uses observed facts from federation or provider discovery to advertise owned /32 paths through BGP. Operators keep the control plane declarative by editing only MobilityPool; per-address advertisements and provider trap action plans are derived by the controller.

For same-provider cloud router maintenance, SAMNodeSet.spec.nodes[].placement.group elects one non-drained active capture member by priority and then nodeRef. Its maintenance.drain: true removes that member from active selection, so only the active member emits provider trap actions while every member can advertise its BGP standby path. Distribute the same SAMNodeSet identity and placement source to every node; each node's MobilityPool keeps only its local self overlay, which preserves deterministic placement without repeating remote member details.

North-Star Field Reference

spec.values : Non-secret local values used while normalizing this node's config. Use this for region names, compartment IDs, resource group names, subnet IDs, NIC names, and similar identifiers. Do not put credentials, tokens, private keys, or account secrets here.

spec.profiles.cloudCaptures.<name>.capture : Reusable defaults for a local cloud provider-secondary-ip capture. A member can opt in with members[].profileRef. Explicit member fields override the profile.

spec.profiles.cloudCaptures.<name>.ownershipDiscovery : Reusable defaults for provider private-IP inventory scanning. If ownershipDiscovery.providerRef is omitted, it inherits the effective capture.providerRef.

members[].profileRef : Applies a named cloud capture profile to that member. Use it for the local self member. It is invalid on remote members.

members[].capture.targetFrom : Maps generated provider action target keys to keys in spec.values. Explicit capture.target entries win when both are present.

members[].ownershipDiscovery.subnetRefFrom : Resolves ownershipDiscovery.subnetRef from spec.values when the explicit field is empty.

SAMNodeSet.spec.nodes[].placement and .maintenance : Declare deterministic active/standby capture placement and drain state in the shared node registry. MobilityPool.spec.members is only the local self overlay after membersFrom; do not repeat remote identity, placement, or maintenance fields there.

The shared membership belongs in SAMNodeSet. The older inline shape, where a pool repeats remote-node identity or provider details, is not a valid authoring model; a pool imports the node set and carries only its local overlay. Provider, capture, and discovery details belong only to the corresponding node's local pool.

Transport Profile

SAMTransportProfile is the higher-level transport profile for BGP-mode SAM. It derives the per-peer TunnelInterface, endpoint /32 IPv4Route, and BGPPeer resources that carry mobility paths. Current CloudEdge examples use IPIP as the default SAM delivery plane. WireGuard, when present, is an encryption underlay only: generated or hand-authored WireGuard peers should keep AllowedIPs to transport endpoint prefixes, not mobility /32s.

Each router must declare spec.selfNodeRef; routerd does not infer the local node identity from hostname or BGP router ID.

spec.addressingMode controls /31 slot derivation:

• edge-index (default): profiles use the same SAMNodeSet through spec.peersFrom. The controller sorts that shared node list and ranks each unordered node pair before allocating a /31 from spec.innerPrefix.
• pair-stable: each peer edge derives a slot from a stable hash, so leaf/router profiles can select only their actual peers without changing the shared node set.

For production fabrics, prefer /20 or larger innerPrefix where practical; smaller pools such as /24 (128 /31 slots) collide more easily under hash+mod allocation.

spec.peersFrom can reference SAMNodeSet/<name>, SAMPeerGroup/<name>, SAMEnrollmentPolicy/<name>, or a fetched SAMRRSet/<name>. A SAMNodeSet source contributes every spec.nodes[].nodeRef to the resolved topology. It creates peers for every non-self node with a samEndpoint, unless nodeRefs narrows the source to the listed adjacent nodes. The generated peer uses that samEndpoint as remoteEndpoint. A generic SAMPeerGroup source contributes reusable transport peers through the peer-group sync cache. SAMPeerGroup is runtime-only, not a top-level spec.resources kind.

An enrollment policy can also offer an opportunistic direct-leaf group. The leaf still consumes its fetched SAMRRSet first, then adds the policy-scoped group with direct: true:

peersFrom:
  - resource: SAMRRSet/pve-rrs
  - resource: SAMPeerGroup/pve-direct-leaves
    direct: true

This second form is delivered atomically with the RRSet by the authenticated enrollment fetcher; it is not a static resource and does not use the generic sync cache. Direct mode requires addressingMode: pair-stable, leaves routeReflectorClient false, and gives direct imports a higher local preference than RR imports. A direct profile must explicitly set the ordinary RR importPolicy.localPreference. This is deliberately BGP local preference, not an administrative-distance or AS-path heuristic: RR reflection in iBGP does not reliably express the extra forwarding hop in AS_PATH. The direct default is 200; set the two values explicitly when the fabric has another preference convention:

bgp:
  importPolicy:
    localPreference: 100 # RR peers
    nextHopRewrite: peer-address
  directLocalPreference: 200 # direct peers; must be greater than RR

For the RR import, leave nextHopRewrite unset (its default is peer-address) or set it explicitly as above. Do not use unchanged on a direct profile: an RR-reflected route may have been originated by a third leaf, but its reachable next hop is the immediately connected RR tunnel. For a legacy direct profile, routerd safely normalizes an explicit unchanged to peer-address at runtime so the daemon can start and the YAML can be replaced. Update the YAML to the explicit form above; non-direct profiles retain their normal unchanged behavior. This setting only chooses the forwarding next hop after a route is learned; it does not affect whether an optional direct peer is admitted.

A missing, expired, incompatible, or unreachable direct group does not make the profile pending and never removes the RR-generated tunnel or BGP peers. In short: direct is an optimization; RR is the fallback.

The controller resolves all sources at reconcile time. Direct peer and topology lists are not accepted: SAMNodeSet is the single identity/topology/endpoint input, while nodeRefs expresses only adjacency. If a required peersFrom source is not yet present, the profile reports Pending; optional sources are ignored until they arrive. If the source was fetched before and only its dynamic-config TTL has expired, routerd reports the source as Stale and keeps using the last-known-good peer group instead of removing generated transport artifacts.

SAMNodeSet entries may provide either a static samEndpoint or samEndpointFrom. The latter reads a status field such as DHCPv4Client/<name>.currentAddress or Interface/<name>.primaryIPv4, strips any prefix length, and feeds the resolved IPv4 address into generated peer remoteEndpoint values. When the source is not resolved, the transport profile stays Pending instead of generating a tunnel with a stale endpoint.

apiVersion: mobility.routerd.net/v1alpha1
kind: SAMTransportProfile
metadata: { name: cloudedge-transport }
spec:
  selfNodeRef: pve-rt01
  mode: ipip
  addressingMode: pair-stable
  innerPrefix: 10.255.0.0/20
  underlayInterface: wg-svnet1
  localEndpointFrom:
    resource: Interface/wg-svnet1
    field: primaryIPv4
  bgp:
    routerRef: BGPRouter/mobility
    peerASN: 64512
  peersFrom:
    - resource: SAMNodeSet/svnet1-nodes

Spine or route-reflector profiles can set spec.publishPeerGroup: true. In that mode routerd publishes a SAMPeerGroup DynamicConfigPart with this profile's selfNodeRef and concrete local endpoint. localEndpointFrom is resolved before publishing so leaves receive a direct remoteEndpoint value.

When routerd serve runs on a node with publishPeerGroup: true, it also serves published peer groups over the transport network on TCP port 19652 (GET /v1/peer-groups). A leaf with a missing required peersFrom group tries to query WireGuard peers reachable through spec.underlayInterface; a matching group is stored locally as peer-group-sync/<group-name> with the normal dynamic-config TTL. If the publisher disappears or the group expires, the leaf does not tear down the generated tunnel or BGP peer. It reuses the expired record as last-known-good input, reports the source as Stale, and keeps the existing transport artifacts rendered. Stale peer sources include a status warning for long-lived fail-static operation. A never-seen required group still reports Pending.

MobilityPool membership is resolved directly from SAMNodeSet resources. It is not published through the transport sync service, so BGP placement receives one canonical identity/topology input and only SAMPeerGroup remains dynamically synchronised.

apiVersion: mobility.routerd.net/v1alpha1
kind: SAMNodeSet
metadata: { name: cloudedge-nodes }
spec:
  nodes:
    - { nodeRef: onprem-router, site: onprem, role: onprem, routeReflector: true, samEndpoint: 10.252.0.1 }
    - { nodeRef: aws-router-a, site: aws, role: cloud, samEndpoint: 10.252.0.2 }
    - { nodeRef: azure-router, site: azure, role: cloud, samEndpoint: 10.252.0.3 }
---
apiVersion: mobility.routerd.net/v1alpha1
kind: SAMTransportProfile
metadata: { name: cloudedge-transport }
spec:
  selfNodeRef: aws-router-a
  mode: ipip
  encryption: wireguard
  innerPrefix: 10.255.0.0/24
  underlayInterface: wg-hybrid
  localEndpointFrom:
    resource: Interface/wg-hybrid
    field: primaryIPv4
  bgp:
    routerRef: BGPRouter/mobility
    peerASN: 64512
    timersPreset: fast
  peersFrom:
    - resource: SAMNodeSet/cloudedge-nodes
      nodeRefs: [onprem-router]

Core routers can set spec.bgp.routeReflectorClient and spec.bgp.routeReflectorClusterID; those fields are copied to each generated BGPPeer. When routeReflectorClient is true, routerd also hardens the generated peer import policy for that leaf: imported routes must be /32s under the configured importPolicy.allowedPrefixes or, when that list is omitted, the declared MobilityPool prefixes. They must carry that leaf's node-identity community and must not carry another topology node's node-identity community. This keeps the RR admission boundary tied to the declared SAM topology; a leaf cannot claim another node identity or advertise a broader mobility prefix through the generated RR session. A compromised leaf can still advertise a pool-local /32 with its own identity; constraining per-node ownership requires a separate authorization signal beyond this route filter. Edge routers can leave the RR fields unset and use ordinary iBGP sessions.

When a static RR intentionally has no local MobilityPool, its generated RR-client peers also provide the typed transit FIB authority. Set spec.bgp.importPolicy.allowedPrefixes explicitly to each mobility prefix and set both prefix-length bounds to 32; validation rejects an unbounded no-pool RR. routerd then installs only an exact owner/return mobility /32 received from that generated peer, with the peer's required/forbidden node-identity communities. A generic BGPRouter import policy is not used as transit authority, and an RR must not add a placeholder MobilityPool merely to forward leaf-owned routes.

Peer removal replaces the profile's generated DynamicConfigPart with the new resource set. Profile deletion replaces the old part with an empty active part, so effective config drops generated tunnel, BGP peer, and endpoint route resources. The generated resources then clean up through normal owner-reference GC and resource-specific teardown.

CloudProviderProfile describes provider capabilities and how an external tool would authenticate. The mobility planner does not call provider APIs directly. For cloud capture it emits dry-run ActionPlan records such as assign-secondary-ip and ensure-forwarding-enabled; the separate provider-action executor path may import and execute those only when explicitly allowed by ProviderActionPolicy.

OverlayPeer identifies a remote routerd peer and underlay for ordinary L3 HybridRoute delivery. Mobility /32s use BGP delivery through SAMTransportProfile.

Capture And Delivery

Supported capture types:

Type	Meaning
provider-secondary-ip	The cloud fabric captures the /32 through a provider-owned secondary address object or equivalent.
proxy-arp	A site router answers ARP locally for a selected address.

Reserved capture types rejected by MVP validation:

Type	Status
static-host-route	Reserved for a later dataplane design.
garp	Reserved for a later dataplane design.

MobilityPool always uses BGP delivery. Owned addresses are advertised as IPv4 unicast /32 paths; non-owners import the BGP best path into the local FIB over the selected overlay next hop.

members[].capture.target carries non-secret provider target hints copied into generated provider ActionPlan.target values. Put identifiers such as region, compartment ID, resource group, NIC name, or IP config name there; credentials, tokens, and private keys must stay in provider auth mechanisms.

Cloud provider-secondary-ip capture derives ordinary secondary-ip behavior from its type; do not configure that redundant strategy. Azure may instead set captureStrategy: route-table: Azure writes a UDR in capture.target.routeTableRef with NextHopType=VirtualAppliance and requires capture.target.nextHopIPAddress. Provider inventory must confirm that the route table points at the local router before routerd advertises the captured /32 to BGP. This provider-observation gate is specific to the route-table strategy; secondary-ip capture does not use route-table observation to decide when to advertise the overlay holder.

The current lab certification covers secondary-ip capture only. The route-table strategy is uncertified until provider observation, BGP advertisement coupling, and provider API delay behavior have been validated in the release lab. Its design intentionally waits for provider route-table observation before advertising the captured /32, so ARM/API latency can delay overlay convergence for this strategy.

Same-subnet limitation (validated in #516): AWS rejects VPC-internal /32 route destinations, and OCI rejects intra-subnet VCN route rules. For the primary same-subnet lift-and-shift use case, route-table is effective only on Azure. AWS and OCI must use secondary-ip.

For BGP-mode on-prem proxy-arp capture, members[].capture.sourceAddress optionally declares the router's local sender address on the capture interface. routerd lowers this to an IPv4StaticAddress /32 and uses it as the capture-prefix route preferred source. This is useful when the capture interface otherwise has no IPv4 address: Linux ARP for local same-subnet clients then uses an address inside the mobility prefix instead of falling back to an unrelated management address.

If that sender address is owned by another lifecycle manager such as DHCP/IPAM, use members[].capture.sourceAddressFrom instead. For example, resource: DHCPv4Client/svnet1-source with field: currentAddress uses the leased address as the capture-prefix route preferred source without lowering it to an IPv4StaticAddress, so routerd does not duplicate ownership of the same address.

Use members[].capture.excludeAddresses for local-only addresses inside the mobility prefix that must never be proxy-ARP captured across the extended segment. On PVE Simple SDN, for example, each host may own the same local gateway address such as 192.168.123.1/32; excluding it prevents generated BGP proxy-neighbor capture for that address and splits the capture-prefix route so Linux does not send local gateway ARP across the SAM capture path.

SAM does not provide transparent DHCP broadcast extension. Keep DHCP ownership with the local fabric, VPC/VNet/VCN, or PVE IPAM. A DHCPv4Client used by sourceAddressFrom should usually set useRoutes: false and useDNS: false when it exists only to learn the capture-interface source address. DHCP lease observations can participate in ownership discovery, but they should be combined with arp-observer, on-demand-arp, or PVE svnet observations when the IPAM source is outside routerd.

Passive sources cannot prove that zero clients exist. By default, an on-prem L2 discovery member with no observed clients remains pending. To make an empty segment an explicit operational policy, set ownershipDiscovery.allowEmptyAfter to a duration such as 2m; after the sources have been armed for that duration, routerd marks discovery Complete and publishes discoveryObserved: 0 plus freshness timestamps in status.

on-demand-arp also performs a conservative proactive sweep of the mobility prefix: one ARP target is probed per source scanInterval, using the same probeTimeout, probeRetries, probeCooldown, and sourceAddressFrom settings as demand-triggered probes. This lets quiet, already-running L2 clients become observed without a manual arping or ping from the owner side. Keep scanInterval conservative on broad prefixes; for /24 lab validation a short interval such as 1s gives fast convergence while still limiting traffic to one active ARP probe per second.

For faster discovery across separate leaf L2 segments, configure authenticated Event Federation listen and peer delivery on the pool's EventGroup. A leaf that sees an unresolved ARP request records routerd.mobility.arp.request.observed with a 45-second lifetime. Every remote pool member validates the group, source membership, pool, prefix, discovery scope, subject, and expiry before asking its local on-demand-arp observer to probe only that target. The source leaf is skipped, received facts are not re-forwarded, and the observer's normal target cooldown suppresses duplicate packets. This follows the EVPN Proxy ARP model of suppressing known bindings and flooding a lookup miss, while keeping SAM's dataplane L3/IPIP rather than carrying raw ARP frames over the overlay. If EventPeer.spec.types is non-empty, add routerd.mobility.arp.request.observed; an empty list already carries the new fact. Without Event Federation peers, the proactive prefix sweep remains available as the compatibility fallback.

For proxy-arp capture on Linux, routerd:

• enables net.ipv4.conf.<capture-interface>.proxy_arp=1 through the normal sysctl controller,
• installs a proxy neighbor entry equivalent to ip neigh add proxy <address> dev <capture-interface>, and
• enables net.ipv4.ip_forward=1 through the normal sysctl controller.

BGP-mode on-prem proxy-ARP members can opt in to a GARP after the node is observed as the /32 holder. Remote capture remains silent, so an on-prem to cloud move can still leave a bounded stale-neighbor window on the old on-prem segment until existing neighbor cache entries expire.

For provider-secondary-ip in BGP mode, the provider fabric owns address capture and routerd never assigns the mobile /32 to the local OS. The cloud provider secondary IP is only the provider-fabric ingress owner, and the Linux FIB forwards the packet through the selected overlay path instead of treating it as a local destination. On Linux routerd removes that specific address from local interfaces if cloud-init, netplan, or another guest agent adds it back. It then ensures IPv4 forwarding, explicit proxy-neighbor state for provider ingress when needed, and per-interface forwarding state; the overlay /32 delivery route comes from BGP best-path import. routerd does not re-add the address when the capture is removed, because it never owns the guest OS assignment.

Status reports this as captureOSAddressAbsence. enforced: true means routerd is actively enforcing that the captured address is absent from local OS interfaces. lastReconcileRemoved: true means the most recent reconcile actually removed the address; it is normally false in steady state once the address is already absent. reason identifies the BGP provider-secondary no-local-address projection.

Inspecting Ownership

MobilityPool status exposes one ownership view:

• ownershipResolverControlPlaneOwnerTable is the only per-address operational projection. It is used by doctor sam, FIB policy checks, and operators. It keeps one deterministic row per observed mobility address and includes the selected owner node/provider/NIC/subnet/resource, local evidence node/provider/NIC/subnet/resource/source, capture state and final captureDisposition/captureReason, advertise/suppression state, and conflict reason/winner/resolution when present.

Use routerctl mobility owners to inspect the control-plane table without pattern-matching raw status JSON:

routerctl mobility owners
routerctl mobility owners --pool cloudedge --address 10.77.60.10/32 -o json

Use routerctl mobility explain to render one owner-table row together with pool-level provider status for one address:

routerctl mobility explain --pool cloudedge --address 10.77.60.10/32
routerctl mobility explain --pool cloudedge --address 10.77.60.10/32 -o json

OwnershipResolved comes from that row. Provider status is intentionally address-scoped only when the pool status names the address: providerActionFailedAddresses makes ProviderActionApplied=False, and providerObservationPendingAddresses makes ProviderObserved=False. For all other addresses those conditions are Unknown; a pool-level provider phase is not incorrectly projected as a failure for every address. FIB installation and reachability remain external checks, reported as Unknown until doctor sam or a dataplane probe verifies them.

Rows are sorted by pool and address. When a remote provider owner overlaps local evidence, or when two fresh provider owners claim the same /32, the row state is Conflict and conflictReason explains the condition. Expired ownership events are not retained as live conflicts. Duplicate provider-owner conflicts also include conflictWinnerNode and conflictResolution: the healed BGP owner wins when present; otherwise the lowest stable owner key wins (nodeRef, provider ref, resource ref, NIC ref, subnet ref, then address), independent of provider scan recency. Losing nodes with an observed local provider-secondary capture report loser-release-local-capture and release only that local capture after the stale-capture hold-down. routerctl doctor sam consumes the same ownership state for conflict checks and, with host checks enabled, compares endpoint-owned local rows with the Linux main FIB. Provider-secondary BGP capture-holder rows are not local endpoint owners, so they are not required to resolve as local/cloud routes; delivery/forwarding checks and dataplane probes prove that path.

FreeBSD and other non-Linux hosts do not have live SAM local capture. The MobilityPool planner rejects proxy-ARP capture on FreeBSD rather than accepting a configuration that cannot safely provide the Linux contract: owned proxy-neighbor capture, gratuitous ARP, and the narrow routerd-owned forwarding path. Provider-secondary capture configuration and the surrounding SAM/BGP control plane remain portable, but their local capture status stays CaptureUnsupported until an equivalently owned FreeBSD dataplane is approved.

The live Linux dataplane has been smoke-tested in an Azure + PVE same-subnet lab. Treat it as pre-release behavior and validate the exact provider and firewall policy before production use.

Reverse Path Filtering

Strict reverse-path filtering can drop forwarded SAM traffic because the mobile /32 may appear to belong to a directly attached subnet while the return path is through the overlay. routerd does not silently change rp_filter for SAM, because that is an invasive interface policy decision.

routerctl doctor hybrid reads net.ipv4.conf.<capture-or-tunnel-interface>.rp_filter when host checks are enabled. It warns when the value is strict (1) and recommends evaluating loose mode (2) on the affected interfaces.

Provider Capabilities

Provider	MVP capability descriptor
Azure	NIC secondary IP plus IP forwarding enabled on the router NIC. Route-table (UDR) capture also available for same-subnet /32 (NextHopType=VirtualAppliance, limit 1000).
AWS	ENI secondary private IPv4 plus source/destination check disabled. Route-table capture rejected for VPC-internal /32.
OCI	VNIC private IP object plus source/destination check disabled. VCN route-rule capture rejected for intra-subnet /32.
GCP	Alias IP or route capability, gated by the declared provider profile.

The profile is declarative. The mobility planner can produce provider ActionPlan records, but address assignment and NIC flag changes remain gated by the provider-action execution policy and executor plugin. The planner itself never mutates provider state.

Same-Subnet Flow

In a 10.0.0.0/24 lab, suppose 10.0.0.7/32 is the cloud VM address and 10.0.0.9/32 is the on-prem/PVE VM address. The goal is for the cloud VM at 10.0.0.7 to open a TCP connection to the on-prem VM at 10.0.0.9 while both VMs keep local default gateways and no NAT is introduced.

1. The cloud VM sends to 10.0.0.9.
2. Azure NIC secondary IP capture directs packets for 10.0.0.9/32 to the cloud routerd node.
3. The cloud routerd node delivers the packet over the generated IPIP SAM transport; if encryption is enabled, that IPIP packet rides over the endpoint-only wg-hybrid underlay.
4. The on-prem side forwards it to the owner of 10.0.0.9.
5. Source and destination IPs remain the original endpoint addresses.

The reverse path for 10.0.0.7/32 is captured on the on-prem side with proxy-ARP. PVE LAN hosts reach .7 through the on-prem routerd node, which delivers the packet over the same generated SAM transport to the cloud routerd node.

The split example configs are:

• examples/cloudedge-mobility-demo/ contains the current MobilityPool-based cloud/on-prem configuration.

Firewall And NAT Composition

Selective Address Mobility lives in the ordinary switching/forwarding plane. It does not contain nat, preserveSource, firewall, or zone fields. Address transparency is intrinsic: the source and destination addresses are preserved.

To firewall or NAT a mobile address, reference its literal /32 in the existing FirewallZone, FirewallRule, or NAT44Rule resources. The current model has no cross-kind reference from firewall or NAT kinds to MobilityPool; the coupling is intentionally loose by literal address. A named reference can be added later if it proves useful.

SAM-forwarded traffic still traverses the existing firewall and conntrack path like any other forwarded traffic. Independence means the mobility resources do not configure arbitrary firewall or NAT policy; it does not mean bypass.

Conntrack cleanup design note

routerd briefly exposed a manual, scoped conntrack cleanup option for BGP SAM failover. It has been removed. In the reference SAM leaf configuration, routerd does not draw dataplane rules that engage conntrack for the delivered overlay flow, so the leaf-side scoped cleanup was a no-op and did not address failover flow anomalies.

The problem statement remains valid for future stateful SAM leaf designs: a router that deliberately tracks forwarded mobile /32 flows may need a scoped recovery hook when it becomes holder. Reintroduction should be designed around detecting a routerd-managed ct-engage dataplane and enabling cleanup automatically for that case. Do not reintroduce the feature as a manual opt-in flag.

In particular, the delivered /32 traffic crosses the Linux firewall FORWARD chain between the capture interface and the tunnel interface. Permit that forwarding path for the captured address explicitly when the router has a default-drop forwarding policy. The managed exceptions are narrow: WireGuardInterface opens its Linux UDP listen port in INPUT, and the typed MobilityPool local capture plan opens the capture-to-tunnel FORWARD path it owns.

Overlay And Federation Addressing On Cloud Nodes

The Event Federation transport (the routerd-eventd listen address and each EventPeer.endpoint), BGP/BFD peer addresses, and the SAM transport endpoint / inner addresses generated by SAMTransportProfile must use address ranges you control end to end on every node. If you place WireGuard underneath the SAM transport, its interface/peer endpoint addresses have the same requirement. On cloud instances, do not draw overlay, BGP/BFD, or federation addresses from ranges the provider reserves for its own internal use:

• 169.254.0.0/16 (RFC 3927 link-local). Cloud instance metadata (IMDS) lives at 169.254.169.254, and some images reserve the entire block: Oracle Cloud's Linux image routes all of 169.254.0.0/16 through an InstanceServices chain, so a federation SYN to a 169.254.x overlay address is pulled to loopback and reset even though ICMP to the same address succeeds. AWS and Azure also use 169.254.169.254 for IMDS. Symptom: local ownership facts are present, but routerd-eventd or BGP/BFD sessions never connect between nodes.
• 100.64.0.0/10 (RFC 6598 carrier-grade NAT). Used by CGNAT on provider underlays and by Tailscale (100.x tailnet addresses, MagicDNS). An overlay in this range collides with any Tailscale membership and with carrier NAT.

Use RFC 1918 ranges you reserve for SAM transport endpoints, the SAMTransportProfile.innerPrefix, any optional WireGuard endpoint addresses, and the routerd-eventd listen / EventPeer endpoints and BGP/BFD peering addresses. Keep them distinct from the mobility pool /24 (the captured addresses) and from every cloud-reserved range above. This applies to all providers (AWS/Azure/OCI); OCI is simply the strictest at enforcing the link-local reservation.

Client Endpoint Addressing vs Router-Overlay Reachability

A globally-unique /32 on a client guest's lo/dummy interface is not reachable across cloud fabrics just because the guest OS owns the address. The cloud fabric (VPC/VNet/VCN) only delivers destinations within the provider subnet CIDR to a client ENI/NIC; a destination outside the VPC CIDR is dropped by the fabric before it reaches the client, regardless of overlay routes on the routers.

Concretely, in a distinct-addressing 4-site test:

• Router endpoint /32s on the overlay itself are reachable end-to-end (the routers carry them over WireGuard). A distinct-mesh of router endpoints passes 12 directed ping+SSH.
• Client dummy/lo /32s outside the VPC CIDR are not — the cloud fabric does not deliver them to the client ENI even with overlay routes and provider forwarding enabled.

Therefore: treat distinct-mesh shortcut endpoints as router endpoints only. To give clients globally-unique, cross-fabric-routable addresses you need either provider-routable client subnets or provider-assigned client IPs (a secondary IP / captured address that the fabric actually delivers), not a guest-local dummy /32. Do not confuse router-overlay reachability with client-fabric reachability when designing a multi-site lab.

Out Of Scope

The MVP does not implement full L2 extension, EVPN, BUM forwarding, broadcast/multicast extension, automatic ungated cloud API mutation, dynamic patch/replace semantics, or automatic rp_filter changes.