跳到主要内容

CloudEdge SAM Internals

This page explains the architecture, internal implementation, and configuration fields of CloudEdge SAM (Selective Address Mobility) at a level where operators and implementers can both follow "what happens inside". Read What is CloudEdge SAM for the conceptual introduction and Selective Address Mobility for how to author the config first.

The implementation lives in pkg/controller/mobility/. The descriptions here are kept consistent with that code (notably planner.go and controller.go).

Architecture: two planes​

CloudEdge SAM cleanly separates reachability from cloud ingress.

Plane 1: overlay reachability — the BGP best path is the truth​

Each owned address in a MobilityPool is represented as an IPv4 unicast /32 BGP advertisement.

  • The holder of a /32 is the node that wins the BGP best path for that prefix.
  • Non-holder nodes learn remote owned addresses from the BGP best path and install delivery routes via the overlay next hop into the FIB.
  • Address movement is expressed as BGP withdraw / advertise and path preference changes. Operators never hand-author leases, per-address ownership records, or provider actions.
  • Failure detection is accelerated by BFD (FRR bfdd); when BFD is unstable, BGP hold timers remain the non-destructive authority for route withdrawal.

This is the decision in ADR 0012, which replaced the older bespoke ledgers (AddressLease / ownershipEpoch / captureEpoch).

Plane 2: cloud ingress — provider operations are background reconciliation​

Packets entering from outside through a VPC / VNet / VCN follow the cloud fabric's routing, not the BGP overlay. So routerd:

  • assigns the target /32 to the holder VM's NIC as a secondary IP, and
  • enables forwarding on that NIC (AWS sourceDestCheck=false / Azure ipForwarding=true / OCI skipSourceDestCheck=true / GCP canIpForward=true).

But these are not the source of truth for reachability; they are operations reconciled eventually in the background from the BGP mobility view and provider inventory. Even if the provider API lags, overlay reachability recovers from BGP convergence alone.

The BGP community taxonomy​

The BGP communities that mobility attaches to /32 advertisements are the signal wires that tell other nodes a node's role, the provenance of an advertisement, and whether it is the holder. They are defined in pkg/controller/mobility/controller.go.

CommunityConstantMeaning
64512:100…CommunityOwnerthis advertisement is a mobility owner /32
64512:101…CommunityRoleOnPremadvertising node's role is on-prem
64512:102…CommunityRoleCloudadvertising node's role is cloud
64512:110…CommunitySourceObservedprovenance: observation-derived advertisement
64512:111…CommunitySourceStaticprovenance: a static owned-address advertisement
64512:112…CommunitySourceHandoverprovenance: advertisement during handover
64512:120…CommunityFailovera seize advertisement during failover
64512:121…CommunityActiveHolderholder-beacon: attached only by the active holder
(per node)node-identity communityidentifies which node advertised (derived from nodeRef)

LOCAL_PREF is set relative to bgpMobilityLocalPrefBase = 200, so an active advertisement carries a higher preference than a standby's make-before-break advertisement.

The holder-beacon (64512:121) is the linchpin​

bgpMobilityPathAttrs (controller.go) attaches 64512:121 only when the advertisement is from an active holder.

On the receiving side, bgpObservedGroupHolder (planner.go) treats a node as the group holder only when the best path for a /32 carries both the node's node-identity community and 64512:121. This means a:

  • standby's weak (lower-preference) make-before-break advertisement, and
  • just-booted (cold-start) advertisement that is not yet active,

are not mistaken for holdership. It is an authoritative holder signal that is plugin-independent (BGP is always present) and best-path-independent (only the active node emits the beacon).

Design history: earlier attempts inferred holdership from next-hop matching or a provider self-scan. Both failed — "the next hop is the tunnel underlay, not the SAM endpoint" and "a node cannot observe its peer's NIC holdings", respectively. Concentrating on a dedicated beacon community on the BGP best path resolves both, including the cold-start mutual-defer deadlock.

Placement: deciding active/standby​

Each SAMNodeSet node has placement.group and placement.priority; the MobilityPool imports that shared placement through membersFrom.

  • group — the unit that competes for active/standby (e.g. azure-edge).
  • priority — a lower number is higher priority. Members left at 0 (unset) are auto-numbered 10, 20, 30, … within the group by autoPlacementPriorities.

The decision logic (evaluatePlacementWithIncumbent)​

  1. Order the non-drained members of the same group by priority ascending, then nodeRef ascending.
  2. Take the head as the active candidate.
  3. No-preempt tie-break: on an equal-priority tie, prefer the current holder (incumbent) over the lexicographic nodeRef winner, so a returning peer does not reclaim a live holder and cause a pointless handoff.
  4. But a strictly higher priority (lower number) member still reclaims — the incumbent override applies only when the incumbent shares the top priority.

When incumbentHolder is empty the logic is pure priority/nodeRef ordering, which is also how the group bootstraps before any holder is observed.

Three mechanisms that reconcile no-preempt with failover​

On top of the bare placement decision, three mechanisms suppress return-time accidents and switch churn (all in planner.go).

1. Startup fence​

placementSettleStart  = time.Now()        // captured at process start (resets on restart)
placementSettleWindow = 120 * time.Second

fencePlacementForStartupWithReadiness defers an active assertion when all three hold: "about to assert active", "no incumbent peer observed yet", and "startup readiness is not complete". Readiness means the local BGP control plane has completed an initial observation and, for provider-inventory-backed captures, provider self-observation has completed. placementSettleWindow remains as a conservative fallback for callers that do not provide readiness signals. When readiness is known but remains incomplete, the fence is bounded: after placementSettleWindow * 3 (360 seconds by default) routerd releases the active assertion even while readiness remains incomplete, so overlay liveness is not blocked forever by a provider API or observation failure.

  • A just-returned node would otherwise win the equal-priority tie-break and reclaim holdership before its fresh BGP RIB / provider observations converge. The fence prevents this.
  • A node whose BGP/provider observations have completed can leave the fence before the wall-clock window expires, which avoids crash-loop nodes remaining artificially passive after every restart.
  • A node whose observations are still incomplete remains fenced even after the wall-clock window, so a partitioned or blind node does not assert active merely because time elapsed.
  • A node that already observes an incumbent peer is not fenced; the normal no-preempt placement tie-break already defers to that holder.

2. Holder retention​

applyHolderRetention keeps a node active while it physically holds its group's captures (selfHolds). It applies when:

  • the node is not already active,
  • selfHolds is true,
  • yieldToHigherPriority is false (see below), and
  • the startup settle window has elapsed (so the fresh self-capture observation is trusted rather than a returning node's stale "I used to hold" memory).

Thus a live holder does not surrender ownership to a deterministic tie-break winner or a transient peer observation (the ADR 0016 principle: yield only on losing your own holdership, never because a peer was observed).

3. Unequal-priority auto-restore (higherPriorityHolderActive)​

higherPriorityHolderActive returns true when the holder observed via the BGP holder-beacon is a strictly higher-priority peer (lower priority number) than self. It feeds the yieldToHigherPriority argument of applyHolderRetention.

  • At equal priority it is always false → retention holds and the result is no-preempt.
  • At unequal priority, the low-priority interim holder releases retention and yields once the high-priority node returns and starts emitting the beacon → the configured auto-restore proceeds.

The handover moves /32s one at a time, so the dataplane never dips.

Fencing: rejecting stale provider operations​

Provider operations (secondary-IP assign/unassign, etc.) carry the mobility path signature (mobilityPathSig) at generation time, plus the desired holder and the observed provider/journal transition. On reconcile, operations whose desired BGP path no longer matches are skipped. The old ownership/capture epoch tables are gone.

Seize (the takeover during failover) has dedicated hold-downs:

  • bgpSeizeLivenessMissingHold = 30s — suppress seize when the liveness marker is missing
  • bgpProviderMissingRetryHold = 30s — suppress retry when the provider observation is missing
  • bgpTrapRIBMissingHold = 2m — retention when the trap route is absent from the RIB

Dynamic RR sync is fail-static​

RR nodes may publish SAMPeerGroup resources over the TCP 19652 sync endpoint so leaves can bootstrap their transport peers. Fetched peer groups are saved as dynamic config parts with ordinary TTLs:

  • peer-group-sync/<name> for SAMPeerGroup

TTL expiry does not mean the data plane should be dismantled. If a leaf has a previously fetched peer group and the RR publisher disappears, routerd treats the expired record as last-known-good input, marks the source Stale, and keeps the generated tunnel and BGP peer rendered. Only a source that has never been seen remains Pending. MobilityPool membership is resolved from static SAMNodeSet configuration. The stale marker is an operator signal that topology freshness is no longer being refreshed. Status also includes a warning field on stale sources so long-lived fail-static mode is visible without tearing down the working data plane.

当 policy 配置了 directMesh.peerGroupRef，并且 leaf 的 claim 已签名且选择了 directMesh: true 时，RR 会把第二个、带 policy 范围的 SAMPeerGroup 放进与 SAMRRSet 相同的 dynamic-config part。该 group 只包含符合条件的远端 leaf，必须匹配 本地 transport fingerprint，并携带由签名 claim 投影的 IPv4 /32 列表；尚未拥有地址的 leaf 的列表可以为空。

这个 direct group 是可选的加速器，不是新的 L2 overlay。SAMTransportProfile 保留 SAMRRSet source，再把 direct group 作为最后一个 direct: true source 加入。只有 direct BGP session 存活时，它的路由才会取得比 RR 更高的 LOCAL_PREF。若 group 缺失、过期、 不兼容或 underlay 无法到达，routerd 不会创建该 direct artifact，仍使用 RR peer。这样 RR 同时负责启动和安全回退。

刚加入时，已签名 leaf 的 mobility.ownedAddresses 可以为空，这是正常状态。routerd 仍会 建立已认证的 direct BGP session，但会给该邻居加上一条明确的 全部拒绝 import 规则：空 claim 不会通告 mobility route，也不会从这条 direct link 接收 route。之后出现已签名的 /32 时，才只允许该地址并使用 direct preference。这样无需编造 IP，通信在准备期间仍安全地经由 RR 转发。

routerctl mobility leaf-config 也可以省略 --owned-address 来生成此状态。此时它不会生成 local service address，也不会生成任何 BGP export/redistribute prefix，因此生成的配置和已签名 claim 一样不会通告路由。

Capture strategies (how cloud ingress is built)​

capture.type selects the normal ingress mechanism. capture.captureStrategy is only an explicit route-table override for provider-secondary-ip.

configurationprovidersbehavior
type: provider-secondary-ipAWS / Azure / OCI / GCPassign the /32 to the NIC as a secondary IP
captureStrategy: route-tableAzurepoint a UDR entry at the holder's NIC
type: proxy-arpon-premcapture on the L2 segment via proxy-ARP/GARP

Current release lab certification covers secondary-ip capture only. The route-table strategy is uncertified. On Azure it requires capture.target.nextHopIPAddress, and routerd waits for provider inventory to observe the UDR pointing at the local router before advertising the captured /32 to BGP. This coupling is specific to route-table; ARM/provider API latency can delay overlay convergence for this strategy. secondary-ip capture is not gated on route-table observation.

认证不会采用 write-accepted gate。route-table write 被接受只证明 provider API 接受了 mutation；它不能证明实际生效的 route table 已经把该 /32 steering 到本地 router。若在 write acceptance 后立即广告 BGP，retry、throttling 或 inventory propagation 延迟造成的 write-to-observation window 中可能出现 black-hole。更安全的契约是 provider 已观测到 ingress 后再发布 overlay advertisement。

本 release 中已认证的 hybrid strategy 仍是 secondary-ip。它同样通过 provider self inventory 确认，但观测对象是 NIC secondary-address attachment，而不是 route-table entry。 将来若要认证 route-table，在移除 uncertified 标记前必须包含 large-pool behavior、 failover rewrite ordering、inventory UDR 解析，以及 ARM/provider delay 或 throttling 证据。

Every capture is accompanied by a forwarding-enable action so the NIC can forward packets that are not addressed to itself.

Provider split-brain reconciliation​

BGP remains the control-plane truth for overlay reachability, but provider inventory can temporarily report the same /32 as owned in more than one cloud fabric after a partition. When fresh provider-discovery facts disagree, the ownership resolver marks the address Conflict with conflictReason=duplicate-provider-home-owners and includes all observed owners.

The resolver also records a deterministic conflictWinnerNode:

  • if the healed BGP RIB has a home-owner path for the /32, that BGP owner wins;
  • otherwise the lowest stable owner key wins (nodeRef, provider ref, resource ref, NIC ref, subnet ref, then address), independent of provider scan recency.

Losers do not create new provider capture actions. If the losing node observes the conflicting /32 still attached to its own provider-secondary capture, the status records conflictResolution=loser-release-local-capture; after the same stale-capture hold-down used for trap cleanup, routerd emits a scoped unassign-secondary-ip for that local capture. Nodes that do not hold a local capture report loser-withhold-local-capture. The winner reports winner-retain-local-capture.

The generated RR-client admission policy is a route-admission boundary, not a per-address authorization system. It requires the advertising node's identity, forbids other topology node identities, and limits accepted routes to /32s inside the declared MobilityPool prefixes. A compromised leaf can still advertise a pool-local /32 with its own identity; preventing that requires an additional ownership authorization signal outside this BGP filter.

On-prem LAN authority is unchanged​

BGP decides remote overlay reachability, but it does not replace the local L2/ARP authority. On the on-prem side, the following remain in force as local safety mechanisms:

  • VRRP-master gating,
  • proxy-ARP / GARP,
  • non-master fail-closed behavior, and
  • the duplicate-holder doctor check.

Graceful stop (make-before-break handover)​

routerd serve --graceful-stop-timeout (default 20s) makes a node, on SIGTERM/SIGINT, wait up to this long for the mobility make-before-break handover. 0 disables it. On a planned restart, the new holder establishes its advertisement before the old holder steps down, avoiding a dip.

Status fields​

MobilityPool status 是显示投影，而不是 desired-state 传输通道。它包含 prefix、groupRef、placementActive、placementActiveNode 和 placementGroup；ownershipResolverControlPlaneOwnerTable 提供面向操作者的 逐地址所有权视图。BGP liveness marker 会被解码为类型化 runtime snapshot， 不是另一份 status 契约。

这些字段可通过 routerctl doctor 的 SAM 诊断和 routerctl show 查看。

Behavior observed on real hardware (for reference)​

Measured on an unequal-priority pair (priority 10 vs 20, Azure hardware):

  • A1 failover: stop the high-priority node → the low-priority node seizes all three /32s in about 132 seconds → full dataplane recovery.
  • A2 restore: bring the high-priority node back → it reclaims the three /32s one at a time (no flapping). Client ping at 1-second intervals during the reclaim had 0% loss.
  • For an equal-priority pair, no-preempt held for 561 seconds with no holder swap, no split, no dip, and no cold-start deadlock.