Large-scale AI training executes a single task across hundreds to thousands of accelerators, and each step completes only after every participant reaches the same barrier[3]. However, the underlying infrastructure remains a collection of servers coordinated by software, whose delay varies with load, address layout and thread scheduling. This is because the network-centric organization behind resource disaggregation treats each server as an independent unit. Such an assumption is far from what tightly coupled computation requires.
A recent review in Nature Reviews Electrical Engineering argues that this gap should be addressed at the level of structure rather than bandwidth[1]. The review was written jointly by silicon and research teams at Panmnesia and the infrastructure group at Meta (Meta Infra). We summarize the argument here in our own words, and note where we think it deserves a harder look.
Latency variation, not latency
Inside a chip, per-hop delay is a design parameter. Wires are short, pipeline depth is fixed and arbitration is resolved in hardware, thereby keeping the timing stable enough that the rest of the design can rely on it. In contrast, the same transfer in a datacenter traverses cables, interface cards, switch queues and one or more software synchronization layers. Each of these stages incurs delay that depends on the state of the system elsewhere. Measurements collected in production report a heavy-tailed round-trip distribution, in which the 99th percentile is roughly 5× the median[4].
Raising per-link bandwidth does not address this. A 400 Gbit/s link improves the throughput of a single path and leaves the variation of the traversal untouched. Specifically, for workloads with dense cross-device dependencies, six times as many participants can leave each of them in the mid-80% range of its earlier efficiency. Note that this loss is time spent waiting rather than any link running out of capacity.
Note that the failure model shifts as well. In a request-driven service, one slow server degrades a fraction of requests. In synchronized training, a stalled device holds the step boundary for everyone else, and the common response is to roll the entire group back to its last checkpoint. Since the recovery granularity is a group rather than a device, the question changes. It is no longer how many devices remain available, but whether the group can advance at a rate that anyone can plan around.
Where disaggregation stops paying off
Conventional server-centric integration places CPU, memory, storage and accelerators in a fixed enclosure, which results in resource utilization below 50% and mismatched upgrade cycles. Disaggregation addresses these limitations by separating resources into function-specific pools connected through a network, and it works well when servers are close to independent and cross-device dependencies are weak.
Unfortunately, AI workloads violate that assumption at every step. Computation stages reference state held on other devices with little tolerance for delay, while coherence maintained in software imposes jitter that varies with address layout, control-flow path and thread scheduling. As more devices participate, these factors interact and the distribution becomes wider. That is, the loose coupling that made disaggregation attractive is exactly what dense synchronization cannot exploit.
What CXL addresses, and what it leaves open
CXL organizes device interaction through three subprotocols: CXL.io for initialization, configuration and I/O (preserving PCIe compatibility), CXL.cache for device access to the CPU cache hierarchy, and CXL.mem for memory decoupled from the CPU package. Their combinations define the Type 1, Type 2 and Type 3 device classes. While this classification describes device behavior well, it also carries a server-centric assumption: every component is modeled as owning a fixed share of the address space and a fixed coherence responsibility.
The trajectory of the standard is a gradual relaxation of package-centric constraints. CXL 1.0 opened hardware-managed access from a CPU to memory held on external devices. CXL 2.0 introduced switching, letting many devices share one domain. CXL 3.0 brought fabric-attached memory, a single shared address space and direct peer-to-peer access, such that traffic patterns begin to resemble those of intra-chip systems. Lastly, the CXL 4.0 white paper builds on the seventh-generation PCIe physical layer. It doubles the per-lane rate to 128 GT/s, allows as many as four retimers to stretch a channel, and bundles several physical ports into one logical port[2].
Two properties of this evolution matter for scale-up. First, coherence bookkeeping grows with the address ranges under each device’s own management rather than with a single fabric-wide structure. The metadata footprint thus stays comparable to the mapping tables and state that server storage stacks already maintain. Second, later revisions added error detection with retransmission at each hop, completion and ordering guarantees per transaction, and channel-level isolation with recovery paths, which keep device-to-device transactions correct as the fabric grows.
However, the specification does not fix the part that determines timing. It leaves fan-out, pipeline depth and per-hop traversal latency to the implementation. A compliant device can thus be built in software, firmware or dedicated hardware, whose hop-level behavior differs by orders of magnitude. In addition, the supported topologies (e.g., mesh, dragonfly and torus) are inherited from high-performance networking, where the design goal is connectivity and throughput rather than a fixed hop count. Compliance is therefore necessary but not sufficient, and the design space that this review targets resides in the implementation freedom that the standard leaves open.
Three hardware structures
High-fan-out, non-blocking switch. Fan-out determines the branching factor of the fabric and thus the hop count. With limited fan-out, the fabric can only grow by adding switch hops, which lengthens traversal distance, complicates routing and limits how densely devices can be packed into the same physical space. Non-blocking behavior maintains the ingress-to-egress bandwidth ratio at 1 under concurrent load, thereby allowing many accelerators to exchange parameters or activations without internal contention. Since every port employs the same forwarding pipeline and the traversal lengths match, the switch can anchor a rack-scale network-on-chip (NoC).
Link acceleration unit (LAU). Every hop repeats the same standard-defined work: it forms and converts headers, manages transactions and decides where the traffic goes next. This work is far from trivial. The boundary hop maps local addresses and metadata onto a fabric-wide address space that can reach 4 PB. In addition, transaction handling spans 8 CXL.io, 6 CXL.cache and 12 CXL.mem types, while congestion and deadlock constraints are enforced on entry. When these stages ride on firmware, the time each of them takes moves with control-plane activity, event processing and transient contention even though the physical path is unchanged. Note that a typical management interface carries over a hundred commands and dozens of event-log types, which makes this sensitivity real in deployed devices. Placing the stages on a fixed hardware pipeline eliminates the software-induced jitter without altering protocol semantics. In addition, the LAU leverages the freedom that the standard leaves open: it monitors link conditions and adjusts transport parameters (i.e., retry window, queue admission and congestion indicators) as traffic changes. The moderate and severe congestion levels default to 10% and 25% of egress occupancy, respectively. Tuning them at runtime trades a small amount of local waiting for fewer replays and less downstream variation.
Fabric controller. A single CXL fabric can span up to 4,096 devices, and thus correctness and ordering become a fabric-wide concern rather than a device-local one. If every controller decides with its own local rule, how a request is ordered or prioritized depends on which controller happens to handle it. This behavioral diversity in turn grows with the controller count. Applying one prioritization policy across the domain gives the fabric what a transaction and ordering engine gives a single die.
The floorplan of a switch die exposes the same reasoning in silicon. Control logic for the entire domain resides at the center, taking the role that arbitration and transaction engines play in a monolithic SoC. The ports form a symmetric ring along the die periphery, which gives every attached device the same electrical distance and prevents physical-layer skew from degrading timing uniformity. Each port integrates its own LAU, controller slice and buffering, while a crossbar or multistage interconnect underneath provides non-blocking traversal.
Tray, pod and fabric
The deployment side generalizes chip floorplanning to rack scale in three levels. A tray gathers accelerators, CPUs, memory or switch resources into a modular block that exposes external CXL links. A resource type is thus composed in the way a block is composed on a die. A pod then groups multiple trays so that any two of them reach each other across one CXL switch hop, which is the rack-scale analog of tile-to-tile interaction. Parallelized switch lanes provide multiple identical traversal options within a pod. The system can therefore reroute among equivalent one-hop paths and localize the impact of a failure, while the hop envelope remains fixed. Lastly, the fabric interconnects pods through upper-tier switches spanning racks, employing ring, mesh, torus or plane-style layers. A plane-style layer can be built as a multi-tier Clos or fat-tree that maintains bisection bandwidth. In practice, oversubscription can be tolerated at the pod tier to pack in more trays, while upper-tier switches remain non-oversubscribed such that system-wide traversal stays uniform.
Coherence follows the same hierarchy. Inside a pod, its local switch serves as the ordering point for request, response and snoop packets, which provides the ordering and visibility view that a tile has inside a chip. Upper-tier switches supply the tier above, and matched hop counts across pods bound the snoop round-trip variation. Forward progress in turn relies on mechanisms that never consult software timing. Requests and data traverse separate channels, credit-based flow control prevents a sender from outrunning downstream resources, and predefined routing eliminates cyclic dependencies.
Numbers, and what they are worth
The review quantifies its envelope in three places, employing the NVLink-C2C-based GB200 NVL72 as the reference network-based platform[5].
Coherence-domain scope expands by roughly an order of magnitude. The reference architecture pairs each CPU directly with two accelerators. Under a conservative coordination assumption for CXL.cache, one CPU can instead coordinate up to 16 accelerators within one address space, which is 8× more. A rack-scale deployment envelope accommodates roughly 60 such groups. The resulting coherence domain can therefore include up to 960 accelerators.
Data access changes in class rather than in degree. RDMA-style remote access typically incurs microsecond-level variation, whereas a fabric that enforces hop count and traversal symmetry can hold a single-hop round trip in the several-hundred-nanosecond range.
Fault isolation becomes finer. Since trays are disaggregated by function, the unit that is repaired or replaced is a device rather than a server. In addition, support for hot-plug and runtime reconfiguration allows the replacement to proceed while the job continues.
We should read these values carefully. They are architectural envelopes derived from the mechanisms, not measurements of a deployed 960-accelerator system, and the review states this explicitly. That is, they describe what the structure permits.
Limits we should be honest about
Reach imposes the first boundary. Attenuation and jitter give an electrical CXL link at 128 GT/s a per-hop reach of just a few meters. Two retimers in series extend this to about 7 m, which covers six to seven racks in standard layouts. Beyond that distance the substrate has to become optical, which mirrors the way long on-die paths go optical where copper wiring runs out.
Physical layout imposes the second. Cabling a rack vertically keeps the average path short at the cost of airflow, while cabling it horizontally does the opposite. Positioning switch trays near the middle of a rack narrows the spread in wire length, for the same reason that a floorplanner pulls routers toward the center of a die. Thermal and power density impose the third. Accelerator and CPU trays generate enough heat that they have to be spaced out for stable operation, while memory trays occupy additional routing area. Thermal drift that is left uncontrolled returns as a change in link behavior.
Security is the boundary that is easiest to defer and most expensive to retrofit. In multitenant environments where training and inference execute concurrently, coherent sharing creates cross-tenant paths that are not explicit. Addressing this requires isolation domains that are declared rather than assumed, together with access control enforced in hardware at both devices and switches. CXL’s integrity and data encryption (IDE) feature underneath provides the baseline for confidentiality on the wire.
Optical technology extends the reach limit but does not remove the rest. CXL-over-optics, co-packaged optics and optical backplanes can carry the same logical coordination framework across a much broader signaling domain, such that the protocol semantics above do not require a redesign. In practice, however, the adoption schedule is decided by everything around the optics: how deeply they integrate, how dense the links can be made, qualification and supply-chain maturity, day-to-day manageability, and what the total cost of ownership works out to. For hyperscale operators, this argues for early pilots and staged deployment rather than abrupt architectural replacement.
What we take from it
We believe the framing worth arguing with is the one about who pays for predictability. A network-centric fabric leaves ordering, visibility and forward progress to software, which is flexible and whose cost grows with the number of participants. A one-chip-like fabric instead spends silicon (i.e., switch fan-out, a hardware protocol pipeline and a single fabric-wide ordering policy) to bound that cost. In return, it accepts a hard physical limit at six to seven racks. Whether tray-pod-fabric is the right factoring, and whether a coherence domain of 960 accelerators is desirable compared to explicit partitioning, is exactly the discussion that the review invites. We posit that the test it proposes applies to any proposal in this space: a design is worth its complexity only if the coordination cost grows more slowly than the system itself.
Source and attribution
This article is an editorial summary prepared for Silicon and Systems. It restates the argument of the review cited below in our own words. No text, figures, tables or glossary entries from the published article are reproduced here, and the figures on this page were created for this summary. For the authoritative version, including all figures and the full reference list, please follow the DOI: 10.1038/s44287-026-00315-5. The original article is (c) Springer Nature Limited 2026.
Note: Silicon and Systems is edited by Myoungsoo Jung, corresponding author of the review. Ten of the review’s twelve authors are with Panmnesia (Daejeon and Seoul) and two are with Meta Infra at Meta (Menlo Park). The competing interests declared in the original article apply here as well, since most of its authors are employees of Panmnesia, Inc.