VAIA MeshSpec

Large language models continue to grow more capable, but inference remains expensive and often slow. The industry’s standard answer is to scale the central server: more GPUs, larger clusters, faster runtimes. This paper asks a different question, whether the coordination logic behind peer-to-peer systems can be repurposed to make LLM inference faster, cheaper, or more scalable, without surrendering the central control an enterprise system requires. We take the early architecture of Spotify as a starting analogy, examine why autoregressive generation resists a literal transplant of that architecture, and propose VAIA MeshSpec, an anchor-verified design in which a single authoritative model retains control of correctness and policy while a mesh of trusted peers performs speculative decoding, retrieval, and caching work in parallel. Our contribution is architectural and experimental rather than empirical: we state the hypothesis precisely, specify the system design that would test it, and lay out a staged experimental program built to determine where, if anywhere, this approach earns its added complexity. We also spend real space on how the idea could fail, because a proposal that only argues for itself is not a research program.

Large language models are getting more capable, but inference is still expensive and often slow. The dominant industry response has been to scale the center: larger GPU clusters, more aggressive batching, faster model runtimes. That approach works, and it will keep working for a while, but it treats the inference server as a monolith. It leaves an obvious class of resources sitting idle: nearby workstations, underused accelerators, and cheaper regional compute that never enters the picture because the serving architecture has no place for them.

This paper asks a narrower and more testable version of that observation:

We approach this question by first looking at Spotify’s early hybrid delivery architecture, a system that solved a related coordination problem in a different domain, before examining why autoregressive language generation makes the same solution much harder to copy directly. The rest of the paper follows that arc: the inspiration, the reason it breaks down for language models, what the existing literature already tells us, the gap between that literature and our question, the architecture we propose to fill it, and the experiments we are running to find out whether it is actually worth building.

The starting point for this paper is a dramatized one. In Netflix’s 2022 limited series The Playlist, particularly the episode “The Coder,” [1]Spotify’s early technical challenge is portrayed as a problem of latency and distribution. Instead of depending entirely on a central server for every request, the young company combined central infrastructure, local caching, and peer-to-peer distribution to make playback feel instantaneous. The dramatization should not be mistaken for a technical source, but the underlying engineering claim holds up well against the historical record. Spotify’s own engineering retrospective describes a client that “hybridized client-server and P2P technology” [2], and a 2010 measurement study by Kreitz and Niemelä found that only 8.8 percent of music data played in the studied deployment came from Spotify’s own servers, with median playback latency, including cached tracks, of 265 milliseconds [3].

What matters here is not music streaming itself, but the design principle it illustrates. A central system does not necessarily need to perform every piece of work; nearby machines can contribute useful computation or data while the central system remains responsible for coordination and correctness. Spotify never asked peers to replace the service. It let peers and caches absorb a narrow, well-defined class of work that was redundant, transferable, and cheap to validate. Music chunks are unusually well suited to this kind of delegation because they are static, content addressable, and can be replayed without being recomputed.

That is exactly where the analogy has to stop, and the distinction needs to be made explicit early rather than discovered later, once an architecture has already been built around a false equivalence. A token generated by a language model is not an inert chunk of data. It is the output of a sequential computation whose internal state depends on every token that preceded it, which means it cannot be cached, replayed, or verified the way a music chunk can. The remainder of this paper is concerned with identifying which parts of LLM inference behave enough like a music chunk to be safely delegated to a peer, and which parts do not.

An autoregressive language model generates one token at a time, and each new token is conditioned on every token generated before it. This sequential dependency is the entire source of the difficulty, and it is worth stating plainly because it is easy to underestimate from a distance. If a Transformer’s layers were partitioned across a chain of machines, with an early block of layers on one peer and the next block on another, every single output token would need to traverse the full chain of peers before the next token could even begin. This is precisely the model used by internet-scale collaborative inference systems such as Petals [4], and exo uses a similar peer-to-peer topology with memory-weighted partitioning across heterogeneous devices [6]. This is genuinely valuable: it lets a model run at all when no single device has enough memory to hold it. It is a different objective, however, from making a single response faster.

This is the structural reason naive peer-to-peer LLM inference can end up slower than centralized inference rather than faster. In a file distribution system, every additional peer is either neutral or helpful: it adds a redundant source of data that can simply be ignored if it turns out not to be useful. In an autoregressive decoding loop, there is no equivalent slack. Network communication is repeatedly inserted directly into the critical path, so every additional network hop is paid on every single generated token, not amortized across the response. Unlike streaming media, adding peers to the naive design does not just risk failing to help; it can actively make the system slower than not distributing the work at all.

For interactive speed, the challenge is specifically the decode loop. If a response has N output tokens, a simplified baseline for centralized decoding is:

A naive peer-to-peer layer pipeline changes the decode term so that every token also pays for multiple network transfers along the chain of peers. The result can be approximated as:

Comparing these two expressions makes the earlier claim precise: partitioning layers across peers can still improve capacity, or make a large model possible on smaller devices, but it is not automatically the best way to reduce time per token, because the summed network terms in the second equation have no equivalent in the first. The design goal for MeshSpec is therefore to move peer work off the serial decode path whenever possible, rather than inserting it directly into the path as a naive layer pipeline does.

This structural constraint sets the first research question this paper works from:

The relevant prior work falls into three largely separate lines of research: work on distributing model execution across machines, work on making centralized serving faster and more efficient, and work on speculative decoding. Each teaches a distinct lesson, and, as we argue in Section 5, no single line of work directly answers the question posed above.

4.1 Distributed model execution

Systems such as Petals [4], SWARM Parallelism [5], and exo [6]demonstrate that very large models can be distributed across multiple machines, including heterogeneous consumer hardware. Their contribution is real and should not be understated: they pool memory across devices, make large models accessible without a single high-end GPU, tolerate node failure gracefully, and put otherwise idle distributed compute to productive use. What they do not automatically provide is lower latency for a single request. Adding a slower or more distant peer to such a pipeline can reduce single-response speed even as it increases the system’s aggregate model capacity, which is a fundamentally different objective from the one this paper is optimizing for.

4.2 High-performance centralized inference

A parallel and largely independent line of work improves centralized serving without distributing it across untrusted or external machines at all. This includes vLLM and PagedAttention [7], and phase-scheduling systems such as DistServe [8], Splitwise [9], Sarathi-Serve [10], and Mooncake [11]. What unites this body of work is the insight that intelligently separating and scheduling the different phases of inference, including prefill versus decode, cache placement, and batching policy, produces large gains entirely on its own, without adding a single additional machine to the trust boundary. It is a useful reminder that architecture matters at least as much as raw compute, and it directly informs how a peer mesh should treat caching and phase separation rather than treating every request as one undifferentiated unit of work.

4.3 Speculative decoding

The third and most directly relevant line of work is speculative decoding. In this approach, a small, fast draft model proposes several candidate tokens ahead of the current position, and the large target model verifies those candidates in a single parallel pass rather than generating each token in strict sequence. Wherever the draft model’s predictions are accepted, several sequential decoding steps are effectively collapsed into one verification pass.

This mechanism was introduced by Google Research, which reported roughly a 2 to 3 times acceleration over standard decoding on T5 XXL, while provably preserving the target model’s output distribution [12]. DistillSpec subsequently showed that a draft model distilled specifically to align with the target model improves acceptance rates further, reporting an additional 20 percent speedup over standard speculative decoding [13]. More recent work has extended the idea through block-level verification [14], which verifies an entire proposed block jointly rather than token by token, and speculative cascades [15], which combine model cascades with speculative execution so that easy continuations are handled cheaply while only genuinely difficult continuations reach the largest model. Google’s 2026 research update describes block verification and tree-structured drafting as part of the techniques now used in its production inference infrastructure [16].

The observation that matters most for this paper is a structural one, not a performance number:

That is precisely the kind of work a peer, rather than the anchor itself, could perform, and it is the single fact on which the rest of this proposal is built.

Positioned against this literature, the gap is straightforward to state precisely. Existing research treats distributed model execution, speculative decoding, and disaggregated serving as largely separate problems, each solved within a single trust domain. A model is either distributed, in which case its serving is not particularly speculative, or speculative decoding is used, in which case it is typically confined to one machine, or serving is disaggregated, in which case it is disaggregated only within a data center the operator fully controls. No line of this work asks whether a central, authoritative model can treat an external, only partially trusted peer network as a source of speculative and auxiliary compute while retaining full responsibility for correctness.

That distinction is the actual research hypothesis of this paper:

We call the proposed system VAIA MeshSpec. It is a hybrid inference architecture with a deliberately asymmetric trust model: one authoritative anchor, surrounded by peers whose contributions are always optional, always verified, and never authoritative in their own right.

6.1 The VAIA Anchor

The anchor is the single authoritative server in the system. It owns the authoritative model, performs final verification of every candidate a peer produces, enforces access control and security policy, maintains the audit trail and user session, and produces the final output returned to the user. A peer never becomes authoritative under any circumstance. The anchor can always complete a request entirely on its own, which is precisely what makes the mesh an optimization rather than a dependency, and what allows the whole system to fail safely toward centralized inference whenever the mesh is unavailable, untrusted, or simply not worth using.

6.2 Draft peers

Draft peers run small models on available machines whose job is to predict likely future tokens ahead of the anchor. Given an anchor context such as “Revenue increased because…”, a draft peer might propose a continuation along the lines of “…operating margins improved following lower input costs.” The anchor then checks, in a single batched pass, whether those proposed tokens agree with what the full target model would itself have generated. Draft peers do not need to be identical to one another. A general-purpose draft model may work well for ordinary prose, while a finance-tuned draft model may achieve materially higher acceptance on investment language, and the scheduler can learn this difference over time from measured acceptance rates rather than from any assumption baked in ahead of time.

6.3 Retrieval peers

Retrieval peers do not generate tokens at all. Their role is to search a vector index, run lexical retrieval such as BM25, extract tables from source documents, rerank retrieved passages, or resolve named financial entities, all while the anchor is occupied with other work. This is one of the lower-risk points in the architecture to validate first, because retrieval output is independently inspectable and never requires target-model state to leave the anchor at all.

6.4 Cache peers

Cache peers store reusable computation: repeated prompt prefixes, document embeddings, frequently used context, and, where technically appropriate, KV-related state. Enterprise workloads reuse system instructions, tool schemas, and portfolio context far more than open-domain workloads do, which makes prefix and embedding reuse a comparatively easy early win, and a natural second experiment once draft peers have been validated.

6.5 Scheduler

The scheduler is arguably the most important component in this proposal, because it is the component that decides whether the mesh should be used at all. For every candidate task, it must estimate whether peer execution, plus network latency, plus serialization, plus verification, would be faster than simply letting the anchor do the work itself. When that sum is not favorable, the peer should not be used, full stop. This is the central design commitment of MeshSpec: the mesh is opportunistic rather than mandatory, and every path through the system must degrade gracefully to anchor-only execution when the arithmetic does not work in its favor.

Consider a user asking: “Compare Company A’s FY25 and FY26 performance and explain the major drivers.” In a conventional serial pipeline, the system would understand the question, retrieve the relevant filings, extract the relevant figures, generate a response, and validate it, one stage strictly after another. In MeshSpec, the anchor still owns understanding the question and producing the final response, but retrieval of the annual reports, extraction of the relevant financial metrics, drafting of likely continuation tokens, and lookup of cached document context can all happen concurrently on separate peers while the anchor works.

The anchor consumes whichever of these parallel results arrive in time, verifies them, and streams only the accepted output. This is the clearest illustration of where the architecture’s potential latency improvement actually comes from. It does not come from making any single step faster in isolation. It comes from overlapping steps that a serial pipeline would otherwise be forced to perform one after another, purely because nothing was available to do that work concurrently.

It follows that the key scheduling metric is not raw peer FLOPS. It is expected useful work per unit of wall-clock time. A peer with a smaller GPU but 1 ms latency and a highly aligned draft model may be more valuable to the scheduler than a much faster GPU sitting across a 40 ms link, because the faster peer’s advantage is entirely consumed by the extra round trip before its output is even usable.

We can make this precise for the speculative case. For a speculative block of k tokens, let a denote the average number of accepted tokens from each verifier pass. A rough MeshSpec decode model is then:

MeshSpec wins only when the accepted-token multiplier a is large enough to offset the transfer and scheduling overhead in that expression. This equation is what turns the architecture into a direct experimental target rather than a vague claim that distribution should be faster: it tells us exactly which quantities (accepted tokens per pass, transfer time, verification time) our experiments in Section 11 need to measure.

A more experimental extension of the design replaces a single draft model with several. Rather than one peer proposing one continuation, multiple peers can each propose a different continuation from the same position, potentially with different domain biases, so that a general peer and a finance-tuned peer diverge after a shared prefix. Where these proposals overlap, they can be merged into a single candidate tree of branching continuations, and the anchor can verify a compact version of that tree in one batched pass, accepting whichever branch turns out to be the longest valid continuation.

We want to be explicit that this is a research hypothesis, not a claim that tree-structured, multi-peer speculation will improve performance. Verifying a tree is more expensive per pass than verifying a single sequence, and whether the additional accepted tokens outweigh that added verification cost is exactly the kind of question our experimental program (Section 11) is designed to answer, rather than a conclusion we are assuming in advance.

VAIA’s workloads are unusually well suited to this architecture because they contain a large amount of naturally parallelizable work: long financial documents, retrieval across multiple filings, table extraction, financial calculation, sustainability framework retrieval, investment memo generation, due diligence, and report generation. An investment-committee memo request, for instance, does not need to be treated as a single, giant, serial call to one model. Financial analysis, market research, risk analysis, ESG assessment, management review, and comparable-company analysis can each be assigned to a different peer, with the central model synthesizing the results into one coherent deliverable rather than producing each section in turn.

This is the point we want to emphasize more than any raw decode-speed number: MeshSpec’s most valuable outcome may not be faster token generation at all, but faster workflowcompletion. An investment professional’s experience of speed is measured by how long it takes to receive a finished, defensible deliverable, not by how quickly an individual response streams to the screen. A system that decodes tokens twice as fast but still takes an analyst’s full afternoon to assemble a memo has not, in any way that matters to the client, gotten faster.

Peers in this architecture must be treated as potentially unreliable compute; the anchor is the only component that is trusted by construction. Every peer interaction is therefore built around signed requests, scoped context, and encrypted transport, with the anchor performing verification and writing to an audit log on every exchange, regardless of how trivial the delegated task appears.

Sensitive financial information must not be broadcast across the mesh indiscriminately, which means the scheduler has to treat data classification as a first-class input rather than an afterthought bolted on once the architecture is already built:

Everything described so far is a systems hypothesis, not a validated result. We are running a staged experimental program that only advances to wider and less predictable networks once the previous stage has demonstrated a clear win on controlled infrastructure, so that a failure at any stage is cheap and localized rather than discovered late, in production, against real traffic.

Experiment 2 deserves a more precise statement, because it defines the break-even condition that every later experiment ultimately depends on. Let T_peer denote draft generation time on the remote peer, T_network the round-trip network cost of sending context out and candidates back, T_verificationthe anchor’s cost of verifying the candidates, and T_saved the decoding time that would otherwise have been spent generating those same tokens sequentially on the anchor alone. Remote speculation is worth using only when:

If this inequality does not hold, remote speculation adds latency rather than removing it, and the scheduler should route the request through the anchor alone, as if the peer had never been available in the first place.

For Experiment 3, our working hypothesis is that latency as a function of peer count is not monotonic. We expect performance to improve as peers are added up to some point, then degrade as coordination and network overhead begin to dominate whatever additional compute those peers contribute. If that hypothesis holds, the practical implication is that MeshSpec has an optimal mesh size for a given network and workload, rather than a curve where more peers is unconditionally better.

Experiment 7 is included deliberately as a check against a common failure mode of systems research: a design can post an impressive tokens-per-second number in isolation while providing almost no improvement to the actual, end-to-end time a user waits for a finished deliverable. We treat that end-to-end measurement, not the isolated decoding benchmark, as the metric that ultimately decides whether this architecture was worth building.

We deliberately do not define success as “peer-to-peer inference works.” That framing is untestable, and it would let a marginal or workload-specific win be over-generalized into a claim the evidence does not support. We use a narrower and more falsifiable definition instead:

A negative result is treated as useful, not as a failure of the research program. If distributed speculation turns out to be slower than centralized decoding but distributed retrieval cuts end-to-end workflow latency by 30 percent, that narrower finding becomes the actual architecture we ship, and the speculative decoding component is set aside rather than forced into production on the strength of the original hypothesis alone.

This section is kept prominent deliberately, because the architecture is exploratory, and there are several distinct ways it can fail even if every individual component works correctly in isolation.

MeshSpec should be treated as an experimental architecture, not a predetermined solution.

The long-term ambition behind this work is not simply to run a language model over a peer-to-peer network. It is to build a compute fabric in which an intelligent scheduler dynamically decides where every part of an AI workload should execute, whether that is a datacenter GPU, an office machine, or an edge device, with the anchor verifying whatever comes back before it ever reaches the user. Different machines would contribute different capabilities to the same request, rather than each machine running an independent copy of the same pipeline in isolation.

Framed this way, the project looks less like a traditional distributed-inference system and more like an AI compute network: a scheduling and verification layer that treats compute, wherever it happens to sit, as a resource to be allocated rather than a location to be assumed. We think that framing, more than any single latency number, is the strongest long-term argument for this line of work.

VAIA MeshSpec begins with a simple question: can ideas that made distributed systems such as early Spotify efficient be reconsidered for the way modern AI systems execute? The answer is unlikely to be as simple as distributing an LLM across several machines. Autoregressive generation introduces constraints that streaming media never had to contend with, particularly around sequential computation, communication latency, and model state, and any architecture that ignores those constraints will underperform a well-tuned centralized system.

Our proposed approach therefore keeps a central, authoritative model in place while allowing surrounding peers to perform work that can happen concurrently: speculative generation, retrieval, document processing, caching, and other auxiliary computation. The central hypothesis is that the real opportunity may not lie in distributing every Transformer layer across the network, but in intelligently distributing the work that surrounds inference, while a trusted model retains verification and control over everything the peer network contributes.

VAIA MeshSpec is currently an experimental research architecture. We are running initial experiments to evaluate local speculative decoding, remote draft models, multi-peer speculation, heterogeneous compute, network sensitivity, and parallel financial AI workloads. Some of these ideas may work. Others may prove slower than conventional inference. Both outcomes are useful, because the objective of this work is to establish, experimentally, where distributed compute genuinely improves LLM systems and where it simply adds complexity without a corresponding benefit.

The architecture described in this paper should therefore be read as a starting hypothesis, not a finished system. Experiments are ongoing, and we intend to publish results, benchmarks, and what we learn as VAIA MeshSpec develops.

1. Netflix, The Playlist, episode listing and description (2022).
2. Spotify Engineering, Four Lessons We Learned from Creating Spotify’s Desktop App (2021).
3. G. Kreitz and F. Niemelä, Spotify: Large Scale, Low Latency, P2P Music on Demand Streaming, IEEE P2P (2010).
4. A. Borzunov et al., Petals: Collaborative Inference and Fine-tuning of Large Models (2022).
5. M. Ryabinin et al., SWARM Parallelism: Training Large Models Can Be Surprisingly Communication Efficient, ICML (2023).
6. exo, P2P distributed AI cluster for heterogeneous devices, project repository.
7. W. Kwon et al., Efficient Memory Management for Large Language Model Serving with PagedAttention, SOSP (2023).
8. Y. Zhong et al., DistServe: Disaggregating Prefill and Decoding for Goodput-optimized LLM Serving (2024).
9. P. Patel et al., Splitwise: Efficient generative LLM inference using phase splitting, ISCA (2024).
10. A. Agrawal et al., Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve (2024).
11. R. Qin et al., Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving, FAST (2025).
12. Y. Leviathan, M. Kalman, Y. Matias, Fast Inference from Transformers via Speculative Decoding, ICML (2023).
13. Y. Zhou et al., DistillSpec: Improving speculative decoding via knowledge distillation, ICLR (2024).
14. Z. Sun et al., Block-level verification accelerates speculative decoding, Google Research (2025).
15. H. Narasimhan et al., Faster Cascades via Speculative Decoding, ICLR (2025).
16. Google Research, A New Era of Discovery: Google Research at I/O 2026.