Mesh LLM, a newly released distributed AI platform, enables teams to pool GPU and memory resources across multiple machines to execute large language models as a single system. By functioning as an OpenAI-compatible API at http://localhost:9337/v1, the software allows users to run models locally or distribute workloads across a peer-to-peer network. This system addresses the limitations of centralized cloud providers, where users often lose control over model updates, data privacy, and variable monthly costs. The architecture is designed for pluggability, supporting over 40 models ranging from small versions suitable for laptops to massive 235B parameter mixture-of-experts giants. For models exceeding the capacity of a single GPU, Mesh LLM employs a split mode called 'Skippy' that partitions models by layer ranges into stages. Activations are streamed between modest hardware nodes, allowing a pipeline of multiple machines to run models that would otherwise be impossible to host on individual devices.

The underlying infrastructure relies on iroh, a networking tool that creates authenticated, NAT-traversing QUIC connections between nodes without requiring a central server. Every node in the mesh acts as an iroh endpoint identified by a public key, handling communication through three specific protocols: mesh-llm/1 for gossip and routing, mesh-llm-control/1 for owner configuration, and skippy-stage/2 for latency-sensitive activation transport between pipeline stages. To ensure reliable connectivity across the internet, the platform uses two iroh relays in different regions as fallback paths for nodes that cannot establish direct peer-to-peer links. The software itself is lightweight at approximately 18 MB. Future development plans include adding support for the emerging agent communication protocol (ACP) and releasing a mobile application built on the iroh Swift SDK to further expand the capabilities of private, peer-to-peer AI infrastructure.

Mesh LLM, a newly released distributed AI platform, enables teams to pool GPU and memory resources across multiple machines to execute large language models as a single system. By functioning as an OpenAI-compatible API at http://localhost:9337/v1, the software allows users to run models locally or distribute workloads across a peer-to-peer network. This system addresses the limitations of centralized cloud providers, where users often lose control over model updates, data privacy, and variable monthly costs. The architecture is designed for pluggability, supporting over 40 models ranging from small versions suitable for laptops to massive 235B parameter mixture-of-experts giants. For models exceeding the capacity of a single GPU, Mesh LLM employs a split mode called 'Skippy' that partitions models by layer ranges into stages. Activations are streamed between modest hardware nodes, allowing a pipeline of multiple machines to run models that would otherwise be impossible to host on individual devices.

The underlying infrastructure relies on iroh, a networking tool that creates authenticated, NAT-traversing QUIC connections between nodes without requiring a central server. Every node in the mesh acts as an iroh endpoint identified by a public key, handling communication through three specific protocols: mesh-llm/1 for gossip and routing, mesh-llm-control/1 for owner configuration, and skippy-stage/2 for latency-sensitive activation transport between pipeline stages. To ensure reliable connectivity across the internet, the platform uses two iroh relays in different regions as fallback paths for nodes that cannot establish direct peer-to-peer links. The software itself is lightweight at approximately 18 MB. Future development plans include adding support for the emerging agent communication protocol (ACP) and releasing a mobile application built on the iroh Swift SDK to further expand the capabilities of private, peer-to-peer AI infrastructure.