A developer successfully implemented a 28.9 million parameter language model on an ESP32-S3 microcontroller, a device typically priced at $8. This system runs entirely on-chip without requiring server connectivity, achieving a generation speed of roughly 9.5 tokens per second. The model is optimized for the hardware's constraints by storing the vast majority of its parameters in slow flash memory rather than the device's 512KB SRAM. This approach uses Per-Layer Embeddings, a technique originally associated with Google's Gemma models, allowing the device to read only the necessary 450 bytes of data for each token while keeping the core processing logic in fast memory.

The model occupies 14.9MB when quantized at 4-bit, with 25 million of its 28.9 million parameters stored in a flash lookup table. This represents a significant increase in parameter count for a low-cost microcontroller; previously, similar hardware implementations were restricted to approximately 260 thousand parameters. The hardware configuration includes 8MB of PSRAM and 16MB of flash storage. While the system demonstrates a new method for scaling model sizes on constrained edge devices, its output capability is limited because the reasoning components remain constrained by the chip's small SRAM. The model was trained on the TinyStories dataset, resulting in coherent, simple story generation rather than general instruction-following or factual query responses.

The project relies on specific design principles to function within the ESP32-S3 memory architecture, effectively separating the 'thinking' core in SRAM, working memory in PSRAM, and the large lookup table in flash. The developer documented the full methodology, firmware, and ablation experiments in an open-source repository. This work builds upon existing research, including TinyStories for dataset generation and the llama2.c project for running models in C. The developer noted that the final results were verified after correcting an earlier accounting bug in parameter reporting, with full measurement data available in the project's documentation.

A developer successfully implemented a 28.9 million parameter language model on an ESP32-S3 microcontroller, a device typically priced at $8. This system runs entirely on-chip without requiring server connectivity, achieving a generation speed of roughly 9.5 tokens per second. The model is optimized for the hardware's constraints by storing the vast majority of its parameters in slow flash memory rather than the device's 512KB SRAM. This approach uses Per-Layer Embeddings, a technique originally associated with Google's Gemma models, allowing the device to read only the necessary 450 bytes of data for each token while keeping the core processing logic in fast memory.

The model occupies 14.9MB when quantized at 4-bit, with 25 million of its 28.9 million parameters stored in a flash lookup table. This represents a significant increase in parameter count for a low-cost microcontroller; previously, similar hardware implementations were restricted to approximately 260 thousand parameters. The hardware configuration includes 8MB of PSRAM and 16MB of flash storage. While the system demonstrates a new method for scaling model sizes on constrained edge devices, its output capability is limited because the reasoning components remain constrained by the chip's small SRAM. The model was trained on the TinyStories dataset, resulting in coherent, simple story generation rather than general instruction-following or factual query responses.

The project relies on specific design principles to function within the ESP32-S3 memory architecture, effectively separating the 'thinking' core in SRAM, working memory in PSRAM, and the large lookup table in flash. The developer documented the full methodology, firmware, and ablation experiments in an open-source repository. This work builds upon existing research, including TinyStories for dataset generation and the llama2.c project for running models in C. The developer noted that the final results were verified after correcting an earlier accounting bug in parameter reporting, with full measurement data available in the project's documentation.