Kuber.studio published “Recreating Minecraft Is Not a Benchmark” on September 6, 2026, arguing that viral AI launch demos such as recreating Minecraft in one prompt no longer measure model capability reliably. The post says GPT Astra had been released “a couple of days ago,” and within the hour the author’s feed filled with the same five examples: Minecraft recreation, MS Paint self-portraits, a pelican riding a bicycle as an SVG, a ball bouncing in a rotating box with believable gravity, and an SVG game controller.

The author calls these examples “demo-benchmarks”: visual tasks that are easy for broad audiences to understand but finite enough for the next model to become “perfect” on. The post argues that a fixed, famous target plus eight weeks of runway lets AI labs optimize directly for those demonstrations, so the result measures launch preparation more than general capability.

The same problem appears in public evaluations, according to the post. Thinking Machines’ Inkling Small scored within a point of its flagship sibling on the Artificial Analysis Intelligence Index with less than a third of the parameters, and beat it on Humanity’s Last Exam, GPQA Diamond and SciCode. The author says public, static and famous test sets can leak into training data and fine-tuning choices.

The post points to rotating or hidden evaluations as stronger alternatives: LiveBench rotates questions, ARC-AGI keeps a private set, and Humanity’s Last Exam holds part of itself back. The author still says strong demos matter because social media can notice quickly when a bicycle finally has pedals or animation, but demo-benchmarks should not be used for grading model quality.

Kuber.studio published “Recreating Minecraft Is Not a Benchmark” on September 6, 2026, arguing that viral AI launch demos such as recreating Minecraft in one prompt no longer measure model capability reliably. The post says GPT Astra had been released “a couple of days ago,” and within the hour the author’s feed filled with the same five examples: Minecraft recreation, MS Paint self-portraits, a pelican riding a bicycle as an SVG, a ball bouncing in a rotating box with believable gravity, and an SVG game controller.

The author calls these examples “demo-benchmarks”: visual tasks that are easy for broad audiences to understand but finite enough for the next model to become “perfect” on. The post argues that a fixed, famous target plus eight weeks of runway lets AI labs optimize directly for those demonstrations, so the result measures launch preparation more than general capability.

The same problem appears in public evaluations, according to the post. Thinking Machines’ Inkling Small scored within a point of its flagship sibling on the Artificial Analysis Intelligence Index with less than a third of the parameters, and beat it on Humanity’s Last Exam, GPQA Diamond and SciCode. The author says public, static and famous test sets can leak into training data and fine-tuning choices.

The post points to rotating or hidden evaluations as stronger alternatives: LiveBench rotates questions, ARC-AGI keeps a private set, and Humanity’s Last Exam holds part of itself back. The author still says strong demos matter because social media can notice quickly when a bicycle finally has pedals or animation, but demo-benchmarks should not be used for grading model quality.