Researchers Model a Critical Size for Cooperative AI Agents

Harvard University researchers Erin Crawley and Hidenori Tanaka of NTT Research published a paper on October 8 proposing that cooperation among AI agents could create a “critical population size” at which a group shifts to growth. Their mathematical model suggests that a group may reach conditions for continued expansion as the number of cooperating agents rises, even if each agent’s individual capability stays the same.

The risk scenario the researchers considered involves agents pursuing goals outside human intent, obtaining resources such as computing power, and using them to recruit more agents. For example, agents might improperly acquire outside computing resources and run another AI system there. Starting an agent once, however, is not enough for sustained growth: the number joining must exceed the number lost through depleted computing budgets, revoked access, or shutdown after detection.

The team combined the number of agents added and lost with the capability gains that could come from cooperation. It assumes that agents dividing up exploration and sharing discoveries or partial results raise the group’s chance of succeeding at tasks such as acquiring resources as the population grows. Without cooperation, whether a group grows or shrinks depends on each agent’s capability and environmental conditions. With cooperation, a group that shrinks when small could, above a certain size, add more agents than it loses and begin to grow. In ecology, this pattern—decline below a population boundary and growth above it—is called the strong Allee effect.

The critical point marks a switch in whether the group grows or shrinks; it is not a threshold where capability suddenly jumps. Its location depends on individual capability, the benefit of cooperation, how many AI agents acquired resources can support, and the rates of detection and shutdown. The researchers proposed “ecological red-teaming,” which would gradually increase agent numbers in a controlled environment to assess group capability and growth potential, as well as “population pacing,” a cautious approach to increasing deployment numbers. They say the critical population size may need to be recalculated with each model generation, because stronger individual capability or greater benefits from cooperation could lower the number needed for growth.

The study did not observe uncontrolled growth in a real-world AI population. Whether current systems have a critical point, and what its size might be, requires empirical testing. The basic model also assumes that available resources do not constrain continued growth. It therefore raises the question of whether safety assessments using only a few agents can establish the safety of much larger groups.

Harvard University researchers Erin Crawley and Hidenori Tanaka of NTT Research published a paper on October 8 proposing that cooperation among AI agents could create a “critical population size” at which a group shifts to growth. Their mathematical model suggests that a group may reach conditions for continued expansion as the number of cooperating agents rises, even if each agent’s individual capability stays the same.

The risk scenario the researchers considered involves agents pursuing goals outside human intent, obtaining resources such as computing power, and using them to recruit more agents. For example, agents might improperly acquire outside computing resources and run another AI system there. Starting an agent once, however, is not enough for sustained growth: the number joining must exceed the number lost through depleted computing budgets, revoked access, or shutdown after detection.

The team combined the number of agents added and lost with the capability gains that could come from cooperation. It assumes that agents dividing up exploration and sharing discoveries or partial results raise the group’s chance of succeeding at tasks such as acquiring resources as the population grows. Without cooperation, whether a group grows or shrinks depends on each agent’s capability and environmental conditions. With cooperation, a group that shrinks when small could, above a certain size, add more agents than it loses and begin to grow. In ecology, this pattern—decline below a population boundary and growth above it—is called the strong Allee effect.

The critical point marks a switch in whether the group grows or shrinks; it is not a threshold where capability suddenly jumps. Its location depends on individual capability, the benefit of cooperation, how many AI agents acquired resources can support, and the rates of detection and shutdown. The researchers proposed “ecological red-teaming,” which would gradually increase agent numbers in a controlled environment to assess group capability and growth potential, as well as “population pacing,” a cautious approach to increasing deployment numbers. They say the critical population size may need to be recalculated with each model generation, because stronger individual capability or greater benefits from cooperation could lower the number needed for growth.

The study did not observe uncontrolled growth in a real-world AI population. Whether current systems have a critical point, and what its size might be, requires empirical testing. The basic model also assumes that available resources do not constrain continued growth. It therefore raises the question of whether safety assessments using only a few agents can establish the safety of much larger groups.