A New Framework for Understanding Emergence in Complex Systems

Stay tuned at the end of this article for a practical guide on how to apply the ideas from this paper to your life and work.

Scientists and philosophers have grappled with the concept of emergence in complex systems for a long time. Emergent phenomena are those fascinating instances where the whole becomes greater than the sum of its parts, which gives rise to unexpected behaviors and properties.

We’re familiar with examples of emergence such as the collective intelligence of ant colonies or the consciousness emerging from billions of neurons, for example. Even though we've been able to observe and describe emergent behaviors, predicting them has remained largely out of reach. This means that when it comes to dealing with complexity we are left being more reactive than proactive.

Recent research has given us some new tools. By bridging concepts from information theory, computational mechanics, and complex systems science, researchers have developed a theoretical framework that, for the first time, provides a formal foundation for understanding and potentially predicting emergent phenomena.

The researchers introduce several key concepts and demonstrate their interrelationships—causal, informational, and computational closure. These concepts help us understand when a system's behavior at one level can be fully explained or predicted without needing to know all the details of what's happening at lower levels. Here are the key concepts in more detail:

• Causal Closure: When a macroscopic process (a higher-level description of a system) contains all the causal information needed to predict its own future.
• Information Closure: When knowing the microscopic details doesn't improve predictions about the macroscopic process beyond what the macroscopic process itself provides.
• Computational Closure: When the computations at a macroscopic level can be described independently of the underlying microscopic processes.

Let’s make these abstract concepts more tangible. Imagine a flock of birds. Causal closure would mean that we can predict the flock's future movement just by looking at its current shape and direction, without needing to know the exact position of each bird. Informational closure suggests that knowing the detailed movements of individual birds wouldn't give us any better predictions about the flock's behavior than we can already make from watching the flock as a whole. Computational closure implies that we can describe the rules governing the flock's behavior without referencing individual birds at all.

To help analyze these concepts, the team developed mathematical tools called ε-machines and υ-machines. Think of these as different ways of mapping the same territory—one from ground level and one from a bird's eye view. These "maps" help us understand how processes work at different scales in a complex system.

The researchers also demonstrated that emergent processes form a hierarchical structure. This is like realizing that in a city there are patterns of behavior at the level of individuals, families, neighborhoods, and the city as a whole. Each level builds on the last but is not entirely predictable from the one below.

By connecting their ideas to the concept of "lumpability" in Markov chains, the team built a bridge between their new theory and existing mathematical tools. Markov chains are mathematical systems that model step-by-step processes where the probability of each step depends only on the state attained in the previous step. "Lumpability" in this context refers to the ability to simplify these chains by grouping states together without losing their essential predictive properties. This connection is like discovering that a new, exotic alloy has properties that can be described using well-established principles of metallurgy. It allows insights from the new theory to be interpreted and applied within the framework of familiar, widely-used mathematical models, potentially accelerating the adoption and practical application of these novel concepts across various scientific disciplines.

Perhaps one of their most intuitive contributions is the idea of viewing emergent processes as "software-like" phenomena running on the "hardware" of lower-level systems. This analogy helps us understand how similar patterns can emerge in seemingly different systems. For example, think of a traffic jam. The emergent behavior of traffic flow (the "software") can be similar whether the "hardware" is cars on a highway, people in a crowded coffee shop, or data packets in a computer network. In each case, we see patterns of congestion, flow, and gridlock emerging from the interactions of individual units, despite the underlying systems being vastly different at the microscopic level. This perspective helps explain why insights from one field, like traffic engineering, can sometimes be applied to seemingly unrelated areas, such as managing data flow in telecommunications networks.

Finally, the researchers demonstrated the broad applicability of their framework by applying it to various systems. They showed how these ideas can be used to understand phenomena as diverse as the behavior of ant colonies, the dynamics of financial markets, and the emergence of consciousness from neural activity.

What I find fascinating about this work is that the researchers—themselves from a wide diversity of fields—created a mathematical and conceptual toolkit for identifying, characterizing, and potentially predicting emergent phenomena in complex systems. This offers promise for us being able to ground the often nebulous concept of emergence in a more formal and quantitative theory that we can apply to more situations.

A Practical Guide to Applying Emergence Theory to Complex Problems

Here's how you can apply these concepts to tackle complex problems:

1. Identify Potential Emergent Levels:
   Examine your system at multiple scales, looking for levels where behavior seems to stabilize or take on new characteristics. These might be candidates for emergent phenomena. For instance, in a retail company, examine sales patterns at individual store, regional, and national levels. You might find that regional trends emerge that can't be explained by individual store performances alone.
2. Assess Predictive Power:
   Test predictions at different levels of your system. If you can make accurate predictions about a level's future state using only information from that same level, you may have found a causally closed level. For instance, for a hotel chain, test if you can predict overall occupancy rates and revenue for the entire chain using only aggregated data from each city or region, without requiring information about individual hotel performances or customer bookings.
3. Evaluate Information Sufficiency:
   Determine if additional lower-level information improves your ability to predict higher-level outcomes. If not, you've likely identified an informationally closed level. For instance, for a manufacturing company, determine if detailed data on individual machine performance improves overall production line efficiency predictions compared to using aggregate production line metrics.
4. Map Information Flow:
   Trace how information moves through your system. Look for bottlenecks or levels where information seems to be aggregated or transformed. For instance, in a multinational corporation, trace how strategic decisions flow from global headquarters to regional offices to local branches, identifying where information is aggregated or transformed.
5. Design Targeted Interventions:
   Use your understanding of closed levels to design interventions that are more likely to have system-wide effects. Focus on manipulating variables at levels exhibiting closure properties. For instance, in a customer service department, implement changes at the team level (like new collaboration tools) rather than focusing on individual employee performance, if team-level metrics show closure properties.
6. Simplify Models Strategically:
   Develop simplified models of your system at levels exhibiting closure. These models might capture essential dynamics without needing to incorporate all lower-level details. For instance, for a supply chain network, develop a model that predicts overall delivery times using only warehouse-level data, without needing to incorporate individual package tracking details.
7. Analyze System Decomposition:
   Use closure properties to guide how you break down complex systems into more manageable components. Components exhibiting closure might be treated as coherent units. For instance, in a large e-commerce platform, break down the system into components like user interface, recommendation engine, and payment processing, treating each as a coherent unit if they exhibit closure properties.
8. Look for Transferable Patterns:
   Identify patterns of emergence that seem similar across different systems in your field. These might represent fundamental organizational principles. For instance, identify similar patterns in how product teams collaborate across different industries, which might reveal fundamental principles of effective teamwork regardless of the specific product being developed.
9. Anticipate Emergent Behaviors:
   Use your understanding of closure to anticipate where and how new behaviors might emerge in your system as it evolves or as you make changes. For instance, for a global fast-food chain, leverage closure concepts to predict how the introduction of a new menu item might lead to emergent trends in customer ordering patterns, potentially affecting the popularity of other menu items or creating unexpected combinations that become popular.
10. Assess System Robustness:
    Evaluate how resistant emergent behaviors are to lower-level perturbations. Highly robust behaviors might indicate strong closure properties. For instance for a water supply network, assess how well the overall water supply system maintains consistent pressure and quality despite variations in individual pump stations or treatment facilities. A system that remains stable despite local fluctuations could demonstrate information closure at a higher level.

By applying these approaches, informed by the concepts of causal, informational, and computational closure, you can develop a better understanding of complex systems in your field. This can lead to more effective analysis, prediction, and management of emergent phenomena, whether you're dealing with organizational behavior, ecosystem dynamics, or technological systems.