I'm continuing on with running through the (probably not real) Ilya Stuskever deep-learning reading list and this week read the first law of complexodynamics. It's a fairly short blog post, and the ideas in it are a little ill-defined (intentionally) but quite fun.

If you haven't read it, my summary is going to be a lot less helpful than just reading it yourself, especially since it isn't that long. But the main aim of the blog post is to try to flesh out this curiousity:

even though isolated physical systems get monotonically more entropic, they don’t get monotonically more “complicated” or “interesting.”

The main example in the article is a cup with coffee in the bottom half, milk in the top. In this state, it's low entropy (since the liquids are fully seperated) but also quite simple, and not especially "interesting" or "complex". As the coffee and milk mix, we get "tendrils of milk dispersed in the coffee" and some more interesting / complex interactions. Finally, once the coffee and milk are fully mixed, we have a state that is now low complexity, and high entropy.

The curiousity here is that while entropy increases over time, complexity increases but then decreases. The blog post is mostly focused with posing this problem, and sign-posts potential avenues for exploration, but doesn't directly attempt to answer it.

I was left puzzling a little about why this was on a deep learning reading list, possibly because neural networks interact together to produce increasingly complex behaviours, but that may be a stretch. I'm not sure neural networks are particularly entropic systems, if anything, as they're trained, they tend towards a lower state of entropy rather than a higher one, because of the highly specified rules for weight distribution.

It's an interesting puzzle nonetheless - I found it very hard not to try and cache out the problem in terms of emergence rather than entropy. Where following simple rules can lead to more complex behaviour. I think emergence is a great concept to explore, and I'd urge anyone interested in systems to read up on it generally, but in terms of the coffee cup mix, I think we're seing something like this:

• State one, full seperation of milk and coffee: Very low interaction between elements, low entropy, and virtually zero emergent behaviour
• State two, mixing of milk and coffee: Interaction of elements in "complex" and "interesting" ways, such as "tendrils" of milk caused by effects like heat pockets in coffee. This is the impact of small rules, like heat distribution, causing emergent behaviour, like tendrils.
• State three, full mixed milk and coffee: High entropy, interaction is no longer emergent due to stable state of drink.

Assuming this is a good explanation of why complexity increases then decreases, what we're seeing is something like "rule/interaction collapse" caused by the increase of entropy.

What's interesting, to me at least, is that a lot of systems don't conform to this, particularly ones that are self-enforcing. If i think of something like Grime's "Evidence for the Existence of Three Primary Strategies in Plants and Its Relevant to Ecological and Evolutionary Theory" (I PROMISE this paper is a fascinating exploration of plant ecosystems following emergent behaviour, and a lot less boring than it sounds), systems like plant ecosystems become more complex and more interesting and they continue. Evolution is a great example of this too, where species tendency to "fill gaps" leads to increasingly complexity over time.

Of course, on a big enough time scale, everything just descends into the end of the universe, but that's probably not a good analysis of meadows etc, and just a fact of universal entropy over time.

Overall, the blog post has some really interesting and fun ideas in it. It's left me wondering about which systems do see rule collapse, and which dont', and why that might be.