There is a design shift as software starts moving to the cloud, from an in-prem situation.
There are a few changes that you have to make in your design. e. g. the database may not always be available at all times, so retrying a connection makes sense.
One of the side-effects of this is that we can use some design patterns from nature, or the hardware world.
Using Hadoop or (as it it is called) HDInsight in Azure, we can have very simple functional blocks perform very complex things.
This is kinda why Map/Reduce was invented in the first place by Google.
Anyway, what this means for our software design, is to have more modular blocks (or classes and methods), and they should perform a set of tasks very well. They won’t even know of the big picture, and don’t need to.
Any good software design more or less is designed this way but try to make the units kind of self-contained, rather than always having a master-controller somewhere.
Not having a master-controller is the ultimate path to infinite growth, because then each unit does not have to come back to the master.
Only some factory method needs to be able to be called and produce these identical worker units.
More and more, this kind of system resembles of the best examples in nature viz. ants.
Take a look at this article and video here.