We’re starting to get inquiries about complexity. Key questions are how to evaluate complexity in an IT organization and consequently how to evaluate its impact on availability and performance of applications. Evaluating complexity wouldn’t be like evaluating the maturity of IT processes, which is like fixing what’s broken, but more like preventive maintenance: understanding what’s going to break soon and taking action to prevent the failure.
Volume of application and services certainly has something to do with complexity. Watts Humphrey said that code size (in KLOC: thousands of lines of code) doubles every two years, certainly due to increase in hardware capacity and speed, and this is easily validated by the evolution of operating systems over the past years. It stands to reason that, as a consequence, the total number of errors in the code also doubles every two years.
But code is not the only cause of error: Change, configuration, and capacity are right there, too. Intuitively, the chance of an error in change and configuration would depend on the diversity of infrastructure components and on the volume of changes. Capacity issues would also be dependent on these parameters.
There is also a subjective aspect to complexity: I’m sure that my grandmother would have found an iPhone extremely complex, but my granddaughter finds it extremely simple. There are obviously human, cultural, and organizational factors in evaluating complexity.
Can we define a “complexity index,” should we turn to an evaluation model with all its subjectivity, or is the whole thing a wild goose chase?
It’s rumored that the Ford Model T’s track dimension (the distance between the wheels of the same axle) could be traced from the Conestoga wagon to the Roman chariot by the ruts they created. Roman roads forced European coachbuilders to adapt their wagons to the Roman chariot track, a measurement they carried over when building wagons in America in the 19th and early 20th centuries. It’s said that Ford had no choice but to adapt his cars to the rural environment created by these wagons. This cycle was finally broken by paving the roads and freeing the car from the chariot legacy.
IT has also carried over a long legacy of habits and processes that contrast with the advanced technology that it uses. While many IT organizations are happy to manage 20 servers per administrator, some Internet service providers are managing 1 or 2 million servers and achieving ratios of 1 administrator per 2000 servers. The problem is not how to use the cloud to gain 80% savings in data center costs, the problem is how to multiply IT organizations’ productivity by a factor of 100. In other words, don’t try the Model T approach of adapting the car to the old roads; think about building new roads so you can take full advantage of the new technology.
Gains in productivity come from technology improvements and economy of scale. The economy of scale is what the cloud is all about: cookie cutter servers using virtualization as a computing platform, for example. The technology advancement that paves the road to economy of scale is automation. Automation is what will abstract diversity and mask the management differences between proprietary and commodity platforms and eventually make the economy of scale possible.
Most, if not all, technology improvements need what is commonly referred to as “complementary inputs” to yield their full potential. For example, Gutenberg's invention of movable type wouldn't have been viable without progress in ink, paper, and printing press technology. IT innovations depend on complements to take hold. The use of internal cloud differences will affect applications, configuration, monitoring, and capacity management. External clouds will need attention to security and performance issues related to network latency. Financial data availability is also one important cloud adoption criteria and must be addressed. Without progress in these complementary technologies, the benefits of using cloud computing cannot be fully developed.
Internal cloud technology is going to offer embedded physical/virtual configuration management, VM provisioning, orchestration of resources, and most probably, basic monitoring or data collection in an automated environment, with a highly abstracted administration interface. This has the following impact:
More than ever, we need to know where things are. Discovery and tracking of assets and applications in real time is more important than ever: As configurations can be easily changed and applications easily moved, control of the data center requires complete visibility. Configuration management systems must adapt to this new environment.
Applications must be easily movable. To take advantage of the flexibility offered by orchestration, provisioning, and configuration automation, applications must be easily loaded and configured. This assumes that there is, upstream of the application release, an automated process that will “containerize” the applications, its dependencies, and its configuration elements. This will affect the application life cycle (see figure).
A few days ago I read an interesting article about how organizations need to adapt to virtualization to take full advantage of it.
If we consider that this is, in fact, the first step toward the industrialization of IT, we should consider how the organization of industry evolved over time, from the beginning to the mass-production era. In fact, I think IT will reach the mass-production stage within a few years. If we replicate this evolution in IT, it will go through these phases:
The craftsperson era. At the early stage of any industry, we find a solitary figure in a shop soon complemented by similarly minded associates (this is me, 43 years ago). They create valuable and innovative products, but productivity and cost per unit of production is usually through the roof. This is where IT was at the end of the 1960s and the beginning of the 1970s. The organization landscape was dominated by “gurus” who seemed to know everything and were loosely coupled within some kind of primitive structure.
The bureaucratic era. As IT was getting more complex, an organizational structure started to appear that tended to “rationalize” IT into a formal, hierarchical structure. In concept, it is very similar to what Max Weber described in 1910: a structure that emphasizes specialization and standardization in pursuit of a common goal. Tasks are split into small increments, mated to skills, and coordinated by a strong hierarchical protocol. The coordination within the organization is primarily achieved through bureaucratic controls. This is the “silo” concept.