Chaotic Innovation: Give Mavericks the Keys to the Castle

This post is inspired by a recent article by Joe McKendrick, one of the more fair and balanced SOA bloggers in the 'sphere, entitled, "Is 'rogue' IT always a bad thing?" He points out that, on occasion, employees outside the ordained chain of IT command are guilty of developing ad hoc technology solutions that do not conform to prevailing corporate standards. In the two examples he provides, the rogue solutions provided clear benefits to the business. However, recognizing that such vigilantism by maverick techies runs the risk of undermining important governance controls, he asks his readers whether IT departments should crack down on this behavior. This question cuts to the heart of my theme of Chaotic IT, so I'll proudly offer my take, arguing that IT should not only allow it but encourage and reward it.

First, we must understand that businesses are complex systems; each is composed of myriad independent "parts" (employees, information systems, teams, customers, partners, and so on) that work together to meet business goals. What makes them complex is their endless variety of parts (e.g., no two employees are the same) and the sophisticated web of relationships they make up. This inherent complexity makes the behavior of the business largely unpredictable, and generally uncontrollable by those relatively few parts in leadership positions (just as ant colonies or bee swarms are not controlled by their respective queens). When we are at peace with this, we are at peace with the fact that no one individual, or group of individuals, runs our businesses--we all do, collectively.

One important business theme of the modern era is agility. We must innovate and adapt faster than ever before because, if we don't, our competitors--or a twenty-year-old with a wild hair and a Ruby on Rails book--will. So how do we make our businesses more fluid, adaptive, responsive? It would be great if the answer came in a box. Then we could simply buy it, plug it in, go to the beach, and giggle into our Treos as we watch our stock blow the roof off of Wall Street. But, alas, there is no shrink-wrapped silver bullet. (Any CIO that has invested in SOA technology should know this first hand by now.) The good news, though, is that many of the resources and infrastructure companies need to become agile are already baked into their businesses. They just need to be discovered and exploited.

Some notable thought leaders lately have put forth the concept of initiating "change from within" or "inside-out" transformation to achieve new levels of competitive advantage. While the terminology differs slightly among them, the idea is consistent: that the solutions to a company's strategic innovation challenges already exist within the organization, most likely within the minds of employees down in the tranches, not the leaders at the helm. Whereas the leaders may have a mental picture of the solution and an inclination to spark a project to bring it to fruition, employees at the front lines may already possess finished, tested products that are ready to put before real customers. It stands to reason that if this is indeed the case, companies are better off seeking out this innate knowledge and cultivating it rather than attempting to invent it from scratch. Tim O'Reilly calls these key individuals "alpha geeks" that exist at the "edges" of their companies. Malcom Gladwell, in his book The Tipping Point, calls them "Mavens." Richard Tanner Pascale and Jerry Sternin, in a May 2005 Harvard Business Review article, call them "secret change agents" and "positive deviants." (It is this last term that, to me, resonates most closely with McKendrick's notion of "rogue" engineers.) These authors regard individuals outside mainstream IT as critical to the success of corporate change initiatives, even though the knee-jerk reaction to their efforts often is to apply negative labels to them, such as "deviant," "rogue," "vigilante," "subversive," or "illegal." These individuals, while they seldom occupy positions of high authority, have a deep understanding of both technology and the business drivers that fuel innovation. What's more, they exist at all levels of the organization and can significantly outnumber the mainstream leaders. It is as if every company has a secret underground reservoir of extreme innovation talent that, when tapped, can bring about significant positive changes in agility levels. By leveraging the efforts of these so called mavericks, a company, at the very least, is able to cast a wider net over its sea of innovation possibilities.

Finally, I would like to add my own point of view to this topic--the chaotic take. As I mentioned before, businesses are complex systems. As such, they behave much more like living organisms than machines and are constantly influenced by forces beyond anyone's direct control. (See my previous post entitled Snowball Oriented Architecture in which I introduced the concept of corporate physics. I'll also expand on this in more detail in future posts.) Since businesses exhibit the fundamental characteristics of living things, I look to the natural world for clues about how businesses--and their IT departments--may operate more competitively. One observation I have made is that living things--all of them-- are inherently agile. If we learn what makes them agile perhaps we can apply the same principles to our businesses.

From cells to bacteria to people to herds to flocks to swarms to entire ecosystems, all living systems, like businesses, must undergo constant transformation in response to changes in their environments for the sake of maximizing their odds of survival. It's a game of evolution, no less real for corporations than it is for Galapagos finches or the influenza virus. We all play by the same rules.

So what makes living things naturally agile? It is simply the ability of their constituent parts to rapidly self-organize in the face of some external threat. This is a two step process. First, the threat must be detected quickly. Second, a positive transformation must take place quickly to counteract the threat. The first step is made possible by the awareness of each part to its immediate environment, and the second by its ability to react without first obtaining "permission." For example, when a person touches a hot burner on the stove, sensory cells in the hand register the threat and react immediately by sending an electrical impulse to the brain. Also, in order for evolution to work, individual strands of DNA must be allowed to mutate on their own, without direction from the brain or other external source. These small, local deviations from the norm are then detected by nearby elements and propagated if beneficial or rejected if not. This cycle of "selection" continues until all components have been transformed or the variation has been eradicated, depending on its overall usefulness. The important things to remember are that 1) adaptive changes are nearly always initiated by components at the edges of complex systems, without the knowledge of a central source of authority and 2) that positive change sweeps through these complex systems from the bottom up.

So, "rogue" initiatives are healthy, natural processes that foster a spirit of constant experiementation and reinvention within our businesses, which is crucial for building agility. Sometimes the results of this experimentation will be beneficial, sometimes not, which is why governance models should incorporate feedback mechanisms to discover and evaluate the "fitness" of innovations that flow from the bottom up. In my opinion, this practice is much more contructive than cracking down on rigid policies that keep the flow of creativity traveling down a one way street from the top down. So empower the mavericks, turn them into sensory receptors and change agents for the business, and let them know that the leaders are listening.

I didn't originally intend for this post to be so lengthy, but I believe the decision to allow or disallow "rogue" innovations will have a critical impact on most companies' quest for agility. It may seem counterintuitive to grant employees the autonomy to act according to their own judgment based on cues perceived only by them. In fact, some would probably say that it invites chaos into the business. My point exactly.

SOA: Snowball Oriented Architecture

I don't consider myself a cartoon freak, but I have always had a warm place in my heart for them. I was practically raised on Loony Tunes and today I can't get enough of a disturbingly addictive little show called Happy Tree Friends. In one recent podcast episode, From Hero to Eternity, was a scene where a cute little chipmunk, disoriented after almost being blown up by a flying squirrel, stumbled off the edge of a cliff. She landed on a steep, snow-covered hill and tumbled down it, gathering snow as she went. Faster and faster she spun until she was encased in a snowball. The snowball soon became gigantic, sucking every living creature in its path into its swelling core, and threatened to pummel a small, hapless village full of cutesy mountain animals. I'm no buzzkill so I'll stop there with the details.

Now, I've seen this "snowball barreling down a hill" bit a hundred times in cartoons. (Who hasn't?) But I grew up in Colorado, so I know a thing or two about snowball physics, and it just doesn't work this way in real life. Anyone who has tried this knows that if you push a little snowball down a hill it does not speed down a straight path and become a perfectly spherical, gigantic version of itself. What really happens is that the little snowball gathers snow in odd places as it's rolled, quickly becomes lopsided and pulls in one direction. When it picks up enough mass to roll on its own, it looks more like a lumpy log than a smooth snowball. At this point, gravity and inertia take over and it's anyone's guess where the thing will ultimately end up and what it will look like when it gets there. So all those cartoons ignore the subtle, cumulative effects that various environmental conditions have on the outcome of the event - things like the stickiness of the snow; the steepness and shape of the slope; the shape, weight, and speed of the snowball; friction, wind, and even the outside temperature. In the cartoon world things are simple and predictable. In the real word they are very complex, even for things as seemingly simple as a snowball rolling down a hill.

I believe many people involved in Service Oriented Architecture are doing the equivalent of pushing a cartoon snowball down a frictionless slope, trusting that if they form a small initiative and give it a nudge in the right direction it will blossom into large-scale success under its own momentum and meet business objectives dead-on at the end of the journey. However, in reality, all change initiatives, SOA in particular, are subject to the constant effects of "corporate physics" - politics, culture, skill sets, methodologies, technology, sponsorship, governance, communication, vendor relationships, budgets, strategy, and so on - just as real snowballs fall under the constant influence of physical laws of gravity, motion, friction, and chemistry. Additionally, unexpected subtleties of any of these effects can lead to dramatic shifts in direction. For instance, a rock buried beneath the snow can throw a rolling snowball wildly off course. Likewise, an unexpected budget cut, missed deadline, or overlooked requirement can trigger a chain of events that can jeopardize an SOA program. The only way to ensure a rolling snowball stays its course is to follow it down, nudging it back on track and reshaping it when it begins to stray. Similarly, the only way to ensure SOA stays its course is to follow it every step of the way, be tuned in to the laws of corporate physics, and reshape the initiative as chaotic events begin to derail it.

SOA holds a lot of promise for companies striving to build agility; it is designed to harness chaos and use that energy to fuel innovation and rapid change. However, before the promise can become a reality companies must build suitable, change-ready SOA infrastructures, which takes patience and careful planning. The keys to successful SOA implementation will be thorough understanding of its fundamental principles, attention to how it influences, and is influenced by, corporate physics at each stage of its evolution, and a deep commitment to following through. Otherwise, the SOA program runs the risk of getting stuck or, worse, barreling out of control through the organization causing mass hysteria and flattening innocent co-workers.

Let the Chaos Begin

I believe most of us by now are in violent agreement that the shape of business is changing for the 21st century. This is an age of rampant unpredictability where stars are born (and fizzle out) overnight. No one knows when the next startup with a 20-year-old CEO will invent an entire industry or when the next stalwart 20th century empire will crumble. But now we all know that it is possible (if not probable), and that it can happen anytime, anywhere. Hence, we are all faced with a choice: to embrace and attempt to leverage the chaos or to ignore it and hope it blows over.

This blog is dedicated to those of us who have chosen to face the beast head-on, those of us who believe it is here for a reason and is here to stay, those of us who are compelled to study it with the goal of understanding it, taming it and, ultimately, harnessing its power to use to our advantage. We understand that its power is fierce and nondiscriminatory, and that this presents despair for the meek but opportunity for the bold. We understand that we brought this chaos upon ourselves through our own innovation and that its path will be influenced by our continued innovation. And as innovation requires change--breaks from the norm, divergent thinking, non-conforming attitudes, creativity--we are the change agents, and with the power of change at our fingertips we will move mountains.

In this blog I will explore the immutable forces of chaos acting upon every company and attempt to offer insights into how this chaos can be leveraged to defy the threats it imposes and capitalize on the opportunities it uncovers. Those individuals that understand these forces will be poised to become heroes, and companies poised to rocket to the top of their markets. Those that do not will watch in bewilderment as their careers and businesses (slowly or quickly, but assuredly) burn out and fall to the ground.

The playing field has been leveled and we all have been given uniforms but we must each decide for ourselves whether to play the game. I hope you'll suit up and stay with me as I explore this fascinating new world of unprecedented uncertainty...and opportunity.

Let the chaos begin...