Agile Innovation: A Secret History
23 April, 2018 / Articles
Some trace agile methodologies all the way back to Francis Bacon’s articulation of the scientific method in 1620. A more reasonable starting point might be the 1930s, when the physicist and statistician Walter Shewhart of Bell Labs began applying Plan-Do-Study-Act (PDSA) cycles to the improvement of products and processes. Shewhart taught this iterative and incremental-development methodology to his mentee, W. Edwards Deming, who used it extensively in Japan in the years following World War II. Toyota hired Deming to train hundreds of the company’s managers, eventually capitalizing on his expertise to develop the famous Toyota Production System — the primary source of today’s “lean” thinking. Iterative and incremental development methods were also a major contributor to the successful creation of the X-15 hypersonic jet in the 1950s.
In 1986, one of us (Takeuchi) and coauthor Ikujiro Nonaka published an article in Harvard Business Review called “The New New Product Development Game.” Studying manufacturers that were releasing successful innovations far faster than competitors, the authors identified a team-oriented approach that changed the design and development process for products such as copiers at Fuji-Xerox, automobile engines at Honda, and cameras at Canon. Rather than following conventional “relay race” methods of product development — in which one group of functional specialists hands off its completed phase to the next functional stage — these companies were using what Takeuchi and Nonaka called a “rugby” approach, “where a team tries to go the whole distance as a unit, passing the ball back and forth.”
In 1993, another of us (Sutherland) faced what seemed like an impossible task: Easel Corporation, a software company, needed to develop a new product to replace its legacy offerings in less than six months. Sutherland already had a strong background in methodologies such as rapid application development, object-oriented design, PDSA cycles, and skunkworks. He hoped to create a skunkworks-like culture in the middle of corporate headquarters, blending the benefits of both organizational separation and integration. So he began by learning everything he could about maximizing organizational productivity. Reading hundreds of papers and interviewing leading product-management experts, he found himself intrigued by several provocative ideas.
One came from a Bell Labs article on the Borland Quattro Pro team, suggesting that short daily team meetings increased group productivity dramatically. But the capstone concept for Sutherland was the discovery of Takeuchi’s and Nonaka’s rugby approach, even though it focused on manufacturing rather than software. Borrowing many of the HBR article’s key ideas and filling in specific operational practices, Sutherland created a new way of developing software; honoring the rugby imagery, he dubbed his approach “scrum.” Scrum methods enabled him to finish his seemingly impossible project on time, under budget, and with fewer bugs than any previous release. He then collaborated with longtime colleague Ken Schwaber to codify the approach, and in 1995 the pair presented scrum to the public for the first time.
Of course, Sutherland and Schwaber weren’t alone in their search for innovative methods. The Information Age was exploding. Disruptive technologies were terrorizing slow-footed competitors. Start-ups and incumbents alike sought better ways to adapt to the unfamiliar and turbulent environment. Software was becoming an integral part of nearly every business function, and many creative software developers were working hard on better methods of programming to increase adaptability.
In 2001, 17 developers who called themselves “organizational anarchists” met in Snowbird, Utah, to share their ideas. Sutherland and other proponents of scrum were among them. But the group included advocates of several competitive approaches, including extreme programming (XP); crystal; adaptive software development (ASD); feature-driven development (FDD); and the dynamic-systems-development method (DSDM). All these approaches were often known as “lightweight” frameworks because they used fewer, simpler rules to allow faster adaptation to rapidly changing environments. Not many of the attendees found the “lightweight” terminology flattering.
Although they disagreed on much, the group eventually settled on a new name for the movement: agile. The word was suggested by one attendee who had been reading the book Agile Competitors and Virtual Organizations: Strategies for Enriching the Customer. The book gave 100 examples of companies — including ABB, Federal Express, Boeing, Bose, and Harley-Davidson — that were creating new ways of adapting to more turbulent markets. Name in hand, attendees then forged consensus on a call to arms dubbed the “Manifesto for Agile Software Development,” which spelled out four key values that everyone agreed on. Later in the meeting, and continuing over the next few months, they developed 12 operating principles, called “Principles Behind the Agile Manifesto.” From 2001 on, all development frameworks that aligned with these values and principles would be known as agile techniques.
Once the Snowbird meeting had canonized a creed for agile innovation, the agile movement spread rapidly. The signatories posted their document online and invited others to add their names as supporters. Most members of the original group, joined by a number of new adherents, reconvened later in the year to discuss ways to disseminate agile principles. All agreed to write and speak on the topic. Several attendees wanted to form a more permanent working group; so they established a nonprofit organization called the Agile Alliance to support the movement. Today, the Agile Alliance has nearly 30,000 members and subscribers.
Meanwhile, agile methodologies continued to evolve. In the late 1980s and early 1990s, researchers from MIT had begun to study Japanese manufacturing systems, especially the Toyota production system. They coined the term “lean” to describe the system’s methods of improving productivity by eliminating waste (“muda”) through reductions in uneven work flows (“mura”) and destructive overburdening (“muri”). Although lean methodologies were not presented as agile frameworks in Snowbird, formal lean and kanban software-development systems emerged during the 2000s. At first, some agile purists refused to recognize these approaches as agile methodologies. But lean advocates intensified their focus on customer collaboration, and eventually more agile practitioners came to accept lean, kanban, and their hybrids (such as scrumban and lean scrum) as legitimate applications of agile values and principles.
Success has many fathers, and agile innovation has a colorful heritage. While agile’s complex family tree sometimes provokes passionate debates among agile practitioners, two things are clear: first, agile’s roots extend far beyond information technology and, second, agile’s branches will continue to spread to improve innovation processes in nearly every function of every industry.
The scientist and innovator, Fernando Fischmann, founder of Crystal Lagoons, recommends this article.
