Systems theory is the study of systems in order to discover successful patterns which can be applied more broadly when presented with similar situations in the future. In their reflections on the IBM/360 development effort (A model of large software development), Belady and Lehman describe a classic deconstruction of such a system:
A system, a process, or a phenomenon may be viewed from the outside, by acts of observing; clarifying; and by measuring and modeling identifiable attributes, patterns, and trends. The scientific method has made progress in revealing the nature of the physical world by pursuing courses other than studying individual phenomena in exquisite detail. Similarly, a system, a process, or a phenomenon may be viewed from the outside, by acts of observing; clarifying; and by measuring and modeling identifiable attributes, patterns, and trends. From such activities one obtains increasing knowledge and understanding, based on the behavior of both the system and its subsystems, the process, and its sub-processes.
When used in this context, behavioral patterns are recurring structural configurations that have been shown to address problems which arise within particular contexts, contribute to the coherence of a system of interest, and provide a set of capabilities that contribute to some purpose. These patterns are usually discovered as a result of reflecting on historical performance and recognizing that one class of behaviors has been more beneficial than others in particular situations. Most often, these situations require attention to qualities of evolutionary growth, incremental repair, or application of experience-based learning. This means that patterns are tightly bound to architectures as deployed in operational usage, especially where they form rules which structure team activities. But combinations of patterns must collective fitness to the situation in order for desired behaviors to emerge.
These patterns have often been expressed in a pattern language, which provide the semantics for applying particular patterns. These patterns thus are triggered by the recognition of particular situations, since patterns depend on context; as we apply patterns, the context may change. As a result, the actual path traversed depends on both circumstances, history, and the choices made in shaping organizational behaviors in the past and along the current path. Such patterns can thus provide a structured (but flexible) means to shape our responses to system complexity, recurring risks, and chronic impediments as they arise. They help us realize that some decisions produce better outcomes than others in different situations. Patterns thus form reusable collections of behaviors that can enhance the ability of a team or organization to align itself with its environment. Of course, like other things which are labeled as best practices which do not have a body of evidence backing them up, embracing the right patterns themselves can be quite subjective, and thus leave us exposed to self attribution errors.
Many of the ideas behind patterns in design and development originated in the book A Pattern Language: Towns, Buildings, Construction. In it, Alexander believes that order in any system fundamentally depends on the process used to build the system. which is why the fundamental process is important. Each step in the process preserves structure, focuses the effort, and gradually reinforces local responses to situations that unfold over time. The process thus involves step-by-step adaptation with feedback. Simply following the pattern language doesn't give you a clue about how to handle the feedback. That's why an over-arching fundamental process must exist: to grant complete freedom to the design process so it can attack the weakest part of the system, wherever it may be.
In The Benefit of Patterns, Linda Rising contrasts the use of patterns with the process specifications that often are the product of dedicated process development groups:
Most process-intensive organizations look to a process specification document as the final word on development activities. We noted three problems with this approach in practice: lack of empirical conformance between practice and process specifications, incompleteness of process models, and inability to capture long-term stable process abstractions. Many processes exhibited such broad variation in behavior that it was difficult for process specifiers to agree on a process that represented the typical scenario. Many organizations informally built process specifications from anecdotal process experience instead of developing the baseline process model with empirical models and data. Many organizations we studied created an ideal specification instead of capturing empirical practices, and organizations used these specifications as a baseline for improvement despite the mismatch. Because many process specification models were divorced from empirical practice, they could not reliably drive real development practice.
Second, process models were often incomplete and inconsistent. Most process models focused on the task and event perspective, leaving artifacts, roles, actors, and agents as secondary abstractions. Much of this task perspective was driven by a preoccupation with interval prediction and reduction on one hand (management of the overall interval by focusing on individual intervals) and quality on the other (methodical reviews form obvious task/event benchmarks). Task models fit well with the waterfall-based development model that predominates in most development cultures. Furthermore, task models held up the promise of process automation. We note that the disconnect between process tasks and the artifacts they produce continues to plague most of the organizations we work with today.
Third, many organizations built their process-improvement programs around the task or event dimension of process. Well-understood processes (e.g., bug report flow) often can be regularized, but the core processes of software architecture, design, implementation, and validation are poorly understood from a task perspective. We have found that task ordering changes rapidly in a high-technology development organization, so it can't be counted on as a stable component of process structure. One large organization we studied surveyed its developers and found that 80 percent of them were working under officially granted process waivers instead of the official common process, largely because the project's process standard didn't capture the essential stable structure of the process. One reason that task chain models don't capture the stable structure is because of the high degree of concurrency present in modern software development. It is interesting to note that iterative and incremental design cultures were demonstrating success at about the same time that process consciousness was growing. Project managers still find it difficult to reconcile iterative and incremental techniques with process standards that prescribe process steps. Many organizations we studied exhibited concurrent engineering practices, where requirements, design, and implementation activities proceeded in parallel. Few organizations intentionally applied concurrent engineering
Much as a tailor uses different patterns to design and assemble raw fabric into different garments, a pattern within a culture provides them with a way to recognize and solve a class of problems by bringing conflicting tensions into balance. In his book, Organizational Patterns for Development, Jim Coplien describes how the structure of these patterns is shaped by natural forces:
Human endeavors share common elements of human nature, which to some degree limits the range of human undertakings. For example, most organizations have leaders, and cliques, as well as their own physical space and organizational structure, and hundreds more. The organization of any major human endeavor follows basic laws of efficiency of communication, span of control, xenophobia, specialization, and other sociological forces that drive similar undertakings to implement similar practices and structures.
Patterns thus provide a language for cultures to communicate idealized behaviors and reduce them into simplified constructs suitable for segmentation, serialization, and orientation. This language can only achieve its potential when roles that actors must assume learn to subsume the patterns which are relevant to their responsibilities, and use them as shortcuts with others across the value chain. Since patterns can either be grooves or ruts, they must also become the basis of status reporting and decision-making, where they can serve as a rich, semantic communications token for a complex set of behaviors. But until they are embraced, practiced, shaped, and reinforced, they can be just one more fluffy abstraction of behavior that is disconnected from the 'real work' of the organization. As a result, endeavors seeking to exploit patterns should be grounded in process management, since as observed by Senge, such foundations are necessary to provide a consistent framework for evaluating performance. The need for this consistency is the basis for creating standards for performance, evaluating applications of those standards, and isolating unwarranted variation.
A repository of organizational patterns can be found here.