What is experience worth?

imagePeter Norvig, Director of Research at Google, has written an interesting essay on learning to program in 10 years. In it, he points out the limits of what can actually be achieved within training programs and development opportunities. He uses the example of learning the syntax of a new programming languages, and contrasts this with how long it takes to actually become proficient in applying the language to design problems. Although his expectations of education may be low - consider his performance evaluation of Albert Einstein - his insights about the challenges of developing competencies for higher levels of performance are right on target.

Norvig brings up a critical question in his analysis: how important is experience in being able to perform certain tasks effectively? This question is crucial because its answer dictates how long it will take to become fully effective in programming and similar intellectual pursuits. This 'spin-up' time impacts many aspects of project and skill management, from how quickly new members can make meaningful contributions on a team, to how to plan and estimate reliably, to what the compensation and retention policies should be for an organization.

For years, professional educators have used Bloom's taxonomy of cognitive skills to evaluate the levels of knowledge necessary to perform different kinds of tasks. Obviously, the more complex the level of knowledge, and the more demanding it is to apply that knowledge, the more difficult it will be (in both a formal and the 'on the job sense') to get someone ready to perform that task. Once these levels of knowledge are determined, knowledge tables can be constructed to lay out a set of objectives for classroom and job experiences. Bloom's taxonomy establishes the following levels:

  • knowledge, the most basic area, which is the ability to recall memorized information
  • comprehension, which is the ability to recognize when to apply such knowledge to a given situation
  • application, when an individual can apply this comprehension to a particular situation effectively
  • analysis, when the individual is able to recognize patterns across many different situations and applications and discern & demonstrate their knowledge of these insights and broaden the body of knowledge for others to use
  • synthesis, in which these insights can be woven together into fundamental new approaches that add to institutional knowledge
  • evaluation, in which the individual can assess the application, analysis, and synthesis of knowledge by others, identify gaps in knowledge, skills, or behaviors, and compare, support, explain, or defend their evaluation.

As Norvig points out, the best an engineer exposed to a new technology will be able to do after a few weeks of exposure (such as a new tool or a new programming language) is to have memorized basic facts about it. While they may have been exposed to simple situations, it is unlikely that they will comprehend how to effectively apply that knowledge, or more importantly, when and why to apply it, in a more complex situation, until they have actually had the opportunity to use it repeatedly in different situations, and learn from that experience. They may well have awareness of the underlying concepts, and will be able to speak the words, but may not actually be capable of fulfilling the intent of what is needed in a particular situation.

As an example, consider cardiac surgeons, who typically require 4 years of college, 4 years of medical school, a 1-2 year residency program, and an additional 3-6 years of surgical internship until they can begin to perform the delicate task of cardiovascular surgery. Note that even after this extensive training, they are still not ready to perform all surgical tasks - for example, complex, demanding tasks such as open heart surgery will take many additional years of practice and mentorship. While a physiologist may have similar knowledge of how the cardiovascular system works, they will not have the skills to perform the delicate tasks required to perform cardiac surgery. Similarly, while a cardiac surgeon may be able to operate on a heart patient, they may not be sufficiently skilled or qualified in other medical branches, such as neurology, or orthopedics, such that they could perform surgery in those fields; some basic skills will be transferrable, and they may be able to recognize and diagnose certain classes of problems in those fields, but given a choice, a more experienced surgeon in the specialty area would much be preferred. Our cardiac surgeon may be quite proficient as a surgeon, but would still need other types of learning opportunities in order to work in research areas, or even to train other surgeons and ensure that they themselves achieve competency in performing surgery. As this example suggests, the competency an individual needs to demonstrate is highly dependent on the responsibilities one is expected to fulfill in a given occupation at a particular point in time.

This example highlights how Bloom's taxonomy is applied to a particular body of knowledge:

  • knowledge, in this example, about how the body and diseases work, and how treatments can be used, within a particular branch of medicine
  • comprehension, which involves being able to recognize the relevant disease processes at work in a particular patient, and understanding how that will progress over time
  • application, when the surgeon can apply this comprehension to a particular patient with symptoms - connecting a problem to a solution
  • analysis, when the surgeon is able to investigate and recognize patterns across many different patients, and select and optimize treatment plans for those patients
  • synthesis, in which a combination of treatments is integrated to produce an unprecedented new approach to treating a disease
  • evaluation, in which the surgeon works at a teaching hospital, and mentors others in all preceding levels

More recently, the Bloom's framework has been further refined in a new book that incorporates an additional dimension. In this revision, Bloom's original cognitive elements are further refined in another dimension, forming a matrix based upon four additional hierarchical elements. The base of this added dimension contains factual knowledge, which is knowledge of terms, details, symbols, etc. At the next level is conceptual knowledge, which includes classification, categorization, and structure. The hierarchy then advances to procedural knowledge, when one understands how to apply these facts and concepts over time. The last element of this hierarchy is called meta-cognitive knowledge, when self-awareness is achieved relative to others, and alternative strategies can be evaluated.

So what does this mean to what an older worker's value is relative to younger workers? Obviously, it depends upon the situation. First, it is important to recognize that there is a wide range of differences in attitudes and value systems across generational groups (see Geeks and Geezers). But despite these differences, if the facts and concepts are simple, and procedures to apply this information are straightforward for a particular task, it may only take a few days or weeks to learn, comprehend, and apply that knowledge. However, if the factual, conceptual, and procedural details are complex, and the time to learn to apply them is long (many months or even years), the importance of retaining experience is worth a lot - especially when that experience can be transferred to other adjacent areas. For example, while you can pick up the syntax of a new programming language in a few days or weeks, you won't necessarily be proficient in applying that syntax, and thus may not be particularly effective in mentally reading someone else's code, and recognizing errors in that code. And even if you have proficiency in one domain area (say, building software for a web-based application), it may not be relevant to another domain, such as building a real-time system.

It is important to also recognize that ten years of experience can either be a single year's experience repeated ten times over, or can instead be a broad exposure to many different situations over that ten year period. When it is really the latter, the patterns that arise in diverse situations can be comprehended, analyzed, and internalized. This is the true nature of what experience is worth. Can people see holes in plans, and know how to fix them? Do they understand why particular aspects are important, or are they just blindly applying slogans without insights? Can they troubleshoot difficult problems in situations that they are unfamiliar with, and develop a hypothesis and a logical plan to evaluate that hypothesis? Do they have the insight required to create new approaches, while still evaluating and making good decisions about the risks involved? Are they only able to tackle problems of limited complexity, or can they break even complex problems down into a robust decision tree? These are the kind of situations that experience is important in, and it never seems like there are enough people with these experiences to go around. Yet the track records of most companies in recruiting, evaluating, and retaining such talent is about as good as that of professional football teams in finding quarterbacks - not good enough!

Of course, like with all other aspects of talent, the Pareto principle also seems to apply. As a result, the majority of useful work is often being done by a relatively small group of people in the organization. As a result, an important decision for any organization to make is whether to design your people strategies and education programs around basic knowledge (i.e. to utilize average talent), or instead to build a strategy to utilize the full measure of all people's contributions (regardless of level), and grow and leverage everyone's contributions with time. Such a long-term talent development strategy is only possible once it's underlying dimensions, as described in the above cognitive framework, are acknowledged and aligned with the organization's business practices.