There are a number of factors that distinguish successful measurement programs from those that are not so successful. This article highlights those critical success factors that organizations have found to be instrumental in the development and implementation of an effective measurement program. It also demonstrates some of the lessons learned by organizations encountering obstacles in this endeavor. Future articles will further articulate these factors.
Define Measurement Objectives and Plan
All too often organizations attempt to establish a measurement program because it is the "popular" thing to do. Before establishing a measurement program, it is essential to define the objective you are attempting to accomplish. This objective should outline the problem or need you are trying to address and what the "ideal state" should be when the program is implemented.
The smooth establishment of a measurement program also requires a strategy and plan. The strategy should address many of the critical success factors described in this article. The resulting plan should be flexible and evolutionary. Measurement should grow with an organization and not overwhelm the culture.
Obtain Senior Management Sponsorship
Senior management sponsorship should be active and genuine. It does not simply mean that they are willing to let you establish a measurement program. It means they will actively sponsor it and participate in making it happen. It is important they convey the message that they intend to utilize the results of the measurement program to manage the software organization.
Dedicate Resources
A core team of qualified support staff is required to facilitate and/or direct the development of the measurement program. They should provide expert guidance and direction to the staff in the development and use of the quality and productivity metrics as well as the tools supporting them. The size of this staff is dependent on the size of the development staff they have to support and the scope of measures to be developed and maintained.
Train, Educate and Market
Marketing measurement to staff and management is critical to the program's success. Often this marketing is in the form of training and education. Proper training and education enables the staff to understand and use the results of the measurement program. This allows it to become a means by which managers and staff manage their departments, applications and projects rather than just a "report card."
Integrate Measurement
Measurement, to be accepted and actively supported, should be viewed as an integral part of everyone's job. It must become an accepted part of the organizational culture. It should be clearly understood and accepted that everyone is expected to take an active role.
In order to gain acceptance, measurement activities need to be integrated with the development lifecycle. It then becomes no different than other scheduled activities such as designs, inspections and testing. When not fully integrated and scheduled, it becomes an optional activity that is rarely performed.
Focus on project team results
A common denominator across the software organization is the project. Focusing on the effectiveness with which you complete projects and maintain applications enables you to do comparative analysis at the project level. This allows you to more readily identify and address influencing factors that are of manageable magnitude, such as process issues.
Focusing on the project level enhances the project managers' ability to effectively manage projects. Aspects of project management that are enhanced include: estimating, project scheduling, project planning, change of scope management, risk assessment and post project reviews. Focusing on the project level also provides benefits back to the staff who often collect the data.
Do Not Measure Individuals
Projects are team rather than individual efforts. Measuring the performance of individuals tends to focus in on the individual as the "cause" of high or low productivity or quality. In reality, individuals have little control over the primary factors that influence productivity and quality. Factors such as the development environment, tools, methodologies and project type have a much stronger influence on project performance than an individual's performance.
Additionally, measuring individual staff performance can actually be counterproductive. If individuals are rewarded or punished based on measurement, their daily activities and attitudes are influenced. They often focus on individual output which can detract from the team- work which is required for success. This could also discourage open and accurate reporting of data.
Define and Report
Measurement programs often fail due to poor definition of meaningful metrics and lack of good analysis and reporting. Many organizations have found themselves collecting data for years without a clear understanding of how to analyze, report and use the resulting information. Measures need to be defined by numerous audiences within the organization. These audiences should determine their unique goals and objectives and define measures which will help achieve them. This both ensures meaningful metrics and widespread involvement.
Automate
A comprehensive measurement system will result in the need to capture, analyze and publish large amounts of data and information. An organization can become inundated with data that must be entered and processed. A key to the success of a measurement program is the ability to expedite the transformation of data into useful measurement information. This often calls for the automation of many facets of the measurement process.
This automation of the process should be approached cautiously. Organizations often purchase expensive measurement software before understanding their requirements. Consider the use of homegrown spread sheets and data bases as a temporary solution until the longer term needs are determined. When you are ready to consider more sophisticated solutions, you might want to start with PQMPlusT from QPMG. This is a package that can assist in data collection (Function Point Analysis), serve as a project and user history repository, and apply benchmark data in generating useful estimates throughout the project lifecycle.
In Summary
Numerous organizations have embarked on the establishment of measurement programs. They have met with varying degrees of success. Those that have enjoyed the greatest success have focused on many of the factors outlined in this article. Although they have experienced obstacles along the way, they have been able to overcome them and move forward. Also, be patient. Developing an effective measurement program requires time and commitment. This effort however, can yield significant benefits in terms of improved productivity and quality.
About the author
Scott Goldfarb, President Q/P Management Group, Inc., wrote this article. © Copyright 2000-2002. Q/P Management Group, Inc. Proprietary and Confidential. All Rights Reserved.
A Primer on Metrics
by gary t. smith
March 6, 2004
Metrics are fast becoming the essence of how organizations measure and manage performance against objectives. However, organizations tread a dangerous path if they do not understand fully what metrics are-and what they really tell.
Metrics are fast becoming the means by which businesses measure and manage performance. What are they? What do different kinds of metrics tell you? Here's a practical guide.
Performance management is the focus of tremendous attention today. Whether it's called business, corporate, or enterprise performance management, metrics and their use in dashboards and other reporting formats form a key part of how organizations are to convey concise and timely information. Unfortunately, amid the buzz, very few words have been written focused on what metrics are, how they should be defined, and how you can determine their relevance and validity. You might find such discussions in academic writing, but not in the trade press.
In this article and the next installments, my goal is to focus on metrics themselves, beginning with a basic discussion of what they are. With accepted definitions, organizations can make true progress with performance management.
Metrics Defined
Metrics are measurements. For this discussion, they are measurements of process elements (inputs, activities, and outputs) in relation to an explicit benchmark — a specified level of performance. Therefore, metrics are process-based. These processes can be in research, manufacturing, finance, operations, marketing, services, or any other process-based activity. No doubt, the more discretely you've defined a process, the more accurate your results will be. Generally, however, you can define metrics for all but the most amorphous of processes.
Metrics by themselves merely state process parameters. In other words, when used to measure performance, users view them relative to a stated policy or benchmark — something I will discuss in more detail in the second part of this article. Ideally, organizations establish benchmarks using statistical methods. In a few cases (for example, the introduction of a totally new process) the management team, perhaps in concert with consultants and subject-matter experts, will arbitrarily establish the initial benchmark. Once established in this manner, however, actual measurements and statistics will refine the benchmark over time.
Metrics must be selected to give the greatest amount of information while using the least amount of resources required to report them in the most effective manner .
Processes Defined
What is a process, exactly? Merriam-Webster defines a process as: "a series of actions or operations conducing to an end." (See Merriam-Webster Dictionary Online at www.m-w.com.) For our purposes, we will also include defined input and output attributes for the process in the definition.
Processes have interface points and workflow. Where processes exchange inputs/outputs — that is, where touchpoints exist — there will be an interface. Some methodologies refer to this interface as material flow, though the connotation that the interface involves something concrete does not always apply. In relationships from customer to company, regulatory agency to company, company to markets, and company to vendors you will see examples of external process interfaces. Other examples of relationships involving internal process interfaces include accounting to finance, operations to accounting, shop floor control to master production scheduling, and manufacturing to distribution. With a little thought, it becomes obvious that myriad processes and interfaces are relevant to any enterprise.
Within and between processes, there is workflow. Workflow is a series of discrete tasks connected by dependencies; workflow consumes resources. Each task consists of operations performed by resources upon components to create a discrete product.
It is possible to measure each operation, task input, task output, dependency chain, interface point, and resource. These measures are the base components for metrics that will provide business intelligence regarding enterprise performance. However, this is not to say that every identifiable operation, task input, task output, dependency chain, interface point, or resource should be measured and monitored. The effort required to collect, compile, and report on metrics must be cost effective. Realistically, you can't report on every measure.
Types of Measures
There can be as many measures as there are things to measure. It may be stating the obvious that a measure must be quantifiable, but you might be surprised at how many subjective "measures" you will find in a given environment.
You can describe quantifiable measures using discrete unit descriptors. For example, a measure of distance could be "inches" or "feet." A temperature measure could be degrees Celsius or Fahrenheit. It is the unit of measure that defines a measure, not what is being measured or the quantity of the measure. For example, "inches" is a unit of measure: "the distance traveled by an object," or the quantity of units traveled by an object in a given distance, are not typically units of measure.
If used only once, measures are of very little use. However, if they are part of a collection taken from a statistical sample population, measures can be extremely useful. If I say that the measure of the current temperature is 60 degrees, I'm not giving you very much information. You might ask where the temperature reading was taken, if the temperature is taken indoors or out, if it were taken in July or January, and whether this is the " normal " temperature. (In this context, normal implies some type of time-series analysis and comparison to a baseline or benchmark, and any variation from that baseline.) However, if I qualified my statement by saying that it was taken on the beach on the Mediterranean coast of Spain in June, you would know that this measure is unusual and warrants further investigation.
Measures can also have a type. Examples of measure types are illustrated in Table 1. Days of sales outstanding (DSO) is a good example of a binary measure commonly found in manufacturing, usually accompanied by a set of limits, such as 30, 60, or 90 days. The statement "DSO is between 10 and 30" requires a determination of either yes or no, exclusively, even though it is stated as a range of values. DSO is either within the range, or it isn't.
Table 1 Measure types.
Additive metrics are characterized by arithmetic counts, differences, intervals, or products. These metrics are best suited for discrete integer-value data. Some examples would include the number of (whole) units produced per hour/shift/day; sales volume (dollars, units) for a given period; and activity-based costs. (The subject of Activity-Based Costing is beyond the scope of this article.)
Simple ratios (proportions) and averages determine the mean value of a population, or the ratio of one measure to another measure. Some examples would include:
- Average age of people who drive Corvette automobiles (for example, (oldest age) — (youngest age)) / 2
- Average sales for the month of August (for example, sum of sales for every August) / (number of August months for which there is data)
- Proportion of August sales for 2002 to all sales for 2002 (for example, August sales for 2002) / (All sales for 2002)
- Productivity; for example, resource utilization / number of units produced; or number of units produced per period) — (scrap + rework)
Descriptive statistics, which include methods to describe data, are often used with the previous types of measures. Measures and measure types are the tools and raw material for constructing metrics and process indicators.
Metric Categories
You can categorize metrics into simple or compound types. By "simple," I mean singular and direct. These metrics stand alone; they are not combined with other metrics and are for measuring individual attributes of the entity of interest. An example would be the time required for a particular loan processor to process a bank loan. The entity (processing a bank loan) is measured individually (without regard to other loan processors) for the attribute "time." When averaged over time, you could see the performance 
standard for that loan processor, and his or her variance from that standard. The metric is direct: that is, not derived from other metrics nor evaluated relative to the behavior of other loan processors.
Compound metrics are more complicated. These can be derived (that is, indirect), composite, and/or layered (hierarchical). Compound metrics are not mutually exclusive; overlap is possible. However, overlap is generally discouraged; instances of overlapping metrics need to be uncovered and the offending metrics combined or eliminated as part of a metrics maintenance process. Different and overlapping metric types should be converted to a common unit of measure (such as dollars, units, occurrences, or time granularity), with an appropriate conversion factor (dollars/hour, units/hour, dollars/unit, or units/dollar).
Compound metrics are, of course, the most complex type. They must be constructed carefully and understood clearly to maintain their validity and relevance. A poorly constructed metric of this type can do irrepairable damage to the business — not to mention the credibility of the business analyst(s) responsible for the metrics portfolio effort.
Compound metrics can be further categorized:
Weighted and composite averages. Although these averages are easy to use (within certain rules), I categorize them as complex because they are compound. Weighted Averages have myriad uses and are developed easily. However, it is also easy to devise weighted averages that, while appearing to be valid when initially developed, produce invalid results. If the underlying assumptions upon which weight assignments are based are incorrect, then they can't produce correct results.
Composite averages represent the mean of a group of averages. Composite averages can also be assigned weights. Where this is done, you can distribute the weight of the composite average back down to the composite average's components as percentages of the total of the component average values. In this way, you can determine each component's weight contribution and its sensitivity impact. This method will work at any level of hierarchy, although in short order it can become extremely complex.
Statistical analysis. Beyond the area of descriptive statistics lies the rich field of inferential statistics. Included here are regression and forecasting, correlation and variance, and other analytical tools. The breadth and depth of this topic lies beyond this article, but the importance of these tools to the application of Compound Metrics can't be overstated. In fact, for applications such as Six Sigma, these tools are essential.
Layered metrics. An associated class of composite metrics is layered (or "consolidation") metrics. These are characterized by a hierarchical relationship among the metric components. For a more detailed discussion, I would refer readers to the available literature on the Analytical Hierarchical Process (AHP) as an application of hierarchical metrics. The AHP illustrates how subordinate metrics (or sets of metrics) can influence superordinate metrics.
Thresholds and triggers. With these metrics, you can initiate some action based upon an out-of-bounds value or group of values. Perhaps the best example is statistical process control (SPC), most commonly found in manufacturing, where high volume, long-running processes produce large numbers of identical units.
In SPC, measurements are taken periodically (such as hourly), or quantitatively (for example, every 100 units), and plotted on a series chart. This chart would define upper control limits (UCL), lower control limits (LCL), and the target (baseline) value. When a measurement exceeds the UCL or LCL, the system can initiate an exception activity to correct the problem that generated the exception. Additionally, a variety of series types might indicate a marked trend in either direction (for example, four increasingly positive, or increasingly negative, measurements) that can trigger a corrective action, even though the UCL or LCL measures are not yet exceeded.
Stay tuned for Part II: In the next installment, I will focus on the relevance and validity of metrics. This knowledge will help in creating an overall metrics strategy and in choosing the best metrics for your performance management goals.
About the author
Gary T. Smith [gtsmith_x@hotmail.com] is a consultant with 25 years of experience in all areas if IT, including positions as IT directorship, project management, and consultant to support global enterprises working with BI, data warehousing, and Oracle database management.
