June, 2007
by Peter Baxter
About the Author
Peter C. Baxter
Peter Baxter is the founder and President of Distributive Management, where he directs the design and delivery of performance management and measurement products and services. He is actively involved in the development of measurement standards and guides in leading professional organizations, including IEEE, INCOSE, PSM, ISO and the SEI. Peter is a frequent author, trainer and presenter on the subject of measurement and metrics at events such as STC, Better Software, INCOSE, StarEast, TUGC and other symposium and professional meetings related to management and measurement.
(540) 891-8811
In the first installment of this paper readers learned about the purpose, function and operation of an effective measurement process within the context of the SEI's Capability Maturity Model Integration, or CMMISM. Organizations implement CMMI because it produces tangible results, as evidenced by these examples drawn from an SEI report, Demonstrating the Impact and Benefits of CMMI:
Following on from part one of the paper, which ended by describing the capabilities that a capable measurement process should provide, this part provides recommendations for selecting a performance measurement tool. This guidance includes characteristics, features and capabilities your organization will want to seek out when looking for software to automate its CMMI Measurement and Analysis process.
A preferred CMMI tool enables an organization to implement a measurement process by identifying and defining information needs and makes possible the delivery of that information to knowledge workers - senior executives to technical managers to developers and staff - in a way that supports decision-making and improves overall program management.
A measurement tool should assist organizations of any size to implement a measurement program that manages systems, software and IT development and also supports process evaluations such as CMMI. The tool should be a scalable, flexible and easy-to-use solution that helps the organization:
Specifically, a measurement tool should provide solutions to the primary challenges encountered when satisfying the Measurement and Analysis (M&A) requirements of the CMMI by:
The remaining subsections describe how a software solution can simplify and speed a measurement process implementation.
2.1 Supports All Activities in the CMMI M&A Process Area
There are eight activities and four primary work products identified in the CMMI M&A process area. Table 1 below, describes briefly how a good measurement tool provides each required element.
|
Element
|
Type
|
Tool Capability
|
|
Establish Measurement
|
activity
|
Provides a framework for defining organizational measures and serves as central location for defining the structure of your measurement plan. Pre-built information needs can serve as ready-to-deploy templates for your organization.
|
|
Specify measures
|
activity
|
Captures the measures needed by managers as information needs, which include graphs, series, analysis, status, formatting, etc. Should include metrics commonly used by CMMI-rated organizations.
|
|
Specify data collection
|
activity
|
Integrated tools to connect to external data without writing scripts or integration code.
|
|
Specify analysis procedures
|
activity
|
Provides tools for easily building and automating analysis within your measures – color-coded status indicators, alarms, statistical analysis, aggregation, and more. Pre-defined measures come with standard analysis built in. Analysis is performed automatically each time measurement data is updated.
|
|
Collect data
|
activity
|
Allows managers to get data updated, automatically, using the schedule required to meet their management responsibilities. Allows manual collection anytime, or batch processing to be scheduled. Automatically determines sources and data to collect, then updates all associated reports.
|
|
Analyze data
|
activity
|
Summarizes status and progress into effective information displays that help you quickly decipher what’s going on, using clear and concise visuals that are easy to interpret. Highlighted troubled areas and direct access to supporting details save time in determining courses of action.
|
|
Store data and results
|
activity
|
Uses proven technology to create a repository for your measurement process. This repository serves as an ongoing storage place for all your key organizational measurement data – the historical data that improves planning and estimating, that provides proof of compliance, and that contributes to lessons learned for process and performance improvements.
|
|
Communicate results
|
activity
|
Communicates measurement results instantly via the web to all levels of the organization that will use the information to make sound decisions. Status views are straightforward and available when they are needed, providing not only quick-look summary status, but also supporting analysis and details and management notes. Easy, secure and flexible sharing of information
|
|
Measurement plan
|
work product
|
Provides automation for all aspects of the measurement process, essentially yielding an actionable measurement plan. Combines the specification of measures and procedures with integration and then graphing.
|
|
Measurement indicators
|
work product
|
Provides the tools needed to define, apply, and maintain the measurement indicators used across the organization. Also provides a method for combining sets of indicators into templates so that they can be applied consistently and quickly.
|
|
Measurement repository
|
work product
|
The database captures all the information needed for historic analysis and organizational use, as well as to provide objective evidence of use.
|
|
Procedures and tools
|
work product
|
All procedures and tools needed to collect data are configured inside the tool. Provides a consistent interface into disparate tools then automates the procedures for validating and gathering data.
|
2.2 Provide a Set of Pre-Built Best Practices
By using best practices gathered from leading software organizations, managers can deploy proven measurements/metrics in their project. Extensive automation is desirable in a measurement tool as it allows managers to decide for themselves which techniques to use, without needing process improvement experts or consultants.
The information needs should be pre-assigned to CMMI level 2 process areas. This assignment provides a starting point for establishing your organization's management and measurement processes. The assignment of information needs to process areas is contained in Appendix B.
2.3 Allow Tailoring for Organization Specific Management Practices
As an organization matures, a measurement tool should capture best practices and share them across an organization quickly and effectively. Management techniques that save time, money or resources become competitive advantages. Methods for reducing requirements creep or raising product quality can be captured so that the organization can address other critical strategic needs, and not re-learn the lessons of past performance.
2.4 Eliminate Time Consuming Documentation and Integration Tasks
A typical measurement process starts with the measurement process described in a management or measurement policy document. Once the policy is developed, integration scripts and reports are created based on the policy. The problem with this approach is that the policy can easily become disconnected from the integration scripts and the reports. Another problem with this approach is that updating the measurement process is time-consuming, as all three elements - the policy document, integration scripts and reporting - must be updated manually.
A desirable measurement tool will eliminate this disconnect by automating the approach so that the policy document is a by-product of defining and selecting metrics within the software application, and integration and reporting are automatically configured based on the metrics needs. The measurement policy is always up-to-date, always accurate.
2.5 Present Project Status Using Management by Exception
A measurement tool should present managers with key color-coded status data, allowing them to quickly spot trouble areas in the projects and programs for which they are responsible. The measurement process is typically used for multiple projects or programs simultaneously, delivering a set of graphs to each project manager on a periodic basis. For example, your organization may have 50 projects being measured, each with 5 information needs and 25 graphs, for a total of 1250 graphs. To aid in finding and spotting problems quickly, the measurement tool should provide a number of management-by-exception features that allow managers to efficiently find trouble areas.
2.6 Support for High Maturity Organizations
High maturity organizations, those at Levels 3 or higher in the CMMI, rely on more advanced measurement process capabilities, as compared to organizations at Levels 1 and 2. As an organization becomes more mature, their managers rely on a measurement process to provide information at a greater breadth (e.g. more process areas and work products) as well as greater depth (e.g. more detailed analysis of resources). A measurement tool should support the unique and more demanding needs of these organizations.
2.7 Data Repository
A measurement tool should capture and store all data required for generating metrics and management reports. It should provide a data mart which captures the raw data needed to construct measurement values, as well as the historical plans and progress data contained in reports.
This data repository should be used to establish organizational capabilities by examining organizational performance in completed projects. For example, by analyzing the historical data for requirements growth in completed projects, an estimator or project manager would have a very good indication of future performance.
The data repository can also be used to compare the current performance with previous performance. One example is a measurement report which shows current progress along with a three month trend. Yet another example is a measurement report which compares current progress to the same period in a previous year (or quarter).
2.8 Quickly Initiate a New Software Development Project
Software project startup can cripple an organization embarking on process improvement. In the first weeks or months of a new project, managers must get authorizations, establish measurement plans and policy, and setup a myriad of new tools and environments. When time runs short, the measurement process is often left undone, leaving managers to deliver critical technology without the tools they need to monitor and control their project to success.
A measurement tool should save managers valuable time during project startup by automating the time-consuming and repetitive tasks of establishing an effective metrics capability. Managers should be able to select the desired metrics from a list of best practices, and see those techniques in use within minutes.
2.9 Track Compliance with Standards, Models and Policy Documents
A measurement tool should address compliance by integrating the measurement process with compliance tracking. Managers should be able to stay informed about the compliance of their projects by reviewing automatically generated alerts or periodically checking a set of compliance indicators available within each project.
This section presents resources for the measurement process, as well as process improvement.
|
Measurement Guidance
|
|
|
Software Engineering Institute
|
|
|
Distributive Management
|
|
|
Practical Software Measurement
|
|
|
Crosstalk Magazine from STSC
|
|
|
Scott Ambler’s Web Site
|
|
|
Karl Weiger’s Web Site
|
|
A measurement process provides valuable guidance for improving your organization's capability to plan, monitor and control programs and projects. Software managers use the measurement and analysis process to enter and establish plans, to review the progress of actual performance against the plan, and finally to take management action to identify and resolve issues of poor process or product performance. The measurement process is the mechanism that the organization uses to provide information that managers need to manage the way that the organization requires them to. The CMMI contains a Measurement and Analysis process area that outlines key elements of a successful measurement process.
A desirable measurement tool should provide an off-the-shelf measurement solution that directly addresses the needs of the CMMI Measurement and Analysis process area. It should support the CMMI in these essential areas:
For more information, please contact the author, Peter Baxter, at Distributive Management through the following:
Distributive Management
800-779-6306
http://www.distributive.com
109 Olde Greenwich Drive, Suite 102
Fredericksburg, Virginia 22401
This appendix describes each of the tasks recommended by the CMMI for a measurement process.
Establish Measurement: The organization determines the resources, people, facilities and techniques to be allocated to the measurement process. Organizational management oversight is identified, either a single executive or a team who is responsible for deploying, supervising and verifying measurement process implementation. The organization establishes the criteria for which business areas and business functions must use measurement as part of their management functions. This criteria includes identifying specific programs, projects and other business area, generally called "management units". The resources allocated to measurement and the requirements for using measurement across the organization are documented in a measurement plan.
Specify Measures: In this task, the specific measures to be used for each management unit (identified in the previous task) are identified. Measures are documented using information needs, where each information need contains one or more measures. The complete set of information needs is captured in the measurement plan, which contains high level organizational objectives for measurement. Each information need contains guidance, reference information, measures (including plan data), stakeholders and reporting requirements which aid the organization in achieving its goals and objectives. Additionally, each information need will contain a description of how status, thresholds, targets and analysis are created.
Specify Data Collection: With a measurement plan in place and information needs defined, the organization is able to plan the physical collection of measurement data. This step identifies the sources of data for each information need used within each managed unit. Note that while the source of data may vary from managed unit to managed unit, the information need definition does not. Specific data includes the source of data, the method for collecting it, and the frequency of collection are documented in the measurement plan. Team members and other stakeholders who work with the data to be measured are trained or informed of procedures for ensuring that correct measurement data is obtained.
Specify Analysis Procedures: The organization establishes "rules" against which the measurement data will be evaluated for gaining greater insight and spotting trouble early. Approaches include: statistical analysis, mathematical manipulation of the data (summing, averaging, etc.), alarms, comparison of actual against plan, and tracking policy and regulatory compliance.
Collect Data: Actual and plan data is collected on a periodic basis for the specified measures. Data may be entered manually or collected through an automated process and is generally gathered monthly, weekly or even daily, depending on the needs of the organization.
Analyze Data: The collected data is evaluated according to the analysis procedures specified previously, after which information users review the results to assess the impact of recent changes and also to determine if trends in data and status are significant enough to warrant further investigation. Digging deeper into the chain of data to follow problems to their root cause is a key to evaluating alternative action plans.
Store Data and Results: The collected data and resulting measures and analysis is stored in a secure, accessible place. Herein lays the ability to share the data and results across the organization and with customers, and also to use historical data as a valuable contributor to lessons learned and to future planning and estimating efforts. Stored data may also serve as documented proof of process and policy compliance.
Communicate Results: The results of having collected and analyzed measurement data (graphs, dashboards, alarm indicators, management notes, etc.) are shared within teams, across departments and up the corporate management chain so that each measurement stakeholder uses the information to improve decision-making and control performance toward organizational goals and objectives. It is critical that the results are up-to-date, accessible and easy to interpret.
This appendix contains a matrix which maps information needs to Level 2 process areas in the CMMI. Since your actual management and measurement processes are based on your engineering processes, this matrix should be tailored to add and remove information needs, and to modify the mapping so that all processes can be managed. Note: Purpose overviews for each of the CMMI process areas shown below are included at the end of this appendix.
|
Candidate Information Needs CMMI Process Areas
|
|||||||||
|
Information Need
|
Graph Name
|
CMMI Process Areas
|
|||||||
|
REQM
|
PP
|
PMC
|
MA
|
PPQA
|
CM
|
SAM
|
|||
|
Requirements Management
|
|||||||||
|
Requirements Progress
|
Requirements Allocation
|
X
|
|
|
|
|
|
|
|
|
|
Requirements Approval Status
|
X
|
|
|
|
|
|
|
|
|
|
Requirements Growth
|
X
|
X
|
X
|
|
|
|
|
|
|
|
Requirements Object Type
|
X
|
|
|
|
|
|
|
|
|
|
User Requirements Actual Progress
|
X
|
|
|
|
|
|
|
|
|
Requirements Stability
|
Requirements Change Summary
|
X
|
|
|
|
|
|
|
|
|
|
Requirements TBDs
|
X
|
|
|
|
|
|
|
|
|
|
Requirements Volatility
|
X
|
X
|
X
|
|
|
|
|
|
|
Schedule
|
|||||||||
|
Schedule Accuracy
|
Completion Variance
|
|
X
|
X
|
|
|
|
|
|
|
|
Task Compliance
|
|
X
|
X
|
|
|
|
|
|
|
|
Milestone Compliance
|
|
X
|
X
|
|
|
|
|
|
|
|
Schedule Size
|
|
X
|
X
|
|
|
|
|
|
|
|
Schedule Performance
|
|
X
|
X
|
|
|
|
|
|
|
|
Schedule Compression
|
|
X
|
X
|
|
|
|
|
|
|
|
Late Task Aging
|
|
X
|
X
|
|
|
|
|
|
|
|
Duration Variance
|
|
X
|
X
|
|
|
|
|
|
|
Schedule Loading
|
Resource Loading
|
|
X
|
X
|
|
|
|
|
|
|
|
Schedule Slack
|
|
X
|
X
|
|
|
|
|
|
|
|
Task Loading
|
|
X
|
X
|
|
|
|
|
|
|
Cost
|
|||||||||
|
Cost
|
Budget At Completion
|
|
X
|
X
|
|
|
|
|
|
|
|
Cost Performance
|
|
X
|
X
|
|
|
|
|
|
|
|
Cost Variance
|
|
X
|
X
|
|
|
|
|
|
|
Software Size / Lines of Code
|
|||||||||
|
Software Size
|
Total Files
|
|
|
X
|
|
X
|
X
|
|
|
|
|
Total Lines
|
|
|
X
|
|
X
|
X
|
|
|
|
|
Code Growth
|
|
|
X
|
|
X
|
|
|
|
|
|
Source Line Change Summary
|
|
|
X
|
|
|
|
|
|
|
|
Source File Change Summary
|
|
|
X
|
|
X
|
|
|
|
|
Software Configuration Management (SW CM)
|
|||||||||
|
SW CM
|
CM Change Summary
|
|
|
|
|
X
|
X
|
|
|
|
|
Files by Status
|
|
|
|
|
|
X
|
|
|
|
|
Files by Category
|
|
|
|
|
|
X
|
|
|
|
|
File Detail Grid
|
|
|
|
|
|
X
|
|
|
|
|
Change Count
|
|
|
|
|
X
|
X
|
|
|
|
Software Testing
|
|||||||||
|
Test Case Development
|
Test Case Growth
|
|
|
X
|
|
|
|
|
|
|
|
Testing Progress
|
X
|
|
X
|
|
X
|
|
|
|
|
|
Test Executions by Status
|
X
|
|
X
|
|
|
|
|
|
|
|
Retest Count
|
|
|
X
|
|
X
|
|
|
|
|
Risk
|
|||||||||
|
Risk Management
|
Risks by Status
|
|
|
X
|
|
|
|
|
|
|
|
Risk Status
|
|
X
|
X
|
|
|
|
|
|
|
|
Risk Cost
|
|
X
|
X
|
|
|
|
|
|
|
|
Risks by Probability
|
|
|
X
|
|
X
|
|
|
|
|
|
Active Risk Grid
|
|
|
X
|
|
X
|
|
|
|
|
Defects
|
|||||||||
|
Defect Productivity
|
CR Close Rate
|
|
|
X
|
|
|
|
|
|
|
|
CRs by State
|
|
|
X
|
|
|
|
|
|
|
|
Defect Implementation Rate
|
|
|
X
|
|
|
|
|
|
|
|
Defects Close Rate
|
|
|
X
|
|
|
|
|
|
|
|
Open vs Closed CRs
|
|
|
X
|
|
X
|
|
|
|
|
|
Percent Defects with Estimated Effort
|
|
|
X
|
|
X
|
|
|
|
|
|
Total Allocated Defects by Severity
|
|
|
X
|
|
|
|
|
|
|
Defect Quality
|
Defect Arrival Rate
|
|
|
X
|
|
X
|
|
|
|
|
|
Defects by Phase Injected
|
|
|
X
|
|
X
|
|
|
|
|
|
New Defects
|
|
|
X
|
|
|
|
|
|
|
|
Open Defect Age-Severe
|
|
|
X
|
|
|
|
|
|
|
|
Open Defect Age-Showstopper
|
|
|
X
|
|
|
|
|
|
|
|
Open Defects
|
|
|
X
|
|
|
|
|
|
|
|
Open Reported Defects
|
|
|
X
|
|
X
|
|
|
|
|
Defect Schedule
|
Percent Phase Remaining
|
|
|
X
|
|
|
|
|
|
|
|
Total Estimated Effort for Defects By Severity
|
|
|
X
|
|
|
|
|
|
|
|
Total Estimated Hours
|
|
|
X
|
|
|
|
|
|
|
Enhancement Productivity
|
Estimated Effort per Subsystem
|
X
|
X
|
X
|
|
|
|
|
|
|
|
Total ERs with Estimated Effort
|
X
|
X
|
X
|
|
|
|
|
|
|
|
Total Allocated ERs by Priority
|
X
|
X
|
X
|
|
|
|
|
|
|
Enhancement Quality
|
Open Allocated ERs
|
X
|
X
|
X
|
|
|
|
|
|
|
|
Open ERs
|
X
|
X
|
X
|
|
|
|
|
|
|
Enhancement Schedule
|
Percent Phase Remaining
|
X
|
X
|
X
|
|
|
|
|
|
|
|
Total Estimated Effort for ERs by Priority
|
X
|
X
|
X
|
|
|
|
|
|
CMMI Process Area Purpose Overviews
|
CMMI Process Areas
|
Purpose
|
|
Requirements Management (REQM)
|
To manage the requirements of the project’s products and product components and to identify inconsistencies between those requirements and the project’s plans and work products.
|
|
Project Planning (PP)
|
To establish and maintain plans that define project activities.
|
|
Project Monitoring and Control (PMC)
|
To provide and understanding of the project’s progress so that appropriate corrective actions can be taken when the project’s performance deviates significantly from the plan.
|
|
Measurement and Analysis (MA)
|
To develop and sustain a measurement capability that is used to support management information needs.
|
|
Process and Product Quality Assurance (PPQA)
|
To provide staff and management with objective insight into processes and associated work products.
|
|
Configuration Management (CM)
|
To establish and maintain the integrity of work products using configuration identification, configuration control, configuration status accounting, and configuration audits.
|
|
Supplier Agreement Management (SAM)
|
To manage the acquisition of products from suppliers for which there exists a formal agreement.
|