Is Your Investment in IT Project Management and Software Process Improvement Paying Off?

By Joe Kolinger, CTO - OfficeWork Software - 866-331-4534

A note from the author:

Ok.  It's not cheap to invest in project management personnel and training.  Are you spending too much?  Are you being too stingy?  How can you know capability is actually improving - apart from subjective anecdotes?  And, are you sure you're spending money on the right people, programs and tools?  What can you expect in return?  How can you measure?

Sorry for so many questions - but if you're going to build the capability of your project people to meet critical company goals you can get some ideas on how by reading this article. 

Introduction

The Software Engineering Institute (SEI) Software Development Maturity Assessment Methodology is used to assess the software development capability of organizations. Research by Lawrence Putnam of Quantitative Software Management (QSM) demonstrates a strong relationship between Capability Maturity Model (CMM) maturity level and the QSM ‘Productivity Index’ (PI). Specifically, rising CMM levels result in higher Productivity Indices, which result in lower development costs.

In a nutshell, higher Productivity Index values are associated with projects that cost less, finish faster, and have fewer defects. Ideally the CMM process improvements should be associated with more efficient projects and better quality. What’s covered in this article is that the QSM methodology, benchmark database, and tool set measure of the benefits of CMM improvements.

This article points to the economic benefit of effective software process improvement, and the role that measurement plays in proving it.

Background

Many companies have undertaken software process improvement (i.e., Software Engineering Institute’s CMM/I) with the hope that better process will somehow produce better results. For example by moving to CMMI level 3 they would expect to experience projects with shorter schedules, reduced costs, improved reliability, fewer emergencies, etc. However, without a metrics plan and a quantitative toolset in place – apart from “anecdotal evidence” – they will never really know. Even worse, process focus, without the right measures encourages individuals to default to rote compliance with ‘process.’ Eventually, lacking any believable measures of improvement the organization abandons the improvement effort altogether. After all, process improvement requires discipline and continuous investment.

If you’re going to pursue software process improvement, protect the investment from the outset with a solid metrics plan and benchmark database.

Start a Basic Metrics Plan

Rather than trying to measure too much, organizations need a basic ‘starter set’ of metrics for basics such as duration, effort, size, and defects. Furthermore these metrics should not be used to measure individuals, but rather to better understand the software development process, making continuous improvement for repeatable success. Metrics misapplied will submarine data quality and put a halt to improvement. Fear drives out learning. Without learning there is no improvement.

Apply a Measurement Framework

Software projects are ‘different’ from other projects, such as construction. Software does not obey the laws of physics and science. Rather it requires human learning, discovery, problem solving and communication, which makes schedule prediction more difficult. (When asked to estimate “how long” to complete an unfamiliar programming task the developer responds, “Don’t know. How long does it take to catch a fish?”)

Software development projects – or most design-type problem solving projects follow a non-linear resource staffing pattern. The sample Raleigh Curve below (Figure 2) shows the common pattern. At the beginning of the project there is a staffing ramp-up (Physical Design), the project peaks at the conclusion of programming and the beginning of testing, and finally the long tail (Testing and Debugging) represents the effort to find and remove bugs over a considerable time frame.

Software development and circuit design projects tend to follow this effort/staffing distribution.


[Figure 2] – The entire lifecycle of a software project follows a curve of rising and then falling manpower. The long tail of the curve represents the many years of so-called software maintenance.

Also included in the QSM Measurement Framework are these measures:

•       Effort – Person hours of work
•       Duration – Elapsed days
•       MBI (Manpower Buildup Index) – Rate at which people are added to the project
•       Defect Density – the number of bugs to be removed
•       Size – Some characterization of what is delivered.
•       Productivity Index – a macro measure of the organization’s development efficiency

Understanding the Productivity Index at a Glance

The software process Productivity Index (or PI) is a QSM metric, representing the level of an organization's software development efficiency. The PI is a macro measure of the total development environment. PI values from 1 to 40 have been adequate to describe the full range of projects seen so far.

Low PI values typically are associated with poorer working environments, poor tools and/or high product complexity. Higher values are associated with good environments, tools and management and well-understood, straightforward projects [Ref. 1, 5].

"Productivity" encompasses many important factors in software development, including:

•       Management influence
•       Software development process and methods
•       Software development tools, techniques and aids
•       Skills and experience of development team members
•       Availability of development computer(s)
•       Complexity of application type

Note that the PI is calculated from the size, schedule and effort that were applied to a completed project. This means that the PI is objective, measurable and capable of being compared on a numeric scale.

Projects normalized around the PI can be meaningfully compared to one another. Without this normalization projects’ performance cannot meaningfully be compared. For example, project A took 6 months to complete, and project B took 4 months to complete. What conclusion can be drawn? None. But if both projects have PI’s then we might see that one had greater size, or greater complexity – or had a team that ramped too slowly. In any event, an organization that gets a bead on its PI has incredibly valuable information to help estimate future projects.

CMM Transition Breakpoints

Research conducted by QSM on the relationship between CMM Level and PI shows in table 3. The benchmark database consisted mostly of Level 1 and Level 2 projects (which characterize the world), therefore the relationships are statistically strongest at these levels. In the Business Systems column we can see that average PI improves from a 12 to a 17 as the organization graduates to Level 2.


[Table 3] Transition Breakpoints for Three Application Types - Note: * Estimated


PI Improvement Reveals Economic Savings

The graph in Table 4 shows results for three Business Systems projects of similar size and complexity. Process improvements – which are related to PI improvements - have lead to more efficient development capability, and a much lower cost. The transition from CMM Level 1 to 2 shows a 50% cost reduction. The transition from Level 1 to Level 3 shows a 75% reduction in costs and a 250% improvement in reliability.

But if we’re to realize a 50% savings, how could it be reasonably spent? How about these options:
•       Finish faster
•       Use fewer people
•       Deliver more scope
•       Use the saved money on other projects
•       Give back to the Business




[Table 4] – The Economic Value of Software Process Improvement for Business Systems

[NOTICE the cost at level 1 vs. cost at level 2]

The related graph in Figure 5 displays the Raleigh curve effort distributions for the three Business Systems projects. Note that project duration and peak staffing decrease with CMM level improvements. This is good news.



[Figure 5] – The Economic Value of Process Improvement (Courtesy of QSM)


Is your software process improvement (& project management) effort paying off?”

Companies rightly undertake process improvement initiatives looking for some kind of improvement in delivery and cost. To some the method of choice is the CMM, to some it’s Agile methods, to others it’s custom processes and project management offices (PMO). But a word of caution: the project and organizational processes that can be implemented have the possibility – but not guarantee - of improving software delivery. Organizations frequently become totally lost in process, and confuse ‘process sophistication’ with ‘real maturity’.

Using the right measurement framework we can actually tell the difference between process improvement efforts that are paying off and those that are not. Are we finding more bugs earlier in the lifecycle? Are the schedules becoming shorter? Is the Productivity Index increasing? Are costs dropping? Do we require fewer people to get the project completed? Is the user finding fewer bugs? Which techniques are giving us the best return on investment? Are project estimates improving? Are fewer dates slipping? What are the priority areas we should next focus on? The point here is that the right measurement framework helps us to know and intelligently manage the investment in process improvements to yield the best return on that investment.

You may have process improvement efforts from which you are already seeing benefit. It’s possible that what you are seeing is only a fraction of what is possible. Without a benchmark database you will not know the extent to which you are performing beneath your capability.

Conclusion -

QSM’s research suggests a strong relationship between CMM level and the QSM Productivity Index. These improvements lead to significantly reduced software development costs. With a metrics plan and the right measurement tool set, meaningful measures position an organization to manage significant economic benefit.

Considering that the QSM tool set has a framework including industry data from over 8,000 projects, and a method to normalize project experiences, it is suitable as a software project management estimation and analysis tool.

This article points to the business case for software improvement. If we understand the nature of CMM level 2 and 3 improvements, they are primarily focused on project management. For this reason this article also suggests the business case for investing in project management.

References

1. Putnam, Lawrence H. and Ware Myers, Measures for Excellence: Reliable
Software On Time Within Budget, Prentice Hall, Englewood Cliffs, NJ, 1992, pp. 32-36.

2. Humphrey, Watts, David H. Kitson and Tim C. Kasse, The State of Software Engineering Practice: A Preliminary Report, CMU/SEI-89-TR-1, ESD-TR-89-01, Software Engineering Institute, Carnegie-Mellon University, Pittsburgh, Feb. 1989, 27 p.

3. Putnam, Lawrence H., “The Economic Value of Moving up the SEI Scale”,
Managing System Development, Applied Computer Research, Inc., July 1994

4. Putnam, Lawrence H., Arlyn D. Schumaker and Paul E. Hughes, Economic
Analysis of Re-Use and Software Engineering Process, (Final Draft Report) TR-
9265/11-2, prepared for Standard Systems Center, Air Force Communication
Command, Maxwell Air Force Base, Alabama 36114, under contract FO1620-90-D-0007, February 1993.

5. Putnam, Lawrence H. and Ware Myers, Executive Briefing: Managing
Software Development, IEEE Computer Society Press, Los Alamitos, CA, 1996, 79 p. Linking the QSM Productivity Index with the SEI Maturity Level,
Version 6, 2000

6. Putnam, Lawrence H, Linking the QSM Productivity Index with the SEI Maturity Level. July, 2000

6. Kolinger, Joe, Seven Signs You Have a Bad Project Estimate, Project Management Institute SF-Bay Area Chapter, January 2010



About Us

OfficeWork Professional Services trains project managers and positions organizations to run successful IT projects. 

We are experts at showing you how to save a ton of money and time getting your projects done.

Call us at 866-331-4534 for a free discussion on how to implement measurable improvement in your company.

Sign up for our training course, "Successful project estimating": 

http://www.officeworksoftware.com/estimating-course.php

And finally, some advice

Spend your money a little differently … and get a much better result.

Also see, “7 Signs You Have a Bad Project Estimate … and What to Do About It”: http://www.kolinger.net/2010/02/03/7-signs-you-have-a-bad-project-estimate-and-what-you-can-do-about-it/

Go to www.officeworkps.com for more details.  966-331-4534.