All about Metrics and formulae

Definition: Metric is a unit of measure.

(IEEE) A quantitative measure of the degree to which a system, component, or process possesses a given attribute

Why measure?

1. to get better understanding of the attributes of models

2. to gain insight into the design and construction of the software

3. focus on specific attributes of software engineering work products resulting from analysis, design, coding, and testing

4. assess the stability of a software product

5. determine the quality of the current product or process

6. estimate the cost & schedule of future projects

7. evaluate the productivity impacts of new tools and techniques

8. establish productivity trends over time

9. forecast future staffing needs

10. anticipate and reduce future maintenance needs

Categories of metrics

1.     Projects

·         Productivity and schedule related

2.     Products

·         Assesses the state of the project

·         Track potential risks

·         Uncover problem areas

·         Adjust workflow or tasks

·         Evaluate teams ability to control quality

3.     Processes

·         Activities related to production of software

·         Lead to long term process improvement

Metrics definitions and formulae

Metric Name	Definition		Unit of Measure		Formula
Project Related Metrics
Effort Variance	Effort Variance is used to keep the track of the variation in effort for an activity with respect to planned effort		(%) of Hrs		(Actual Effort – Planned Effort) / (Planned Effort ) X 100
Schedule (Delivery) Variance	Schedule Variance is used to keep track of variation in number of days required for each deliverable in the project with respect to planned days (%age)		(%) No of Days		(Actual End Date – Planned End Date) / (Planned End Date – Planned Start Date +1) X 100 OR (Actual Duration – Planned Duration)/Planned Duration  X 100
Milestone Variance	Milestone Variance is used to keep track of number of days required for difference milestones in the project in %age		(%) No of days		Actual End Date – Revised End Date)/ (Revised End Date – Planned Start Date) X 100
Productivity	Productivity is the effort that we are putting in developing one unit of size which helps in planning activity		(%) No of Hours		Size / Effort (Size is defined here as Budgeted Hours)
Process Related Metrics
Testing Efficiency		Testing Efficiency is used to track efficiency in terms of number of test steps authored  per one unit (hr) of testing effort		Test Steps authored + Executed		Test Steps Authored + Executed / Testing Time (Testing time is considered as QA’s time which include test case authoring, execution, automating scripts and automation testing)
Review Efficiency		Review Efficiency is used to keep track of efficiency in terms of number of defects per one unit (hr) of testing effort		Defects / hr		(Actions + Defects) / Review Time
Testing Effectiveness		Testing Effectiveness is used to keep track of defects that are leaked to subsequent phases with respect to defects injected in that phase		Defects Rate		No of Defects / (No of Defects detected + leaked to later phases) X 100
Review Effectiveness		Review Effectiveness is used to keep track of defects that are leaked to subsequent phases with respect to defects injected in that phase		Defects Rate		No of defects detected / (No of defects detected + leaked to later phases) x 100
Product Related Metrics
Defect Density		Defect Density is used to keep the track the product quality at the end of each phase in the project		(%) No of Defects		No of Defects / Size (Size is defined here as Budgeted Hours)
Defect Removal Effectiveness (DRE)		DRE is used to track the % age of defects effectively removed during a phase		Defects		No of Defects identified in the current SDLC stage / workstream / (No of defects injected in the current SDLC stage / workstream phase + No of defects leaked in to the current SDLC stage / workstream phase from previous phase) X 100
Cost of Quality (COQ)		COQ is used to track effort spent in reviews, testing, rework, defects prevention, training and audits with respect to total project effort		(%) Hrs		Review Effort + Testing Effort + Rework Effort
Cost of Poor Quality (COPQ)		COPQ is used to track the rework for the defects during review and testing in a project with respect to total project effort		(%) Hrs		Rework Effort
Cost of Poor Quality Index (CPQI)		CPQI is used to track the amount of rework effort spent against Customer Budgeted hours		(%) Hrs		Rework Effort / Budgeted Hours X 100

ISO

ISO 9001:2000

Model Overview

Guides how to implement the quality systems and software engineering activities.
The ISO 9001:2000 standards are based on the following eight quality management principles.

1. Customer Focus.
2. Leadership.
3. Involvement of People.
4. Process Approach.
5. System Approach to Management.
6. Continuous Improvement.
7. Factual Approach to Decision-Making.
8. Mutually Beneficial Supplier Relationships.

CMMI

Software Engineering Institute Capability Maturity Model Integration (CMMI®)

On March 1, 2002, the integrated model for systems and software engineering, Integrated Product and Process Development, and supplier sourcing (CMMI®-SE/ SW, version 1.1) was released.

Maturity Levels

There are five maturity levels. Each maturity level defines an approach of process improvement and stabilizes an important part of the organization’s processes.

The five maturity levels are:

Level 1: Initial

The process capability at Level 1 is considered ad hoc because the software development process constantly changes as the work progresses. Schedules, budgets, functionality, and product quality are generally unpredictable. Success depends on having exceptional people who are working in the organization for quite some time.

Level 2: Managed

Level 2 organizations know about basic management controls and have adopted them. Policies for managing a software project and procedures to implement those policies are established. Planning and managing projects is based on experience with similar projects. Software costs, schedules, and functionality are tracked.

The capability of Level 2 organizations can be summarized as disciplined, because the ability to successfully repeat planning and tracking of earlier projects results in stability. The following six key process areas:

• Requirements Management
• Software Project Planning
• Software Project Tracking
• Software Subcontract Management
• Software Quality Assurance
• Software Configuration Management

Level 3: Defined

There is documentation on standard engineering and management processes for developing and maintaining software across an organization. There is a group responsible for the organization’s software process activities.

The capability of Level 3 organizations is summarized as standard and consistent because engineering and management activities are stable and repeatable.

The following are key process areas:

• Organization Process Focus
• Organization Process Definition
• Training
• Integrated Software Management
• Software Product Engineering
• Inter-group Coordination
• Peer Reviews

Level 4: Quantitatively Managed

A Level 4 organization sets quantitative quality goals for both software products and processes. Productivity and quality are measured the capability of Level 4 organizations is summarized as predictable because the process is measured and operates within measurable limits.

Quantitative Process Management and Software Quality Management are the two key process areas.

Level 5: Optimizing

At Level 5 the entire organization is focused on continuous process improvement. The organization has the means to identify weaknesses and strengthen the process proactively, with the goal of preventing the occurrence of defects. Root cause analysis and defect prevention steps are taken.

Key process areas of Level 5 are:

• Defect Prevention
• Technology Change Management
• Process Change Management

Criteria To Select A Model

Criteria To Select A Model

Any IT organization should take in to account the below mentioned four criteria when selecting a model:

• Applicability of the model to the organization’s goals and objectives.

Before adopting a model, the organization should check for the consistency of the criteria imposed by the model with the organization’s goals and objectives. It is important that each criterion in a model be directly related to accomplishing the organization’s goals and objectives.

• Management commitment.

Management commitment is the key factor to get success with any model implementation. The style of management should vary from traditional way to a quality perspective manner to check if the models are implemented properly or how to implement the compliance or different proven methods of doing work.

• Need for baseline assessments.

An organization should know how effective and efficient it is. For this a measurement programs have to be created and goals have to be set that become the baseline for the measurements. It could be both subjective and objective in nature.

• Need for measurable goals and objectives.

Organizations that desire to improve, need goals and objectives to measure that improvement. If a model is accepted as the means for continuous improvement, then measurable goals and objectives can be defined that describe movement toward meeting the model criteria.

Quality models

Purpose of a model

A model is a concept to be accomplished that is ideal. These have specific standards defined under national or international standards organizations. And when these are to be implemented for a purpose the standards can be tailored. These models have to be accomplished for compliance and are measured in "pass/fail" terms.

Compliance assessments can be through first-party, second-party or third-party audits or other types of evaluation.

Organizations choose to adopt a model for any or all of the following reasons:
• Meet business goals and objectives.

Ideally by implementing a model by an organization can refocus the goals and objectives to meet them in a more appropriate manner.

• Requirements are imposed by key customer.

There are scenarios where the key customer imposes the adoption of model in the IT organization to improve the efficiency and effectiveness.

• For competitive reasons.

Few customers’ select IT organizations that is in compliance with a model to do business.

• For continuous improvement.

All the quality models show a way to improve the existing process and how to lead a path for continuous improvement. So having a model would help IT organization to have roadmap for continuous improvement.