MeasureIT – Issue 4.03 – Using the Analytic Hierarchy Process (AHP) to identify performance scenarios for Enterprise Applications by Nidhi Tiwari



Using the Analytic Hierarchy Process (AHP)
To Identify Performance Scenarios for Enterprise Applications
March, 2006

By Nidhi Tiwari

About the Author


[Hide]

What is the AHP?

The Analytic Hierarchy Process (AHP) is a quantitative technique for multiple criteria decision making. It provides a convenient way to quantify the qualitative aspects thus eliminating subjectivity in the outcome. The technique begins by clearly identifying the objectives, criteria and alternatives for a given problem situation. For each of the criteria (qualitative or quantitative) a vector is created that gives the relative ranking of its alternatives. Weights of the criteria’s are determined from the problem context. These inputs are then used to relatively rank the alternatives. Detailed description on AHP usage can be obtained from [Haas 1999]. Some AHP tools are: expertchoice (http://www.expertchoice.com/) and Hipre (http://www.hipre.hut.fi/).

Using AHP in performance modeling

The success of any performance modeling exercise hinges on verification of the target system’s architecture using a set of performance scenarios. Proactive performance management recommends the development of a performance model to evaluate the target system’s performance and scalability. Ranking application workloads is not straight forward as these criterions have different weights and many of them are qualitative. In determining the performance scenarios, it is required to rank the application workloads based on these multiple criterions. This article will list some important performance scenario selection criterions appropriate to enterprise-class applications and demonstrate the use of Analytic Hierarchy Process (AHP) to objectively select scenarios.

These performance scenarios represent the application’s workloads, critical in terms of business and performance. One example of a performance scenario for an on-line trading application could be a login page, as it is frequently used, business critical and has stringent performance objective of very low response time. Likewise, based on the application context there could be several other criterions to be considered such as the workload type, number of external interfaces or resource demand. Furthermore one criterion may be more important than the other.

Methodology for Identification of Performance Scenario using AHP

This section depicts the methodology to determine the performance scenarios using the AHP technique. The steps involved in the method are as follows:

Step 1: Determine and Rank the Criterions for the Application Context
Several publications and books [Jain 1991], identify attributes of workloads which influence its selection for performance modeling:

  • Business Criticality (can be enunciated as user-type, revenue-impact, etc)
  • Performance goals such as response time.
  • Workload intensity (measured as hits/sec or number of concurrent users, payload size)
  • Resource Intensiveness (system utilization)
  • Workload Type (Online/Batch/Messaging/Report Printing).
  • External Interfaces
  • Ease to build and test.

AHP theory classifies these criterions as benefit criteria and cost criteria [Saaty 1994]. Benefit criteria are those criterions where higher values are desired. In the presented list, business criticality, workload intensity, number of external interfaces and resource intensiveness are benefit criterions, as the application workloads with higher values of these are preferred candidates for performance modeling. Cost criterion have lower desired values. It is implicit that response time is a cost criterion.

Based on the application context, one criterion may be more important than the other. A pair-wise comparison and the eigenvector of priorities is used to rank the criterions for application context. Consider a simple case of ranking benefit criterions for a camera:

Step 1.1 Pair-wise comparison with relative ranks
The relative importance of one criterion over another employs a pair-wise comparison. Pair-wise comparison examines the relative importance of each option against each of the other option in pairs. This can be represented using a square matrix of options.

Step 1.2 Eigenvector of ranks
AHP technique uses eigenvectors to mathematically determine the ranks of options from the pair-wise matrix [Haas]. The eigenvector of priorities is calculated by squaring the pair-wise comparison square matrix and normalizing the row totals. The eigenvector of the absolute ranks of benefit and cost criterions is calculated separately from their corresponding pair-wise comparison matrices.

Step 2: Rank the Application Scenarios for each criterion

Once the criteria values for scenarios are available, the next step is to compute the relative ranks of the application workloads for each criterion. For quantitative criteria scenario ranks are derived by normalizing their values, to make them comparable with other criteria values.

However, for qualitative criteria, AHP advocates pair-wise comparison of scenarios. If there are "n" scenarios and "m" qualitative criterions then (n* (n-1)/2) * m user inputs would be required. For an application with just 10 scenarios and 1 criterion, 45 inputs from users need be taken. Since most applications consist of a large number of application scenarios and multiple criteria, getting inputs from the user for pair-wise comparison may not be feasible.

Alternatively, AHP recommends the ratings approach. According to this, a scale of intensities (i.e. low, medium, high) is established for qualitative criteria like business criticality or resource intensiveness. Pair-wise comparison of these intensities is done to get the relative importance of one over another and their relative ranks are computed. Then user rates the scenarios on these scales of intensities for the corresponding criteria [EPA 2002]. These scale intensities are then replaced by their priorities for further processing. Thus only n*m+ (i*(i-1)/2)*m (where i is the number of scale intensities for m criteria) user inputs would be required. For the case described above, the number of user inputs required for pair-wise comparison would be 3, besides the scale ratings of scenarios for each criterion.

For the camera example discussed above, the ranking for style criterion, which is qualitative, is computed as follows using the ratings approach.

Steps 2.1 through 2.4

Step 3: Compute overall ranking of Application Scenarios using cost-benefit analysis.

In the first two steps, the scenario ranks for each criterion and criteria weights have been computed. In step three, we will aggregate the scenario ranks for all cost and benefit criterions separately. In the camera example, the overall ranks of the options is obtained by taking the ratio of their benefit and cost ranks.

Step 3.1 Ranks of Options for benefit criteria
To achieve this, the application scenarios ranks for all benefit criteria, obtained in step 2 are represented together in a matrix form. This matrix is multiplied with the eigenvector of benefit criteria created in Step 1 to get vector of the total rank of scenarios based on all benefit criteria. Thus, if there are "n" scenarios ranked for "m" criteria, then a matrix of order (n * m) is formed and multiplied with (m*1) criteria vector, consisting of ranks of these "m" criterions to get a (n*1) scenario ranks vector.

Similarly, a vector of scenario ranks based on all cost criteria weights is derived.

Step 3.2 Ranks of Options for cost criteria

The final step is computation of an overall rank for scenarios by taking the ratio of their benefit and cost ranks [Saaty 1994]. The scenarios with higher values of ratio are selected.

Step 3.3 Aggregated ranks for Benefit and Cost criteria and the overall Benefit/Cost Ranks for camera options

Case Study

In this section we present the re-engineering of a shipment application. The existing application had performance problems and needed to support a substantial increase in workload due to business requirements. Performance modeling was initiated for the new application architecture.

As a result, a minimum set of application scenarios was required. Therefore with the objective of determining critical and frequently used workloads, a workload modeling study was conducted. The study helped to baseline the arrival rate, arrival pattern, and response time for the 200 odd application workloads. Table 1 shows a snap shot of the workload modeling results and the projected workload metrics for the target system based on inputs from the business stakeholders.

Table 1 - Modeling results and projections

Table 2 - Criterion rank for application context
A mix of OLTP, batch, message and report printing workloads was observed. It was decided to take performance scenarios of each workload type for modeling. Here the application of AHP method for identifying OLTP performance scenario is discussed. Table 2 shows criterions, eigenvector values and normalized ranks (step 1.2) determined from the pair-wise comparison (step 1.1).
Table 2 - Criterion rank for application context

Table 3 - Ranking of criterion application scenarios
While the business criticality, throughput and resource intensiveness were considered benefit criterions, response time was regarded a cost criterion. Following the procedure shown in step 2, ranking of scenarios for the qualitative criterion (business criticality and resource intensiveness) was determined (using the ratings approach). A scale of low, medium and high intensities was established and pair-wise comparison (step 2.1) of these scales was used to get eigenvector values (step 2.2) and normalized ranks (Table 3).

For quantitative criteria (response time, throughput) normalized ranks of the scenarios was obtained. From the procedure described in step 3, the weighted benefit and cost rank of each scenario was calculated by matrix multiplication. Subsequently, the overall ranks of scenarios was obtained by taking the ratio of their benefit and cost ranks (step 3.3). Table 4 shows these benefit/cost ranks in terms of percentage values.

Table 4 - Benefit/Cost ranks (%)

From the above analysis it was determined that the ratio of 9.42 in scenario 4 indicates the most representative performance scenario for the OLTP workload type.

Additional scenarios from other workloads (batch, message and report printing, etc) can be picked having high values of benefit/cost ratio. Using the same process, the different workload types resulted in selection of the performance scenarios as shown in table 5.

Table 5 - Selected performance scenarios with highest benefit/cost ratio

Conclusion

The AHP methodology described in this article can facilitate and accelerate the selection of the most valuable performance scenarios. This can focus performance modeling on a key few performance critical scenarios, rather than evaluating hundreds of application’s scenarios. Some critical factors for choosing performance scenarios have been suggested. However, more criteria can be taken into account based on system, user and business requirements. By following the outline of the steps described, selection of performance scenarios can be made more objectively, based on scientific method of AHP.


References

[Haas] An Illustrated Guide to the Analytic Hierarchy Process. Dr. Rainer Haas, Dr. Oliver Meixner. Institute of Marketing & Innovation. University of Natural Resources and Applied Life Sciences, Vienna. http://www.boku.ac.at/mi/

[Jain 1991] The Art of Computer Systems Performance Analysis. By Raj Jain. Published by John Wiley & Sons, Inc. 1991.

[EPA 2002] Appendix A The Analytic Hierarchy Process and the Expert Choice Mercury Retirement Model. http://www.epa.gov/ORD/NRMRL/pubs/600r03048/600R03048appA.pdf

[Saaty 1994] Decision Making in Economic, Political, Social and Technological Environments with the Analytic Hierarchy Process. Thomas L. Saaty and Luis G. Vargas. RWS Publications. http://www.boku.ac.at/mi/ahp/ahptutorial.pdf

 



Last Updated