May, 2008
by Scott Chapman
In a continuing proof of the value of exploring performance opportunities, we recently reduced the CPU utilization of our largest CICS application by approximately 15% by simply changing an LE run-time parameter. This savings will not be available to every application, but if you have CICS applications that are written in COBOL and do CALLs between COBOL modules, then you may want to read further.
In the fall of 2007, I noticed our CICS monitor's detailed performance counters for some transactions indicated hundreds of "PUSH HANDLE" and "POP HANDLE" EIP requests. This sparked my curiosity because they weren't immediately familiar to me. A quick scan of the application's source library didn't turn up any "PUSH HANDLE" or "POP HANDLE" calls. So I traced a few transactions and my confusion deepened-the module that the trace implicated didn't have any such calls in it, although it did CALL other COBOL modules, although none of those included a PUSH or POP either.
After discussing this mystery with our CICS sysprog, he found a related LE run time option: CBLPSHPOP. If this option is set ON, when program A CALLs program B then LE adds a "PUSH HANDLE" before the CALL to program B and does a "POP HANDLE" upon return to program A. Why? To save the current state of any HANDLE ABEND or HANDLE CONDITIONs that program A may have done. If program B resets those conditions, the POP HANDLE will restore A's conditions upon return. That process happens for every execution of every CALL statement.
That's pretty handy if program B contains EXEC CICS commands. However, way back when I was an application programmer we had to use EXEC CICS LINK to call programs that were going to do CICS commands. That may not be the best performing strategy today, but we have a lot of COBOL CICS code that still adheres to that philosophy. For such applications, if there is a CALL to program B, program B wouldn't be expected to have any CICS commands in it, so the PUSH/POP pair is unnecessary. In fact, it turns out that those PUSH/POP pairs can consume a fair bit of CPU, and CBLPSHPOP(ON) is known to cause potentially significant CPU consumption.1 Of course if a transaction doesn't do any CALLs then the state of CBLPSHPOP is moot.
Fortunately we have a test environment where I could readily play with the options and run some limited tests. For various reasons, I couldn't do very broad tests, but the limited tests still showed very significant improvements for CBLPSHPOP(OFF) for our largest application-up to a 20% CPU savings for some transactions. Clearly CBLPSHPOP(OFF) is important to this application.
We explained the changes to the application team and proceeded to implement CBLPSHPOP(OFF) in their development and test regions. As expected, no functional differences were noted.
Eventually we moved the changes to production where we did find one minor functional difference. This application does a HANDLE ABEND in the main program invoked by the transaction. Previously, the PUSH negated the HANDLE ABEND as soon as the main program CALLed the next program. Now the HANDLE ABEND remains in effect so when a program lower on the calling chain generates an abend condition CICS tries to return to the paragraph that the original HANDLE ABEND condition pointed to. LE won't allow that and generates an additional APC2 abend. The transaction was already abending though, so the additional abend was not a functional problem-merely an additional dump to ignore.
The final result, after running with CBLPSHPOP(OFF) for a couple of weeks, is a very noticeable reduction in average CPU consumed per transaction. The following two graphs compare the same hour of the day and same day of the week for the beginning of March and April for this application. For example, Tuesday, April 1st was compared to Tuesday, March 4th. On the first day, only one of the seven AORs that service this application had the new LE parm, and you can see there is not a significant overall difference. About 12% of the transactions ran in that AOR and saw about a 13% reduction in CPU utilization, impacting the overall CICS CPU consumption by less than 2%. However, starting on April 2nd, all the AORs had the new LE parm and the CPU reduction becomes obvious.
The following chart shows just the percentage improvement after implementing the new LE parms.
It would of course be nice if the workload was a little more homogenous to make the results crisper. I don't have a good explanation for why there was more of an improvement in the second week. However, it is likely that there was some beginning of month activity the first week that was not present the second week, and that could account for the difference.
Despite the variability, I think we can conservatively say this saved us about 15% off the CPU utilization of this application. That savings is significant enough to show up on the daily utilization graphs, which are not adjusted for transaction volume. This equates to shaving our peak daytime CPU utilization by approximately 3% of our overall installed capacity. That may not sound like much, until you realize that it's approximately 30% of the capacity that would be added by upgrading by one general purpose CPU engine. I'm not going to speculate how long that might delay our next upgrade, but this change certainly has real value to us.
So to summarize, check your LE options to see that they are set optimally for your application. Although not discussed here, there are other LE options (such as STORAGE and ALL31) that may also impact CPU usage. A curious attitude and some investigative work can sometimes yield significant savings. For another example of that, see Geoff Adams' article in the February 2007 MeasureIT.2 We also saved about 2% of our installed capacity by tuning the system options he documented. So the final lesson here is that you should participate in CMG and read MeasureIT-but you already knew that!