z/XPF Reviews
Innovation Data Processing Review
Fred Forst - SAS Institute, Inc. Review
Mike Shaw - Chicago-Soft Review
Terry Fuller - Serena Software Review
Innovation Data Processing Review
When your business is providing software products that improve performance and total cost of ownership for your customers' computing, it's mission-critical to deliver highly-optimized tools that can achieve results quickly. This is the ongoing task for INNOVATION Data Processing, a leading international provider of enterprise data protection, business continuance and storage resource management solutions. Since its beginnings in 1972, INNOVATION's focus has been on combining high performance design with non-disruptive virtualization technology, easy to use and extremely reliable backup, recovery, data replication and privileged information protection that help eliminate downtime and data security concerns for enterprises around the world.
"All of our products are focused on improving application performance for our customers," said Rich Morse, INNOVATION Senior Software Developer, "The goals are to lower resource utilization by reducing CPU cycles and I/Os and by raising overall performance. If our clients can do that with the help of our products, they gain efficiencies and lower overall costs of operation."
Morse and fellow Senior Software Developer Terry Thomas collectively have 70 years experience at INNOVATION developing software products that solve enterprise computing performance and optimization issues. "We've worked on virtually every IBM mainframe operating system and language," said Thomas. "The IBM System z is an outstanding enterprise computing platform, and we pride ourselves in delivering solutions that assist sites in getting the most work they can out of the systems they have."
In its product development lab, INNOVATION runs an IBM System z9 machine that hosts various levels of z/OS, the primary System z operating system. "We use the System z9 for product development and test, and we also work hand in hand with other hardware and software vendors as well as our customers on performance and optimization," said Thomas.
Problem #1
One challenge INNOVATION faces as a solution developer is testing new product releases in every possible system scenario that customers employ. "I had noticed an increase in CPU time when I was doing development on FDRVIEWS, a product that uses FDREPORT extensively as the primary data gatherer," said Thomas. "In our shop, the increase had little proportional impact on the wall time. It was more of just a slight increase, so I ignored it. However, when a beta client called in with a significant increase in CPU time that had a much larger proportional impact on the wall time…as much as 150 percent, if I remember correctly…it caught my attention."
Thomas compared the source releases, examining all changes, line for line. "I found a few things, but nothing that added up to even a few percentage points in CPU time, so I asked Mike Geake, who works with me on the product, if he knew anything about this old software profiling product we had in-house, to see if it could locate the problem areas," said Thomas. "Mike ran the old profiler against three release levels to see if we could find some hot spots where the CPU time had markedly increased, but nothing major popped up. All in all, we were looking at output generated from instructions in six separate modules. We went over all of the entries that were listed, and again nothing more was found. This went on for more than a few weeks, looking, checking, chasing other problems, but always coming back to the listing with no significant results and with a second beta site reporting, now having to now tell two clients that I had found nothing that could account for their problem."
Coincidentally, Lou D'Agnolo, the manager of the INNOVATION Systems Group who was already familiar with ColeSoft's z/XDC interactive debugging tool that INNOVATION had been successfully using for years, was reviewing information on ColeSoft's new z/XPF Profiler and passed it around for comments.
Frustrated and perplexed, Thomas decided to put ColeSoft Marketing Inc.'s new z/XPF software profiler to the task. "The first thing I tried with z/XPF gave me a great place to start," said Thomas. "I ran the same jobs with z/XPF that the other profiler had monitored. Using the ISPF panels provided with z/XPF, I requested z/XPF to schedule a Profile Data Capture session. It was easy. I took the future start option, entered my job name, and since I'm a heavy user of excp, set the excp sampling interval. When the job was complete, I again went to the panel and requested reports from the data that was collected. The Top CPU Consumer's Report was where I started."
Thomas said that he and his team had already spent two or three weeks off and on going over their modules with the other profiler. "We were pleasantly surprised," said Thomas. "After only a few days of using the CPU report and detailed listings from z/XPF, we were able to uncover the causes of our performance issues and implement the necessary changes."
Problem #2
Thomas's colleague and co-developer Rich Morse also found benefits from using z/XPF in problem resolution.
"I encountered a complex Locking problem that was causing delays in processing in our IAM/RLS file server address space, due to contention for a z/OS Local Lock. It was resulting from a high level of I/O activity from concurrently running jobs and CICS regions," said Morse. "One of our customers had also reported elongated run times for some critical batch work that was resulting in their not meeting schedule, and which caused delays in other critical work."
Initially, Morse tried to use the older profiling tool to determine why the local lock contention was occurring in z/OS, but he was unable to quantify how much of the delay was being caused by the contention.
"I decided to try the new z/XPF instead," said Morse. "z/XPF provided the details to quantify the amount of the delay that was being caused by the lock contention. It also helped to identify the functions that were causing the contention…..The z/XPF Work Unit Summary Report has a section on delays caused by contention of various resources." z/XPF is also the only tool of its kind that can report on Lock Suspends. Morse shared this example:
"We had already spent a few weeks speculating about what was causing the delay, and trying to use the older tool to determine what portion of the delay was due to lock contention," said Morse. "When we tried z/XPF, we discovered that z/XPF had this capability, and we were able to project from our internal testing and the z/XPF reports what portion of the delay was caused by lock contention. We accomplished this in a couple of days."
| TOTAL CONTENTION RELATED WAIT/PAUSE TIME. INCLUDES SVC ENQ, ISGENQ, LOCK SUSPEND, LATCH, AND WAIT FOR CPU………………………………………….. | 00.00.00:87.5597 |
| TOTAL NUMBER OF CONTENTION EVENTS OBSERVED ………. | 1,276 |
| TOTAL ELAPSED TIME SUSPENDED WAITING FOR A LOCK ….. | 0.00.00:26.0320 |
| NUMBER OF TIMES OBSERVED LOCK SUSPEND …………… | 547 |
| AVERAGE ELAPSED TIME SUSPENDED FOR LOCK …………. | 00.00:00.0475 |
| The z/XPF product provided the details to quantify the amount of the delay that was being caused by the lock contention, and help identify the functions that were causing the contention. The z/XPF Work Unit Summary Report has a section on delays caused by contention on various resources. From the example report shown above there was .876 seconds of delay, of which .260 (30% of total) was due to local lock contention, and 70% delay was due to other causes. | |
What is special about z/XPF
Because z/XPF is an event-driven profiler, it is able to obtain all of its information on what is occurring within an address space from outside of that address space, looking at the address space as a processor would. This contrasts with the "state" measurement techniques traditionally employed by other profiling products that actually stop the target address space for measurement-taking, which in turn extends elapsed time and can mask underlying causes which then adversely impact trouble-shooting and time to problem resolution.
"This is a great product, and it's fast becoming an important tool in our portfolio," said Morse. "It also comes with the enthusiasm, ease of contact, and the high level of support that ColeSoft provides that makes z/XPF easy to use."
Mary Shacklett - Transworld Data
Fred Forst - SAS Institute, Inc. Review
z/XPF has been very valuable in pinpointing places in our code that are suspect when comparing different releases of our software or comparing on/off scenarios of options in our software. It allows us to drill down to the offset in the code where our code seems to be observed executing excessively. With this detail information, it allows the developer to greatly reduce the possible number of suspect locations in the code. This has allowed us to proactively fix more bugs (or performance problems) before a customer is affected.
The level of service we receive from Colesoft for z/XPF is what sets it apart from other vendors. They have been very responsive to make changes when we have made suggestions. To name a few situations:
- The ability to call z/XPF from within our code (ZXPFTRAC) inserted records in the detail data that could be analyzed for performance.
- Time index reporting capability allowed us to specify start and end times for reporting.
- Mapping of LPA modules was necessary with our 64-bit code.
- History command in the new dynamic ISPF reporting logic.
- Re-write of z/XPF to make data capture logic more parallel in nature.
- Addition of CPU reporting of the target application.
Colesoft has also been very helpful in examining some of our data captures and helping us understand them. They respond very quickly, usually within a day.
Fred Forst - Senior Software Developer, SAS Institute, Inc.
Mike Shaw - Chicago-Soft Review
We use z/XPF to detect and eliminate hard-to-find inefficiencies in our code. z/XPF is the only tool that lets us trace execution of our application in real time and at normal execution speed. z/XPF lets us record and analyze our code’s use of system resources, including I/O patterns, SVCs, and CPU time, allowing us to locate subtle errors that manual code reviews just cannot find. Our product is faster and more efficient now, thanks to z/XPF.
Mike Shaw - Lead Developer of Chicago-Soft’s MVS/QuickRef product
Terry Fuller - Serena Software Review
Serena Software helps organizations orchestrate their application development, IT and business processes. The world's largest organizations depend on Serena's ALM and Process Management solutions to deliver great applications with confidence and to automate important processes for speed, auditability and efficiency. For more information, please visitwww.serena.com, or call (800)457-3736.
Recently, Serena Software installed and tested the z/XPF Event Profiler. Ultimately they kept the product, and here is what they have to say about it.
A large-scale user went into production on our new release. With 300 users logged on consistently and peaks to 700 users they drove the system hard and were very sensitive to performance. When their regular reporting showed a 4x increase in CPU consumption they called for help. The case was escalated to the development team very soon after being opened.
The customer had a market leading "State Profiler" and began sending us reports from the peak usage periods. The reports from the "State Profiler" weren't definitive, but did indicate a problem with one of the master file update routines. The development team spent hours running update scenarios attempting to reproduce the behavior of the customer's system on our system; we have several expert users of the customer's profiler, and each of them weighed in with suggestions for parameter changes to improve the results, but to no avail.
Meanwhile the escalation was making its way up the management chain. Inevitably the question came, "Is there anything else we can do?" We'd recently had a demonstration of z/XPF and knew that it operated on a different principle from the sampling profilers. z/XPF is essentially an "Event Profiler" – a completely different approach to profiling. We put in a call to Cole Software and got a copy.
The day we installed z/XPF we ran our customer-emulation test case and (despite some teething problems with the installation) we were able to identify a hot spot in the master file query code (not the update code that the event profiler had been indicating).
It eventually turned out that this was just one of several performance issues which had masked it from the State Profiler, but z/XPF was able to identify it despite the "noise" provided by the concurrent problems.
Terry Fuller - Software Engineer, Serena Software International
