Everyone goes to a different CMG.

That’s a paraphrase of one of my mom’s sayings about parties. She and her pals would have coffee the morning after a neighborhood party and talk about what each heard at the same party. The differences in recall and detail were astonishing, hence their phrase, "Everyone goes to a different party".

So, when Mark Friedman asked me to write a trip report for MeasureIT, I worried that what I found interesting and exciting may not ring as true with all of you. As usual, there were always at least two sessions in each time slot that interested me, and the added burdens of speaking and vendor influence activities carved up a lot of my time. Still, I’ll take you through what I view as a common occurrence after I go to a CMG national conference; the things I learned at a CMG conference directly helped me on the job.

One of the innovations at CMG2003 were the half hour sessions (I’m a fan) where an author could present a significant finding briefly. Three of these sessions were combined into one session (I’m not a fan) so I often ended up sitting through two to hear the one I wanted. I hope that these innovations continue, perhaps with a morning of distinct 20-25 minute sessions beginning on the half hour, where we could dart merrily around, grabbing coffee as we zip between sessions or sneaking over to the vendor area to try out some products. There’s a lot of great work out there, perhaps a shorter format would draw out more folks to talk about theirs.

At one of these half hour sessions, (look for paper 3225 on the CMG web site) "Performance Implications of Hyper-Threading" Yiping Ding described a workable model of how to use queuing theory to describe the effects of Intel’s Hyper-Threading on performance. I found the elegant and simple "tandem-queue model" that they developed to encapsulate both the overhead derived from the choice of virtual processors and then the actual processing matched my observations of production systems very closely.

While processors with hyper-threading enabled may appear to some of our measurement systems as two distinct processors, with most business workloads, there is no way we will get a full two processors worth of work through them. (I typically use 1.3 as a first guess estimate with our typical workload mix, yours may vary.) Innovative research like this paper presents helps all of us to make far more accurate performance models and forecasts for our firms. If only we could keep track of which of our thousands of machines had these processors and which had hyper-threading enabled or disabled!

Another astonishing thing I saw in paper 3187, "PetStore-WS: Measuring the Performance Implications of Web Services" was a microscopic view of how a java process consumes resources. It’s always great to see someone show a tool being used on a real problem. What stunned me was just how much time was consumed in java garbage collection or memory "clean-up" activities in their test system, on the order of 8.5% (0.085)!

Some of you may have been infected by my efforts to find process pathologies automatically, and one of the easiest ones to find is a loop, where a single process gets stuck in some code that eats an entire CPU for extended periods without doing business useful work. (Read paper 3027, Automating Process and Workload Pathology Detection for more details.) When I returned from CMG, I was investigating an application performance issues and was stunned to see a process that looked a lot like a loop to my eye, but was consuming CPU in amounts just under what it took to qualify under my current rules.

Looking further, what I found was a java process that averaged around 91.5% ((1-0.085)%!) utilization all the time, just what a CPU loop that had to give up 8.5% of it’s time would do! Even if your loop is not allocating memory (few do), it must still stop for garbage collection logic, so it’s anticipated "looping consumption" must be reduced by the garbage collection time. I’m now in the process of changing my loop detectors to allow a lower loop threshold only for java processes. Here again, a small part of someone else’s work presented at CMG can spur us on to new useful tools and methods for our firms. I’ll keep you all posted on how this experiment turns out.

Finally, there are sessions that stand out in your mind as wonderful ways to communicate the issues that we all face. While you will not get the benefit of the wonderful speaker, I urge you to read paper 3070, "Performance Forensics" by Bob Sneed. Whether you use Solaris or not, a great analogy is often the key to getting all the folks investigating a problem to pull their heads out of their niche and focus on the big picture. Bob’s paper is a wonderful blend of wisdom and facts that in my mind represent the best of what CMG can do.

CMG isn’t all sessions and technology. I also got to spend time with great friends, both old and new, at dinners and PARS. I recall fondly the great Tex-Mex food we had one evening, but the name of the restaurant escapes me, possibly due to the effects of another wonderful discovery, a fabulous margarita martini discovered and perfected by one of my pals. (2.0 oz Blue Agave Reposado Tequila, 1.5 oz Cointreau (no substitute), juice from one fat freshly squeezed but not too tart lime, shaken with cubed ice and served with a lime wedge. He prefers no salt on the rim, I like salt, but we both strongly urge that you walk home after the experiment.) I’ve tried this at several venues, including the spinning bar atop Dallas’s Reunion Tower, and can report consistent good reviews from all dedicated co-experimenters. After a few CMGs, you will grow accustomed to fellow attendees sending extremely detailed instructions to hapless bartenders!

I closed many of my talks with this challenge to the audience. Think about the conference you just went to and write down the title of a paper that you wish had been there. Sketch out an outline of what it might contain. Then write that paper and submit it for next year’s conference! I can tell you from personal experience that I learned more from the paper writing process than I could ever have imagined. If you have a topic beyond your own abilities, do what I did; offer to co-write with your favorite expert or performance idol. Providing data, hard work and test time is often exactly what experts need to help discover something new. You’ll definitely get back far more than you put into it.

Write a paper for next year!

Check out Ron's papers from CMG2003

'Business Metrics and Capacity Planning' (PDF 220 KB)

'Automating Process and Workload Pathology Detection' (PDF 316 KB)

'Time Stable Workload Characterization Techniques' (PDF 228 KB)