August, 2007
by Michael A. Salsburg, D. V. (Doctor of Virtualization)
Michael is responsible for the strategic direction of the overall server product line, with a focus on System Performance and overall Architecture.
He received his Bachelor of Science from the University of Pittsburgh (magna cum laude) in Mathematics in August 1972. He received his Masters of Science from the University of Delaware in Computer Science (Software Engineering and Artificial Intelligence) in 1982. He received his Doctor of Philosophy in Mathematics from Drexel University (Probability and Statistics) in 1992.
He has been awarded two international patents in the area of performance modeling algorithms and software. In addition, he has published dozens of papers and has lectured world-wide on the topic of computer performance evaluation. His current interest is in the area of High Performance Computing.
About the Author
Michael A. Salsburg, Ph.D., Technology Director, Unisys Corporation
Aren't you sick of hearing the marketing hype about computing being like a utility? The good Doctor is. Utility computing as a vision is certainly commendable, but over-zealous marketers have to keep in mind that it's not fully baked yet. Meanwhile, the hype-meisters want to put it on the menu already. Meanwhile, down here in the real world, we live with everyday SNAFUs, including typical phone conversations that pause with ..."oh, the computer must be down or slow or..."or whatever... And while we're in the midst of this swamp, you read another article about Valhalla, where computing power is a utility and parceled out like kilowatts or something. It would be easier to envision this dream if our darn workstations didn't have to be re-booted every other day. Enmeshed in this vision is the idea that computing resources are available at a set level of service. And if we just look a little more closely, we realize that there has to be a highly available system before you can even dream of getting consistent service times. But the good doctor is ranting, so he will calm down now.....
So, you may ask, what does virtualization have to do with this discussion? In this article, I would like to explore how server virtualization can address one facet of utility computing, which is carrier-grade availability. Availability is considered carrier-grade (as in the carrier of telephony) if its availability meets or exceeds 99.999% available. When we pick up a phone (I mean a land-line phone, not a cell-based phone), we would be surprised NOT to hear a dial tone. Wouldn't you like to live in a world where if an IT service is not responsive, you are surprised?
Now, some of you mainframers might be clucking at this point that mainframes can achieve five nines in availability. This is true. But today's computing relies on many more components than just the mainframe. There are web and application servers involved in most internet-based commerce. Computing is not about a single, fully integrated and reliable system called the mainframe. It is about the entire web of systems that must all behave to achieve that coveted five nines.
There are numerous ways in which server virtualization can improve availability. This article will discuss improving availability by reducing the time to recover from a failure in the system. Many of the ideas are drawn from the seminal work of David Patterson.
You may know him as the leading author of a little paper called, "A case for redundant arrays of inexpensive disks (RAID)"1. This paper started the whole RAID ball of wax and has had a profound effect on data storage reliability. His more recent paper, "Recovery Oriented Computing (ROC): Motivation, Definition, Techniques, and Case Studies"2 is compelling and thought-provoking. One of the important points in the paper is that, even though failure rates continue to improve, an area that has not been investigated and addressed enough is the time to recover after the failure. Remember that availability is a function of both the failure rate and the time to recover:
where MTTF is the mean time to failure and MTTR is the mean time to repair.
Note the following limits:
In English, availability approaches unity (an infinite number of nines) as either the mean time to failure becomes infinite or the mean time to repair approaches zero.
The reason we have consistent dial tone is not that all of the components are infallible. The reason is that, when an error occurs, recovery is near-instantaneous. There is a lot of complexity involved in near-instantaneous recovery. One of the factors is how well the fault is isolated. If a single component fails and it only affects a few circuits, the impact is not felt globally. Imagine that, every time a telephone component failed, all of the global network of circuits would have to be re-initialized. So, redundancy and isolation are factors in quick recovery. Virtualization can help in both of these areas.
In today's environment, recovery from a server failure can take about 15 minutes using a "gold image". This gold image is an image of the disk that is made after the operating system and applications have been fully configured. It is used to make "clones" of the configured server. For example, you could have multiple web servers that are all cloned from a single gold image. So, consider a scenario where a server fails. Another server, that will replace the failed server, can be fully functional in 15 minutes. That's the mean time to repair. If there are multiple servers involved (such as one of many web servers), one failure will not cause a system failure, but service will degrade until the re-purposed server is brought up. With virtualization, the image of the system is an image of the memory of the system, not its boot data. The time it takes to move an image of a virtual machine and then make it available can be measured in seconds. This is two orders of magnitude faster. Traditional clustering technologies rely on having additional resources running and available for the failover. The failover is quicker than booting from a gold image, but the capital expenses are much higher. Just like performance, availability can always improve with a healthy injection of money. The virtual machine image approach does not have to rely on having spare or running servers dedicated to failover. Virtual machines can be created on running servers that are already running virtual machines.
But this is fairly straight forward. The good doctor would like to advance a new concept by lifting a few ideas from recovery-oriented computing's David Patterson who, in turn, lifted some ideas from George Candea3,4,5 . Here is a quote from 4:
Fine-grained partitioning enables bounded, partial restarts that recover a failed system faster than a full reboot.
Throw in the capabilities of server virtualization, and voila. The idea here is to re-think how applications and their components are grouped together on a server. When a single process fails by corrupting memory, we must re-initialize to get back to a known state. The server may be used to support many independent processes, but memory corruption cannot be isolated to the offending process. By using virtualization, we can create small-grained virtual servers that contain a process and its dependent processes - and no more. When there is a failure, that virtual server can be restarted (in seconds) without disrupting other independent processes. This is similar to the previous scenario where a single telephony component does not impact the global network.
The real issue here is to identify "fault trees" that would help us group together dependent components and processes and isolate them from others than can remain operational over a reboot. This is not a simple exercise and, if taken seriously, could change how we engineer applications in the future. But then, if we ever hope to achieve a future of utility computing, there need to be some bold steps taken.
This article has presented how Recovery-Oriented computing concepts, mixed with a little server virtualization, can be used in the ultimate recipe for utility computing. There are many other ingredients, but, hopefully, these ideas have got your noodle cooking in overtime. Perhaps you have some other ideas to share - after all, the good Doctor can't think of everything (but it appears I'm thinking about lunch). So, until next month, feel free to send me e-mails with your comments and suggestions at:
And remember - I'm not a Real Doctor, I'm a Virtual one....