Capacity Concerns in a SaaS and Cloud World

February, 2010
by Ron Kaminski

About the Author

Ron Kaminski, Safeway Inc.

Ron has been a capacity planner and performance analyst since the mid 1980s, on probably every platform you can name besides a mainframe. A dedicated workload characterization junkie, Ron enjoys using multiple vendor and "home-grown" tools to collect, reduce, analyze, display and manage large-scale performance and capacity planning, as well as sharing ideas with fellow capacity planners and performance analysts.

Capacity Concerns in a SaaS and Cloud World

By Ron Kaminski

Fellow capacity planners, if you haven't already heard it, you probably will hear one or all of these statements in a meeting soon:

"We found that we can deploy a new _____ system in only three months worldwide using a Software as a Service (hereafter SaaS) provider."
"We don't need to buy any IT infrastructure for this new venture; we'll just put this in a cloud somewhere and use their resources."
"We need to give people more access to their personal e-mail and web services from within the corporate network, and it had better be fast."

It will be said with confidence by a senior person in the firm and there will be much nodding and positive grunting. Some IT staff will start wondering how long until they are eliminated, and they will hurry home to start a job search.

However, SaaS, clouds and wider web access may have an Achilles heel.

As capacity planners, we need to plan for SaaS and cloud techniques, and the inevitability of people demanding ubiquitous web access to their private data. These can be great tools to make our businesses more agile, profitable and serve our customers better. Notice that I said "can be" as these also can be disasters, and it turns out that capacity issues will become paramount in which way many of these go. It is our duty to inform our management of what may actually happen. Let's try an example:

Your global firm has decided to go with a new SaaS personnel system and the vendor assures them that it will deploy world-wide in 3 months. Managers knowledgeable about the costs and time frames to develop a new application internally start to salivate, seeing potential savings and assuming that little or no investment will be required "in house", thinking that all that the potential users need is a web browser. Everyone in management and business literature seems to be so positive about this trend, what could go wrong?

Well, we capacity planners need to inject some patient reality into the mix. Let's do some comparisons, based on the following assumptions drawn from my personal observances, your mileage may vary:

The average corporate windows or *nix server running discrete applications uses between 4 and 11% of their CPU resources, barely half of their installed memory, and many probably suffer I/O wait because the applications are connected to cheap RAID 5 or slow huge locally attached drives that were purchased by someone who thought that "cheapest cost per megabyte" was the driving concern.
90% of the time, the peak usage of the week is not the application; it is more likely some maintenance activity like backups or antivirus/security scanning. The business application maybe uses 3%-5% of the machine at peak. A particularly awful recent development is the arrival of the "Indexing services" that some O/S vendors offer as a convenience to their users. If the user clicks on 'Yes, I would like future searches to go faster" on one of your servers, be prepared to lose 3.09% of a CPU to that, 24 hours a day!
The firm's internal network is reasonably fast, but the firewalls and such that you need to go out of the firm have "issues" at times. Note that these issues do not impact the in-house applications.
Many applications are "chatty" when interacting with their users meaning that the applications send and receive many small packets of data when in use. This is tolerable as network latencies based on distance are small; the server is right in the same building.
Most internal applications do not bother to encrypt data, as nobody can get a machine on the corporate network without a huge (but safe) process.
Many desktops are being replaced by cheap monitors and keyboards that run connected to some distant virtualized server, so local desktop machines are disappearing.
New web and bandwidth intensive applications are appearing (and are demanded) with ever increasing frequency. How many of you are integrating iPhones and other advanced personal devices into your networks, mail systems and calendars?
Backups and data just keep growing. We all talk about the data lifecycle, but honestly, how many of you have heard a manager say, "Well, disk is so cheap, we'll add disk now and work on a delete strategy later" and later never comes.

Given that basis, what needs to change in our firms to cope with a SaaS world? The actual software doing the work functions the same, and uses the same resources, right?

It turns out, quite a bit will change, and it is all about capacities.

That new personnel system is full of all sorts of information that evil doers will want to get, so encryption/decryption will be required at both ends. We aren't concerned much about the technical aspects of this, safe technologies exist, but what resources do these methods cost to run? Figure on 3% to 5% of the machine's resources will be needed for encrypt/decrypt duties.

These machines will be accessing the World Wide Web, so the machines will need security scanners and antivirus products, and need them active during prime use business hours, figure on 4%-12% of the machine's resources.

The SAAS user's web browser is unlikely to be a simple display; there could be java applets, or other web experience enhancement technologies that add the sizzle that make potential users buy it, and those consume resources as well, both bandwidth and CPU. Figure on 7% to 20%.

As much as we try to avoid it in these green times, there will likely often be printing locally, which means printer drivers and the like. As grim experience from remote offices has taught me, the way mis-configured printing issues usually show up is looping spoolsv processes on windows machines, but let's think positively, nobody in our new SaaS world will ever make a printing setup mistake or leave a printer configured that no longer exists (your author looks up nervously to dodge a lightning bolt) and say that printing will take 1-3% of the machine.

Backups are an interesting part of the puzzle. True, the vendor will be doing them (and storing your presumably encrypted data, because you did negotiate that in the contract, didn't you?) at their site, but again, you are going to have to endure at least the encryption overhead. Also, smart vendors will charge by the terabyte and add a profit, so your firm's sloth will become their revenue. Figure on another 15%-30% for all that encryption during off-peak backup and maintenance times.

Network issues may also be lurking. How many of you know the current capacity and looming demands of your...

Network bandwidth to the internet?
Firewall appliances (many of which the vendor does not allow you to run capacity collectors on)?
Inter-building, or metro network segments?
Office building networks, now coping with fat graphical web applications instead of tiny local data packets?

At some point, and depending on your tools and planning, it might be quite soon, network spending might be needed for success. In these modern times, chargeback is viewed as a weird retrograde artifact from the 60's mainframe world, unneeded in the cheap hardware current world. But, when the network slows to a crawl from the 4^th SaaS application, who gets to pay for the new network items? Also, remember all of those other internal chatty applications? When several of the applications slow due to increased network use, who gets the call?

The old rule is true, "Whenever you add complexity, you add staff, either with forethought or hindsight". Does anyone think that the SaaS, Cloud or the web accessible world will be less complex?

So, we went from having 4% to 14.09% utilized internal firm servers with low network latency and little overhead for security to SaaS servers that will be on 15% to 43.09% used machines with potentially high network latency for that and possibly other internal applications, sluggish I/Os due to encrypt/decrypt overhead, and escalating storage charges. Another way to look at it is - if your firm ran the software internally, in a virtualization cloud, you could run on roughly ¼ the hardware that running the SaaS solution will demand. Try to find a SaaS vendor that highlights the true costs that lie ahead.

Oh by the way, you may end up with multiple SaaS and/or cloud applications. What are the chances that every vendor will all use the exact same security and encryption methods and versions? If some of the vendors use different methods, how much more load will you be enduring to encrypt, decrypt and re-encrypt and re-decrypt...?

In all but extremely optimistic scenarios, SaaS and cloud applications will probably take more IT resources to run than it would to run it in house, and remember, the vendors will price it to run at a profit, not just at cost. Check my numbers with your firm's and see what it tells you. Then tell your management, quick.

Summary

Clearly change is upon us, inevitable in the face of all of that trendy marketing hype. While we used to worry mostly about "on machine" issues in the distributed systems space, (like CPU use, CPU wait, I/O use, I/O wait) the future capacity issue is the network (bandwidth, latency) and the effects of all of that new security that will be needed to traverse the dangerous, hacker-filled web as your sensitive data goes to and from the SaaS or cloud vendors and personal e-services. The network is the computer, or at least doing the computing over the network will likely take most of the computer.

If there will be SaaS or Clouds at your firm, be the first mover who claims the savings (while absorbing any current network surplus capacity) before the bill comes due on later applications! Consider investing in network tool companies and/or firms that realize the true costs and offer to run SaaS on your firm's hardware. They are going to have some great sales years in the decade ahead.