Because we are a Church, we are considerably more cost conscious than a typical enterprise would be. In an effort to save power, cooling & hardware costs in our data center, we've begun implementing virtualization. We've been using virtualization for quite some time with our AIX servers, but we've now begun to virtualize Windows and Linux boxes.
Most of you know what virtualization is, but for those who don't, I'll try to explain.
<virtualization_explanation>
In the past, when a team needed a server in the data center, we would purchase them their own server. Their application might use, on average, 10% of the capabilities of that server. Or less. It was a waste, not just for our shop, but for the entire industry. A potential, but not always viable, solution was to load multiple solutions onto one box, but those solutions shared the application server and operating system and consequently could interfere with each other by crashing the operating system or the application server and/or just utilizing too much of the computer's brain (CPU).
Virtualization allows us to load multiple instances of an operating system on a single box. So on one machine, we could load several solutions, each with its own instance of the application server (Websphere) and the operating system (typically Linux). So if one solution crashes its instance of the OS, the other solution is just fine because it's running on top of its own instance of the OS.
And because each solution was previously only using around 10% of the total resources of the server, you can run, say, 5 solutions on one server and still only use around 50% of the server's resources. You just cut your power, cooling and hardware costs by roughly 80% (minus a little overhead for the virtualization technology itself).
Pretty amazing.
</virtualization_explanation>
We have seen significant, and maybe even extraordinary, savings on our "per server" costs as a result of virtualization.
The unintended consequence has been "server sprawl." The ease, speed and extremely low cost of creating a new server (because it is virtual) has increased the demand for servers. Without great governance and management tools, this is becoming a problem for most enterprises.
How are you dealing with "server sprawl" in your shops?
Across the street from you [literally] we have some of the same problems but on a much smaller scale. We have put some pretty strict criteria in place for creating a new "server instance". Once we explain to a user the criteria and the reason for it we usually receive little or no push-back. As a result of our policy we have really received another untended but positive result, we train and help potential users on why we need to keep our environments clean and uncluttered and we invest them in our governance. This would be harder in your enterprise but still a valid strategy.
ReplyDeleteI'm working on switching over all sysadmin tasks to a program like Puppet. It only works on *nix's for now, but tools like these can extend the influence of a single admin to manage the extra installs you get when you switch to VMs.
ReplyDeleteIf you're having a problem with virtual server sprawl, there were probably issues with physical server sprawl as well - they just weren't as visible. After all, even virtual servers consume resources so a sprawl there eventually leads to a need for more physical servers.
ReplyDeleteIn my experience, the problem normally results from not having enough good infrastructure resources (people) close enough to your end users (those requesting the servers).
If those end users are of the less technical type, they're not going to understand their needs that well, and may not know where to turn to get the proper talent allocated to their need so that their need can be properly defined - the also often love and appreciate help of this nature when it is made available to them. Doing so can help keep your server count down.
If those end users are of the more technical type, often they're programmers or some such that while they are solid technical people, don't always have the needed infrastructure skills to properly define their need. If they do have the skills, it's almost a certainty that they don't have the needed visibility into other projects to see how they can be more efficient: "Our design change is going to require a database, and we need a server to host it on." - "There's a better way, team X over here already has a DB server that's not being fully utilized - we can host your DB on that server."
In addition, if any of these people have their needs neglected or ignored long enough, they have an tendency to start hording resources whenever possible - kind of like squirrels storing nuts for winter. This isn't efficient, but if their needs aren't being met, they may not care about your needs (efficiency).
Then there always seem to be a few people somewhere who just don't properly track the resources they have been allocated or their use of them. Infrastructure people close to the problem can spot it, point it out, and push back to help these individuals clean up and organize their space a little better - resulting in better efficiency
All in all though, it still comes down to having good talented people on the ground close to the problem. At least that's been my experience across the isle from you.
The biggest problem with server sprawl is that management costs are linear with number of servers, virtual or not UNLESS you have some kind of automation. The best bet in that regard, as far as I'm concerned, is Puppet, IC Classify, and other infrastructure automation tools. Organizations figure out how to use these will reap all the benefits of virtualization without incurring the full expense of management.
ReplyDeleteYou are talking about server sprawl, but that is only an indicator of a larger problem at hand.
ReplyDeleteIn most organization the discussion that never takes place is the discussion of
how the software is going to be maintained, especially in the long run. If that is
never a consideration you will always see server sprawl, because it is easier to
treat it as a standalone artifact.
Or phrased a little bit differently, if you don't make a development group think
about the bigger picture they will treat their own requirements for resources the only requirements in existence and thus you as the manager would enable them
and send them down the road of server sprawl.
Most people rant and rave about virtualization and that term is typically an abbreviation from the most commonly known type of virtualization otherwise known as "hardware virtualization". With this type of virtualization, you are basically virtualizing the BIOS and thus are creating multiple guest instances of a hardware environment that happens to require an OS and all of its applications to run. While I think this form of virtualization is useful and deserves its de facto notion of 'virtualization', I think another form of virtualization is far more powerful and will be the future notion of the abbreviation; I am referring to Application Virtualization.
ReplyDeleteThe problem with Hardware Virtualization is that from an operational standpoint you still have just as much to maintain and some, such as myself, would argue that hardware virtualization makes it too easy to provision more 'machines' thus increasing the number to manage. With greater numbers comes greater complexity.
Application Virtualization, on the other hand, is where the application itself runs in a virtual process within the host OS (or even guest OS'es). This enables a scenario where you have one OS to manage/maintain but potentially infinite instances of applications running in virtual processes as if the application was dedicated to a single instance OS/machine. The exciting thing about this technology is that the provisioning is even simpler than that of its Hardware Virtualization uncle and that it provides a clear path to a multi-tenant architecture where homogenized computing resources can be used as a utility and it is the application that gets provisioned on demand when needed. This is also a much cleaner path towards backwards compatibility and a legacy application lifecycle strategy; if the application is hosted in a virtual process, that virtual hosting environment can make the application think it's running on whatever version of OS with whatever configuration, hardware, et cetera. The cool thing is, that legacy application can run side-by-side (SxS) with its more contemporary progeny indefinitely. I imagine this is something the church would be all over given its legacy applications and attempts to optimize fiscal resource. I can go on and on but I am sure you get the point...
Admittedly, I'm an AIX geek, so my opinion here is a bit biased.
ReplyDeleteMy organization supports over 1000 AIX operating system instances, close to another 5000 other UNIX-type OS instances, and over 15000 Wintel instances.
We have hardware virtualization throughout and are starting to roll in application virtualization.
This hasn't saved us any time in software/OS management, except that from a hardware perspective, it's easier to deploy a new logical partition. The main reason our clients like virtualization is for the hardware cost. "Sprawl" doesn't go away when you consider the OS and app instances -- just the hardware instances (but in many cases, formerly routine tasks like a firmware update on a Series P box are now more complex with the multiple OS images on the same hardware).
I'm not as bullish on automation tools are Phil is, unless you constrict them to a homogenous OS (particularly where IBM is concerned). What works great on Linux/Sun, etc. doesn't work well with HPUX, AIX, etc. [If the product documentation says "UNIX", I'm automatically turned off. It needs to explicitly say "AIX" for me to even consider it to be a viable option for managing AIX environments.] Of course, it's people who buy generic *IX management products who keep me employed, because someone's got to make it work with AIX...
I think there is a short term solution to server sprawl and a long term one.
ReplyDeleteShort term, you just need to start saying no, or if not no, ensure they have a solid business case for their server request, ask them why they really need one, and make sure they have a really good case. This will work for a while, but you run the risk of giving the business a bad message about IT
Longer term, I think something like SOA is the answer. Start looking at your application landscape and see if you can simplify it, into a more SOA environment. A smaller number strategic building blocks, that can flex up or down with the business, and can service all the needs of the business.
[...] met with VMWare this week. They own ninety-something percent of the virtualization market. They’ve now purchased SpringSource. SpringSource is all about simplifying Java [...]
ReplyDelete