I just spent about 4 days troubleshooting some disturbing behavior on a new SQL server 2000 installation. I realize SQL 2000 is pretty long in the tooth so I'm not exactly blowing the lid off a shocking story but I thought it's worth mentioning anyway. We have a client that is running on some older software and hardware; Websphere 4 and SQL 2000. We decided to upgrade their database server from an old IBM X series to a DL580. After we got everything installed I started to run some load tests and found downright terrible performance; pages that took 3 seconds on the old database server were taking up to 40 seconds on the new one.
Actually, when we initially began testing the server wasn't even completing the tests. It would just stop and hang out with no SQL being issued and no threads in a running state. Since we were going from a quad processor machine to a quad proc, quad core machine we started messing around with parallelism settings, which allowed us to complete our initial tests. SQL 2000 is not particularly smart about using parallelism, so we theorized that the overhead associated with running a parallel querey on 16 processors was causing more problems than it was fixing. We found our best results with 4 processors used in parallelism, although the times were still abysmally bad.
As we approached the end of our ropes we started looking at differences between the servers a little more closely. The main differences were an OS upgrade to Windows 2003 Server, and the differences in hardware. We also noticed that when SQL was installed, it was upgraded to SP4, whereas the existing server was on SP3 (with the important security updates, of course). We removed SQL and reinstalled it with SP3 and the performance was night and day. Now the pages were loading in under 5 seconds every time, which probably means we are not running into parallelism issues any longer, either.
We just did this last night around 10:00pm so I haven't really dug into why this occured, but if we find out exactly what about SP4 caused us problems I'll post an update. Also, we noticed a pretty big performance detriment after installing the post-SP3 patches but nowhere near the difference between SP3 and SP4.
Thursday, September 4, 2008
Friday, August 29, 2008
VMWare ESX Server Templates on the Cheap
When it came time to expand our internal server hosting beyond what the air conditioning and power in our small server room could support, VMWare ESX Server was a relatively easy sell. We are currently running 3 DL580 servers with about 24GB of RAM each and as many processors as we can jam in them. This solution is hosting roughly 65 environments, mostly quality assurance and user acceptance testing Java environments running WebSphere 5.1.1.8 and IIS. Generally they run happily, even though we don't constrain the CPU and memory usage of the VMs.
The main reason we don't restrict resources is because we have no central management of our VMs since we didn't purchase virtual center. We would want to use resource pools to allow our various teams of developers to allocate their allotted resources as appropriate for where they are in their release cycle. For example, I would create a resource pool for "ClientX", and then allow ClientX to determine if they want their QA environment to be allocated more hardware resources because they don't have an application in UAT at the moment, while still ensuring ClientY has their own resources and are not affected by the activities of ClientX. That's all great on paper, but when ClientX is spread out with environments on two or three different servers it really detracts from your ability to allocate accordingly without a way to centralize the resources.
Getting back to the topic at hand, not having VirtualCenter also prevents us from using templates. A large part of the appeal of VMWare is that it speeds up server deployments; what used to take us weeks of acquiring hardware, building, and racking now takes hours. However, without templates we would find ourselves building each OS from scratch, and that's lame.
What we did was build a VM called "2k3_template" on a SAN drive, installed our OS, patched it, installed our all standard software and tweaks, and then ran sysprep. Once it shut down, I removed it from the inventory but left the files. Now when I need a new server I ssh into the VMWare server, make an empty folder on the san that corresponds to the new server name, and run this:
vmkfstools -i /vmfs/volumes/san_volume/name_of_template/name_of_template.vmdk /vmfs/volumes/san_volume/new_server_name/newservername.vmdk
That will create a copy of the template called newservername. Then, you go into the Virtual Infrastructure console and create a VM, making sure to select a "custom" VM named the same thing as you named it from the command line. When it comes time to create a new VMDK or use an existing one, use the one you created with vmkfstools. Power it on, go through mini setup, and you have a new server ready to go.
The main reason we don't restrict resources is because we have no central management of our VMs since we didn't purchase virtual center. We would want to use resource pools to allow our various teams of developers to allocate their allotted resources as appropriate for where they are in their release cycle. For example, I would create a resource pool for "ClientX", and then allow ClientX to determine if they want their QA environment to be allocated more hardware resources because they don't have an application in UAT at the moment, while still ensuring ClientY has their own resources and are not affected by the activities of ClientX. That's all great on paper, but when ClientX is spread out with environments on two or three different servers it really detracts from your ability to allocate accordingly without a way to centralize the resources.
Getting back to the topic at hand, not having VirtualCenter also prevents us from using templates. A large part of the appeal of VMWare is that it speeds up server deployments; what used to take us weeks of acquiring hardware, building, and racking now takes hours. However, without templates we would find ourselves building each OS from scratch, and that's lame.
What we did was build a VM called "2k3_template" on a SAN drive, installed our OS, patched it, installed our all standard software and tweaks, and then ran sysprep. Once it shut down, I removed it from the inventory but left the files. Now when I need a new server I ssh into the VMWare server, make an empty folder on the san that corresponds to the new server name, and run this:
vmkfstools -i /vmfs/volumes/san_volume/name_of_template/name_of_template.vmdk /vmfs/volumes/san_volume/new_server_name/newservername.vmdk
That will create a copy of the template called newservername. Then, you go into the Virtual Infrastructure console and create a VM, making sure to select a "custom" VM named the same thing as you named it from the command line. When it comes time to create a new VMDK or use an existing one, use the one you created with vmkfstools. Power it on, go through mini setup, and you have a new server ready to go.
Subscribe to:
Posts (Atom)