How to build a Quad-Core Xeon VMware ESXi Server for less than an ‘82 Civic!
Not long ago I was in need of a decent Virtual Lab machine and my only budget was whatever I could get from selling off some old computer / networking equipment that was just taking up shelf space in the laundry room and seriously annoying my wife. The really tricky part was also meeting the following criteria:
- 4 Cores, 6-8GB of RAM, 500GB+ Storage
- Meets the NAZI-like requirements of the ESXi HCL (unofficially if not officially)
- Has BIOS enabled virtualization support (for Citrix XenServer)
- 64-bit processor(s)
- Name brand for easy to find replacement parts (Dell, HP, IBM)
- Cheap enough not to piss off my wife (ie: not a damn dime over what I got from selling my old equipment!)
I already had two 500GB SATA drives sitting around from an old server pull so I mostly just needed a decent chassis with CPU and RAM. Sadly, even building a decent desktop for this ranged anywhere from $400 to $800 since I wanted to go Quad-Core with at least 8GB of RAM. Then I stumbled upon some REALLY cheap PowerEdge 860’s on eBay (yes, I know… eBay…). So, I did my homework and found that although it was LGA775 it WOULD accept the Quad-Core Xeon models X3210, X3220, and X3230 as well as 8GB of RAM. Also, since it takes unbuffered ECC RAM it’s a lot cheaper than FB-DIMMs. So for the price of a cheapo desktop I could have a “real” server. The PE860 was going for $249obo so I told the seller that I didn’t need the two 250GB HDDs and the 2GB of RAM and would offer $200 for just the Chassis with the processor (since I needed the heat sink) and he accepted. I then purchased an X3220, RAM and Slim IDE DVD-ROM from Newegg. I ran into an issue with the DVD-ROM where I had purchased a slim IDE drive but didn’t have an IDE to Slim adapter and had to harvest one out of another old server I had lying around. Because I already had 2 HDDs my total cost was $515 which was only $15 more than selling off the Cisco ASA5505 and two old servers (just don’t tell the wife). So, if you’re looking for a small (1U) super-low-budget 64bit VMware server that actually performs, here is a detailed list of components:
| Component: | Purchase Location: | Cost: |
| Server Chassis – Dell PowerEdge 860 | eBay | $200 – $250 |
| CPU – Intel Xeon X3220 Quad-Core 2.4GHz | Newegg – Link | $200 |
| RAM – 4 x 2GB Unbuffered ECC PC2-5300 | Newegg – Link | $116 |
| HDD – 1TB SATA | Newegg | $79 – $99 |
| CD/DVD – Slim SATA DVD-RW (if your server doesn’t have one) | Newegg | $49 |
| Total | $595 – $710 |
Additional items of note:
- Don’t let the CD/DVD ROM catch you off guard! Most of the cheaper 860’s available DON’T come with one so you have 3 options:
- Find a PE 860 with a CD-Rom
- Use one of the two onboard SATA connectors to install a slim SATA DVD-RO
- Buy an IDE slim drive and find a slim to 3.5″ IDE adapter board (I did this and was a PITA since they’re usually slim to 2.5″ adapters
- Try to negotiate with the seller on the PE 860, I was able to get mine for only $200 by telling the seller I didn’t need the RAM and two 250GB HDD’s it came with (original price was $250 with a Pentium D processor)
- I SPECIFICALLY mention the Xeon X3220 because you’re limited to LGA775 at 1066MHZ BUS ONLY!!! Therefore, the best bang for the buck comes out to be the X3220 since the X3210 is the same price and the X3230 is $299
- The performance bottleneck in this server is definitely the HDD, if you have a slightly higher budget you can find Dell SAS5i/R cards for $50-$100 which will give you RAID 1/0 and let you use faster SAS drives instead of SATA
- The 860 was available with 2 different riser cards (make sure you know which one you’re getting!):
- 2 PCIe 8x slots
- 1 PCIe 8x slot and 1 PCIx slo
***This is good to know because if you are adding peripherals the PCIx versions are sometimes cheaper***
- Although all the onboard hardware WILL work with both ESXi 4.0 and Citrix XenServer 5.5, the onboard SATA controller is not compatible with ESXi 3.5 or 3.0.
- If anyone wants me to test out ESX (Not ESXi) version 3.0, 3.5, or 4.0 let me know.
Greetings,
Thanks for the article! I recently acquired a Poweredge 860, and have been trying to decide which virtualization platform I want to run on it.
I was wondering if you know which processor originally came with the PE 860 that you got, and if you have had any issues with overheating with the quad-core. Dell did (does?) offer the 860 with the X3220, so I know the airflow engineering will support it, but I’m curious if the heatsink I got with mine (on the Pentium D 925) will be sufficient if I throw it on the X3220.
Your insight is appreciated!
I found that both ESXi 4.0 and Citrix XenServer 5.5 both work quite well on the 860 (just make sure CPU support for Vitualization is turned on in the BIOS). If you go with Hyper-V or Xen be sure and let me know how it worked out
Mine came with the same Pentium D 925 and I haven’t run into any cooling issues. I just cleaned off the old heatsink, picked up some Artic Silver from Newegg and applied it thin and evenly. I’m tempted to run a torture on it and report back with temps… check back tonight and hopefully I’ll have some real numbers for you.
Thanks for the update. I’d be curious to see some hard numbers on the X3220, if you have time whilst juggling parental responsibilities and dodging evil Legos. If you do run some stres tests, I’m also curious about fan speed/noise. Right now, my 860 runs pretty quietly (for a server), although it occasionally boots up with a short reminder that it’s capable of sound levels somewhere between a shop vac and a DIY hovercraft. I’m probably a little too excited about quad-core performance, but if it means the fans are going to be running at speeds capable of supplementing my HVAC system, I might have to pass.
I think I’m going to run with ESXi for now, since I’m more familiar with it, but I may very well get bored and try out some of the others. If I have anything useful to report, I’ll definitely let you know.
Although the CPU temp software wouldn’t work on the VM’s so I wasn’t able to get exact CPU temps I was able to get this data via ESXi hardware health status along with a highly tuned temperature instrument known as my right hand:
Before torture test with no VM’s running:
All items green
Both fans running at 3000rpm
System board temperature at 27C
So, after 3 hours and 34 minutes off 100% CPU usage (2 VM’s with 4 cores each running prime95):
All items Green
Both fans running at 3000rpm
System board temperature at 28C
Note: This is in a rack in a server room at 70F. It was still blowing out less heat than the 3 Dell 2950’s below it that were barely doing anything.
Also, since I couldn’t get an exact CPU temp I thought I’d make you feel better by resting my face on the top of the case right above the CPU heatsink and… OUCH!!! just kidding
It felt the same as the rest of the case. So, although the test was about as scientific as duct taping a muffler together I personally feel confident that the heatsink is more than adequate.
Thanks for the additional time, effort, and info! Sounds like I need to bite the bullet and copy your setup. Of course, now you just cost me more money…. =)
Keep up the good work on the informative articles!
It’s the least I could do since you’re the first “real” commenter on my new blog
. I have some cool ideas for upcoming articles involving VMware, PowerShell, Open-Source SMB Solutions, etc. Check in from time to time, I’m all for comments/suggestions/requests.
Well, I thought I’d post back with my update on cpu temps, since your evil powers forced me to send a couple hundred more dollars to NewEgg. =)
After installing my new X3220 (stepping G0) and 8 Gig-o-rams, I went ahead and threw Windows Server 2003 on for a bit so I could run some stress testing utilities and temperature monitors on the cpu. Here, roughly, are my results:
Load Software: Prime95 (Pegs all 4 cores for as long as you want, basically.)
Temp Monitor: RealTemp 3.00 & CPUTempWatch 1.0.9
Temps started in the mid-to-low 30’s celsius, and for the first 20 minutes or so, stayed below mid-50’s. For the next 20 minutes or so, the temps bumped up to 65 celsius a couple of times, but would drop back down to 60-63 the rest of the time. I ran Prime95 for about 45 minutes, and the fans never sped up (I don’t *think* I would need the Dell utilities for Windows to control the fans, but I dunno).
Granted, low 60’s is plenty hot for sustained usage, but from what I’m seeing of the G0 stepping version of the X3220, it’s still a solid buffer from this processor’s “danger zone,” and I really doubt I’m going to be pushing 4 cores at 100% with my home virtualization projects. Not consistently and for an extended period of time, anyway.
Also of note is that within literally a second or two of stopping Prime95, the temps on all cores had dropped about 10 degrees, and were back to mid-to-low 30’s within 3 or 4 minutes. This is all with the same stock fans and heatsink that were on the 925, just like your setup.
In summary: you put together a solid little virtualization platform, and thanks for the recommendation; I’m glad I copied it!
(Thanks also for letting me bug you about the temps and my rambling here that may be more information than you care to have on this thread, but perhaps some other folks will find it helpful) =)
MUA HA HA HA!!! If only I could get Newegg to read your comment…
You’re very welcome! I’m glad it worked out.
As far as the fans go though I have run into an issue where the RPM’s never changed (it was on a PowerEdge 2950 III) and a BMC firmware update fixed the issue. As far as I know the BMC controls the fans by temperature and I wouldn’t think that OpenManage would be required but I’ve seen stranger dependencies. Might be worth installing if you have a copy lying around though just to find out
I have OpenManage running on all of our production machines (Windows, Linux, and ESX). On a related side note, I did get an ISO of ESXi 3.5 U4 from Dell once that supposedly had OpenManage embedded but it gave me errors during install and i never spent any more time on it.
As far as all the information you posted, I couldn’t be happier. The more anyone has to add to any post of mine here the better. Like you said, it could come in very handy for someone else, so thank YOU
VMWare does have a “Dell customized” version of ESXi 4 for download. I’ll give that a shot and see if I spot any differences vs. the standard build that I had on there before the hardware upgrade. If I were going to run Windows natively on this box, I’d definitely prove to myself that the fan controls work, but I’m satisfied with the cooling as-is, so I’ll forge ahead with ESXi at this point.
I also tracked down a couple of things that may allow for more detailed monitoring of the host hardware than what the vsphere client gives you; I’ll continue stretching this thread if I find anything interesting to report on that front.
Sounds good, keep it coming
Apparently Dell pulled the ESXi 4 link due to “Issue with Release”. However, I did download the standalone OpenManage ESXi 4 update but haven’t got it working yet. Actual “Work” keeps getting in the way…
Damn. . .i could have really used that Dell custom ESXi 4 download! I found this blog because i got 3 860s in the mail today and am ready to take the plunge!
My question is: I’ve got 4G of ram in each, stock Pentium D 925 Dual Core 3.0GHz CPU and am running a 250G and a 1T SATA drive in each. What kind of performance are you guys getting out of yours? How many VMs? I’ll mostly be serving up LAMP servers with centos or ubuntu. Perhaps a Windows 2003 server as well on one of them.
VMWare’s download page for ESXi still has the Dell customized binary in the “Optional Binaries” page. I haven’t had a chance to play with mine for the past week-ish, so I’m still not sure what additional/different functionality one gets out of this build. I did notice that the vsphere client gave me the asset tag number of the Dell, which I think was something new vs the non-customized build (same vsphere client, though).
Regarding performance, my un-helpful advice would be to load them up and see how it runs. Fire up some VM’s, keep an eye on your CPU and RAM, and then decide if you want to keep adding more VM’s to that box. (Told you that wouldn’t be very helpful). I was told by a friend that his user-grade hard drives died rapidly in his esxi box, so using server-grade drives and/or bumping up your ram could help prevent that. More ram is always good with virtualization, anyway… =)
Drives died fast. . .doesn’t sound promising. Not sure what the 250G are (OEM to the box) but the 1T drives are Western Digital Caviar Black drives. . .~$100 each.
I was looking for 1TB hard drives for another box at home, and I noticed way too many newegg reviews complaining about longevity of consumer-grade terabyte drives (unfortunately, there seem to be lots of complaints there about those caviar black drives, too…). Seems like the technology isn’t “fully baked” in that area yet. Western Digital and Samsung both have server grade drives available for a slight premium (or a huge premium if you order through Dell)–I ended up going with a Samsung “Raid class” drive that has a 7-year warranty. I’ll see how it goes…
James: First of all, Aaron is dead on about the RAM, just make sure you get the right RAM! Maxing it out at 8GB will definitely give you the best bang for the buck. Second, although the newer Core2’s and Xeon’s will be NOTICEABLY faster than that Pentium D, the real bottleneck here is HDD access and speed. Although it wasn’t bad running 5 VM’s that weren’t very active I noticed that boot times were a little slower and VM Conversions or moves take significantly longer than over iSCSI or on local 15k SAS drives. This can be helped by using a RAID controller (the SAS 5iR was an option on the 860) which will be a little faster even with SATA drives because of the cache on the controller as well as the option for RAID 0 if you really wanted to squeeze out as much performance as you can. So, as far as your LAMP servers go you can probably run 5+ light web servers but if you’re doing anything that’s heavy on the MySQL side you’re going to see some serious performance hits. Unfortunately, this is one of those things that would take FAR longer to predict and spec out than it would be to just create 10-20 VM’s and starting turning them on until you find your ceiling. A perfect real world example would be one of my clients. 2 2950’s, 8cores, 32GB RAM, 15k SAS drives. Engineering server runs 5 VM’s and stays between 50-80% CPU usage at all times. Product Test server has 57 (yes, 57) web/app servers running and pretty much stays at 60-80% CPU usage at all times. What those servers do has a DRASTIC outcome on VM performance. Hope this really long-winded reply at least helps a little
Aaron: nice! I’ll have to take a peek.
A general note on user-grade drives in servers. My original intent was to get around this by installing a used SAS5iR card and running RAID 1 but I got a bad pull and never got around to finding a replacement. With server class SATA drives I’ve had mixed luck and honestly the best thing about them is their warranty and fast replacement. This budget setup probably shouldn’t be used in production but would be perfect for testing, home certification lab, etc.
Hi all,
First thank you for posting this. I’m about to buy several of these, to setup a small cluster. Since I’m a complete nOOb to this, and you seem to know what you are doing. I figured I’d post and see what you thought. I would like to run Sun’s Virtualbox using Ubuntu 9.04 on these. One box will be used to serve about five to ten web sites, using either Apache 2.x or Cherokee, and of course the rest (PHP + Zend Framework, PERL, Tomcat, etc. etc.), One box is going to be dedicated to MySQL, and the last server will be dedicated to FFMpeg/FFServer, Adobe FMIS or Red5 or Wowza or CRTMP. This is for a startup, with zero traffic. I’m hoping to handle at least 1000 simultaneous connections, and remove some of the burden by using CDN from Cache Fly (scripts, media, and static content) and UVault (Flash Media Interactive Server). I plan to upgrade all of the servers with quad core processors and eight gigs of ram, except for the web server which I beleieve can run fairly well with just 4 gigs and the 925 processor. Since I believe only two of the five to ten web sites will get the majority of the traffic.
I’m thinking I will equip most of the systems with one WD VelociRaptor 300 GB SATA Hard Drive and one 1 TB Samsung HDD. I’ve also contemplated using either SSD or CF storage, but have shyed away because of some cost issues and un-familiarity.
Any insights you could give me would be greatly appreciated.
Jon,
You’re very welcome! I’m glad it has been helpful. Providing that Ubuntu/VirtualBox doesn’t have any issues with the 860 hardware, I would imagine it’ll be a nice setup. If you have the budget for REAL SSDs storage don’t bother with this setup
. Although I can’t say for sure obviously, I would imagine that the setup you describe should be able to handle your situation nicely. Since yours will be a production environment just make sure you have a decent backup solution thrown into the mix (Amanda and Bacula come to mind for this setup). If you have the budget, it might not be a bad idea to buy a spare of each type of drive to minimize downtime as well. Normally I don’t condone any kind of NON-RAID setup for ANY production server but you can get around it with the above as long as a few hours downtime isn’t an issue. Otherwise, go with the SAS 5iR and run RAID 1.
Maybe I’ll order a new SAS 5iR card and get that going this week if I can sneak it past the wife
If you’re waiting to see how Ubuntu and VirtualBox run before you purchase, my 860 will be available again for testing mid next week sometime. Let me know and I’ll set it up.
If you go ahead with the setup I’d love to hear how it went. Good luck!
Hi, thaqnks for the quick reply. I checked http://webapps.ubuntu.com/certification/hardware/200712-194/ for hardware compatability with Ubuntu 9.04. Not sure about VirtualBox, guess I will have to try. Hopefully it will work. I have no experience with SAS. Can you recommend drives/controller and a ball park figure on what it will cost? I found this, http://www.directron.com/mg129.html is this correct? Seems like a good price
Amanda/Bacula or RSYNC for off line storage is planned, thanks for the reminder tho….
Again, thank you so much for sharing. I really appreciate it.
Hopefully the sellers call me back soon on these systems I want to purchase. So that I can get started early next week.
Jon,
No problem, actually I’m sitting in a server room waiting for a long file copy to finish after what might be the mother of all vmware issues (definitely a post coming up about it soon).
That’s the card, might want to contact Dell to verify part number just in case there are version variations (I know there is more than one part number for this card even though they look the same). Also, keep in mind that cheap price is because it’s used and that’s the small RAID card Dell offers that doesn’t have the 256MB memory module and battery, also only does RAID0 and RAID1. It’s perfect for a server with only two drives.
Although 15000rpm SAS drives rock! They’re very expensive. However, SAS controllers (at elast the Dell ones) will support SATA drives! So, using a SAS5iR controller you could use two SATA drives and have the redundancy of RAID1
EvilEmuofDoom,
Thanks again. I will check it out with Dell. I’ve gotten a hold of the person selling these so I will be picking them up an a couple of hours. Do you have some memory recommendations, brand, part #, etc. that you perfer? Also, Can I use the Q6600 in these, or will they run to hot?
I don’t want to take advantage of you. You have been very, very helpfull.
Thanks!
Not at all! This is EXACTLY why I created this blog. To share some of my experiences and the things I think are cool and get feedback from others that may try it or even have a better way than I’ve found. I’m really looking forward to hearing how this goes for you.
The Q6600 and X3220 are very close and the only real difference I can find is their thermal rating. Supposedly the X3220 can operate at higher temps (85C) while the Q6600 is rated a little lower (71C). There are a ton of online arguments over these two processors with no clear outcome. Either should work fine although the X3220 MIGHT be a little safer bet in case you don’t have good A/C where these are going. If you want my personal opinion if I were in your shoes… Unless I could find the Q6600 for $40-50 cheaper I’d probably go with the X3220. Then again, I can get a little paranoid about these things
Although you can use plain DDR2 667 PC2-5300 ram you really SHOULD use “Unbuffered” ECC PC2-5300. I linked the exact memory at Newegg in the original post.
Wow.. .for $56 i’d splurge for that RAID card if it really would support 2 SATA drives. Can’t seem to tell where you’d plug the sata drives into it though?
My “redundancy” on these machines right now IS vmware. I’m running it mostly so my machines will be platform independent and NOT so I can get a million machines. I’m replacing 4 machines with 3 of these 860s right now. One a buddy of mine will control and two of them are “mine.” I’ll keep copies of the VMs off and ready to go. Should one machine go, i’ll have copies of its VMs on the other machine that I can bring up temporarily while it’s being repaired.
Do you have any thoughts on backing up VMs snapshot style on ESXi? Looks like I can take the machine down and make copies of the files, but it’d be nice to be able to do some kind of ‘hot’ snapshot backup.
-James
There is a single large connector on the top of the card that requires a cable like this: http://www.newegg.com/Product/Product.aspx?Item=N82E16812198020
Dell has a 2-connector cable but I couldn’t find one for a decent price and just purchased this one. BTW, the SAS5iR actually supports 4 drives but will only do RAID sets by 2 (2 separate RAID0 sets or 2 separate RAID1 sets).
You know, I never really looked into a bare-metal backup solution for ESXi. It might be beyond the scope of ESXi since it doesn’t have the full linux backbone but it’s definitely a very interesting question. If you didn’t mind the 30-45min downtime you could always use the VMware standalone converter to copy between ESXi servers. We have only two production machines running ESXi (instead of ESX) and they have Windows VM’s being backed up by Backup Exec. I’m curious now… this one will take some thought
Hi,
I picked up the servers on Friday. I thought I had found a solution to the CD-ROM drives but it turned out not to work. What CD-ROM drive did you buy? In your list it says SATA, but in your description it says IDE with a converter board. I’m stuck, please advise.
Thanks!
I bought an IDE DVD-RW before I realized an adapter would be needed. I went through three out-of-production servers before finding on that fit correctly. I had found a good link to these adapters and meant to include it in the article. I quick search on google found this: http://www.cables4computer.com/Cabling_Products/IDE-Cables_Internal-PC-Cables_Slimline_CD-Rom_Drive_to_IDE_Adapter.html although I know I’ve seen them cheaper. The key is to make sure it is 3.5″ IDE and NOT 2.5″. The easiest way to tell them apart is the power. 2.5″ IDE carries data and power while 3.5″ IDE requires a separate power cable (look for the 4 pin power connector).
The “easy” way around it is to just use a SATA CD-ROM but then you’ll either be limited to one SATA HDD or need to use the RAID controller. Another thing to note is that if you use the RAID controller you’ll need a SATA power y-cable since there are only 2 SATA power connections.
If you’re going the scavenging route like I did you’ll have better luck with really old servers since newer ones tend to use 2.5″ IDE for CD-ROMs. I believe I found my adapter in P3 HP Netserver.
Hope this helps. Sorry for the lack of info on this before.
Great thread/blog going here.
I went through the hardware decision about 6 months ago. However, I chose the Dell PowerEdge 1800 because it’s easier to expand and can either be rack mounted for floor standing. Supports 12gigs, has the CERC SATA 1.5/6ch controller (including all six expensive Dell cables). And they seem to be selling on eBay for 200-300 regularly. I picked mine up for ~$250 with dual-core Xeon 3.2Ghz procs, 4gb mem and 2 x 250GB. Which later was upgraded to 12GB and 3 x 1.0TB for VMware readiness.
Hardware wise; Works great and is on almost every HCL guide for anything. All in all makes my life easier in the end.
Anyhow, I’m attempting to installed ESXi v4.0 but doing some research first and found this blog.
I did install ESXi 4.0 however, didn’t realize there was a Dell specific for this download at the time (had it for 3.5) so I’m going to re-install with the Dell specific 4.0 install CD tonight (if the download doesn’t take forever…) because I’m going to have to adjust my RAID configuration but without OSMA I have no way to do it non-destructively.
Ok, sorry, didn’t actually provide anything useful here but felt compelled to say thanks for the info! Hopefully I can add my experiences to the collective.
Image of ESXi v4.0.0 Build 181792 on Dell’s website, just ran across it looking for other documentation.
http://support.us.dell.com/support/downloads/download.aspx?c=us&l=en&s=gen&releaseid=R232776&formatcnt=1&libid=0&fileid=335587
However… Major note regarding OMSA!!!!
“These images do not include Dell OpenManage Server Administrator as part of the ESXi image. For detailed instructions to install and use Dell OpenManage Server Administrator, please refer to the Dell OpenManage support documents at http://www.support.dell.com/”
Doh! That’s what I was hoping it had!
Zoobie,
Thanks for the great comment! Actually, I think your reply could be VERY helpful. An alternate server choice is always a good thing and if anyone has any more to add I would highly encourage it. Especially other brands as I deal almost exclusively in Dell servers these days. I had thought the 1800 wasn’t 64-bit but a quick search just now has proved otherwise. Can you do me a favor and check the BIOS to see if it has the option for Virtualization Support (or similar, can’t remember the exact wording). I THINK that CPU/BIOS is required for running 64-bit VM’s but confirmation either way would be very useful especially since the 1800 supports 12GB of RAM.
I downloaded an install for OMSA last week that’s supposedly for ESXi but didn’t work. Now I’m wondering if it requires the ESXi build you posted. It can be found here: http://support.us.dell.com/support/downloads/download.aspx?c=us&cs=555&l=en&s=biz&releaseid=R227503&formatcnt=1&libid=0&fileid=326356
I wish my 860 wasn’t in use right now so I could test it out… This might get really interesting
Quick update. I just read the compatibility list on the ESXi OMSA download and the 860 isn’t listed. I might try it anyway to see if it gives at least partial functionality though.
FYI, here’s the list:
PowerEdge 1950
PowerEdge 2900
PowerEdge 2950
PowerEdge 2970
PowerEdge M600
PowerEdge M605
PowerEdge M805
PowerEdge M905
PowerEdge R200
PowerEdge R300
PowerEdge R410
PowerEdge R610
PowerEdge R710
PowerEdge R805
PowerEdge R900
PowerEdge R905
PowerEdge T300
PowerEdge T410
PowerEdge T605
PowerEdge T610
PowerEdge T710
Yeah, only reason I purchased the Dell is… work uses them almost exclusively so it will make my learning and applying curve a little easier by keeping it the same.
My goal is to VM our work environment. Unfortunately I am the IT dept. (haha lots of hats to wear) and we only have live servers with nothing to experiment on. So, I’ve taken it upon myself to learn at home but, it’s gotta be cheap (no budget for this stuff).
I’m going from memory on this but I’m almost 100% sure the BIOS has the VT-Enable option. I will double check when I get home though.
I hear ya, I’ve done the lone IT guy thing before. The home lab is a fantastic way to do it. If you want to play around with iSCSI storage in VMware you can always setup FreeNAS on a spare junky desktop. Until recently I was taking up about 1/4 of the laundry room with a 28U rack housing my Cisco Lab and some old servers I had thrown together to learn various technologies. You can imagine how much that absolutely thrilled my wife
I think you’ve found a great alternative to the 860. If you get a chance, let us know if it supports VT.
So my NEW strategy is to put one of the machines in there and move all of my critical sites off of the dell desktop (gx260) machine that i moved them to in a panic when the 800MHz machine that they were in died and onto it. With machine number two i’ll wait until I can get that RAID card and set it up with 2 500G drives (anyone got any recommendations for reliable 500G drives that fit the budget status of these machines?. . .i.e. ~$100 or so a drive). Once that’s in place, i’ll move the VMs to the RAID system and set up the original machine as a RAID system and start moving stuff off of it.
The second machine that i’m replacing is a dual Xeon PIII/800MHz with 1G of ram. You can see that these are not powerful machines that i’m replacing here! That’s why I’m thinking that these 860s with 4G of ram can handle a few VMs each without me noticing a performance drop. What I did notice, though, is that ESXi seems to do static memory allocation, right? You can’t over allocate the memory? My previous experience with VMWare Server was that I could (on a 4G machine) allocate 2G each to 3 different machines and have it run fine. . .as long as they weren’t all at capacity.
My other issue is with compact flash cards. I noticed that these machines came with a CF to IDE adapter hooked up and in the slot where the CDROM goes. I was thinking that it would be a great place to install ESXi. Thoughts on that?
-J
I take that back about the CF cards. Looks like they’re pretty expensive just for 32G and that’s not enough for VMWARE. Oh well. . .was a neat thought though!
any thoughts on these:
http://woo.ly/neweggdrives
That’s searching for drives 400G to 800G and sorting by their rating.
-J
Sorry for another comment. . .but is this the right RAID card? $30 free shipping seems almost free:
http://woo.ly/ebayraid
Are you sure about the CF space? I thought you could install ESXi to USB/Flash with only a few hundred MB of space. Might be worth looking into further.
Not completely sure about that SAS5iR card. It looks like the correct one (all the components look right) but it doesn’t have a backplate on it. Backplate probably isn’t the right word… you know, the metal piece that gets screwed or clipped into place to hold the card in.
Personally, I’ve had good luck with two WD Caviar Black drives so far and have only dealt with WD Warranty once (on an external HDD) and they were pretty good. I’ve got Samsung 1TB drives in my 860 now but that’s because Newegg had them for $69.99 each a while back and I knew it wasn’t going to be used for production. To be fair, all the server HDD”s I’ve used in production were all OEM (mostly Fujitsu and Seagate) server drives through Dell due to the 4hr or NBD replacement time.
Wow! I didn’t even notice the lack of. . .a. . .yeah, let’s just call it a plate. Minor issue as long as the card’s right. I could steal one from something the same size and make it fit.
As for the CF, I think you’re confusing it with a USB Flash drive. What i’m talking about is this: http://woo.ly/compactflash
Looks like the largest NewEgg sells are 32G and the cheapest one is about $74. So I’ll opt out of that.
Mine came with a 512mb CF as well. I just thought that ESXi only needed 300-500mb on USB and figured that would translate to CF as well. Haven’t looked into it at all though.
they have 512M CF? My bad. . .i didn’t even know. Seem to be expensive, though.
ack. . .mixing up MB and GB in my head like an idiot. of COURSE it would fit. DUH.
Any thoughts on using it like that?
-J
like this one:
http://woo.ly/cflash
Hi,
I guess my last post never made it. Looking for suggestions for rack mounts for a four post. 19×25 I believe. Any leads?
Thanks again…
I’d be interested in rack rails as well. I’ve seen them on ebay, but at $40 a set i’d rather just sit them on top of the existing servers or use two rack shelves. . .one in front and one in back.
-J
James: Yes, although I’m not sure of the size or speed requirements
Jon and James: Your best bet is going to be Craigslist and eBay unfortunately. Since rails are usually around $200 new, $40 isn’t bad at all. I see them go for $100 used a lot
. Also, Dell alone changes rail design frequently and along with square vs circular hole designs makes it hard to track down exactly what you need. My home setup is a 28U 2-post rack and only one server and my Snap NAS have their rails. Everything else is either network gear with ears or sits on shelves that I got for free with the rack (which btw, I found on Craigslist for $50).
Ok, sorry for not getting last night as I intended. Too late, too tired, fell asleep at the dinner table.
None-the-less; I did check my bios on the 1800 and I do not see the VT setting in there I could have sworn I saw before. I thought that was the first thing I enabled when I bought it (since VM’ing it was my goal.)
Now, for the odd part. I already have installed ESXi v4.0 on it and have it running, granted I didn’t create any VM’s as I realized I couldn’t mount RAW devices on the EVAL version so I couldn’t transfer any files between the old OS (NTFS) and VMFS. This is when I started getting serious about a plan for installing VMware. Much easier to do with empty hard drives – which isn’t an option for me right now.
Cont’d odd part; So I started doing some research this evening to see if in fact the 1800 does or does not support VT hardware enablement. Can’t find anything that specifically says yes or no. BUT… There is a new BMC update from Dell that came out on 6/30/2009 (v1.84 A09) and the description says “Fixes and Enhancements: Fix the incorrect entity instance number of CPUs and PSUs shown in the inventory info of VMware ESX”
So what does that mean??? Obviously this must support ESX to some extent. Will ESX run without VT-Enabled in the BIOS (since I don’t see that setting)?
Does my babble make any sense? I’m kinda tired so not sure if I’m explaining clearly.
Regardless, I’m going to move forward with my ESXi v4.0 installation and convert my file system to VMFS, still have to figure out where to temporarily move my data (2TB worth, yuck!!!). I will post back as I progress.
Here’s what looks like a pretty good “whole vm” backup solution. Looks like it can be done live (although i’m not sure if that’s wise).
http://communities.vmware.com/docs/DOC-8760
-James
Thanks for the post about the backup script. I decided to cheap out on the last piece of my ESXi setup, and not get a RAID controller (for now, anyway). My plan is to back up my VM’s to another (physical) box via a dedicated gIgabit link, so “ghettoVCB” sounds right up my alley. This is all a home setup for me, anyway, so I can get away without real RAID.
Yeah. . .i’m hoping it will run well with “live” VMs. I can’t be shutting them off every night for backup. My plan right now is to have two VM machines backing up to eachother with this script. If one machine gets canned, the other can be brought up.
As far as the RAID goes, i’m ordering those cards on ebay. I’m putting one machine with a 1T drive in there now (I need to replace the temporary desktop machine i have in there now) for now and will replace it with the RAID setup later.
-J
Sorry for late response, insanely busy day. I’ll DEFINITELY post a reply with real info this evening.
FYI: Newegg has WD 1TB Caviar Green drives on sale for $74.99 for 24hrs today. Personally I have the WD Blacks but then they say those aren’t good for RAID either, but for home use I don’t care.
Zoobie: So, ESX 3.5 is 32bit and ESX 4.0 is 64bit meaning you need a 64bit processor. VT is only required if you want to run 64bit VMs. I was pretty sure this was the case with VT and some quick reading points towards that as well. So, unless you’re planning on 64bit VM’s this probably isn’t an issue with the PE 1800.
James: Thanks for the great script! I’ll definitely have to check that one out as VM backup is something that comes up all the time and is hard to provide for smaller VM setups due to budget limits.
Aaron: The only thing you’ll have to take into account for your situation is downtime cause by a potential HDD fialure without RAID. Sounds like you have that pretty much covered though.
Zoobie: I saw that sale today as well and was so tempted… I’m putting together a project for a cheap ITX box that runs BackupPC and Nagios for SMB sites that it would be perfect for. I still have a lot of work on the project though so I can wait.
One thing I’ve noticed from reading COUNTLESS reviews on Newegg is that with few exceptions Hard Drives still seem to be somewhat of a crapshoot. People have bad experiences with the best of drives. Most of the good drives seem to have similar ratios of good ratings to DOA/”died in a week” reviews. Therefore I’ll probably stick with WD for now because of my experience with them and warranty claims. That could always change though (10 years ago I was a Seagate fan).
I’m actually at our datacenter right now. Drove 40 minutes from NJ,through the Lincoln Tunnel, down to 8th ave (we’re in the same building as google), plugged in the 860. . .BEEEEP BEEEEEP BEEEEEEEEP!!! I brought the WRONG ONE! I picked up the one without any memory in it! I can’t even sit here and throw vmware on it just so I have something to work with!
Drives. . .yeah i’m a big seagate fan, but they’ve been catching a lot of flak for a LONG time now. i wouldn’t want to chance it. . .even in a RAID configuration.
My main plan right now, though, is to get these 4 ecommerce sites off of the dell desktop machine (that’s taking up 4U!) and onto a vm. Once I build a new VM, even if it’s on an 860 w/ a 1T drive and no RAID, i can take a little time building the mirrored system and have very little downtime when it’s time to move it over. Once they’re moved over I can bring the 1T machine home and redo it.
My biggest hurdle has been working with the vmware client. I’ve moved away from using windows (Ubuntu on one laptop, OSX on my MSI wind and my wife has a macmini) and there’s no client for other platforms. big mistake on vmware’s part! Fortunately, after gaining access to a windows machine i was able to set up a windows VM to use the client. Then i can just remote desktop to that machine and it’s fine and dandy. what a real pain in the rear!
-James
I haven’t used Seagate in a few years because I was getting bad drives. I also thought it odd that for a system that is built on Linux there is no Linux client… makes no sense. Although, I guess VMWare is more popular with the Windows crowd than with Linux admins.
In our Datacenter setup I have the VI client installed on the vCenter server and a VM. I usually RDP into the VM and run the client from there because the vCenter server is also the BackupExec sever and is only a PE 750 so it’s usually very slow. Actually, I’d love to replace it with one of these 860’s since it would run circles around the underpowered PE 750 (did I mention I hate 750’s?)
Hi,
I ended up ordering a few of those Dell PCI-e SAS 5/IR UCS-51 Controller cards from eBay. I will go and buy the end brackets some where. Any way, I was wondering if this is the correct cable for them; http://www.newegg.com/Product/Product.aspx?Item=N82E16812710002
I also ordered the rack rail mounts as you suggested off of eBay as well.
Your help is much appreciated.
Nice! I might have to order one myself. That is the correct cable.
Let us know how it goes!
Check it out!!! Dell OpenManage Server Administrator LiveCD! http://linux.dell.com/files/openmanage-contributions/omsa-55-live/
Downloading now but can’t test it yet since the 860 is still in use. If anyone tries it out, let me know.
Also, checkout just how many Linux resources Dell has at http://linux.dell.com/files/
I might just have to make a dedicated post for this…
Hi,
Thanks for the links. I’ve been able to get (on order) most of my system components. Would it be helpfull to anyone for me to list out where I bought my parts? If so let me know and I would be happy to post them here, with EvilEmuofDoom’s permission, of course.
Does any one know of a good source to research co-location? Like questions to ask, such as requirements for mounting servers, power requirements, band width metering etc, etc.? This is my first time in this situation, so sorry for the off topic questions. I guess what I’m looking for is kind of like a check off list of things I need (to ask) prior to deployment. Any and all suggestions is much appreciated.
Jon: Go for it! The more options the better
Kinda lengthy but great article on the subject: http://www.technewsworld.com/story/61485.html?wlc=1251489848
Did anybody get this working on a PE860? Any luck with ESXi OMSA or XenServer 5.x?
Here is a description for a $639 Dell (entry level) Server to play with VT . . .
AND
Only $540 with this CPU (no HT) Intel® Xeon® X3430, 2.4 GHz, 8M Cache, Turbo
NEW shipping included. Plus 5% off with Bing coupons
PowerEdge T110 with Intel® Xeon® X3440, 2.53 GHz, 8M Cache, Turbo, HT , and with 8GB Memory (4×2GB), 1066MHz, Dual Ranked UDIMM , then an onboard SATA, 1-4 Hard Drives connected to onbaord SATA Controller -No RAID – with one 160GB 7.2k RPM Serial ATA 3Gbps 3.5-in Cabled Hard Drive (I have 2 250GB WD SATA’s too). For NIC – On-Board Dual Gigabit Network Adapter (or for more cash – Broadcom 5722 1GbE Single Port NIC, PCIe-1) and 1 DVD Drive, Internal .
What are your thoughts? Tear it up !
BTW – Thanks for the much needed blog, you filled a void with knowledgable representation, tell your wife how much you are appreciated : )
ESXi on USB internally with this Pedestal server.
I have been researching and searching for months, also with a $500 budget and parts laying around. I even have an old Stratus ftServer 3300, although it’s 32 bit. A boat anchor.
Look forward to your feedback.
Nice! I haven’t seen the T110 yet. If you get it, let us know if the onboard SATA controller works with ESXi or ESX. A client recently purchased a T105 (the AMD version of the T100) and ESX didn’t recognize the SATA controller. Although, looking at the Dell site you can add a RAID 1 capable card for $200. Looks like a solid choice for an entry level VMware server especially if you want something with a warranty!
Thanks for the post!
Hi,
A few months ago I posted about adding some resources to the list on where you might find CPUs, memory, etc. etc. but never followed up, sorry. I will list them now, although a quick scan has shown that most of the prices (some dramatically) have risen, so you may have to search to find the best deals.
CPU & Memory: SuperBiiz.com paid $176.99 for CPU and $30.49 a stick for ram. Memory price is over $50.00
now.
DVD writer: slidirect.com paid $49.00 for the DVD (Panasonic UJ-850 Slim DVD Writer IDE (Tray Load) ) $6.00 for conversion I/F board. They call them selves something different now. http://www.digistore.com, I think.
Perc 5/i SATA/SAS RAID 1/0 card: AMB Systems, paid $29.00 for the card, it comes with the bracket. Try a search on eBay, they still have them listed as the date of this post. For $10.00 more they have the 256 MB cache version as well.
I found the appropriate cable at Newegg.com for $16.49. SNT 30″ 32pin SAS to 4 x SATA 7pin cable Model CABLE32SAS47 – OEM.
That’s the main pieces, other items such as the IDE cable to the DVD conversion board are a couple of bucks at most local computer hardware stores around my area.
I still have not deployed my systems, probably won’t before next year so I don’t have much to offer from “real world” performance numbers.
Things I plan to try:
1. CF boot drive – I purchased an 8 GB x266 card and a IDE to CF card R/W interface at the time of purchase of the servers.
2. I bought a couple of OCZ 60GB SSD drives to use with the Database server, these will be used with WD VelociRaptor 300GB 10K SATA II drives. The 60GB SSD drives will hold the DataBase (MySQL) and be used with a PERC SAS 5/IR card in RAID 0 configuration. The WD VelociRaptor’s in RAID 1 will backup these SSDs continuously. So the SSD are acting as a psuedo ‘cache’ for the database. I would really like to use ZFD for this, it’s built in to the filesystem. but sadly, I’m going to use Ubuntu OS and their are license issues.
3. I recently found some Hitachi 450 GB 15K SAS drives at an almost bearable price of $325.00. From the reviews that I’ve read they are reasonably responsive. So I might scratch number 2 idea above and just pickup a few of these.
I hope this post helps someone in need of piecing a system together.
Thanks to EvilEmuofDoom for all of the help in getting things together.
Hi,
Sorry it is http://www.digistor.com. Drop the ‘e’ at the end. They have the DVD/CD R/W drive I am currently using.
Hi,
Can any one help me out on connecting my servers together? I’m a noob when it comes to this stuff. I need to connect my Web Server to my MySQL server to my application server. I’m thinking that gigabit ethernet is going to be the least expensive way (via a Gigabit ethernet switch such as the Dell power connect 2808 switch). I would also think that fiber would be an alternative, but I’m not sure of driver support or cost. Any thoughts would be very much appreciated.
The way you describe it, sounds like you’re talking about three physical servers. In that case, gigabit ethernet (assuming all of your servers have gigabit ethernet cards) would probably be the way to go.
Could you tell us more about your systems? Why do you have 3 separate systems, where will they be hosted, what kind of hardware, what will you be hosting, etc?
-James
Fiber is really only practical in certain situations. Usually where distance is an issue or FC SAN connection although iSCSI is quickly replacing that in the most common setups as well. Gig Ethernet is definitely the way to go and if you need more bandwidth but don’t have the ridiculous budget required for 10GE then you could look at teaming the NICs together depending on your hardware. VMware allows you to team NICs but relies on a round-robin scheme instead of true load balancing.
Let us know your physical setup and what you’re trying to accomplish and we’ll help you out.
Hi,
Thanks for the replies. Let me get some advice from you as a follow up. Knowing the limited space of the 1U case and that I am already running a SAS 5/ir card, what would you recommend then?
1. DRAC remote management card.
2. Dual 1Gb Ethernet card.
Thanks in advance.
I bought a Dell T110, patly based on what I read here.
Question: Is there a major difference between the capabilities of the Broadcom 5709 and 5722?
Dell offered a free 5722 to make up for the lack of the advertrised dual gigabit newtwork capability.
Is it worth ttying to get the 5709 instead or are these differences really unimportant to most everyday uses.
A case in point would be an application server ,run as an ASP connectiion to many users over the Internet.
Any guesses on the relative connections a 5709 could handle versus a 5722, assuming that only the NIC is the bottleneck? It may be that the 5709 was motivated to help less capable processors than a X3440, and that it is virually impossible to saturate a X3440 with Tcp/IP chores?? Can anyone point to synhetic number of packets/response time tests using a X3440 and Broadcom 5709 versus 5722?
Thanks!
on perc 5/i vt-d is not supported.
if you vt-d enable on bios you can’t add datastore on vmware 4.x
RE: iq100 “I bought a Dell T110, partly based in what I read here. … ”
THANK YOU! – I barely noticed the lack of the dual gigabit, ASK AND YOU SHALL RECEIVE, because they haven’t offered this to me after the fact! I will call them on this and request one too. IS THERE AN OPTION OF THE TWO (5722 & 5709) YOU MENTIONED?
Initially, I would opt for a dual head controller to net 3 interfaces, this provides a more efficient use of resources. I am not in a production environment, just R&D for learning and development course development, as I am an IT Instructor and trainer for the last 10 years (IT for 23 yr).
It did surprise me to see 2 internal USB ports, I thought Dell mentioned only one internally. That was useful.
I still need a solution for ESXi installable for the file system compatibility limitations. I have plenty of SCSI drives, (well, SATA drives too) yet no SCSI controller. Yeah, Craigslist for $20 – $50, but budget is gone.
NOTE:
* IDE RAID and SATA RAID are not supported for the VMFS file system.
and . . .
* ESX 4.0, ESXi 4.0 Embedded and ESXi 4.0 Installable are equivalent products from an I/O device compatibility perspective. (per VMware’s website)
These are two facts that are commonly overlooked. VMFS (Virtual Machine File System) is a cluster file system. Used by ESX Server and VMware Infrastructure (VI). It’s used to store virtual machine disk images. It is not mandatory to use VMFS with VMware; an alternative is NFS.
Thanks for the TIP iq100!
* * Great Forum, I hope to contribute more, if encouraged to do so? * *
BTW
Broadcom 5709 and 5722 difference?
5709 = dual head (2 x 1GB Ethernet ports per card) $149 ea. @ Dell
5722 = single head $89 ea. @ Dell
Sorry for the long absence all… it’s been a lot of long hours and long weeks.
At this point the 860 is probably an aging option even though it’s only been a year. I’ll keep my eye out for other great budget ESXi options.
For those with the T110’s. Have you had any problems with them? I’ve run into some strange network issues with VM’s running on a T105 and T110 at a small client site (using on-board NIC only). No packets to the host are dropped but pinging the VM’s causes and occasional 20-30 second connection loss.
I’ve also recently built 2 Hyper-V R2 servers for a client of ours who is a MS Gold Partner (they have tons of free MS licensing and don’t want to pay for vmware). So I might have some good info about that experience coming up soon.
In other news, I finally got around to testing and acquiring my VCP4 certification (yay!)