Today I decided I’d talk about some of what I do at work!
I work in a data center; installing, configuring, maintaining, and decommissioning servers as needed. I do plenty of other odds and ends kind of stuff as well, but server decommissions are done pretty frequently. I did some just a few days ago, so I decided to take some pictures and show you what I do.
Before I start throwing out terms, I’d like to explain a bit about how our servers are racked. In the below image you’ll find several racks with several servers. This particular image is of our J row, which holds cold storage servers that have lease life left but are not in use. For the entire data center; each rack is 2 feet wide, about 8 feet tall, and labeled by row and number. This varies a bit as some rows don’t have 15 racks, some rows start at 2, and some rows have breaks in the middle.
Each rack has a number of units (referred to as Us, pronounced “yous”) depending on the height of the rack. Most of our racks have 42-45 units, and the largest enclosures usually take up ten units. Normal “pizza box” style servers, like our staple HP DL360’s, take up one unit. Larger DL380’s take up 2-4 depending on configuration. Best practice dictates that we not stuff a rack entirely full of production servers as they tend to get hot, and heat is the enemy of electronics.
Usually, I’ll get an email with a request that I get rid of a server or group of servers. I grab the name or serial number of the servers to be removed and double check with the owner to make sure I’ve got everything right before I start pulling wires. I have a spreadsheet that has the rack and unit number, so I run the name or serial number against that to find my location. For instance, one server I pulled for this post was in H4 U25-26. Once I’ve got the location, I update another spreadsheet with the location, server name, serial number, whether it’s going back to our leasing company or for recycling, and when it was removed. I also keep the port numbers that the machine was plugged in to, in case we need to reinstall the server for any reason.
Now it’s time to start pulling cables! I start with power cables, making sure the machine is powered down nicely first. I usually have the owner do this, or log in through a crash cart and shut it down nicely if I have the rights to. Once power cables are out of the way, I trace the network ports, enter them in my spreadsheet, and pull those. That part can get fun.
It’s nothing compared to tracing cables under the drop floor though!
Once I’ve got the cables disconnected from the machine, I remove them from the rack to keep things nice and tidy. Then it’s time to pull the machine out. This server has a nice easy latch system on the front that you pull down on either side. Then you just slide the server out towards you.
Each company has their own method of keeping their server in the rack, keeping the server on the rails, and keeping those rails secure in the rack. That can make it interesting trying to figure out how to get the machine off the rails. The aforementioned DL360’s slide into their rails, and have a catch that you press to release them when pulling them out. This machine is a bit different, it has a set of holes that are two different sizes. You plant the machine on the rails using the larger holes, then slide it forward to secure it in place. There is a latch that holds the machine in place, so you have to release both and slide the machine back to get it out. Once that’s done, I lug that big guy out of there, and throw it on the cart along with the other servers.
Next, the mounting hardware comes out. These can be all sorts of fussy as there are several different methods of getting these to stay in the rack. Some will cause you loads of trouble, and some will kindly detach themselves. This setup is one of the later. There is a tab on the front that you push towards the outside of the rack, then pull the rail back and it’s done. Repeat this process where it attaches to the back of the rack, and then throw those on the cart as well.
That’s one machine done! Depending on the setup and how much room you have to work, it can take anywhere from 5 minutes to 20 minutes to remove one machine. The next step in the process is to cover up the holes left by the departure of the machine with blanking panels.
These are used to control airflow, as we have a cold isle and a hot isle. The cold isle is the front where the machines draw in their cool air and the hot isle is naturally the other end where the hot air is collected and drawn out of the building. Plus they make everything look nice.
I usually try to keep our lease servers separate from those that are going to be recycled, for obvious reasons. In this instance, I had the cart loaded with machines that were all destined for recycling, so I headed down to the loading docks to get them ready for pickup.
That’s a fancy way of saying I throw them all on a pallet, put the mounting hardware in between, and throw a sticky note on to make sure everyone knows where they’re going.
If you’re wondering where the hard drives have gone, worry no more! I remove all hard drives from decommissioned machines and wipe them. Those going back to our leasing company need to have their drives replaced after wiping. Drives from recycle machines are either kept to refill our on hand stock, or tossed in a bin that gets ground up for data security. I use a 3 pass wipe though, so data security shouldn’t be much of an issue. Here’s my lovely little DL360 that I commandeered for disk wiping duties.
This can be pretty time consuming, but I can kick off and monitor the progress from my desk via the miracle that is HP’s Integrated Lights Out, or iLO. So I just need to run down and swap disks out when they’re data has been destroyed. While that’s happening, I wind up the network cables according to color and length and stash them accordingly. The powered cables are also stored in a labeled bin so we know where everything is.
And that’s that, all done! Other than a few emails to schedule a pickup and make sure the lease servers go back in time, I’m all done. Until the next email requesting a decom, that is.
Here are a few bonus images from when things get busy:
Next time I’ll go over configuring a server, specifically a blade server. See you then!