Power Outage Update
Power has been restored to the building and all servers are back up and running. There currently is an issue with a blade in one of the core routers and it is being replaced, so the servers are still unreachable until this issue is corrected. We are working to get service restored as soon as possible. Our apologies for the inconvenience.
- UPDATE 04:51 PST -
The supervisor in one of our core routers went kaput when the power went off. It was replaced shortly after the original posting. Further problems arose with corrupt configurations left over from the old supervisor. These have been restored and we have re-established connection to seven out of our eight uplinks. We are currently working on the eighth uplink and everything is coming back to normal operation now. We are close to being 100% and thank everyone for their patience. As always, check this space for updates.
- UPDATE 11:21 PST -
The router is up and healthy, and the majority of our network is back up and running. However, a few of our servers are having issues because their file servers are not talking to the rest of the network correctly. There is a link down from one part of the datacenter back to the routers and that is causing the file server issues. If you are on a server whose name is a beverage, this will effect you. We are working very hard to resolve this issue, and we apologize for the inconvenience.
.
February 25th, 2007 at 2:57 am
Keep the good work! .. first!
February 25th, 2007 at 2:59 am
Hurry Up Guys , You are again losing you repu as last year
Best Of Luck
February 25th, 2007 at 3:00 am
Nothings working for me,
February 25th, 2007 at 3:00 am
That post seemed pretty contradictory to me … how can the servers be back up and running if they’re still unreachable?
February 25th, 2007 at 3:00 am
Yay! I’m so lost with my websites down.
February 25th, 2007 at 3:02 am
Sites & mail still not working…
February 25th, 2007 at 3:02 am
I was curious why my site wasn’t working, and when I couldn’t get to the panel, that was even worse. But glad to see ya’all know about it and are working on a fix.
February 25th, 2007 at 3:03 am
so they have power just no routers
February 25th, 2007 at 3:03 am
I guess what they meant was: the building has power again, but our servers still don’t work.
February 25th, 2007 at 3:04 am
Now the minimum that dreamhost can (must?) do is increase our disk space or send us a cold beer
Nice work guys…
February 25th, 2007 at 3:04 am
still not working
February 25th, 2007 at 3:05 am
Surely the faulty blade in the router couldn’t have been caused by the (controlled) powering down and up again (could it?), so wouldn’t that problem have been evident before?
In any case, I’m glad you were on hand to shut down the machines safely, and thank-you for keeping us all informed.
February 25th, 2007 at 3:05 am
@Huh?: The servers are up but their connection through our pipe out of the building is the issue, so they are still unreachable to the outside world. It is being worked on right now.
@CAFxX: The dreamhost.com and other dreamhost sites besides dreamhoststatus.com are on the network with the core router issue, they will be back up when it is fixed.
February 25th, 2007 at 3:05 am
Mail is working, but not Panel or my websites
February 25th, 2007 at 3:07 am
The servers may be up internally, meaning the computers they run on are up and running, but the blade that is down is preventing people form the outside to access them, or the data is inaccessible until the blade is repaired. From experience, they are definitely working on this, not just blowing smoke up our butts.
Keep up the work guys, it’s appreciated. (Gals also if there are any working, sorry, it’s slang)
February 25th, 2007 at 3:07 am
@huh: servers run independently of the routing equipment. If the servers are up, but the routers are down, then you still won’t get anything. Even if the servers were up and the routers were up, there’s still plenty of places to cut the link and make your server disappear.
It is still rather odd to think that DH has 3 datacenters with all lines to the internet loading into one of them… what was that saying about eggs and a basket?
I do sympathize with DH though - they’ve had some real bad luck in the last year.
February 25th, 2007 at 3:09 am
No irony here then that I was about to post a blog entry about the state of electrical power supply in my town ( Horsham , Sussex , England ). The power had been off for 3 days and is even now intermittent. Its like living in a 3rd world country. So there I go to blog about it and har de har har the power is out at Dreamhost. I love the irony in that , I really really do. Anyway to the Dreamhost guys whom are currently working had to get things running I do hope your not getting those annoying interupting questions like ” Will it be much longer ? or Hows it going then ?” and keep up the good work( hard ) work .
February 25th, 2007 at 3:09 am
Thanks for telling us!
and to the n00bs that say “you have too much downtime” “when are the sites coming back up” “Hurry up”
Think of the price and what you get and at my old host, site goes down for a day, comes back up and no news!!
February 25th, 2007 at 3:10 am
My mistake, mail was not working
February 25th, 2007 at 3:11 am
@Justine
How come this blog runs while every thing on ur network is down ??
also… Hurry up…
its been a while 
since sites are down… unfortunatelly i have to say, i see every now and then some thing is always wrong with DreamHost hardwares…..
February 25th, 2007 at 3:11 am
pues a mi no me funciona ninguna.
February 25th, 2007 at 3:12 am
How can all servers be up if it is still unreachable?
February 25th, 2007 at 3:13 am
I’m not complaining … just the wording sounded weird to someone who doesn’t know a lot about that sort of thing, was all I was trying to point out … “servers” means the same, regardless of how anyone explains anything. I actually found it more humorous than anything else. Maybe I’m the only person who has a sense of humor at 5 am …
February 25th, 2007 at 3:14 am
Ni a mi… Pero esta gente trabaja rápido!
February 25th, 2007 at 3:15 am
Still can’t access DH main page, though.
Neither my sites, BTW.
February 25th, 2007 at 3:15 am
Actually, I was just pointing out the irony since “server” means “server” means “server”, no matter how you look at it. I found it humorous; I wasn’t complaining … Maybe I’m the only person who has a sense of humor at 5 am?
February 25th, 2007 at 3:17 am
I appreciate the fact that this page is here for updates. Building management waited for an appropriate time to do this upgrade if you are in the PST time zone. Next time it could be good to fire off an email (though it might just make people who never knew about the outage jumpy, so on the other hand I understand not sending out an email).
February 25th, 2007 at 3:20 am
so, the unreachable server includes the server in fiji ?
February 25th, 2007 at 3:20 am
Here in Brazil is 7 am Sunday morning. It is the first time I go to my DH hosted site and it is down. Hope this core router problem be solved ASAP (the outage is solved now).
Bye!
February 25th, 2007 at 3:21 am
don’t worry… be happy
February 25th, 2007 at 3:21 am
A lot of hosting outfits would not bother to keep clients up-to-date with what’s happening. On that score, congrats to the Dreamhost team. Hope I still feel the same at the end of the weekend — by which time everything should be back in place for Dreamhost to maintain street cred.
February 25th, 2007 at 3:21 am
If they have 3 datacenters and one goes down, shouldn’t the other 2 have all the data stored and take over the load? If this is not the case then what is the point of having the other two data centers, might as well just have one and charge us less money for hosting…
February 25th, 2007 at 3:22 am
@Saurav Basu
Because this site is hosted completely off of Dreamhost systems, so that even if the entire system goes down, they can still notify us.
February 25th, 2007 at 3:23 am
Lets all just go to bed ^_^
February 25th, 2007 at 3:24 am
hope it’s fixed soon…
*sigh*
February 25th, 2007 at 3:24 am
Hmmm…. so who does dreamhost use when even dreamhost goes down?
February 25th, 2007 at 3:26 am
Most of my emails are working fine now (a couple aren’t, but s’okay) though the websites are still unreachable. I don’t mind downtime when it’s something I’m prepared for, and even then, I think that (for me) the fast and courteous tech support I’ve received makes up for a few down-times. My last host never, ever warned anyone about down-time, and they couldn’t be bothered to tell anyone when there was a system-wide problem, and I was paying much more for their service. There’s always a trade-off. I rather like the one I’ve found with Dreamhost.
February 25th, 2007 at 3:26 am
I agree with harrisony, at least you get told when things are down so you know they are working on it. It’s a shame it’s down at the moment (and it’s 10:26 am here (GMT), so it’s causing me problems) but clearly they are working on it so let’s just sit tight!
February 25th, 2007 at 3:28 am
I work for an ISP and one of its main revenues is data centre and hostings - and though we charge -a lot- more, we have generators and what not to pick up the slack etc
having had to put up network status messages on a regular basis, when a time isn’t specified it usually means they have no idea
I’m with Aixsyd, bed time. My f5 key is getting worn out ^.^
February 25th, 2007 at 3:29 am
It looks like things should be up soon now. It’s a matter of less than an hour.
February 25th, 2007 at 3:29 am
that means it shouldn’t be long now.
remember this is probably not their datacenter alone.
It is likely shared. these kinds of facilities are too expensive for only one host to be in there.
So you can bet some non-DH people are also down.
February 25th, 2007 at 3:30 am
Three things:
1) To those that say one shouldn’t complain because the service is cheap: not everyone is paying $8/month. Also, it seems to me that since even http://www.dreamhost.com is down, it’s not a matter of price. If they don’t have redundancy for themselves, who do they have redundancy for ?
Maybe things should be quantified: what kind of uptime for each class of accounts does DH aim to provide ?
2) Is Apple Store really down because of DH ?
3) What provider are they using for this site ?
Thank you,
Davide
February 25th, 2007 at 3:30 am
Wow, I was expecting for you to have much more downtime than this (even though I still can’t access anything)!
Thanks for the speedy work!
Lee
February 25th, 2007 at 3:33 am
I don’t think you understand. When you have a business and customers are trying to reach your website and they can’t, what does that say about your business? Thats like you going to a 7 Eleven and the doors are locked… What’s the first thing your going to think? WTF?!??!?!?! Obviously the people that are cool with this are the people that have personal websites about their cats hosted here.
February 25th, 2007 at 3:33 am
not yet
February 25th, 2007 at 3:35 am
Thanks for the update. Good luck on the blade.
And how about having an open house to make it up to us customers and have King Taco do the catering?!
February 25th, 2007 at 3:35 am
>>Ken Says: February 25th, 2007 at 3:30 am
>>Of course this has to happen on the night my site makes it to the homepage of digg.com >>…. why god why?
KARMA
February 25th, 2007 at 3:35 am
Why dont you email us about this? the LEAST you could do is send an email out to all your clients to let us know.
I have just been abused and had my head torn off cause one of my customers sites was down. You guys do make me look incompetent at times eh! and I’ve always got to search for the problem that you already know about.
Please consider emailing us. Maybe the employee owned model does not work well, whos there to keep you all on your toes.
February 25th, 2007 at 3:35 am
Waaaaaaa!!! I want my site back now!!! I want my mommy!!!
come on guys, just pay the electricity utility their money and flick the switch to ON
February 25th, 2007 at 3:36 am
I was worrying here… I thought my site was down for another reason.
Hope it gets back up ASAP…. It still can’t be access via web browser!