Server outage

Status
Not open for further replies.

chris

Administrator
Administrator
Jun 10, 2006
11,813
1,779
113
Long Island, NY
#1
We had the servers go down last night around 11:30 pm EST. They automatically reboot the servers, but for some reason the databases were not connecting to the web server. Unfortunately, the server admins did not recognize the issue (since the servers were technically up) and I just realized this morning.

The issue has been resolved and I'll do what I can to see what caused the problem, so that it can be avoided in the future. At a minimum, we should be able to avoid any significant downtime should this happen again.

Odd, because the servers at EIC are pretty powerful and made it through some major events (Macworld, etc). Could have been an issue within the host that affected a large slate of servers -- at least that's my guess.

Anyway, sorry for any inconvenience and welcome back to EIC.

-Chris
 

IphoneTony

Zealot
Gold
Nov 10, 2007
2,163
18
38
38
New Jersey
#3
A few other sites were down yesterday also. Apple's site was showing an error message when I was trying to click the sdk download
 
H

Hayesimus

Guest
#4
I thought I had finally been bannished!



Glad to be back!


Not yet haha. but who are we kiddin, you could get away with murder, right?!
 

themanofthedark

New Member
Gold
Jan 5, 2008
2,674
0
0
26
Boston
#5
why wasn't everythingtreo or everythingq down?

and thanks for getting on the issue..


I almost registered at everythingtreo, just to have a forum to chill on..
 
H

Hayesimus

Guest
#6
do you possess a treo?
 
S

Starry

Guest
#8
Chris... you realize with the outage lastnight I HAD to clean house and do some OTHER stuff instead of sitting on the forums... MY GOD there is a WHOLE world out there... but I like this world much better :)

You're GROUNDED don't do that again! LOL... like you had any control huh :)
 
H

Hayesimus

Guest
#9
lol can you image sweet starry. Never phased by anything. And then EIC was down. She probably snapped and started throwing things and cussing!
 

psylichon

Genius
Moderator
Oct 31, 2007
16,591
751
113
41
Philly
#10
How do you think I felt? I'm at work thinking "No major updates till June... just did a warranty exchange for a new iPhone, I have off tomorrow...I'm gonna do it... I'm gonna JAILBREAK tonight!"

So I rush home to review all those looooong threads on the subject before diving in and... no EIC. It's the first time I've ever visited ModMyiPhone.com. I didn't care for it.

So yes, one man jailbreaks alone... super easy, didn't NEED your help, so THANKS ANYWAY!

Hehe, don't know what took me so long...
 

chris

Administrator
Administrator
Jun 10, 2006
11,813
1,779
113
Long Island, NY
#11
So I rush home to review all those looooong threads on the subject before diving in and... no EIC. It's the first time I've ever visited ModMyiPhone.com. I didn't care for it.
EIC down and MMI up. There's a first for everything.:p
 
H

Hayesimus

Guest
#12
haha. you know i was thinking the same thing. i mean how many times have they been down recently. oh well. at least we are back
 

x999x

New Member
Gold
Aug 6, 2007
1,656
0
0
#13
We had the servers go down last night around 11:30 pm EST. They automatically reboot the servers, but for some reason the databases were not connecting to the web server. Unfortunately, the server admins did not recognize the issue (since the servers were technically up) and I just realized this morning.

The issue has been resolved and I'll do what I can to see what caused the problem, so that it can be avoided in the future. At a minimum, we should be able to avoid any significant downtime should this happen again.

Odd, because the servers at EIC are pretty powerful and made it through some major events (Macworld, etc). Could have been an issue within the host that affected a large slate of servers -- at least that's my guess.

Anyway, sorry for any inconvenience and welcome back to EIC.

-Chris
Hey Chris,

I think something might have been going around on the Web yesterday, or maybe it was just a coincidence.

I had a full cluster go down on me yesterday. Had to set the firewalls to ignore bot traffic and that seems to have cleared things up nicely.

I don't know if it was some huge Denial of Service push from pinging zombie desktops or what yet, all I know is that its stopped.

Thanks for getting the site back up!
 

connie

Evangelist
Gold
Jul 11, 2007
2,839
104
63
So Cal
#14
How do you think I felt? I'm at work thinking "No major updates till June... just did a warranty exchange for a new iPhone, I have off tomorrow...I'm gonna do it... I'm gonna JAILBREAK tonight!"

So I rush home to review all those looooong threads on the subject before diving in and... no EIC. It's the first time I've ever visited ModMyiPhone.com. I didn't care for it.

So yes, one man jailbreaks alone... super easy, didn't NEED your help, so THANKS ANYWAY!

Hehe, don't know what took me so long...
Funny you should say this because I also decided to jb my phone again last night. I already had ZiPhone file from before so I re-jailbroke and then I came here to get some helpful hints on what to do next.........uh-oh what now, so I had no choice but to hang out at MMI for awhile.

And Chris, it said to contact administrator on the server error screen last night. It allowed to send you an email to let you know. I almost did it but thought, no probably hundreds of EIC people have already emailed him to let him know. Maybe you could of known last night had I done it?:oops:
 

Buffy

Zealot
Gold
Jan 20, 2008
2,216
3
36
#15
Yes, the outage sucked. I actually got an early night.
 

acosmichippo

Genius
Platinum
Sep 10, 2007
15,385
1,089
113
DC
#16
chris, don't be sorry... just don't let it happen again.

:)
 

acosmichippo

Genius
Platinum
Sep 10, 2007
15,385
1,089
113
DC
#17
How do you think I felt? I'm at work thinking "No major updates till June... just did a warranty exchange for a new iPhone, I have off tomorrow...I'm gonna do it... I'm gonna JAILBREAK tonight!"

So I rush home to review all those looooong threads on the subject before diving in and... no EIC. It's the first time I've ever visited ModMyiPhone.com. I didn't care for it.

So yes, one man jailbreaks alone... super easy, didn't NEED your help, so THANKS ANYWAY!

Hehe, don't know what took me so long...


welcome to the dark side.

drinks are over there->
 

chris

Administrator
Administrator
Jun 10, 2006
11,813
1,779
113
Long Island, NY
#19
I think it was a bigger issue than the EIC servers. Problem is that checks weren't in place to be sure the actual site is up and running. The server admins will have these checks going forward, so any outages should be limited in nature. If anything major were to happen, ie. hard drive failure, hacking, etc, then it would require a new box and restoring backups. That would of course take a few hours, but that's not typical. Possible? Yes, but everything has been done to prevent those types of issues.

FWIW, I was sleeping during all of this.:sick: The admins got the servers up after the initial outage, but the checks weren't done. Hence, the extended downtime.

-Chris
 

chris

Administrator
Administrator
Jun 10, 2006
11,813
1,779
113
Long Island, NY
#20
BTW, there are a few server happenings today that are planned to help complete the migration from EIP to EIC. I don't suspect there will be any downtime, but will do my best to keep it limited.
 
Status
Not open for further replies.