The Ramblings of an Incoherent Administrator

What happened (quick and dirty version)

What happened (quick and dirty version)

[info]squeaky
4/16/08 02:37 pm
Previous Entry Add to Memories Tell a Friend Next Entry
Here is a quick and dirty version of what happened. I will post a detailed account to annoucements later. I am posting this here for rumor control.

Monday
- Called host inquiring about how to cancel servers under contract (databases were still under contract web servers were not). Was informed by faxing in a form I could cancel the contract. I was led to believe that I would be able to cancel servers separately at a later date.
- I faxed the form to get the ball rolling.

Tuesday
- Woke up to multiple text messages and email stating site was down. Logged into to hosting control panel to see both servers deleted.
- Called support to try to end cancelation, they informed me that only billing can handle that and billing is only open M-F 9-5
- Called billing and was told that the forms I sent in were for server termination and they did just that without emailing or calling me. I gave them quite a piece of my mind, but was told that they did nothing wrong. Even though their rep gave me bad info.
- Eventually got ahold of support and managed to get them to save my hard drives from the old servers.
- Had to order new database servers

Wednesday
- New database servers were finally ready
- After wrangling with support for about 4 hours I got the old drives in the new servers
- Got servers up and cleaned the databases and made backups of them immediatly
- Brought site up
- Fixed error with 1 webserver causing issues
Previous Entry Add to Memories Tell a Friend Next Entry

  • [info]ladybug218
    4/16/08 11:52 pm (UTC)
    Damn.

    So how much money do we need to raise to move?

    I have a few ideas of ways to fundraise in addition to the perm account sale and the insane userpics sale.

  • [info]majmunka
    4/16/08 11:54 pm (UTC)
    We love you, you're fantastic. Thanks for the twitter. *hugs*

  • [info]techno_goddess
    4/16/08 11:55 pm (UTC)
    I never lost faith in you for an INSTANT! Thanks so much for keeping us updated via Twitter. That's a great idea.

  • [info]jokesoftragedy
    4/16/08 11:56 pm (UTC)
    Thank goodness it's all up and running again and nothing was lost! I was really worried when the site stayed down for more than 24 hours. But we have faith in you, Squeaky! You were a great admin to welcome everyone from GJ and we knew you wouldn't let IJ just die like that.

  • [info]joereaves
    4/16/08 11:56 pm (UTC)
    Wow they suck but well done on getting things sorted so fast. I was beginning to have withdrawal symptoms! *hugs IJ tightly*

  • [info]aurora
    4/16/08 11:58 pm (UTC)
    Yeah, a permanent account sale would be a great way to raise money to move. Because that's really a shame, that it had to happen.

  • [info]randombastary
    4/16/08 11:59 pm (UTC)
    ::clings::

  • [info]ardath_rekha
    4/17/08 12:00 am (UTC)
    *HUGS LOTS!* I had a smaller-scale version of that happen to me a few years ago, when the server my little-ish site was located on got buggy. I asked to have the server moved, and they were supposed to tell me when it was about to happen so I could freeze my databases for the transfer. Instead they notified me of my "new" IP address... and next thing I knew, there were two versions of my site running concurrently, with their databases collecting conflicting information that was impossible to re-integrate. I was so furious.

    It's like hosting companies lose track of the idea that, in order for us to make our sites function properly, we need to be consulted and included in the timelines for things, especially transfers and cancellations. I'm so relieved that you were able to recover everything. This definitely wasn't you fault.

  • [info]esmestrella
    4/17/08 12:00 am (UTC)
    Wow. Whatta headache. But at least they hadn't actually erased anything.

  • [info]angelicphrase
    4/17/08 12:02 am (UTC)
    I'm using an RP account right now, but THANK YOU SO MUCH for being so quick and efficient about this, especially when it came to letting us users know about the situation ASAP. You're awesome! :D

    -Krissy

  • [info]porridgebird
    4/17/08 12:03 am (UTC)
    Yeeh-haah! What fun, not!

    (thank you for all you do)

  • [info]alisanne
    4/17/08 12:03 am (UTC)
    Thank you so much for all you do for us!

    Seriously, just let us know what we can do to raise funds our whatever.

    And just so you know, YOU ROCK!

  • [info]agtiger
    4/17/08 12:04 am (UTC)
    I'm glad you got the service back online. After monitoring the site for a while to convince yourself all is well again, go have several pints of ale, and then go get some sleep. (I doubt you've slept much in the past 48 hours.)

    Then content yourself that no data was actually lost, and take away from this some good bits of knowledge: Verbal statements mean nothing compared to what's in writing. Always have the replacement for something up and fully functional before turning off the old service.

    Now seriously... Beer and sleep.

  • [info]devotfeige
    4/17/08 12:05 am (UTC)
    A friend said it best when they told me: "And the funny part? It's STILL somehow better than LJ."

    Thanks for the Twitter, man. *thumbs up* It was much appreciated. Not to mention all the hard work on everyone's behalf.

  • [info]ktoth04
    4/17/08 12:06 am (UTC)
    Shouldn't it say
    Create an InsaneJournal Account on the bar?

    Hrm, if its from the LJ code, the a is appropriate, but for insanejournal it should be 'an'

    ~amateur grammar nazi

  • [info]angelsndevils
    4/17/08 12:07 am (UTC)
    You're doing a great job. Remember to breathe.

  • [info]aristoboule
    4/17/08 12:08 am (UTC)
    UGH! I really hope your week gets better! I have a suspicion that this might be the same host that basically held a friend's site for ransom ($10K no less) when they changed terms without notifying anyone.

    PS: You are AWESOME.

  • [info]ramdonomo
    4/17/08 12:09 am (UTC)
    I was really impressed with the way you kept us all informed constantly. Thanks so much for being on the ball like that. I'm sure it was was seriously stressful.

  • [info]ladybahiya
    4/17/08 12:10 am (UTC)
    You are a GOD among men. For this you have my undying loyalty and the firstborn child of whichever of my children breeds first. :D

  • [info]jaybee65
    4/17/08 12:12 am (UTC)
    Thanks for the Twitter updates! It was great knowing what was going on.

  • [info]maliciousintent
    4/17/08 12:12 am (UTC)
    I just want to say thank you for working so hard on all this, Squeaky. Your devotion to getting everything back in one piece was wonderful and I am so very, very thankful for what you did.

    I should mention that I agree with the comment above - How much do we need to shift to your new 'home'? Perhaps a little telethon ticker would get people motivated to donate? :) All I can say is that, even though I can't afford much, I'm donating so that these problems can, hopefully, go away.

  • [info]pre_raphaelite1
    4/17/08 12:13 am (UTC)
    *HUGS and ADORES*

    Thanks for all the hard work. And for the updates on Twitter. You are totally awesome.

  • [info]mercurychaos
    4/17/08 12:14 am (UTC)
    You are freakin' awesome. :D

  • [info]madlodger
    4/17/08 12:15 am (UTC)
    All my accounts look perfect, logging in and looking through was fast. Good job on damage control. I especially appreciated your Twitter info on happenings. Well handled :)

    I would like to get another perm account at the next sale also.

  • [info]prefect
    4/17/08 12:22 am (UTC)
    Thanks for keeping us updated. :)
Powered by InsaneJournal