Wednesday, April 28, 2010

An Open Letter to HostingRails

Dear HostingRails.com,

I am unsure who to contact as I have tried on numerous occasions today to contact you by phone, email and through your support system. Given the lack of response from your team I am understandably disappointed.

Let me say this first: I have had two websites that have been offline all day, during business hours. From 9am PDT to 8pm PDT. Actually, only one site is up and running at this time.

One of these sites is a client site, the other is a personal site.

Both of these issues are related to a server upgrade you had planned on your servers.

I received emails from you informing me of a server upgrade over a period of "24 to 72" hours and that my static IP address would be changed. This type of upgrade is pretty standard save for (1) the fact that it happened during business hours, (2) the fact that it happened during the week, (3) the fact that you provided no time when it would happen, and (4) the fact that you gave a 48 hour window over which the upgrade would happen.

However, I was assured that "While we do not anticipate any issues related to the upgrade, we are available to answer any questions that you may have once the move has been completed." This is a quote from the email you sent me.

As soon as I found out that my sites were down I immediately filed tickets through your new ticket system (which now requires me to login to my account v. just filing a ticket from the website).

I called at 10:11am to your new phone support system. I actually talked to someone who was friendly but suggested I file tickets through the support system. I had already opened one for one of my sites; he suggest I open a new one for the other site and he would get this into the queue with "priority."

I also updated my nameservers with the new information as I was instructed to do.

I checked the site during the day with little progress on the tickets.

I called back at 4pm PDT to check on the status of my tickets. There had been no progress. In one of my meetings during the afternoon I worked myself on trying to get the site up and running and updated the tickets as I uncovered more information.

Not only was the support rep unhelpful on this call he was entirely uncaring about my situation. I explained that I had a client website that had been down all day - no sympathy at all. No information he gave me that suggested he knew anything about the tickets in process - only that "one of the engineers is looking into the issue." Given that you had major hardware upgrades happening today your support staff fielding calls didn't seem to know anything about it.

Oh, I also attempted to contact you on twitter as well, no reply. It was interesting to see a litany of replies to you from very unsatisfied customers.

Sorry, time for a little tech talk. I do this to show that your migration was not well planned and so that I may document the steps I had to go through to get my application working.

While my tickets were "being looked at" I was able to SSH into my server and get my Rails application up and running. One of the features of your upgrade was to allow your hosting clients the ability to choose a specific version of Ruby with which to run their applications. I was previously on a Ruby 1.8.7 server. Naturally when I logged in my server was running 1.8.6. I got this all working according to the tutorial on your site (your engineers BTW had completely hacked a solution that did not work b/c they any existing mongrel process would never be stopped) but was shocked that my migration had not preserved the version of ruby I had previously been running.

This of course was not enough. Once my application was running I discovered that my database, while in existence, was empty. No data, no tables. Nada. When I logged into cPanel I was unable to see my database or any database user. I was thus stuck again.

I finally got an email reply from one of your "engineers" that told me they'd been having permissions issues and that "reassigning users to the database" should solve the problem. Clearly, they had not understood my ticket update - I could not see my database. I had attached an image just in case a visual would help.

Ultimately I solved this problem myself by restoring my development database into the production database. Fortunately this is a new website with very little in the way of dynamic content at this point and so I was able to reasonably restore the system. I figured based on the email response I had just received it would be several more hours before I was able to have my backed up database restored.

The punchline of course is that at 1:16 pm PDT I was notified that my server upgrade and migration had been completed.

During the day, because I had plenty of time to just twiddle my thumbs waiting on your support team, I did a little research and found out that you have recently been purchased by Jumpline. I don't know anything about Jumpline but if today's events are a result of your management then I am duly unimpressed.

Let me summarize this here for you:


  1. I have been a loyal customer to you for over two years. I have recommended many people to you for solid and inexpensive Ruby on Rails hosting.
  2. You used to have competent engineers that would resolve issues quickly and through email, not some arcane and unhelpful ticketing system.
  3. You used to have engineers that would take the initiative to resolve issues. Now you have engineers that throw support tickets back at clients that now have to resolve issues themselves.
  4. You used to resolve issues in a timely manner. Now apparently 12+ hours of downtime is perfectly acceptable.
I doubt there is anything you can do to maintain my business. I am actively shopping for new hosting providers. While I hope to find an organization that can exceed my expectations as a customer I most certainly can find one that offers far superior service than you.

Regards,

Jeff

11 comments:

gogo said...

Hi, I've had sites on hostingrails for quite some times, and did experience a lot of problems during and *after* the upgrade, basically nothing works like it used to; they've been having a server load of 14.xx (which sometimes mysteriously goes down to 1.55). Note that they've been bought by Jumpline and that's why their customer support changed. Anyway after a lot of research I've now decided to move everything to RailsPlayground.

Anonymous said...

i am having the same basic issues starting yesterday. i am so upset that no notice of the acquision was done.

also i just lost one of my clients to another host...

@jobrieniii

Anonymous said...

what happened to raju? william? those guys were sharp

Jeff in West Seattle said...

update - I have since called HostingRails and let them know that it's been 26 hours since my site has been down. They "explained" that they are working furiously to solve these migration issues.

I'm not sure what the rush was here to get these servers migrated but they sure didn't test this migration plan at all and have handled this whole situation so poorly it's ridiculous.

Anonymous said...

I haven't used HostingRails so cannot speak to their quality of service past or present. I can however point you at slicehost.com, who has provided excellent documentation and service in the year that I've used them. Heroku is another option that provides an excruciatingly easy deploy process if you have familiarity with git.

- Chris

Lindsey said...

At least I understand this tech blog more than the normal running blogs :P I don't do rails, but I had a similar prob with fatcow, their solution was to charge me $20 for the next year of hosting. -LRJD

Jason said...

I couldn't agree more! I have been happily using hosting rails for almost three years and recently had two accounts 'frozen' - tech support couldn't even login and had to 'create a ticket'. Not good when tech support has to create a ticket for themselves! The websites have been offline for at least three days (perhaps longer, since I don't check them every day).

Worse yet, I recently created two new accounts for some customers. I seriously regret my choice to go with these guys again. Had I realized the issues created from their acquisition by jumpline I would have looked for a different host.

hostingrails used to be awesome! It's too bad the sale to jumpline has caused some many problems.

danimataonrails said...

I am having the same issues this week. Even worst: after getting everything straight they have dropped my apps and recover a backup of my machine so I have lost all the history/data/logs for the last three days.

Of course, I am not continuing with them.

RandallMit said...

Me too! I left Jumpline a few years back for this very reason, and it seems I'm stuck with them again.

Site has been up and down for four days, and all I get is "we're working on it."

I'm getting off their service as fast as I can.

Any lawyers on this board?

Anonymous said...

I've had a dedicated server with HostingRails for 4 years and am currently going through a similar nightmare.

This move was supposedly to put our machine in a separate network class that should take 30-60 minutes. It had to be done this week during business hours. We scheduled for 2 days to get the timing right as it is a high volume site. The site has been down for over 4 hours and support has been completely absent in replies to our tickets and calls.

Only during this have I found out about the new ownership of hostingrails. I'd not previously had any problems, but this was the only site I manage that was being run on something not EC2.

After this, this site will be on EC2 as soon as humanly possible.

Ngoc Lan said...
This comment has been removed by a blog administrator.