View Full Version : 11/29 - enterprise MySQL
Please stand by while we troubleshoot a MySQL issue on enterprise.
enterprise is being rebooted at this time. More information to come.
Just wanted to update everyone that enterprise is running an fsck right now (file system check), and that we continue to work this issue as quickly as we possibly can. Updates will be provided in this thread as soon as possible.
Please be sure to do the instructions in the following thread:
Automatically receive notifications
http://forum.myriadnetwork.com/showthread.php?t=113
Enterprise is back up -- websites and email should be functioning at this time. If your sites are not working, please allow a few minutes.
NOTE: MySQL is being worked on at this time. I will update this thread when it has been fixed.
CyberDezign
11-29-2006, 11:02 AM
Error message:
Warning: mysql_pconnect() [function.mysql-pconnect]: Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2) in /home/landowne/public_html/search/adodb/drivers/adodb-mysql.inc.php on line 285
NOTE: MySQL is being worked on at this time. I will update this thread when it has been fixed.
Sorry for the delay everyone -- cPanel staff have been logged into the server for some time now and have been making every attempt to address this issue. Their response time after initially opening the ticket was within minutes, and they have been providing us with regular updates.
At this time, MySQL is still experiencing issues on enterprise, and we continue to make every attempt to correct the issue. Your patience is appreciated, and we sincerely apologize for the issues :(
All,
cPanel staff are continuing to work the MySQL issue, and we will provide more information as we have it.
The technical details of the issue (which we usually release after everything has been corrected and have confirmed everything is working, but are going to give now since this is taking a while) are as follows:
One of the hard drives in the RAID kicked offline. RAID (http://en.wikipedia.org/wiki/Redundant_array_of_independent_disks) basically means that we are using multiple hard drives in the servers. If one hard drive goes bad or drops offline for some reason, the 2nd drive will take over, since it has the same exact data on it that the other did. This type of RAID is referred to as mirroring, since the same data exists on multiple disks.
When a drive goes bad, that second drive will take over, and will keep things up and running just fine. Then we go to the datacenter and place a new drive in the server. This is all completely transparent to the customer, their customers, and everyone else, and no downtime is incurred due to a faulty disk.
While the 2nd drive did take over when the 1st drive kicked offline, the RAID controller went haywire, which caused some filesystem issues (note: nothing under /home was affected since that is mounted on its own partition). We fixed a number of things, and now cPanel is taking care of the MySQL issues.
Your continued patience is very much appreciated, and again we sincerely apologize for any inconvenience. Please be assured we continue to work on this as quickly and efficiently as possible, and will provide updates as soon as we have them.
More technical details and an update:
As a result of the RAID controller going bonkers and the fs issues, a number of perl modules and other things associated with perl got hosed. This is what was preventing MySQL from starting. As such perl had to be reinstalled, as did MySQL.
I suspect this is very close to being resolved, and you will all be updated soon.
Just waiting on 1 more update from cPanel folks.
All, I think we're good to go now, as Apache and PHP had to be recompiled to top this off.
CyberDezign
11-29-2006, 03:34 PM
Thank you for your diligent service, as always.
Kerrie Carre';)
Sorry for the "we think this will be resolved soon" update then the delay -- at that point everything was just about done. cPanel was telling we would need to have each customer re add their users to their databases, but we felt confident that they wouldn't need to after we simply imported the latest backup, which is what we did.
However, after importing the latest backup, users' databases were still not working. I went through all the motions of creating the database in cPanel, setting up the username and password for the database, and then assigning the privileges to the user and assigning the user to the db, but still could not get the test site to display properly.
Tom asked me for my thoughts on rebuilding Apache and PHP, and I told him I didn't think that would be necessary. I was wrong. He pressed forward and recompiled both, and now everything appears to be working again - without any work required on the part of a single customer.
As far as we can see, everything is up and running fine at this point. If you experience any issues for any reason (which you shouldn't, given we have a new perl install, MySQL install, Apache, PHP, etc), please do open up a ticket with our helpdesk.
Once again we profusely apologize for the issues from today. I am pleased to say that, per the usual, we had begun working on the issue before the first ticket was opened, and before the first phone call. Our monitoring system caught it within 60 seconds, and we went to work on it immediately. I can live with a few hours of MySQL downtime every 4 years. I don't like it, but..
Thank you for your diligent service, as always.
Kerrie Carre';)
You're welcome Kerrie. I was going to update you personally via your ticket as promised as soon as I got done with my last post above, but you beat me to it :)
CyberDezign
11-29-2006, 03:45 PM
Sorry to say this but Cpanel is still having a problem.:confused:
Error Message:
Internal Server Error
The server encountered an internal error or misconfiguration and was unable to complete your request.
Please contact the server administrator, webmaster@exoticrecovery.com and inform them of the time the error occurred, and anything you might have done that may have caused the error.
More information about this error may be available in the server error log.
Additionally, a 404 Not Found error was encountered while trying to use an ErrorDocument to handle the request.
Please open up a new ticket with us an I will provide you with a secure URL to use to submit your login details so that we can attempt to duplicate the issue, thanks.
Just FYI - I was able to before, and am still able to log into cPanel on the server without issue. We'll just right on the ticket when it comes in Kerrie.
edit -- er "jump right on the ticket" is what I meant to say.
FYI Jeffro,
The Internal 500 error only shows up when you try to go to site.com/cpanel or site.com/webmail. Logging in through the Myriadnetworks.com support page works fine. Also, thanks for fixing everything up!
Interesting. Thanks for that Will. I was only using SSL:
https://x.x.x.x:2083
For anyone having issues logging into cPanel, please try using SSL. For WHM (if you have a reseller account), you can use SSL via the following URL:
https://x.x.x.x:2087
Thanks Will!
I believe this has been fixed :)
CyberDezign
11-29-2006, 04:52 PM
I tried: http://www.exoticrecovery.com/webmail and I'm still getting "server encountered an internal error or misconfiguration (500 error).
Also tried: http://209.8.40.218:2082/webmail
to get a clients email and it won't login to webmail this way.
Just letting you know. I'm sure your head is about to explode by now.:eek:
Kerrie
ack, didn't think to try webmail.
Please use this for now:
https://site.com:2096
Will update as soon as possible.
BALDINOCOM
11-29-2006, 09:36 PM
When I noticed a problem today, I came here and saw that the issue was posted. Saved me the time of trying to figure out what the problem was.
Still, no complaints here. I can never say enough nice things about Myriad service and support. When I signed up, I was bouncing from one hosting company to another because every other day there were hours of down time.
Sure, it happens but every other day? Since I been here, I cannot remember the last time my sites were down. The folks at Myriad could teach others a thing or two about customer service.
As I was writing this, I checked in on my last hosting company - the domain is now a google adsense link site - LOL
Keep up the great work guys. I know in your line of work all your going to hear about are the things that are not working right - as you probably heard today. I just want you to know that I am very greatful for your service and the support you have provided for myself over the past few years and in the years to come. Have a happy holiday!
Regards,
Michael Baldino
Baldino.com
Thanks a bunch for the kind words Michael! I was talking to Tom on the phone as you posted this and mentioned it to him so he could have a look. He was very appreciative as well, as are the rest of the Myriad staff.
Today was not one we care to repeat for quite some time (ok - never), but I have to say that we too are very thankful for the understanding from all of our customers. Not 1 person who called or opened a ticket yelled, said anything negative, or anything else in that regard. I'm sure some folks may have been muttering under their breath as they anxiously waited for MySQL to come back up, and understandably so. However, we received a lot of kind words and support as we worked to get everything fixed.
That is what makes me the proudest to be part of Myriad Network :)
Thanks again Michael, and Happy Holidays to you too!
vBulletin® v3.7.1, Copyright ©2000-2009, Jelsoft Enterprises Ltd.