Upstream Provider Issues

[Jeff]

Hi,

Approximately 30 minutes ago a customer stated that they were experiencing issues reaching their websites. Since the servers are up and I was able to reach each of the sites without issue from an offsite location (not connected to Myriad Network), I asked the customer to provide me with a traceroute. The traceroute indicates that our upstream provider is currently experiencing issues. This means that, the path that is taken from your computer at work or at home to reach your website may not be working, but others are.

I have contacted BTN at 9:29PM EST, to which they responded at 9:30PM EST stating they are investigating the issue and will update us as soon as possible. As soon as we know something, we will post it in this thread.

I apologize for any inconvenience - this issue is out of our control, as the network that is currently experiencing issues does not belong to us, and as such is not managed by us. More to come as we hear from BTN.

[tomp]

A quick update. This appears to be affecting some folks but not others. We are holding with our provider to determine the cause and work towards a speedy resolution.


Thank you for your patience.

[Jeff]

Hi,

After speaking with BTN, they stated that after clearing the arp table on one of the routers everything started working fine again. Apparently it was cleared about 30 minutes ago. Once again we do apologize for any inconvenience this may have caused.

[Jeff]

2 customers have reported this issue earlier this evening. I am waiting for a response from one, and the other I spoke with about 20 minutes ago after I was informed they were still having issues. BTN has been contacted again and we are currently waiting on a response from them. Updates will be posted in this thread as soon as possible. Again we apologize to those that are affected by this. It appears some are, while others are not.

[Jeff]

The latest is that BTN believes the issue is resolved. If you are still experiencing issues, please open a ticket with our helpdesk via one of the following methods:

1. http://my.myriadnetwork.com

2. email support@myriadnetwork.com

In the ticket, you will need to provide the following:

1. A traceroute to your site:
- Go to Start
- Go to Run
- Type: cmd
- Hit "OK"

- Type: tracert yoursite.com
- Copy and Paste the results in the ticket


Thank you.

[Jeff]

To further elaborate on this issue, which should be finalized well by now, I will post the email we received from BTN about an hour ago, then explain what it means:



I've isolated this to one of our Savvis links which is up/up but not passing
traffic and is still a part of the 4Gig link bundle to them because of an
active status on the link.

I've removed this faulty GigE to them and everything should be working fine
now.

Unfortunately the link didn't drop and the route table still had it as a
part of 4gig link bundle to them and was trying to route traffic across this
faulty link which is what was causing problems for you.

Our NOC will contact Equinix and Savvis now to isolate the problem with this
faulty link and once again sorry for this inconvenience caused on something
which was not so predictable and easy to catch.



In short, concise, and understanable terms, the issue was this: there was a router (http://en.wikipedia.org/wiki/Router) on the Internet which was faulty. Although it should have been flagged to not be used, so to speak, it was still showing as active. As such, other routers were attempting to pass traffic through it, not realizing it was not functioning properly. Had it been flagged as being unoperational, the routers that attempted to pass traffic to it would have started using other routes/routers for passing traffic.

I would just like to reiterate that this issue did not affect everyone. In fact, we only had 2 support requests open for this issue. The issue was on the Savvis (http://www.savvis.net) network - Myriad Network was operational per the usual.