Service Announcement
March 11th, 2014
Local Domain Name Server (DNS) Resolution Failures
Some markets have continued to report network issues connecting to our servers. We first received reports of this on 3-08-2014. Depending on what you're accessing, we have servers in Dallas, New York, and Chicago. We also have monitoring all over the USA and Europe to help us maintain visibility over our network traffic and up-time.
Our services have been 100% up throughout all reports since last Saturday. See for yourself, here:
https://rpm.newrelic.com/public/charts/bkGRgJEaIAM
and here:
https://rpm.newrelic.com/public/charts/i1bl2Uv3Vup
The response time has consistently trended at around a quarter of a second. We use that to show that we're responding to requests consistently without interruption.
We've been informed by our hosting provider that Level 3 has reported incidents impacting multiple regions, but we've taken steps to find a workaround for those affected.
For those experiencing issues:
So your internet connection tries to find a path to our servers when you type signal88.inteliguide.net into your browser address bar. This is how all domain names resolve from their names (like google.com) to their machine IP addresses (like 74.125.239.98). Sometimes the details get lost in translation if records are changed too often, or not cached properly, or the network en route from your computer to our servers breaks down somewhere along the line.
That's essentially what's been happening. Local DNS has been unable to connect to our servers, meaning there's a breakdown in communication somewhere between your ISP and our service. This is beyond our control, but there is something we can suggest. Use a national DNS provider!
You will need Admin access to your Toughbook or desktop computer to complete this workaround.
- Go here for Windows: http://answers.microsoft.com/en-us/windows/wiki/windows8_1-networking/how-to-change-ipv4-dns-server-address-to-public/56548e8d-eb9c-4fc5-807c-eefca5d278d0
- Here for Mac: http://www.howtogeek.com/howto/38793/how-to-switch-mac-os-x-to-use-opendns-or-google-dns/
And follow the walkthrough to enter Google as your DNS provider on your active network connection.
This is not just a temporary solution. You can keep it in place to help avoid this sort of issue in the future. In the eight years that we've been providing web services, this is only the second time this has happened, so it's generally been a very rare occurrence, but this workaround will help if/when it happens again.
Note: This is a confirmed workaround, but will only apply in cases where DNS is failing to resolve from your ISP. If there's a broader internet connectivity issue, or you lack any solid connection to the internet, this workaround does not apply.
Our response
We received the following from our hosting partner late Saturday night in response to tickets we logged shortly after hearing our first reports of issues:
"I was informed that there have been multiple notifications that the Internet Service Provider 'Level3' (which is a major back-bone provider) has had issues world wide, with severe packet loss to other providers noted last night and earlier today. These backbones providers included Cogent, Sprint, Savvis, ATT, and Verizon."
We're continuing to follow up with these providers to see if we can get an ETA on resolution of these issues, and will post updates here as they come in.
1 Comments