Board timeouts
0

20 posts in this topic

Board timeouts

Hi all,

Many have already seen these, but in case you haven't - we are currently seeing brief slow downs and timeouts on the boards on a somewhat regular basis. We filed a ticket with the board hosts on Wednesday 4/12 to resolve this when their frequency seemed to go up.

Initially we needed to isolate the problem to the board host, as they suspected at first that our CDN (network infrastructure that improves performance) was the problem. We didn't think it was, but we needed to prove that to them.

We also began tracking the outages by polling the boards with an automated tool once per minute in order to establish to the hosts the frequency / urgency of the situation. We're seeing something like 8 of these timeouts or slow downs per day right now lasting between 1 and 5 minutes.

Tech support at the board host asked us to uninstall a third party plugin, and we did some trouble shooting with other plugins to try to resolve the issue, but those efforts didn't resolve the problem.

We have done some diagnostics to try to isolate the problem and focus the troubleshooting efforts. They've been investigating database load, but we believe the problem to be at a higher level (at the web server or network level for those technically inclined, because even a TCP handshake fails).

At this point they have escalated the ticket to higher level resources on their end and are continuing to pursue it.

We're sorry for the inconvenience these slow downs are causing, and we're working to get this resolved as quickly as possible.

 

Share this post


Link to post
Share on other sites

Excellent, thank you Arch (thumbsu

Do keep us posted...

Share this post


Link to post
Share on other sites

Thanks for the update Arch!

Share this post


Link to post
Share on other sites

Posted (edited)

3 hours ago, Architecht said:

Initially we needed to isolate the problem to the board host, as they suspected at first that our CDN

Damnit! I knew it was the Canadians  

BowGIeCLedzY0QDXTxCP.png

Edited by Jerkfro
BlowUpTheMoon and Senormac like this

Share this post


Link to post
Share on other sites

I'd say the frequency of slowdowns is approaching, if not surpassing, the worst days on the old boards.  I know there have been several times in the past week when I've given up after waiting a dog's age (translation:  30 seconds or so!) for a post or PM to go through.

Share this post


Link to post
Share on other sites
1 hour ago, Sqeggs said:

I'd say the frequency of slowdowns is approaching, if not surpassing, the worst days on the old boards.  I know there have been several times in the past week when I've given up after waiting a dog's age (translation:  30 seconds or so!) for a post or PM to go through.

Agreed - just endured yet another slowdown, this login was a last attempt after repeated failures.

Hopefully this issue can be resolved sooner rather than later.

Share this post


Link to post
Share on other sites

Posted (edited)

On 4/18/2017 at 11:21 AM, Jerkfro said:

Damnit! I knew it was the Canadians  

BowGIeCLedzY0QDXTxCP.png

:acclaim:You guys south of the border blink slower so it doesn't effect you as much. 

Edited by Silver Surfer

Share this post


Link to post
Share on other sites

Just got 504'd twice in succession. :pullhair:

Share this post


Link to post
Share on other sites

Invision has made an adjustment that they think will let certain queries run faster. I don't know if that's truly the source of the problem, and I haven't had enough time for monitoring to confirm/deny improvement, but I wanted to update everyone that progress is continuing.

I am continuing to run independent monitoring so that if this doesn't resolve things, we will know it.

 

Share this post


Link to post
Share on other sites

Thanks for letting us know. My mobile experience has definitely suffered lately.

Share this post


Link to post
Share on other sites
On 4/18/2017 at 5:24 PM, Sqeggs said:

I'd say the frequency of slowdowns is approaching, if not surpassing, the worst days on the old boards.  I know there have been several times in the past week when I've given up after waiting a dog's age (translation:  30 seconds or so!) for a post or PM to go through.

+ 1. I think this is the first time I have ever even brought up one negative thing about the boards. Hopefully you guys can resolve this issue :wishluck:

Share this post


Link to post
Share on other sites

I've gotten 3 diff 504 errors in last hour, fyi

Share this post


Link to post
Share on other sites
On ‎4‎/‎18‎/‎2017 at 4:24 PM, Sqeggs said:

I'd say the frequency of slowdowns is approaching, if not surpassing, the worst days on the old boards.  I know there have been several times in the past week when I've given up after waiting a dog's age (translation:  30 seconds or so!) for a post or PM to go through.

for me it's no comparison, never had problems like this on the old boards. Missed out on some sweet books this week in a thread :pullhair:

Share this post


Link to post
Share on other sites

The Boards were inaccessible from 20:50 (UTC) until 21:04 (UTC) on 4/20.  I posted the ten screenshots in the "I hate the new boards" thread in Comics General.

error.png

 

error10.jpg

Share this post


Link to post
Share on other sites

 

Share this post


Link to post
Share on other sites

504 is the new 404 :banana:

Share this post


Link to post
Share on other sites
15 hours ago, Marwood & I said:

 

All quiet on the pictures of time outs front.  Fingers crossed, the problem has gone away and we can all get back to posting high quality comic related observations free from the terrors of Billy Buffering ?

Share this post


Link to post
Share on other sites

Just an update for everyone.

It looks like we have turned the corner on this issue. We did have an outage on the morning of 4/22. This was actually not an outage due to the board software, but was at a different level of network infrastructure. Sorry for that inconvenience, but the good news is that the more intractable problem of regular board timeouts caused by database load seem to be behind us. Other than the known issue on 4/22, we've had pretty clear sailing according to the web monitors.

 

Share this post


Link to post
Share on other sites

Promise?

Share this post


Link to post
Share on other sites

And here I thought this thread was about posters getting sent to the corner with a dunce cap . . . :)

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now
0