Site is slooooow

[Jimbo]

Still working on things... the search index was corrupt so I am reindexing it as we speak.

Thank You Scott !

Blue line went across the top this time, if that tells you anything.

 

[Scott Greczkowski]

I have to say it is getting frustrating for me. I have the entire site running great and leave it alone. Then wake up one morning and everything is running like crap... and I didn't touch anything.

The issue we had a few weeks ago was a hardware issue.

The stats software was running this morning and was using a lot of CPU. That normally runs at 3 in the morning and normally only takes a few minutes to run. I do not know why it was stuck. I killed it and the speed came back. But when I got to work I see speed issues are evident again.

I am going through the server again trying to figure out what's going on but I am also trying to work my real job as well.

I always want this place to run at the best speeds possible, and as we have seen normally it does. It just pisses me off when it does not and I have no reason why.

 

[TRG]

Ok I am here at work, and still not as snappy as I would like.

It appears the databse server is trying to repair the search index. The process list is showing

Repair by sorting

REPAIR TABLE `xf_search_index`

I would like to kill it, but I am reading to leave it alone and let it finish. As killing it can cause other issues.

I am keeping an eye on it.

BTW a new version of Xenforo (our forum software) was released yesterday. I am holding off for about a week to make sure it is stable.

Any word from the developer regarding the advanced search feature we had before? I really miss that.

 

[mackie99]

Still slow for me. It took about a minute just to open this post.

Edit...took about 30 secs. to post this.

 

[Scott Greczkowski]

Yes seeing it here... letting the search rebuild finish

 

[Scott Greczkowski]

I am starting to think the issue is at the ISP or with the hardware again.

Why am I thinking this?

I am watching the search index rebuild normally it does 500 posts every 3 seconds. Now its doing 500 posts every 45 seconds.

So reading around someone suggested checking the hard drive speed. We are on a server with NVME drives so the drive speeds should be flying.

However if I do a test, the speeds are lower than a regular standard hard drive. Something is not right.

dd if=/dev/zero of=/tmp/tempfile bs=1M count=1024 conv=fdatasync
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 12.7285 s, 84.4 MB/s

12 seconds to copy a 1 gig file at 84.4 MB/S does not seem right to me.

Any of you unix guys what are your thoughts?

I am going to lower the TTL on our DNS just incase, and if they offer to move us to a different datacenter I am going to pick the East Coast.
(I have not put in a ticket yet with them.)

 

[ncted]

W

I am starting to think the issue is at the ISP or with the hardware again.

Why am I thinking this?

I am watching the search index rebuild normally it does 500 posts every 3 seconds. Now its doing 500 posts every 45 seconds.

So reading around someone suggested checking the hard drive speed. We are on a server with NVME drives so the drive speeds should be flying.

However if I do a test, the speeds are lower than a regular standard hard drive. Something is not right.

dd if=/dev/zero of=/tmp/tempfile bs=1M count=1024 conv=fdatasync
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 12.7285 s, 84.4 MB/s

12 seconds to copy a 1 gig file at 84.4 MB/S does not seem right to me.

Any of you unix guys what are your thoughts?

I am going to lower the TTL on our DNS just incase, and if they offer to move us to a different datacenter I am going to pick the East Coast.
(I have not put in a ticket yet with them.)

Well, the NVMe SSD in my desktop, which is not enterprise-grade, is almost an order of magnitude faster:

$ dd if=/dev/zero of=/tmp/tempfile bs=1M count=1024 conv=fdatasync
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 1.41086 s, 761 MB/s

 

[navychop]

We could burn some butter, bow to the East and pray to the West. If you think it’d help.

Sometimes, it seems that….

 

[FTA4PA]

I was just going to post about site slowness this morning but see Scott is already trying to address it. Good Luck!

 

[comfortably_numb]

Site seems to be coming back online... a half hour ago I couldn't log in at all

 

[Scott Greczkowski]

Site seems to be coming back online... a half hour ago I couldn't log in at all

Its not like it should be. Still working.

Just lowered all the DNS TTL just in case.

If we continue to have issues and cant post here I will start posting updates to our SatelliteGuys X / Twitter account. (Yeah I know a lot of people hate that place but its the only place where I can easily update everyone)

In doing more tests, I will be rebooting the server in a few minutes, in which we will be unavailable for a moment.

 

[Scott Greczkowski]

Server Reboot Complete.

 

[FTA4PA]

If we continue to have issues and cant post here I will start posting updates to our SatelliteGuys X / Twitter account. (Yeah I know a lot of people hate that place but its the only place where I can easily update everyone)

Any possibility of shooting out a mass email - to those who are interested in receiving it - regarding down times and progress updates? I don't mind visiting X/Twitter to see a post from time to time but the last few times I did when this was happening it was requiring me to set up an account before I could view it, which I will not do.

Anyways, just a suggestion to see if it is an option. Thanks for all you do Scott!

 

[Scott Greczkowski]

I don't believe you need a login for Twitter (they required it for awhile) Try going to x.com and see if you can see the posts without logging in.

If I sent Mass Emails they would come from this server. And last time I sent out a mass email we sent so many emails were were automatically added to many email blacklists. So I don't want to do that again. And with 180,000 emails to send it would take a long time as well.

 

[Tampa8]

This has been over a few days not slow enough to report every time but noticeable...

 

[mackie99]

Still very slow.

 

[FTA4PA]

I don't believe you need a login for Twitter (they required it for awhile) Try going to x.com and see if you can see the posts without logging in.

If I sent Mass Emails they would come from this server. And last time I sent out a mass email we sent so many emails were were automatically added to many email blacklists. So I don't want to do that again. And with 180,000 emails to send it would take a long time as well.

No dice. Still requires an account to see your SG posts. Here is a screenshot after following your link:

 

[Scott Greczkowski]

Definitely noticeable today.

Still running tests but still appears to be hardware / hard drive related.

If I tell them to move us to another region I do not know how long it will take. Again I will update on X / Twitter as necessary. (And again I know (and understand) many people don't like that site, but unfortunately its my best option.

 

[Scott Greczkowski]

Ticket has been submitted. To the ISP.

God only knows how long it will take them to answer it.

 

[comfortably_numb]

I don't believe you need a login for Twitter (they required it for awhile) Try going to x.com and see if you can see the posts without logging in.

Doesn't work for me