Reindexing of posts and topics after upgrade

[Tom_K]

I upgraded from 4.1.13.2 to 4.1.14 and now my board has been reindexing topics and posts for more than 24 hours.

This has happened on one of the previous minor version upgrades as well.

Why is reindexing needed so often? It's an issue for large boards. Will reindexing be needed for each upgrade in the future?

[Danny Tipple]

Same as above.  Big issue on my board with 2.2million posts.

I have background tasks to run via CRON every minute but the loop size seems too small to do this efficiently. Is there a command line tool i can run to process them constantly until complete?

[Charles]

The cron script automatically loops itself. Just run it every minute and let it go. A 2.2 million post rebuild should only take 6 - 8 hours.

[Tom_K]

19 minutes ago, Charles said:

The cron script automatically loops itself. Just run it every minute and let it go. A 2.2 million post rebuild should only take 6 - 8 hours.

Yeah, in theory. I contacted support a while ago (ticket #965772) and they said everything appears to be OK but it's still slow. Never solved the issue.

[Danny Tipple]

2 hours ago, Charles said:

The cron script automatically loops itself. Just run it every minute and let it go. A 2.2 million post rebuild should only take 6 - 8 hours.

The cron script appears to execute once (~5-10 seconds) then exit.  The it will run again on the next minute.  Meaning it takes an awful lot longer than it should.  The site also seems to slow while its running as if tasks are also running inline?

I'm currently running the following which is running them much much faster:

while :; do /usr/bin/php -d memory_limit=-1 -d max_execution_time=0 /path/to/board/applications/core/interface/task/task.php <cron_key> <background_task_id>; echo -n '.'; done

 

[Charles]

If it's executing and exiting that quickly then either your queue task is locked or something else is wrong.

Doing a loop like that on the cron will not actually make it run any faster because:

1. Only one cron can run at a time so if the previous one is not done running the next execution will not start
2. The cron should run at least 45 seconds between goes all on its own as it does internal looping.

If you are having any issues please submit a support ticket.

[maddog107_merged]

1 hour ago, Charles said:

If it's executing and exiting that quickly then either your queue task is locked or something else is wrong.

Doing a loop like that on the cron will not actually make it run any faster because:

1. Only one cron can run at a time so if the previous one is not done running the next execution will not start
2. The cron should run at least 45 seconds between goes all on its own as it does internal looping.

If you are having any issues please submit a support ticket.

I saw the same behavior on my ~4million post board, where the script only runs for 5-10 seconds then exists, I have a fairly powerful machine (24 threads, 64gb ram, SSD's) so I assumed since it only processes a few posts at a time (50?) it finishes quite quickly. I setup cron to run every 10 seconds ( sleep 10; php......,   sleeep 20, php .....)

Just FYI, it works well for me. 

[Charles]

2 hours ago, maddog107_merged said:

I saw the same behavior on my ~4million post board, where the script only runs for 5-10 seconds then exists

That is not how it is supposed to work. If you have the ability to reproduce please submit a ticket so we can see.

[Danny Tipple]

I ran the loop simply in a screen session to complete things quicker.  I just timed it and apologies it does take 30-40 seconds per run when specifying the 'background tasks' task id (running it without the task id seems to favour other tasks which run much quicker?).

Even at 40 seconds its still wasting 33% of the time it could be processing.  It would be nice to get a proper task we can run in a terminal after upgrades which gives some output and feedback.