We've got a queue incident on the Mastodon instances.
Some users on plemora pushed HUGE mp4 videos, and the queue was stucked to process them.
That gave processes like 97.2% CPU and 300 Mb RAM running dozens of minute hour each (the server isn't optimized for video rendering).
And so while video convert occurred, all the workers are busy, and we don't have timeline updates.
No data is lost, as all is saved in Redis as jobs to process.
Timeline is currently updating again.
I increased the workers amount, provisioning a new machine to temporarily help to process the queue. With that, we currently process 30 tasks / seconds.
That means with 130 000 tasks to process, plus a little bit for the new tasks incoming, in 2 hours, queue will be empty.
Nasqueron is a budding community of creative people, writers, developers and thinkers. We focus on free culture, ethics and to be a positive change. We share values like respect, justice and equity.