351 559 tasks has been processed this Sunday, our queue is now fully processed.

Our instance had issues with Redis cache and then PostgreSQL indexes.

The issues are solved, and the situation is monitored.

@lizsmells Thanks for your support :) The donation is appreciated.

#maintenance Show more

Dereckson boosted

Maybe we were beautiful- the way the first waves of tsunami are, the way the flames consuming dying embers are, the way two stars before collision- like a tragic wildfire, destined to consume.

- poetry by me!

Queue backlog has been fully processed at 19:46 UTC.

Since then, our instance process events normally.

The home feeds have been rebuilt so we're sure they are coherent and in chronological order.

All looks good since.

So the big question, how to prepare a Nagios check triggering when a job is stuck in a Sidekiq queue after a delay.

And a check the queue is greater than a specified threshold.

Timeline is currently updating again.

I increased the workers amount, provisioning a new machine to temporarily help to process the queue. With that, we currently process 30 tasks / seconds.

That means with 130 000 tasks to process, plus a little bit for the new tasks incoming, in 2 hours, queue will be empty.

We've got a queue incident on the Mastodon instances.

Some users on plemora pushed HUGE mp4 videos, and the queue was stucked to process them.

That gave processes like 97.2% CPU and 300 Mb RAM running dozens of minute hour each (the server isn't optimized for video rendering).

And so while video convert occurred, all the workers are busy, and we don't have timeline updates.

No data is lost, as all is saved in Redis as jobs to process.

Hey a green fireball-like spotted from E to ESE :) Cool view.

Dereckson boosted

So a kick-start, it's a badge, a bingo card and some challenge words to put in the story.

Still one Python workshop to give then I can go to our local kick start event.

An evening to eat chat play board games meet each other. Then at midnight we start writing.

@Fralambert Depuis la mise à jour, j'ai l'impression que la colonne Local timeline se charge sans aucun souci (mais vu que c'est la première qui s'affiche parfois, j'imagine que c'est juste qu'elle est en cache du navigateur en local storage ou autre ?)

Show more
Social Nasqueron

Nasqueron is a budding community of creative people, writers, developers and thinkers. We focus on free culture, ethics and to be a positive change. We share values like respect, justice and equity.