Dereckson boosted

Maybe we were beautiful- the way the first waves of tsunami are, the way the flames consuming dying embers are, the way two stars before collision- like a tragic wildfire, destined to consume.

- poetry by me!

@benoit Online has always had issues with their consoles.

There is a fun variant: when the serial console rate is too fast and the OS logs out the console session every 20 seconds.

You've 20 seconds to do your task :p

Queue backlog has been fully processed at 19:46 UTC.

Since then, our instance process events normally.

The home feeds have been rebuilt so we're sure they are coherent and in chronological order.

All looks good since.

@thaj @johnnynull There is still 95 402 jobs to process

Here how it works:
1. it pulls the status
2. it prints it on the federated timeline
3. it populates new jobs for notifications, timeline updates, check links, check replies, etc.

I'll rebuild all the home timeline when the queue is at's 0.

@StuC @lizsmells @thaj Yup that was the main issue: DM is also a timeline to push to.

So the big question, how to prepare a Nagios check triggering when a job is stuck in a Sidekiq queue after a delay.

And a check the queue is greater than a specified threshold.

Timeline is currently updating again.

I increased the workers amount, provisioning a new machine to temporarily help to process the queue. With that, we currently process 30 tasks / seconds.

That means with 130 000 tasks to process, plus a little bit for the new tasks incoming, in 2 hours, queue will be empty.

We've got a queue incident on the Mastodon instances.

Some users on plemora pushed HUGE mp4 videos, and the queue was stucked to process them.

That gave processes like 97.2% CPU and 300 Mb RAM running dozens of minute hour each (the server isn't optimized for video rendering).

And so while video convert occurred, all the workers are busy, and we don't have timeline updates.

No data is lost, as all is saved in Redis as jobs to process.

@TheFerridge (stats for the main queue, that's why it's below the 119k from the previous tweet)

@TheFerridge We decreased from 117 259 to 114,660 last hour.

At this rate, all should be in order tomorrow.

I'm currently trying to see how to speed up the process running more workers.

@TheFerridge Hi, we're actually well connected.

The issue is the federated timeline isn't updated as the update jobs are actually queued (there are currently 119 372 jobs to process).

And the operations are stucked.

@deadsuperhero yeah.

On IRC we also use youtube-dl to get video Metadata if a link is posted without title information

@quota_atypique Pas spécialement.

Ennuier 10 000 étudiants pour les dizaines de millions d'€ de budget qui en réalité viendront de bourses de l'état et du fonds social des universités ça vaut le coup ?

@quota_atypique donc en gros on a des règles compliquées pour juste 5% ...

Show more
Social Nasqueron

Nasqueron is a budding community of creative people, writers, developers and thinkers. We focus on free culture, ethics and to be a positive change. We share values like respect, justice and equity.