Lemmy Server Performance

2 readers

1 users here now

lemmy_server uses the Diesel ORM that automatically generates SQL statements. There are serious performance problems in June and July 2023 preventing Lemmy from scaling. Topics include caching, PostgreSQL extensions for troubleshooting, Client/Server Code/SQL Data/server operator apps/sever operator API (performance and storage monitoring), etc.

founded 1 year ago

MODERATORS

RoundSparrow@lemmy.ml

RoundSparrow@bulletintree.com

How lemmy.ca took on finding why PostgreSQL was at 100% CPU and crashing the Lemmy website this past weekend. PostgreSQL auto_explain (lemmy.ml)

submitted 1 year ago* (last edited 1 year ago) by RoundSparrow@lemmy.ml to c/lemmyperformance@lemmy.ml

6 comments fedilink hide all child comments

IIRC, it was lemmy.ca full copy of live data that was used (copy made on development system, not live server - if I'm following). Shared Saturday July 22 on GitHub was this procedure:

...

Notable is the AUTO_EXPLAIN SQL activation statements:

LOAD 'auto_explain';
SET auto_explain.log_min_duration = 0;
SET auto_explain.log_analyze = true;
SET auto_explain.log_nested_statements = true;

This technique would be of great use for developers doing changes and study of PostgreSQL activity. Thank you!

top 6 comments

sorted by: hot top controversial new old

[–] Shadow@lemmy.ca 7 points 1 year ago (1 children)

Yeah this was the only way I could get visibility into the queries running inside the triggers.

[–] RoundSparrow@lemmy.ml 5 points 1 year ago (1 children)

Thank you for sharing, I had not come across the auto_explain technique and was manually pulling out SQL from pg_stat_statements and trying to match up to runtime parameters for $1 $2 etc with EXPLAIN

[–] Shadow@lemmy.ca 4 points 1 year ago (1 children)

Oof yeah, that's not gonna be a fun way to troubleshoot!

Thanks for taking the ball and running with this, looks like it's getting a lot of attention now and hopefully next release will have some juicy sql improvements!

[–] RoundSparrow@lemmy.ml 2 points 1 year ago (1 children)

I've really been bothered by the server crashes, especially on lemmy.ml since even before I created my first Lemmy account. so far, I haven't seen anyone able to reproduce the overloads/crashes outside of production servers.

If we can get pg_stat_statatements with on 0.18.3 with pg_stat_statements.track = all out of a real Lemmy database run for 12 or more hours, that is having crashes, I think we might find something that's being overlooked.

[–] Shadow@lemmy.ca 3 points 1 year ago (1 children)

We don't get any crashes on ca since I disabled account deletion. Things are stable.

[–] RoundSparrow@lemmy.ml 2 points 1 year ago

ok. back to the drawing board.