this post was submitted on 24 Aug 2023
1 points (100.0% liked)

Azorius (software)

1 readers
2 users here now

This is about the software. Development, operations, etc. Even the occasional bug report.

founded 1 year ago
 

So I finally got tired of the hit and miss of using azorius on the same subdomain as my old lemmy install. I assumed it was a key issue so I killed the DB and re initialized the build, and am now hosting it under the new subdomain deddit.

I subscribed to all the same communities but this time I noticed when loading a new community no posts were shown, though the logs clearly show json dumps with posts from those groups. I didn't think anything of it and kept subscribing to my groups. Sometimes a couple of posts would show but usually zero posts.

Today, I see a few commeints but no new posts. Even though the groups are active with new posts, etc.

I see a lot of these entries in the logs:

2023/08/24 16:02:45 no post for comment: https://programming.dev/comment/2291160

Anything I can do to troubleshoot this? Are there any key files stored I should have wiped out or something? I did delete the entire -datadir and recreated it.

top 14 comments
sorted by: hot top controversial new old
[–] peter@deddit.petersanchez.com 1 points 1 year ago

Eyy, at least I see this posted correctly to the azorius.net group :)

[–] tedu@azorius.net 1 points 1 year ago

That's a pretty normal log entry. We don't (currently) attempt to fetch parents for comments. When you subscribe to a new group, you'll get a bunch of comments for posts you don't have, but as the group rolls over and new posts come in, everything should work.

I think you just need to wait a bit longer. If you're getting comments, even if we're not currently saving them, that's a good sign everything is working. The posts will get there eventually.

The slow start is not ideal, but I don't think it indicates anything wrong.

[–] tedu@azorius.net 1 points 1 year ago* (last edited 1 year ago)

How did you follow these groups? When you search for the group, it should pull the outbox. It skips this if you search for one post. But prog@azorius is empty. I didn't think that was possible. And I wrote it. :)

You found a bug! 🎉 Recent change to treat background refresh nicer ended up breaking things.

[–] peter@deddit.petersanchez.com 1 points 1 year ago (1 children)

Hah I knew it! Because I got like 4 new posts overnight when there had to have been 100+ for all my groups.

So yea, I search for the group by full url,it loads the page (but shows no posts, which didn't happen before) and I click the subscribe button. Logs show successful post to remote group inbox (which didn't work before on old subdomain).

So is the fix pushed? Will I need to unsubscribe and resubscribe?

[–] tedu@azorius.net 1 points 1 year ago

Everything should be good. The existing groups will backfill some posts after a day or two as well. Just make sure you have change 1e0138409da4.

[–] peter@deddit.petersanchez.com 1 points 1 year ago

OK, updated. I'll watch it over the coming days. Thanks!

[–] peter@deddit.petersanchez.com 1 points 1 year ago (1 children)

So is there some like polling that happens after X time? Because I notice I do get new stuff but it's bursts by groups. So like, the first 10 most recent are from a single group, the next 10 are from another group, etc. It seems like things aren't being pushed to me but pulled from Azorius.

[–] tedu@azorius.net 1 points 1 year ago (2 children)

Yes. Every day or so, we poll group outboxes looking for posts that we missed somehow. This also helps fill in groups that don't have any subscribers, and would appear dead otherwise.

The code is in refresh.go.

[–] peter@deddit.petersanchez.com 1 points 1 year ago

Thanks! I'll keep an eye on things but it seems I have a delay of up to 24 hours for any new posts in groups I'm subscribed to. I keep referring to world@lemmy.world because it's a pretty active group but it seems anything pushed to me isn't sticking. I'll dig around the logs a bit too and update if I find anything odd.

[–] peter@deddit.petersanchez.com 1 points 1 year ago (1 children)

After a big of digging through logs it seems it's lemmy.world that is not posting updates to me. Trying to unsubscribe produces this error:

2023/08/29 19:16:01 failed to post json to https://lemmy.world/c/selfhosted/inbox: http post status: 403

Are the keys stored in the database (azorius.db) or on disk? I did destroy the db and re-create it when moving to a new sub domain but maybe the keys survived if they're stored on disk somewhere other than the db? Could that be the issue (though I guess it doesn't make sense it works with other lemmy instances)

[–] tedu@azorius.net 1 points 1 year ago (1 children)

That's a little weird, but lemmy.world is always having problems. :(

In my testing, a bad key would result in a 400 status.

The keys are stored in the database. Every instance will have new ones.

You may be stuck for a bit in the delivery retry queue. After a failed response like 403, we'll save that and try it again for a while. But new messages will line up behind the failed message. So effectively, you can't talk to lemmy.world until the 403 message times out. This is by design, but if there's a federation problem, it's not great.

You can peek in the database at the deliveries table, and delete anything with a rcpt of lemmy.world. Then try subscribing again.

[–] peter@deddit.petersanchez.com 1 points 1 year ago

Well, I just went and unsubscribed and resubscribed and finally I got 200 OK responses and Accept posts back. So maybe things are all worked out now. Wish I could say what the issue or resolution was but as the saying goes, time heals all wounds.

[–] tedu@azorius.net 1 points 1 year ago (1 children)

Your trouble aside, what I like about this post is it inspired me to write the sub to post feature so I'd get notified of comments, and then you replied and I didn't get the notification because I mixed up the self logic.

[–] peter@deddit.petersanchez.com 1 points 1 year ago

Hah, happy to be of service