I found FediSearch, and also this post basically saying that a fediverse search engine would just be used as a tool by trolls.
this post was submitted on 21 Dec 2023
270 points (97.5% liked)
Fediverse
28480 readers
737 users here now
A community to talk about the Fediverse and all it's related services using ActivityPub (Mastodon, Lemmy, KBin, etc).
If you wanted to get help with moderating your own community then head over to !moderators@lemmy.world!
Rules
- Posts must be on topic.
- Be respectful of others.
- Cite the sources used for graphs and other statistics.
- Follow the general Lemmy.world rules.
Learn more at these websites: Join The Fediverse Wiki, Fediverse.info, Wikipedia Page, The Federation Info (Stats), FediDB (Stats), Sub Rehab (Reddit Migration), Search Lemmy
founded 2 years ago
MODERATORS
It's worth noting that since FedSearch, Mastodon has actually natively implemented opt-in search on posts.
That’s a good point. But those people can be banned? I guess Reddit handles this by moderation and archiving old posts.
Yes, but moderation teams on the fediverse are very small, and by nature of it, can make hundreds of account of different servers all trailing that would need to be individually sought out and banned.
It is a game of cat & 100 mice
People will take the harassment off site especially if they are dedicated enough or use it to scrape for potential personal info to publicly release.
How is that different from Reddit? If trolls want to search and scrape and find information on people, they're going to. You can't put your information on the open Internet and not appreciate there's always a danger of that.
load more comments
(1 replies)
load more comments
(13 replies)
Well, please make sure it respects post privacy at least but also realize that on the microblogging side of the fediverse, they may not take kindly to this prospect at all. People who start these kinds of projects are often harassed or at least receive passive hostility. Making it opt in instead out of opt out in some capacity is best.
I disagree. Post privacy sure, but the internet is by definition public. Anything you put out there can be used for pretty much everything, the original rules of the internet apply. I'd be happy to see an easy opt out on the engine to remove yourself, but if everything is opt in it'll never get off the ground.
That's not how the fediverse functions and approaching it that way is a problem waiting to happen. I'm stating so as a warning to be mindful of the culture of the way the fediverse itself functions. This is not Reddit, we share the fediverse with other software with different uses and features and we need to be mindful of that especially when building these kinds of tools. Making it opt out not only places a burden on smaller instances but presents a potential harassment risk for instances with vulnerable people on other fediverse platforms. As well, it is contrary to the entire way specific other activitypub instances operate. The fediverse is like a city we share with others, if Lemmy is not mindful of that city's culture then people will promptly give them the boot.
I'm not saying user by user opt in either, but instance by instance. Lemmy needs a tool of archiving especially. There is already cultural clashes I see occurring with the rest of the fediverse. Post like these of potential tools when it seems like the creator doesn't know the messy history behind previous projects like them in the fediverse make me fearful of the clashes coming to fruition.
Well that’s why I’m asking for input. And I won’t launch this on every instance without letting them know. Baby steps.
My matrix is open if you want/are actually interested in doing this in a way that won't make the rest of the fediverse flip shit. I support this tools creation especially for lemmy, but if it isn't done the right way it'll be received poorly. Making it behave differently on lemmy compared to other software as well might be an idea too.
But ActivityPub already publishes all of the data out. I don't think this is going out to servers asking for data, it's listening to public data being broadcasted out. If people are broadcasting over activitypub then they're okay with it being shared.
If they don't want it shared then they don't have to publish ActivityPub to anyone. They can defederate from the search federation. Those tools already exist.
As the fediverse is almost exclusively run by volunteers that are paying server bills and being admins, I could see some larger instances not taking kindly to this, especially depending on how much stress it would be putting on some already at capacity servers.
load more comments
(5 replies)
Why wouldnt people want do have search engine? Without it Fediverse stands no chance against non-free internet. Everything posted here would be much more valuable if it was searchable. Now comment posted once is viewed only until post gets less popular. Any other site of this kind displays answers decades old. Privacy isnt issue as everything posted here is available to everyone on internet.
I mean they are posting on the public internet, they should know that it can be read by anyone. I like the idea of users opting out.
If you give users the choice to opt out, all the privacy-focussing communities won't be searchable.
What if someone who opts out posts a comment and someone who opts in answers?
The Fediverse is public, a search engine doesn't show anything that isn't already open for anyone to see.
Good point. They should know they are making public comments. If you want it private then send a private message.
Is this something you can point yacy at?
I heard it’s not optimized well but I’ll take a look at it.
I'd be interested to know what "not optimized" means.
I heard it will send out requests as fast as possible essentially creating a DOS attack but I could be wrong. Also the UI can use some improvement.
You should federate the search engine so that folks can defed from the search as desired.
But then we would need a search engine for all the search engines...
Everything you post on the Fediverse is public.
If you don't want to show up in an internet search, post your stuff on your private server and only give access to the people you want to invite.
Yep
I love the idea, especially from a technical standpoint!
How big is the fediverse today? How many posts are there? What kind of algorithms atmre you using to store the results? Do you scan sites and then their connected sites or do you have a premade list?
More technical information please 😊!
The fediverse is a few thousand servers, from Mastodon, Lemmy, etc. Can't say the amount of posts but there are a lot.
So on the more technical side, I plan on using a light weight fast search engine called Sonic (It's written in rust). I have already used it in other projects and it can handle billions of messages / posts. But it has a cost it doesn't have faceted search, like for example if you want to exclude certain texts from the results. I think this is a fair trade off. The other solution would be to use something more mature like ElasticSearch but it'll be expensive (I'm assuming not much money will be made from this and I'm talking about donations)
For scanning sites there are premade lists to start with and it'll be possible to scan new sites from other instances if found. So a bit of both.
There is already Sepiasearch specifically for Peertube.
It's limited to only Peertube and it's not the most intuitive. I want to work with them on expanding this.
I support bigger picture. Rather than an independent site, wouldn't it be more practical to work with current fediverse app developers for lemmy, mastodon, etc to integrate search engine within the app?
I’m reaching out to see their thoughts. But there are limitations to what they can index.
The fedi elite will snipe you from a bell tower.
I don't know anything about the technical side of this. But I would (possibly naively) think that it would be simpler to have a filter that you could automatically apply to sift bog-standard search engine results for Fediverse instances? Like adding "site:uk" to the end of a normal search, except that your filter term would check a list of Fediverse instances to return the relevant results.
And make it an app/add-on so that people can use it with their usual search strategies.
People been doing that now but it comes with limitations. Thanks for the input.
view more: next ›