Technology

I've recently been looking into downloading offline copies of important data, since I don't expect that today's freely available information will continue to be freely available and accessible in perpetuity.

One problem I quickly ran into was that e.g. wikipedia downloads are not in an easily browsable format.

I found a project called Kiwix that packages datasets from a variety of free sources, like Wikipedia and Project Gutenberg, along with a reader application that can read these "zim" archives. The different data sources are available via torrents or direct downloads. https://wiki.kiwix.org/wiki/Content

I'm particularly interested in freely downloadable archives of scientific papers. A lot seems to be paywalled, or at least free-account-walled, even though the papers themselves are theoretically open-access. I would love to know of any sources out there to download an entire database locally.

I made a Hacker News post about human knowledge databases and got some interesting data I'd like to share back to this thread.

• Wikipedia Single 100GB File Thanks to @hersh@literature.cafe
• http://sci-hub.wf/
• Arxiv Download Script
• https://wholeearth.info/
• https://the-eye.eu/

Related:

Major cyber attack could cost the world $3.5 trillion

Bringing the big brain out on me! This is off the cuff.

1. I was not aware of Universal Design principles but a quick look suggests Principle 5: Tolerance for Error is most applicable to Software Engineering. Why not 1-5 bedrock? Because, in my opinion the general state of software is that it is more functional yet just as unreliable in decades past. What is the first thing a little experienced user to do when an error occurs? Yeah, they quit. No access. The micro-service paradigm has made the situation emminently worse as even finding the blame/responsibility for "no access" is fruitless.

2a. With anything of this type, the most obvious risk is to my own reputation. Security is a field burdened with responsiblity, people come to rely on it, what if they get hacked using my repo? I only took on things I spent months understanding and testing absolutely everything by hand. I limited myself to only distributions I could juggle, use daily, so I could be responsive to needs.

2b. Risk is competing objectives. FreeBSD and thus it's reliants, Ghost and Dragonfly, are in a strange position right now. FreeBSD is Linuxifying itself and adding more Corporates Sponsorships than ever in a path away from traditional BSD security. This presents itself a potentially competiing ethos situation for me, but not yet.

2c. OpenBSD is used by world security intelligence agencies and I hear the DoJ. Am I without my knowledge picking sides here and favoring some entities over others? Famously DARPA and FBI backdoor right? I researched the OpenBSD Sponsorship list carefully and asked around. The OpenBSD availability (at least of the version we use!) is equitable and I purposely put out an OpenBSD honeypot to see which entities would try to compromise it! Results: Fair.

3. This is a can of worms because what we are really talking about is the Linux-Effect. Started out community home-grown to now be a Corporate Globally Mega-Corp sponsorship vehicle estimated to be worth $100 Billion. Even Apple is now a Silver Linux sponsor. What am I saying is Corporate dominance is think-tanking and policy making. Data selection is inherently profit focused instead of Humanity Progression focused. Bodies like the UN, EFF, et al. are wholly ineffective.

The paths forward on that are gruesome to be honest as what would be best would be something like a randomly selected group of High School Science Fair finalists and Waitresses to form a Governance body with teeth to dissolve Corporations completely for profiteering off populace private data, genetic data, financial data, and the engineering decisions that are ubiqutously driven by them when determined that a Corporation or other Government body is acting against out future.

Interesting timing, myself having spotlighted Corporate greed in my screed. The internet is afire, bringing the website offline at times, with the supremely influential updated tonight https://usdebtclock.org/ covering up all it's valuable financial data in a foreboding hint with the following quotations:

“You are a den of vipers and thieves. I intend to rout you out, and by the eternal God, I will rout you out. If Congress has the right under the constitution to issue paper money, it was given them to be used by themselves, not to be delegated to individuals or corporations.”

“The mischief springs from the power which the monied interest derives from a paper currency which they are able to control, from the multitude of corporations with exclusive privileges...which they have employed for their benefit"

• Andrew Jackson.

Oh yeah? How often are you able to prevent the windows installer from ignoring your partition tables that you just configured manually, with the fucking Windows partition tool?

Because I'm up to about 50/50, and I've installed more Windows, Linux, and BSD systems than anyone ever should.

Point is, can anyone really truly and completely understand the temperamental beast that is the Windows installer?

Also, this is BSD, not Linux. So there's that...

I don't think I'd necessarily agree with that, but just to note, op is recommending the BSDs, not Linux.

Also his (so far unsupported) shots against Mozilla. How does that help in a world where >90% of the Desktop world is still dominated by Microsoft and web and mobil by Google products?