harsh3466

Not as much time to tinker this week, but I still had some fun and learned some things!

How to run a memory test using memtest86+

My error message is back, which means my nuke and pave approach didn’t solve the problem. So, yay to having a record of the error message?

Here’s the error:

echo 0 > / proc/sys/kernel/hung_task_timeout_secs" disables this message.
INFO: task txg_sync: 3557 blocked for more than 241 seconds.
Tainted: P 0 5.15.0-94-generic #104-Ubuntu

I decided to do a little more searching and found that the txg_sync is a zfs task. I know zfs uses a lot of RAM as part of it’s processing. As a result/starting point, I decided to do a memory test to see if I messed up any of my RAM modules when I knocked the shit out of my server.

Running a memory test was really easy. I downloaded the latest memtest86+ ISO, used balena etcher to flash it to a usb stick, booted from that stick, and let the test run.

I let it run for two full passes and got no errors.

So as of right now I know that the error is being caused by zfs writes/activity, but I don’t know why the error is happening, other than that I fucked something up when I knocked the shit out of my server.

How to set up a wireguard tunnel

This also has been on my list of things to figure out for quite awhile, and, turns out, with the wg-easy project, it is exactly as easy as the name implies. I found out about wg-easy through the Awesome Open Source YouTube channel. I’ve learned a lot from the guy that runs that channel, so I always check there when I want to learn more about something. Timing was fortuitous, since he had just dropped his video on wg-easy.

I’ve got wg-easy running on my vps, and I’m planning to connect my playground server to it so I can ssh in and play around during my breaks at work.

Grav CMS is pretty nice

I’ve kind of idly been looking for an alternative to Wordpress. I found out about Grav while bouncing around Linux YouTube looking for things to learn about/try. I’ve already tried 11ty and Hugo ssgs and neither worked for me.

Grav on the other hand was easy to get up and running, is easy to theme, (Theming was the problem I kept running into for both 11ty and Hugo), can be managed through cli or webui, and can have content added through the webui, or, more importantly for me, from markdown files on the server.

Whether or not I’ll actually use it to deploy a site remains to be seen. I’ll continue to tinker with it while I decide if I want to migrate my wordpress site over to it.

Edit: I've made an account here on lemmy.ml as I routinely can't comment or post from my account on lemmy.world.

Bit of a week! As usual, had a lot of fun tinkering. Here’s my takeaways from this past week(ish).

I finally learned how to set up a cron job with elevated privileges

This is something I've had on my , "I should really get this figured out" list for about two years now, but instead have been inconsistently typing my rsync commands (Since I've also been too lazy to set up the aliases for these commands).

I spent a couple of days rebuilding my server from the OS up (for reasons which I will explain momentarily), and since I'm up on a fresh OS with all my containers and services up and running, I figured it was time I figure out this cron job thing.

The approach I took was to write a simple bash script for my backup. The script is four lines. Three of which are sudo rsync ..., and the last of which is a curl -d ... command.

The rsync commands are to incrementally back up my server data, cache, and docker volumes.

The curl command triggers a notification through my ntfy instance, (link is to ntfy, not to my instance), to let me know the backups have successfully completed.

In order for that to run properly, I also had to learn....

How to update sudoers privileges

After reading about crontab and privileges, I know I could have just edited /etc/crontab and run my script as root, but what would be the fun in that when I could also learn about changing privileges through sudoers! So I learned how to modify sudo priviliges by creating a new file in sudoers.d with the command:

sudo visudo -f /etc/sudoers.d/name-of-my-sudoers-file

And why that and not just editing /etc/sudoers directly with nano or vim or emacs? That was my first question when I saw that command and thought, "Oh, shit, I'm going to have to brush up on my vi/m."

Turns out, if you just rando edit sudoers (or add a file to /etc/sudoers.d/) with any old editor, you can fuck up the syntax, and if you fuck up the syntax, you can fuck up your ability to use sudo, and then you can't do anything requiring sudo on your machine without going through tremendous headache to fix it.

However, if you use sudo visudo ..., you get syntax verification to prevent you from breaking sudo.

And, on Ubuntu server, visudo uses nano by default, which meant I didn't have to worry about vim just yet (vim is on my roadmap of things to learn)

(Also, you can change the default editor visudo uses, but I don't remember the command because I won't be changing it until I get a grip on vim and can make a decision about which editor I want to use.)

With all that being said, I created a file in /etc/sudoers.d and added a line to allow my backup script to run with elevated privileges without requiring a password with this syntax:

username ALL=(root) NOPASSWD: /path/to/my/script

Good documentation/notes will save you like good backups will save you

This isn’t something that’s new to me, or to linux (Arch wiki ftw) but it’s something that 100% made rebuilding my server from the OS up a pretty worry free breeze.

So why did I rebuild my server again a little over a month after rebuilding it from the OS up? Turns out when you accidentally kick a stool while carrying a heavy box and that stool knocks the fuck out of your server, your OS can get fucked up.

This happened a few weeks ago, and boy was I panicked when I first kicked that stool into the server. After putting down the boxes I turned on the monitor and the screen was freaking out. It looked like a scrambled Max Headroom. I held the power button to force a shutdown, and after rebooting the server everything came back up and I thought, ”Holy shit, I dodged a bullet!”

(Bonus lesson, I learned to not leave the stool in front of the server rack!)

But, all was not well. My server data and cache are on zfs pools, and every time I tried to bulk add some of the shows or movies I prepared to the data pool, I would get this procsys kernel panic error. I had repeatedly been checking my zpool status, and everything was good there. So I was furiously searching trying to figure out what the error meant, and I kept finding folks with the same or similar errors who talked about checking logs, but whenever I checked logs I couldn’t find anything to indicate what was actually going on.

Finally, after a few days additional searching, I ran across a comment on a thread that said this particular error (I neglected to save the error, I wish I had) was usually a hardware related issue, like a loose connector, and I thought, ”Holy shit, that makes perfect sense after knocking the shit of my server!”

So I shut it down, opened it up, and sure enough there was a loose cable on the motherboard. I reseated it, checked the rest, rebooted, and over the course of the next week, it seemed all was well.

But I kept getting these weird errors. Not actual error messages, no more kernel panics, and data wrote to the zpool just fine. It was little things not working as expected. Commands that typically ran very speedily (like ls) were lagging, opening a file in nano took multiple seconds instead of being near instant, stuff like that.

I decided to go for a nuke and pave approach, rebuilding from the OS up again, which is where the documentation comes in. Since I started messing about with self hosting 2-3 years ago, I’ve kept meticulous notes on everything I have done and learned so that if I had to re-do it, I could open up Joplin, search for whatever I needed, and proceed. This has saved my ass multiple times over the years as I tinker, break shit, and fix it using my notes.

So yeah, in addition to having a good backup system, you should also keep good documentation for yourself.

edit: removed extra 4 from post title