I wonder if you could feed the paper into a printer and have some LLM without guardrails print out the names and likenesses of all the wealthy people on earth so they all die at once..

The rules don’t say you can’t do that, you just need to write the name and have the image of the person in mind, and this might just fulfill those requirements, if just looking at someone while writing would do it. It’s not like you are capable of a perfect representation in your mind anyway, so ai might be close enough..

I feel like that’s morally fairly clean since you didn’t do anything but fill a paper tray and write a prompt..

I’m immediately researching ask the major political figures of the world and most of the richest people too- I think I need 8-10 books, probably.

It's obviously broken.

Not sure I really buy this. You have two choices: use it or don't. If you use it, it will either kill somebody or it won't; if you don't use it, it surely won't kill anybody. The probability of the former case is obviously extraordinarily low, but the probability of the latter causing a death is surely many times lower. The only thing that matters for morality IMO is the ratio of these probabilities, not the absolute difference.