Technology

Have you seen the great gatspy with Wizard too? That's what always comes up when mine goes too far. I'm working on compiling llama.cpp from source today. I think that's all I need to be able to use some of the other models like Llama2-70B derivatives.

The code for llama.cpp is only an 850 line python file (not exactly sure how python=CPP yet but YOLO I guess, I just started reading the code from a phone last night). This file is where all of the prompt magic happens. I think all of the easy checkpoint model stuff that works in Oobabooga uses python-llama-cpp from pip. That hasn't had any github repo updates in 3 months, so it doesn't work with a lot of newer and larger models. I'm not super proficient with Python. It is one of the things I had hoped to use AI to help me learn better, but I can read and usually modify someone else's code to some extent. It looks like a lot of the functionality (likely) built into the more complex chat systems like Tavern AI are just mixing the chat, notebook, and instruct prompt techniques into one 'context injection' (-if that term makes any sense).

The most information I have seen someone work with independently offline was using langchain with a 300 page book. So I know at least that much is possible. I have also come across a few examples of people using langchain with up to 3 PDF files at the same time. There is also the MPT model with up to 32k context tokens but it looks like it needs server machine ram in the hundreds of GB to function.

I'm having trouble with distrobox/conda/nvidia on Fedora Workstation. I think I may start over with Nix soon, or I am going to need to look into proxmox, virtualization or go back to an immutable base to ensure I can fall back effectively. I simply can't track down where some dependencies are getting stashed and I only have 6 distrobox containers so far. I'm only barely knowledgeable enough in Linux to manage something like this well enough for it to function. - suggestions welcome