Basically wondering if someone else has already done the calculations to work out when a lower damage attack with the Agile trait is better than a higher damage attack without it. Is there a good rule of thumb here?
The specific scenario I'm looking at is one of my players' Badger animal companions, which can take 2 actions. If it's in a situation where both those actions are going to be attacks, is it better to bite twice (1d8 without Agile) or bite and then claw* (1d6 with Agile). I am mainly wondering about the general rule of thumb though, if there is one.
* I'm assuming my reading on the rules here is right. If the first attack is not Agile, but the second one is, that's still only the -4 penalty, right?
Most people here have already given you good answers. I'll just add to this by saying it depends on too many other factors, and add some of my own experience and calculating to support what other people are saying.
TL;DR : It doesn't matter, it's like around 1 DPR (damage per round) difference. If you care, use a DPR calculator to work out your ideal options
As far as I am aware, there are no agile attacks that have a damage dice higher than a d6 (for PCs), so my math will be done assuming that.
The Third Attack
The way the math works out, you should want to use an agile attack for the third attack. It will be very unlikely to hit or crit, but it grants a better DPR than using a non-agile attack.
The Second Attack
As for the second attack, it depends heavily on multiple factors:
1. What is the difference in dice size between the agile and non-agile attack, and how many damage dice do you have?
Higher differences in dice sizes will make the non-agile attack more preferable, as though you're 5% less likely to hit or crit, the damage if you do hit statistically makes up for that. Larger amounts of damage dice exacerbates this effect, as your non-agile weapon will pull a larger gap to the agile weapon.
2. What is your damage roll modifier?
A larger damage roll modifier rewards hitting more often, so agile weapons become more favorable if the modifier is larger. This also includes things such as the damage runes, where the damage dice of your weapon is irrelevant.
3. What chance do you have of critting or hitting the target?
If you have the chance to crit on rolls that are not nat 20s, your DPR with the non-agile weapon begins to trump the agile weapon. This is due to the crit doubling the difference due to damage dice size. The exact point at which this occurs depends on the above two factors as well.
Weirdly the DPR starts to swing the other way as the target's AC gets higher. As you exceed a roll of 9 needed to hit a target, your chance to crit is static at 5% (nat 20). This reduces the effectiveness of the agile weapon, but after that, the agile weapon starts to claw back its effectiveness as the non-agile weapon's ability to hit shrinks. Again, the exact point at which this occurs depends on the above two factors.
Graphs
I plotted a few graphs to illustrate the above considerations. The attack roll modifier refers to the attack roll without any MAP, and the DPR calculations have factored in the MAP for the agile and non-agile weapons.
The first is a graph with a d8 non-agile weapon, a d6 agile weapon, and a +4 damage roll modifier.
Note the scale of the axis on the left, it doesn't make a significant difference for this set of stats.
Now what about a level 4 character without any class bonuses to damage, with the same weapons with all the expected fundamental runes, doubling their damage dice?
Note the fact that there isn't even a window where the agile weapon is good against weaker enemies, but it claws back much more effectiveness. This is what we'd expect due to the larger amount of damage dice.
Lastly to illustrate the effect of flat modifiers, I put up a level 7 swashbuckler with an Dueling Spear (d8 finesse) against a Shortsword (d6 agile finesse). They will have panache, and are making these strikes non-finishers. The weapons both have the expected fundamental runes, and a flaming rune.
Note how there is a larger range of ACs where an agile weapon is better.
Conclusions
The answer is it depends, as it always does. I can't give you a definite answer for every single scenario, because there are way too many, and I'm not your dad. In general the following you should make the second strike with an agile weapon if any of these conditions are true:
Otherwise, use you should use your non-agile weapon to make the second strike.
Note that the difference in DPR is small (it's like 1.2 at most), so it doesn't really matter which you choose. I'd go with an agile attack if I wanted consistency, and non-agile if I wanted burst.
That's it! If you read through all of this, thanks for reading my insane ramblings. There are many more things I probably forgot to account for, but I can't think of them right now, and I accidentally overwrote the sheet for the second graph. I'm gonna go do something more productive now.