Advancing the compound in the z-direction as you describe is how I've been doing it for 20 years when using non-topping (partial profile) inserts. I've had to start using topping (full profile) inserts now that I have a solid tool post.your "it fits but doesn't" was a recurring issue for me as well.
A few points to consider:
1. An A60 insert is not intended for coarse pitches., most listings are wildy optimitic about their range. Too wide for fine pitches, too narrow for coarse pitches. On my A60's the tip width is typically .008" where a 13 tpi thread wants .0094" at the root (.125*pitch). You can fake this with compund parallel to the Z axis and offsetting slightly on a finish pass.
2. External thread depth is typically .6134* pitch, internal .5413*pitch, but it still requires proper tip width. A60 might not be able to go deep enough at coarse pitches without topping the threads.
3. Carbide bits don't like running in reverse, did you chip yours by running the spindle in reverse to back it out without winding the cutter clear?
4. Measuring your external thread with wires should confirm the above.
This is one time that the Machinery's Handbook is worth looking at, the thread proportions are fairly simple once you digest them.
For your desired pitch AG60 is a better bet, but again the tip width matters.
I generally end up putting the insert on my optical comparator, measure tip width and pass it by the d-bit grinder for correction depending on the thread I need.
Gerrit
NB I think your math is off regarding the root diameter for the 13 pitch, I believe it should be 0.019". Which is all the more reason to be able to have the ability to widen the root diameter of the thread, either with the compound, or full profile inserts.