IMO, once you have a hole that you want to then enlarge by larger diameter drilling, the section view becomes essentially a variant of one of the pictures I posted. The prior hole will present a 90-deg edge to the 118-deg or 135-deg or whatever drill angle so there is a good chance of catching just like the spot drill example. I suspect that's why some advocate just drilling with the big drill to begin with, but that's not always possible. So I don't know of a reliable solution other than drill selection & a bit of start hole technique. Stub drills have less stick out so will be more rigid for the same material. Carbide is stiffer than HSS but cost prohibitive, assuming they even make them. I've heard the 135-deg drills focus their forces more obliquely which could be helpful & often have different relief geometry than conventional drills. Whether that cures all problems or just helps I cant say. Maybe one could chamfer a matching drill angle on the crest of the prior hole, but now that's another tool/step. Not as convenient or time efficient. Pecking the initial drill contact can be helpful. I find that holding some friction on the TS barrel can be beneficial too. Its a crude form of anti-backlash & might reduce the propensity to draw the drill into the cut (snatching conditions). So on a lathe, this could be a TS feed screw wear issue whereas a mill will have a different feed mechanism & be different.
Personally I consider drilling conventional holes to be a roughing procedure. Accuracy typically comes with boring bar type tooling. So its maybe a matter of personal choice as to when to stop with the drills & start with boring, probably dictated by material type & machine capacity. Some people acknowledge the drill kick & basically don't care if they are going to be boring anyways. Not ideal, but kind of pick yor battles strategy. Deep drilling like gun drilling is whole different subject I have no experience with. For large diameter conventional holes up to say 2" depth, I actually prefer annular cutters. You are removing much less material vs a drill & even have a resultant core slug to use for another project. But this has depth limitations obviously unless you can flip the part in which case 2x cutter length. So as usual... depends...
