After further experimentation, I am sure that my biggest mistake was going too slowly. I am also sure that my mill does not have enough power to use this cutter properly with this material.
HSS center drill - cuts like butter
1/4" HSS drill - cuts like butter. Cuts better with a faster feed
a couple more HSS drills to widen the hole to 1/2" (the largest drill I have) - cuts like butter. Cuts better with a faster feed.
1" HSS countersink - Skates on the surface until enough pressure is applied. Then cuts well for a short time. Then goes back to skating (feed to slow) or stalls the motor
I than made a make shift boring bar from a chunk of 3/8th square HSS - I have no way to grind it, so it was a very crude tool. I managed to enlarge the hole from 1/2 to about 7/8ths, but the tool is so imprecise that I don't want to get any closer even when the fit up will be with wood
This all tells me that while the metal might be bad, it can't be _that_ hard. I think I either need to find a 1" drill, or figure something better to bore with than a square chunk of HSS gripped at an angle in a 3 jaw check. Having done this once, I don't recommend that anyone try this method