According to https://www.inchcalculator.com/horsepower-to-kva-calculator/ the 5kVA transformer should easily handle the 1HP 3 phase motor on my mill?
I think perhaps you missed the nuances of my comment. Although the transformer might be sized and rated for this task. The wiring may not be. To be more clear, the amperage on the low voltage side will be inversely proportional the voltage ratio. If the wires and terminals on the low voltage side are not sized for the current required on the low voltage side to generate the output power for the mill on the high voltage side, they will be seriously compromised.
It seems very odd to me to think that the transformer was sized that big though. I can't appreciate that there was any need for that. It seems more like an over designed system assembled by someone who didn't really know what they were doing and chose to err on the safe side or perhaps even more likely by someone who did know what they were doing but chose to use what was available rather than to buy and use ideal components. We might never know.
In any event, I can't really see any advantage to using the transformer to step up the voltage from 120 or 220 to 575. To do so will require 120 or 220 3 pH and careful analysis of the wiring limits as described earlier. Also, to be very clear, the 3phase output of a VFD is not a suitable input for any transformer. At a minimum, transformer influenced feedback to the VFD will be misinterpreted to the detriment of VFD output voltage control and at worst could result in serious damage to the VFD circuits.
On the other hand, using the transformer to generate single phase low voltage output to power other devices such as DROs and power feeds would work quite well and given that they exist on the subject machine I am reasonably confident that this is how the transformer was used - albeit overdesigned.
Without detailed knowledge of the previous usage of the system all of the above is just educated speculation on my part.