What are the down sides of using a milling machine as your drill press?
Milling machine to replace drill press not drill press to replace milling machine....Not very good.
Drill press bearings aren’t designed for very much radial loading and
without a drawbar your chuck will work loose. YMMV