I think it is more with the linkage to the chuck (or maybe the chuck itself) than with the bearings, and it doesn’t take a very significant degree of misalignment to have a big impact on the drill accuracy. If these parts of the drill are wobbling it probably won’t put that much wear and tear on the motor itself (and if it does, in a cheap drill, any failure will probably be attributed to the cheapness of the drill rather than the wobble).
Here’s an example in a non-budget drill:
