(POTENTIAL BOOK SPOILERS)
Castle Black doesn’t have the greatest defences on the South side because it’s the law. A long time ago, an ex-commander tried to carve the Wall and the Gift into his own little kingdom. Since then, it’s been decreed that the castles at the wall will have no significant Southern fortifications.
Like Harry Potter, Lord of the Rings, and any number of other movie/TV adaptations, stuff generally makes more sense in the original books. If you’re looking for more depth, pick up the source material.
