Um, no, that’s never actually stated; there are three traditional explanations for Jesus’ brothers:
First, that Mary and Joseph had sex.
Second, that Mary didn’t, but Joseph did, so then, Joseph might have had another wife, who had died young, and Joseph would have remarried.
Third, that neither of them had sex, but they had cousins.
