The rules really need to be explained better. If swapping is a normal and expected part of the game then your opponent asking you if you want to swap is not so weird. Especially if there are asymmetric roles, like, if your opponent is the “house” and never gets to deny a swap but always offers.
But having heard the answer and thus assuming equal players, I found my way to the same answer via another route. I tried to get at a generalized answer: if the card you have is a 3 or lower, you would statistically be better off swapping. But if you factor in that your opponent has also arrived at that rule, you may want to conclude that you won’t swap when you have a 3, because that means you opponent must have a 1 or a 2 (since there is only one three). And then you infinite regress from there.