Yeah, it’s not much harder to open a lock without the key–as long as you have access to the ‘inside’ of the lock. You need to be able to remove the springs from the top of the lock ‘bible’ and well as the followers that they push down. Some locks make this hard to do, so I can’t give you specific advice. It’s also pretty easy to just replace the lock mechanism in most handle sets.
I’ve rekeyed four houses by now and it’s very much worth doing. There is no lowering of security by having one key for the whole house. Criminal entry will either break the door in (keys don’t matter) or they will pick the locks (keys don’t matter). Depending on the age of the home, you may end up needing to replace some locksets so that you have the same keyway for all of your locks, but then rekeying is a maybe 10 minute/lockset task once you get confortable with it. Many hardware stores sell rekey kits that come with the tools you need. If you need to rekey more locks than that kit supplies, you can get all the pins you need on ebay, amazon, etc. Just get pins for the right brand as there are subtle differences in diameter, length, and other dimensions.
If you’re going to go through a lot of trouble like this, maybe combine the effort with replacing the locks with better ones. Though the security of home and commercial locksets is not much considering the way that most locks are bypassed. It’s just as easy to kick in a good lockset as a cheap one if the door is the part that fails.