Verbal fillers are a ubiquitous human linguistic tic. Like, Uhm, Actually, Basically, and yes, So are all just part of talking to one another without spitting out the first thing that surfaces in your mind. In Arabic, it's yani, which even I picked up when I lived in Egypt and was speaking Arabic on a daily basis.
IMO, any of these are actually a good thing. It means that the person you're talking to is thinking through what they are saying. Maybe they are making sure they are accurate, or trying to keep from offending you. Maybe what they are talking about has some gravitas, or they're not totally confident in how they are portraying their own thoughts.
Any verbal filler has the potential to "undermine your credibility", but most people don't hear it in day to day conversation, as they shouldn't.