The GNUTLS code is terrible though, they ignore common coding conventions and have it set up in a way that makes it difficult to integrate into real programs. It's just badly designed. The fact that an error like this existed was almost inevitable.
This is going to be a bit technical, so hold on.
The error is a mishandling of an error code from a function. By convention in C, you return 0 on success, and non-zero in case of failure. The number you return is generally either -1, or an error code. GNUTLS did the opposite, 0 is failure and 1 is success. When a different programmer in 2005 went and refactored the code, he forgot to account for this and programmed the thing assuming the certificate check returned normal error codes. It's a stupid bug, but it's one that is far too easy to make by accident given the rest of the codebase.