Asbestos may be an even better analogy. Just inert fibres, how can they harm anyone? CFC:s is another example of a perfectly safe chemical.
The basic rule should be that unless you can show how it is degraded into natural components, assume that anything we make will accumulate until it sooner or later reach concentrations where it is harmful.