Originally published at: https://boingboing.net/2019/08/07/u-s-warned-tesla-to-stop-givi.html
…
2 Likes
Tesla: our cars perform better at NHTSA’s tests than cars of other manufacturers. This means our cars are the safest.
NHTSA: no, it doesn’t mean that.
6 Likes
One could then argue that the tests don’t mean much of anything.
Edit:
I would say that this is a much better summary of the disagreement than most others out there that use many times more space.
3 Likes
With luck, this will end in the NHTSA being forced to put fresh cadavers, or maybe even prisoners, in the crash tests.
NHTSA: We know our DATA says Tesla is better than everyone else, but you can’t tell anyone that.
TESLA: But…
NHTSA: 5 STAR TROPHIES FOR EVERYONE!!!
3 Likes
if you read the correspondence provided at the link above, the nhtsa is saying that there is no validity to comparing vehicles results if the vehicles differ in weight by more than 250 pounds. tesla might have a leg to stand on if they claimed they were the safest vehicle in their weight class but to haul off and say they are safest over all vehicles is wrong and dangerous.
6 Likes
I did read the linked correspondence and would point out that Tesla does not reference any particular type of crash or vehicle weight and simply makes the assumption that - on average - a lower scoring car is safer. If this is true then Tesla’s claims all make sense and follow quite organically from the test scores. If a lower score does not have any correspondence to safety and likelihood of injury, then I think we have a bigger problem of the testing being unrelated to safety!
This happened quite a while ago and the blog post by Tesla is still live. To me this means that while the agency doesn’t like automakers referring to anything beyond the “star rating”, they don’t think Tesla has actually said anything incorrect or misinterpreted the data.
2 Likes
If they were dealing with the FDA rather than the NHTSA, they would have received a Notification, and if the false labeling was not removed, Tesla would be facing fines and even a consent decree, including federal agents being sent to their production facility to lock down the building until they establish compliance. Just because the blog post is still up doesn’t mean it’s correct.
The NHTSA test does it’s best to simulate certain crash conditions, but it can’t be extrapolated to overall safety in every situation. Tesla’s statement is a blanket statement that doesn’t actually reflect the science behind the testing, and thus misrepresents its products to consumers.
5 Likes
This is a pretty tenuous hypothetical and implies that the NHTSA would have acted if only it could. The poor agency is being beat up by the big bad corporation that should go have a time out.
So we cannot say that a “5 star” vehicle is safer than a “1 star” vehicle? How about saying that a “5 star” is better than a “4 star” vehicle? Let’s not bend over backwards to persecute a car company for shouting from the rooftops that they got the highest score on the safety test by the car safety testing organization.
I personally don’t find it unreasonable to call the lowest (highest) scoring vehicle safest.
1 Like
The NHTSA doesn’t have as much regulatory authority as the FDA to enforce labeling standards. That’s the point I’m making. Their available mechanism is to go to court and sue Tesla for false advertisement. That may not be a priority for them, or they may not have the funding to do so. If Tesla were dealing with a different agency, one with explicit enforcement power, they’d be in a very different situation.
That’s not what Tesla is claiming. They are claiming that their 5 star vehicle is safer than all the other 5 star vehicles, and the NHTSA is saying that claim is not backed up by their test methodology.
3 Likes
Specifically the NHTSA is identifying that heavier vehicles fare better in collisions with lighter vehicles and that the test methodology relies on a fixed barrier (equivalent to a frontal impact with a similar weight vehicle). This is really the only specific complaint the agency makes and it is specifically addressed by the Tesla response.
These are the specific claims that Tesla makes about their car which did receive the best score of any vehicle of any kind tested by the NHTSA. I don’t think that any of these claims are outlandish or misleading given the outstanding results achieved by these vehicles.
1 Like
But that’s not how experimental design and statistical analysis work. Just that first statement violates so many rules of communicating the results of experiments, it boggles the mind.
It’s simple to say that the NHTSA testing does not prove any of those things.
If Tesla had stuck to, “The Model 3 achieved the best score ever in NHTSA testing” they would be implying many of those things, but not outright saying them. If I were in the NHTSA, I would be uncomfortable with that and might send them a letter objecting to their use of my testing, but not block their use of it. Since the 4 statements you list outright lie about the meaning of the test results, yeah, NHTSA has every right to demand their removal.
7 Likes
That’s why every other manufacturer is careful to state that they got “the highest rating” or “scored best in its class on [specific tests]”.
The more I read about Tesla, the more it resembles a long con.
6 Likes
So we have tests that are designed - to the best of our knowledge and understanding - to evaluate the safety of vehicles that are currently for sale on the market. These tests are designed to evaluate the likelihood of injury of an occupant and to evaluate the engineering of the vehicle at addressing the types of collisions that are most likely to cause injury.
Do the tests not evaluate these things? Does the score of a vehicle on these tests not demonstrate the performance against these metrics?
Certainly there is error in measurements and so this information should be provided but this isn’t what NHTSA is complaining about.
If I have designed a suite of tests to evaluate how well a winter jacket keeps you warm in the cold under various types of scenarios, would the maker of the jacket that scored the best on my tests not then be able to say that their jacket will keep you the warmest? Especially if I take this and boil it down to a single weighted score? This seems pretty fair!
Right. Because one day it will turn out that all these people who bought the cars were actually sold toy cars and don’t actually have a car at all? How did they ever fall for such a con! Or maybe one day you find out that Tesla has been mining bitcoin on your car all this time! Or maybe Tesla was a front for Big Canabis and was using your car secretly to do drug runs over the border while you were at work!
I think the idea that Tesla is some sort of long con is preposterous. The company has been in business for 16 years and has done things people said were impossible many times. They catch a lot of flack for it in some surprising places.
Tesla doesn’t play nicely with regulators. This is true and is, perhaps, a sign of Musk’s leadership style. It is also, maybe, the attitude that has enabled them to prosper and innovate when the world was telling them it couldn’t be done.
1 Like
The NHTSA designs tests to make accurate but gross estimation of the likelihood of injury in a collision. They manage to do so by simulating very small number of collisions, because the tests are very expensive and time-consuming to analyze. They report the results by placing vehicles in 5 categories.
In order to state with authority that one vehicle is better than any other vehicle, they would need to run a lot of tests to reach that level of comparative certainty. Like, hundreds or thousands. In order to compare vehicles that have different mass or geometry, they would have to run a number of variations on the tests.
Again, comparing the NHTSA testing to clinical trials, in order to test whether a therapy works ok, you might only need to test it in a hundred or so patients. To claim “it’s the best evar!!$#!” you would need to test it in hundreds of thousands of patients with very specific comparative controls with every other drug.
Let’s run with your winter jacket analogy. The NHTSA tests are powered to rate winter jackets into 5 categories of insulation. Tesla’s jacket was rated in the highest category, along with many other jackets. But since the test subject’s perception of heat and cold is subjective and varies from one person to the next, while Tesla’s jacket is definitely warm, the experiment is not powered to compare it directly with other jackets that receive the same rating, and extrapolate that comparison to all conditions. Tesla’s might be the best for tall people in the wind, but is not as good as others for short people in the rain. Nonetheless, Tesla claimed their jacket is the warmest ever tested. That misrepresents both the design of the experiment and the analysis of the results.
ETA:
I’m a fan of Tesla’s cars and what they’ve done to push the industry to do with EVs. That doesn’t excuse dodgy marketing claims.
6 Likes
I think I sat next to you in archeology class.
11 Likes
I think this may be the weirdest and coolest way I have ever been made fun of. Like ever. Gold star Kathy!
1 Like
So what you are saying is that regardless of the precision of the actual underlying rating, the accuracy of the rating is only valid to the 5 stars and any implication that one 5 star rating is better than another is invalid based on the error implicit in the test results due to the “power” of the testing completed. I can understand this complaint and if this is what the acronym salad agency is angry about then they are doing a very poor job of explaining that.
The lawyer didn’t go into the statistical details, because that’s not his job or area of expertise. He did state, very clearly, that comparison of vehicles within the 5 star safety rating is not allowed, and that is explained in detail in their Guidelines for Communication. This is also the second time they’ve done this, and have been called out on it.
ETA: This is what the Advertising and Communication Guidelines document says about competitive comparisons:
Competitive Comparisons
Advertisements that competitively compare frontal crash star ratings or the Overall Vehicle Score of two or more vehicles should occur only if the vehicles being compared are within 250 pounds of each other and within the same vehicle class. Comparisons for frontal crash ratings or the Overall Vehicle
Score should not be implied between vehicles where the difference in weight exceeds 250 pounds or the vehicles are of different body styles (e.g., comparisons between a sedan and an SUV, even if of similar mass).
1 Like
