It’s not clear what would satisfy the FDA – that’s the problem. The technology that 23andMe uses is exactly what scientists use every day. There isn’t a serious debate about whether it works. Instead the argument is about how it is somehow irresponsible for the public to know what is in their genome without going through a physician. Yes, having a SNP correlated with a disease doesn’t mean you will get the disease, but that’s true whether or not you learn of the SNP directly or from your physician.
1 Like
A physician is likely to do a better job of explaining the actual risk – and how it compares to the risk without this information, and how it compares to the risk without this specific factor,
Purely personal concern: Knowing that they have the technology unfortunately doesn’t prove they’re using it correctly. Massachusetts just had a huge scandal over a gal in the evidence analysis laboratory who was “dry-labbing” – ie, guessing and falsifying reports rather than actually doing the proper analysis. I’d want to know what their QC procedures are before I’d be comfortable with them going retail. If you have a doctor look at the results, they get sanity-checked against other evidence, which again reduces the odds of erroneous information.
I honestly don’t think this is a matter of protecting physicians or their labs. Heck, if the service was cheap and trustworthy many physicians themselves would be glad to take advantage of it rather than using more expensive services.
1 Like
From my experiences with the 510(k) process, whole genome sequencing would be a prohibitively cumbersome method to get through FDA, as the applicant would need to demonstrate functional equivalency to existing FDA approved methods (predicate devices). Essentially this would mean that they would need to compare to each and every test which is reasonably determined mostly by genes, such as sickle cell disease or thalassemias, metabolic disorders, and so on. Arguably the better solution would be the promulgation of a rule for whole genome sequencing, but FDA does not have the budget to conduct very many of these. I have seen manufacturers go through years of trials and tribulations within the 510(k) process to make quite reasonable and well-founded claims; this action does not surprise me at all.
3 Likes
Well, as one of the scientists whose job was to read the validation reports of the scientists who did this testing every day, I have to say a) Thank you for your faith in us, but believe me there’s quite a lot of things that can wrong and b) I have a fairly good idea what FDA wants and it will be incredibly expensive, but not impossible if Google is still backing them. I’m sure 23andMe have an even better idea, since they’ve met the FDA and talked it through.
As an example, Acute Measles testing (not the genetic test) is one of the oldest and simplest tests on the books. Most - more than 50% - of positive acute measles tests are false positives. This is using FDA-approved “devices” (tests), which are validated and claim the tests are between 85% to 95% specific. How can this be? Well, ever since vaccination became commonplace, almost nobody has measles. When you test a population that just doesn’t have the disease, the positive results you do get are more likely to be in error. There’s a wikihow piece about it here - positive predictive value. So if you’re a pregnant woman and you have Measles IgM positivity and you don’t know this - what’s your next step?
Another example: CleverLabs are doing genetic testing, they’ve purchased the finest database on the planet, and it’s validated up the wazoo. Their sequencers never fail and get 100% reproducible answers. Jim from IT writes the code that sends the raw data from what’s identified as your sample to the database, gets the answer from the database and attaches it to the name and ID number on your requisition. Did Jim get it right? Did he get it right under all conditions? Not on the day Sharon in specimen accessioning typed your name as ,hbadger by accident, where the comma was a field delimiter, knocking your result and the next hundred…or maybe thousand… into the wrong cells in the database, sending the wrong result back to all of you?
I have about a hundred more of those examples, but I’ll spare you.
5 Likes
The California examiners (inspectors) I’ve worked with have given me other instances of dry-labbing. Yes, it’s hard to believe but some people are quite happy to use a random number generator and bill you for it. One described being in a lab when the chemistry analyzer just started spitting out a tape of “results” despite there being no specimens on the carousel at the time. (This is usually Medicare fraud than simple fraud, though, in my experience.)
You can go read the Warning Letter right here:
http://www.fda.gov/ICECI/EnforcementActions/WarningLetters/2013/ucm376296.htm
These are not sent unless there is an egregious or repeated citation from a (less public) FORM 483 finding. Basically, 23andme was using a lower scrutiny filing (510c) and has dodged the Agency’s requests for validation of their methods. This is pretty basic shit in the pharma industry and indicates a compliance problem at that company.
The Warning Letter explicitly cites the dangers associated with false positives (think BRC1) and false negatives (think …BRC1). They need to have methods in place that prevent this, and those methods need to be validated.
3 Likes
Lyle, why are you always blaming Sharon?
1 Like
That Sharon! Don’t get me started!
Actually one of the best IT folks I knew is called Jim and one of the best laboratorians I knew is called Sharon, so maybe I should change my default names.
Sorry Sharon!
While it’s all well and good for the FDA to give companies who don’t comply with verification requirements a hard time, I find it absolutely ridiculous that homeopathic “medicine” and magnetic bracelets can be sold without any consequence.
If homeopathic practitioners had to prove their methods of diagnosis were accurate, and that their “remedies” worked, there simply wouldn’t be any.
/off topic rant
What I mean to say is, why does 23andMe get hassled and sent stern warnings, when there’s people selling water and sugar pills as if they were medicine, and there are naturopaths telling their customers to discontinue their chemotherapy in favor of coffee enemas and magnetic bracelets?
4 Likes
Yeah…no. There’s nothing there about “diagnosis.” Nada.
I agree with you … but unfortunately “hey, he’s getting away with it” is not a valid defense either when pointing at your brother or when pointing at another company.
(I suspect that part of the difference is that the homeopathics etc. have learned to be extremely careful about what claims they make, staying just barely inside the law… whereas these folks weren’t so careful because they had something that isn’t a pure scam.)
2 Likes
Yeah I know it’s something like a tu quoque argument (I think?). But it still feels like the cops are busy ticketing jay-walkers while ignoring that there’s a guy shooting at cars from the crosswalk.
The problem arises if they tell a subscriber “You’ve got BRCA1” by mistake, and then that person goes and gets their boob lopped off to avoid getting breast cancer.
This is a problem, because 23andme doesn’t actually sequence DNA, they measure fluorescence from a large array of markers that look for short sequences of DNA. Then, they say “this pattern of fluorescence correlates (say) 99% of the time with having BRCA1” - they need to fold in the underlying prevalence of the gene in the population, and the rate of false positives, to have useful information.
Case in point, here is a customer of 23andme who was told he had a fatal form of muscular dystrophy, but in fact the test was a false positive. The correct diagnosis is apparently that the guy is a carrier of two copies of two separate, faulty genes that are linked to the disease, but not two faulty copies of the same identical gene - ergo, he’s a carrier but he will not develop the disease himself.
But 23andme provided the wrong information, namely “you’re gonna die of muscular dystrophy” to their customer. If you knew you were likely to die in the next 10 years you’d live your life differently - so it’s important that 23andme actually get their facts right. The FDA is doing the right thing in my view. Google probably has the resources to carry out a proper evaluation and play the game properly. More to the point if they launch an ineffective test, and people start relying on it to make medical decisions, they’ll be sued to hell if it’s faulty.
Is running a panel of 240 tests really a sensible idea? Suppose we want the false positive rate to be less than 5 in 100 customers. Then each individual test has to be accurate to 99.978% to avoid having at least one false positive amongst the panel of 240.
5 Likes
I pee what you did there
2 Likes
If 23andMe tells you that you are going to get breast cancer and you chop your breasts off as a result: A) you are a fucking idiot B) the doctor that did that surgery should lose their ability to practice medicine because they are somehow stupider than you are and C) your insurance company that will pay for the surgery is stupider than both you and your doctor combined and is going to go bankrupt momentarily.
23andMe gives information. Anyone who has used it has to sign like 40 fucking forms that declare in bold blinking letters that this is not a perfect test, and if you find anything in there that is worrying, talk to your fucking doctor. If you get through the half a dozen disclaimers and still do something that is on its face stupid, and find a doctor stupid enough to humor you, you are so actively dumb that the government trying to save you from yourself is a futile act. You are probably going to blow your face off trying to remove a zit with a shotgun regardless of what anyone does to save you from yourself.
I can’t get a round of genetic testing from my doctor for random stuff for $80. My doctor won’t give it to me, and even if they did, they are going to charge thousands of dollars. I can get it from 23andMe. The options are either get no testing, or get not-so-awesome testing. Not-so-awesome testing is superior. Not-so-awesome testing can certainly give you a false positive, but the obvious response to a positive result is to hit up your doctor. Your doctor asks how you know you have some result, you tell them, and the next thing the doctor does is order a follow up to either confirm or reject the results. There is no practicing doctor that would do otherwise. Hell, there is no insurance company that would cover whatever drugs or surgery you think you should get without official results. The entire idea that someone is going to get a false positive and do something stupid is laughably absurd.
Vastly more likely is that people will get true positives, contact their doctor, and get superior care. People with false negatives will do exactly what they are doing now, assume that they are okay. People with true positives will see a fucking doctor. There is no loser here, other than the FDA which apparently feels the need to swing their dick at things and keep the cost of medicine as high as possible.
6 Likes
If you chop of your boob off over a 23andMe kit, you are an idiot, your doctor is an idiot, and you obviously ignored the 40 disclaimers that scream in bold letters “no dude, seriously, talk to your doctor”. This isn’t fine print shit that they put in either; it is big old block letters, a summary page, and more.[quote=“anon68287401, post:34, topic:15035”]
But 23andme provided the wrong information, namely “you’re gonna die of muscular dystrophy” to their customer. If you knew you were likely to die in the next 10 years you’d live your life differently - so it’s important that 23andme actually get their facts right.
[/quote]
Again, if someone tells you that you are going to die in 10 years and you don’t see a doctor, you are probably going to die in 10 years regardless, but through epic stupidity. Even if you manage to ignore the half a dozen disclaimers about how the test isn’t perfect and that if you see anything scary you REALLY need to go see a doctor, seeing a doctor when being told you are going to die should be the natural reaction of anyone that isn’t criminally stupid.
Of the 240 things run, the vast majority are boring. Learning that you likely have small breasts based upon the random boob gene they are testing for isn’t going to end your life, especially when you look down and realize that they are in fact wrong. The vast majority of the stuff is a curiosity. For the things that matter, if you get a false positive, a doctor can quickly re-test and negate it, or perhaps more importantly, you can get an actual positive, and your doctor can confirm it and you can take appropriate steps.
I’ll happily take a false positive that is easily disproved with a followup test, over failing to receive a true positive result because I was never tested. A false positive is a brief scare. A true positive that you never are never told of can make you dead. I’ll take being scared for a week or two over being dead. The FDA can fuck off.
3 Likes
I was working with a doctor’s office getting their EHR to talk to Quest Labs; against my better judgment I used myself as a test patient. (Almost all fake demographic info, but my real name and phone number.)
Well, inevitably one of our test tests got “processed” as if it were real, and they ended up sending me to collection (because, of course, I’d supplied bogus insurance information.) When the collection agency eventually tracked me down, how we laughed!
All of this without, y’know, any actual specimen being drawn or tested. Even better: in the fake test results that got sent back through the interface, and stored in the EHR, I apparently have a yeast infection.
1 Like
So we’re agreed then, the test is medical in nature, therefore it needs to have a demonstrated efficacy and quality controls to monitor it’s implementation, and it should never be available direct to the general public without a licensed doctor to interpret the results in the context of a full medical history, and advise as to what steps to take. This is the same reason it’s illegal to supply prescription medicines direct to the general public without some consultation and prescription.
People who are curiosity-driven can hire a private doctor to perform possibly unnecessary medical tests. A basic standard would be to have a peer-reviewed study that quantifies the efficacy of 23andme’s test against each of the 240+ different conditions that the kit claims to detect, compared to the current known-best diagnostic procedure where one exists, with each of the 240 trials generating enough data to be statistically significant to understand the rate of false positives for the panel as a whole. Then, in the instances that it’s actually more effective than the existing diagnostic procedure, it should supplant the existing technique, and where it’s less effective, the information should be suppressed or used to order a more accurate test.
If it’s going to run as an on-going service, the FDA needs some way of verifying quality control - even if 23andme are an honest business and their product works as claims, what stops another company charging $50, tossing the spit sample in the bin and generating random diagnostic reports?
2 Likes
Yikes. Given my previous job, this sounds like one of my nightmares come true.
Sure – but those are the same sort of people who will start taking actions based on a segment of the evening news.
1 Like
