25 Likes
900 million and counting
I would argue that details of positive tests of Covid 19 are pretty much the definition of data that needs to be updated in real time
Apparently as a result of this mess over half of 50.000 contacts were not told to self isolate in the foolowing week and the rate of infected per 100.000 which is the metric used to trigger local lockdowns was not updated correctly
3 Likes
You are a super-hero. (I get all dewey-eyed around librarians.)
7 Likes
It’s a peculiarly kind of UK government gross incompetence that we used to create sitcoms about except the very real tragedy of further spread that this gross incompetence will result in causes all the comedy to evaporate. So… spaff millions on a data sucking app that does’t work and then scrap, hand out multi-million pound PPE contracts to your cronies, fail to provide frontline workers with adequate PPE and again outsourcing key parts of this infrastructure to hopeless private companies that simply can’t manage the volume of data or even keep it secure.
I’m inclined to include Powerpoint in the Great Old Ones software bundle for the brain eating invocations it can spawn in those unfortunate enough to sit through a presentation.
6 Likes
The BBC is reporting its actually an issue with the number of rows exceeding 65k - a limitation removed in Office 2000.
3 Likes
I think Excel is a fantastic tool - for what it was designed to do - which isn’t masquerading as a database.
A couple of weeks ago my annual pension statement fell foul to someone using Excel as a database.
Somehow, someone had selected one of the column, decided to sort that column, then sent the whole sheet off for printing. Which meant (hilariously) I got someone else’s statement details including the value of their pot, their account number and date of birth - and presumably someone else is even now sobbing over my statement wondering how they are going to cope in the future.
And this is not a small pension scheme. I hoping the Information Commissioner’s Office will take them to the cleaners - though they seem to be giving everyone breaching data security a free pass now because of [rolls Magic 8 Ball] - oooh pandemic.
5 Likes
We have a whole profession called Data Management now.
Should I be surprised that nobody thought to bring in somebody who knows how to do this stuff?
6 Likes
Huh. Apparently there are loads of labs that could have been used that did have integrated workflows into an existing PHE database. But, it was outsourced to a bunch of new labs run by friends of the government with no requirement to use those interfaces or provide detailed data.
So yes, the choice of CSV and Excel might have flaws but they’re only using these workarounds because the whole process had been torpedoed at an institutional level.
11 Likes
Classic Dom!
PS - it’s not just data processing incompetence:
6 Likes
I’ve used Excel 10 years ago, and it seemed nice.
Word on the other hand was absolutely horrible. When I started writing my PhD thesis there was a strong push to use Word, and I said that if I can’t use TeX, I quit immediately.
5 Likes
What a Hancock-up: Excel spreadsheet blunder blamed after England under-reports 16,000 COVID-19 cases
Some more background info, links, and knowledgeable comments thread.
2 Likes
Hancock-up
Love it!
2 Likes
If the data got updated in bulk properly every 6 hours, do you think it would be a major difference to real time? The local lockdown decisions are always going to be based on a batch of data, rather than live data. If the positivity rate goes above X for 15 minutes, it’s not going to trigger a local lockdown.
Then with contact tracing, you only really need the bulk data collection delay to be less than the backlog in contacts. Got a day and a half backlog on contact tracing? There’s going to be no impact on performance if you only get the data once a day.
1 Like
When my wife was a grad student, they collected their field research data in CSV format, then immediately entered it all into Excel spreadsheets for long term storage and analysis. They never kept the source data. I could not even fathom why one would keep data in a proprietary format like that. It was like amateur science.
6 Likes
but everyone who uses Excel knows this is human error: they put each case in a new column instead of a new row.
No. No. No.
Everyone who knows anything about data management knows you NEVER USE EXCEL for it.
You don’t have transactions.
You don’t have data integrity.
You don’t have atomic commits.
You don’t have optimized queries.
You don’t have access controls.
You don’t have proper querying capability.
Whomever made the decision to use Excel should be fired and banned for life from ever making money “in computers”. What they did was the equivalent of printing the US Constitution on pizza dough because it was easier. Their level of incompetence is so staggering as to be criminal.
3 Likes
This topic was automatically closed after 5 days. New replies are no longer allowed.
