Newest thoughts →

February 2023

Phew. And the importer of Internet Archive item-level data -- from a 600GB JSON file -- is off.

Getting my parser for this large JSON import to be performant was a little tricky.

Ended up using the msgspec library, which is a Schema-based parser for JSON that I had not used before. It lets you specify only the fields in which you are interested, giving substantial speedups vs. a parser that imports every field.

I then used list comprehensions to filter everything in a single pass and have batched database updates into groups of 1,000 records to lower DB hits.

No, this is not a disentanglement from, but a progressive knotting into...

50 years, today, since the publication of Thomas Pynchon’s Gravity's Rainbow.

Today I wrote the initial spec. document for a new system, dubbed Project Op Cit, for "self-healing DOIs". It's basically a preservation and monitoring system that attempts to detect when a DOI endpoint has failed and automatically resurrects the preserved copy.

Remembering one of the funniest pictures of Toby here, refusing to go out for a walk in the rain because he didn't want to get wet!

It's almost as though this "huge u-turn" might be economically advantageous to the publisher, having drummed up loads of media interest through a very public scandal.

So the Michel Pastoreau history of the color White book is a lovely physical object, but it has no index. Urgh.

Thought maybe I could grudgingly buy an ebook, also, so I could search it. Then saw the DRM limitations on their eBooks and it's a "no" from me.

Did my six-monthly check in to see if Impossible's meat-free burger is yet available in the UK. Alas, still no 😔

However, my search history did remind me of the rabbit hole I went down trying to understand "impossible differential cryptanalysis" a whole ago...

Further, today I found the Internet Archive dataset that I needed! It's a 300GB compressed JSON file 😱 Rigged up an EC2 box to download it, decompress, and process into a relational database.

Spent this evening editing the transcript of an interview about contemporary digital publishing. Asking the questions was @jamescosullivan. Looking forward to seeing it out, soon.

Top role. Though you may have to work with colleagues who are using chaos, madness, and gambling to hinder equality and human rights.

And that's a wrap. Had to use the Internet Archive's Keepers KBART file in the end as the item-level snapshot didn't contain external identifiers and the container snapshot didn't contain preservation extent. Still, I have equivalent data to the Keepers Registry now :)

Can anyone please recommend a good UK online photo printing service that you have used? We want to put together a physical album of Toby pictures.

Suspicion: The Keepers Registry drastically under-reports the extent of preservation of scholarly artifacts in the Internet Archive.

This lovely sympathy card for Toby from @drlennardlee, @MarkOak04974342, and @NikolaBrigden has set me off all over again. Thank you so much.

Today's work, with any luck, is to write the importer for the Internet Archive's FATCAT catalog. 150m scholarly artefacts, digitally preserved...

In less brilliant news, my kidney function has declined again. It's a rollercoaster.

BK virus levels are down. I've suffered massively to try to reduce the immunosuppression. But the kidneys just keep failing. My projected graph predicts dialysis by July.

Some good independent-label music out today through which to jam out the Friday. First up is the new single from Payfone & Kyd Nereida: I Feel You. Very funky. https://lengrecords.bandcamp.com/album/i-feel-you-incl-mudd-remixes

Then there's Duncan Gray's The Demon Haunted World. Nice Sagan reference. Played by Andrew Weatherall back in 2016 and now getting a release. https://ticitaci.bandcamp.com/track/the-demon-haunted-world?from=fanpub_fb_trk

Then there's the @HardwayBros remixes of Holy Youth Movement's Better Together. Uplifting stuff! https://holyyouthmovement.bandcamp.com/album/better-together-hardway-bros-remixes

Speaking of good paper titles...

Of course, this paper does much more than that. There are loads of metadata assertions also embedded in that page...

Today, I fixed the HathiTrust preservation status importer and began work on the 150m records in the Internet Archive. After this, there's just French National Library to go & I'll have the coverage to draw conclusions about the preservation of the scholarly landscape as a whole.

I love the titling on some 1960s journal articles that give no indication whatsoever of the contents. "Two Tantalizing Biographical Fragments of Historical Interest".

The preservation database had an edge case where incomplete metadata led to false preservation assertions, with no easy way to resolve them. Latest version expands the ISSN list using ISSNL-ISSN mappings so that we catch all title variants.

Crying on the train back from hospital here because we said goodbye to Toby today. We adopted him at age 15 after his previous owner died. We chose him because so few people want to adopt older dogs. We knew he wouldn't last forever but he had a good last few years with us.

I find this whole media trend of reporting on people not giving up seats quite worrying. I often feel I'm getting dirty looks for being a man in his mid-30s who takes a seat. But I can't stand or walk without a stick at the moment with my rheumatoid arthritis.

I remember once in my twenties, during a flare up, a man telling me to stand up to give my seat to someone else... Despite the fact that I had a walking aid.

The other thing that isn't to be tolerated are transient conditions. Most of my life, I haven't needed a stick. Then the RA flares and I need it. But there's an in between state. I can stand, but it's painful. I have to take the stick in public as a "prop" in this state.

Without the stick, people assume I'm selfish. But, for those who haven't used one, it's actually *really* inconvenient using a walking stick. You lose the use of a hand in everything you do. But it's better than getting abuse on the train.

It's hard to explain to people who don't know the condition how, with rheumatoid arthritis, if it's controlled with biological therapies, you can be well enough to dance... Then when you're off the meds, you can't walk.

At 3am, Toby started barking and howling, clearly in pain. Then he started having spasms and couldn't stand up. We went to the emergency vet who gave him methadone. We'll just have to see how he goes today.

Urgh. Hit the first snag in my digital preservation research project: a major bug in my parsing of ISSNs in the HathiTrust importer. I think this is now fixed in database version 0.0.42 but am re-running to check. A shame when all was going so smoothly. #CrossrefLabs

I can definitely title this article about the extent of digital preservation behind persistent identifiers "nevertheless, she persisted", right?

My current routine is to wake up at 5.30am to take steroids and painkillers and then to go back to sleep for an hour and a half while they do their thing. I forgot today and now feel awful. So much pain. It's going to be a tough morning.

I was just asked by national media to comment on the Dahl censorship fiasco. I don't really want to be a pawn in various media culture wars and so I replied that although I would love to comment, I have only seen the controversy at second-hand myself, through the media

and so would merely be voicing outrage at something that I have not, myself, closely observed. I have not read the new editions side-by-side with the originals and so cannot determine the extent to which this is a genuine bowdlerization or a sensationalised scare item.

The books chime with social mores of a quite-nasty contemporary Britain. And I don't know whether editing the books will fix that, because people like the nastiness, and want their children to like the nastiness too.

Dahl's works are not about tolerance for others. They're full of malign characters who have external appearances that project their inner evil. The stories basically re-enforce playground bullying hierarchies, with which children are well acquainted.

In general, I am suspicious of the censorship of literary texts by post-publication editorial intervention, though I would not say it should never happen.

Of course, liking nastiness in fiction doesn't mean that you like it in reality.

I have concluded that Airflow is probably the worst at developer experience I've had in years 😂 (sorry, Airflow)

But I also can't believe that there's really no really simple serverless cron service for python. Unless someone can recommend one?

Had an hour-long conversation yesterday about normalising scholarly/persistent identifiers. It's surely only applicable to edge cases, but we have to cater for those.

Basic questions around non-PID URLs: should http and https be treated as synonymous (breaks the spec but avoids

URL context, for example. So if someone accidentally cites the former, it's a challenge to return results also for the latter, but the digital spec defines them as different resources.

fragmentation)? DOI handbook is explicit about non-case sensitive comparison... Yet the mechanics of automatic case switching across languages and character sets is really complicated. URLs by contrast have to be case sensitive. Bell_hooks is not equal to bell_hooks in Wikipedia

WIth no offense meant to middle management, the cryptic crossword I looked at last night had an anagram of "pinheads" for university "Deanship"s. 🤣

I still find the official MS Word file spec document hilariously opaque:

Each data element in the Plcfbkfd is associated, in a one-to-one correlation, with a data element in the Plcfbkld at offset fcPlcfBklFactoid. For this reason, the Plcfbkfd that begins at offset... etc.

The web page detailing this near-indecipherable stream of jargon has, conveniently, a useful feedback tool attached, where the user is asked to rate, with a thumbs up or thumbs down, whether “this page is helpful.”

Also, massive thanks to @Lever_Press / @doc_chocula / all the institutions that support them for publishing this book open access, without author fees, making my research life about a billion times easier. I love what they do.

Today I hope to:

* Edit some of my book
* Do final revisions to a book chapter (_The Cambridge History of the British Essay_)
* Tidy my office
* Observe the televisual broadcast about dog adoption, _The Dog House_

If time:

* Make some banging techno

I will be on BBC News South East this evening, starting at 6.30pm, talking about the recent Evusheld decision and NICE drug-appraisal processes.

A conjecture: most of my colleagues who have voiced concern at the sustainability/persistence of digital journals have no idea how to check whether a title is digitally preserved. There's very low awareness among academics on this topic.

MEDSCAPE article on NICE and Evusheld, with comment from me, @drlennardlee, @FionaCLoud, and @rowntron.

https://www.medscape.co.uk/viewarticle/nice-rejects-covid-drug-evusheld-nhs-use-2023a100038j

Owing to government & process delays, this committee was pushed back & back while COVID continued to mutate. By the time the committee met, almost a full year after the MHRA gave marketing authorisation for Evusheld it was no longer proven to work against the most recent variants

Overly bureaucratic and slow cost appraisal processes for COVID drugs are useless and are outpaced by the virus. And it's costing patients their lives.

It is not acceptable in an ongoing pandemic for drug appraisal processes to take 11 months. We have immunocompromised people in our group who have now been totally shielding for 3 years. The virus will continue to mutate.

Had a dream last night that my random sampler always (and only) returned articles from a Tottenham Hotspur Football Club fanzine.

Ah drat. ITV have cancelled at the last minute. National news often does this due to scheduling constraints (at 9am I asked if we were definitely on and got a "yes"). A shame how difficult it is to get stories run about the ongoing Covid vulnerable. :/

My personal experience with having Evusheld privately: I was hospitalised with Covid in January. I couldn't clear the virus after a month with it. However, I did NOT end up on a ventilator which was the feared outcome given that I am in 3 of the most vulnerable categories.

So, I'll never be able to prove it, but while Evusheld didn't stop me being hospitalised, it's possible it saved my life. As I say, my outcomes were much better than clinicians expected.

I want this protection for everyone. Roll on Evusheld 2/SUPERNOVA.

For me, the scandal with Evusheld is the delay. So many other countries gave their immunocompromised a year's worth of protection with the drug. Our government put it through bureaucratic processes and ran down the clock until it didn't work with new variants.

None of this is the independent medical panel's fault at NICE. The latest evidence doesn't look good for Evusheld. But if you ask WHY that is... It's because of the year-long delay since MHRA authorisation in March 2022. And that's inexcusable.

This cannot happen again. A process that takes 11 months to deliver lifesaving drugs to vulnerable populations in an ongoing pandemic is not satisfactory. Patients are angry and feel abandoned.

Here's the NICE announcement of new processes.

NICE says evidence that COVID-19 treatment Evusheld is effective in protecting vulnerable adults against current variants is lacking as it announces new rapid update process for COVID-19 medicines
https://www.nice.org.uk/news/article/nice-says-no-evidence-that-covid-19-treatment-evusheld-is-effective-in-protecting-vulnerable-adults-against-current-variants-as-it-announces-new-rapid-update-process-for-covid-19-medicines

Well, "we lost the battle but won the war". NICE has published preliminary recommendations against Evusheld on the NHS. I'm not surprised: they delayed so long that the virus evolved. Our shielding members could have had its benefit when it did work. But no.

The good news...

NICE agrees that its process is too slow and not fit for purpose. Evusheld 2, in clinical trials now, will be appraised within 90 days of MHRA authorisation. That's still quite slow! But we're pushing for the fiasco they've put patients through this time never to happen again.

Today's final task was to write the preservation database randomiser. It will return you <count> number of supposedly preserved items in any specific archive, and an equal number that it thinks are not. This will allow for spot checking the database to verify its rough accuracy.

The UK's National Institute for Health and Care Excellence (NICE) will announce its decision on Evusheld tomorrow.

ITV National News should be coming to do a television interview with me about this process and the future.

Saw a headline: "Labour will not give bonuses to water bosses who dump sewage in the ocean" and thought how low the bar is set for policy announcements.

Another kidney clinic today. Where it goes from here is anyone's guess. Abatacept for the rheumatoid could lead to the BK virus getting worse. Or it could be better than baricitinib. In any case, I'm being sent to the low clearance clinic to monitor for if/when we start dialysis.

Urgh. I have a nasty python memory leak that I just can't track down. It's tracemalloc time. (Not as good as "hammer time".)

Import routine working fine on local machine. Got it working on Airflow... Except that the managed cloud instance of Airflow has far less memory than my home dev box. Out of memory killer invoked on these massive preservation databases. Doh.

I love the fun of trying to decode ONIX fields.

"See code list 103S for permissible values for
<ResourceVersionIDType>".

OK. Where the heck is that?

*MUCH GOOGLING LATER*

More kidney and rheumatology monitoring and clinics today. BK virus levels have come DOWN to 7m copies (previously 16m and before that... 93m 😱). But kidney function continues to decline. I'm now being sent to "low clearance clinic". Frustrating.

Sigh. People who don't use railways and who most likely have pictures of Thatcher on their walls decide that the misery of train strikes for other people is the price worth paying.

Wrestled yesterday with VPC/connectivity challenges on a managed Aurora database for way too long so didn't get all the Airflow stuff rewritten. Today is another trip to the hospital, though, for both renal and rheumatology. Might as well do a tour of the whole hospital!

Using .gitignore to ensure you don't leak files with secrets can definitely be exposing at the pip packaging stage if you forget to exclude those same files :|

There's also a bigger task for the week: generate a random sample of works and validate the preservation status, as test cases.

If that proves accurate then I will be able to answer the question: how well digitally preserved is the scholarly record?

Today's jobs:

1. Package the digital preservation library up on pip so Airflow can import it.

2. Write the Terraform infrastructure code for an Aurora postgres database and non-local secrets storage.

3. Rewrite the import routines as Airflow jobs.

When this is done, I will have the basic project structure for injecting any new prototype fields into the Crossref API in future, running as scheduled Airflow jobs.

Continuing to airflow-ize the digital preservation sampler for Labs and to expand its features. One of the crucial missing parts was... test cases. There's no point us providing digital preservation statistics unless we can cross-validate that the data are correct.

But right now: I need a cup of tea. And more painkillers as the others are wearing off.

So I now have this system, where you can specify whether a DOI should or should NOT be present in a particular archive, with a set of test cases. A next task is to expand this list with arbitrary samples and validate against the Keepers Registry.

Unsure re all calculations here, but sharing as it's an interesting thought.

Apparently sending one fewer one-line emails (e.g. "thank you") in Britain would be equal to saving 81,1522 flights to Madrid in carbon emissions.

https://www.ovoenergy.com/ovo-newsroom/press-releases/2019/november/think-before-you-thank-if-every-brit-sent-one-less-thank-you-email-a-day-we-would-save-16433-tonnes-of-carbon-a-year-the-same-as-81152-flights-to-madrid

And yes, I know, it's another displacement of the blame/responsibility for carbon emissions onto individuals, rather than systemic change in the energy sector etc.

It's probably placebo or just random flux in the rheumatoid arthritis as it's too soon for the drugs to have worked, but I woke up today feeling... not entirely dreadful. Quick! Edit some book!

The rest of the day will be cross-validation of my digital preservation database.

They really just don't get that, actually, what people want are decent, functioning public services paid for by general taxation and free at the point of use.

Mandate?

You didn't win a general election with a manifesto setting out what you would do. Your own party members -- a tiny percentage of the population -- reluctantly elected you.

But I guess that's what these wreckers think democracy is these days. The will of Tory party. https://x.com/trussliz/status/1621985629251813378

A mandate to rule because they think they are born to rule.

Suggestion: nobody should listen to this total moron who has caused massive damage to the country's finances already. She's got a bloody nerve to think she's still right.

Today it's more book editing, making some electronic funk music, and going for a short recovery walk by the sea.

Current painkiller regime appears to be working relatively well. I sure know about it when they wear off, but am enjoying getting stuff done.

Phew. I got the Airflow workflow working. Now generating preliminary digital preservation statistics for all Crossref members.

Still much to do on this project, but I'm getting closer to answering the question: how persistent is most content with a persistent identifier?

Urgh. Kidney function results from yesterday turn out to be not good. On a bad trajectory due to the BK virus, despite withdrawal from immunosuppression. Sigh.

Today @gbilder built a cool reverse proxy/annotation system for the Crossref API and I'm using it to patch in the preservation data I've collected. This structure, once we have it working, will allow Labs rapidly to prototype new metadata fields and for the community to try them.

At last, I've had the new meds at hospital. No allergic reaction. Unbelievable logistical hurdles all day though. Absolutely shattered and in much pain with no disability accommodation in hosp processes. Still have to get home with no trains running. Miles to go before I sleep.

This hospital trip is a grim fiasco. First, I go for kidney blood tests. By luck I notice that a load of tests I've been told to have are missing. Much frustrating to and fro between nurses to get it added.

Second, I go to rheumatology. Queue to check in. I am the only person in the queue. I can't stand for long and need a stick. Receptionist ignores me for 5 minutes then midway through checkin starts talking to someone else. I tell her I'm about to collapse.

So I traipse to the pharmacy. Where I have to stand in a queue for 20 minutes just to get the prescription converted from the electronic system to the digital system. I'm then told there's a wait of an hour to collect.

Third, my rheumatology appointment is on time and with the very helpful and kind registrar who really knows my case. However, now the prescription is "done"... I have to go collect it from the hospital pharmacy. It also transpires that a kidney blood test wasn't done earlier.

Once I have the prescription - hopefully soon - I will have to go back to clinic to have the first injection. Then I must wait an hour.

Urgh. Everything today has been difficult and painful to organise. I am totally exhausted and in mega pain.

So I go back to the 3rd floor kidney blood room. I get the missing blood test done. I come downstairs and get a cup of tea. Again, I have to stand to wait for this for 10 minutes.

The only other option would be to get a wheelchair. But then everything takes 20x as long, waiting for hospital porters - needing someone to accompany me etc. But the hospital processes for people with some mobility but who cannot stand for any period of time are dire.

So I queued for another 20 minutes in the pharmacy "collection" queue - which is another standing only queue - only to be told that the funding hasn't been authorised and they have to contact rheumatology.

I want to cry right now from pain and despair at hospital systems.

So now it turns out that, having got the drugs, the pharmacy haven't given me the needles that need to be attached to the epipens. And they don't have them. Doctor is urgently phoning pharmacy now.

Today's the day I start the new rheumatoid arthritis drugs. There might be an end in sight to the constant pain, nausea, fever, and immobility. But they take between 3 weeks and 3 months to work. So I have to hold out a while longer. But it's a start.