Security Now 968 Transcript
0:00:00 - Leo Laporte
It's time for security now. Steve Gibson is here and, yes, we are going to talk about one of the most interesting and, in many ways, scary security flaws on the Internet and I'm not talking about AT&T, although we'll talk about that as well Steve's explanation of the XZ flaw. We will talk about AT&T. What's taking them so long in telling us what's going on? Clearly there's a problem. What's taking him so long in telling us what's going on? Clearly there's a problem. There's also a lot more to talk about, including problems with NPM and PiPi a great Spinrite success story in Steve's own household. That and a lot more. Oh, and also some scary numbers about how many people use ad blockers that and a lot more coming up. Next on Security Now Podcasts you love From people. You trust.
This is Twit. This is Security Now with Steve Gibson, episode 968. Recorded Tuesday, april 2nd 2024. A cautionary tale it's time for security now. You wait all week for this. I know you do. It's a long seven days. But yes, tuesday is here again and so is Steve Gibson, the man in charge of security. Now Hi back, hello, newman. Hello.
0:01:28 - Steve Gibson
Great to be back with you again, as always. And you asked me right off the bat are we going to be talking about XZ? And I said, of course, but I'm, you know, and of course this is what was discovered that almost I mean actually I was going to say almost got into the outside world, but actually it did and there's had to be some rollback. But I titled today's podcast a cautionary tale because you know, we're seeing more problems like this as we're moving forward, more problems like this as we're moving forward.
0:02:07 - Leo Laporte
You've been talking about supply chain attacks for ages, like PyPy and all of these libraries In fact, pypy had a problem, NPM had a problem.
0:02:18 - Steve Gibson
There's more pressure being turned up, so, anyway, we have a great podcast today being turned up. So, anyway, we have a great podcast today. We're going to look at a different reason why all Linux users need to update their systems, if they haven't since February. What 73 million current and past AT&T customers all have in common? We're going to answer the question of what additional and welcome, though very different new features await Signal and Telegram users. Which major IT supplier just left Russia early last week?
What did Ghostery's ad blocking profile survey reveal about Internet users? Whatever happened with that incognito mode lawsuit? That was a class action suit that actually looks like it did more than generate money for the attorneys. What happened with that? Uh, and how are things going in the open source repository world? As I already, you know, suggested, not well. And then I share something kind of special that happened day before yesterday, on Sunday, involving my wife Spinrite and her laptop, and it's probably not what you think. We're going to take a look at another rather horrifying bullet that the Internet dodged once again, but it begs the question. You know they're coming faster and the dodge is barely working. At some point one might hit us, and what would that be like? It's an amazing story.
0:03:54 - Leo Laporte
I can't wait to hear you talk about it, and it really is completely serendipitous that we did dodge that bullet, because it wasn't even a security researcher who discovered it. It's just mind-boggling how close we came on this one, and it's inevitable that, uh, it's not going to always be that way well, and leo, had we not, and we'll get into this, of course, but the bullet that would have hit is serious.
0:04:19 - Steve Gibson
I mean, it's like, not like, oh, if, if you, you know, tap your heels three times in some you know, and in some nonstandard configs that almost no one has, this would have been, it would have slammed the world.
0:04:38 - Leo Laporte
Every bit of this story is a novel. I mean, it's just there should be a movie about it because it's so mind-bogglingly wild, but anyway, I can't wait to hear you tell it. Maybe, if you do it just right, we can option it out and make some money. Steve's coming up with many great security stories and, of course, the picture of the week, but before we do that, I want to welcome a brand-new sponsor to our airwaves. One big thing.
Now here's the dirty little secret that I think a lot of small and medium-sized businesses know. If you're a mid-sized, high-growth organization, your job, pretty much entirely, is to focus on your offerings, right To serve your customers, to make your business go. You don't have the volume of work to keep a full-time privacy and AI team busy, and the dirty little secret is you probably can't even attract or afford the top talent. They're going to Microsoft, google, you know. So it's tough. Fortunately, there's a solution, and that's where one big think comes in. Privacy and AI compliance is here. This is, you know. We're just going to be table stakes from now on. Regulations, though, are constantly changing around the world Constant flux, new regulations developing daily. I just saw, in fact, a whole new AI accord signed by a couple of the big players this week. Organizations you've got to embrace concepts like privacy by design, transparency, purpose limitation, data minimization, data subject rights all of these issues and you can do that with one big thing. With their services, organizations gain the capacity and the ability of a DPO, that's a data protection officer and an AI expert, giving you all of the stuff you need at a fraction of the cost, while maintaining independence requirements. It's kind of a win-win-win. A DPO is an enterprise security leadership role responsible for overseeing data protection strategy compliance and implementation to ensure compliance GDPR, ccpa with California, cpra and so on of their data protection obligations and compliance requirements, serving as the primary point between the company and relevant supervisory authorities, and a whole lot more. But there is some way forward with this and it's One Big Think To learn more about how to give your organization sustainable privacy and AI compliance that works, that you can afford.
Visit OneBigThinkcom alliance. That works. That you can afford. Visit one big thinkcom. That's the number one b-i-g-t-h-i-n-k. One big thinkcom. We thank him so much. Welcome one big thinking. We thank him so much for supporting the good work that mr g is about including, although we have yet to find a sponsor for it. The picture of the week.
0:07:46 - Steve Gibson
Oh yeah, Nobody wants to sponsor this picture, except maybe the underwriters lab.
0:07:53 - Leo Laporte
Wow, all right. What is this? What's going on here?
0:08:07 - Steve Gibson
So this is roughly related to last week's disastrous. Let's create, let's use coat hanger wire hanging over the prongs of a of a usb charger to to hang our cables. This is what happens when you've you're in a room that only has a single one of those european style you know, this looks like a prison room.
0:08:25 - Leo Laporte
I'll be honest with you it doesn't look it was a little depressing yes, I agree it's.
0:08:30 - Steve Gibson
It's a little horrifying. So if you you know the, the european style plug that looks like a vampire has has you know, it's a uk, this is a uk.
0:08:41 - Leo Laporte
Oh, it's a uk, okay, I think so, yeah, um right. Yeah, I'm not. No, no, it isn't.
0:08:45 - Steve Gibson
No, you're right, it's eu, you're right yeah and so so it's just two round holes um and and. But you've got three things that you're trying to put into the to the round holes. So some enterprising individuals said, well, okay, I don't have, you know, one of those AC expansion boxes.
0:09:08 - Leo Laporte
You know what I do have. I have a coat hanger.
0:09:19 - Steve Gibson
I've got some spare wire Physical expansion using some wire wrapped or successively wrapped around three sets of these two pin plugs.
0:09:31 - Leo Laporte
This is not a surge suppressor, this is a surge aggressor.
0:09:36 - Steve Gibson
This is a surge disaster. Yeah, not good. So anyway, just you know, I mean, this exists in the real world. I thank our listeners for finding these and saying okay, steve, I saw this photo and thought of you, this definitely needs to be something to be shared. Do not try this at home, no matter where in Europe you may live. Nightmare, nightmare, wow, wow, yeah. And, needless to say, this is all exposed. Right, you don't want to touch any of this, because the whole point of having those two pins is there's really no way you can hurt yourself electrically. In the US and I'm sure this has happened to you, leo, back in your earlier days, you'd be pulling a cord out and inadvertently let a thumb touch across the two metal prongs and go get yourself a little zap.
0:10:28 - Leo Laporte
Yeah, and you know so I, I frankly know the feeling quite well you know somebody miss somebody in our uh discord is pointing out, uh that in fact chickenhead says with great foresight, they've placed it very close to a metal bed frame. This may be some sort of torture device. I don't know what's going on. It's not good not good, okay.
0:10:53 - Steve Gibson
so, uh, last friday which of course was good friday, uh was not a good friday for linux. Not only did the world first learn uh which we'll be talking about later of a near miss on the dissemination of an SSH backdoor. You know again, this will be the Linux kernel from at least versions 5.14 through 6.6.14. Running the exploit as a normal user, which was released as a proof of concept on GitHub I have a link in the show notes. If anyone wants to go look on a vulnerable machine will grant root access, allowing the exploiter to do whatever they wish. Now, the good news is a local-only exploit, so not remote. This could be used, however, by rogue insiders or perhaps by malware that's already on a computer, allowing it to cause further damage and problems. The affected distributions include Debian, ubuntu, red Hat, fedora and doubtless many other Linuxes. And it had I love this a 99.4% success rate, given 1,000 tries. The author of this tried it 1,000 times. In other words, it worked 994 of those 1,000 times and only failed six times. I noted that the public learned of it last Friday because it was responsibly disclosed. That's the good news to the Linux community insiders back in January, and then, since the end of January. Updates to this, which was given a CVSS of 7.8 flaw, have been rolling out. You know the fix was clear and simple, didn't take long to patch or test. It was very obvious. Once everyone saw it it was like ooh yeah, that's not what we meant or we intended. So it's because the updated code had been available by last Friday for two months that its discoverer felt it was time enough time had gone by and the updates had been pushed out for two months that he was able to disclose it all on github along with a proof of concept. So, and boy, he has a super detailed description of what he found, again links to all that in the show notes. The takeaway for anyone using linux, uh, who may not have updated since before February, meaning that their current build is probably still buggy, is that knowledge of this is now public and widespread. So if anyone untrusted might have or be able to gain access to any unpatched machines, it would be a good time now to get them updated because, as I said, the word is out.
Techcrunch has been on top of the news of a significant breach of very sensitive customer data from AT&T. The story begins five years ago, back in 2019, which TechCrunch first reported 10 days ago, on the 22nd of March, they gave it the headline and this is the first of two stories, because then they updated their coverage. So this first report they gave the headline. At&t won't say how its customers' data spilled online and in fact, that's still the case, but that's a little more difficult, as we'll see, for them to be denying it. So TechCrunch explained, they said, three years after a hacker first teased an alleged massive theft of AT&T customer data, a breach seller this week meaning three weeks ago dumped the full data set online.
It contains and this is the real concern the very personal information of 73 million AT&T customers, including me, I'm sure, yeah, and I was an AT&T customer back in the day. I, finally I had AT&T landlines until all they were ever getting was, you know, robocalls and I thought, oh, come on, why am I paying for this anymore? So a new analysis of the fully leaked data set, which contains names, home addresses, phone numbers, social security numbers and dates of birth In other words, everything you need for identity theft, points to the data being authentic. Some AT&T customers have confirmed their leaked customer data is accurate, but AT&T still hasn't said how its customers data spilled online.
0:16:12 - Leo Laporte
They're like well, I don't know.
0:16:13 - Steve Gibson
You know what we know. We don't know if it was us. August of 2021, which was two years after he got the data to have stolen millions of AT&T customers' data only published a small sample of the leaked records online, hoping to use that as proof, but the sample was so small that it was a little difficult to verify that a much larger breach was authentic. That a much larger breach was authentic. At&t, the largest phone carrier in the US, said back in 2021 that the leaked data quote does not appear to have come from our systems unquote, you know, because we would like it not to appear that way, but AT&T chose not to speculate as to where the data had originated or whether it was valid.
Now Troy Hunt, who we know well, a security researcher and owner TechCrunch, wrote a data breach notification site. Have I Been Pwned? Recently obtained a copy of the full leaked data set. Hunt concluded the leaked data was real by asking AT&T customers if their leaked records were accurate. In a blog post. Analyzing the data, troy said that, of the 73 million leaked records records, the data contained 49 million unique email addresses, 44 million social security numbers and customers' dates of birth. When reached for comment, at&t spokesperson Stephen Stokes told TechCrunch in a statement. Quote once again we have no indications of a compromise of our systems.
0:18:08 - Leo Laporte
Everything's working great here. Folks Very angry, that's just awful.
0:18:14 - Steve Gibson
Yeah, he said. We determined in 2021 that the information offered on this online forum did not appear to have come from our systems. So still just head buried in the sand. He said this appears to be the same data set that has been recycled several times on this forum, meaning this is nothing new. We'd prefer to close this statement now.
The AT&T spokesperson, they said, did not respond to follow-up emails by TechCrunch asking if the alleged customer data was valid or where its customer's data came from. So, basically, again just head in the sand. As Troy Hunt notes, they wrote, the source of the breach remains inconclusive and it's not clear if AT&T even knows where the data came from. Hunt said it's plausible that the data originated either from AT&T or a third-party processor they use, or from another entity altogether. That's entirely unrelated. Again, we just don't know and they're not saying TechCrunch said. What is clear is that even three years later, we're still no closer to solving this mystery breach. Nor can AT&T say how its customers data ended up online. Investigating data breaches and leaks takes time. Conclusion of the first posting AT&T should be able to provide a better explanation as to why millions of its customers' data is indeed online for all to see. Okay, so that was 10 days ago. In follow-up reporting just last Saturday under their headline AT&T resets account passcodes after millions of customer records leak online. I guess maybe AT&T, you know, lumbers along and just takes them a while to say, oh, maybe this is new.
Techcrunch has exclusively learned that phone giant AT&T has reset millions of customer account passcodes after a huge cache of data containing AT&T. On Monday that the leaked data contained encrypted passcodes that could be used to access AT&T customer accounts. So, whoops, it's no longer you know all just five years ago and we don't know what it is, or even if it's actually our data. Suddenly, whoops, we're going to mass reset passcodes. A security researcher who analyzed the leaked data told TechCrunch that the encrypted account passcodes are easy to decipher. Now, that's not quite correct technically. I'll explain what happened in here.
In a second, techcrunch alerted AT&T to the security researchers' findings. In a statement provided Saturday, at&t said quote AT&T has launched a robust investigation Right, not a moment too soon Supported by internal and external cybersecurity experts oh, they have their own internal experts. Based on our preliminary analysis, the data set appears to be from 2019 or earlier, impacting approximately 7.6 million current AT&T account holders and approximately 65.4 million former account holders, and you know, count me among the former category because I left AT&T. The statement also said quote AT&T does not have evidence of unauthorized access to its systems resulting in exfiltration of the data set. Yeah, five years ago, right TechCrunch withheld the publication of this story until AT&T could begin resetting customer account passcodes. At&t also has a post on what customers could do to keep their accounts secure. At&t customer account passcodes are typically four-digit numbers that are used as an additional layer of security when accessing a customer's account, such as calling AT&T customer service in retail stores and online.
0:22:58 - Leo Laporte
We've told people to avoid SIM swapping. Set a PIN with your carrier.
0:23:04 - Steve Gibson
That's the pin, right, yep exactly, and I know that when I did have occasion to call at&t, they would say what's your pin right? And I would tell them what it was, and it was indeed four digits. So and that was, you know, not optional, it was. There was like four, you know, that was all you could do. So so this, writes TechCrunch, is the first time AT&T has acknowledged that the leaked data belongs to its customers, some three years after a hacker claimed the theft of 73 million AT&T customer records, which now has been. You know, even AT&T Vera said yep, we're not happy with that number, but you know, we have to tell the truth when we're confronted with no choice. The AT&T had denied a breach of its systems and actually they're still saying well, we don't know how it got out there, but it does look like it's ours. On Saturday they said it is not yet known whether the data in those fields originated Wow, they're still denying it from AT&T or one of its vendors.
Now, security researcher Sam Crowley told TechCrunch that each record in the leaked data also contains the AT&T customer's account passcode in an encrypted format. Crowley double-checked his findings by looking up records in the leaked data against AT&T account passcodes known only to him. Crowley said it was not necessary to crack the encryption cipher to unscramble the passcode data. Why we're going to see here in a second. He took all of the encrypted passcodes from the 73 million data set and removed every duplicate. The result amounted to get this 10,000 unique encrypted values. In other words, starting at 0000 and finishing at 9999. That's 10,000. So they're encrypted. But after he sorted all 73 million of them and removed duplicates, what do you think he got? Well, he got encrypted representations of every passcode from 0000 to 9999.
Okay, so, in other words, at&t, we now know, used a fixed cipher with no salt to encrypt the entire data set and the encryption never changed. Every encryption maps one-to-one to some four-digit passcode. This means that there's no mystery here, techcrunch wrote. According to Crowley, the insufficient randomness of the encrypted data means it's possible to guess the customer's four-digit account passcode based on surrounding information in the leaked data set. Okay, in other words, since all 73 million users were restricted to using four-digit passcodes and all passcodes are identically encrypted, once any single user's passcode can be guessed and we see what that encrypts to, we also know everyone else who chose the same four digits because they will have an identical encrypted code because of the lack of salt. Correct exactly, it's kind of a mess, yeah, tech crunch wrote. It's not uncommon for people to set passcodes, particularly if limited to four digits that mean something to them. That might be the last four digits of a social security number which is in the data set, or the person's phone number in the data set.
The year of someone's birth in the data set or even the four digits of a house number in the data set.
0:27:25 - Leo Laporte
By the way, this was AT&t's practice I think it still is to set your voicemail passcode to the last four digits of your phone number. So I'll bet you anything that people didn't change it.
0:27:37 - Steve Gibson
It's in the data set nobody changed it.
Yep, yep all of this surrounding data is found. They wrote in almost every record in the leaked data set. By correlating encrypted account passcodes to surrounding account data, such as customer dates of birth, house numbers, partial social security numbers and phone numbers, crowley was able to reverse engineer which encrypted values matched which plain text passcode. In other words, zero protection. At&t said it will contact all of the 7.6 million existing customers whose passcodes it just reset, as well as current and former customers whose personal information was compromised, and that's 76 million in total. So let's hope that when they do this, like this passcode reset, they have a new and improved means of encrypting passcodes and improved means of encrypting passcodes, hopefully using a unique per-user random salt, so that it's no longer possible to create a simple one-to-one mapping of passcodes to their encryptions. And you know, doing this without salt is really so. Last century, gosh and Leo. We're going to talk about signal and telegram, but let's take a break for a sponsor. Thank you.
0:29:11 - Leo Laporte
This is. That's so depressing. Oh, from at&t. Yeah right, you, someday you will be hacked by at&t, you will? You? So sad. All right, let's talk about something a little bit happier.
And of course, it's our fine sponsor, our great friends at Collide K-O-L-I-D-E and I know you've heard us talk about Collide. Did you hear the news? Yep, collide was just acquired by 1Password, and that is actually really good news. Both companies dedicated leading the industry to create security solutions that put users first. For over a year, collide Device Trust has helped companies with Okta ensure that only known and secure devices can access their data right, and that's what they're still doing. But now they're just doing it better as part of 1Password. So if you have Okta and you've been meaning to check out Collide, don't no, don't put it off. Now is a great time.
Collide comes with a library of pre-built device posture checks. You can write your own custom checks for just about anything you can think of. Plus and this is a great feature you can use Collide on devices without MDM. So that means pretty much everything your Linux fleet, your contractor devices, all those BYOD phones and laptops in your company they can all be protected, which means you can be protected and now that Collide is part of 1Password, it's just going to get better. Check it out Collidecom slash security now. Watch the demo today K-O-L-I-D-E Collidecom slash security now. We thank them so much for supporting Steve's great work here at the Security Now headquarters in beautiful Irvine, california.
0:31:05 - Steve Gibson
Wherever that is, wherever that may be. So I haven't and really don't have much need for cross-platform secure messaging. Imessage does everything I need. But, as I mentioned a few years ago, the talented PHP programmer, rasmus Vind, who's a listener of ours, he's the guy who helped integrate Squirrel into the Zenforo web forums that GRC uses and, by the way, I'm so happy with that choice. If anyone is looking for forum software, I could not be more pleased with the decision I made many years ago. Could not be more pleased with the decision I made many years ago. Anyway, he suggested that we use Signal to converse while we were working to get all of the details worked out of getting Squirrel to log in. So that exposed me to Signal and to its somewhat awkward restrictions which, leo, you and I have talked about from time to time because you know it's kind of a pain. So I was interested and pleased to read that Signal is beginning to loosen things up a bit. We talked about another instance of that loosening, I think a week or two ago the site SignalUpdateInfocom, which apparently exists just to watch and report on Signal happenings.
They wrote cloud backup status in development. They said there has not been any official announcement but it appears the signal is currently working on cloud backup for iOS and Android. This feature would allow you to create cloud backups of your messages and media and media. While we have known they wrote about cloud backups since a commit to Signal iOS on October 20, 2023, a recent commit to Signal for Android has revealed many more details. It looks like there could be both free and paid tiers. The free tier would provide full message and text backups, but only media backups from the last 30 days. The paid tier could provide full message, text and media backups, with a storage limit of one terabyte. From the commit they wrote. We also know that signal backups will be end-to-end encrypted and completely optional, with the ability to delete them at any time. Cloud backups could significantly improve usability on all platforms by preventing complete data loss in the event that you lose your phone or encounter some other issue, and apparently it also suggests that there would be a means for migrating content between devices, which would really be a nice boost in usability. So you know, bravo for signal. I don't know if they're feeling competitive pressure. If they're feeling like well, you know there's like their own users are saying why can, can't I do X, y or Z and they're saying well, yeah, ok, that's probably a good point. We need to make this better.
On the telegram side, telegram users initially located in Russia, ukraine and Belarus have received a welcome new feature which allows users to restrict who can message them. Last Wednesday, telegram's founder, pavel Durov, sent the following message through Telegram. Now, when I looked it up, it was in Russian, so I fired up a copy of Chrome in order to bring it up under Chrome and have Google translate it for me, and I have to say it did a pretty good job. So the English translation of what Pavel wrote in Russian reads Four days ago, russian-speaking Telegram users began to complain about messages from strangers calling for terrorist attacks. Within an hour of receiving such complaints, we applied a number of technical and organizational measures in order to prevent this activity. As a result, tens of thousands of attempts to send such messages were stopped and thousands of users participating in this flash mob faced permanent blocking of their Telegram accounts From the beginning of next week and he wrote this last week, so this week, all users from Russia, ukraine and Belarus will be able to limit the circle of those who can send them private messages. We are also implementing AI solutions to handle complaints even more efficiently. And he finishes. Telegram is not a place for spam mailings and calls for violence. And you know, bravo to Google Translate for coming up with that in English for me from the Russian, because you know that was flawless.
But speaking of Google Translation, when I fired up Chrome to view the page and had it perform the translation, I then copied it and posted the result over to Firefox and I just sort of forgot about Chrome. I left it running. Got about Chrome, I left it running. So there was one page open doing nothing. Yet it caused my machine's fans to spin up to full speed, which I thought what the heck. And that you know. When I noticed that, I opened Task Manager and saw that Chrome was at the top of the process utilization list, consuming half of my very strong multi I think I got 16 cores or something while it was doing nothing. So I had forgotten that it used to do that when I was using it there for a while before I switched back to Firefox and my machine became blessedly silent. So thanks anyway, chrome, wow.
Also speaking of Russia, hp has ceased all operations and has exited the Russian market ahead of schedule. Hp had terminated all shipments into Russia. Two years ago, in February, a little over two years ago, in February of 2022, after Russia's invasion of Ukraine Three months later, in May of 2022, hp began the slow process of winding down all of their corporate operations and had planned to make their final exit next month, in May 2024. So basically, they said, over the back in 2023, in May, over the next two years, 24 months, we're going to be winding things down. You know, maybe they hope they wouldn't have to leave, they hope that this mess with Ukraine would get resolved and, and you know, maybe they hope they wouldn't have to leave. They hope that this mess with Ukraine would get resolved and, and you know, they could stay. But no, that hasn't happened. So HB pulled out of Russia last week, at the end of March, two months ahead of its planned departure.
The move surprised Russian companies, which are now no longer able to update drivers or contact HP support, and I I I have to say I was wondering what about those printers that you know tend to be rather persnickety? I wonder what's going to happen with. You know, if they try to put non-HP ink into an HP printer in Russia, you know what will happen, who knows? Okay, this was. I got a kick out of this next piece because of what we learned about advertisers advertisers use of ad blockers. Ghostery published what they called their tracker and ad blocker report, which was the result from research conducted by an independent third party research firm-wide. It found that and there's four main bullet points individuals who have experience in advertising programming and cybersecurity are significantly more likely to use ad blockers than average Americans. That's hysterical.
0:39:32 - Leo Laporte
Isn't that? I love it. It's just great. I guess they know something. That I love it. It's just great. I guess they know something huh.
0:39:38 - Steve Gibson
These industry insiders are more skeptical of their online safety, underscoring concerns about the current severity of user tracking. Third bullet point Americans are underestimating the dangers of health and political tracking. And, finally, lesser known big tech players, so distrust among these experts. Okay, so I have an infographic that I duplicated from the report. It shows that, whereas 52% of all Americans use an ad blocker, that number is actually astounding, that's more than half. Wow, it is a far higher number than I would have ever guessed. Wow, that's like, yes, surprising.
0:40:29 - Leo Laporte
Cory Doctorow has called this the largest consumer boycott in history, and I think these numbers bear it out. It's incredible.
0:40:36 - Steve Gibson
Yeah, okay, so 52 of all americans. On the other hand, 66, so two out of every three of experienced advertisers block at their own ads, you know, or you, theirs and everybody else's, and I suppose they don't wish to be tracked either. Yeah, okay, the infographic divides all of those surveyed into four groups Everyone that was that. All Americans. Everyone experienced advertisers, experienced programmers, you know, coders and cybersecurity experts, and that's also the order of increasing use of ad blockers. So everyone advertisers, coders and security experts, with the, with the percentages percent 66, 72, and 76 percent. So just over three quarters 76 percent of cybersecurity experts surveyed are using ad blockers. The other interesting question each group was asked was why those who are using ad blockers are doing so. They were asked to choose among one of four reasons for the protection of their online privacy to block ads, to speed up page loading or none of the above. Interestingly, for every group of users those groups the ranking among those four reasons privacy, freedom from ads, web speed or other was identical and in that order, the only difference was in the distribution among those reasons. The generic all-Americans group was the most evenly split between privacy and not wishing to be confronted with ads, at 20 percent and 18 percent respectively, and only 9 percent were using ad blockers for speed experts. Up at the other end, the split was much greater, at 30% wishing for privacy and 19% wanting not to see ads. Now, with the changes, google has not only been promising, but is now well on the way to delivering with the implementation and enforcement of their privacy sandbox technologies in Chrome, the connection between advertising and privacy encroachment is truly being broken for the first time ever, but it's a significant change that we can expect the recognition of to take quite some time to spread and, I would argue, probably much more time than we even expect.
I've heard some of our listeners who simply don't believe it. They don't really care about the technology behind it. For them, it's just blah, blah, blah, blah, blah. Even if they understand it, they're so jaded that they cannot conceive of a world where we are not implicitly paying for the web by relinquishing our personal information. On my side, you know, I believe in technology that I understand technology that I understand, and thanks to this podcast, which forced me to invest in acquiring an understanding of it so that I could share that understanding, I get what Google has wrought. You know. It is good, it is right and it makes sense. If it was me doing this, you know, like me doing Squirrel, it wouldn't matter how good it was. But it's not me, it's Google and it's Chrome, and that means it's going to matter. It's just going to take a long time I mean, you know, a long time for the rest of the world to catch up, and you know that's not a bad thing either. Right, inertia creates stability and that's also good. You know, and all of those techies who have been coding the technology underlying the tracking, you know they're going to need some time to find new jobs, so this will give them some time.
As I noted recently, it's the presence of advertising that's financing the web and that shows no sign of changing. Advertisers appear to be willing to pay double when they know that their ads are being well targeted, and double can easily make the difference between a website being financially viable and not. But, as the statistics which goes to recollected showed, everyone knows that the way advertising initially evolved has not been privacy friendly. So we've needed a long term solution for giving advertisers the ad targeting that they're willing to pay for, while not trading away our personal information and privacy. Google's final solution, which amounts to moving all advertising, auctioning and ad choice into the user's web browser. I think is brilliant. It flips everything on its head, but it's a massive change and that will not happen overnight. So we're getting there, but it's going to take a while.
And speaking of Google and privacy, remember that confusion over just exactly how incognito those using Chrome's incognito mode really were. Google's defense was that its users, who were upset to learn that they were still being tracked and profiled while using incognito mode, had simply failed to read the fine print about what exactly incognito mode did and mostly did not protect them from. Well, it's time to massively delete, it turns out, everything that google learned about their chrome users while they believed they were in fact, incognito and weren't so much. The hacker news just carried this morning a bit of news and what's been learned during the lawsuit discovery, which you know many companies try to avoid because you know their employees go under oath to be deposed and the truth comes out Right. So here's what Hacker News said. Here's what Hacker News said.
Google has agreed to purge billions of data records reflecting users' browsing activities to settle a class action lawsuit that claimed the search giant tracked them without their knowledge or consent in its Chrome browser filed back in 2020, alleged the company misled users by tracking their internet browsing activity while they thought it remained private when using the incognito or private mode on web browsers like Chrome. In late December 2023, it emerged that the company had consented to settle the lawsuit. The deal is currently pending approval by US District Judge Yvonne Gonzalez-Rogers, a court filing yesterday, on April Fool's Day, said the settlement provides broad relief, regardless of any challenges presented by Google's limited record-keeping. As part of the data remediation process, writes the Hacker News, google is also required to delete information that makes private browsing data identifiable by redacting data points like IP addresses, generalizing user agent strings and remove detailed URLs within a specific website. You know, retain only the domain-level portion of the URL. In addition, it has been asked to delete the so-called x-client-data header field, which Google described as a Chrome variations header that captures the quote state of the installation of Chrome itself, including active variations, as well as server-side experiments that may affect the installation unquote. What's significant there is that this header is generated from a seed value, making it potentially a random seed, making it potentially unique enough to identify specific Chrome users. In other words, there's a serial number in the query headers that Chrome has been using, whoops.
Other settlement terms require Google to block third-party cookies within Chrome's incognito mode for five years, a setting the company has already implemented for all users. The tech company, meaning Google, has separately announced plans to eliminate tracking cookies by default by the end of the year. Of course that's the whole privacy sandbox thing. And of course that's the whole privacy sandbox thing. Google has since also updated the wording of incognito mode as of January of this year to clarify that the setting will not change. Quote how data is collected by websites you visit and the services they use, including Google. So you know they're being more clear and more explicit. And here's the biggie that came out of the depositions. The lawsuit extracted admissions from Google employees that characterized the browser's incognito browsing mode as, as quote, a confusing mess, effectively a lie and a problem of professional ethics and basic honesty. Unquote Ouch. It further laid bare internal exchanges in which executives argued incognito mode should not be called private because it risked exacerbating known misconceptions.
0:51:40 - Leo Laporte
Like that it was private. Yeah, oh, yeah, that one, that one Right.
0:51:46 - Steve Gibson
Just a little problem, a minor, and finally, we will be talking shortly about the problems of malice in the open source world. It's a problem everyone is having to deal with. The NPM repository discovered eight new malicious packages last week, and PyPy was again forced to disable new user account creation and package uploads after being hit by a malware submission wave. Hundreds right, there was a ton of them, yes, yes, and Ubuntu's caretaker, canonical, has just switched to manual reviews for all apps submitted to the Ubuntu OS App Store. They needed to do this after multiple publishers attempted to upload malicious crypto wallet applications to the store over the past couple of weeks. The problem focused enough upon crypto wallets that they planned to create a separate app submission policy for cryptocurrency wallets going forward. So, yes, it's kind of a mess out there it's bad.
These people are bad. And, leo, I want to share with everybody something that happened on Sunday. But let's do one more sponsor break. Yes, and then I will do that Gladly.
0:53:09 - Leo Laporte
We like to tell our sponsors we're grateful that they're here, and in this case, it's one that's been with us for many years, and I know you know the name Melissa. Melissa is the data quality expert. Melissa has helped over 10,000 businesses worldwide harness accurate data with their industry-leading solutions, processing over 1 trillion addresses, emails, names and phone records. I mean, they are so good. In fact, g-true, in its 2022 grid report, has once again recognized Melissa as leaders in data quality and address verification. And because Melissa cares about your bottom line, they offer free trials, they offer sample codes, flexible pricing with an ROI guarantee. You get unlimited technical support to customers and there are customers all over the world. And here's some good news about security and privacy. Melissa is a FedRAMP authorized address data resource. That means you've got the highest level of security for your customer data, of course, gdpr and CCPA compliant as well. It meets SOC 2 and HIPAA high trust standards for information security management. Your data is safe with Melissa. Now you can also improve your e-commerce operations with Melissa. They're an Esri partner, which means Melissa's cloud-based tools standardize, validate, verify and enhance data on which top business operations rely, ensuring address data is seamlessly location optimized and you can try it yourself for free.
Don't forget to download the free Melissa Lookups apps. They're on Google Play and the App Store. There's no signup required. Just search for Lookups in your app store. Get started today with 1,000 records cleaned for free. Melissacom slash twit. That's Melissacom slash twit, steve. We have one more ad break to do. I know we're getting to the fabulous story of XZ, but I want to hear what you did for Lori the other day.
0:55:13 - Steve Gibson
Okay. So the day before yesterday my wonderful wife, lori had her first experience of what Spinrite can mean for long-term solid-state storage management and maintenance. It meant a lot to me so I wanted to share it with our listeners. Sunday morning she set up a Dell laptop to take a Q that's what we call a QEEG, which is short for quantitative electroencephalogram. Wow, this is the first phase for delivering EEG biofeedback therapy to her clients. A client sits motionless for 15 minutes with their eyes open and another 15 minutes with their eyes closed. They're wearing an elastic cap covered with EEG electrodes recording the surface electrical potentials across the exterior of their skull. Lori does these on Sunday mornings since there's no noise from nearby gardeners and it's important to have a quiet setting. While she was getting everything ready, on Sunday she turned the Dell laptop on to boot it up and she began waiting and waiting, and waiting, and I don't recall whether I happened to wander by or she called me over, but you know those. The spinning dots were circling and the little hard drive light icon was on solid. Uh, she was a little bit impatient finally, and beginning to get a little worried, like she depends upon this, and the people were on their way, and was she going to be able to do this? I pointed to the hard drive light that was on, solid, and just told her that, you know, the machine was still working on getting booted up but finally, after a truly interminable wait, windows showed some signs of life when the desktop finally appeared. She started moving the cursor around, but the desktop was incomplete, and I pointed to the little hard drive light, again showing her that, you know, even though the desktop was visible, windows was still working to finish getting itself ready. And you know, those of us who have known Windows for what seems like our entire lives know that Microsoft used to get complaints over how long Windows was taking to boot. So they arranged to put the desktop up, like immediately, as fast as it possibly could, even though Windows wasn't actually ready. Drives me crazy, oh, I know, so's so cheap, it just yeah, exactly. So after like another full minute or so, the rest of the tray icons finally populated and the drive light began flickering, meaning spending some time being off as Windows finally began to wrap up its work and get settled.
The program that she uses to record EEGs is a true monstrosity. It was written, unfortunately, by the engineers who designed and built the multi-channel EEG amplifier. The hardware is a true work of art, but the software that goes with it not so much. So she wondered how long that would take to start up. She launched it and again waited and waited, and waited and it too finally showed up. She was relieved and knew that when the family you know that she would be working with arrived, she'd be ready. Having observed this, I said it looks like it would be worth running Spinrite on that laptop. So a few hours later, after a successful session of EEG recording, she brought the laptop to me. I plugged it in, booted into Spinrite and set it going on level three to perform a full rewrite refresh of the drive, and I had to push past 6.1's new warning screen reminding me that I was going to be rewriting mass storage media that preferred being written to being read or being read to being written. So I did that and started Spinrite running on Laurie's EEG recording laptop.
Looking at the real-time monitoring display, where the reading and writing cycles are very obvious, most of the time was being spent reading, and once a 16-megabyte block had been successfully read, rewriting it was just a flicker on the screen. It has a series of bars. Reading is the top bar, the final rewrite is the bottom bar, and so there were little bursts of rapidly alternating reading and writing, but most of the very long pauses were just the SSD sitting there trying to read its own media. This was a 256 gig SSD and I had looked. About 89 gigs of that were in use, and I noticed that by the time Spinrite was about halfway through, the reading and writing phases were flickering back and forth almost without interruption, with occasional pauses during writing, while the SSD buffered, flushed and rearranged its data. So by that point I knew that we were through with the region that mattered.
I interrupted Spinrite, removed the little USB boot drive and rebooted, and the difference was truly astonishing. Windows booted right up and was ready in maybe like 15 seconds or so. So I shut it down, moved it back to where the rest of her EEG equipment was that she'd been using earlier. Before we left the house for Easter dinner, I told her that Spinrite had finished with her machine and invited her to sit down and boot it up. Needless to say, she was astonished by the difference.
She then launched the EEG recording software and it too started almost immediately. So you know, she's put up with me for the past three and a half years, working every day and weekends and nearly every evening, without a break or vacation. She's taken a few trips to visit friends, while I've stayed home to work, and she's let me because she knows it's what I've wanted to do more than anything else. But for this to work, for this work I've done, to have made that much difference in the performance and operation of a machine she depends upon. Well, it was a gratifying moment and you know, yes, honey, your husband's not insane after all you earned your keep, Steve.
1:02:19 - Leo Laporte
You got another 10 years.
1:02:20 - Steve Gibson
Good job so we discovered this phenomenon early in Spinrite's development, after I wrote the ReadSpeed utility, which was meant as just a platform for testing. Spin writes new drivers. We know and expected that spinning hard drives transfer data more slowly as we move toward the end of the drive, but we naturally expected solid state storage to have uniform performance across its media and brand new SSDs do. What we discovered was that SSDs that had been in service for a while had grown astonishingly slow at the front, where files live that are often read but not often rewritten. You know that's the operating system, which you know has gotten ridiculously large. What's fascinating to me as an engineer is that the SSD in Lori's laptop could have become that slow while still eventually managing to return error-free results. No, there were no bad spots found, it was just astonishingly slow.
1:03:43 - Leo Laporte
That's interesting because on spinning drives usually there's something bad that can hardly read the sector and has to try a lot. But this isn't that.
1:03:52 - Steve Gibson
Well, we know that inside SSDs are analog bit storage cells where each cell contains a voltage that represents some number of binary bits. Originally it was just one bit, so the cell was either all on or all off. You know, fully charged or fully discharged, high or low, there's always pressure to achieve higher density. But the cells were already tiny, so designers decided to store four voltages per cell instead of only two voltages. Since four voltages could be used to represent two binary bits instead of just one, that allowed them to instantly double the storage density for the entire system. Same number of cells just store two bits per cell. But why stop there? Eight voltages could store three bits, and 16 voltages God help us could be used to store four.
Well, there's a well-known problem with SSDs, known as read-disturb, where reading from a block of cells subtly disturbs the storage of neighboring cells, of neighboring cells. We're witnessing that as time passes, ssds gradually require more and more time to perform error correction, or error correction is being used across a greater and greater portion to fulfill a request. And it may also be that an SSD's error correction is not a fast thing to do, since it's assumed not to be needed often, and it's not a stretch to imagine that, if things are allowed to get bad enough, a solid-state drive will eventually report that it's unable to read and recover the contents of a block of data. This is why preventative maintenance makes just as much sense. It turns out for solid-state media as it always has for spinning platter drives.
This was not something that anyone really understood and appreciated until we began working on Spinrite 6.1. An SSD is not only being restored to factory fresh performance, but the reason its performance is restored is that the voltages in the storage cells that had gradually drifted away over time from their original ideal levels will be reset. That means no more extensive and slow error correction, much faster performance as a result and presumably restored reliability. So anyway, as I said, it meant a lot to me that Lori was able to finally see what I've been up to for the past three and a half years and experience some of the benefit you know firsthand.
1:06:59 - Leo Laporte
Did she look at you and go oh, it really works.
1:07:03 - Steve Gibson
She was astonished. It's funny too, because she immediately said oh, we should have taken a before video. Yes, so we would have a, and then? And then she said is there any way to put it back? Oh, I said no, honey, it's like we'll have to just wait a few years for it to slow down again. Yeah, and then you know.
1:07:24 - Leo Laporte
So every what, every year or so do? You should do this just to refresh.
1:07:30 - Steve Gibson
I mean, you know people we've had people say, like, who are using MacBook Airs that, like you know it was, I'm sure it was faster when it was new. Yeah, yeah, yeah, yeah, it turns out. Yes, it actually has. It's not. We know, with an SSD it's actually the SSD itself is spending more and more time doing error correction. As the sectors are becoming softer, the actual bits are softening and a rewrite is what they need. So what's exciting is that 6.1 works, but it's a bit of a blunt instrument. I mean, it's all we have now. What I'm excited about is that 7 will be able to find the individual slow spots and surgically selectively rewrite them.
1:08:28 - Leo Laporte
This is not trim, though this is not everybody in the chat room saying, oh, this must be trim. No, this is separate from trim, nothing to do with trim. Yeah, yeah, is not everybody in the chat I'm saying, oh, this must be trim.
1:08:34 - Steve Gibson
No, this is separate from trim.
1:08:35 - Leo Laporte
Nothing to do with trim yeah, yeah, it's just voltage, uh, fluctuation and error correction. That makes a lot of sense actually. Yeah, so you're, in effect, you're, you're just erasing that sector and rewriting it so that it's back to the way it was.
1:08:49 - Steve Gibson
Yes, I am exactly I'm. I'm putting the data back, but after the error correction. So the individual bits had drifted and error correction was able to, because, basically, error correction is extra bits and so by using the extra bits that are not part of the data, it's possible for the entire thing to still be reconstructed. But that takes time, which is why reading slows down. But because SSDs know that they're fatigued from rewriting, they don't do it themselves. Alvin once told me. You know, alvin.
1:09:36 - Leo Laporte
Toto, yeah, yeah.
1:09:39 - Steve Gibson
Alvin Toto told me that no SSDs ever rewrite themselves, so it takes an external agent to do that, and so Spinrite serves that function. Basically, while the sector can still be corrected, it takes the corrected data and rewrites it, which basically resets all the bits firmly so that it no longer needs correction and thus it can be read much more quickly.
1:10:07 - Leo Laporte
It would also probably be accurate to say you don't want to do this too often. I mean the reason they don't do it is because they don't want to wear themselves out too often. I mean the reason they don't do it is because they don't want to wear themselves out.
1:10:15 - Steve Gibson
On the other hand, you look at the number of rewrites that SSDs are able to Nowadays it's hundreds of thousands, so one a year is not making a big difference.
1:10:27 - Leo Laporte
Would that be your recommendation, maybe once a year, or when you see it slowing down, I guess? Yep. Well, as soon as you get this working for the Mac, let me know, because it's true, these MacBooks tend to slow down over time.
1:10:41 - Steve Gibson
Yep Well, and the one that will work on the Mac will be the UEFI-based because Apple's earlier ones would boot on older Macs, so people who have Macs from 2008 are able to run Spinrite on them. It runs on older Macs but not on new ones, because Apple went pure UEFI and it will not run on ARM-based Macs probably ever oh.
1:11:06 - Leo Laporte
So, yeah, well, that pretty much eliminates everything that's been sold in the last couple of years.
1:11:11 - Steve Gibson
I mean, it is a PC-based maintenance tool. Yeah, it's Intel. Yeah, yeah, yeah it's.
1:11:16 - Leo Laporte
Intel Because you write in x86 ASM there really is. It's not a high-level language. We're talking here, so should we do our last break yeah. Let's do the last break, because I'm very excited. I want to hear more about.
1:11:30 - Steve Gibson
I knew you would cover this.
1:11:31 - Leo Laporte
And, as I said, this is a movie baby. I can't wait to hear about XZ. But first let's talk about Vanta. Vanta is your single platform for continuously monitoring your controls, reporting on your security posture, streamlining your audit readiness. Managing the requirements for modern security programs is getting more and more challenging and more and more time-cons consuming, which is why you'll be glad to have Vanta. Vanta gives you one place you can centralize and scale your security program. Vanta quickly assesses risk, streamlines security reviews and automates compliance for SOC 2, ISO 27001, and more. You can leverage Vanta's market-leading trust management platform to unify risk management and secure the trust of your customers. Plus, use Vanta AI to save time when completing security questionnaires. Let it fill it out for you.
G2 loves Vanta year after year and, by the way, some really good reviews there, like this one from a CEO. I'll say it in a CEO voice. Vanta guided us through a process that we had no experience with before. We didn't even have to think about the audit process. It became straightforward. Actually, CEOs don't sound like that anymore, do they? And we got SOC 2, Type 2 compliant in just a few weeks. How about that? Help your business, scale and thrive with Vanta. To learn more. Watch Vanta's on-demand demo at vantacom slash security now. That's V-A-N-T-A dot com slash security now. Thank you, Vanta, for supporting Steve and his great work, and I've been looking for it all week, ever since this XZ vulnerability surfaced, to hear your telling, because it's a story with so many layers, so interesting. Oh, I've got to turn your microphone on, otherwise we won't hear it. Go ahead.
1:13:28 - Steve Gibson
And it's got some cool technology too. Yes, it does. So the runner-up title for today's podcast, which I decided I settled on a cautionary tale yes, it does. Malicious developer that the scheme was discovered. What was discovered was that, by employing a diabolically circuitous route, the system SSH daemon, which is to say the SSH incoming connection accepting server for Linux, would have had a secret and invisible back door installed in it that would have then allowed someone anyone anywhere, using a specific RSA public key, to authenticate and log in to any Linux system on the planet and also provide their own code for it to run. So to say that this would have been huge hardly does it justice. Okay, now to be clear, unlike last week's coverage I mean again, I want to put this in the proper context Unlike last week's coverage of the GoFetch vulnerability, where Apple was informed 107 days before the public disclosure by a team of responsible researchers, in this case someone actually created this extremely complex and very well hidden backdoor and had it not been discovered by chance due to some small errors made by this malefactor, it would have happened. This would have happened the Linux distros would have been updated some actually were and refreshed, and they would have gradually moved out into the world to eventually become widespread, and this code was so well and cleverly hidden that it might have remained unseen for months or years. Fortunately, we'll never know. I imagine it would have been eventually discovered, but who knows what damage might have been done before then discovered? But who knows what damage might have been done before then? And imagine the frenzied panic that would have ensued when it was discovered that all builds of Linux with publicly exposed SSH services contained a remotely accessible keyed backdoor.
Well, that's a podcast I would have liked to deliver, but fortunately we dodged this one. So I went with the title A Cautionary Tale, because the bigger takeaway lesson here should not be this specific instance, which I'll describe in further detail in a minute. It should be that there's nothing about this that's necessarily a one-off. This is the threat and the dark side of a massive community of developers working collectively and largely anonymously. There's no question that by far nearly every developer is well-meaning I'm certain that's the case. But the asymmetry of the struggle for security, where we must get every single thing right every time to be secure and only one mistake needs to be made once to introduce an insecurity, means that unfortunately, the good guys will always be fighting an uphill battle. Same asymmetric principle applies to large-scale software development, where just one bad seed, just one sufficiently clever malicious developer, can have an outsized effect upon the security of everything else.
Okay, now I'm going to give everyone the TLDR. First, no-transcript. How do you go about hiding malicious code in a highly scrutinized open source environment which worships code in its source form, so that no one can see what you've done? You focus your efforts upon a compression library project. You focus your efforts upon a compression library project. Compression library projects contain test files which are used to verify the still proper behavior of recently modified and recompiled source code. These compression test files are, and are expected to be, opaque binary blobs. So you very cleverly arranged to place your malicious binary code into one of the supposedly compressed compression test files for that library, where no one would ever think to look. It's I mean again. One of the points here that I didn't put into the show notes is, unfortunately, once something has seemed to have been done, people who wouldn't have had this idea originally wouldn't have the original idea. They're like oh that's interesting. I wonder what mischief I can get up to. So we may be seeing more of this in the future.
In their reporting of this, ars Technica interviewed Will Dorman, who we've referred to before in the past. He's a senior vulnerability analyst at the security firm and an allegiance A-N-A-L-Y-G-E-N-C-E an allegiance Will noted that because the backdoor was discovered before the malicious versions of XZUtils were added to production versions of Linux, he said quote it's not really affecting anyone in the real world. All caps. That's only because it was discovered early due to bad actor sloppiness. Had it not been discovered, it would have been catastrophic to the world, unquote.
Okay, so all of this erupted last Friday, on Good Friday, with a posting to the OSS security list by the Microsofty who stumbled upon this Andreas Frund, f-r-e-u-n-d. As a consequence of this being a huge deal, security firms and developers all plowed into and reverse engineered all of the available evidence. Now I'm sure that Ars Technica felt last Friday, being a news outlet, that they needed to say something. So Dan Gooden wrote what he could, given the, at the time, limited amount of available information. But then late Sunday morning, dan published a second piece that did a far better job of pulling this entire escapade together and characterizing what's known, as well as some of the intriguing backstory, and actually Kevin Beaumont, who posts on Mastodon. Now from his Gossy the Dog profile also had something really great that I'll share. Anyway, I've edited from Dan's second posting. So here's what we know thanks to Ars Technica and Dan's update.
He wrote on Friday a lone Microsoft developer rocked the world when he revealed a backdoor had been intentionally planted in XZUtils, an open source data compression library available on almost all installations of Linux and other Unix-like operating systems. The person or people behind this project likely spent years on it. They were likely very close to seeing the backdoor update merged into Debian and Red Hat, the two biggest distributions of Linux. When an eagle-eyed software developer spotted something fishy, spotted something fishy. Software and crypto engineer Filippo Velsorda said of the effort which came frightfully close to succeeding. Quote this might be the best executed supply chain attack we've seen, described in the open, and it's a nightmare scenario. Malicious. And it's a nightmare scenario. Malicious, competent, authorized upstream in a widely used library. Unquote. Dan writes XZutils is nearly ubiquitous in Linux.
It provides lossless data compression on virtually all Unix-like operating systems, including Linux. Unix-like operating systems, including Linux. Xzutils provides critical functions for compressing and decompressing data during all kinds of operations. Xzutils also supports the legacy LZMA format, making this component even more crucial. So what happened? Andreas Frund, a developer and engineer working on Microsoft's PostgresQL offerings, was recently troubleshooting performance problems. A Debian system was experiencing with SSH, the most widely used protocol for remotely logging into devices over the Internet. Specifically, ssh logins were consuming too many CPU cycles and were generating errors with Valgrind, a utility for monitoring computer memory. With sheer luck and Frun's careful eye, he eventually discovered the problems were the result of updates that had been made to XZutils. On Friday, frund took to the open source security list to disclose that the updates were the result of someone intentionally planting a backdoor into the compression software.
It's hard to overstate the complexity of the social engineering and the inner workings of the backdoor. Thomas rocha, a researcher at microsoft, published a graphic on mastodon that helps visualize the sprawling extent of the nearly successful endeavor to spread a backdoor with a reach that would have dwarfed the SolarWinds event of 2020. And for anyone who's interested, I've reproduced the infographic on page 15 of the show notes. So what does the backdoor do? Malicious code added to XZUtil's versions 5.6.0 and 5.6.1 modified the way the software functions.
The backdoor manipulated SSHD, the executable file used to make remote SSH connections. Anyone in possession of a predetermined encryption key could stash any code of their choice in an SSH login certificate, upload it and execute it on the backdoor device. No one has actually seen code uploaded, so it's not known what code the attacker planned to run. In theory, the code could allow for just about anything, including stealing encryption keys or installing malware. Answering the question about how a compression utility can manipulate any process as security-sensitive as SSH any process as security sensitive as SSH Dan explains any library can tamper with the inner workings of any executable it is linked to. Often, the developer of the executable will establish a link to a library that's needed for it to work properly. Open SSH the most popular SSHD implementation does not link the LibLZMA library, but Debian and many other Linux distributions add a patch to link SSHD to SystemD, a program that loads a variety of services during the system boot up, and system D in turn links to live LZMA, and this allows XZutils to exert control over SSHD. Okay, and then, of course, everyone wants to know how all this happened. Dan writes, and this should give everyone some chills, and we'll get more into this with what Kevin wrote. Dan said it would appear that this backdoor was years in the making In 2021, someone with the username J-I-A-T-7-5, and I'll just refer to them as G, made their first known commit to an open source project.
In retrospect, the change to the libarchive project is suspicious because it replaced the safe underscore fprint function with a variant that has long been recognized as less secure. No one noticed at the time. The following year, gt75 submitted a patch over the XZutils mailing list and almost immediately, a never-before-seen participant named Jigar Kumar, j-i-g-a-r-k-u-m-a-r. Jigar Kumar joined the discussion and argued that Lass Collin, the longtime maintainer of XZutils, had not been updating the software often or fast enough. Kumar, with the support of Dennis Enns, also never before seen and never since, and several other people who had never had a presence on the list and were never seen again, pressured Lass-Colin to bring on an additional developer to maintain the project. In January 2023, gt75 made their first commit to XZUtils.
In the months following GT75, who used the name GTAN, became increasingly involved in XZUtils affairs. For instance, tan replaced Colin's contact information with their own on OSS Fuzz, a project that scans open source software for vulnerabilities that could be exploited. Tan also requested that OSS Fuzz disable the iFunc function during testing, a change that prevented it from detecting the malicious changes tan would soon make to XZ utils. Okay, now I'll just note that that sort of request is not unusual in itself. It's a bit like a website using the robots. In itself it's a bit like a website using the robotstxt file to keep spiders out of places where they might cause some damage to the site or get themselves hopelessly lost and tangled up. You know, it's possible that aggressive, random fuzzing which is hoping to detect unknown problems might inadvertently trigger a known, deliberate problem and behavior. So asking for fuzzing exceptions is not inherently suspicious. In this case the perpetrator did have an ulterior motive, as it turned out. So Dan finishes.
In February of this year, tan issued commits for versions 5.6.0 and 5.6.1 of XZ Utils. The updates implemented the backdoor. In the following weeks, tan or others appealed to developers of Ubuntu, red Hat and Debian to merge the updates into their OSs Eventually. According to security firm Tenable, one of the two updates, both of which had the back doors, made their way into Fedora, rawhide, fedora 41, the Debian testing unstable and experimental distros, versions 5.5.1 alpha 0.1 to 5.6.1, hyphen 1. Opensuse, both Tumbleweed and OpenSUSE Micro OS and Kali Linux. So it was well on the way to going into wide distribution. Also, additional reporting and research has found that Arch Linux, alpine, linux Gen2, slackware, pc Linux OS and others were also infected. However, this was all recent enough that most of the affected distros were still in their experimental slash unstable releases, so the malicious code did not make it into any widespread production systems. The Mac OS Homebrew Package Manager and OpenWRT router firmware also included backdoored XZUtils versions.
Some other reporting into the deeper backstory and various motivations wrote. All of this started two years ago, in April 2022. The attacker contributed code to XZUtils, gradually built his reputation and trust over time and used various sock puppet accounts to pressure. Xzu welcomed the help, since they were a single developer on a highly popular project. And, as I said, I want to share one more piece of writing about this, a wonderful narrative by oh, it was by Michael Zalewski. He's a Polish security expert, white hat hacker from Poland, a former Google employee. We've quoted him in the past. He's currently the VP of security engineering at Snap Inc.
So here's Michael's take on this most recent, near catastrophic misadventure. In his posting Sunday, which he titled Techies vs Spies the XZ Backdoor Debate, diving into some of the dynamics and the interpretations of the brazen ploy to subvert the LIB-LZMA compression library, michael wrote Well, we just witnessed one of the most daring infosec capers of my career. Here's what we know so far a relatively obscure open source compression library was a dependency of OpenSSH, a security-critical remote admin tool used to manage millions of servers around the world. This dependency existed not because of a deliberate design decision by the developers of OpenSSH, but because of a kludge added by some Linux distributions to integrate the tool with the operating system's newfangled orchestration service, systemd. Equipped with this knowledge about XZ, the aforementioned party probably invented the persona Gitane, a developer with no prior online footprint, who materialized out of the blue in October of 2021 and started making helpful contributions to the library.
Up to that point, xz had been a single maintainer I mean sorry, had a single maintainer, lass Collin, who was dealing with health issues and was falling behind. Shortly after the arrival of XZ, several apparent sock puppet accounts showed up and started pressuring Lass to pass the baton. It seems that he relented at some point in 2023. Since then, xi diligently contributed the maintenance work, culminating in February 2024 with the seamless introduction of a sophisticated, well-concealed backdoor tucked inside one of the build scripts.
Full analysis of the payload is still pending, but it appears to have targeted the pre-authentication crypto functions of OpenSSH. It's probably safe to assume that it added master key functionality to let the attackers access all affected servers at will, and I'll note that's exactly what was indeed being discovered over the weekend while Michael was writing this. The exploit provides for remote code execution with root privileges. Michael continues Some time after getting the backdoor in Xi, along with a new cast of sock puppet accounts, started pinging Linux distro maintainers to have the backdoored library packaged and distributed to end users. Linux distro maintainers no, I'm sorry, to end users. Linux distro maintainers no, I'm sorry to end users.
The scheme worked until Andreas Freund, a PostgresQL developer in the employ of Microsoft, reportedly decided to investigate some unexpected SSH latencies caused by a minor bug in the backdoor code. And this entire exploit if he says this entire exploit timeline is correct, it's not the modus operandi of a hobbyist In today's world. If you have the technical chops and the patience to pull this off, you can easily land a job that would set you for life without risking any prison time. It's true that we also have some brilliant folks with sociopathic tendencies and poor impulse control, but almost by definition, such black hat groups seek instant gratification and don't plan major heists years in advance. In other words, all signs point to this being a professional for-pay operation, and it wouldn't be surprising if it was paid for by a state actor, estate actor. So, as we might imagine, this has really shaken the entire Linux open source community, because everyone understands just how close these malicious modifications came to being incorporated into all mainstream Linux distributions and then filtering out into the world. Were it not for Andrea's friend happening to wonder why SSH latencies were slightly higher than expected in what was referred by someone as a micro benchmarking, you know how much further might this have gone.
Kevin Beaumont posted from his cyberplacesocial Macedon account. He said before everyone high fives each other. This is how the backdoor was found. Somebody happened to look at why CPU usage had increased in SSHD and did all the research and notification work themselves. By this point the backdoor had been there for a month unnoticed.
That if GCHQ aren't introducing backdoors and volns into open source, that I want a tax refund. It wasn't a joke, and it won't be just GCHQ doing it. So all of this begs the question what will be next? And will someone catch that one too, before it gets loose? Lately we appear to be dodging more bullets, which are coming more often. In fact. Our recent episode 962 from February 20th was titled the Internet Dod dodged a bullet when researchers stumbled over a way of bringing DNS to its knees. If I hadn't used that title just six weeks ago, I probably would have used it today, because we did again just dodge another bullet. Amazing, given the inherently precarious nature of security and what we appear, and that we appear to be dodging more bullets recently. I won't be surprised if one comes along that we don't dodge in time. Yeah, I wonder what lessons we'll learn from that. Fortunately, we'll be here.
1:41:01 - Leo Laporte
Stay tuned it's probably, I'm thinking it's not far off in the next year or so.
1:41:07 - Steve Gibson
That's my feeling too, Leo. That's my feeling. There is so much pressure now.
1:41:12 - Leo Laporte
And I think it's probably a nation state that was going to use this in a targeted fashion against specific people that it wanted, not us, probably, but doesn't mean Not you and me, right, but you know dissidents, journalists, you know people the state doesn't like, and so it's not a general threat to all of us, but it is a serious threat and I do hope they this just doesn't seem to much will, or maybe even ability to check these repositories.
1:41:47 - Steve Gibson
Well, and in this case, the fact is, the library did not need updating, right? There is a ton of very old code, Right?
1:41:58 - Leo Laporte
Yeah, sometimes it's fixed, it's done, it's done, it's finished.
1:42:01 - Steve Gibson
It's like RTOS 32, which I bought from that guy when he was going to take it off the market. You know, he hadn't been able to sell licenses because there were no more bugs, Right, and so it was like, oh well, you know, no more revenue from fixes. So I said thank you very much.
1:42:17 - Leo Laporte
The proud owner of his own operating system. Ladies and gentlemen, we're going to call it I don't know GDOS, gibson, dos, something like that. Steve Gibson, not GibDOS, gibdos.
1:42:28 - Steve Gibson
DOS, gib DOS.
1:42:29 - Leo Laporte
Gib. He is at GRCcom, the Gibson Research Corporation. That's where you'll find the latest version of Spinrite 6.1. And it is definitely the world's best mass storage recovery. But now we really have to underscore a maintenance utility as well. Yeah, maybe, if we need all have a copy to run on our SSDs every year or so, go on and get yourself one. You can also get a copy of this program there.
He's got the 64 kilobit audio kind of the standard audio format, but also 16 kilobit if you want a much smaller file size at the loss of some quality. But you know, sometimes people don't care. We also have transcripts. Those are very small but very useful if you like to read along or search, you know, or just use it as an additional piece of information. And of course the show notes are there too, a very nice PDF he crafts every week.
Grccom we have audio and video at our site twittvsn. There's a dedicated YouTube channel for Security. Now you can watch there. It's also a good place to get little clips to share. People seem to like that. Share it on social or share it with friends, share it with coworkers. That YouTube channel is very useful for that.
If you are a regular you probably should subscribe, and that's easy to do. It's free. Just get a podcast client and subscribe to Security Now. You'll get it the minute we're done editing it up later today and every week from now on. Past episodes, of course, are at the website, going all the way back to episode one. Believe it or not, 968 episodes in total. Now I should mention that a lot of this show and everything we do here is supported by club members. Yep, like most podcasts, we have a club. It's, I think, pretty affordable given the number of shows, the amount of content. You get $7 a month. Add free versions of all the shows. You get access to the Discord. You get a lot a month ad-free versions of all the shows. You get access to the Discord. You get a lot of additional benefits and you keep this thing going because it is getting harder and harder to do that. 52% of our audience probably more runs ad blockers, right.
1:44:39 - Steve Gibson
Oh, more than this audience. We're at the high end, probably 90% right.
1:44:43 - Leo Laporte
So you know, ads just aren't working anymore and it's getting harder and harder for us to sell them. We get less and less money for it. It's expensive to do what we do. We want to keep doing it. So please join the club and help us out. We really appreciate it. There are family memberships as well as individual memberships. There are corporate memberships and, of course, if you say you know it's worth more than $7 a month for four episodes of Security, Now to me you can give more, you can increase that amount. We're not a nonprofit so you can't deduct it on your taxes, but you certainly get that warm feeling inside knowing you're helping us out. We appreciate it.
Twittv slash club Twit. We do this show on Tuesdays right after Mac Break Weekly. That's about 1.30 Pacific, 4.30 Eastern, 20.30 UTC. You can watch us do it live. We stream on YouTube. That's open to all YouTubecom slash twit. Of course, club members get to watch in the Discord too. Thank you, Steve. Have a wonderful week, Enjoy the gentlemen and we'll see you next week on Security Now. Thanks, buddy, See you next week.
