Security Now Episode 852 Transcript

Please be advised this transcript is AI-generated and may not be word for word. Time codes refer to the approximate times in the ad-supported version of the show. 

Leo Laporte (00:00:00):
It's time for Security Now we start off the year with a handful of yes. Log4j updates, including another fix from Apache. Why couldn't they get it right, Steve'll talk a little bit about how you can write secure software Alibaba's in the doghouse and an underwhelming announcement from the department of Homeland security, that and a look at Microsoft's bizarre fix for another exchange server flaw. It's all coming up next on Security Now. Podcasts you

... (00:00:32):
Podcasts you love, from people you trust. This is TWiT.

Leo Laporte (00:00:39):
This is Security Now with Steve Gibson episode 852 recorded Tuesday, January 4th, 2022, December 33rd. This episode of Security Now is brought to you by progress. Progress has the technology you need to secure, analyze and integrate your applications, network and processes. Find out more and load of free trial at progress.com/security-now, and by bit warden, get the password manager that offers a robust and cost effective solution that can drastically increase your chances of staying safe. Online. Get started with a free trial of a teams or enterprise plan or get started across all devices as an individual user@bitwarden.com slash TWiT and by ITProTV. Are you looking to break into the world of IT? Get the introduction you need with ITProTV, visit it pro.tv/security. Now for an additional 30% off all consumer subscriptions for the lifetime of your active subscription. When you use a code S and three at checkout, it's time for security. Now the show we cover the latest security news a brand new year, and nothing's changed at security. That's the guy in charge, Steve Gibson, grc.com, doing everything you can to through our security, you know, and I guess it's inch by inch bit by bit. Happy New Year!

Steve Gibson (00:02:10):
It's happy new year to you. 2022 has arrived well for most of us.

Leo Laporte (00:02:17):
What?

Steve Gibson (00:02:17):
Uh there, there is an exception from which this podcast received its title

Leo Laporte (00:02:24):
December 33rd. That's right. That's today.

Steve Gibson (00:02:29):
We're gonna explain, we're gonna explain why it's the 33rd in some locations of the web or the world. This is security now episode 852, as we count down to the end of, well, we don't know what 

Leo Laporte (00:02:49):
Nothing is ended. That's the, that's the problem. Everything continues.

Steve Gibson (00:02:53):
That's it sure does. So we got a bunch of fun things to talk about. We're gonna start off of course, with a handful of continuing Log4j updates we've got, believe it or not another fix the fifth one from Apache, still trying to get this right. We have some false positive alarms have been generated, which caused consternation. It was unwarranted. Alibaba is in the doghouse as a consequence of their actions in this drama. And we have an underwhelming announcement from the US department of Homeland security, keeping us secure but probably not their own network so much. We're gonna note the postponement of a critical industry security conference, an interesting aspirational announcement from Duck Duck Go CEO, which will be interesting to see. We we'll talk about that. And also the soon to be rising costs of cyber insurance, based on all of the feedback that the, the mean bond analysts are producing. We've also got a bit of miscellaneous, a quick update on spin, right? Since it's been two weeks since we've talked and things have happened and then we look at the surprising technological decision that has forced the official creation of December 33rd. <Laugh>

Leo Laporte (00:04:23):
A day that will live in infamy.

Steve Gibson (00:04:24):
And did that not be the last extra day we see in December. We'll see how that goes. So, oh, wow. I think another fun podcast for our list. Awesome.

Leo Laporte (00:04:33):
Looking forward to it. And also we, I should mention some good news that the James web telescope has opened up finally, the fifth and final membrane of their solar shield. And yeah, this is we're wa this is like the suspense story of our time watching this incredibly complex device, millions of miles away, slowly deploy. And if anything goes wrong, that's, you know, a lot of broken hearts. <Laugh>

Steve Gibson (00:05:04):
Well it had a lot of money

Leo Laporte (00:05:07):
10 billion down the tube shot up into. Yeah, yeah. But, but gosh, you know, if it works, we're gonna, it's very exciting. Anyway, that's good news. I we'll continue to report over the days and weeks to come as as those benchmarks. I

Steve Gibson (00:05:19):
Just, I I'm so happy Leo, that we're still doing some science. I agree that we just haven't completely given up on. I agree. Like, you know, things that don't have an immediate economic return, but, you know, may <laugh>, we may be needing some place to go eventually. So yeah. You know, having something out there, looking around saying, Hey, what about that ball over there?

Leo Laporte (00:05:43):
Well and you know, we're both science fiction fans, I think much of our audience fans of science fiction. So it's nice to report on some science fact. That's also pretty darned exciting. So that's that's some good news. I thought I passed that along. That happened about four hours ago, but I just, just found out cuz you know, it takes a while for that. Yes. I was disappointed when we, when we decided not to build that next generation. Cyclotron in, it was gonna be oh, in Texas. Yeah, Texas. Yeah. Yeah, yeah. Of course the Europeans got all the attention by building it instead. Hey, one, a brand new sponsor to the security now podcast one you'll all wanna know about. And probably many of you already do. I know whenever we talk about progress, our, our chat room lights up.

Leo Laporte (00:06:29):
Oh, I, I love that. Why use that? Whether you're a security professional, looking to keep your data safe or an it individual, a person and working in it tasks with keeping end users secure, you should know about progress. Progress. Has you covered progress? P R O G R E S S just like the word has been enabling enterprise experiences for decades and has assembled the technologies that are gonna empower businesses to thrive in this post COVID world. And even in, you know, still COVID <laugh> frankly, most companies don't have the resources to invest in technology. Not the resources the big guys have they need to use technology off the shelf sometimes to create differentiation with a smaller investment. Right? Well, you can achieve it, turn to progress as your trusted provider with progress. Any organization can achieve the level of differentiation that's critical in today's business environment.

Leo Laporte (00:07:25):
I'll give you some examples, move it, the move it managed file transfer from progress provides security control and visibility over all file transfer activities. As you know, one of the real challenge challenges in medicine in the legal profession in many businesses is transferring confidential information through insecure channels over the public internet and elsewhere, and it can open you to liability. And it's just, you know, problematic with move its managed file transfer solution. You can securely send and receive sensitive data with a managed file transfer software, like move it you'll have complete visibility and control. You can you'll get tamper evident logging. So you, you know, you know, you can have some confidence that's made it safely. You'll get centralized access controls to meet operational requirements. That's just one of progress. As many products flow on network intelligence. How about, about that? It's a network detection and response solution NDR that detects threats hidden in network traffic focuses on minimizing your it's attack surface by bridging the gap between perimeter and endpoint security all while employing multiple detection methods to alert on network anomalies and provide early detection of advanced persistent threats with network monitoring and anomaly detection.

Leo Laporte (00:08:42):
You can eliminate the blind spots in your network and stay proactive rather than reactive. I know that's something we're all talking about all the time and really want to do these days progress. It's got the technology you need to secure, analyze and integrate your applications, your network, your processes. Find out more, download a free trial progress.com/security dash. Now don't miss out. Visit progress.com/security dash. Now don't forget that-progress.com/security dash. Now you'll even get at your very own progress swag bag. Yes, progress.com/security. Now the swag bag awaits, but maybe more importantly, the free trial. So you can see what progress could do for your business progress.com/security. Now, now it's time for our picture of the week.

Steve Gibson (00:09:36):
So this was just something fun that I had in the queue of stuff. Since there was nothing particularly more relevant that I ran across. Some, one of our listeners ran shields up on his Tesla. What, and <laugh>

Leo Laporte (00:09:51):
Well, Tesla runs not windows runs Linux. He must have done some trick to do this.

Steve Gibson (00:09:56):
Well, no, cuz I did the browser. Of course. Yeah, yeah, yeah. It's just a browser. Right, right, right. Yeah. So yeah, so the good news is you may not be stealth on the road, but you can be invisible, you know, very stealthy on the internet,

Leo Laporte (00:10:09):
You at all that green completely stealthed.

Steve Gibson (00:10:12):
Yeah. And I don't know whether, like I would be surprised if you got your own IP address as you're driving around. I would bet that you're being proxied through Tesla central. So that IP address, I mean he shows the IP address, right? 31, 1 61, 1 91 0 3. That's a weird address.

Leo Laporte (00:10:32):
That's a weird looking address.

Steve Gibson (00:10:35):
What's you know, hundred, you know, IP address of course. But 31, I don't know what that is. Yeah, yeah. But anyway, so anyway, just sort of a cool fun picture of the week showing yes, no ports are open on your Tesla. 

Leo Laporte (00:10:51):
Anybody can do this cuz the Tesla comes with a browser. You can do this on advice with a browser, right? Yeah. Yeah. That's cool. Yeah.

Steve Gibson (00:10:59):
So, okay. Last Tuesday, the 28th between Christmas and new year's the illustrious <laugh> maybe a little less lustrous than before Apache software foundation released another batch of log for J patches batch because they've got, you know, job of 5, 6, 7, 8 still, or I guess six, seven and eight, I think five finally completely bit the bullet, but addressing yet another arbitrary code execution flaw, which has been discovered in blog for J which can again be used by bad guys to run their malicious code on vulnerable machines, making this the fifth security shortcoming to be discovered in log for J in the span of a month. And you know, I, I, I suggest more than anything this, again, highlights know the fact that we keep finding problems. What happens when really smart hackers, carefully examine code that really smart hackers haven't yet carefully examined. They're gonna find things.

Steve Gibson (00:12:14):
So, you know, writing code that works is entirely different from writing code, which is secure. They're really distinct a and you know, completely separate goals. One is easy. The other is surprisingly difficult and sets the bar far higher. You know, it's possible. And I think one of the reasons is you don't really get credit for writing secure code, right. You know, it's possible to directly see features. And so those get bullet points and of course everyone says, oh, it's super secure, but you can't see a lack of security in the implement of those features. So does it really have to happen until problems arise? So the good news is the CVSs severity score in this fifth case. And like, you know, those more recent ones keeps dropping with these success of, of discoveries. None of these follow on problems that had since been fixed sport that initially eye popping 10.0 <laugh> that we had this latest one got a CVSs of 6.6 and all of the log four J versions, which precede it from 2.0 alpha seven through two point 17.0 for Java eight are vulnerable.

Steve Gibson (00:13:50):
So once again, they all need to get fixed. And you know, this is the problem with these rolling updates more than anything else is that it's easy to say, oh, we found another problem. Everybody should fix it. But as we talked about, thanks to Google's research about the depth of, of the dependency tree, of the, of the Java libraries, which depend upon this, this is a nightmare. So you need to go if you know anybody who's in charge of this, or if you're digging in to see what dependencies are important 2.1 7.0 now needs to be 2.1 7.1. So it was a, you know, a dot release. And although Apache didn't acknowledge the researcher who reported the issue, checkmark security researcher Yev Nery has claimed credit. And I'm sure it's due for reporting the vulnerability to Apache. And this was impressive the day before their release last Monday, the 27th.

Steve Gibson (00:15:02):
So it took them like no time, one day that this, this two point 17.1 was released the next day he has published a detailed blog posting to back up his claim that it was he who provided this. He said, the complexity of this vulnerability is higher than the well, of course, cuz it would be, it would be hard for it to be much lower. He said it's higher than the original CBE, 20 21 44, 2 28. Since it requires the attacker, he wrote to have control over the configuration. Unlike log back in log for J there is a feature to load a remote configuration file. Of course <laugh> why wouldn't there be my Lord or to configure the logger through the code. So an arbitrary code execution could be achieved with a man in the middle attack. For example, user input ending up in a vulnerable configuration variable or modifying the config file.

Steve Gibson (00:16:09):
In other words, yet another feature that log for Jay probably didn't ever need, which created yet another vulnerability that we also wish we didn't have. Now we don't, but it's still a big nightmare. The problem, as I was just saying is, you know, the need to keep all the affected libraries within the entire sprawling, somewhere between seven and nine layer, deep Java dependency tree updated with the latest release. And this has made so much more problematical, problematic, problematic when Apache needs to keep updating their patches as like new issues. The like this is increasingly small, right? So still it needs to get fixed. So it's a mess. The developer and security communities, which have been doing their best to like, you know, stay current are getting a little stretched feeling. They reacted to this latest news, which was first school closed, not even by Apache, but by misery's own public tweet with an explosion of reaction traffic.

Steve Gibson (00:17:21):
One user replied. I hope this is a joke. I hope so much another tweeted. We are long in all caps past the point where the only responsible thing to do is put up a giant flashing neon sign that reads log for Jay cannot be fixed. <Laugh> do not use it for anything. And even Kevin Beaumont, who tweets as goy the dog labeled the instance, another quote failed log for Jay disclosure in motion because no time was given to allow people to patch this before misery, you know, completely disclosed how this could be abused may. That was part of what got Apache off the dime and immediately put, put out two point 17.1 you know, in between Christmas and new year's they weren't waiting. But again, the problem is it's one thing for Apache to say, Hey, we've changed our minds. <Laugh> about two point 17.0.

Steve Gibson (00:18:29):
Kidding. When now you need two point 17.1, getting it like actually out is a huge problem. And you know, during these past few weeks we've really been treated to a ringside seat to observe everything that's wrong with the current software is being built, maintained, secured, and deployed well. And after deployment it's management for the most part, the system works and, you know, in a fairness, what it does is truly amazing. The ability to openly and freely build upon the work of others creates incredible economies, but it is nevertheless a brittle system which breaks down the moment, something as ubiquitous, as log for J being found to be untrustworthy. And in fact, very readily exploitable, thus the 10 pole 0.0 occurs. So we're not done with this yet. So

Leo Laporte (00:19:36):
On time, I would love to spend some time with you talking about what would one do if one wanted to write secure software? What is there a, is there a process or is it just a mindset? You know, do you have to have to think a bad guy to, you know, think about all the ways this could be abused,

Steve Gibson (00:19:55):
You know? Now's as good a time as any you, you and I have often talked about this really interesting phenomenon that anyone coding has experienced. You can't see your own mistakes. Yeah. You, you, you just can't and you know, and it's not ego. It's not that you, I mean, I don't have any ego. My all, everybody who follows me in the, in the news group and who watched me, you know, nail every bug that squirrel had and has Jimmy mean do the same thing for, for spin. Right? I want to find these problems. I'm I'm there's no, no denial anywhere. Yeah.

Leo Laporte (00:20:40):
But you can't see 'em you can look and look and look exactly. This is why in, in the business, the process is usually involves code review by your peers where you, you, you, you say, well, look at my code and I'll look at your code. It's hard to do it in your, in, in, in solitary,

Steve Gibson (00:20:58):
You know? So yes, I, and I would, I would incr I would say I would use a word different than hard. I mean, well not I impossible. Yeah. It's and this has happened to me where I, I look at the, at a, at a function, which I know is broken and I look at it again and I read it, knowing that there's a problem, which increases my focus, cuz like, okay, there's a problem here somewhere. So there's no longer presumption that it's pristine. It's okay. There's something evil hiding. And so that gives you a one like change of attitude, which is valuable. And often that's all it takes, you know? And I've, I've, I've wondered about that phenomenon that when, when the compiler finds a or the assembler, in my case lines, a bug, I'll go back to the code. I just wrote and go, ah, you know, it's like, it's like just being told no, it's, it's like, then I'll look at it thinking, okay, wrong.

Steve Gibson (00:21:59):
And I'll go, oh, of course, look at that. How could I have done that? And then I'll fix it. Well, I, why didn't I see it before I hit the assemble button? I don't know. But being told no, that, that helps me to go. Oh, but so, but despite all of that, even knowing that there's like a problem, I can't see, it will sometimes be single stepping through the algorithm until I hit the thing and I go, oh, it's a mean, it's like you, it has to be just rubbed in your face before you see it. And, and so I, Leo I think that it's this concept of a red team. What I would say is that somebody other than the author or the team, cuz you know, you know, maybe other team members could like crosscheck each other's code, but, but somebody is gonna find the problem.

Steve Gibson (00:22:56):
Who do you want that to be? And, and, and so it's, it's why I really think that this the, when in the case of log for Jay, the, that horrible initial 0.0 drew the, the interested focus of a bunch of other talented security aware coders. And there is, there certainly is something to being to this idea of having like a higher level, like a security trained code. Who's just seen all the things that can be done wrong and, and looks at code from a, from not from the standpoint of assuming it's right. But in like challenging every assumption that the, that, that the original coder made about what his code was gonna do. And, and it is, oh my God, it's far more laborious to, to do that than it is just to make it work. If assuming that all the conditions are what you want them to be, it's, it's a completely different approach.

Steve Gibson (00:24:04):
But I, I, I think the idea of a red team of, of having a a, I mean, and, and really that's what bounties are, right. Bounties are yeah. Paying good guys to find the problems you and your enterprise and your coders cannot find. And no one's gonna do it for free. I mean, it's, it's sort of the, it's the failing of the open source effort, right? I mean, it's the, it's that, that wonderful pyramid of blocks resting on one little toothpick down at the bottom. It's like, you know, some unpaid guy, you some far off land who no one's ever met. And yet we're all dependent upon 'em, you know, and it's thankless. I did I, there, there was a plea from Wikipedia over the holidays. I'm a, I'm in a, I, I do $10 a month because I just, I wanna support them.

Steve Gibson (00:25:01):
Absolutely. And I wish there were some, some way of associating my identity with that. So that cuz I feel guilty. I know I get bugged by <laugh> <laugh> yeah. I don't feel guilty, but I know I'm giving him money, so I just play it. Okay. Yeah, exactly. Okay. And, and, and the guy that does tree style tabs, which I, which I also use every week, I depend upon it on, in Firefox, you know, he was begging for some money. So I gave him some but still most of this open source stuff is just done by hobbyists because, you know, they're well-meaning, but they're often not security trained, you know, open SSL is a disaster until it finally got, you know, deeply looked at, as we talked about years ago and all kinds of problems were found with it. And in fact, you know, as, as we also said Amazon just wrote their own from scratch.

Steve Gibson (00:25:55):
They said, forget that we, you, you can't even understand this spaghetti, you know, what, what this thing has turned into over time. So we're just gonna, you know, TLS, isn't that hard, we're just gonna do one. And you know, they did, which was like, what was it? One, one 20th the size and had all the same functions, you know? But just none of the extra Cret that had, you know, the barnacles that had grown onto it over time. So I, I just, I think that the one, one solution, and, and we've also talked about this is taking C out, out of people's hands. You know, because it's, it is, it's still the, the most popular solution and it is just powerfully dangerous. I mean, it is, it is so, so dangerous. And in fact it's probably the, what caused the naming of this podcast December 33rd as we'll be getting to oh, interesting. Yeah. It's that kind of problem. Yeah.

Leo Laporte (00:26:58):
I think we are, we are moving towards safer languages yep. Which, you know, know kind of nudge the programmer in the right direction, but that's only a certain kind of flaw and then there's higher level stuff that can happen that you just have to kind of, I don't know. I think you have to have a

Steve Gibson (00:27:12):
Mindset. Well, and we have Microsoft, we have Microsoft wondering whether it's necessary to have the deep optimization where they noticed that chromium is all of its prob half of its problems come from the, the, the super secure optimization. Yeah. You know, we've got processors, you have to, you know, pour liquid, he on them pretty hard to keep them, you know, operating. Yeah. You know, we just don't need that level of optimization anymore. So

Leo Laporte (00:27:41):
No. Yeah. Good. It's a good subject. I'd love to talk more about that over over time. Yeah. And making your software it's more secure. Yeah.

Steve Gibson (00:27:50):
Yeah. And, and the problem is budget and time. I mean, I'm living in this world where I don't have anybody telling me that I have to ship anything on a, by a certain date. And if you do every thing changes because nothing ever happens in the, at, at, in the time scale that you expect it to. Right. It just doesn't and you end up having to ship stuff. What is it, Mike? You know, we hear Paul and Mary Jo talk about it. Oh yeah. Windows there's 10,000 bugs and they shipped it. It's like what? <Laugh>, that's just incredible. So okay. So speaking of Microsoft and bugs last week, and exactly, and, and actually this is a time constraint story. Microsoft's defenders newly, and perhaps two hastily have deployed log for J scanner has been triggering false positive alerts, which, you know, hasn't been helping anyone's nerves.

Steve Gibson (00:28:56):
You know, everybody's already on edge about log for J and then suddenly Microsoft defender is saying, log for J log for Jay. You you've got some. And it's like, what? So this was Microsoft defender for endpoint. It was showing sensor tampering alerts linked to their newly deployed Microsoft 365 defender scanner for log for J processes. So, yeah, log for J the alerts are reportedly mainly shown on windows server 2016 systems and Warren quote says possible. Sensor tampering in memory was detected by Microsoft defender for endpoint. And that's created by something called the open handle collector XY process. And according to field reports, admins have been dealing with this issue since at least the previous week in response to these panic inducing, false positives, Tor teller. Who's the, I love this title too. The principle group PM manager at Microsoft enterprise security posture. Now, you know, people have been comparing Microsoft to IBM and boy, if you've got the principal group PM manager at Microsoft enterprise security posture, okay.

Steve Gibson (00:30:30):
Yeah, that's a, that's a company that's been growing Tor explained that although this defender processes behavior is tagged Des malicious, there's nothing to worry about. <Laugh> move along. Since these are false positives. Well thank you Tor, but still it's upsetting everyone. He indicated that Microsoft is currently looking into this Microsoft 365 defender issue and working on a fix, which of course is what the principal group PM managers at Microsoft say, when anything goes wrong and that this company should soon deliver a fix for effective systems Tor. Explain. This is part of the work we did to detect log for J instances on dis the team is analyzing why it triggers the alert. And it shouldn't of course, exactly. One week ago, last Tuesday, Microsoft said that their newly deployed log for J scanner was rolled out with a new consolidated Microsoft 365 defender portal log for J dashboard for threat and vulnerability management.

Steve Gibson (00:31:47):
So yeah, they added a whole big log for J dashboard thing to 365 defender, the new dashboard designed to help customers identify and remediate files, software and devices expose to a tax exploiting log for J vulnerabilities. Now, of course, this happened quickly, right? Microsoft didn't have a lot of time to work on this. And so of course, to place this into perspective, this is not the first time such a thing has happened. So October, 2020 windows admins have had to deal with other defender for endpoint false positives, including one that marked office documents as E mote malware payloads. One that showed network devices infected with cobalt strike when they weren't. And another, that tagged Chrome updates as PHP back doors now in a world where very clever bad guys have forced malware detectors into the use of heuristic algorithms, false positive become a real problem, right?

Steve Gibson (00:33:00):
I mean the only solution is to test all new code extensively to make sure that benign services don't raise false positive alarms, but the need to rapidly push something out into the field to protect full enterprise users in the face of an emergency, such as log for J can make the time required for extensive testing, difficult to justify. So I get it right. There's a trade off, you know, doesn't help for Microsoft to wait six months and then release this, but boy, is it good? <Laugh> so let's just hope that amid the reports of some false alarms, this new log for Jay scanner was also able to correctly detect unsuspected instances of log for Jay residing on disk, as it was designed to. And that many disasters will have been averted as a result. So on balance, a good thing, Microsoft did this and pushed it out.

Steve Gibson (00:34:05):
And yes, in the case of log for J at 10.0 on the CVSs, probably better sooner than later, even if it's at the cost of some false positives, you know, maybe it would've been nice if, if they'd said, you know, don't panic when this goes off, because you know, we, we got, we figured you wanted to have something rather than nothing. So here it is. But you know, it has to guess a little bit and it could guess wrong. Speaking of being maybe wrong, the Chinese government is annoyed with Alibaba. Remember how we reported three weeks ago that the Apache software foundation was first informed of the ultra critical log for J vulnerability by of all entities, Alibaba in China. It turns out that China's not happy <laugh> that they weren't to be, that they weren't told first China's internet regulator, the ministry of industry and information technology.

Steve Gibson (00:35:11):
M I, I T has temporarily suspended a partnership with Alibaba cloud, which is the Alibaba cloud as the cloud computing subsidiary of, you know, the Alibaba group, the e-commerce giant, you know, they're like their China's equivalent of Amazon. This six month suspension was the result of Alibaba's failure to promptly inform the government about the log for J security vulnerability. Huh? Gee, I wonder what China would want with early access to that. A <laugh> the suspension was disclosed by Reuters and the south China a morning post citing a report from the 21st century business Herald, which is a Chinese business news daily Reuters wrote Alibaba cloud did not immediately report vulnerabilities in the popular open source law logging framework, Apache log for J version two to China's telecommunications regulator in response, M I, I T suspended a cooperative partnership with the cloud unit regarding cybersecurity threats and information sharing platforms.

Steve Gibson (00:36:37):
Okay. Now the fact that it wasn't a permanent, like termination was a suspension suggests kind of a slap, right? I mean, like they clearly need this relationship and they're not willing to give it up completely. And as we now, as we know this log for shell, which was the name given to the exploitation of the log for J vulnerability first came to light after she <inaudible> of the Alibaba cloud security team sent an email alerting the Apache software foundation of the critical flaw on November 24th. That's when this first all began Chan added that it in his email quote has a major impact. And just as the patch was being ready for release details of the vulnerability were shared on a Chinese blogging platform by an unidentified actor on December 8th, which at the Apache team scrambling. So it had become suddenly public from November 24th to December 8th, which is why Apache scrambled to release the patch.

Steve Gibson (00:37:53):
Two days later on the 10th, December 10th. Now one might suspect that if officials within China's ministry of industry and information technology are mostly embarrassed, M I I T said in a belated statement published on December 17th. So exactly one week after the Apache release that quote, this vulnerability may cause a de ice to be remotely controlled, which will cause serious hazards such as theft of sensitive information and device service interruption, and the ministry added that it was only made aware of the flaw on December 9th, 15 days after Alibaba's initial disclosure to Apache. So this pushback from M I, I T occurred months after the Chinese government issued stricter vulnerability disclosure regulations, which mandate that software and networking affected with critical flaws alongside entities or individuals engaged in network, product security, vulnerability discovery, report them firsthand to the Chinese government authorities M within two days. I remember we talked about this at time.

Steve Gibson (00:39:15):
This was, you know late summer of last year, and last September, China followed it up by launching the quote, cyberspace security and vulnerability professional databases for the reporting of security vulnerabilities and networks, mobile apps, industrial control systems, smart cars, I OT devices and other internet products that could be targeted by threat actors. So, you know, if they'd made that official and that's where they wanted that report submitted, not too patchy. So feeling duly chastised by the ministry's action, according to a follow up report from the south China morning, post Ali cloud said that it would work towards improving its risk management and compliance. <Laugh> saying that it did not fully comprehend the severity of the flaw Uhhuh, and that it did not share the details with the government in a timely fashion. So, so we're sorry. We won't do that again. <Laugh> right.

Leo Laporte (00:40:22):
And let's take a little break to talk about our sponsor, the fine folks at bit warden, you know, there was a big to do last week about the other big password manager. I see. Think it finally came down to an issue of what they called credential stuffing where people had reused their, their master password 

Steve Gibson (00:40:45):
And apparently some false alarms also there,

Leo Laporte (00:40:47):
And maybe some false

Steve Gibson (00:40:48):
Alarms their system was, was misfiring. So,

Leo Laporte (00:40:51):
Yeah, but that gave everybody a chance to kind of think about, well, what password manager do I want to use and was really glad to jump into those conversations and talk about our sponsor bit warden. I did this on the radio show as well. There are, there are many password managers to choose from the fundamentals. Of course, of any password manager is it has to provide a secure vault for your passwords generate secure one time only password words and then store 'em in the vault. You remember the master password, don't write it down, don't reuse it. We all, we all know that, but there is more, a password manager can do. And, and bit warden is I think, a really good choice. And I'm gonna tell you the reasons I use it and recommend it. First of all, it's the only open source cross platform password manager that can be used at home by businesses on the go it's trusted by millions open source.

Leo Laporte (00:41:45):
To me is a really good thing. And one of the side effects of open source that you might not have thought of the free version of bit warden is always gonna be free. Their business model does not rely on pushing free users into paid versions of it. Now, I actually paid for bit warden. It was only I think, a buck a month, less than that 10 bucks a year because I not cuz I needed the features because I wanted to support 'em cuz they're an open source project, but then I found out bit warden does so much more. For instance, bit warden for business is a really great solution for any business that wants to empower its employees to follow best practices and password management, both at home and at the office because the bit warden business plan begins with your employees creating their own personal vault.

Leo Laporte (00:42:34):
So it really encourages them to use bit warden at home and then you add them to the organizational vault. So that with one instance of bit warden, they get completely securely, completely independently, their own passwords and the business passwords. You want them to keep secure. So create that personal vault it's free to do so by the way, and then create the organizational vault. And you'll like this organizational vault because you have a huge number of enterprise policies you can adapt to fit your business needs. It includes bit warden, sand, which is a fully encrypted method to transmit sensitive information, text files, whatever directly to anyone. They do not have to have a bit warden account to use that to send it, but to not to receive it. Your team members can generate you very important, unique, secure passwords, every single site and bit warden has a great security monitor system that will let you know if you've repeated passwords and give you the opportunity to fix all of that.

Leo Laporte (00:43:33):
So you don't fall prey to that cred. So stuffing problem, enterprise grade security, it's GDPR compliant, HIPAA compliant, California's privacy lodge, CCPA compliant, SOC two compliant. And of course, an end to end encrypted vault that in you could have them host, which I do cuz I figure they know more about a secure hosting than I do. But if you wish you can host yourself not every kind of account is available. You have to bid war and self hosting is not available in the free organization or team plans, but the individual plan and the, and the enterprise plan yes. Self hosting. So if you don't wanna have that centralized vault, if you want absolute control, you can do that. I love that. Now the, the other thing I like about bit warden problem, you know, it's because they're open source, but also because they're really good, they're always adding new features.

Leo Laporte (00:44:24):
They've been asked, they're gonna be rolling out two new features for the enterprise plans. This is something that we do for our employees. And I think it's a really great thing. If you already are part of a bit warden cloud-based enterprise plan, all your users will be granted free bit warden, family accounts. Those are special premium accounts for up to six family members. And that's just part of that. And I think that's a really that's how bit warden is, is effect saying, you know, we want to extend password security to all, not just the members of your organization, but their families as well. They've also added log with single sign on which I really I'm a big fan of SSO. It's a new option for customer managed encryption. It enables a self hosted enterprise. So you, you have to choose the self hosting option then at create a local key server.

Leo Laporte (00:45:13):
And you can use that to manage vault encryption keys. So employees can access and decrypt bit warden through their SSO credentials that eliminates the whole credential stuffing problem, no additional passwords or login information. It's easier. It's more convenient for employees they're, they're using SSO and their device, which in fact, you can enforce this with a policy uses biometrics. So they have that second factor automatically built in. This is just the way a password manager should be created. If you're interested in a business plan, bit warden has many plans. One that'll work for you for sure. The team's organization option $3 per month per user. You can share private data securely with coworker departments, entire organizations. It's very granular bigger enterprises you'll want be warden's enterprise organization plan. That's just $5 a month per user. And again, don't forget, you're gonna get that free family plan for all of your employees, which is really great, the free organization plan.

Leo Laporte (00:46:10):
By the way, you should look at that at least to, to try it out. That's for two users and two users can store and share secure passwords. And all of that bit warden believes everyone should have access to basic password security tools. That's why their basic free account is free forever. And that means an unlimited number of passwords. It works cross platform. You can upgrade it anytime I did this just to support them. Cause I love them for less than a dollar a month. You get a premium account with some additional features. And of course, if you're looking for secure family storage, they've got the family organization option six users premium features three bucks, 33 cents a month, total not per user. So bit warden is very affordable. Again, open source, which gives me some real sense of security. And you never have to worry about them forcing you into a higher paid plan.

Leo Laporte (00:47:01):
And with the bit warden cloud, you can get started in no time at all monitor and manage security vulnerabilities with their bit warden vault health reports. I mentioned those which will identify exposed weak reused, potentially compromised passwords and we'll. And I love this. It'll say any item in your vault that doesn't have two factor turned on. It'll say turn on two factor, get started with the free teams or enterprise trial of a teams or enterprise plan bit warden.com/twi. This is really my choice. It's what I rely on for a password manager. And I have with great confidence can recommend it to everybody. Try it free across devices as an individual user, look at the enterprise plan, the, the family plan, it's all available@bitwarden.com slash TWI. These are great guys. I'm glad we could support them on the show bit warden.com/twi. Okay, Steve. Now it's your turn again?

Steve Gibson (00:47:58):
So <laugh> the hack, the DHS bug bounty was expanded to explicitly include log for J flaw based attacks. And I was like what the USS Homeland security secretary Alejandro Myorca recently announced that the DHS would broaden its new bug bounty program to, to explicitly solicit the discovery of vulnerabilities in its networks resulting from the exploitation of log for J he did this in a tweet. He tweeted in response to the recently discovered log for J vulnerabilities at DHS gov is the scope of our new hack DHS bug bounty program and including additional incentives to find and patch log for J related vulnerabilities in our systems in partnership with vetted hackers, cuz you know, you just can't let anybody poke at the government. <Laugh> the federal government will continue to secure nationwide systems and increased, shared cyber resilience. Okay. So as we know the C I S a CISA demanded that no one go home for Christmas until all federal agencies had addressed their own log for Jay vulnerabilities. Unfortunately, as we've seen doing that in short order, just because some federal bureaucrat demands, it doesn't make it so old and vulnerable versions of log for Jay, you know, used for logging are far too buried, deeply underneath existing systems to be immediately patched overnight, just because Santa is on the way. So let's hope <laugh> there's can Santa write code

Leo Laporte (00:50:09):
<Laugh> ah, good at patching things,

Steve Gibson (00:50:12):
Honey, he's got a lot of elves, so have their coders. Yeah. Let's hope it was a white and not a red Christmas. Oh, so far officials at the cyber branch of the DHS have said they've seen no signs of malicious actors using the vulnerability to breach the systems of federal departments and agencies, but they've warned that a tax utilizing the flaw might still occur. Yeah, no kidding. If a tax really haven't occurred, it's only because there are currently too many, other far more lucrative and juicy pieces of lower hanging fruit to be plucked using the log for Jay hedge trimmers. I mean, that's just the only explanation. However, I have to say that I'm unimpressed by what DHS is willing to pay. My Orca said security researchers participating in the bug bounty program would be paid anywhere from $500 to 5,000, depending upon the gravity of the vulnerability $500, really a cheap government.

Steve Gibson (00:51:28):
<Laugh> I've well also remember you're competing with people who will pay a lot more, exactly $500. Isn't even worth the hassle of reporting the problem, you know, that's ludicrous, you know, you know, when the world is chalk full of cash, rich enterprises just wait to be ransom. What underworld low life is gonna waste his or her time poking around the department of Homeland security for a possible $500 payday, which I assume unlike a big ransom payment is taxable. So I did a, I did a bit more are digging into this and I suspect that the problem is that Myorca or his underlings, cuz I'm sure, you know, he just tweeted something that someone wrote for him really don't have any idea what log for J is, as we know, because we covered it at the time back in 2016, the us department of defense introduced the hack, the Pentagon program.

Steve Gibson (00:52:32):
And after that, the program discovered nearly 140 previously unidentified vulnerabilities on some of the department's websites. Yeah. Right? Like, oh, you forgot the S on HTTP. Can I have 500 bucks please? <Laugh> really not very hard to solve problems. The program was declared a success and was quickly duplicated by other branches of the us military. The problem of course is log for Jay is not some website vulnerability. It's an entirely different scale and class, you know, for which my Orca says yeah, 500 bucks. You know, and besides didn't the CISA declare that Christmas would be postponed until all federal agencies had the log for Jay bug expunged from their networks. Since Christmas arrived on schedule this year, it must be that log for Jay is already a non-issue within the entire federal government. So problem solved, right? Yes. Your us taxpayer dollars hardly at least they're frugal with them in that case.

Steve Gibson (00:53:52):
<Laugh> that's I guess that's true. <Laugh> oh boy. You know, it's just, you know, the bad guys will get to the government in time, but if they're being so cheap certainly no reason to, you know, you know, tell them about the problems you find. COVID has postponed the RSA conference organizers of the annual RSA conference, one of the largest cybersecurity events of the year announced before the end of last year that they're moving the, the annually scheduled February gathering to June over health concerns in an email, two attendees org and organizers the, or, well, two attendees, the organizers said, I guess the attendees were the first to be notified. Everybody else will be told too, that a, that, you know, the, a recent uptake uptick in COVID 19 cases, of course, that everyone's aware of made it difficult to hold an in-person event, which I guess they're still holding onto in near term, the RSA conference was originally slated to be held the week of February 7th and past events have attracted tens of thousands of cyber security professionals to San Francisco's Mosconi center and the surrounding area.

Steve Gibson (00:55:15):
And of course, as our listeners know it was during one of those fateful conferences that I encountered Yuba CO's St Sfard whom no one heard from or heard of at the time, she was sort of for Lauren Lee standing at the top of the Mosconi center, escalators looking around for someone who would talk to her and as a result of their brilliant concept and the listeners of this podcast we were able to play a pivotal role in Yu CO's early start. So conferences still do pay off. Maybe now they're more cyber than they were physical in any event for this year, Linda Gray Martin VP of the RSA conference wrote, we are extremely disappointed to share that we're not able to gather in person this February, but we firmly believe that with the surge in COVID 19 cases around the world, this is the responsible step to take, to ensure our community stays healthy and can focus on protecting our critical systems and business businesses against ever present cyber threats.

Steve Gibson (00:56:23):
And of course at the same time, real world viral threats. So their plan is, and I hope it works out for an in-person conference to be held from June 6th through the ninth. The organizer said they will contact all registered these speakers event sponsors and exhibitors and partners regarding the postponement. And, you know, just as a heads up for anyone who would be planning to attend they're going to hold to some health measures. You will be required all attendees and exhibitors and everybody, you know, physically within the confines will be required to show a proof of vaccine for COVID 19 and the mandatory use of face masks for all indoor activities. So I would not have gone if it's in any way possible to, to see this stuff online at this point, things are a little bit crazy in COVID land.

Steve Gibson (00:57:28):
So anyway, if that affects anybody, if by any chance you missed that email it's not gonna be February. It's gonna be June. Duck, duck go continues to grow. Of course we've talked about duck dot previously and I, I, I don't know, Leo, I still have a difficult time with the name <laugh> you never played the game. That's why I never played the game and I'm not sure that I want to duck it. <Laugh> I know I'm still Googling it. Although I did it did induce me to go take a look at that their page and it sure is cleaner. I mean, once upon a time, it was, it was Google. That was such a clean page. And of course now it's a lot more overtly, a sponsored than it was very sad. Yeah. And it does bug me that when I click a link, it redirects through them.

Steve Gibson (00:58:21):
It's like, why is it any of their business that I click that link? But it's also clear that duck duck go is picking up the pace of its continued growth. I've got a chart in the show notes showing their traffic from 2010 to, to, to now. And boy, I mean, they are going, unfortunately it's, it's only duck dot go. That's gone exponential COVID as we know at moment is surging. But it is really a beautiful growth curve. Now that 2021 has wrapped up. We know how that year went for them for duck dot go it's clearly privacy search engine was used for more than 34.8 billion queries in 2021, making that an increase of more than 47% from 2020, which itself was an increase of similar size or percentage from 2019. So it's going crazy. And given the fact that Google and Bing are known to be rapaciously siphoning and tracking and doing everything in their power to extract every possible bit of revenue from their users, UC growth suggests that many people are becoming increasingly comfortable with using it as an alternative.

Steve Gibson (00:59:49):
Of course, as we know, they initially made a name for themselves being a search engine that had no interest in tracking. And then they further expanded other products, apps, and extensions aimed at enforcing or enabling a more private online browsing experience. And now duck do go is planning to expand its offerings, to include a browser for desktop users. And that's why I've like was brought up short. It's like, wait, what? At the end of their 2021 review blog, post duck, duck go CEO, Gabriel Weinberg talked a bit about their forthcoming desktop web browser. He said like, we've done on mobile duck, duck go for desktop. We'll redefine user expectations of everyday online privacy, no complicated settings, no misleading warnings, no levels of privacy protection, just robust privacy protection that works by default across search browsing email. And, or he says, it's not a privacy browser.

Steve Gibson (01:01:06):
It's an everyday browsing app that respects your privacy because there's never a bad time to stop companies from spying on your search and browsing history. He said instead, and this is what I found really interesting instead of forking in chromium or anything else, we're building our desktop app around the OS provided rendering engines as we have on mobile, allowing us to strip away a lot of the unnecessary Cret and clutter that's accumulated over the years in major browser with our clean and simple interface combined with the beloved fire button from our mobile app, duck, duck go for desktop will be ready to become your new everyday browsing app compared to Chrome duck duck go app for desktop is cleaner way more private, early tests have found it significantly faster too. Wow. So that's interesting. I'm sure we'll all be very curious to see how that develops and I will be keeping my, I peeled for some additional information, you know, and I'm curious, I've often talked about how difficult it is to build and maintain one's own browser.

Steve Gibson (01:02:29):
Now these days it's like a little OS, right? I mean, you have to really, really want to, as Google clearly done, Microsoft wanted to as well, but a web browser wasn't their central mission. So they wisely abandoned the sinking ship of IE and edge classic in favor of aro fork and, you know, created the new edge, which I guess Paul and Mary Jo, is it still CR over the and windows weekly? Oh yeah, I guess <laugh> oh yeah. So anyway, Gabriel referred to forking chromium. So, you know, he, so he talked about that in his note saying, no, we're not doing that. So that's, you know, more obvious choice is which was the path taken by all non Firefox browser, you know, Firefox and the, to browser it was on duck, duck go's radar, but they appeared to believe they can build somehow a lightweight web browser around the hosting OSS, native HTML rendering engine.

Steve Gibson (01:03:35):
Like I said, good luck. You know, I love the idea in theory of what he describes as doing away with a lot of the unnecessary Cret and clutter that's accumulated over the years of major browsers. But I don't know you know, there are many services that contemporary users want and now expect, and in some cases even need from their browsers that make them convenient to use. For example, what about you were just talking about bit warden. Oh yeah. What, you know, that's an add-on, what about password managers? Will duck duck goes browser support the industry's emerging standard browser plugin add-on facility, if not, can a feature stripped desktop browser succeed in today's world. I couldn't use one. So it'll be interesting to see, you know, what they have in mind, how they think they're gonna be able to. I mean, frankly, I would've started with chromium.

Steve Gibson (01:04:38):
I would've done another fork of chromium and yes, then stripped it of, of privacy, intrusive stuff, you know, wrapped her a, a shell around it that prevented it from doing all of that, but still, I just don't know. Well, we'll see how, how they can deliver on what they want to. We'll be interesting. The cost of cyber insurance will likely be rising or maybe insurance will even be terminated given the past several years. And the number of times we've talked about state and local government agencies, including school districts, relying upon their cyber attack insurance to cover the cost of ransoms and after attack recovery, it shouldn't surprise anyone that wall street investors and the insurance markets are worried about the cyber security risks that and local governments face. This is changing the economics of public sector services and not for the better offices that previously provided a limited set of local services, which never really gave much more than a passing thought to the need for cybersecurity are now for the first time, finding themselves targeted by foreign criminal and nation state actors.

Steve Gibson (01:06:05):
And in reaction to all the payouts insurers have had to cough up the rates on cybersecurity insurance, which has been the main line of defense for many state and local governments has started to rise. And in some cases to even become available, at least not at a price that you know, our government agencies have been able to afford, or, you know, just had the budget for. We know from all of our previous reporting that the bad guys are explicitly targeting entities, which are believed to be well because, you know, as with Willie Sutton, that's where the real money is as far as protecting themselves goes for our local budget, strapped, municipal and state agencies, as always, it's a matter of funding. Omid Ramani, who's the associate director for us public, for finance at the credit rating agency fit, which is a biggie told the record in an interview, the landscape is changing quite rapidly now from the cybersecurity insurance and the threat landscape side, which leaves local governments in the middle, dealing with used.

Steve Gibson (01:07:19):
They traditionally haven't had to deal with. And literally no one believes that our agencies are up to the challenge. They now have no choice, but to tackle last month, Hilltop securities surveyed 150 municipal bond credit analyst and specialists excluding those who were at rating agencies, I've placed the bar chart. <Laugh> showing the sad results of one of the survey questions which they were asked the 13th, multiple choice survey question asked was what is your opinion of how prepared state and local governments and other municipal market participants currently are for cyber attacks. So there was, this was multiple choice, very prepared, prepared on their way to being prepared, somewhat prepared, hardly prepared or other. Okay. By far the dominant bar at 63% of the total was hard, hardly prepared. The runner up, which made up most of the remaining balance at 30% was somewhat prepared.

Steve Gibson (01:08:43):
And the remaining 6% selected from the multiple, from the multiple choices was the optimistic on their way to being prepared. But no one, not one of the 150 analysts surveyed chose either they're prepared or very prepared. And, and many of the surveyed analysts cited cybersecurity as a major factor in the current municipal bond market. So what does that translate into higher borrowing cost for all of those who are, who finance themselves by issuing municipal bonds? And of course, it's we taxpayers who fund these bond measures back in April of 20, 20, only 12% of those responding to a similar survey, cited cyber secure among the top five issues affecting the municipal bond market at the time in this updated and just published survey, that number has risen from 12% to 29%.

Steve Gibson (01:09:51):
And of course, this is why none of this bodes well for the futures availability and or the cost to these agencies for the purchase of sufficiently comprehensive cyber attack insurance. We keep seeing that money isn't spent where there isn't a screaming need for it to be spent. And this appears to be especially true in it. You know, which starched shirt C-suite executives have never really understood nor really wanted to need to understand, you know, their feeling was please just let all that confusing mumbo jumbo be somebody else's problem. Eh, it's not until recently random municipal agencies were being attacked that just there wasn't, you know, no reason to attack them. So not only was their true cyber preparedness allowed to go lacking, but their insurance costs were low. However, that's not what the future promises at least insurance is something that bureaucrats understand what this mean means for those of us in the us, at least is either a decrease in the quality or quantity of provided services due to the, you know, need to budget more on a fixed on a fixed budget for cyber insurance, or probably an increase in local taxes.

Steve Gibson (01:11:17):
Since, as I said, somebody needs to pay for this and taxes is where the money comes from. So yeah there's a, you know, a lag in this happening, as we know, the ransom attack have been increasingly prevalent. The last few years, it's getting pushed down the chain and ultimately it will end up raising our taxes. So Leo, two weeks ago, I mentioned that I was excited because the day after that podcast the matrix resurrection, oh God, oh, <laugh>

Leo Laporte (01:12:01):
Hope lives eternal in the matrix fan, doesn't it? Oh,

Steve Gibson (01:12:05):
It does. 23 years ago. The matrix first appeared. Amazing. And I admit to having, oh my God, I admit to it. I mean, that was amazing. Oh yeah. And oh yeah. You know, I'm sure that after spending these past 17 years together, our audience knows that for me, hope does indeed spring eternal. But in this case, those hopes were unrealized and the movie was shown to be a barren money play offering, not a single new idea immediately after suffering through it. I tweeted the matrix resurrections. I am unsurprising that I am DB's viewer rating dropped it from 6.8 to 6.1 after its wide release, I was unimpressed by the seemingly endless and boring Kung Fu the whole rationale was quite a reach. If you feel that you need to see it don't expect much. And for today I just checked the internet movie database. Now PS it at 5.7, which is not surprising. So I think that probably killed

Leo Laporte (01:13:29):
It. It was the matrix was such a, a breakthrough at its time. And if you were lucky enough and I was to have seen it without any prior knowledge, just, I just went to see it. And it was like, I walked outta that theater mind blown. Yes. but now it's so well known. I it'd be hard to have that kind of impact with a matrix movie again, I think.

Steve Gibson (01:13:51):
Yeah, I do agree. I'll I will say however, Leo, over the holidays, Lori and I did encounter one very pleasant surprise Netflixes don't look up. Oh, isn't that fun? Oh, it was yeah, I enjoyed it. Hey. Yeah. I I'm glad I, I meant to, to shoot you and Lisa a note it's a wonderfully exaggerated yet. Disturbingly apropo brilliant commentary on the many facets of creeping denialism that we appear to be seeing around us everywhere these days. It was, you know, I, I, I thought it was brilliant. And so I do recommend it to anybody who has access. I'm glad you guys liked it too. Oh yeah. Yeah. on the spin right front, I've become very comfortable with the project's use of GitLab. And I am very glad I took the time to bring it online. The spin right development community is using it to, to man edge the collection of known problems that I'm whittling away at every day.

Steve Gibson (01:14:55):
And, you know, I was using that GRCs news groups and, you know, like, like I, I think I have something in in one note, well, I, I know that I have some notes in one note that are now living over publicly in GitLab. So I was sort of privately managing the, the, the stuff I needed to, to deal with and get to. And I was often marking messages as unread in the news groups as my own means of managing, but, oh, this is just such a cool way to operate. And the good news is it continues to be true that the new foundation I've written for spin, right, is working robustly for just about everyone. At the same time, I strongly believe it's worthwhile to take the time right now to track down the causes of any misbehavior that anyone finds for one thing I need to know that it's not my fault, you know, no matter how rare it, it might be.

Steve Gibson (01:15:59):
You know, sometimes what we're finding is the trouble is a bug in a chip set that is in no way spin rights fault, but I don't know that until I find it. We found an, an example of that on an old rock motherboard spin, right, is now using many of the features that every chip set was advertising back then, but had apparently not yet perfected. So in those cases, there's nothing I can do. In this case, it only occurred when the chip set was in a H C I mode. So switching other board back to its legacy ATA mode, which by the way, was the bio's default mode for that motherboard. Anyway, it resolved the problem. And even when the motherboard was in a H C I mode, the trouble only appeared to occur. The second or third time spin right.

Steve Gibson (01:16:52):
Was run after boot. Oh, I hate that kind of bug. It was weird. Oh God. You know, so, so, you know, it probably wouldn't even be encountered and it requires non default settings to happen. And if it were to happen, there's a workaround, which is, you know, not running spin right. More than once, or just switching back to ATA mode, which was the, the, the default and Leo, this thing, when I found it, I, I was like, what it's like, I was, I was setting a bit in very, very high memory in an a H C I register bit that's in, like, it's a, the, all the, all of the IO stuff lives at the top of the four gig memory area that, that that's where all the PCI stuff is. And so, you know, so all the dresses start off as F F F you know, E and then something or other.

Steve Gibson (01:17:54):
So there was a point where I was setting a, a bit turn oring a bit on the least significant bit of a word up there and there, and that caused a, a word in low memory to spontaneously switch to 0 1, 0 1 hex. It's like, what? I mean, it's, it's, it's, I mean, I don't even like, you know, a bug right in the hardware. If the hardware, if you don't have hardware to run on you, Don have a computer really you've, you've got, you know, a, a fancy, random number generator. That's not very good, but still anyway, the point is there are those sorts of problems. Yeah. But an example of a different kind over, just this past weekend, I tracked down some really odd behavior that was only were seen on a particular, super micro server motherboard with a Zion processor. It turns out that some Intel processors stop running their timestamp counter, which is supposed to be counting up the, the processor clock cycles.

Steve Gibson (01:19:08):
They stop it while the process is halted. Okay. So servicing a hardware interrupt from a halted state can theoretically reduce a systems interrupt response time since there's no need to wait for a current instruction to be completed, cuz there's no current instruction being executed, the system's halted <affirmative>. So at one point in my code, I was establishing a time-based reference. I wanted to measure the duration between two hardware interrupts the, the slow timer ticking. So I was halting the processor before each of the two successive clock interrupts in order to obtain the jitter free reading of that interval. Now remember, no one else had any trouble. We have hundreds, truly hundreds of people testing the code that I'm writing. And, and often there, each of them are testing on multiple systems. Yet this was only seen on one, on that one system. I acquired one of them because that's the only way to solve this kind of problem.

Steve Gibson (01:20:20):
Okay. Today I'm no longer halting the processor during that time establishment, the problems gone. It wasn't really necessary to do that since that is, as of what I was originally doing since today's processors run so fast relative to the 55 millisecond interval I was measuring, but you know, I'm a perfectionist. So now spin right. Runs on that system perfectly and on untold others that would eventually have encountered spin right in the future. And would've had the same problem. Now they won't. And as I was saying earlier, you know, I realize this is a different approach to developing software, but everyone here Dans that I'm not just creating software to get it shipped, I'm creating it to get it right. And you know, that's how I have my jollies on you know, the holiday weekend <laugh> and Leo let's give our listeners the jolly experience of finding out about another sponsor. And then we're gonna talk about the origin of what will become known as December 33rd. <Laugh>,

Leo Laporte (01:21:37):
You know, I'm doing some math and I think this is January 4th. So it actually, this would be December 32nd, third, fourth, fifth, December 35th. But I guess time

Steve Gibson (01:21:48):
Opt, we may encounter that the the month did need to get extended. <Laugh> let's put

Leo Laporte (01:21:55):
It that way was, was, yes. That's the point? Our show today brought to you by, for your brand new year, a brand new career. How about at it? Pro T V I saw a record number. People quit their jobs in November. People just fed up with low pay, uninteresting, you know, jobs menial jobs. Wouldn't you like to work in it? If you listen to this show, you clearly have the aptitude, the interest I would submit more than enough aptitude to, to get a great job in it. Maybe you just need to get those cert, it's get a little training and that's what it pro TV is all about. Now, you know, when you think about, well, I'd like to get a career in it, it might be a little overwhelming, cuz there's lots of different areas. It's kind of hard to figure out where to start.

Leo Laporte (01:22:45):
Well, I can tell you where to start right now. It pro TV, they've got the knowledge you need. The certificates you'll need to break into the it world, make yourself desirable to future employers. And if you're already an it professional to get re-certified or to upgrade your skills or to learn a new skill. And it's just fun, it's entertaining. It's engaging. It's kinda like TWI. In fact, Tim and Don fully admit, they were freely admit they were inspired by what we do here at TWI to create essentially a, a full time TV station, all about it training. That was their expertise and they've man they've, they've really made it a reality every month. Now for the last six months they've been doing themes, which I love this month, January. It's all about getting started in it. New year, new job, new career January 15th and 16th will be their free weekend.

Leo Laporte (01:23:39):
If you wanna just dip your toes into it, kind of get a sense of what it would be like to study for these certs. They're gonna have a bunch of free courses. Compt is ITF plus and a plus courses. Those are, those are the certs really people often get when they first start an it, especially that a plus cert for desktop support. And I love it cuz they're the official training partner of compt. So they have the best compt training also free on the 15th and 16th line essentials. I know a lot of you are interested in Linux as an operating system. There'd be a great place to start. Microsoft 365 fundamentals is part of the new Microsoft certification process. Ms 900. I love this a hands on PC build from the bench part of their, from the bench series. I love that Cisco CCT routing and switching 100, 4 90 apples certified support professional AC S P for Mac OS 11, the current Macs.

Leo Laporte (01:24:34):
Those are all free. January 15th and 16th. Now you might say, oh, is this gonna be boring? A no. Yeah. I think a lot of it training could be boring, not it pro TV. They're enthusiastic stick. Very smart, very experienced. Ed entertainers are there to give you the information, but to do it in a, in a fun and engaging way. The, I love, they chunk everything up into 20 to 30 minute segments. They're video by the way, which is more engaging. You can watch, you can see you can they do it have Monday through Friday out of their studios. I think they have seven studios now in Gainesville, Florida, and they're always producing live from 8:00 AM to 6:00 PM Eastern. So you can watch live. In fact, they have a chat room just like we do and you can participate live. And it only takes 'em 24 hours to go from the live production into the on demand live or a big library, 15 800 hours of on demand programming, all of it in 20 to 30 minute increments.

Leo Laporte (01:25:33):
Perfect for, you know, just watching during a lunch hour, when you have a break, the content is always up to date. That's one of the reasons they have seven studios because stuff changes. The test change software changes, there's new software, there's new certifications might Microsoft just changed everything. And the nice thing is it pro TV is always up to date. They always keep it up to date and you will be prepared for your exams because you can get virtual labs that let you, you know, try these skills, get some hands on. Even if you don't have the windows server machines or that kind of stuff. They have practice tests to fantastic monthly webinars, January 13th, they'll be discussing cloud. This is, this is a great title. Cloud computing, confidential secrets to leveraging the cloud in your organization. I think a lot of people would like, like to know about that.

Leo Laporte (01:26:23):
If you're looking to break into the world of it, bottom line here, get the introduction you need with it. Pro TV, it's affordable, it's engaging, it's fun. But most importantly, it's really great training that will get you on your way. Go to it. Pro.Tv/Security. Now, if you use the offer, SN three zero, you'll get 30% off all consumer subscriptions. As long as you stay active for the lifetime of your active subscription, SN 30 at it pro do TV slash security. Now 30% off for the lifetime of your active subscription. That's an amazing deal. It pro TV. We love these guys and they're doing such great work. And I know so many people who are working in it now, thanks to them. You could be one of 'em it pro TV build or expand your it career and enjoy the journey it pro.tv/security now, and don't forget SN 30 for 30% off. Okay. Let's talk about December 33rd.

Steve Gibson (01:27:28):
Okay. So prepare for some fun uhoh We all know that 2021 was a rough year for Microsoft's exchange server. Yeah. And unfortunately, 2022 hasn't so far started out much better or as ours Technica headline read quote, Microsoft fixes hair-brained Y two K 22 exchange bug that disrupted email worldwide, a rookie programming air were crashed servers because they couldn't process the year 2022. Oh God.

Steve Gibson (01:28:13):
But Leo, the details are just wonderful which is why this made it into the show's title piece. Okay. So a global mass disruption in exchange server 2016 and 2019 email delivery was a result of a date check failure, which made it impossible for Microsoft's servers to accommodate the year 2022. And this immediately prompted the industry to label this bug Y two K 22 <laugh>. Okay. So get a load of this. This is what was happening in lurking inside Microsoft servers, exchange stores, dates and times. And this is actually for the, this is for the security update timestamp on exchange security packages, stores, dates, and times as 32 bit as a 32, 2 bit signed integer, the largest binary value that a 32 bit unsigned integer that is just like 32 bits can hold. We all know that's, you know, two to the 32 minus one because we start counting at zero, right?

Steve Gibson (01:29:42):
So it's two to the 32 total values. The, and since we start, since we, since zero counts, as one of those that the highest value is 32 to 32 minus one. And we all know what that number is, right? That's that 4.3 billion be because that's 32 bits. The same as the address on an I P V four packet. So it's like 4.3 billion. I P V four internet addresses, right? And so on and anything 32 bits, 4.3 billion, but signed into that is where that, that 32 bits needs to express a, a plus or minus a, you know, a set of negative values which are represented in a format typically known as twos, complement arithmetic. They use the most significant bit as their sign bit. And it, it turns out it's all kinds of cool things happen when you do that. For example, if you have a, a value of all zeros and you increment it in a binary fashion, you know, it goes 0 0 0 0 1 0 0 0 1 0 0 0 0 1 1 0 0 1 0 0 and so forth upwards.

Steve Gibson (01:31:10):
Well, if you decrement it in that same binary sense, it goes from all zeros. You, if you subtract one from that, it's all ones. So in other words, minus one is all ones. And notice that the first bit the Mo the, the me the most significant bit, the left most bit is now a one. So as you continue to subtract ones from that, the least significant bits have that zero pattern dances around while everything else stays all ones until you get to the most negative value that 32 that assigned 32 bit value can represent, which is a, is a one followed by all zeros. Okay? So that's the way values are, are represented in our compilers. When we say we have a signed 32 bit value, okay. So if the largest value that unsigned 32 bits can represent is 4.3 billion, the largest positive value that a signed 32 bit quantity can represent is two to the 31, 1, right?

Steve Gibson (01:32:42):
Not two to the 32 minus one, cuz we lose that high bit. That's now the sign bit. So we can, the largest positive value is two to the 31 minus one. And that's gonna be half of 4.3 billion. Well it turns out it's exactly 21, 4 74, 8 36 47. In other words, 2 1 4 7 4 8 3 6 4 7. Notice that the first two digits are 21 as in 2021. It turns out that Microsoft uses the first two digits of the decimal updates version, which is a signed 32 bit value to denote the year the update was released. And that that version value is stored. As I said, as a 32 bit signed value. Well, that approach has a limitation. <Laugh> that Microsoft just encountered since the largest value, you can start a 32 bit signed integer width is 21. You cannot start a 32 bit signed integer with 22 followed by eight digits. It won't fit <laugh> consequently, when Microsoft released as they did version 22 0 1 0 1 0 0 0 1 on new year's Eve everyone's on premises exchange servers immediately crashed. Yeah, because they were unable to figure out what the heck was going on. The crashes meant that exchange was unable to process messages, which became stuck in transport cues and admins around the world on new year's day were frantically trying to troubleshoot the problem. Hopefully not with a hangover from over partying the night before. Why? Why

Leo Laporte (01:35:14):
Are they being so clever? Is this even this

Steve Gibson (01:35:17):
Just dumb? I is so dumb. Leo it's like, like what? There's so, oh, let's be clever Uhhuh. You're right. It is. It is. Okay. So the troubleshooting was not aided by the cryptic log messages. That exchange was emitting. It was like the was on one of them was the F I P hyphen Fs quote Microsoft's scan engine failed to load. Now the error description had a detail, which if you knew what was going on, would've been a clue cuz it said can't convert 22 0 1 0 1 0 0 0 1 to long meaning that it was trying to do a type conversion from a, from an unsigned 32 to a, a long value that, you know, 64 bits. But it couldn't do it. It was crashing. You know, when you look at the date of

Leo Laporte (01:36:20):
This error message and that error message there, it should have rung a bell. Right. I mean, yeah, yeah. You know, it seems like, oh, that's interesting. I'm seeing a lot of 20 twos in there

Steve Gibson (01:36:31):
Now had Microsoft's clever programmer decided to use an unsigned 32 bit integer. Mm. We could have gone until 2044 <laugh> but the same

Leo Laporte (01:36:46):
Thing would've happened

Steve Gibson (01:36:47):
Then yes, it would might maybe just use along how about that we could have gone until 20, 24 until this meltdown occurred. Yes. Now as a programmer, the only reason, and I can see to encode the year, month and day followed by a four digit serial number like this is that a simple arithmatic comparison could have been used to compare versions, which must be what's going on, but it would've been far safer to reduce the serial number portion to just three digits from four and then use three digits for the year, since three digits for the release number on any given day. That is those, those lower three digits would certainly have been sufficient. You there's a

Leo Laporte (01:37:40):
Million in ways you could have done this better.

Steve Gibson (01:37:41):
I mean, yes, that's true. That's

Leo Laporte (01:37:43):
Just, it's just, they weren't whoever did it was being too clever

Steve Gibson (01:37:48):
And yes. Not thinking about the consequences and of course exchange server has been around for a long time. No, but this is,

Leo Laporte (01:37:54):
You know, this is a very

Steve Gibson (01:37:55):
Common area. I know it's it's

Leo Laporte (01:37:57):
Bad. Yeah. Using the wrong, you know, type to represent a number and then you get these overflows that's that happens all the time. It's just a, yes.

Steve Gibson (01:38:06):
It, yes. It, it, it should have been, you know, a what a, a 10 digit yeah. String issues

Leo Laporte (01:38:15):
Long or so there's a million ways you could do it. Yep. Just that's just too clever

Steve Gibson (01:38:20):
To by, and that's how programmers get in trouble, right by going, oh, I'm gonna get, I thought gonna flex my I'm gonna flex my muscles. Not

Leo Laporte (01:38:27):
Clever. You know, it's really clever the fix.

Steve Gibson (01:38:31):
Oh. In any event, my God, in any event was immediately fixed and the world received its email only a little bit late. And how did Microsoft, oh my fix this so quickly. When, as we know most things, Microsoft does takes months or sometimes even years. Well they punted Microsoft released a power shell based script called reset hyphen scan, engine version.ps, one which needed to be run on each exchange. Mailbox server used for downloading anti malware updates. And what does the little shall script do it adjusts the date back to 21, 12 33. <Laugh> 0, 0, 0 1. Yes. That's right in Redmond. New year's has delayed and December has 33 days.

Leo Laporte (01:39:41):
Well, by now 35 it's I mean, what they're gonna keep doing this. I mean, get, it can never be 20, 22 ever

Steve Gibson (01:39:49):
Again. I'm sure they came up with this clergy hack on top top of ACLU because it has the benefit of not breaking their simple signed arithmatic version comparisons. They need to do nothing more than hold the years' digits at 21. And after all December can last for another 66 days until we get to December 99th, at which point it might be necessary to move to the 13th month of 2021, which I suppose would mean that Microsoft had vetted the leap century. <Laugh> now okay. To put everyone's mind at ease. Microsoft wrote quote, the newly updated scanning engine is fully supported by Microsoft while we need to work on this sequence. Longer term Uhhuh, the scanning engine version was not rolled back. Leo <laugh> rather, they wrote this. I'm not kidding. The I'm quoting them. It was rolled forward into this new sequence. <Laugh> that's right. We didn't, we didn't roll it back. We just rolled it forward a different direction. It's back

Leo Laporte (01:41:13):
To the future. Of course. <Laugh>

Steve Gibson (01:41:17):
They finished by writing quote, the scanning engine will continue to receive updates in this new sequence.

Leo Laporte (01:41:24):
No, they're gonna fix this. Yes,

Steve Gibson (01:41:26):
No. They said that. They said that no, the scanning engine will continue to receive updates in this sequence. So for who knows how long they're gonna leave it starting at 21 and cuz I mean, they got lots of months. You got 99 months and you got 99 days in a month. So yeah. We're just gonna abandon actual dates because our actual dates are broken. Now

Leo Laporte (01:41:52):
I can't wait to see the simple power shell calculation they'll have to do to figure out <laugh> if this is the latest version, suddenly that simple calculation gets very complicated. Oh boy, that's hysterical. That's hysterical. So they, they could go, I mean for years 99. Yeah. 99 months of, of 99 days. I dunno what that think it is,

Steve Gibson (01:42:17):
But they just abandon, abandon their comparison. Yeah. You know, some, I mean let's, if I were them they easiest thing to do would be just to redefine it as an unsigned, then nothing else has to change. Just change the type to unsigned. Then it won't bring their, their comparison. It won't break their, their conversion to long any longer. And what I would, I mean, I that's, that's clearly the simplest thing to do be if they want to maintain for another 22 years until 2044 when this will break again. And that, that ought be time to, for Microsoft to figure out the right way to do it. <Laugh>

Leo Laporte (01:42:58):
Start over rewrite the whole thing <laugh> oh, programmers are so fun. You know, we don't even, I mean, I suppose we'll never know who wrote that code. Just

Steve Gibson (01:43:13):
Incredible. No it's, I mean he's pro probably cashed out or, or, you know, you know, he got, he got, I mean, exchange has been around for so long that it was written back then. Yeah. And I'm sure he had stock in Microsoft and you're right. He's probably

Leo Laporte (01:43:28):
Rich and

Steve Gibson (01:43:29):
Living on an island. He's he's laughing right now. Yeah. He finally, finally got hit by my, you got hit by my bomb.

Leo Laporte (01:43:36):
I left, you know, that's true because if he had written it and was still at Microsoft, he probably would've known that this is gonna be a problem in 2022. Maybe not. He must have, oh,

Steve Gibson (01:43:46):
He, oh Leo, this is the kind of stuff a programmer forgets the next day. <Laugh> I mean

Leo Laporte (01:43:51):
Really again, take notes folks. <Laugh> write things down. That is just a wild story. Especially

Steve Gibson (01:44:01):
The fix. It may not have this. The problem is he, he, he would've and see it would've been, it would've been depending upon the size of the language, it would been, you know, just a assigned variable and

Leo Laporte (01:44:15):
He it's true. He may not have known that it was a, a 32 bit integer. He should have, you should know that about your language. You

Steve Gibson (01:44:22):
Have to. That's why I'm in assembly. Language is nothing's hidden. Yeah, yeah, yeah.

Leo Laporte (01:44:27):
Yeah. I mean, this is integer overflow is so common. It's such a common issue. It came up at the advent of code this months where people realized they had to use a long because there was <laugh>

Steve Gibson (01:44:39):
Leo who's ever heard of a buffer overflow, whoever that ever

Leo Laporte (01:44:43):
Happens, how, how could that happen? It never happens. Welcome to March 675th. <Laugh>

Steve Gibson (01:44:51):
It?

Leo Laporte (01:44:52):
Year 20, 21. Well this could invite redoing your numbering scheme, Steve, and maybe we can find a way to create a power shell script to do that.

Steve Gibson (01:45:03):
Oh, that's right. Who am I laughing at? I've got a, I've got a three get an overflow coming up. Yeah, that's right. Oh, what goes around comes. 

Leo Laporte (01:45:13):
But you have a plan, at least you remembered and you have a plan.

Steve Gibson (01:45:17):
We have been watching this day creep forward slowly 50

Leo Laporte (01:45:22):
To three years. Three years ago. So yeah, I should really figure out what the, as I remember, I did it once everything was January 20, 24, I think, but to I mean, sorry, December 20, 24 roughly three years, but we won't think about that right now. Right now. We'll just be glad that Steve Gibson graces us every Tuesday with security. Now we do this about one 30 Pacific four 30 Eastern, 2030 U 2130 UTC. If you wanna watch the sausage being made, we stream it all@livedottwi.tv. You can chat@ircdottwi.tv or in the club, TWI discord server. As we do this show, after the fact Steve's got several versions of the show that are unique, he's got the 16 kilobit audio for the bandwidths impaired. He's got the 64 kilo audio, just like we do. He also has transcripts, which are very, very handy, searchable transcripts, all of that@grc.com.

Leo Laporte (01:46:20):
That's his home on the web, the Gibson research corporation. You can leave feedback for him there at grc.com/feedback. You should also investigate all of, of the other wonderful things. I mean, if you've got a test son, you haven't run spin right yet. How do you know you're secure? I mean, come on, that's there, that's a free service. And of course his bread and butter, the world's best mass storage, maintenance and recovery utility spin, right, is also theirs version 6.0 available for sale. If you buy it now you'll get 6.1 for free as an up upgrade. And you can participate in the development of the next version. All of that@grc.com, he's also on the Twitter and his DMS are open. So you can message him there at SG, G R C, we have 64 kilo audio and video. That's our E unique version of the show at our website, twit TV slash SN.

Leo Laporte (01:47:12):
There is a YouTube channel devoted to security now as well. And of course probably the best way to get any podcast is subscribe. And that way you get it automatically, you don't have to think about it, audio or video just go to your favorite podcast player and subscribe to security. Now you'll get it automatically. They, if they let you leave a review, please tell the world about this very, very valuable show. The show. Everybody must listen to weekly. By giving it a five star rating, we would really appreciate that you can support the show also by joining club, TWI $7 a month gets you ad free versions of this and all of our shows, access to the club, tweet, discord, a great place to hang out all the time. I was in there even during the holidays, it was so much fun.

Leo Laporte (01:47:54):
And our TWI plus feed Andy, NACO be our next victim. I'm sorry, subject. And our ask me anything next week and you'll get that on the feed or you can join it live when it happens in the discord again, club TWI $7 a month, just go to twi.tv/club TWI to find out more. We just got another corporate member. So I think we have three now. So there's corporate memberships also available. Steve, have a great week. I hope you find something to watch on the tube. <Laugh> we're having a good time. Good. And we will be back next week where it actually is January of 2022, even though it's December in Redmond. <Laugh> that would've been another good title. Maybe we'll save that for tomorrow's windows weekly. It's December in Redmond. It's it's December in Redmond. <Laugh> thank you, Steve. Have a great week. We'll see you next time on security now. Thank you,

Speaker 3 (01:48:43):
Buddy. Hey, you don't have to wait till the weekend to get the tech news. You need join Jason Howell and myself mic Sergeant for tech tech news weekly, where we talk to and about the people making and breaking the tech news security.

Speaker 4 (01:48:58):
Now.