FLOSS Weekly 757 Transcript

Please be advised this transcript is AI-generated and may not be word for word. Time codes refer to the approximate times in the ad-supported version of the show.

0:00:00 - Doc Searls
This is FLOSS Weekly. I'm Doc Searls. This week, Shawn Powers and a call-in from Jonathan Bennett, who knows about this stuff and is dealing with it, joined our guest Philip Griffiths of OpenZiti, which is zero trust networking, fully secure networking. It has the ambition to be like the Linux of secure networking really good stuff, really deep, very involved, incredibly interesting, and that is coming up next. This is FLOSS Weekly episode 757, recorded Wednesday, november 8, 2023,. Noodling around with OpenZiti.

0:00:55 - Leo Laporte
Listeners of this program get an ad-free version if they're members of Club Twit. $7 a month gives you ad-free versions of all of our shows, plus membership in the Club Twit Discord, a great clubhouse for Twit listeners. And finally the Twit Plus feed with shows like Stacey's Book Club, the Untitled Linux Show, the GizFiz and more. Go to twittv slash club twit. And thanks for your support.

0:01:22 - Doc Searls
Hello again everyone everywhere. I am Doc Searls and this is FLOSS Weekly Welcome, and with my co-host here is Shawn Powers himself. Hello, he's come up for those not visually impaired, looking very well lit this morning in the sense of luminance. The green here is growing out or growing in, or what's going on.

0:01:52 - Shawn Powers
I went back subtly my daughter did highlights instead of making my whole head a green Q-tip. I don't know, we'll see. I don't think the green is going away anytime soon.

0:02:07 - Doc Searls
You started with orange, though, right, I mean, just so you get to his history. It was always green. Didn't you have orange at one time? I did not.

0:02:17 - Shawn Powers
I mean my natural hair color, is I mean brownish?

0:02:20 - Doc Searls
You started because your daughter or somebody was doing a color.

0:02:25 - Shawn Powers
She had orange hair.

0:02:26 - Doc Searls
You were in solidarity with her.

0:02:28 - Shawn Powers
It was yep in solidarity with my daughter. Her hair was orange. I think there's probably a picture of me with green hair and her with orange hair that you probably are.

0:02:35 - Doc Searls
Maybe that's the thing. The Halloween theme. I'm in solidarity with bald people at this point. Ant's committed. Our producer here is committed. He actually shaves his head. I just let mine wear out. It's pretty much like snow in North Dakota it doesn't melt, it just wears out. So are you familiar with OpenZD Shawn?

0:03:08 - Shawn Powers
Only in that I looked into it a little bit so that I wouldn't be completely ignorant going into the show. But since it was tough for me to kind of get the feel for it, I'm looking forward to the conversation today to get a little more clarity on exactly how it works, what it does and the advantages and stuff.

0:03:28 - Doc Searls
Let's make that happen. I want to welcome. Our guest is Philip Griffiths. He's the head product lead growth. There's a lot of Pengectival nouns there at OpenZD driving evangelism, outboard, product business development, marketing. Speaks at lots of things DevOps, iot, cyber security. He worked for Atos IT services in various roles. He lives in Cambridge the real one in the UK with his three kids. So welcome Philip. There you are with a bright white light next to you illuminating your head. So tell us about OpenZD and what it does and how we'll unpack it in the next hour.

0:04:15 - Philip Griffiths
Yeah, so thanks for having me and I have that light because we're so north up in, not the real Cambridge, the original Cambridge, so 4pm it's pitch black here. Yeah, I don't live in this country for the weather. Openzd is a super interesting project in our opinion. I am biased because I do work on it, but effectively we started it because we were of the opinion that network insecurity is broken and we think that is shown by, a, how often people get attacked and compromised and, b, that network insecurity is really hard and complicated.

And so, instead of trying to make a better VPN, we went to first principles and thought what if we could put private network in inside applications themselves?

And if we're going to do it, we should open source it so that everyone can have it completely for free. And so that's what we built the ability to put zero trust networks inside applications so the applications can be consumed from anywhere and not have any trust in the network and you do not need to listen to the underlying network, which means you cannot be subject to external network attacks, which means the majority of attacks that happen today they're redundant and they cannot happen. The North Star is obviously application embedded, because that gives a wonderful experience and security, but zero trust is a journey, and so we have a bunch of components as well that allow you to deploy on the host or deploy onto the network. But fundamentally, the mission is how do we make security by default, how do we make zero trust easy and completely free so that people can just adopt it and build it into whatever they're delivering to their own customers?

0:06:17 - Doc Searls
So zero trust is one of those words, or pairs of words, we hear a lot. Did you want to explain a little more about what that is? It's that you're not trusting anything coming in, or you already don't trust anything that's already there. What is that exactly?

0:06:33 - Philip Griffiths
So I would start off with a colleague of mine does a presentation where he has a great slide which is the blank on her name, but there's a lady from oh man, I'm blank on the show now. Anyway, she rolls her eyes. It's like oh, zero trust, because everyone uses the term and it's completely overused to the point that it's like DevOps or cloud. A lot of people just hate the term. The genesis, though, of the idea even though it has been abused, tried to be a tool for marketing departments is that we shouldn't implicitly trust anything. For many years, we set up perimeter networks and said well, if you're on the inside, we trust you. If you're on the outside, we don't trust you. You're the bad internet, don't come inside here, and we'll have parameters between those things, and that just doesn't work in a world of software eating everything and distributed systems. And so zero trust was conceptualized as a term in order to remove implicit trust. And then there's a whole bunch of downstream standards as a wrong word but architectural references from the likes of NIST and various other bodies, which puts a framework as to what are the components, what are the principles, what are the ways in which we build a system and therefore, how does it help a business to reduce risk, et cetera, et cetera. I, as you may have guessed from my introduction, I'm of the opinion that the term is massively overused. There are a lot of organizations say, hey, we've got this zero trust widget, we've got this zero trust thing, and most of it is not true implementations of the principles. In fact, there's a vendor who says, hey, we've got this zero trust thing and the default configuration is default open, so if you fail authentication, you get access. That's not, in my opinion, zero trust. That's quite the opposite. That is very open trust.

I wrote a blog last year which tried to describe it more simply using Harry Potter analogies, because most people have read or watched the movies and I did it with my six year old daughter and we effectively categorized into three non magical zero trust. This is where you may do device authentication or user authentication. Basically checking is the thing or the person, who they are, who they are, that they say they are, but they listen on the network. They're requiring bound ports. Therefore, silly muggles can find them, because the magical world is not invisible. Most vendors, most products fit into that space.

Table stakes, in my opinion, is to make our applications or infrastructure invisible, so silly muggles cannot just turn up and find it like platform nine in three quarters or remote outsuit again, if you're familiar, harry Potter. But the true magical way is to give our applications a port key and invisibility cloak, because that means the application becomes invisible to the network so it cannot be found and exploited. And then B we can just magically move packets from A to B source to destination. We don't need to know how to actually do that, we don't need to own the networks or configure the networks in between. That because ZT provides an abstraction layer to facilitate those packets to move across. I shared this blog far and wide, including to my sister, who was completely a non tech IT person. If you went, I get it. What you do for a job now. So hopefully other people will find that as a useful mechanism to understand Zero Trust, or at least parts of Zero Trust.

0:10:19 - Shawn Powers
So I have a lot of questions because while I was looking at it, there's a lot of I mean. So I'm currently in the midst of cloud training, making cloud training, so I get the frustration with buzzwords and jargon that start to become meaningless after a while. So OpenZD can you explain, like, what it means for an application? Is this zero trust networking, only internal to the components that an application uses? Is this something that you have? You know database here, users here, you know web, front end, maybe that goes out and is publicly accessible? What is it actually facilitating? Is it like a cloud-based VLAN that we trust and then is it all nodes, or however the function works, that you control, or are you using some public nodes to transfer data? And how do you manage speed? Would you like another 12 questions before I stop asking? That's good to say.

0:11:19 - Philip Griffiths
There's a lot of questions packed in there, so let's try and take that on a journey. At its most simple level, zt is a very private, very secure, very obfuscated bit pipe to take packets from A to B using zero trust principles. So we do not trust the underlying network. We do not trust IP addresses, dns, et cetera. You have to use strong cryptographic identity, so ZT comes with its own PKI or CA, or you can bring an external provider, that's no problem. This enables you to connect anything. The way it has to be done is authenticate and authorize before connectivity, because that enables us to implement these policies as well as to have a lot of control and visibility as to how we get packets from A to B. And so ZT has broadly a split into three components. There is the console, where you do your configuration and your policy. There is the fabric, which is a control plane, and the data plane, and that's being built with smart routine and HA and all these wonderful things. And then we have the edge, which are the end points which enable you to intercept packets and egress packets, whether that's operating at an application level, a host level or a network level. Now there are two use cases in my mind that you would never use ZT, for One is an unauthenticated website like the BBC or CNN. You have no value for ZT because you don't care about authenticating your users and implementing strong zero trust principles. You want anyone to find that information Use.

Case two is I've got an application that sits on local host and therefore, if it's not making any network calls, I don't care about the network. This covers a lot of what you said, but then there was that question of well, can we extend it to anything? Potentially, as long as it's speaking IP, zt can facilitate those connections to move those packets. It operates at, let's say, three to five in the OSI stack, though you can set it up to make it look like it's operating that layer too. But then you get into the interest in the area of well, what about if I want to share something on the public internet in the way that I don't for a private application?

And this is where we've continued developing some of the concepts of OpenZiti. So, for example, one of the things we've been working on in the last year we call it ZROC. If you're familiar with EngROC, it's basically that an open source and building some concepts in which they have not, and that enables you to use case two. I want private sorry, use case one. I want private, authenticated access for private applications with OpenZiti. Use case two. I want to be able to share a document, a web hook, an API, a site I'm developing with SON and I just want to give them a URL, a DNS, et cetera. That can be done with ZROC, which is a we call it a ZT native application. It's built on top of OpenZiti and it enables you to just really easily share stuff with other people, and so I think I've unpacked maybe 60% of your questions.

0:14:37 - Shawn Powers
Yeah, and my questions were kind of disparate because I wasn't exactly sure what's happening. But it sounds like and correct me if I'm wrong or clarify where I'm wrong here it seems like WireGuard or more specifically, the tail scale, add on top of the WireGuard idea with things like exit nodes and stuff. It seems like that's sort of a concept, but maybe more baked into the application, as opposed to a separate layer. Is that fair? Is that kind of what's happening?

0:15:06 - Philip Griffiths
We've differences, so a few immediate ones that come to my mind is we are strongly opinionated on ZT being fully open source. Obviously, wireguard is open source, but tail scale is not open source. Yeah, sure, there's head scale, but there's still some proprietary stuff inside there. The reason they created that is because WireGuard is a great VPN. If you want a better VPN, use WireGuard, but Wire doesn't have the control plane, and so TailScale, netbird, all of these companies have built the control plane for WireGuard and most of it's proprietary. There is some open source, so ZT comes standalone with its own control plane and data plane.

So that's aspect number one. Aspect number two is really abiding by the principles of zero trust. Sure, you can do ACLs with tail scale and you can implement potentially micro segmentation and policy et cetera, but it doesn't go to the adherence of NIST 800-207, which is the framework for zero trust and being able to push policy to the they call it the PEP, the policy enforcement point which, with ZT, is inside the SDK or inside the tunneller rather than sitting in the cloud. And so you're able to do micro segmentation, you're able to do least privilege, you're able to deny by default in a much more rigorous way. There is then other things built on top, like posture checks or other security aspects, which again really fits to. You want the most secure solution. If you want a better VPN. I talk a lot to people on Reddit. If someone says, hey, I've got this problem, I need a better VPN, use WireGuard, use tail scale, I would 100% tell you to do that. If you are concerned with having the strongest security, the strongest control, then you're going to love what ZT does. The other aspect that definitely comes into it then is, as you allude into, the software development kit, so that developers can pick it up and utilize it and build it into what they're doing.

For us, this is the North Star because of a couple of reasons. A, it gives you the strongest security. If you're building a zero trust overlay network into the application in memory, then you're not even trusting the host OS network, so even if a malware gets onto my device, it cannot break into the network and literally move. I think it was. So I listened to security now and, looking at the hacks which have recently taken place I said recently in the last year against the password company, a lot of them, or that one specifically one of the hacks, because they've had a couple happened because they got malware into their home lab or something and then it got onto their work laptop and then it literally moved across their VPN into their corporate environment. That becomes impossible, not impossible. That becomes almost impossible if you're building zero trust into your applications because the malware would have to break into the application and then only gets access to the other side of the application, the server side, rather than actually being able to actually move across the network.

So, a we're able to have the strongest security, but we also deliver the best user experience because you don't have to have the additional agent loaded onto your device. It's just part of the app. You don't have to have the user's complain about setting this up, because it's just part of the experience. And so there are cool and interesting use cases that you can do where it's A building into the application. Or, b, we do a lot of deployments into edge runtimes where your system isn't necessarily running Linux it's like some stripped down Linux or it's not even running an OS where, hey, I could build an SDK into that. In addition, we have use cases where we can do we call it clientless, zero trust.

Now, it's not actually clientless because according to our methodology, there has to be a client to do that authentication authorization before connectivity, but the user is tricked into thinking it's clientless. We call it browser, but with a Z, because we put Zs inside everything that we put CT inside of, and effectively it gives you a public SaaS-like experience. I go to a URL, I authenticate to my IDP oof, I get access to the application, but the application's never exposed to the public internet. And we do that by effectively loading in a JavaScript-based SDK with a little bit of wasm into the user's tab running in memory. Because this is our North Star how can we give the highest level security with the highest level of ease of usability and then give it away for free and open source so it becomes the standard?

0:19:57 - Doc Searls
So, oh boy, we have an unusual situation here, in the sense that both Shawn and one of our other co-hosts and other people in our back channel are piling up the questions, and we'll get to those after this break.

0:20:12 - Shawn Powers
All right, I'll just steal the mic again because the questions are on my mind and I didn't write them down. So I get it. So thank you, that does help. I wasn't sure exactly what problems specifically opens that he was trying to solve, so that is extremely helpful. So can you help me with? Like I saw on the website recently, there was a integration with Catty. Now I'm a huge fan of Catty, I run Catty for my web servers, I reverse proxy everything with Catty. It's just the catch me how I just really love Catty. So that excited me. Can you explain the use case though? I mean the example there. Even looking at the example, I wasn't exactly sure what was the end result of the integration. Like, I get that it was kind of built into the network, but then what? Is it only accessible to somebody else, tied into that network, or was it like some fancy remote proxy so that it was accessible from somewhere else? So what is an integration into something like Catty do?

0:21:14 - Philip Griffiths
Yes, and, side note, thank you for pointing that out, because I was actually doing I was putting together some slides earlier where I was looking at, you know what. We're not doing a good enough job of communicating the who and the why of our value proposition. We're going too much into the what and the how. We need to update that and you basically just demonstrated that the. There's actually two bits of work that we're doing with Catty and we've done some work directly with Matt, who's the originator of Catty on it. One which I'll touch on first, but it's not the one you mentioned, is inside Xerox. We're actually using Catty as the front end, so Catty is basically embedded in Xerox in order to make it way nicer and more beautiful and more feature rich to be able to, to you know, share files, share webhooks, et cetera, et cetera, and it really leads towards a bunch of features we're developing called drives, which basically will facilitate a distributed, secure ability to share stuff. I think Google drives, but more secure and more distributed.

The question you actually asked was the blog we did on Catty of where we took our Go SDK and we built inside.

The simple upshot is that you can make it so that, instead of your Catty server being addressable by anyone on the public internet and you require, therefore, inbound ports et cetera. You can have your Catty in a completely private network where it's going to do outbound connections to the ZTO overlay and the only people that will be able to connect, therefore, to Catty is people who are authenticated and authorized to the ZTO overlay, whether they're running a ZT desktop edge or for whatever operating system, or if they're using that browser clientless version. Effectively, it means your Catty server becomes non-addressable, but you haven't explicitly said they should be able to access it, which massively cuts down the attack surface but then also massively reduces operational overhead of again managing access control lists, firewall rules, et cetera, et cetera, and all within the binary of Catty. So rather than, hey, I've got to deploy a separate agent that sits in front, to the side, whatever, of Catty, instead it's just inside the binary.

0:23:32 - Shawn Powers
Okay, so how was the website hosted inside the OpenZD layered network access, like? What do you type in the browser URL? I mean, is this internal IP networks? Is this a DNS? That has resolved somewhere magical what's.

0:23:49 - Philip Griffiths
That is fully your choice. Obviously, the website or the resource that you're looking to access will have a private IP address. Cadd previously required a public IP address. Now it doesn't, it can have a private IP address For the user that's accessing it. You can call it anything you want One, two, three, four. In fact, you could do one dot one dot, one dot one. And it's not going to go to Cloudflare. It will go over ZT, because ZT has its own private DNS, because ZT does not route traffic according to IP addresses. It routes it according to the identity system that ZT has. Okay, therefore, go ahead.

0:24:28 - Shawn Powers
But then how do we get that running in the browser? I mean, I guess I understand that, but say I want to access this. You said that the application is running, you know, like a JavaScript app or whatever, in the browser. Is this a separate, a different use case? I mean, like in my browser, am I going to type if I type one dot one, dot one, dot one, it's going to go to Cloudflare, oh sure.

0:24:51 - Philip Griffiths
Yeah, so well, this depends. If you're running ZT desktop edge and you've defined a rule that, to come to my website, it's going to be one dot one dot one or both in the face or whatever, then the ZT desktop edge, which actually are ZT desktop edge for Windows users, wintown, which is from Wyguard Thanks, wyguard. It's a great way to insert packets. That sees the packets in the network OS first and it goes oh, this is a packet for ZT, we're going to take that over and egress it from wherever the policy says to egress those packets.

0:25:23 - Shawn Powers
Okay, so there is in this scenario a client. The ZT edge client is running on my computer.

0:25:28 - Philip Griffiths
Okay, that can be answered and that takes you to a landing page where it goes authenticate to Google, authenticate to GitHub or whatever the IDP is, that returns the x5 and that loads the JavaScript agent into into your tab and that then connects to the overlay and establishes that end to end encrypted connection from that browser tab to, in this instance.

0:25:53 - Shawn Powers
Is that end-to-end encrypted connection from that browser tap to in this instance, so that one does and is this a self-hosted, like authentication kind of hop-off page kind of thing, where I start that browser with the Z JavaScript app, or is that like something generic that once I'm authenticated then it knows? I mean, I assume that's something that I have to host on like a publicly accessible SSL site and that sort of a thing.

0:26:25 - Philip Griffiths
Correct. So yes, if you're self-hosting browser on OpenZiti yourself, then you host that. We call it the browser gateway, and that does the connections to your IDP and it returns all of the stuff and it does all the magic for you. If you're using Cloud ZT, which is our SaaS version of OpenZiti, then we're hosting all that infrastructure for you and that's just invisible and it all just magically works.

0:26:51 - Shawn Powers
Sure, okay, one last question on this line, and then I'll let Doc talk, because I worry that he might have fallen asleep on his ear. So, oh, dane, and I had a good question and I might have lost it.

0:27:02 - Doc Searls
So, doc, if you want to, Actually this is a good time for a second break, so I'm going to take a break. Then we go to some questions from the back channel right after this.

0:27:13 - Shawn Powers
Okay, I thought of it during the break, so again in this scenario of the browser in Cadi and the OpenZiti network in place, one of the things I love about Cadi is that it really, really wants you to use SSL in crypto traffic by default. Is this something with the encryption and the trusted network of ZT? Is that something that isn't done, or do you still use SSL? And if so, how on earth would you make it a valid SSL cert? Or is just self-signed the only option there?

0:27:45 - Philip Griffiths
Good question. So yes, if you're using browser with a Z, then you need to have a cert so that you get that nice little green icon that says this is secure. If you're using ZT itself, then you can be serving up an application and your browser goes whoa, this isn't secure, but it's really secure. Zt doesn't touch or decrypt the application layer encryption. What we're doing is mutual TLS between every single hop and then we have end-to-end encryption between the source and destination end points, and that's built to be extensible. So by default it's chart, our 20 poly 1305, which is the same as wire guard but is pluggable. So we've had people that have deployed it with FIX 140-2. I know of people that try and quantum encryption with it, etc.

Therefore, you could be doing a completely private connection that from the outside you can't see or understand what's happening while the little icon goes. Oh, this might not be secure because the browsers don't know that. You know that effective, that private CA. There is also another interesting aspect of if you look at a ZT packet on the wire at some point in the hop, there's also ideas of Tor in ZT and so we're encrypting the metadata so that the only thing that you see on the underlay network is what is the next hop of the public IP, of the next hop? You don't know where does it come from, where is the actual source and destination, where may be other hops in between here, so that you have, yeah, a lot of obfuscation built in.

0:29:26 - Shawn Powers
Okay, so you could have a little green lock in your browser. It would just require some thinking about what's actually happening so you know what traffic you're actually using.

0:29:38 - Philip Griffiths
If you're using Caddy, then Caddy can give you that nice little green lock. We have had these internal conversations because people have gone hey, you can have the nice little green lock, and we're like wow, yeah, I mean we could do it. It means implementing this and it's it always comes down to where do we develop? Where do we focus our time?

0:29:56 - Shawn Powers
Cool, okay, yeah, I'm done.

0:29:57 - Doc Searls
sorry, I dominated the whole show You're never done and I don't want you to be done. So we have a whole bunch of questions backed up here. I'm going to take the first two. Is this related to software defined perimeter, and does OpenZiti use FW to knock or single packet authorization anywhere? There's a little backstory about why he's asking those questions, but we'll go with it. We'll start with those and see where it goes. Great question.

0:30:27 - Philip Griffiths
The simple answer is no as to how it's defined in the cloud security alliance. But if you check out Wikipedia for SDP, I updated it to include authenticate before connect because in my opinion it is a valid implementation of SDP, because the concept of SDP is we don't trust the outside world. Therefore we shouldn't let anything in ZT is doing authenticate and authorize before connectivity on the endpoint, based upon the identity that that endpoint has, and so you cannot connect to the overlay without doing that and you know we're outbound connection on both sides. First packet authenticate or single packet authenticate is effectively a cryptographic packet that you send to a certain port. If it is correct, then it opens up a connection. If it's not, the packet just gets black hole. We chose not to go in that direction because then there's implications around how do I manage my file rules, blacklist in whitelist, there's complexities that come into it and, yeah, our opinion was we'll do that authenticate and authorize before connect because that aligns to where we're heading with our product.

0:31:42 - Doc Searls
So I'm looking at the back channel here. Okay, so this, I think, is a follow on what you're answering. Now he says I'm curious how they're running a JS code at the same time as an arbitrary web app. That's a complex question. Adding a weir I'm not an expert but I can.

0:32:05 - Philip Griffiths
So we use JavaScript, but we also use wasm to do like TLS, encryption, offload, so it's faster. We also within browser. There's in the client on the user side, there's a browser runtime and there's also a browser service worker and I think it's the service worker that does the function of how do we do intercept of calls and be able to function. But yeah, there's much more knowledgeable people than me on that topic.

0:32:43 - Doc Searls
So a couple others here. So I was saying that, oh, does magic Lizzero trust you some iteration of port knocking and is that? I don't know if that follows on what you just said or not, because I'm a little bit lost here around the where.

0:33:02 - Philip Griffiths
No, we're not doing port knocking because port knocking is sending a cryptographic token, but in our opinion again, particularly with NAT, you've got NAT behind NAT behind NAT. Nowadays you can't do port knocking because you don't know you're potentially sending to all of those IP addresses that are behind that NAT, and so, no, we're not doing port knocking in the traditional sense, but the Harry Potter blog that I wrote did have a reference to wink, wink port knocking in terms of how's each operation, but it's not how you think of traditional port knocking.

0:33:45 - Shawn Powers
So it's not like just a stateful connection all the time. That I mean, that wouldn't be feasible. So all right, I guess I have to read the blog way to make us read your blog entry, man.

0:33:58 - Doc Searls
I'm not going to get hurrying to go look at that and see if there are more questions or answers to questions. So we also have a question of how onboarding works, noting that it's often a weak spot with ideas like these.

0:34:14 - Philip Griffiths
Yeah, good question. So it depends. Is the is the simple answer, the, as I alluded to, due to our approach of authenticating, authorized for connect and strong identity. You, in most scenarios, need to have a endpoint and an identity. But the question becomes where are you putting it? Where does your trust demarcate? You can deploy a ZT edge router and say I trust everything you know behind this, I trust the whole land site, I trust the whole VPC, vnet, whatever. Yes, you could do micro segmentation, at least privilege off that, but you'll, you'll put in trust in the land environment.

Our opinion is you might start there, but it's better to to extend to the host so you're not trusting the local environment. And so that then means today I need to to put a endpoint where it's Android, mac, whatever your windows, linux on the host, and I need an identity. There are exceptions to this, for example, the browser. One browser makes it magical so that you're not doing any onboarding. The use cases we see that most person for is I can't put an agent onto my device because I don't manage it, I don't have privilege or I don't want that user experience. Simultaneously, if you're building inside the application, you don't see it. You know, my bank is NatWest. If NatWest built ZT into my application I would have absolutely no idea, but now they're clad at cloud API has become dark and unaddressable on the internet, but it doesn't change my user experience One of the pieces of work that we're doing at the moment we've been working on it for a while where we can use external identity providers and so we see this used a lot in cloud native, like Spiffy Spire, and then also IoT with hardware security models, hardware trust, you know PCKS11 compliant identity.

Zt can take that and go. Ah, I know this identity because I'm talking to this other CA and that CA has told me to expect this, this public key, and accept it as an endpoint. One of the pieces of work that we're doing at the moment is to extend that to any OICD or SAML compliant identity system, which reduces that initial barrier to entry, because then you just need a ZT endpoint and you've already got your identity, because most companies already have their own IDP. They have their own identity within their organization.

0:36:47 - Shawn Powers
That was kind of a question about the bank, then I mean, because you need to unload that information, right. I mean it's great to say like, oh, the app is on the ZT network, but you know, you gotta. But I mean, you know it's a catch 22 right, how to get on the network without having the information and you can't get on the network so.

0:37:01 - Philip Griffiths
Nat West or are already using X5 and 9 certificates to do mutual TLS and enter encryption. So sure. If your app doesn't already have that, then yeah, you now need to work with identity and maybe use ZT's identity, maybe you bring your own. But if your application already has identity, then you just put in the SDKs inside. Now it's all invisible.

0:37:19 - Shawn Powers
It just feeds it right to it. Yeah, that makes sense. So can you give me, like, let's say I wanted to start playing with this, because I kind of want to start playing with this. I did look and there's, like you know, the Docker setups. What are the pieces required to make this happen? You know, and let's pretend, not the browser piece, because that's an added complication or you need to develop. So let's pretend I actually want to install the client. You know, like on my the edge piece I think you called it what pieces are required to make all of this happen? Because, again, this isn't just the encryption, this is also that control layer. So what pieces are required?

0:37:55 - Philip Griffiths
And I would also say as awesome as browser is. It's a little bit pre-beta at the moment we have some clients of customers for it but it has some sharp edges, particularly if you're self-hosting. If you're using Cloud ZT, then we've abstracted all the way so it's much easier to test. Likewise, segue with Cloud ZT. We have a completely free tier, so you can sign up and you don't have to host anything, just end point here, end point there. Make a policy, it's all connected.

Assuming you're not using Cloud ZT and again, I really want to self-host this. Well, you require at least three things. You require the orchestration engine. We call it ZAC, the ZT automation console. I think it stands for that is where you put in your policies and your various different things. You require a controller, at least one, which implements policy. It has the PKI and the CA and it distributes control.

You require at least one edge router. The edge router is what the end points outbound connect into to form the data plane and then whatever I'm trying to put on the overlay device over here cloud over there you distribute those edge components to whatever components you want to connect and in your ZAC console you build a policy that says this endpoint can connect to this identity, this identity can connect to that identity for this IP address, this DNS, whatever. Again, an important point to understand with ZT is it's not wire guard, it's not tail scale, so it's not just deploy to end points and poof, it's all just magically connected, that I can do any application connections that I want. You have to explicitly say I want to connect this IP address, this DNS, because it is denied by default, although you can set it up to do 0.0.0, slash 1 and just take all packets. But that's again, that's an intentional design choice to be coarse in terms of your intercept. So console, controller, edge router and then endpoints on whatever you're connecting. If you're using cloud ZT, endpoints on whatever you're connecting.

0:40:07 - Shawn Powers
Is edge router or router, depending on which side of the ocean we're on. Is the edge component the same? I mean, you mentioned that as the third piece, but is that the same in all the places or is that different than?

0:40:21 - Philip Griffiths
So ZT has its routers and they come in different flavors. You can have a router and you can either turn on or off what we call the link listener. If you turn on the link listener, it becomes part of the smart routing mesh which you likely put on the internet. But it could be in a completely air gap network and that becomes think of it as like software MPLS. It has the ability to in fact it runs Dijkstra's algorithms to figure out what is the quickest way to get packets from A to B, and then you can implement policy.

Well, I don't want to take this router for this service. Or again, you can get very granular if you want to. Or you can say, hey, just accept all services. If you have your link listener off, then the edge router sits in your private edge and it does outbound connections. It can see all of the other edge routers but it does not operate as part of the mesh and this enables you to build H&M resiliency in that edge in terms of how you're getting on and off the overlay.

Or you can bind them together and get you know one of the organizations I know using ZT. They're all like oh, what's the max throughput I can get. They put four edge routers together and they're like hey, I can get 6.5 gigabits per second, like that year ago or something. It might be more now, but the point was they wanted high throughput. So they chain them together and they can get higher throughput. On an edge router, you can also get it with a tunneller deployed onto it. So our edge routers are based on Linux, ubuntu to be precise. It used to be Red Hat, but we know what happened there with CentOS.

So if you have a tunneller on it now you can actually do the ingress and egress of intercepting or terminating packets off the overlay. And so some people they get started just by they deploy an edge router with a tunneller on it into their VPC, vnet, dmz, whatever. And now I can say now I can apply zero trust to this environment. The point of that is, yes, we wanted to make a difference between the fabric routers and those which are actually in the edge, but again there's scenarios that you basically want both, so it's plug and play.

0:42:33 - Doc Searls
Cool. So we're going to take another break and after that we're bringing in by audio because he's not visible today. Jonathan Bennett has actually been the source of some of the questions we've already asked. I just thought about our back shell. Why don't you just come on the show? So we're going to do that after this.

0:42:53 - Jonathan Bennett
Hey, philip, I wanted to jump in and ask a couple of questions just directly to you, if I could, and the reason being is I've done development on a project. It's kind of a semi-retired project now, but it's FWNOP, and that's kind of what we call the next generation port knocking, and we got hooked in with the software-defined perimeter guys, because somebody said, hey, we need to do a single packet authorization with this, and we got wind of it and said, oh, we do single packet authorization, and the weird thing was, though, that they wanted to use the SPA name but completely ignore all of the work that we had done with the SPA, and so, when I heard about this show and started listening to it, boy, my spider sense is really pricked up, and so I'm curious. You talk about having services that are completely dark, and I'm extremely curious how do you pull that off? What is the mechanism to make those services dark? Do you use SPA, do you use something like FWNOP, or is this dark just because they're only accessible via a VPN?

0:44:00 - Philip Griffiths
It is dark in that both and more you're able to if you think so. Let's say I've got applications in data center A and applications in data center B and I want them to connect. You can have ZT in the middle in those environments and both of those environments require zero inbound ports. And in fact, going back to the tail scale, why I got a thing. They say the same thing oh, you don't need any inbound ports. It's not completely true, you require inbound ports somewhere.

Well, so yes, with them, because they're doing point to point connections and the same thing that you do as SPA. You require inbound UDP ports. With ZT you don't require inbound UDP ports. You can block all inbound and in fact you can also block all outbound, except for to the overlay. So even if malware got into your environment, it cannot connect to a command and control or X-Fortrait data. How we do that is the ZT endpoints with their identity are told by the controller how they're going to connect into the data plane, and so they do outbound connections into the data plane before you establish a connection. So maybe it's client initiated let's say it's server initiated both the client and the server side of ZT outbound connection to the overlay. The overlay only listens for authenticated and authorized connections and then either the client side initiates over that connection or the server side initiates over that connection, so that effectively, you're able to have the source and destination of being completely dark.

0:45:33 - Jonathan Bennett
Now, how does your controller only listen for authenticated connections? How do you get the authentication before you have a connection?

0:45:42 - Philip Griffiths
That is done using strong cryptographic identities X509s and Jots or JWTs, which function as the core identity of the system. And so if you want to exist in the ZT overlay network, you have to go through the process of bootstrapping trust, whether that's through ZT's mechanism of trust and that's like a five-part blog which goes into this, or you bring in your own external identity system and you've bonded together that identity system with ZT. So expect to receive your identity. But that is a core component that effectively those endpoints their outbound connection to controller. The controller is going well. Here are the services and configurations you're going to send. So if I see a packet, you're going to outband connections onto these components in the overlay network so that to the user you just define services of which should communicate to which of the identities, and everything else is just abstracted away Under the hood. Zt is really complicated. Yes, I'm sure.

And the surface is pretty complicated and it just magically works.

0:46:50 - Shawn Powers
Maybe I'm not understanding what Jonathan's asking, but if it's not listening to anything except for an authenticated ID, or whatever the terminology was, where is it listening for? The authenticated ID, I mean sure?

0:47:07 - Philip Griffiths
So let's be clear the fabric, both the control plane and the data plane, components that sit in the middle they need to exist on an IP that is reachable by the edge. Now, whether that's on the public internet or it's in an air-gapped private network doesn't matter. It needs to be reachable from those components. So if you are using the internet, they have a public IP and therefore, if you try to connect it and you're not authenticated, it goes. You're not able to connect here, You're not authenticated to connect here. But this is where then you may go.

Well, I could, I could dust the controller, I could dust the data plane, blah, blah, blah. That's why we've built resiliency into it, so you can deploy multiple components within the smart routing fabric and if you dust one of them, that causes the latency to spike. Therefore the smart routing mechanism goes whoa. This is a bad path. Let's gracefully move the connections over to other paths which are available and which are better performing, because ZT is an ephemeral network. In fact, if you move resources, the network may move with you in order to facilitate that connection.

0:48:07 - Shawn Powers
And also the plane or the controller is listening for that connection. I mean I just somebody has to be listening.

0:48:15 - Philip Griffiths
So okay, I get it.

0:48:15 - Shawn Powers
You just you don't connect directly to the other edge or whatever it is you have to. The controller has to facilitate that connection. That's the part I missed, and maybe Jonathan understood that I just missed that aspect of it. So thank you.

0:48:28 - Jonathan Bennett
I would imagine, then, that the controller code that is actually listening and doing that authorization step is one of the places where you guys have spent all of your time, a lot of engineering effort, making sure that's secure. Has it been audited by anybody outside? Like how many eyeballs have looked at that code?

0:48:45 - Philip Griffiths
Quite a lot of eyeballs. Yes, we have had some people who I can't say their name, but they're experts in this area and they really put ZT through the ringer in terms of its code security. We have several use cases where ZT is being used in very, very high security scenarios used in a US defense contractor who, in their word, zt provides the best adherence to NIST 800-207 across a wide variety of architectures and these are the type of people that look at your code quality, make sure there's no back doors, no vulnerabilities, et cetera. This is also one of the routes of where we've been putting a huge amount of engineering effort into for the last couple of years, and it's probably I'll CTO. I won't lie, I'm saying this, but this month maybe start up. Next month we'll be in beta and testing with some of our customers HA for our controllers, and we use RAF and Gossip, which are two open source protocols, to be able to share, state and elect leaders so that if, for example, you dost one of the controllers again, they go oh, that one's not available. Let's shift over the control to another leader so that the system can maintain resiliency.

There is actually as well. Here's an interesting one. We work with a university that does some research area around Zero Trust and they came to us and said could I apply SDP to the controller? We had a long conversation. There's pros and cons. We were like the code's open source, so if you implement it, there's a nice implementation. We might adopt it. That would make your controller dark so that you couldn't initially try to find it and do an attack against it. Again, there are pros and cons, but there is a world of interesting things that could be done.

0:50:30 - Jonathan Bennett
Yeah, it seems to me like that would be a perfect place to use something like FWNOT, which I don't know if you've looked into that program particularly, but we kind of pioneered the idea of SDA.

0:50:39 - Philip Griffiths
I don't know what this individual was planning to do, as they looked to do that with some of their PhD students, etc. Right, I could have been with it.

0:50:49 - Jonathan Bennett
Sure. The one other thing that I didn't quite understand is the browser with a Z, with a Z app. Is that an executable? So is this an additional browser that someone would install on their computer, or is this just a set of JavaScript tools that runs in a tab?

0:51:08 - Philip Griffiths
It's just that latter part. It just runs in your single tab. When it's loaded into your tab, it's only that tab which has access to the overlay and the services on the other side. None of your other tabs, none of the rest of the machine has access to the overlay.

0:51:23 - Jonathan Bennett
I actually need to look into this, then, because one of the problems that we ran into and we kind of never got the funding to figure it out was how you make a system like this work with just an arbitrary web application. Because a web app is going to be opening new TCP connections, it may have multiple HTTPS connections and trying to get all of that routed through your data plane and not going out, trying to either getting blocked or going out through the unfiltered internet, was one of the challenges that we didn't work out. So it's pretty impressive if you guys have really nailed that, because that's not an easy problem to solve.

0:52:03 - Philip Griffiths
Big shout out to Kurt, who's our developer that's been working on it for about three and a half years as his sole job. That and creating the Nodejs SDK which goes with it for server side yeah, there's been a lot of complex engineering that's gone into it. I will be honest, it doesn't work with every single application because there are some applications which are such old code that it doesn't have the concepts to be able to use service work and some of the other functions that browser requires to do it. But on the latest dashboard I think I saw it has been tested in various apps. I think it's up to about 90% nowadays. So as long as it's been built in the last 10, 15 years, it's great. If it's really negative, then it's.

0:52:47 - Jonathan Bennett
So you guys are running into some of the same problems that we were looking at. That's interesting. You just got the funding to fix it.

0:52:55 - Philip Griffiths
Yes.

0:52:56 - Jonathan Bennett
OK, so there's one Since I'm on. There's one question I love to ask everybody, and that is what is the weirdest or most surprising thing that someone has done with OpenZD that you're aware of?

0:53:07 - Philip Griffiths
Oh, that's a good question. So is this the most weirdest? There was a university which wanted to build an intent-based networking solution using blockchain and they went and thought, oh, we should do this with SDP. They might have looked at various different SDP options and then they chanced across the team. They were like this is great, if we just use that, then we can focus on the integration and the blockchain stuff. And again, we did some presentations with the CSA and various opinions on using blockchain is not necessarily a good idea for this, that and the other, but it was an interesting thing and, from our perspective, this is why we're open sourcing it, because our vision is to turn ZT into the equivalent of Linux for secure networking, that people just take it and use it Whatever crazy research project they have or some product that they've built. Any time you're looking to build a secure, distributed system, zt just solves loads of problems for you to take it because free and open source.

0:54:20 - Doc Searls
Well, we actually had a moment of silence there, oh my mic must have been muted. Sorry, I have. I'm so glad these two guys are on this show because a lot of this is way outside of my experience. But you used the term identity a whole bunch of times and I'm thinking is that the MAC address? Is that the machine handle? Is it? What is that? And when you say an identity because I know a lot about identity, I co-host an identity conference, but I don't know what you mean by that.

0:54:54 - Philip Griffiths
Yeah, and now we get into a religious conversation of identity identifiers, people of different opinions. But in our context, the identity is the system through which ZT knows who the endpoint is, whether it's coming from the ZTCA PKI of the X599 or you're bringing some external identity provider and you're tying them together. That said, zt does have a suite of posture checks built into the various SDKs which allows you to do more, Like, for example, what is the MAC address of the device? Is it domain joined? I want to do TOTPMFA, I want to check the OS operating system, I want to make sure antivirus or PowerPoint is open and running. There's various different things that you can do to do extra layers of authentication and authorization.

0:55:44 - Doc Searls
It is funny. Also in the back channel, we've noted that this may have. We may have reached the high watermark of initialisms used during your show, or acronyms, if you could pronounce them. That's the difference. With those, we'll have to ship with glossary. What was that?

0:56:03 - Philip Griffiths
We'll have to ship with glossary.

0:56:07 - Doc Searls
That's probably not a bad idea. You chose Apache as your license. What's your thinking with that?

0:56:17 - Philip Griffiths
Correct. We went Apache 2.0 because we wanted it to be permissive. We want anyone to be able to pick it up and use it and get value from it. That's it In fact. We know there is a few. If we are successful, there is a future where there's other organizations that will create their own flavors of open ZT and then we'll have to gift it to a community that runs it in an open manner. We haven't done that yet because by going through a governance framework you lose control and you lose speed. We want to get ZT to the best version before we have to do that. We know that will happen. We don't want to do a terraform, because that removes all the value. If you make a technology that's truly transformation, that can plug into everything, then it has to be open source, it has to be owned by multiple parties. The logical starting point of that, in our opinion, is only Apache 2. That's all different.

0:57:16 - Doc Searls
When you say governance framework, are you talking about a informal agreement? Are you talking about I'm?

0:57:23 - Philip Griffiths
in Linux Foundation so you have a governance board and you have subcommittees that make decisions for engineering and lots of people involved, etc. Which is great, but it also slows down. We didn't want to start there.

0:57:38 - Doc Searls
Okay, this is great. We're pretty much at the end of the show. At this point I'm curious about, before we get off, about your team. How big is it? I assume it's mostly distributed. You probably don't have a central office, maybe you do. I don't know if you're big enough to have that Put your name on a building. I know companies put their name on a building. There's nobody in there, they just bought the building. We are distributed today.

0:58:00 - Philip Griffiths
We have about 60 people. There's around 10 working just purely on OpenZiti. Then we have maybe another 10 who are doing go-to-market of sales, marketing, growth, talking to the community, etc. Then there's management. Then there's another whole bunch of engineers who are doing support or technical account management or building our commercial product, the cloud version of it, Majoritively Charlotte, North Carolina, so close to where it used to be for our headquarters, and then a lot, particularly the ZT team in upstate New York near Rochester, Some people in India and then scattered people across the globe like me and Cambridge.

0:58:48 - Doc Searls
You're one of the scattered. This has been great. I think we should have you back after OpenZiti has become the Linux of secure networking, which I think is how you put it. This has been really great. I can't believe how fast this show has gone. It's been a very quick hour. I really appreciate your hat coming on the show.

0:59:10 - Philip Griffiths
It's a pleasure, thank you.

0:59:12 - Doc Searls
I hope you go out and hit the pub after this.

0:59:15 - Philip Griffiths
Unfortunately I have three kids oh yeah, three kids Bring them down. Before I get. If anyone is watching, this is Ziggy. He's our mascot. Oh cool, he's a ninja. He says trust no one or trust.

0:59:34 - Doc Searls
I thought it was a microphone. I see, okay, that's great.

0:59:38 - Philip Griffiths
My kids love him, though. I'll find books lying around the house and have little Ziggy stickers hidden all over it. It's like, ah, god damn they found more stickers.

0:59:47 - Doc Searls
That's great, that's great. Well, thanks again for coming on the show. We will have to have you back. Thank you.

0:59:54 - Shawn Powers
Hey, mr Serres, I'm going to cut in. We didn't ask him our standard questions. Come on, the people want to know, oh no, my gosh.

1:00:01 - Doc Searls
Oh, you're right. You're right, I'm looking at too many variables. I'm going to bring you back to answer that we always ask these and this is the first time ever I think in four years I have not done it. What are your favorite text editor and scripting language?

1:00:15 - Philip Griffiths
Okay, so I'm going to answer that with a story. I was at Goforcon a few weeks ago which is just developers talking about Go Lang and over a few beers. I said to someone do you want to know a secret? They went what I said. I have never written a line of code in my life. They went, never. Whatever. You've been talking these super technical topics for an hour. Literally I've done a tiny bit of bash on a command line, but I've never written code. I come from, I did a history degree and I've become progressively more technical and I can sound like I know stuff, but I've never actually done it in anger and so I mean I don't word. I like writing something word and PowerPoint.

1:01:03 - Doc Searls
Whoa, that is the first time, anybody said word, I mean it's the first time. Way to go. That's Ed Wing, and you're kind of and boy, I'm not all that technical and at least I use the eye. So there we go.

1:01:24 - Shawn Powers
Yes, siros this is a first. I think you got the guess beat what.

1:01:29 - Doc Searls
It's a rare one, but I've got the guess beat. But you really know your stuff, dude, so I really appreciate it. This is excellent. You're doing really well. You're not faking it, I can tell. So this is awesome. Thanks again. So I have stuff on my tongue or some other body part. So, Shawn and Jonathan, this is unusual. I don't think we've done this that often where we had to bring another co-host on the show to ask questions.

1:02:07 - Shawn Powers
I really enjoyed the explanation because, like I said, I went in kind of getting a lot of the concepts I mean like what they meant about zero trust, networking and stuff but not what it actually did. What was the use case? What does it look like when it's installed? And I got that. For me it was the comparison to Water Guard and Task Scale as far as what it is, but how it's more than that too. So, yeah, I really enjoyed that a lot and I came away understanding it so well. I mean understanding what it is.

1:02:47 - Doc Searls
So do? Either of you want it, either of you like you both deal with enterprises of various kinds, types and sizes. Are you swung? You might say.

1:03:02 - Jonathan Bennett
I'm personally I'm pretty interested to see how they've solved, to look at the details of how they've solved some of these problems. I don't know that anybody that I deal with would make use of this and it's probably a bit overkill for me to use internally, but there are enterprises out there where this is going to be a killer solution for them. I also find it really interesting that they do still have that problem of, well, okay, the controller has to have open ports, right, that is really hard to get away from. Something's got to be listening somewhere. But other than that, it's a really it's a cool solution and neat to hear from him.

1:03:43 - Philip Griffiths
I agree with that, John. I mean, I don't know who your clients are.

1:03:46 - Shawn Powers
Where's that voice coming from? He's gone, he's still here.

1:03:51 - Philip Griffiths
One of the things I am very cognizant of and that we're trying to be cognizant of more cognizant of in terms of our ideal customer profile is A organizations who want the highest security or who sell to people who want the highest security, and then B organizations that need to get access to their customers network and, ideally, have highly distributed environments. The people who really love ZT is the Oracle is currently picking it to build their Zero Trust network platform. These defense contractors are starting to build it into all of the products that they sell to their customers. These types of organizations who, as you say, they need the highest security, they need the highest control. They are in other people's networks. That's where you want it. If you're just connecting your home lab again, I'm using the simple use case use WireGuard, like damn. That's just what you should be using.

1:04:40 - Jonathan Bennett
Yes absolutely.

1:04:43 - Doc Searls
So we're moving on here, so time for plugs. So, Shawn, what do you have to plug?

1:04:50 - Shawn Powers
So I started a new job. I'm a trainer at CBT in Nuggets again, so I don't have anything to currently plug. I mean, you can continue to check out my website, or, yeah, there's my newsletter. There'll be hopefully information there before too long about what's going on with me, but yeah, I'm just in a transitionary phase which sounds far more professional than it is.

1:05:14 - Doc Searls
So, jonathan, what about you, man?

1:05:16 - Jonathan Bennett
So I've got a couple of things to plug. First off is don't forget to follow my security column over at Hackadaycom. It goes live every Friday. We talk about security and other fun things. I wrote a little short story about Perfect Dark Recompile. That's on the screen for those watching. That was a lot of fun, and playing through that game again has been taking up far too much of my time recently. And then, of course, there's also the untitled Linux show. That is a ClubTwit exclusive. You can catch it live on Discord or, if you are a subscriber to ClubTwit, you can get it on the RSS feeds there. Talk about everything Linux, the news. We'll talk about the kernel versions as they come out, what's coming, and boy, there's some really cool stuff coming to Linux. I'm going to be testing out some HDR stuff in the next couple of weeks and I am super excited about that. So make sure to get on ClubTwit and catch the show to get the down low on all of that. And thanks for letting me come along, guys. It's been a lot of fun.

1:06:11 - Doc Searls
I'm glad you were hanging out. Okay, so next week we have Karen Sandler on. She'll be the guest and to talk about the declaration of digital autonomy I hope that's a tease, because I think that's really important. So, but she'll be on next week and until then I'm Dr Searle. This is Floss Weekly. We'll see you then.

1:06:34 - Rod Pyle
Hey, I'm Rod Pyle, editor-in-chief of Ad Astra Magazine, and each week I joined with my co-host to bring you, This Week in Space, the latest and greatest news from the Final Frontier. We talk to NASA, chief space scientists, engineers, educators and artists and sometimes we just shoot the breeze over what's hot and what's not in space books and TV. And we do it all for you, our fellow true believers. So, whether you're an armchair adventurer or waiting for your turn to grab a slot in Elon's Mars rocket, join us on This Week in Space and be part of the greatest adventure of all time.