PBS_2026_08_29

Well, it's that time of the week again. It's time for programming by Stealth and this is Tidbit 19B recorded August 29th, 2026. And I'm your host, Allison Sheridan. And of course, I'm joined by your host and tutor, Bart Bush. Hey, Bart. How you doing today? >> I am doing just fine. Um, it's been a while. >> It has a month. Now, I always feel compelled to explain what tidbits are. Tidbits aren't necessarily small. They're they're just not in a theme of like we're going to do Git now for several weeks in a row or we're doing JavaScript or we're doing Shimoa, whatever we're doing. They are they aren't in the big theme. They're tidbits on the side. But this one is actually going to end up being a three-part tidbit. >> Yeah. And we're not the first because Helma ended up going three parts when she was going to quickly explain Docker to us. >> Yeah. Yeah, that was great. So, >> yes, >> tidbit is a it's it's more of a feeling really than a descriptive word. >> Well, I I would say what makes it a tidbit is that it's adjacent to the series because we're not really learning a new skill. This is an application of what we've done, but it's not really a lesson like the rest of programming by self. >> Yeah. >> So that's why it's a tidbit. >> There we go. >> So when we last left our heroes, we had explained that because of other podcasting stuff, I do a lot of it with you. I need to make links all the time, links to articles on web pages. And so I need to show the headline of the article I'm linking to and where it came from. Because to me, a headline without a source is meaningless because every publication has a point of view and without that context, you don't know what to make of the headline, >> right? And that manual work is a lot of work when you have 200 links in say a let's talk apple set of show notes. So I needed to automate it. And we sort of explained that at the start of the part A of this tidbit and then we took a diversion into the fact that when you're writing JavaScript to solve any problem, the chances that you can solve all the problem with your own code, go to zero as the complexity of the problem to be solved increases beyond simple. And so you end up at the very least with stuff like jQuery or something or Bootstrap or something, but you end up building up dependencies and in the olden days you could just install whatever you liked and it was fine. And that's not today. So then we spent pretty much all of that first part, part A, talking about the practicalities of trying to stay safe while dealing with dependencies cuz you need them, but you also need to be careful with them. So that was a whole discussion of bar, >> right? >> And now I actually had a specific problem to solve and I had written it in plain old JavaScript. Not just And I say old, I mean old because I wrote it nine years ago. >> Wow. >> Yeah. Pree6 using the prototype syntax that we just we just don't do anymore. And I even went back in time in all of our old show notes after we recorded part A and put a link sort of a little warning telling people, hey, this is legacy syntax. Jump ahead to like PBS 128 or something. And yeah, I need to rewrite all the code. And we didn't really go any further than that last time. So this time it's all about that. I need to rewrite all the code. >> Now the name of your little your little code is called linkifier. >> Yeah. So it was a script that I called linkifier.js and that was one giant piece of JavaScript. like oh my god was it long which meant that managing it was a giant pain in the backside because scrolling up scrolling down scrolling up scrolling down it's all in one big file because that's how you wrote things in the old versions of JavaScript so one of the things I've wanted to do for a long time is to modernize the JavaScript but that meant rewriting it which meant remembering how it all worked from 9 years ago And that was a lot of work and I wouldn't have done it until those annoying AI people started to ruin things and my downloads weren't working anymore. So the script worked by downloading the web page at the URL, finding the title and then using that to create the link. If you can't download the page, what do you do? Well, my old script crashed. Just it just did nothing. It didn't handle it gracefully. It just crashed. So I had to rewrite it anyway to do something when it went wrong. So I modernized the code. So we went from one giant big script into a modern ES6 module, a single module where I could contain all the logic together. So the brains of Linkifier is now a single module. And I'm going to give a slight preview to part C because having solved the problem doesn't give me an interface. There's no UI yet. There is a black box that is this module where you can shove a URL in one side and a link will fall out the other side. >> But that's making useful >> precisely. So either I could write a little wrapper script. It would be a much much shorter script, but it still be a little script and I'd have all the same problems that I had with the script cuz I could pipe PB paste to the script and then the output of the script into PB copy and then I have my annoying trailing new line character again. I could fuff about like that, but what I actually did was create a JavaScript CLI app. And that's going to be part C because the CLI apps code doesn't have to do any of the work of converting links. The only thing it has to do is provide a nice human friendly interface. But to a good CLI, there's quite a bit to it. So that's actually enough of a discussion to be worthy of a part C. >> Yeah. I'm not sure we've actually gone through that, right? >> No, we have not. So, I've been meaning to find an excuse to wet people's appetite because we we have all the skills to to to basically learn how to do it, but we haven't looked at it. So, that's why I really wanted to do this tidbit. To be honest, that's part C is the real reason I wanted to do the tidbit. >> Well, you can't get to part C without the other bits. You got to tell a story. So today I want to talk about the design of the updated linkifier. So we have a complex problem to solve and my brain has for decades worked in an object-oriented way. My first programming language was Java which is an object-oriented language and that's just how I think about things. A problem is made up of objects which interact with each other. So what are the moving parts to the problem to be solved? And in this case when I thought about it I came up with three classes for three different types of object. So the first one I created was page data. This represents what it is I pull out of those web pages sitting at those URLs. So that's things like the URL, the content of the title tag at the top of the HTML page. I also make two arrays of all the H1 tags and all the H2 tags because they're the kind of places that headlines tend to go. I also make a dictionary of the metadata. So those uh SEO headers we talked about a few weeks ago, few months ago in the main series just to really go wibbly wobbly timy wimy. And so they're in a dictionary. So that gives us a single object to encapsulate whatever it is we extract from the actual web page at the URL. To construct a link, you need the pieces that make up a link. So I made another class called link data to capture what it is that a link is made of. A URL, some text, and optionally a description which would appear as hover text usually. >> So that's the three possible pieces of a link. So that's a second object. And then the third object is a template for turning the parts of a link into a link because maybe you want HTML sometimes, maybe you want markdown, sometimes you may have different ways you want to do it. So I figured we should have a template that we can just apply. And the template just has two pieces. a template string, which I chose to do in mustache because that's just my favorite templating language for JavaScript. And also, I decided to add the ability to create filters. So, this is just an optional set of functions that can be used to intelligently transform the the you know, the pieces that make up the link. So things like say stripping those annoying query parameters off the end of URLs can be done as a filter you know >> everything after that question mark. >> Yeah because there's a bunch of them are done for tracking purposes and I really want to strip those UTM equals I think they all start with. So I want to strip those out. So >> often often annoyance just when you're sending somebody like an Amazon link and all of a sudden it's three and a half pages long. H yeah, Amazon's a perfect example because if you take everything after the question mark off, it works fine. >> Yeah. >> All the rest of it is tracking >> including the link. >> Well, you can leave it if you like. It won't make it won't break anything. >> Okay. I usually take it off. >> It's cleaner, >> but if you leave it on, it won't break. >> Okay. So those three classes actually have changed very little in the last decade. I mean I've given you sort of a small handful of their properties and if you actually look at the code they have a few extra little fiddly bits to make them go but that is actually the essence of them and has been since the start and that hasn't changed very much. So my initial one giant big file started by defining these three classes and then it went and used those three classes to implement my algorithm which is a three-step process. Step one, download the HTML and pull out the information to create a page data object. So now we have the information about the page. Then step two is apply some sort of appropriate logic to find the right piece of that page data to build the information that we're going to use for our link. So we're going to transform a page data object into a link data object. And step three is turn the pieces of a link into an actual link. In other words, we're going to use a we're going to transform a link data object into a link with a link template. So, HTML page to page data, page data to link data, link data plus template to output, which is >> that sounds pretty elegant. It is a a sensible design which means that the actual functions for doing the work have a sensible structure and even after a decade of not really looking at the code I was actually able to find my way very quickly because you have three classes and a three-step process the code is logically structured. There's just a lot of it all in one file, which is >> okay. Just so much >> optimal. >> Too much to scroll up and down. Let's be honest. >> So when you move to the ES6 universe, you end up thinking of modules. And the first thing you can do with any module is that it you can take each of these classes and put them in their own file inside the module. So immediately I could take the three classes out of the top of my great big long file and put them into three nice separate files. So that's already a nice start. But with a module, where do we put the logic bit? Where does the three-step process go? Well, the answer is you actually need like a master class to tie it all together. And so I created a fourth class that I called linkifier since I've been using that phrase all along. And so the linkifier class makes use of the other three classes to actually implement the three-step process. So that means of all of the classes, it's probably the most complex one, but it's also like the main one which so that's actually what gets exposed by the module. So if a module has a default export when you say import linkifier, you get back one object. What do you get back? Right? You get back whatever the default export of the module is. And so I decided to make my default export an instance of the class linkifier. So this master class is what you get when you say import the module. >> Oh, okay. you get the whole class, >> you get an instance of the class. You get an object built >> using the default constructor for the class. >> Okay? And so that class is actually the most complex of them all because it contains a lot more properties. Some of them are property properties like piece of data. Some of them are functions. Some of them are static. Some of them are instance. So it really is a much bigger thing. So I'm not going to go through everything in that class, but I am going to highlight the most important things that class gives you because when you have an instance of that class, that's what you're working with. So the first thing is this module has a bunch of default behaviors. And so instead of those being hidden away, I chose to expose them through a dictionary called linkifier.defaults. So if you want to know what the default template is, you'll find it in the defaults dictionary. If you want to know basically any default, you'll find it in the defaults dictionary. That's also what the default constructor calls. Instead of it being hardcoded values inside the constructor, the constructor just uses the defaults dictionary. So, it's just easier for everyone because you it's just more obvious. >> Right. Right. >> There's there's also a dictionary that just contains a whole bunch of useful functions. And these are just functions I created that I know I'm going to need over and over again. And I just publish them because they're useful. And so, linkifier.utilities is just a dictionary with lots of useful little functions. When you say you publish them, do you mean that's like a separate thing you could find on GitHub or do you mean it's part of this module? >> It's part of this module. So the this this module the linkifier class has a dictionary named defaults that has these values in it and it has a dictionary named utilities that has all these functions in it. >> Okay. It also has a suite of functions for managing the configuration. So defining templates and all those kind of things. It has a suite of functions for managing the logic we're going to use to do that step two. Turn page data into link data. Well, we set it as one thing. There's actually quite a bit going on there as we're about to discover. And so the logic for doing that has to be configurable. So there's a suite of functions to make that possible. There's also a suite of functions for managing the list of available templates. So if we want to be able to make markdown and HTML and stuff, we need to be able to manage those and you may want to create your own entirely different to what comes by default. So there's functions for that. And then there's functions for doing step one, step two, step three of the three-step process. So that you can basically say okay I want to take this URL give me a link data object okay great I'll take this link data object turn it into sorry take this page data object and turn it into a link data object etc right each step there's a function >> it's functions all >> oh yeah lots of functions and then there's the most important function of all generate link which takes as an argument a URL and it returns a string that is your link So that is the most important function of everything is just the generate link function in the linkifier class that does all the work using everything else >> takes all the credit is the way I would put it. All those other functions they're doing the heavy lifting. It's out. It's just the showy one that gets to go here I made you this. >> Yeah. And it only needs one argument which is kind of the beauty of this process that everything else has been encapsulated in all these other classes. And so ultimately when you want to make a link you just need to call generate link with one argument that is a URL. So I do kind of like that. Okay. Now I've very intentionally glossed over the complexities of these three steps. There are three conceptually simple steps and it's important to think of them as separate steps. Otherwise, you could never solve this problem. But step two, God, that's where the devil is in the detail in that bloody step two. That really is the tricky bit. >> Just just go get that headline part. Come on. How hard can that be? >> The headline. >> It's right there. You can see it. >> But because you might imagine that step one could be tricky, right? Download a web page and turn it into this page data object. But actually, there's modules for that. So there's actually a module that lets you turn a web page into something that looks exactly like jQuery without there ever being a browser involved. You just give it a string of HTML and it gives you a dollar object and you can just query it like jQuery. So all of the jQuery syntax we can use just in this module without there ever being a browser involved. It's great. So that makes it really easy to pull out the title tag and to pull out all the H1s. It's we just use jQuery syntax. So that bit's easy to make the page data object. But yeah, those headlines. Oh boy. So somewhere in every web page that is an article, we can see with our eyes the title, the headline of the article. It has to be in the HTML. There is no way for it to exist on the web page and not be in the HTML. And you might assume that every web page would be similar enough that you could apply a simple piece of logic and that simple piece of logic would just work. And so when I first started writing this script, I naively assumed that the answer was very straightforward. It would be in the title tag, in the head tag, right at the top of the page. >> Sure. >> And that's partially true. The correct information is almost always in thereish. Can we put this in the Yes. And yeah, so it is sometimes it's actually truncated in there, which is uber annoying. That's the most useless they can possibly be is to have it up there but truncated. It just ends in like an ellipsus when it gets too long. So that's definitely useful on the websites that do that. But almost every web page either prefixes the website name and some sort of a special character like a pipe or something or appends it to the end. So just using the title tag is always going to give you the headline with baggage >> or most of the headline. >> I actually have baggage >> a teeny tiny itty bitty little script that just copies my headline and the first thing it has the main thing it has to do is scrape off pipe pod feed podcast. I didn't exactly >> cause it to do that. That's something that WordPress does for me. It creates that little title. So mine works, but I'm solving one problem, one specific instant, looks exactly like this every single time. >> Yeah. If every web page did it exactly the same way, we'd have a chance here. >> But that's not how it works. >> Well, I remember asking going, Bart, geez, I've got like a four-line script that does mine. I don't know what your problem is. >> And if I maybe if I only linked to RS Technica, then it would be easy. >> Sure. or if you only did one link a week >> or that you're doing >> Yeah. >> practically hundreds. Yeah. >> Yeah. Yeah. So my second thought, having realized the title tag was not my friend, was well, it's best practice to put the main title of a page in the first H1 tag in the page because that's better for SEO and most websites like good SEO. So I thought maybe that would work. But that proves to have some annoying problems as well. because that best practice was true before HTML 5, but HTML 5 isn't very new anymore. >> So, now that we have semantic markup like heading uh not body, that should say main, not body in the show notes. My bad. >> Article section, all of these regions now have their own tag. So now the rule is the first H1 tag within the appropriate region is still best practice. Well, that could be the fifth H1 tag on the page on one website and the second H1 tag on the page on another website. >> Oh, really? >> And then >> yeah, because it's just do they do they have the markup for their sidebar before or after the markup for the middle of the page? Where do they put the markup for the heading? I mean the amount of permutations of where that headline for the article could be is quite large. And then you have the fact that some websites choose to give the headline of the article second billing. Their own brand gets first billing in an H1 tag and the article title only gets second billing in an H2 tag. But they could be like the eighth H2 tag. The good goodness only knows when you have H2 tags because the sidebar can be full of them. So when you fall into H2 tags, all bets are off. >> And they've got those little banner things across the top with people have cards that show all the other articles you should be reading instead of whatever it is you were trying to read on this page. >> Exactly. Exactly. And then some websites in order to try game the search engines actually have hidden text in their H1 tags that when you looked at the HTML code, they have about 50 buzzwords after the title of the post, but the script sees those. So I can't I can't remember which website it was, but it might have been the old one of them that went bankrupt. was it um macarel.co.uk or.com one of them went away and they used to have like 50 different tags in the headline only in invisible text and so >> I completely believe it was them because that was the the only page I ever went to that made me think I really will do an ad blocker cuz I literally cannot find your content. You're you're killing me. Actually, no. The other one was wired. the two of those I just stopped reading Wired and but I kept getting links to Mac World and I'm kind of I'm sorry for the people who lost their jobs but man that was awful towards the end so I'm not at all surprised that you included other garbage in their titles. >> Yeah and to be honest I actually stopped using them for Let's Talk Apple because their articles became as garbage their headlines. >> So they really they really really started scraping the barrel anyway. And then finally, some people are so bad at web design that their article titles are just in paragraphs with a bold tag. >> Oh jeez. Seriously? Okay. Never link to them. They don't deserve it. >> So clearly that's not going to work, >> right? So, I did briefly think, well, what if I try the first H1 tag and then somehow with a magic regular expression figure out that that isn't going to work and then try the title tag and then try something else and then try something else. Could I write one big algorithm with a whole bunch of ifel statements that could somehow find the right permutation on every website? Five minutes told me that was a terrible idea. So, no, I couldn't find a pattern. No. >> Wow. >> So that left only one obvious solution. Basically scale up what you do per site logic. >> What? >> So on RS Technica do this, >> right? Your skepticism is perfect because my first thought was what am I doing to myself? >> Yeah. >> But then I looked up my own show notes. I would say 80% of my links come from about 10 websites, 90% of my links from about 15, and almost all of my links from maybe 30 sites in total. Like try to list 30 websites. >> Yeah. Well, now that's one of the things where you always say this is a known good website. I remember I recently uh was unable to find a really important article on any of the big websites. didn't make the Verge. Arsectica didn't make any of those. And you're like, uh, those are those are sites I've never heard of. And I think part of it is like, I got to do those by hand is a little part of it. But you do all the security bit stuff and let's talk Apple and that those are two very different sets of links. That's still 90% and 15. >> Yeah. If you look at security bits, it's probably only five sites I trust enough to link to regularly. It's Bleeping Computer, The Hacker News, Cyber Insider, and strangely enough, the Mac Observer do really good coverage of stuff we talk about on security bits. But actually, the majority of the links are only five or six sites that I trust enough because there's so much hyperbolic >> scare you nonsense for security bits that there's fewer actual sources I trust than for the Apple news. >> Oh, that's really interesting. So that means it wasn't it made more sense to do this tailored approach, >> huh? >> Yeah. Okay. I surprised myself when I started to look at my own show notes. I was like, "Oh, no. This is quite doable." And the great thing is these websites, there's actually a few very standard patterns. So the reason there's a utilities class is cuz one of the most common patterns is the title tag with a regular expression to pull out a string. And so one of the utility functions takes basically the the the title and a string as the second argument and it just pulls the string out of the other string and returns it. So RS Technica use pipe rs technica you have a pipe based one. So that just that one utility function with a different string covers 10 15 sites. >> Yeah. So my actual code for those 10 15 sites is just oneline functions. Call my utility function with this argument. Call my utility function with this argument. So a lot of my customization is one line a oneline JavaScript function and I write it as a fat arrow function so that it literally is one line. >> It sounds like that would actually make it >> an easier way to understand your code because those examples are so obvious. >> True. you know, for some somebody like me to query your code and go, "Oh, I see what he's doing." Because if I look at ours Technica, it does this. And actually, I should I haven't got in the show notes, but I'm going to mention it because our listeners are cool. Um, one of the things that is in the uh GitHub repository for Linkifier is an example config file called real world example. >> That's very real word. That's my actual config that I use every day. And as I update my config, I actually push the changes into that repo because I actually use realworld example.comconfig. >> And so if you're curious how many customizations have I made, if you open that file, you can count them and that is exactly how many I've made. >> Cool. And most of them are oneliners. Big exceptions are when I link to um is it Overcast? When I link to Overcast for podcast, that takes a lot of faffing about because the podcast name has to be pulled out from the episode name and they all have to be jumbled about and XKCD takes a lot of code because I actually pull out the image and make a pretty link with an image and everything in it, but most of them are oneliners. So this per site extraction proved to be the answer. And so that's what I implemented 10 years ago and it's still what I'm doing today. So the ES6 module takes exactly that approach. So extraction logic is just a really fancy way of saying a function. The input to the function is a page data object and the function has to return a link data object. And that's every one of these extractor functions is just one argument, one return value. Page data to link data. And now we have to connect them to the different websites. So I need something for the Mac Observer and something for RS Technica and something for Bleeping Computer. Well, what differentiates Mac stories from the Mac Observer? It's their domain name. One of them's at what did I say? Macstories do well probably com. And the Mac Observer is definitely at macobserver.com. And I'm a CIS admin most of my life. Not anymore. No, I can't say I am. I was a CIS admin. Now I'm a cyber security specialist. Anyway, I was also the DNS admin. So I think in terms of DNS and actually using DNS names has some really nice advantages because there are two features of DNS names that really add power to Linkifier. >> Right before you tell them >> to stop everybody from screaming, it's maxto.net. I just checked max.com is available for purchase though. >> I should gift that to Frederrico. >> I bet it costs a fortune. >> Not available. >> You're right. Is available in two ways. >> Okay. So, the first thing is DNS names are hierarchical. So the period or the dot as we say it in URLs is a separator between the different parts of a domain name and they're they're they're hierarchical. So www.podfey.com is a subdomain of podfeat.com is a subdomain of the top level domain com. >> I never thought of www as a subdomain of podfeat.com but I guess it is. >> Yeah it is >> before the dot. So yeah, precisely. So basically everything on the left is a subdomain of everything on the right. And then something almost no one knows because the internet has done its best to hide it from us is that every URL is actually a subdomain of the root domain which is denoted by a final trailing dot. >> Really? You don't Yeah, you don't see this because the browser is hided from us, but your domain name is not www.podfeet.com. It's actually www.podfeet.com and I'm pbs.bofessor.net. >> Wow. >> And the browsers put them in silently. So when they issue a DNS query, they pop them in and when they get the answer back, they just take it out. And what you see in the address bar is the human friendly version without that silly dot. And when you call most DNS commands and you leave out the dot, they just put it in for you and then they do the DNS work, but it's actually there. And there are a few terminal commands that show it, but DNS admins see the final dot and know what it means, and everyone else assumes it's punctuation. They assume there is just a period at the end of the sentence in the output of the terminal command. They don't associate it as being part of the domain name. But there are actually 24 DNS servers for the dot domain because those DNS servers know who is responsible for com and who is responsible for net and who is the responsible for org. that's actually handled by the root DNS servers and they are the DNS servers for dot wonderful piece of trivia but it comes in bloody useful. >> Oh, so we need to actually know this. >> Well, it I use it as a feature to make link work a little bit more better. Terrible grammar, but you know what I mean. Um, the other thing I should say is that from the very very first time I wrote this script, I always in my head refer to these extraction functions as transformers because they transform a page data object into a link data object. And so I leaned into that naming convention when writing the ES6 module. So in the original script, I had a dictionary where I put the domain names in as the keys and the values were my functions. And that's fine because I was writing a script for me. And so I was putting that trailing dot in at the end for reasons I'll explain in a minute. And it was it was fine because I'm a DNS person and the script was only for me. But now that I'm writing a module and publishing it to the world, expecting having to explain to everyone that DNS names have a secret dot you can't see, that's not something I want to do. So instead of me give making a dictionary and exposing the dictionary and making the user of the module edit that dictionary, I still have a dictionary, but it's a private variable. And what I have instead are functions that automatically add and remove the final period the same way Firefox and Safari do. And it means that the users just call functions with sensible names. And they have no idea what's happening under the hood. And that's how modules are supposed to work, right? They're supposed to be black boxes that you shouldn't have to explain life, the universe, and everything to the user of your module. So we have a function called register transformer that takes two arguments. The domain name you want to apply the transformer to and then the transformer itself which is a function. Get transformer for domain takes one argument a domain name and it will return to you a function. And then if you want to see everything I do actually let you see everything. There is domain true transformer mappings is available as a read only copy of the dictionary for anyone who wants to see them all but that's just for the nerds. So really you register transformer and get transformer allows you to do everything you need to do. >> I would have thought that a user of your module would need to be able to to edit the underlying dictionary for their own use. >> Right. Register transformer. That's what register transformer does. But the domain to transformer mappings, isn't that the one that's that's taking the pipe off of podfeed.com? Okay, that no, but reg. Okay, so under the hood there is a dictionary. Register transformer updates that dictionary for you. >> Oh, >> automatically. >> Okay, >> so the problem you have is I need to add my regular expression for pot.com, >> right? You would say register transformer first argument podfeat.com second argument your function that calls the regular expression >> and that's editing the readonly copy of the underlying dictionary okay >> yeah exactly so why do I say this whole subdomain and secret hidden top level domain matter well I implement fallback in my mappings so So if you add a mapping for podfey.com, that mapping will apply to podfey.com and all of its subdomains unless there's a more specific mapping on the subdomain. So some people might link to rstechnica without the www and some people might link to it with the www. If I just register a transformer for rstechnica.com, it will cover both. There still exist a few really weird websites where they have a mobile website which is usually horrible and a normal website which means I may actually need two different transformers for that website. And I can handle that by having one for say silly.com and another one for m.sillybsite.com sillybsite.com because M is what's usually used for the mobile sites. >> Oh, I forgot all about those that >> those M dots. Yeah. >> Yeah. You see that? You're like, when did you write this? When did you do this? >> Yeah. >> So, before responsive design. Huh. >> Yeah. So as an example of how this fallback works, imagine that we have a very simple universe where all sites have acceptable titles in the first H1 tag or in the title tag except for one website called sumsite.com. This site has a legacy mobile site at m.sumsite.com sumsite.com and a modern website available both at sumsite.com and www.somsite.com. In this very simple universe, we need three transformer functions. One to handle every website on planet earth apart from this one website. This is not our real universe, but bear with me. One to handle the mobile site and one to handle the modern site for sumsite.com. And how does that work then? Well, we we put a mapping on some site.com dot another one on m.site.com dot and the last one for every other site in the universe gets attached to dot which means it's the default transformer for every single web page on the internet. And actually the linkifier module does implement a default transformer so that you fall back to something and that default transformer falls back to if there's an H1 tag on the page use it else use the title tag. There's always a title tag >> and that actually is what the module does as its default last stitch effort because at least then you get a link that has the right information. You may have to clean it, but at least you get something. >> Yeah. >> And then I have my actual mappings >> cuz otherwise the option would be fall over in a heap. >> Precisely. Precisely. So to see how this works, imagine we have the URL sumsite.com/bigstory1. The first thing that will happen is when we pass this URL in, it will say is there a mapping for sumsite.com? There is. I shall use it. So that's a very simple algorithm >> meaning it's in your dictionary. It's in your list of 15 or 20 or 30 sites. >> Yeah. So in our imaginary universe, there are three entries in the dictionary. M.Sight.com and sumsite.com. So this first URL is sumsite.com. Do I have a mapping for sumsite.com? Yes, I do. Use it. Our second example is www.somsite.com. sumsite.com/bigstory2. Well, here we say, do we have a mapping for www.somsite.com? No, we do not. We only have those three mappings above. Do we have a mapping for someight.com? Oh, yes, we do. Use that. >> Okay. Then we get um www.anothersight.net/bigstory. net/bigstory. So, is there a mapping for www.anothersight.net? Nope. Try the parent domain. Is there a mapping for another site.net? Nope. Is there a mapping for net? No. Well, there never will be, but has to try. >> Mhm. >> It's just take off a dot every time. Is there a mapping for dot? Yes, there is. Fall back to our default. apply the default mapping. Hey presto. And then finally, if we have the mobile site, then we would say, do we have a mapping for m.somesite.com? And it would be yes, we do. So that would be a different one to what gets applied to www.sumsite.com. So everything works in our simplified universe with just three functions. There are more than three in my actual real world config, but there are far less than 100. I think it's 20 to 30, that sort of ballpark. A few tens and that covers everything >> that I need. So that's not too bad, you know. >> Yeah. >> So that piece of the logic I was very very pleased with myself a decade ago when I came up with this and that has worked flawlessly for the last decade. But then we get to the cause of all of my recent problems, the bot blocking. Because the AI bots are hoovering up the whole internet and making people cranky, my little script is getting caught in the crossfire. No one cares about my little script, but it's a bot, so it's getting blocked. And I spent a lot of time trying to solve this problem. And for a long time, I thought the only thing I could do that would work reliably is to throw away my code and start over and integrate some new code straight into a real browser because the real browsers are let through by design. So instead of trying to somehow imitate Firefox or Safari, somehow become Firefox or Safari. And I know hypothetic I know for a fact because of certain things I've had to do with my work at that there are APIs for driving the browsers. So you can call an API to tell Firefox to go to a URL and it would be a lot of work that I've never used. It's technology I've never used before. I know it exists and I'm sure with infinite time I could figure it out. I believe headless Chrome is the one everyone uses, but again, I didn't want to do that. >> I've touched headless Chrome and I was never happy with it. But I wonder whether the Isn't there a textonly browser out there that still exists that was designed 100 years ago >> link >> ly link? >> Yeah. >> Well, I hadn't thought of that. Yeah, I could end up with a I could rewrite it in bash and you Yeah, that's a that's a third avenue I hadn't thought of. I will grant you that. That sounds fun. >> Yeah. The other option that I did think of was rewrite all of my code as a Firefox plugin. >> H cuz Firefox can open web pages just fine. Like they're not blocking Firefox, >> but again, I've never written a Firefox plug-in. I could learn, but that's a big project. So, yeah. And then I noticed something. Every URL of every website I actually use has the headline in the URL sort of. They're called URL slugs. So, there's an actual headline from the Mac Observer from a few weeks ago in the show notes. www.mmacobserver.com/news/ iphone-8-promax-cood and heavier due to bigger battery with dashes for every space that actually clearly contains the headline, "The iPhone Pro Max could be thicker and heavier due to bigger battery." It's It's there. It's mushed up a bit, but it's there. So, there's an algorithm for sluggifying URLs. Well, what if you reverse that algorithm? Can you deslugify a URL? Turns out there are modules for that because there's modules for everything. So, okay, this is an avenue that's plausible. But even that didn't prove to be quite as straightforward as I'd hoped. So there is information lost in sluggifying text. For a start, all of the capitalization is gone in that URL, >> right? iPhone 18 Pro Max. It's all lowercase. >> So, okay. Well, there's a convention that just about everyone uses in media called title case where you capitalize the first letter of every word to make your headlines. So, if I deslogify and title case, then don't I get good headlines? >> Except for the little words. >> The little words you don't do that. >> Okay. So, you've hit on a secondary problem. There's actually a bigger problem before then. So, slogification is lossy in three ways, right? The character casing gets thrown away. The title case algorithm can fix a lot of that. So, okay, that's something. Old punctuation evaporates. Everything becomes a space. A colon, it's a it's a dash, which you reverse slugify to a space. Spaces become dashes become spaces. Colons become dashes become spaces. Commas become dashes become spaces. See what happens here. Diiocritics. Now in English we don't have a huge amount of diiocritics. So these are the little twiddly bits on letters. So your axon on the E and your sadia on the C. They're diiocritics. And when you slugify they evaporate. So a ax on becomes a and for cliche for example that means that you don't really get back cliche you get back amateur >> yeah exactly the amateur version and I'm the kind of nitpicky person who wants the yaxon >> um so >> can I interrupt real quick this is a fallback mechanism we're talking about here right >> it is >> you don't you don't have all of your big sites aren't blocking you from pulling the title from >> logical places. >> It got as bad as 50%. Now it's gotten a little bit better. They seem to have backed off when they're blocking. Somehow my script isn't falling into their algorithms as much. So now it's about a quarter. >> Okay. >> That's still a lot of links. >> Mhm. >> So this fall back is still quite important. So the big problems I was seeing was the punctuation currency symbols. Apple News, they're always talking about billions of dollars. Like billions of dollars are everywhere in Apple News. >> All numbers get boogered as well. And they don't get reversibly boogered because 1,1 and 1.001 all become one space 001. How do you know if it's thousands or fractions? >> It'd be bad to be wrong on something like that. >> A teeny bit. Yeah. Um, acronyms get destroyed. NASA becomes NASA with a capital N and lowercase ASA. >> And weirdly capitalized words which Apple has an absolute fetish for get destroyed as well. iPads, iPods, iPhones, m Mac OS, all of them. >> Yeah. And of course, cliche disappears. That's pretty much the only accented word that shows up in many of my headlines, but it does get, you know, it does get destroyed. So, the first of these three problems are actually just not solvable. So, they are always going to need a manual fix and they're not the world's most difficult manual fixes and so I tolerate them. This is a fallback mechanism. If it's 90% there, hey, it'll do. So you can put in colas and commas and uh currency and fix the numbers. >> Yeah, exactly. But actually, I could manually fix everything. But the second three are actually 90% solvable. Not 100%, but still pretty darn well solvable automatically because acronyms, the same ones tend to show up over and over again in my show notes. NSA, uh, CIA for security bits, and the weirdly capitalized words, they're mostly Apple ones, let's face it. So, they they're only a handful of those. >> So, you can just hand type them in >> like to a dictionary or something. >> Yeah, exactly. So at the moment what I have is a dictionary of weirdly capitalized words and what I do is I apply a case insensitive regular expression and replace the matches with the case with the actual correctly cased value from the list. >> So I have a list of all of these words which I save as a JavaScript set. I don't. Did we ever talk about sets in the main series? >> I didn't recognize it when you referred to it. >> Yeah. So, they are they're a data type in standard JavaScript just like arrays and strings and stuff. And they have functions just like arrays and strings and stuff, but they're subtly different to arrays because a set is inherently unique. the set 1 2 3 4 5 if you add five is still the set 1 2 3 4 5. And so for these kind of lists of funny capitalizations, having automatic dduplication by the very nature of the data structure, that's a really big advantage. >> I'm not sure I followed that. How is 1 2 3 4 5 + 5 1 2 3 4 5 in set theory. If you add something to a set that's already in the set, the set doesn't change. An array will just duplicate it. >> Okay. >> So, if I give you a list, >> how does that work in this in this example where you're adding NASA? >> Okay. If I add NASA a second time because I've lost track of the five million acronyms. >> Oh, okay. I would say it would. Yeah. If it's an array, you're always kind of you need to check the array. You'd have a function for adding a new word and you'd have to check your array. But when you use a set, JavaScript just goes, okay, the set waffles, pancakes, oatmeal plus waffles is still the set. Waffles, pancakes, oatmeals. >> I want this in lots of places, Bart. I use the Microsoft To-Do app. For some reason, Steve and I use it for packing certain things. and you'll see crossstitch five times in the same list because I keep going, "Oh, I don't want to forget my cross stitch." But it's not sorted or anything like that. So, it's a really bad way to do a packing list. >> Yeah. And if you're into maths, >> the set class, those things like union and all of those functions you learned about in set theory in school. >> Cool. Cool. >> So, it is actually set in the mathematical sense, but it's a really good way to store lists that you do not want duplication in. >> So, I'm not going to use it everywhere. Having discovered it for this, I I become a big fan of sets. So, a nice excuse to throw them in. >> I'm glad I asked for more detail on that because I didn't follow it at first, now I totally get it. Excellent. >> Yeah. And so, what this means is that with just my set of weirdly capitalized words, all of my acronyms can be easily handled because NASA can be handled just the same as iPod and iPad. And I am the next item on my to-do list and there's actually a GitHub um issue open to myself is I'm going to add a second set of regular expressions so that I can handle things like so called with the dash between it cuz that has to be a regular expression that maps so space called to so dash called >> whereas with the funny capitalized words it's just single strings I need to add to my set but again I can do you know that's on the feature list So with a surprisingly not horrifically long set, I have pretty much got all of the acronyms that show up regularly in security bits and let's talk Apple taken care of now. So all of those happen automatically. So my fallback is actually pretty good apart from the month when it's Apple earnings call. That month I have a lot of corrections to do because there's billions everywhere. But you know most of the time it works out fine. Um then we come to the title casing because that too has some nuance you've already mentioned. So I had naively assumed that because title case is so ubiquitous across the internet there would be like a standard which defines the set of so-called small words. Those words that don't get their first letter capitalized. the an a to right all of these little joiner words they don't get capitalized and I just assumed that at least in the English- speakaking world there will be a standard set of small words no there isn't most people agree on most of them but there's edge cases everywhere and so I've actually ended up I used a really really nice module for doing the title casing that has no dependencies and it's a nice clean module and it comes with what it thinks is the right answer for the set of small words and it doesn't treat it and its as small words and I think capital it t and capital it ts looks ridiculous. So what I what I do in my code is I take their list as the default add my own list of extra words to create a set of the actual words to use which I make editable as linkifier small words. >> Okay. So if I like it's with a capital I could change it for me but you don't have to do it for you. Okay. >> Exactly. So you could call the delete function from the set and pull it out of the set. >> I feel a little bit better knowing that it's not u it isn't really defined and agreed upon because I sit there arguing with myself sometimes. Apparently that's why. >> Yeah. And so the official advice is to have a style guide for your organization. And in my case, the style guide has been cemented into the code for linkifier. I am 100% consistent because I'm not doingistent my script. >> Yeah. Exactly. Exactly. >> I like it. >> Okay. So, that is how linkifier works. And the last thing I want to end on here is to tie us back to part A, choosing my dependencies. So, I did not do everything here from scratch. I've alluded quite a few times to using different modules. So let's go through to implement this three classes and then a fourth master class and a three-step process. What dependencies did I end up needing? How many of them? Well, the answer is six, which isn't a huge amount, but it's not zero. So what did I need? The first one I needed is an amazing module called Cheerio. It's a browserless jQuery. >> jQuery without a browser. Oh, it's a so good. You just >> Wait a minute. jQuery requires a browser. >> Well, if you Yeah, you can't use jQuery in Node.js. jQuery uses the DOM. The DOM is part of the browser. >> Oh, yeah. Yeah. Yeah. Okay. >> So, Cheerio builds its own DOM. So, Cheerio takes as an argument a string of HTML. It builds a DOM and lets you query it with the dollar syntax and it intentionally duplicates jQuery syntax. So, it's designed for jQuery programmers who want to be on the terminal, who want to be out of the browser. >> By the way, you're referring to these as modules, but I'm used to hearing you refer to things like as as dictionaries. >> I'm sorry, as libraries. >> Okay. So, library is a language agnostic term. So you would describe reusable code in JavaScript as a library and reusable code in Pearl as a library and reusable code in Ruby as a library. But in JavaScript, you implement libraries with modules. And in Ruby, you implement libraries with gems. And what was the other example I gave? I don't remember. >> All of these that you're listing here are modules, which could also be called libraries, but they're called modules because you're speaking in JavaScript. Got it. Bing bing bing. Yeah. Library is like a super set. >> Oh yeah. Okay. >> So, Cheerio is the first one. Tens of millions of weekly downloads. Actively maintained, a very active GitHub repo, excellent documentation, and despite everything it does, 11 dependencies. >> Oh, wow. >> That's not too bad, you know. So, my my six has multiplied a bit, but still not too bad. mustache. We love mustache. Millions of downloads. Not very actively maintained, but there's no known vulnerabilities and I think it's not very actively maintained because it's kind of finished. >> It's kind of done. Yeah, that's okay. >> Yeah, it does what it does and it's fine. Um, it has a gate hub repo, but there's not much happening there. It also has zero dependencies, so yay. And the documentation's fine. It's not pretty, >> but it was good enough for us in programming by stealth, so it's fine. Node- fetch actually fetches the HTML for me. It again has not just millions of downloads, hundreds of millions. That was the first time I'd seen such a big number. So, it's like, well, I'm definitely not going out on a limb here. >> That's sick. >> I am with the crowd. Yeah. uh very actively maintained. Um GitHub repo, it's not as active as the module itself, but there's still stuff going on. Only three dependencies >> and superb documentation. >> Okay. >> Uh title case used for converting the string to title case. Millions of downloads. A lot of people want to do this. >> Um it appears to be still maintained. There's not much going on, but I also think it's one of those it's done. >> It's not solving that hard of a problem. >> It's just let's all use this one cuz somebody solved it. >> Exactly. The documentation's a bit poor, >> but it is sufficient. I was able to get what I needed and no dependencies. So again, I haven't expanded my dependency tree here. So far, only uh 14 extra dependencies. Not bad. Uh URL.js JS is an amazing URL parser. >> URI.js. >> Thank you. >> Okay. >> Thank you very much indeed. >> Yeah. So URI.js is an amazing parser. So it lets you take any URL and give me everything after the question mark and all that kind of stuff and strip it out and jiggle it all. >> This ages ago. >> We did. I'm a huge fan. Uh millions downloads. It was very actively maintained. It's not that active these days, but there's no vulnerabilities and it's possible I'm going to have to find an alternative to this sometime in the next 5 years. I get the feeling the developer has lost interest a bit, but for now it's still fine. >> You said it was a decade ago it was maintained. >> Yeah. >> Okay. >> But there's no vulnerabilities, so >> it's still okay. >> What dependencies? >> Zero. >> Okay. >> So I I've still had 14 extra. Not bad. And then URL-slug, which can actually it can make slugs, which I don't care about, but it also has a dlugify function. So that's what I care about. >> That's just fun to say >> again, isn't it? Hundreds of thousands of downloads, very actively maintained, zero dependencies. >> Wow. You got out with 14. I don't think I've ever downloaded anything like in in uh homebrew that didn't have 126 dependencies. But bear in mind, I chose these carefully because I could have very easily picked similar dependencies that would have achieved the same goal that came with like 50 extra dependencies each cuz they were programmed by lazy programmers. >> Okay. >> So, when you choose carefully, you can end up with 6 + 14. That's a total of 20 in my entire tree. That's not too bad. >> Now, what's interesting is that becomes an advantage of your module. Right now, somebody looks at it and they might say, "Well, there are maybe hundreds of millions of downloads a week, but look, it's got 14 dependencies. That's pretty good." >> Yeah, exactly. >> So, that brings us to the end of how the link module works. We have our three-step process, but we don't have an interface to it >> at all, which sets us up for part C. Hopefully, it's a cliffhanger. It's a cliffhanger. And I don't I believe our summer pseudo hiatus is at an end. So I think we're business as usual in two weeks. >> I think it's a miracle. I don't think we really did summer hiatus this time. We've gone a couple of months without it. And I think we we might not be on the perfect every two weeks cadence, but we're we've done far better this year. Yeah, I think we skipped at least two, maybe three, but that's not bad. >> Not in a row. >> Way worse. >> Yeah, considering how much I've been traveling. >> Oh, yeah. Jesus, you've been halfway around the globe. You must have a millionaire miles. >> Yeah, I know this is an evergreen segment, but this audience, the programming by stealth audience, doesn't know that uh a week ago we were sitting in the same room while Bart cooked us dinner. That's the only the second time since we've gotten to know each other that we've actually met in real life. So, what a what a treat >> it was. It was so good to have you guys over. I enjoyed every minute of those four days. Right. Well, at that we shall draw a temporary line under it. So, until next time, happy computing. >> I hope you enjoyed this episode of Chitchat Across the Pond. Did you notice there weren't any ads in the show? That's because this show is not adupported. It's supported by you. If you learned something or maybe you were just entertained, consider contributing to the Podfeed podcast. You can do that by going over to podfeed.com and look for the big red button that says support the show. When you click that button, you're going to find different ways to contribute. You can donate one time through the big donate button with a credit card or Apple Pay, or you can use PayPal. If you want to make a recurring contribution, click the Patreon button. Keep in mind I don't charge Patreon for chithat across the pond or program I buy stealth episodes just once a month for the no siliccast that keeps it simple. If you want to contact me for any reason you can email me at allisonpodfey.com and you can follow me on mastadon at podfey.com/mastadon. If you want to talk to other noilic ways you can do that in our slack group at podfey.com/slack. Thanks for listening and stay subscribed.