Should I use markdown for my site?

In this episode of Search Off the Record, we are talking about markdown, LLMs, and if you should convert your content into markdown or even LLMs TXT files, or if it's not that helpful maybe. Or is it? Or is it not? Find out in this episode of Search Off the Record. >> [music] [music] >> Hello and welcome to a new episode of Search Off the Record, the podcast coming from the Google Search Relations team. My name is Martin Splitt, and with me here is Ma Boss. Hello Mr. Moo. Hello John. How's it going? >> Hi Martin. Great to be here with you. >> Long time no see. I have a question for you because I have been asked this multiple times and I'm pretty sure I have the right answer, but I rather hear a little bit of your perspective as well on this. Should I convert my website into markdown so that LLMs have an easier time figuring it out? >> Wow. >> I I know. Okay. Okay. Hear me out. Hear me out. In my opinion, markdown is an intermediate format. Basically like HTML is an intermediate step to how a website looks like. It's a structured kind of text format. It's just annoyingly tedious to write and can like get the nesting wrong and all of that. And markdown is is doing a lot less to kind of get more or less the same structure into a text file like you can have headlines and you can have bullet point lists and you can have numbered lists and you can have tables and blah blah blah, and links and images even and whatever. But fundamentally, that's just it. And yes, it is easier to deal with markdown than it is to deal with HTML, but all the crawlers that exist today had to deal with HTML for the breadth of ingesting the web. Like the goal was to get the web and you can't just be like, oh, we're just not going to get any of the information that's out there. We're just going to get the markdown files. So, they had to already solve the problem of dealing with HTML. So, I don't think that's a problem that needs solving. >> I don't know. >> Okay. >> I don't know, Martin. You're You're like a smart guy. You've learned a lot about how search works, right? >> Fair. >> Okay. >> I'm not sure where this is going now. Now, I feel like I'm being investigated, okay? >> You said, you know, like the best practices for making good websites, right? >> I think so. I hope so. >> How do you write the content for your website? Do you write HTML or do you write markdown? >> Why am I having a job interview right now? Is Is like Is like something Okay. I do not write the HTML, actually. I do I Well, I have to admit that I actually use a static site generator and because of that I'm writing in markdown. And I've written my own static site generator back in the day to do that. >> Okay. >> Yeah, because I don't want to I don't want to write all these like angular brackets and all that kind of stuff. I just want to have like, here's a link, there's a headline. Moving on. >> Okay. >> It's It's easy. It's less typing. That's That's why. >> So, if people want to rank well in search like you do, which I I don't know. I didn't actually look. >> [laughter] >> I don't think I rank so well because I don't Okay. I think I'm I'm ranking reasonably well for what I care for. >> Oh. Oh. Well, then may- maybe they shouldn't be using markdown. >> Oh, you think that's why? Aha, interesting. No, I don't think that's that's the thing. >> Okay. I I guess maybe we should take a step back and briefly explain what this markdown is. Like where does it come from? What What is it? >> Okay. Okay. Okay. Okay. Yes. Actually, fun fact, I'm not sure where it comes from. I saw it first, I think, on GitHub where you can add a readme.md's or markdown file, and then it automatically is kind of like the home page of your repository on GitHub. But I I guess it's older. I don't I don't know. Have you looked at the history of it? >> I looked it up because I I don't know, preparing for this mock interview about markdown. >> Okay, pressure just went up again. Great, yeah. >> [laughter] >> Yeah. Actually, I asked an LLM. So, >> [laughter] >> this is awkward. So, it's like maybe I got something wrong. But anyway, I will assume it's kind of correct. Apparently, it was created 2004, so quite a long time ago. I imagine we have listeners who are younger than markdown, which is not surprising because we probably have listeners younger than HTML or JavaScript, which I don't even know when JavaScript was made. Long time ago. Uh it was created by John Gruber and Aaron Swartz. So, John Gruber, I think, is still active online. Aaron Swartz is a bit tragic, the whole history around it. He was one of the developers of RSS and Creative Commons, and one of the, I think, co-owners or co-founders of Reddit. So, long time ago. >> Wow. >> Maybe like the Reddit connection is why people assume this is good for AI because AI loves Reddit, so therefore anything Reddit does must be good, right? >> Must be good. Okay, uh maybe that's where it's coming from. >> Actually, Reddit apparently was written in Lisp originally. So, maybe people should be using Lisp to make their websites if they want to be like Reddit. >> Gosh, I've written Lisp during my university studies, and no. I mean, it's a pretty beautiful language, but no. No. No, thank you. >> Gosh, yeah. So, the whole markdown thing was created as a way to have a simple plain text kind of English readable style of creating content that's easy to convert into HTML and easy to convert back from HTML. So, it's is basically it's like if you assume HTML exists, how it could you make it so that it's easier for people to write and understand. Which I think maps kind of well to why you're using it for your website, and I use it for some of my websites as well because it's it's just a lot easier to write and a lot of the structures they're they're just kind of like a natural text file. >> Yeah, yeah. And that's how I see it. I'm remember back in the days when you had like text files as part of copies of computer games you found online. You sometimes got text files with like ASCII art in it that looked like pretty fancy. And it's like it's a way to style and and juice up a text file basically. Like it's a little more structured, it's a little more readable than just like having random text. It's like, "Oh, so this is meant to be a headline. I can pretend that this is a headline." And I think that makes sense. And because of the simplicity you can use tools to programmatically transform it into other things. I've used markdown to write a book. >> Wow. >> And publish a book. So, yeah. And that has like it's over 10 years ago. So, that's nothing new. >> That's crazy. Wow. Huh. >> But I again, I didn't want to write all the overhead of HTML. Like all the brackets and stuff. It's like, "Yeah, and then you forget a closing thing and then everything becomes a headline." >> Yeah, true. And I think what I also find kind of neat about markdown is that it's almost by default like semantic markup. >> Yeah. >> Whereas like, this is a heading, it's not like, this is a big piece of text that could be a heading or could just be big text. It's just very clear. It's like, this is a heading, this is bold, this is a link. It's It's like super straightforward. >> Yep. And as as long as you're not cheating because if you know that the output format is going to be HTML, you can configure a bunch of the Markdown taking things, like the programs that take Markdown as their input and then produce HTML as their output and configure them so that you can actually include HTML. And then if you don't do Yeah, like that's nice. If you want to like show a video or if you want to have a widget in it that uses JavaScript, you can do it that way. But if you do that, then you invite back all the complexity of HTML. If you don't do that, then by definition you're separating style and content. Right? So like the presentation is separate because you you it looks kind of boring black text on white background by default if it's rendered to HTML and you need a style sheet or something that wraps stuff around it to make it look in a specific way. Whereas in HTML, you can say like, "Oh, I want this to just be like twice the size of everything else." And then you kind of sneak in presentation information into the content. And here you kind of have clean separation between actual content and actual presentation. I think that's that's a good thing. >> Yeah, that's that's true. Now that you mention it, it's like with Markdown, you basically provide the structure and the text. >> Mhm. >> But all of the styling information is kind of separated out. >> Yep. >> So you can take this text and repurpose it. It's like, "Oh, I'll put it on my website or I'll create some kind of PDF with it or whatever." And it's like, the text and the structure of the text is transferable. >> And I think that's also why people think it's good for LLMs because you kind of you have less stuff, less tokens. And if you look at an HTML file without a browser rendering it, if you just look at the plain HTML in a text editor basically, then it's hard to read the content because there's so much craft, so much stuff in it, right? There's all these HTML tags and all this maybe even inline styles and all that kind of stuff. But if a markdown renderer fails and you look at the markdown file in a text editor, it's still is structured and readable. >> Yeah. >> Like a link is the word of the link text, like the anchor text, and then in in square brackets and then in normal brackets. It's probably what I would do if text was all I had available, right? If I was writing an email without the possibility to to actually link things, I would probably like mark up some sort of link text and then put some sort of way to say like and this is where you need to go to actually see that. >> Yeah. >> And I think this minimalism is probably what makes people think, "Yeah, this is great for a machine that needs to understand this content, unlike HTML." >> Yeah. I think the other difference is also all of the stuff around the content. Things like headings and footers, sidebars. >> All right. >> Like all of that is basically gone. So when you write your content in markdown, you focus on the text and the links and things like that. And then afterwards, the system goes, "Okay, I will put your piece of text in the structure of a website and create all of the the cross links to other categories and all of these things." >> Okay, so then that all sounds very nice. Should we just make markdown as our like make our websites in markdown basically or >> Huh. >> [sighs] >> I don't know. He's like you already make your website in markdown. It's like >> [laughter] >> it's an awkward cycle of it's like you turn it into HTML and then you're now you're like, "Well, maybe we should just turn it back into markdown." >> Just publish the markdown with no steps no extra steps. >> I think the big thing is that the web with HTML and everything has been around for really long time, longer than markdown. And all of the crawlers out there, they have practice with HTML and so like converting HTML into text is is trivial. Like there lots of libraries out there that can do that for you. So, if you think about what an average web crawler might look for or might need to find on a page to be able to understand it, then probably that's just HTML. >> Yeah. And I mean, the other thing is yes, it's nice that markdown is usually then focusing on a piece of content, but HTML with all the links and the navigation and the headers and all that kind of stuff that kind of gets stripped out in the markdown files that make the website are important to understand the structure and how this connects to the rest of the site. So, I guess that's also a bad thing. If we were to lose this, that's probably not so good for crawling and discovery, huh? >> Oh, definitely. Yeah, yeah. >> Yeah. >> I think when it comes to things like a search engine or probably also in generic LLM system, having a website that uses normal HTML for the pages is critical because a search engine or crawler can just go to that page, it can recognize all of the other links that are within the website, and usually those links are somewhere in the header or in the footer or in the sidebar somewhere where they say, "These are other categories of content, maybe other pages that are available on the website that are not directly linked in the content. And all of that is critical. So, it's almost like if you want to focus purely on being discoverable in search and being discoverable for these AI systems so that they can use your content for training, then having normal HTML pages is basically the main thing you can do. It's almost I don't know. May Maybe it's even the the primary thing that you need to do as a prerequisite in order to be crawled and indexed normally. Obviously, the web is super messy and sometimes people put normal text files online or PDFs and crawlers have to deal with some of that as well. But they definitely know how to deal with HTML pages. That's kind of the foundation of the web. >> And I mean, the other thing is also for users, you can't just publish a set of markdown documents because A, we like colors and images and stuff to kind of like flow in a nice layout and markdown by definition, unless you put a layout on it, doesn't. And markdown doesn't support layouts directly, so you would have to have some sort of mechanism to You're basically recreating the browser. You're recreating HTML parsing in the end. So, you might as well use HTML parsing because as you say, that has been around and has been tried and tested for decades at this point. The other thing is you would duplicate things if you were to acknowledge like, "Oh, users don't want markdown. They want a full-fledged website." And then I create a version just for LLMs, then you're kind of making twice the work having twice the work, no? >> Yeah. I think that's always terrible on the web. And I understand where these ideas come from in that a lot of web pages are just terrible from a structural point of view and hard to use, and it's tempting to say, well, users can see this complex, weird page, and automated systems, they should have it easy. Like, you should just give them the information that they're looking for. But, fundamentally, as soon as you have these parallel versions of your content, then everything becomes so much more complex. You have to maintain those multiple versions. You have to make sure that nothing breaks on a version that a user doesn't see. Uh because users might complain to you if your page doesn't load properly, but if the LLM version of a page doesn't load properly, then no user is going to tell you that something is broken. And a lot of these automated systems, they might not even recognize that something is broken because they see, oh, it's like, there's some text here, it must be what they want us to index. >> Yeah, I think we learned that lesson with dynamic rendering, which was a nice stopgap solution for a while, but we found out in practice it oftentimes caused more problems and was really hard to debug because of this duality of the two different separate versions, and yeah, that's uh that's not great. Okay, while we're on the topic of markdown, should I then just create a text version, like, have like a text file that has all the content in it for LLMs, or is it kind of like the same problem? >> I think you mean the LLMs text file. >> No. >> No, [laughter] the text file for LLMs. Yeah, So, so I I talked with, I think, one of the people who who created that proposal a while back, and the idea was really not to create something that makes it easier for search engines or LLM systems to discover all of your content, But, almost more that if an LLM already knows about your site and wants to find out what else is here, then that might be an approach. And I think the aspect of using this as a way to optimize for discovery by AI systems or discovery by search systems, that doesn't make any sense at all. Uh because it's basically you're telling these systems like, "Oh, I have the best website ever and here are all of the pages that everyone must go to and you must buy all of my products or whatever you put in there." So, an LLM system it basically by design can't trust what is here as a way of differentiating between different websites. If someone is already on your website, maybe some kind of automated system is helpful where if they go like, "I want to go to Martin Split and buy a photograph." then the LLM system can go to your website and can look around like, "How do you buy a photograph? Like, maybe he has some guidelines for me as an agent for buying photographs." That kind of makes sense. But, going off and saying like, "I want to buy a photograph, which website has one?" The system is not going to go to your website and five others and say like, "Who has some automated information?" But, rather they're trying going to try to find the best website first. >> Okay, makes sense. >> I think from that point of view, optimizing as a way of being discovered, that doesn't make sense. But, like what happens when an agent is on your website, I think that also just generally seems to be an open area for discussion at the moment in that there's LLMs that text as a proposal. There are different JSON files and well-known file types that are in discussion. There's Web MCP, which I think tries to do something similar where they say it's like, well, you're on this page now, but we have a programmatic interface for this added specific URL or a specific mechanism. >> Mhm. >> I think those are then almost different discussions. So, the generic SEO angle of how do I find a website that sells me a photograph is almost going to be completely bound to HTML pages and normal web pages. And then, if so a user decides to go to a specific service, then within that service, then there is a little bit more room for maybe helping an agent or an LLM system to find the right approach. But, what is interesting, of course, is like lots of ideas and none of these have basically crystallized as the one thing that everyone will use. So, I'm sure over the next, I don't know, half year, year, or maybe longer, it's going to take a bit, and some of these agentic systems are going to kind of unify around some some standard file type or mechanism or something. >> All right. So, I guess that should settle the debate if we should just go back to markdown. I mean, even if we are if for all the websites that I'm have made in the last, I don't know, 10 years at least. Actually, holy moly, that was like 2012 when I started using markdown to make my website. So, that's at this point 14 years. For the last 14 years, I basically made markdown websites, but no one would have known because you look at the HTML version, and yeah, think that's continues to be fine. >> That's fine. >> Yeah. >> I think the one place where maybe some markdown content on a website could make sense is if you have something like developer documentation. >> Uh-huh. >> Where again, if the agent or the LLM system already knows about your website and the user says like, "How do I use this API?" then if you give the LLM system a markdown file, it's a lot easier for it to understand, "Okay, this is the the mechanism here." So, I suspect for more code or or websites that provide code samples and developer interaction for them, having markdown versions of the technical documentation makes some sense. And then you have that challenge of course of having the parallel versions because users are not going to look at markdown because it's a text file. Yeah, it's not it's not this nice-looking HTML page thing. But maybe for agents, that's something that makes sense. >> Well, I'm I mean, there's a solution to that, which is to publish your repository with the documentation in markdown and then use that markdown documentation to generate the HTML version. >> Exactly. >> And then you don't have to drift. Yeah, yeah. >> Exactly. I I think again, for developer content, I think that makes a lot of sense. But if you're selling shoes it's like you're not going to have a markdown version of your shoe catalog. Like that makes that makes no sense at all. I think the challenge is of course people who are creating websites are developers and using developer tools and they're like, "Oh, I'm using the markdown version of this API to understand how it works. Therefore, maybe my shoe site should also have a markdown API." Which is kind of like that bias, I think, that developers just have. That is like, I do it like this, therefore maybe everyone does it like this. And probably that's not the case. >> And good news is that normally there's more than just the developers involved in making a website. So, hopefully teamwork will make the dream work. >> Yeah. Yeah. >> [laughter] >> Yeah. And the other thing I think is also we've been talking about websites at this point. But, the web platform offers more than just plain old websites. Like, a list of products, you could build applications in there, you could build interactivity in there. And markdown itself doesn't support that, and I don't think it should because again it's for content. And so, I guess the web will continue to be this multitude of things, this multitude of what a website could be. It could be somewhere between application and actual just like a content document. And I guess markdown is just one part of it, and most likely will stay just the middle bit of the pipeline from thoughts in someone's mind to website on on the internet. Yeah. >> Yeah. I think again, like for all of the SEO-related things and discovery of content, a normal HTML website is >> is the best. >> Is like that's not going to go away. I mean, like who knows, but that seems very unlikely that it'll go away. So, that I think is at least the baseline requirement. If you have developer content, doing something markdowny is fine. Try it out, see if it actually brings some value. But, for everyone else, I think markdown doesn't really make sense. >> Yeah. All right. I think that makes sense. And I think we've spoken enough about markdown at this point. And uh I hope that you all out there have a better idea of why Markdown became so popular recently for LLMs and uh what you should do to make your websites and maybe even use Markdown to create the HTML of your website. It's fine, trust me. Right, John? >> Right. Do it. >> Excellent. >> Or don't. >> [laughter] >> Yeah, we're not cops. We're just like random people on the internet. Uh Well, anyway, thank you all so much for listening out there. I hope that it was fun and useful. Let us know in the comments below if you're using Markdown for something and how you're using Markdown. And um in that case, thank you, John. >> Thank you, Martin. Great to be here. >> Thank you for being here with me and uh bye-bye everybody. >> Bye. >> We've been having fun with these podcast episodes. I hope you, the listener, have found them both entertaining and insightful, too. Feel free to drop us a note on LinkedIn or chat with us at one of our next events we go to. If you have any thoughts, let us know. And of course, do not forget to like and subscribe. Thank you [music] so much for listening and goodbye.