Bora Goekbora, BCG | theCUBE + NYSE Wired: AI Factories - Data Centers of the Future

Palo Alto studio connection Silicon Valley and Wall Street. I'm John B here with Dave A, my co-host. Hello, I'm John Furrier, host of the Cube. Here at the Cub's NYC studio, of course, we have our Peloto studio connecting Silicon Valley to Wall Street. It's our NYC wired program and the open community of leaders participating in the revolution of AI infrastructure. just our AI factory series. Uh we got a great guest here from Boston Consulting Group, Bora Gbora, managing director and partner at Boston Consulting Group or BCG. We're not going to talk about the strategic resource map that BCG is well known for. He's an expert in the edge and what's going on in the telecom and the AI infrastructure. Bora, thanks for coming in. Appreciate it. >> John, it's a pleasure. Thanks for having me. >> We we were talking before we came on about many things that you have insight into. But I want to unpack it, but I first want to get your take on the AI infrastructure wave. What's happening? We all know the AI factories are booming. It's our most popular series, but the edge is on the horizon. Nvidia's telegraphing with 6G. It's just a natural progression. We're in a distributed computing paradigm. We've seen this movie before. >> Yes. >> Big node, little node, edge node, human nodes. You're going to have this AI uh network effect. >> I think so. >> You're at You're doing a lot of digging into that. what's happening >> right I think I think John what's been fascinating to see is this is really the evolution of a space that's been in front of us the whole time I think you know traditionally this has been communications infrastructure this stack has always exists I mean this is a traditional IT stack but there's now just so much emphasis on the GPU and the compute level uh layer of that same stack and so what's happening is we're seeing a ton of attention around high performance compute obviously um investors are seeking you know how do I underwrite it what's the longevity of these chips They're also investors that have been in communications and digital infrastructure in some cases for decades now. And so this is now the natural next asset class that they're looking into, but the characteristics of it are very different than the characteristics of some of what they've invested in the past. Things like cell towers, conduit, fiber networks, etc. So the underwriting framework is something that they're deeply deeply looking into again especially about residual value of of these high performance chips. Yeah, on the other end we have telecoms and you know the incumbents of this infrastructure layer that are dealing with things like interest rates and balance sheet strain and generally flattening growth because of how competitive things like broadband have become. So now more than ever there's this tension of how how do those players express their scarcity and value as investors are looking to continue to enter the space. So that's really what I've been working on lately. >> I mean it's fascinating that you the way you describe it because you have the physical plant or the physical aspect of things. You have the communication stack you mentioned that's essentially software now with AI. Um it's interesting because now that's coming together. It's not like siloed, right? They're integrated. I have to ask this question because one of the things we're seeing on the AI infrastructure side and even in the the the generational shift is older folks that have been working on hard problems have an awakening because we see Nvidia and the supercomputing capability high performance clusters providing massive value to actually solve problems that couldn't be solved before just by a function of evolution right so now we're in an era where I talked to folks hey when I was in grad school in the 80s or 90s s I did that now I'm actually doing it because I can do it. >> So things are more attainable. The question is what's the new attainability factor that coming out of some of these markets because they've been around they've deployed physical plants telecomunications it's been around right >> is there a new refresh refactoring kind of mindset where these unsolved problems are being solved >> absolutely and and this is you know very much in terms of BCG and my co my excellent colleagues I mean we are spending a ton of time with corporations about how to actually embed AI into the work that their companies do because sure there's a ton of attention on training and the complexity required for that and the computation and the sophistication of that stack but it really only makes sense if we see inference come through which we're seeing that inference is getting expressed obviously with prompting of things like chat GPT cla but really the transformational level for um enterprises how do they embed it into their workflows such that the work is happening like you said in ways that it couldn't have happened before or at a much faster pace at a much higher you know accuracy rate etc. So that's really where we're spending a lot of our time with with enterprise. Um the other way to take your question is what is the new technical capability of things like the edge and where should the edge sit and where is this compute really supposed to be and so we've been spending time there too and >> what's your vision there because the question would be okay Bora what is injecting intelligence at the edge mean? Yeah, it's a >> Okay, it's kind of a loose statement, but no. Okay, that that's >> that's a good question. And for me, because I spend a lot of time with our our telecom and cable clients, the edge is not a new word. I'm almost, you know, it's time to really really really reinvent it in a way that is probably going to be more built to suit how it's supposed to behave. And so what I mean by that is we've always talked about the edge as really a low latency application or a low latency location where compute should sit such that the end user can perform use cases that are super latency sensitive like milliseconds is what we're talking about. Autonomous vehicles have been talked about >> robotics. >> What we've really seen though other you know autonomous vehicle companies they're doing it all on prem. So that edge never really worked out right in that sense. I think what we're seeing now though is >> when you say on prem on prem to a data center or onrem the caricle the car. Yeah. That's where the sensors are. And so the edge is still it it's skipped a step. It's actually just going to the device. And so a lot of a lot of you know speculators and observers and and and you know folks in innovation are wondering is that edge going to converge back into something more in the middle? And so options for that you've got cell towers they've got edge businesses at their cabinets. Um, you've got telecoms with their central offices and cable companies with their headends. That's powered real estate and it's actually some of the most distributed power real powered real estate we have in the country. There's thousands of them. And so, you know, they're rated up to 1MW, maybe two, three plus, but there's some re retrofitting work that needs to be done, but that seems to me to be, you know, an excellent place for distributed inference workloads to be sitting. Um, >> so if I'm a telco or I'm a carrier or infrastructure provider or a carrier, >> my retrofit decision >> has to factor in economics. >> Absolutely. >> Cost. >> Absolutely. >> Take us through how you're thinking about that because it's pretty clear that it's a no-brainer of them to retrofit from a concept standpoint. So you say, "Hey, you got some dormant power, you got an edge position. >> What's stopping you? >> Where get that intelligence in there? In fact, go faster." >> No, I understand. and and you we've seen them play different ways. Some have said we're just going to sell these assets and so you know if you look around the ecosystem some telecoms have disposed of them and they've given them into hands like private equity and investors and they've said this should be shared we'll do the retrofit and we'll colllocate and we'll actually ring fence you as a sale lease back. It's how they you know can get structured. The other way we think you know it may go is um the cable companies and telecoms they may do it themselves. that may, you know, inject capital from partners, you know, to help with the retrofit, but then they'll have access to compute at that edge. Um, they'll, you know, buy those GPUs. The other way they could do it is they could do a powered shell or a powered land, but they definitely have some options. >> I love that um uh edge story because my story was hyperconverging edge. That was a great example because they are going to the car that that use case. What does a hypercon converge edge look like to you? Because if if it follows the data center pattern, we had storage, networking, comput, they're all hypercon converge. Now they're unconverging uh because of the systems are looking different obviously with with the new architecture. What does hypercon converge look like in practice? So for us when we get questions like this um you really don't see as much replacement right away. You see, like you said, coexistence and then eventual convergence of these different asset types. And so you've got the hypers scale, you know, large facilities, the colo facilities. You'll see an expression of the edge um into things like the tower edge, the device edge, and then like what we talked about, more of a regional point of presence edge around the head end. That's going to act as a mesh of different network layers and compute layers that have different properties. Some are closer to the endpoint, but they're also further from the center. So depending on which hops they have to make, they're going to express their latency. Yeah. They also have different physical um you know form factors. They can only host so much compute. So we're just going to see workload distribution follow that suit, right? There's going to be a perfect place for each workload that optimizes cost, location, latency within capacity constraints and obviously monetary and data sovereignty constraints is another one too. Yeah, you nailed the um you nailed the latency point because if you look at the AI factors gen one generation one of Nvidia Infiniban KVach now Dynamo networking was the key fundamental thing >> that's where the that's where the bottleneck was >> that was the bottleneck and then KV cache became so important because of the of the serving of the tokens pre-filled decode not to get too technical but then that cash got fatter but inference comes in now compute comes in so now you have kind of component subsystems around the mass of HPC or >> cloud system or any AI system. If you take that to the edge that opens up the idea that okay I could disagregate the infrastructure okay and I need networking. So you brought up the the >> and fiber is you know a topic that we've seen actually quite a big um surge in the need for not just fiber inside the data center but also fiber to the data center and the interconnection of data centers. So there's also the concept of clustering which fiber plays a huge role in because the the major constraint has been powered land. We're going to see um facilities go deeper and deeper into where costs are appropriate and land is a available with power. The thing is the fiber won't be there. So that mismatch is actually something we're also >> So you get the land and power but no fiber. >> Exactly. And so what we're going to see is a surge and we're already seeing you know we've seen some some large deals from um Zo and Verizon. they're connecting these data centers at the cluster and this is a big business for Lumen as well. >> Um that was actually unexpected in B2B fire but now there's all these new enterprise companies >> the scale across as Nvidia calls it >> and and and so those data centers are the new enterprise customer and we're seeing not just new uh mileage of fiber but also density of the fiber. >> Yes, >> it's it's I want to get to that this idea of asset light. It's a term that's been kicked around. >> Yes. So capex spilled out, trillions of dollars, billions of dollars being spent. An enterprise doesn't I mean JP Morgan's got a $10 billion IT budget. Okay, they're different, but that's it. Enterprises aren't sitting on a massive financing capable. They want to maximize costs and so they will work with whatever that data center is. To your point, enterprise is the new data center >> and or would you say is the new enterprise? Okay. Because they will use that and create that single tenant experience like we're seeing with tools like Instinct, these new awesome things. Yeah, there's a there's a new part of the stack in managed services which are basically whether it's onrem, colo, hypers scale at the edge. There's definitely a growing um opportunity there for investors too in those managed service space because not every enterprise is built to manage the complexity of all this different infrastructure as well and aentic workloads and high performing AI workloads are only going to demand more sophistication on >> and they'll want to plug in to that intelligence. Absolutely. >> And get the tokens like electricity >> that that's the new commodity >> comput. All right, back to your other example about the edge because I want to get this out there and help my understand. I think you got we're just kind of talking through and we're riffing on this. If I'm in a metro area, just say New York or San Jose because we had a great demo at GTC around San Jose having a bunch of towers and 6G and all kinds of metro cloud whatever they want to call it. Okay, I got homeland security, I got first responders, yes, critical infrastructure, basic consumer services, and then business. If I'm a carrier, >> I'm not selling bandwidth in a family plan here. Hey, the revenue opportunity, whoever can nail that equation, that's basically quality of service and policy based management. That's a managed service. Exactly. >> Why wouldn't I want to be in that business? >> And you you you see >> what's your take on that? >> Yeah, it's it's it's absolutely a required space. The complexity of networking and we didn't even mention satellite, but there's other network layers emerging as well. We're not quite yet at 6G. Um, but still still, >> but is 6G the Infiniban of wireless? >> I think I think there's plenty, you know, to accomplish with 5G. Um, and so this is the what's called like the B2B segment for carriers, and it it's certainly a business that um, you know, there's complexity in it, but there's there's huge opportunity to grow. What it requires is they've got to be able to manage the different network layers. In many case, enterprise doesn't need to know, hey, is it being supported by 5G? Is this a fiberized connection? They just want to know that their use case is going to be supported. And there's also resiliency that they have to manage. >> Is my latency nailing my workflow? Is it secure? Resilience >> diverse. >> I'm going all day long. Well, I don't care where it comes from. >> But that complexity is great for infrastructure owners, those that can provide the services layer on top. I'll take it a step further, too. The the workload specifically, the applications in uh facilities also come with a lot of complexity. Observability is huge right now. threat detection. I mean, we're seeing more than ever. The bar is lowering for anything that's AI enabled. So, now threats are also multiplying as well. So, we're just going to see more and more services like Sassy, um, private networks. >> You know, I I was observing the Nscale investor meeting here a couple month a month ago. They bought a company that we've covered since coming out of Cal Berkeley was any scale. Any scale was acquired by Nscale. They kind of have the same name but the reason why they bought them they were vertically integrated neo cloud or HPC cloud whatever you want to call it but they have because of this new disagregation going on whether it's disagregated serving or subsystems handling >> parts of the workload orchestration's going on you need observability they had to know every piece of resource being touched >> it's hard >> up and down the stack that is extremely difficult >> you have to bake that into the stack >> that is a really huge that's a new dynamic we're starting to see >> and and that's the concept and a lot of the concepts you and I talking they've been around but we're seeing a bit more traction than we expected on things like the edge um B2B fiber but digital twin is another concept you just got into which is can I understand my inventory and my assets in real time what's what's available um what can I sell where are their threats um how should I be rerouting or optimizing customers are going to be expecting that too because workloads have to have their cost managed every CFO >> and CTO is at battle right now around what's the right answer to tokens and AI into the organization but doing it in an ROI based way and so visibility and orchestration are going to have more pressure >> explainability from your roll back standpoint for understanding >> how to handle the resource >> what happened auditing >> what did I get from it am I is my cost to serve decreasing my cost per interaction decreasing >> I'll tell you funny story too I've been inter I've been experimenting with >> AI assistance and what I'm noticing is companies don't necessarily want your agents to call They're saying, "Hey, I'm not ready for that. My handling costs are going to go through the roof." So, the gentification and these use cases are are moving in that direction. >> You guys must be so busy at BCG because there's so much chaos and it's going to be out there, but it'll be rained in. I mean, like you said, there's a lot of concepts. I mean, I was just talking with my daughter the other day about, you know, in England, they drive on the left side of the road, and the you everywhere else it's right side of the road. That's a governance issue. Like, okay. Okay. Okay. those agents. I can't just have my agent driving on the wrong side of the road. So, you're starting to see the sovereign equation become, okay, what are the laws? >> And if you have agents not liking the laws, then you got the British law and the Americans don't want to have them make their own rule of law. >> You have to do it in an IT responsible way. >> So, you're going to have this like, well, agents will just make up their own laws to serve themselves. >> There'll be guards in place otherwise we won't be able to adopt them, right? >> All right. Final question for you. Um you know you mentioned earlier a lot of the stuff's been around for a while computer science and we of course we've seen them a lot of computer science principles a lot of networking principles um communications principles a lot are out there what do you see as the most uh biggest comeback uh besides ontologies which I was just talking the other day at with a graph at a graph engineering event um everyone's like it's not new although palunteer saying it's new because they want to market it but you know there's things like ontologies you know a lot of the AI stuff is primitives we're d we're nailed in the in the 80s. Some of the biggest computer science breakthroughs were in the 70s, >> right? >> What is the hottest comeback hottest less the comeback of of buzzwords or concepts that are that are now front and center and making a real impact? >> Absolutely. Um I'm going to go back to a topic I touched on a bit, but for a long time, you know, uh businessto business fiber and long haul fiber was seen as a commodity. We're now seeing with how much data center capacity is coming online and also the concept you mentioned of data center interconnect where you can interconnect facilities to present even larger clust clusters to your customers. Fiber is arbitrageing some of the constraints we've seen in powered land because you're able to interconnect these facilities and so for me it's really been about the fiberization of these assets and um you know for for some of our carriers >> it's not it's not a commodity when it's a critical path item for turning on a resource that's expensive that produces value. >> Exactly. and uh the throughput you know the bandwidth required at these sites is more than expected and so that's been a you know a great growth opportunity. Well, we could go for an hour. Definitely have you back. I'm sure you're a great great uh commentator and also, you know, sharing insights. We haven't even gotten into the whole grid, >> you know, reliability issues. >> Talk about copper. >> Copper, we'll come back to copper. You know, I think Photonis is going to replace copper in the rack. Maybe it's commoditized back, but for thank you so much for coming on. Great, great to have you. Great insights again. BCG is helping their customers. They're solving problems. This is what they do. This is what everyone's working on. the money and the entrepreneurs and the smart money follow the constraints and we're seeing a lot of them once they get taken away opens up value creation and extraction. We're doing our part on the cube here to share that with you. Thanks for watching.