AI Reality Check: What’s Really Happening Behind the Hype?

There is so much happening in AI right now it is hard to keep track of the latest changes, but are people getting real value out of it. Today we are going to dig into where things are going. Stay tuned.

Today we have with us, Mark Kurtz CTO at Neural Magic where Mark leads a team of engineers and researchers creating cutting-edge solutions for deep learning challenges. Neural Magic was founded in 2017 by MIT professors and research scientists, and most recently the company recently raised a $35M Series A round led by NEA, with participation from Andreessen Horowitz, VMware, Verizon, Amdocs, Comcast, Pillar, and Ridgeline Ventures.

So let's talk with Mark about the state of AI and where things are going.

Welcome Mark

Click Here to Subscribe:

FUTR.tv focuses on startups, innovation, culture and the business of emerging tech with weekly podcasts talking with Industry leaders and deep thinkers.

Occasionally we share links to products we use. As an Amazon Associate we earn from qualifying purchases on Amazon.

Transcript

Chris Brandt: 0:00

There is so much happening in AI right now, it's hard to keep track of the latest changes, but are people getting real value out of it? Today, we're going to dig into where things are going, so stay tuned. Today, we have with us Mark Kurtz. CTO of NeuralMagic, where Mark leads a team of engineers and researchers creating cutting edge solutions for deep learning challenges. NeuralMagic was founded in 2017 by MIT professors and research scientists, and most recently, the company was founded. raised a 35 million Series A round led by NEA with participation from Andreessen Horwitz, VMware, Verizon, Amdocs, Comcast, Pillar, and Ridgeline Ventures. So let's talk with Mark about the state of AI and where things are going. Welcome, Mark.

Mark Kurtz: 0:52

Thanks, Chris. Thanks for having me on. I'm really excited to, uh, to talk through and be on the podcast.

Chris Brandt: 0:57

AI is a, in a funny place now. I mean, literally, I mean, my email mailbox is just absolutely inundated with AI offerings. This AI, that off AI, you know, my company will do a, you know, like produce your generative AI models for you, blah, blah, blah, you know, it's just endless. And, and, you know, like my newsfeeds are just like, Every day, it's something crazy new, you know, development in in the world of AI. We were talking before we started here about how when we talked previously, like so many things have changed in just a short amount of time. It's absolutely crazy. You know, why don't we start at, you know, um, you know, you're, you're at Neuromagic. So you're in the middle of this all the time. What, what are you seeing? As sort of like the state of generative AI right now. I mean, that seems to be the hot topic.

Mark Kurtz: 1:52

Well, there's the community push and then the enterprise push and community is hacking around and doing quite a bit more. And I think it differs a little bit from what enterprises are actually trying to do where community is all about. And the open source ecosystem are all about, you know, agents. How do I optimize my life, right? AI systems, things like these, are there ways to run these locally on my computer? So I can, I can be a little bit more productive in my life. Enterprise, because, uh, they have a little bit more space. Standards and, and, uh, need a lot of control around what they're putting out, how they're processing data, things like that. They're definitely a lot less risk, uh, risk taking and more risk adverse towards, uh, agents and, and things like that. So they're really kind of just trying to scale up and figure out how does this fit into my current ecosystem and, uh, and build on top of it. So they're less working on agents and more just, um. Augmenting a lot of pathways. So most of the common ones are, you know, blogs or image generation, things like that. So content augmentation and even coding generation and quite a few are now getting into, you know, trying to replace chatbots and, uh, and things like that. Um, but a ton of human oversight over the top of these, just to make sure that, you know, they, uh, they have control for the brand and control their image. So I'd say, you know, these two different sides are really kind of weighing and going a little bit in opposite directions. So I don't know.

Chris Brandt: 3:16

And I think their needs are so vastly different too. And I, you know, so like, you know, starting with like the, the corporate side of it. Right. I mean, I think that, you know, probably the biggest push that's been happening, I mean, besides just chat GPT in general, you know, kind of driving the market, but I think in the enterprise, um, Most companies are, you know, being inundated about copilot and things like that, right? And, you know, I think the challenge I've seen, I've seen a lot of companies, they've been deploying copilot, but I don't know that they've, you know, once they get it in, they kind of go, and now what? You know, what do I do with this now? Right? You know, so it's a, it's a question of like, are, do you feel like companies are really seeing the value in some of the deployments they're doing right now? And you know, because it's changing so fast, do you think that there's, there's an advantage to getting into it early or, or, or not at this point?

Mark Kurtz: 4:11

To take the second question first, uh, you know, towards getting in early and advantages there, absolutely, because I think, and this has been just throughout my career, the number one thing that hinders AI adoption in any enterprise, you know, before we had all the computer vision pushes and NLP and now, uh, NLG and the generative AI push, the thing that's hindered it has just had, uh, has just been, you know, business people and, and product people and just kind of holistically across these companies understanding it. Where the risks are and how to implement AI within their products. So I think getting in early and getting those experiences absolutely critical, regardless of if you're going to have immediate success, it's 1 of those things that you have to invest in and be ready to take on. And then looking at what's possible today in the enterprises and what we've seen a lot of it. It focuses around kind of more core traditional use cases that are starting to get replaced. So to take one example, Amazon and Google and all of them have, you know, replaced most of their teams that were traditionally on the Google Assistant and Amazon Alexa, things like that. Traditionally NLP, they've replaced all those. Primarily with each gen AI pushes. So, you know, a lot of invest. And just

Chris Brandt: 5:26

to clear up, uh, NLP is natural language processing. I, I know we, we have a tendency to get, uh, into a lot of acronyms and things. I just want to make sure everybody understands what we're talking about.

Mark Kurtz: 5:37

Um, so yeah, we're, you know, we're looking at a lot of replacement and big bets on gen AI for things that are assistant based and especially translation based. So those are easy fixes, you know, to put in where you do have, you know, you do have assumed risk. For the user coming in, maybe it's not going to be 100 percent correct, right? And also these pathways where you have human moderation over the top of it. So being able to augment and, uh, augment human processes in terms of cogeneration and, uh, and things like that. Those are really easy to plug in. Currently, the ones that get a little bit harder are the ones where we are controlling outbound messaging. So chat messaging. I know a few airlines have. Gone out there and, and replace their chatbots, but they always have that little critical help button, take me to a human. Um, and I think these are just kind of the growing pains that, uh, that happened. And, and, you know, people are getting better about how they can mitigate the risks around GemIIni just because there are issues with hallucination and bias, and those are not going to go away anytime soon. So the big thing is investing in one, your understanding of those and to being able to build out the risk tolerance and pipelines to control those. So that you can scale it in your enterprise.

Chris Brandt: 6:48

You're pointing to Something I think is really critical is, is sort of like, and, and, and maybe it is too with like getting in early is, is a big part of this too, but like getting to the point where you understand what the limitations of it are and, and so that you can. You know, direct it better or or not fall prey to being overly confident and how, uh, how it's it's performing. But, you know, one of the things, too, is, you know, when we talk about sort of like a lot of the use cases we see, um, there's there's a conversation that's happening around the idea that, um, We, a lot of these are creative processes that we're seeing coming out of generative AI, and the idea was AI was supposed to relieve us of all the mundane tasks so that we could do the creative work. And I don't think we've quite, I don't think we've quite gotten there yet. Um, What are what are your thoughts on that? A

Mark Kurtz: 7:42

few thoughts on that one is, uh, one is just, you know, in terms of the immediate creative needs and, uh, and things like that. So, you know, putting out blogs or, uh, augmenting cogeneration, things like that, those are podcasts.

Chris Brandt: 7:57

Which Google has offered up to everybody.

Mark Kurtz: 7:59

Yeah. The notebook LM. That's uh, it's an interesting, interesting place. Um, but, uh, but yeah, all of that is a lot easier to plug in right now because it is an iterative process and it's something that easily has human oversight over top of it, right? The things that are harder to plug in, uh, out of the box right now are the ones that are fully automated pathways, right? So controlling a robot to go through, you know, in this. 60 space and make sure that it's not bending itself over backwards or going through your part, things like that. Um, those, those become a lot harder because there is no tolerance, right? You have to be, uh, exact and same thing applies to JSON schemas. If you're taking in payloads, things like that, right. Very strict setups where a difference in a token can make or break things, right? Whenever we're in the creative industry, a difference of a token doesn't. Doesn't really matter as much provided that it is convincing to a human, right? And that's the thing is that just the interfaces are a lot more lax on the creative side But the other side of that is that all of these generative AI models are trained on just immense Masses of data, and, uh, to the point that we're almost running out of clean data. Well, I think we're within an order of magnitude of running out of clean data, uh, to be able to train on. And, um, and yeah, it's one of the things that it's memorizing that average and predicting based on that, you know, average that it collects. So it's, uh, you know, the new processes and new generation trying to influence new pathways in culture and society. I don't think Gen AI is close to that or going to be doing that anytime soon, but I think it will help augment humans to get, get to the next pathway.

Chris Brandt: 9:42

I want to like highlight what you're talking about because I've been hearing from some folks in the, in the. AI space that they've actually sort of tapped pretty much the maximum cap capability of all the available training data right now. And so now, now we're, we're getting into a world where, um, one, we have to start taking a look at different approaches, I think, to, you know, solving some of these challenges. Um, but you know, there's, there's also like this, this issue of synthetic data now that we're, we're getting where you have to like, there's not enough. Training data. So let's create some training data. And then thirdly, we have this issue of um, Rights over content. I mean, especially when we're talking about like the creative stuff, right? We've had so many issues. I mean, like literally the strikes in Hollywood were about, you know, AI. Um, and we're seeing a lot of lawsuits happening about, you know, image galleries and things like that, that have, that have been taken. And I think you can really see that too. Um, you see like. When you see generative AI, I can see the faces of celebrities in the mix of things, you know, so they're, they're definitely leveraging, you know, those things very heavily. Um, but so, so like what, what are your thoughts on, you know, like the, the challenges around training data, what are our pathways out of that, uh, and, and, and synthetic data? What, what do you think about that?

Mark Kurtz: 11:06

Starting with just the train data and running out of it. I think this is, it's something that was very easy for the big companies to scale, right? Uh, just to start off, just, it's something that they've done a lot of, they have a ton of data available to them already, and then scale out on top of that. And it's pretty much just an OpEx, right? Throw some cash into it and we're going to scale up and get the best model. And then we'll see where it goes from there. And, uh, you know, along those lines. Uh, Zuck and Zuckerberg and all of these other people have come out and said, you know, this is a significant investment that we're doing, but it's one that we can't miss out on. And even if it doesn't pay off, we can recycle all this hardware. So the current state that we're in and the fastest pathway to get something that was impressive was just the scale data and scale compute. And now that we're at the limits of that. I think we have to start getting a lot smarter on our algorithms. I don't think it's going to be the, uh, the old one pathway and, you know, trying to throw in more contextual tokens. Cause there's just, there's a lot of research coming out about China soft prompting and things like that and how that can help certain regimes, especially ones that are well structured and, uh, you know, getting into. Map or, you know, if you're trying to prove out logic or coding, things like that, things that are very easily testable, then can easily push, push that boundary, but that's a ton of compute. You have to have an environment that you can test them, and it doesn't solve. You don't have that in every problem that I was trying to apply to. So I think we have to get a lot more creative and smarter and move path. Standard gradient descent gradient descent is just moving through this huge optimization space. I mean, just absolutely massive to the scale that we've never seen before moving through this optimization space. And it's memorizing the average between that. And luckily, it doesn't over fit to that data, right? Is trying to figure out that trend that extends to the next. Next side of batches that are coming in. And so ultimately, I think the big thing is it has to be a double down on at least moving forward a double down on smaller data, smarter data and smarter algorithms and for people are trying to implement with what's out there right now. It's kind of the same thing, right? Get cleaner data, scale down your data, and make sure that you can, um, not try to solve everything with a single model, right, be able to focus these models on what your core needs are and put some pipelines around it to, uh, to ensure that it's going to meet those needs. So I think that's the biggest thing.

Chris Brandt: 13:35

I've been really intrigued by the idea of like smaller models, because when you start looking at the performance of some of these AI models versus the cost to run them, um, you know, you can see these. Um, You know, they kind of, uh, there's definitely a trajectory that they're on. And I think that, you know, some of these smaller models that are more heavily trained, more tokenized, you know, then, then some of the, like the really, really big high parameter count models, I think there's some really compelling things going on there. And I think that, you know, for You know, like a open AI, you know, for example, there's an incentive there to build the really big models because it's a competitive advantage. Not everybody can have a data center full of, you know, servers and, you know, things and running, running constantly. So, um, but I think, you know, we're seeing really interesting results from some of these smaller models that are just more trained, um, and, and, and, and putting them together with agents, uh, you know, which I would love to get your, your take on, you know, um, smaller models, agents, um, you know, moving it from the data center more back towards the edge. Um, you know, what, what, you know, obviously I, I know you're a proponent of these, these smaller models. Could you, could you speak to how that all works and how like integrating multiple models and agents and all that kind of brings it together?

Mark Kurtz: 14:57

The big thing is, and as you said, open AI has a vested interest in these larger models and be able to have a chatbot interface that ideally does everything right? And as you've seen, there's quite a few issues that come out of that and trying to plug in with that one is just that you have to put a lot of safeties around it. You have to you essentially create a new position for prompt engineers. Right, and coming in, which is just another ability to code. You're trying to figure out how to change this model and tweak, tweak the context that it gives you the output you expect. And, um, you know, there's some risks there as well, which is if they change the model, all those prompts that you've done need to be reworked, right? Things like that. So, whenever you can come in and limit the scope, because not everybody needs this model that can do everything. And also, whenever you scale up these model sizes and try to solve every single problem. It's just the amount of surface area that you have risk around grows exponentially, right? So if you can minimize that surface area, try and specialize these models into the core task that you're worried about, it means that you can get dedicated teams, dedicated evaluations and control the life cycle. And also these smaller models make it tractable for you to train and fine tune and control, right? And that's kind of the big thing is that are you at the mercy of. Whatever open AI or well, I guess, yeah, open AI, if you're doing the open API or meta, if you're doing the open source, right, you're at the mercy of whatever data they collected, what that model learns from that data, rather than being able to tweak what it's learned to what you want it to do. Right. And what, what identity you want it to have and be able to push out. So I think these are the big things is that. Yeah. Whenever you scale down the model size, you can specialize the more, becomes a lot more tractable to be able to do model life cycles on these, right? Rather than spending millions of dollars to train a large one. Uh, you can, you know, you can do it for a few thousand for, uh, some of these small models to fine tune them to, uh, a small data set that becomes very tractable for, uh, startups or, uh, truly tractable for large enterprises to be able to put a life cycle around and actively improve that and build it into their product, right? And all of that trains. So, uh, you said, with agents and things like that, because the more we can limit the scope on these LLMs, the more we can, uh, it's not that we're necessarily going to get rid of bias or hallucinations or anything like that. But it makes it a lot easier to build safeties around it. So we know when they're acting up and when we need to reevaluate, right? Because if we're just trying to solve, you know, push out a model that solves everything. Again, we're at that just creative interface where you need a human interpreter. Um, be able to come over and interpret what the output is. And if that's correct, but if we can scale down the risk profile now, we can get to the point of automation and be able to put. Uh, some safeties around it, both in terms of our people, you know, if it's exposed to the Internet, are people doing prompt injection, but more importantly, on the end of it, right, we can do quick tests on, is this a, uh, some, you know, an accurate, uh, accurate model, so, or an accurate output. To give an example, if somebody's coming through and trying to parse, you know, unstructured data and pull out some figures, maybe they're trying to fill out a loan, uh, application for somebody, you know, across all these different websites. Right. If you can come in and say, hey, uh, for what it populated, you know, if they said the max value that they're trying to get is 1. 2 million or something like that, uh, you can go, you can go through and say, it does that exist in the original input, right? So the output that comes out, so there's a lot of easy things like that, that we can control a lot more as we scale down the size and, uh.

Chris Brandt: 18:41

Having the agent based approach to things and to the point where like, I think, you know, like having, um, some intentionality about, uh, going out and seeking the kind of answers that you're, you know, being proactive about sort of like overnight, here's, here's all the things that happened that you should be paying attention to, you know, maybe this model is now better, so I'm going to shift my, you know, workloads over to this model and, you know, like piecing together this whole much larger ecosystem that may have a lot of, you know, variable components in it really seems interesting. And, you know, we've seen different approaches from the big model people around this, you know, like the mixture of experts and things like that in there and, and, and, and some of those, those types of things. And I think when you look at, OpenAI 01. Which has a very different approach to, um, it's, it's kind of, you know, we, we, we talk about vibe checking, you know, like running it, which is kind of similar to what you were talking about on the, the insurance, you know, example you gave, you know, go and checking and see, does this make sense kind of thing, but, you know, like checking to see if, does this sound like it's AI? Does it, is it full of hallucinations? Is it, you know, doing, doing those kinds of things? It, internally does a lot of that, um, and, and, and checks itself and, and, and, and tries to come up with better answers based on some of those, those, uh, checks. Um, do you feel like that has, that has kind of not, uh, achieved what they, they wanted to achieve? And how, how do you think some of these big, bigger models are going to look at some of this, the newer developments like that?

Mark Kurtz: 20:22

For the big model companies, I mean, and actually this is mainly specifically at OpenAI, uh, just because Sam, Sam Altman has come out multiple times and said he's trying to do gen AI and he wants one model to solve everything, right? Yeah. Uh, if you look at Meta and Yama Kun. Uh, his, his initiative, he very much sees these, I think, aligned with how I see these, which are, they're another tool that we can tweak and train, and if we scale up the size, they become a little bit more general, they can memorize more data, right, but, uh, and they're an interesting tool, but I don't think they're on the path towards, uh, towards Gen AI, and this kind of gets into the O 1 or the strawberry release, which it, um, and it's also And how many hours

Chris Brandt: 21:01

are in that strawberry? That's right. That's right. Yeah. That's right. That's right. For those of you who are not familiar with that, there was a big problem of, uh, open AI or AI's in general, identifying how many hours are actually in strawberry, typically coming up with tube. And that's largely because it's all tokenized, and they're not looking at the actual word. They're looking at the. You know, the token. So anyways, I'm sorry. I didn't mean to interrupt you.

Mark Kurtz: 21:24

It was, uh, it was a great, great name that was blowing up. Um, but, uh, but yeah, whenever we're looking at, you know, Owen strawberry and especially looking at how open AI is shifting very much to a commercial company, something like strawberry is a great move for them. Because it's just processing more and more tokens to get a single answer. You know, we're going up by 40 X processing, uh, 40 times the compute to get a single answer. So it's a great move for them just to get more capital coming in. And obviously they probably, whenever they initially started it, they probably weren't going after that. But, um, but it does, well,

Chris Brandt: 21:59

Sam Altman's great at raising money for sure. He's got that reality distortion field around.

Mark Kurtz: 22:05

So they're, they're all going after these one model. Uh, solves all and, uh, and becomes again. It's just a huge, huge surface area to try and track and work with. And whenever we get into strawberry or things like these, it doesn't necessarily get rid of the hallucinations. And ultimately, the problem statement again has to be. Logically testable by that by that model and even within that unless if the model wasn't trained on that specific format, then we're going to hit issues with with hallucination and by even within its logical thinking. So it's kind of gets into is strawberry an efficient way to solve. Those problems, because there were multiple that came out in the past that were more, um, uh, that were more evolutionary cell algorithms or reinforcement learning, things like that. That solves all these math problems. Very well. Uh, you know, strawberry starting to set new benchmarks, but it's 1 of those things that if we just. scale the reinforcement learning that had toward your earlier point of how many hours are in strawberry. The reinforcement learning side has a very specific mathematical language coming in. That has a strict structure around it. It makes it a lot more testable as compared to natural language. Right? So if we just applied the same scale to those techniques, where would those end up? Right. And is that going to be more efficient than trying to have next token prediction, solve everything. Right. Yeah. Just based on natural language.

Chris Brandt: 23:29

Yeah. And I, I think scale is an interesting, uh, discussion around that too, because you know, like open AI, right? So it's 6. 6 billion of VC funding. Um, and like you mentioned, Sam Altman has said, stated that I'm, my intention is to build an artificial general intelligence, right? Which is, you know, really a big deal. Um, yeah. And at the same time, you're seeing a lot of people leaving OpenAI. I mean, you know, like Ilya is gone and, you know, who's sort of the conscience of OpenAI, a lot of people, you know, would, would think. And, you know, like, so a lot of the talent has sort of left at the same time they're raising all this money and, and, and trying to build an AGI, which means that they would have to massively scale. And, you know, just in terms of like pure compute power, the cost of that, I think is going to, you know, It's far exceed the 6. 6 billion that they've raised. So you know, I, I, I just wonder, you know, it looks like a huge amount of money in this, you know, potentially multi trillion dollar, you know, market that's out there that, you know, hasn't materialized yet, but is in theory going to materialize. I, you know, I just see, I just see that kind of as a bit of a risky bet, you know, just going as big and big and big as you can.

Mark Kurtz: 24:49

Uh, if I was in charge of a fund that big, you know, through, uh, through these people at OneHand, uh, I, I would not have invested, uh, to that level. I think OpenAI was, uh, Well,

Chris Brandt: 24:58

that Microsoft and Apple backed away from it, right? So, I mean, Absolutely. I think that says something.

Mark Kurtz: 25:05

Yeah. Uh, and I mean, Sam Albin is, as you said, probably, he may be the best fundraiser that, uh, uh, that we've seen in kind of the modern, modern startup day.

Chris Brandt: 25:14

He's got that kind of Steve Jobs ish, you know, like flair about it. You know, it's like, Hey, this is going to be amazing. You know, one more thing, you

Mark Kurtz: 25:22

know? Absolutely. Um, but, uh, but yeah, I mean, towards your point, it's, Yeah, they're trying to scale out compute. The question is, is that the right pathway to go down? My opinion is no, because fundamentally, when we're working with any kind of a gradient descent or deep learning, we've seen these hallucinations, we've seen these issues through every single stage of model and application that these have come out there. They're not going to go away. Without another technique and, um, and also, I mean, it's just it takes so much to scale. Right? I mean, there definitely is a scaling law, but it's also that scaling laws of power law. Right? So to get to the point of the benchmarks that we're looking at now to increase those, even by 2 percent we're looking at going from 405 billion. Parameters now up to maybe 800 billion or a trillion, right? And it's something that doesn't become scalable to deploy, and it also doesn't become scalable to train and towards your point earlier again, about running out of training data, what is that compute going to go to, right? So, you know, if you can, if you can generate data that memorizes the world. Uh, then it's a perfect bet. It's a perfect play, right? And you can create that data. You can push it out. But as we start running out of data and we start running out of specializations, it's one of those things that these agent based systems, these specialized systems to be able to solve specific problems and be able to control the interactions between them become a lot safer of a bet because you don't have to spend anywhere near the amount of money, anywhere near the amount of training data, uh, to be able to create these. And, uh, yeah, it's, it's one of those things that I don't think is gonna, you know, work out in, uh, in the near future for them. Uh, and I think they're, you know, they're, this is probably why they're going to that commercial play.

Chris Brandt: 27:15

Well, and I, I gotta imagine Sam wants, wants to get paid for all the work he's done. Oh, absolutely. That's probably a big, big part of, of all of it. Um, so, you We talk a lot about OpenAI because they're kind of like the 800 pound gorilla in the space at the moment. Um, but there's a lot of other really interesting companies. I mean, there's a million other companies out there, but I mean, in terms of the model space, you mentioned Meta, which has Lambda, which, um, you know, when you look at, you know, MMLU benchmarks and things like that. And, you know, to, to the, to the parent, the, their, their performance based, you know, relative to their, uh, cost, you know, they're, they're, they're actually doing really well. And, and as an open source model, I think that, um, that is a really compelling sort of thing, but then there's, you know, the anthropics of the world and, you know, and then, you know, like you have Ilya's company that's going to be doing, you know, focusing on a safe. You know, version of a I, you know, right? And so it's gonna be interesting to see what comes out of that. And at the same time, we're seeing, you know, people leaving open a I and, you know, publishing the models like here's how you build jet GPT to here's all the scripts and everything you need to just create your own version of it, right? Um, So I think on the one hand, this, um, some of this stuff is getting easier to do. I don't think at this point, you know, creating a large language model is really that difficult, you know, like making it. useful and improving it and tweaking around the edges certainly is, um, do you see, you know, like, do you see the open source world winning out in this battle? I mean, I, I feel like it's, this is something that is going to become so pervasive and, uh, it just having some sort of public kind of model of that sort really benefits everyone.

Mark Kurtz: 29:10

I see you open source. Essentially critical towards towards these pathways. And this is primarily on two fronts. One is from a safety perspective, right? Just being able to come in. And I mean, you know, obviously, a lot of these companies like, like meta and Google and others that are pushing out public weights, they're not pushing out the data sets and, you know, into copyright issues and legal issues if they were to do that. So, yeah, I mean, Yeah, definitely not doing that and makes it a little bit less open, but it's still one of those things that you can see the way you can inspect them. You can do whatever you want to try and perturb them and figure out where where these prompt injection attacks lie and things like that. Right? So you can figure out a lot of the risk exposure on these models, which we can't do to just a closed API. So it's kind of one of the one of the sides is just around safety. Um, But the second one is just around the whole point of what I was raising earlier. Is that the more you can test and evaluate these models, the more likely it is that you can deploy them into these specific scenarios. And I don't, you know, I've never talked with a young lagoon or anything like that, but I imagine or Zach, but I imagine that's really where their focus is. They can push out these open source models that they have a large investment in that they want to use internally, but they want. As much testing on these as possible, right? And that's what you get. And additional

Chris Brandt: 30:34

people developing for it too,

Mark Kurtz: 30:36

right? Absolutely. Absolutely. The stack around it, everything. I mean, these are immense projects that they're putting in millions of dollars into, and just for a single training run. And yeah, they want to get as much legwork on that. They put out as possible and it goes towards exactly as you said, the stack that they can put around it and get from open source to test and they can get from it all of that, which just push the pushes them leaps and bounds ahead of where closed source is going to go. I would say in probably the next year or two.

Chris Brandt: 31:03

I do want to like touch on one other aspect to, um, of all this. That there's a lot that A. I. is doing behind the scenes. That's really interesting. I mean, because what you know, at the start of this, we talked about how just rapidly everything is changing and you know, there's a lot of, uh, you know, and I saw some stat about the amount of code that's being generated by A. I. Now it was like 57 percent of code on GitHub or something like that is being generated by A. I. But you know, the thing is that there's a lot of. Uh, grunt work in, in developing software that, that AI is actually really doing a good job, like just from testing and, you know, like continuous integration, continuous deployment models and things like that. And the rate at which, you know, AI is helping people sort of turn over some of these issues and identify issues and, and resolve issues. And I think that's part of why we've just seen this, um, it almost feels like an exponential leap in development. In some regards, you know, I just like the amount that that has changed in the tech space in the last like three years is like more than I've ever seen in the industry and and I just feel like that's just still accelerating, which is both good and bad. I suppose. Right. Um, do you, what do you see as you know, somebody who's in in the mix of it, you know, doing the development work and helping people develop, you know, their, their Yeah. Their systems. How do you see A. I. Impacting that end of it?

Mark Kurtz: 32:38

Just from a personal anecdote. I use it all the time. I have both both copilot and chat GPT and all those just to well, and we have some internal models as well to, uh, To try to bring down bring down some of those costs, but I use it all the time just because it is 1 of the things that you said is doing a lot of the current work and getting a lot of these things that I don't want to think about a specific name of a variable or remember how to write a for loop across 6 different languages. things like that, right? So, um, or all the syntactic differences and, and things like that. So, well, you can, you can

Chris Brandt: 33:15

actually write in your language of choice and then have AI converted to whatever you need the project to be in, right? Yeah,

Mark Kurtz: 33:21

absolutely. Absolutely. So I think that's, that's something that we'll continue to build. And, uh, and you're absolutely right. And in terms of, uh, what I've seen a lot of as well to be able to augment. The core code and be able to plug in, uh, CSP program or CSP projects and testing projects and all that good. Right. So it is very much starting to augment and inject itself in our current world. But obviously there are, you know, a lot of IP and data privacy issues that companies need to figure out themselves to go through, uh, as they expose that to, uh, to their internal. Internal engineers and things like that. But the 1 thing that I would caution with and say is that it is a great tool if you're experienced, right? Yeah. And it's mainly just because it is going to make mistakes. And unless you, unless you Um, you know, you have that guiding knowledge to go over the top of it and say, Hey, no, fix this, this way, fix this, this way. It's something that can lend itself towards a lot of potential, uh, mistakes and, and open yourselves up to some liabilities and things like that. So, you know, it's a great tool to, as you are an active coder and have gotten experience, but I definitely would not use it to scale up on maybe to do some quick projects and give you an example. But you really got to play around and, and, you know, inject yourself into that to understand what's wrong with the different pieces. And if you're just getting, you know, 2000 lines of code dumped out at you, it's hard to go, it's hard to go through and learn exactly what is, what each of those lines is doing. So that's the one caution that I would give around.

Chris Brandt: 34:58

Well, and, and it raises the, the question about You know, like more supply chain type of attacks too, because if you've got a model that you can inject some, you know, malicious code into what it's building, you know, somehow through the prompt engineering or messing with, you know, the training data or whatever it is, you know, messing with the model, um, that, you know, and, and people, you know, at some point are just going to be like trusting the output and just moving on with things. And I think that's, that's a very dangerous thing. I saw an interesting, um, It was a teacher who came on and she was talking about all this existential dread and sort of like, what is even the point of me doing any of this stuff? Do these kids just want to use AI to, to do that? And then, you know, somebody then came back and said, well, I would say that you're teaching methodologies are, are, are, are wrong. are failing you because this is sort of the new thing in the world, new paradigm. And what you need to do is teach your kids how, how to use this. And I, I think, I think that's kind of what you're talking about here too, right? It's, uh, cause there's this one teacher I know, a professor, she was having her, her students write a paper using AI, and then they were bringing them back and sort of reading through them and saying, you know, like, how can we, you know, Edit this, how can we identify things that are weak writing? How do we identify the source of these materials? How do we identify, you know, like just really leveraging AI, but you know, like really going through that process of, you know, vibe checking it as you know, is often referred to, um, You know, do you, you know, as somebody who's working with a lot of people in AI and probably hiring a AI, do you, do you find that like, um, people are coming to this kind of lacking those kind of, I don't want to say like critical thinking skills, but I guess it is kind of a critical thinking skill, you know, to tackle a, these AI challenges.

Mark Kurtz: 36:47

It's a great, great thing to bring up and a great thing to talk to. I think, um. As you said, it is another tool that people need to adapt to and want, you know, want to adapt to, right? I think, uh, I forgot when I saw it, but this was, uh, uh, there was some quote from a old, old history book talking about, how people were moving toward, you know, pencil and paper, right? Rather than just memorizing everything. And it's one of those things that, you know, technology is going to keep advancing. People will be hesitant to adopt. But I think you're absolutely right that this is something for that I think will help out. A lot of people, because it is one of those things that it fills in a lot of that ground that legwork for you, and you can come in and then focus your efforts on what we need to be doing. And as you said, that's really, I think, and, you know, I'm not nowhere near an educator. So definitely don't take my opinion for, um, uh, you know, as a truth here, but I would say, you know, be able to adapt and say that here's the groundwork that. These tools are giving you and be able to do the critical thinking on top of it, be able to adapt it, be able to personalize it to your tone of voice to your opinion. Things like that. Those are really critical because if you don't adapt, people are just going to do it behind your backs and you're either going to race to try and build out the tools to detect it, right? Or you're going to race to try and build people.

Chris Brandt: 38:11

Well, now there's always, there's a, there's a lot of discussions around like which AIs are best for doing those vibe checks to, to make them sound less like AIs so they're virtually undetectable. Yeah. Yeah. So, so tell me, like, uh, so what, what's, what's next for, for neural magic? What do you, where, where are you guys going?

Mark Kurtz: 38:30

Yeah, yeah. So looking at, I mean, looking at everything we've talked through here, we do see a big investment happening in Gen AI, uh, across everyone. And, uh, we've always been very focused on efficient deep learning and, and being able to make that both cost efficient and energy efficient. And this is where we're continuing to double down on. So, you know, today we're actually pushing out, uh, compressed models, Of all the top gen AI models. So these are ones that we've come in and removed a lot of the redundancies from these models. And this is also kind of, you know, call back towards the earlier point. If we need to find better algorithms, all of these models we can compress significantly. So there is a lot of space left over and these models that is wasted me training optimization space that we can essentially either, um, Reduce the precision of through quantization or just outright move the connections through pruning. And so this is what we really specialize on. And ultimately, this means that we can get, you know, for 4X up to, um, and some of our earlier numbers, not on the latest models, but working on getting those up to date now. But, uh, yeah, up to 10X on, uh, on some platforms. So, right. So a 10X improvement in your performance. 10x cheaper, 10x more energy efficient. So this is where we're really focused on is being able to make all of that as easy as possible. So you can build out a control plan essentially around your Gen AI applications. Put all of these pipeline pieces around that. Make it, uh, make it a system that you're confident in deploying and make sure that it's not going to break the bank or, you know, significantly, uh, hurt the environment ideally. So this is everything that we're working on and really pushing heavily on. Uh, and, you know, a lot of that is made up of open source contributions as well as, uh, some enterprise licensing.

Chris Brandt: 40:18

If somebody wanted to get some of that, uh, compressed model goodness, where, where should they go?

Mark Kurtz: 40:23

Check out, I think the number one place to start would probably be our HuggingFace org. Uh, so neural magic on HuggingFace. We have all of our open source models pushed up there, uh, with instructions on how to deploy them, all the evals around them. So you can check them out and run them locally and, you know, ultimately, uh, hopefully not need. Uh, 120, 000 hour server to deploy one of these models, um, so that you can run them locally on your machines and, uh, and on smaller servers.

Chris Brandt: 40:50

Yeah. Hugging, hugging face seems to be ground zero for AI right now. So yeah, that's, that's pretty cool. And it's cool that you're, you're, you're distributing them open source and doing all that too. I, I, I think that's super valuable to the world. So thank you Yeah, definitely.

Mark Kurtz: 41:07

Definitely.

Chris Brandt: 41:08

Well, it's, Mark, it's been absolutely awesome talking to you. I, uh, could start on about 30 more topics that I would love to talk to you about right now, but I don't think anybody would still be here when we're done. But, uh, it's been absolutely, uh, enlightening talking to you and, uh, great fun and, uh, good luck with everything you're doing in the, uh, world of the wild world of, uh, of AI.

Mark Kurtz: 41:33

Thanks, Chris. I appreciate it. And, uh, yeah, excited, excited to see who else you bring on because I've been a fan for a little bit and it's been really great seeing everybody you've been bringing on and all the insights you've been bringing.

Chris Brandt: 41:44

Awesome. Thanks so much. Thanks for watching. I would love to hear your thoughts on the state of AI in the comments below. And if you could give us a like, think about subscribing, share this with a friend, and I will see you in the next one.