1:01All right. Hey, everybody. Super, super pumped to have you here. I'm Jacob on the DevRel side. We've also got Alex here, too. Howdy, howdy. So Alex and I have been doing lots and lots of really fun diving into semantic layers and context, and how the heck do you get all these pieces to work together?
1:25And so what we want to share today is just a little bit about what we've been up to and what we've learned. You know, I think this is in as much as this is a like a conversation between Alex and I, you know, I think it's pretty open.
1:39Right. We're we are, you know, Alex and I both kind of come into this work as people who have worked at pretty big companies on the analytics kind of as analysts. Right. So people who are like, how do we apply all this really cool stuff? Right. And so I think like a lot of what we're thinking about today is like, hey, like this is an exploration of like applied.
2:03Right. And so we'd love for the for the comments to really focus on, you know, or not focus on, but like, you know, Q&A. It's like, let's talk about like how we apply these things. What does it mean to use AI with your database? Like, how do you actually make it work? I think those are all like the really fun things that, you know, are getting us really pumped about this.
2:20You know, in the in the context here is. Let's see. We are talking about, does AI need the semantic layer? Right. Which is super fun. And I'm going to jump in just just a few slides just to to put everyone in context about what MotherDuck is real quickly here.
2:43So I'm gonna throw this on the stage. So MotherDuck, we like to talk about infrastructure for answers. Right. So how do you actually get the value you want from the data you have? And I'm going to go through a couple of slides here. We're not going to spend too much time on the slides. Obviously, this is why we're all here. Right. It's all up.
3:05Everything up. Everyone's using more agents. It's wild. It's wild and crazy. You know, from five percent to 40 percent from twenty twenty five to twenty twenty six. That's that's wild. That's a big change. That's a big change really fast. And what we've also observed is that, like the way that a human and an agent query these things together are very different.
3:29You know, humans are kind of smaller scale, consistent, small printable workloads. Agents are just going to jam queries straight through this thing. Right. And so that means old architectures or as I like to call them, legacy architectures. I love that word. This is my favorite marketing. We are modern.
3:51Yeah. Agents don't query like humans do. So what do we do? Well, we we need to think about a new architecture. And so this is what MotherDuck does. We we are building an architecture that works really well for agents and that is backed by DuckDB. So, you know, I'll talk about three things, three things here.
4:13And then I think we just hop in and really talk about the meat of the conversation here. Fully isolated compute. Right. What's really cool about this is it means that if I'm running a query or my more likely my agent is running a query, I don't block Alex. That's great. My favorite one is if the CEO is asking a question, they get their answer just as fast as if the entire data team is asking questions.
4:36Yeah, exactly. Or like a big if a big pipeline is running and like, you know, the CEO, the CFO is up in the middle of the night and there's, you know, all your data is refreshing and they need to get some data for a board slide. They're not just waiting and spinning forever. Right. Not that we've been there or anything.
4:51Yeah, not that I've been there. It's like, yeah, where's your slide? Where's your slide, man? It's like, I don't know, like SQL servers running. We scale up and we scale out, you know, so what that means is you can run on really big nodes. Our biggest node is 192 cores. Wild. And you can get as many ducklings as you want on the same data set with zero copy cloning.
5:16Makes a lot of sense. And then, you know, you can limit these workloads by resource. Again, we talk about no, no resource contingent, which is excellent. OK, we will just we'll talk about this to all your friends get to play with MotherDuck also. A lot of fun. Right. Get all these pieces in. You probably see a programming language that either you know how to code or your agent knows how to code in there.
5:38You see tools that are UI first and text first. Lots of really, really cool ways to fit all this together. Because what is a flock without friends? All right. I'm going to stop sharing slides. I mentioned as well, just it's not on that slide, but it's also a bring your own agent scenario. That's right. That's right. You know, I think a very important ecosystem in 2026 as well.
6:01So we're not locking you into a choice of agent or model or anything like that. We slot in with a remote MCP so that way you can you can bring your own agent, your favorite work environment. And wherever you're already getting your questions answered, we can help. That's right. Absolutely. That is exactly right. All right, Alex. So we're going to talk a little bit.
6:21You know, I think I think the first thing I want to talk about actually is like semantic layer and context layer. Like what? What is the difference between the two? You know, what does it mean? Like is it marketing? Is there real differences? Alex, why don't you just take us into into that? Yes, absolutely. I think this is a space where if you're confused, I'm right there with you.
6:45You know, I was very, very confused just even like a couple of months ago on this topic where it is evolving very, very quickly. And I think that we're on the leading edge of this in a lot of ways where we're we're we've been doing a lot of experimenting. We're going to share with you some of our experimental results as well. You know, so it's very much an evolving environment. I'd say the part that's been around the longest is the semantic layer.
7:07So this is something that's been a concept since, you know, the Bob Jay days, business objects, MicroStrategy before they became a Bitcoin company. Right. The original MicroStrategy days. So really, the concept of a semantic layer and a context, they're actually independent.
7:24And so a semantic layer historically was really a way to translate drag and drop point and click UI's into SQL queries to then be run. So that's sort of the historical side of it. Now, modern times has changed a little bit. Exactly. The main focus of that right focus of the semantic layer now is not around enabling draggy droppy UI's.
7:48Really, it is a translator where you can take a set of columns and a set of metrics and then compile it down and kind of convert it into SQL. That has some advantages as well. And it is an extra layer there. Right. So that's the the give and take of it. It's an extra layer. There's some benefits. There's some some pros and cons either direction. How does that really relate to a context layer?
8:11I think the context layer is a little bit more modern, unless if you think back a little bit, what this used to be called were just docs. Now, that's minimizing it a little bit. Right. A context layer is more than docs. But, you know, we we're here for the hot takes. Right. So hot takes are that this is not entirely new either.
8:29The context layer is about how do you navigate your business's relationship with data? That's not new. What's new is that agents are having to navigate it and not humans. And so a lot of what the context layer is about is not just documentation in a way where a human can navigate it.
8:46It's the relationships between concepts, relations between documents and a way for agents to navigate it. So what we believe is really that these are not identical. And the key differences there are that the semantic layer is that translation layer.
9:07The context layer is a lot more about where to look, how to understand the data. What's the right way to use a semantic layer? What's what is the how do all these tables and views and other data sets fit together? So both of them are read by agents. Both of them are used in, you know, in the modern modern way.
9:26We don't believe that either one really, you know, they perfectly overlap. Yeah. How would you how would you add that to that, Jacob? I would just add that, like vendors, you know, in the modern data set space or the duck stack, as we like to call it, kind of use them interchangeably.
9:50And so a lot of the time it can be confusing. I think snowflakes, snowflakes, internal context layer or what I would call context is branded as semantic snowflakes, semantic something. So, like, it's, you know, I think a little bit hairy there.
10:10So just something to know. I would also add that, like, you know, there's very valuable purposes, I think, for for semantic layer around a couple of concepts that we like to think about in the space.
10:22And we that we saw, like, for example, with, you know, Looker back in the day, I guess, like, you know, 10 years ago now, but it feels like forever, you know, which is really it's great to have that contract between your consumers and producers and keeping that together. I see a comment here, which is how does context layer fit in with ontology? Alex, you want to talk about that?
10:43Well, first, I need to put my pinky out and, you know, really get fancier here to talk about ontology. Right. I think I think, you know, ontology is a similar way to think about this. It's it's not entirely unrelated. I think, you know, if you're talking to someone not in this data world, they would call all of these synonyms. Right.
11:04So I think ontology is one of those three in that same bucket. A lot of it is around the relationships is the ontology. Yeah, that's right. Yeah. That's the key piece of it. We don't think that's the overarching term. We think that's more of a specific subset of context. If we had to think about breadth, right, we'd say context is maybe the broadest ontology is a subset of that semantic layer. Similarly, a subset.
11:27That's right. Yeah, I would just I would just add that, like, I think, like the main thing in like how do these tie together? So if you're using Markdown, some of you, some of you may already be kind of building your own ontologies in that. Right. Which is using the front matter to link. Hey, this is related to these other concepts. Right. I see a comment here on LinkedIn that says when you talk about semantic layers, talking about defined aggregations, like traditional QB talking about knowledge graphs.
11:49I mean, great question. I think this is all like in the same space. Like, it depends who you're talking to. I would say that I would talk about defined aggregations, but like people talking about knowledge graphs would be like, oh, yeah, ontology. Right. Really? That like, you know, a semantic layer is part of the ontology or whatever. Right.
12:08I think like these things are all very interrelated. And so like pulling them apart is actually fairly difficult because any good context layer will include ontology because that's how it's useful to your agent. Right. If you just give it like a like a single file with, you know, hundreds of thousands of lines, it can't actually navigate it. Right. And so we use we need we need some sort of way to make it useful and break it down into smaller parts.
12:30I really think kind of about I think about a lot of this stuff is like I think about a book. Right. A book has a table of content, a table of contents. Right. It tells you where to go to find certain things. And then back it has an index. Right. It says, hey, these concepts can be found here. And I think we can apply that very, very linearly or like not linear, but like directly. Right.
12:48That's like that's how context and ontology works here, too. Right. It's just like telling you where to go to find the thing.
12:56Anyways, so so when we're when we're talking about semantics, what we mean with semantic layer is specifically like defining, you know, how your data is related in in a kind of semi technical language that, you know, expresses joins and pre aggregations mostly. Anything you want to add there, Alex? Yes.
13:17And, you know, some some filters as well. So not not too dissimilar. I think one of the key differences there is that when I think of a semantic layer, we're establishing the relationships, but not strictly the granularity. And so the key use case for that, of course, historically was because depending on what people would drag and drop into your pivot table, that would redefine the granularity on the fly.
13:41So that's a key value of a semantic layer is that redefines granularity. I will say that that's a very concrete mechanism for it. But AI does this on the fly with context as well. If you put a SQL snippet in your context layer and you ask a slightly different question, modern models are able to remove columns from your select statement and your group statement and give you an aggregate, a different level of detail.
14:05It that same concept can be done in a context layer, whereas, you know, two years ago, the models really weren't smart enough to do that effect. That's right. I mean, it might even be I mean, I think, like, really, the watershed moment for a lot of this stuff was Opus four, five or four, six kind of November 2025.
14:24Like all of a sudden, I think we really saw a big step change in the ability for it to accurately navigate context and be able to give solid answers. They were all they were already kind of pretty good at, like, writing SQL in the abstract, like it could write SQL that would run. But like adding like having some intelligence to navigate around it, I think, is relatively new.
14:47Yeah. All right. So hope that answers the questions around there. Obviously, feel free to feel free to throw some more stuff into the chat if we want to really kind of drill into it. But I think what we're going to talk about next year is just like I want to give you all context for how we thought about an experiment that could test these notions of kind of semantic layers versus context and how to use them with agents.
15:11Diagram, I've got a handy. Yeah, let me. Yeah. You want to you want to pop it up, Alex? And actually, why don't you drop a link? Can you drop a link to the article you wrote about it, too? Because, I mean, we've published quite a bit around this and I think it's useful for just sweet.
15:23Thank you, Alex. So we've been running a lot of this on a somewhat recent benchmark that was built for this called Dabstep. When it was really shipped, our models were about to get 10 percent, I think, 10 to 12 percent correct from a question and answer perspective.
15:46So fairly difficult for the versions of agents like I think for maybe three point five or something. And, you know, saw it around the same that point when they launched this. And so the way we think about it is we distill it into 26 question answer pairs that broadly represent kind of the scope of all of the questions.
16:10And we take that along with a manual that defines relationships that they provide as part of the benchmark and we turn that into semantic models using Fable. And we are using we're building this on Malloy, but I think any, you know, any open kind of open query language would would work fine on this.
16:30And I'm sure you could probably like dial in Aussie or whatever, too. But we use Malloy. Obviously, we are big fans of that work. So within that, you know, now we have a generated generated model and we can evaluate that against the remaining held out set of 400 question and answer pairs.
16:52So we want to see is like how well are these questions general? You know, is this how well does a specific question generalize to a broader set of questions? Right. How do we train on if you think about like analytics in general, like how?
17:07The thing we really care about from context and for semantic layer is like how well does can we find questions that generalize well to a broader subset that we actually don't know. Right. This is a discovered set. Right. We do. I think analytics in some extent is a little bit like a little bit like archaeology.
17:25Right. You have to kind of dig to find the thing you're looking for. You don't know it until you find it. You just know there's something there's something down there. We just got to, you know, break out the pickaxes and look. Anyways, so that's this is kind of generally how we set it up again.
17:42So the core primitive that we leaned on was was these 26 question answer pairs plus the manual and then just, you know, running that in a loop with with our best friend Fable and saying, OK, now we have a set of of semantic.
17:59Models that we can now run against a broader set and see if it works. We did this very recently. Right. So we were running this on Fable and Luna. So really, really cool to see how it works and a little bit that I made this diagram and I, you know, since then said, oh, I wish I put another box on there. So I think one simplification that this makes is that we created this semantic model here.
18:22But we actually ended up creating a semantic model and a context layer. That's right. So if you look at that data manual step at the beginning, we actually ended up converting that into a series of mother guides. So mother guide is is is the context layer that mother provides. It's a way for our MCP and your agent to be able to go and retrieve context about your data when you ask a question about it.
18:46That's right. And it has a set of tags to where it can navigate because we just had a semantic model. The natural question for the agent is which part of the model to use, which model of these many individual table models or kind of entity models, rather like a customer model or a revenue model. Right. Which do I pick?
19:06And in many cases, the context layer helps you then accurately use the semantic model underneath. Yeah, that's right. I think like the thing that I would just add there is is a lot of the times we use very imprecise, imprecise terms. Right. And so what the context around the semantic models lets us do is say, hey, when someone says customers,
19:29that means this model is the model to use, for example, or when someone says active customers use this one over here. And if they don't specify ask. Right. And so there's lots of ways to kind of tune these things and make them work together. Again, this is this is kind of just the perspective we took from from the experiment here.
19:50Awesome. OK, Alex, when you pull that down or take that off and let's let's talk about what we've learned, like how we how we discovered this. Actually, before we do that, there's a really good question that just popped in from Nick. I'm going to throw up here. This is for you, Alex. Data manual could, in principle, be rather unstructured, even multiple snippets, since the runtime reference is the derived downstream
20:14artifact semantic layer. That's a great question. There is not a requirement for the manual to be all in one document. That's right. It's definitely a series of documents. And we did split it apart as a part of building our context layer. So, Jacob mentioned earlier as well, we don't want to hand a model all of the documentation at once. It'll blow through its context window.
20:37It'll either exceed the context window if you're at a large enough, more complex enough data environment, or it'll just slow it down and confuse it. So it's really a study in progressive disclosure. That's right. Give the model exactly what it needs. And so there is a destructuring that has to happen just to work well. And I think you're hitting on exactly right on that topic. It's how do you give it enough at each step
21:00and how do you break it apart in an efficient way? That's absolutely all part of it. Yeah, absolutely. All right, Alex, let's talk about what we discovered. First, what we learned as we ran this eval with different models and different approaches. You bet. I think the first thing is that we built a semantic model
21:23using Malloy and we wrote zero lines of Malloy. Yep. And I know some Malloy, but I could not have written the Malloy that was written. It was using some pretty advanced things and it proved that the models can write Malloy. And that was our key thing that we set out to answer is we know models can write SQL. Can they also write Malloy?
21:48Because it's a different language. And we know that it's not consistent across languages with these frontier models. And we found that it could. So concretely, how do we justify, prove that as well? When we did our SQL benchmark on the same dataset, SQL was able to score 100% using the Gemini 3 flash model and similar scores for LunaMax and others.
22:11With Malloy, we did find that it was unable to get exactly up to 100%. It was at best at a 95% level in the experimentation that we did. I think that we could have continued on and brought it all the way up to 100. I think for the amount of effort that we put in across those two things, we were able to bring it all the way up to a 95% accuracy level. That was interesting.
22:33We had expected that to be able to bring it up to 100% exactly the same way. I think we have a number of different assumptions around that. It's tough to be conclusive. As we know, LLMs are probabilistic. That's what we love about them. But as a part of that, it means we don't have an exact answer for why. Part of it is there's a lot of SQL on the internet and models are very good at it because they've seen a lot of it.
22:58Well, I think, Alex, it's worth splitting actually kind of concretely on that point, right? Sure. Why, so when Alex was talking about in terms of comparison here is we gave it a context that was context and semantic models, and then we gave it context that had SQL snippets in it, right? And so what we found is that,
23:24what we found was basically that the models were better at manipulating the SQL from the training set to make it cohesive for the held out test set. Whereas with the, with the Malloy or with the semantic layer, right, where it's like more specific language,
23:48it was less able to kind of manipulate that to find the answer. And so that really explains where the gap was. The gap was 100% in, not that the models weren't smart enough, but they did not have the facilities at that time to manipulate the kind of abstraction on top of SQL, which was the semantic layer to actually kind of find it. Really interesting question actually coming up here in chat, Alex, I'm gonna throw to you too.
24:12Sure. So if the output is Malloy for the agent, what's running the SQL? You're preventing the agent LM calls from running any SQL directly. This is a great question. Yes. So that's it. The answer to the second question there is yes. So what we ended up doing is we had the, the central model was a set of context, markdown documents that we were hierarchically organizing
24:34in guides and then Malloy models, dot Malloy files. And so that's what the output of kind of the fable class build step was. Now the evaluation step using a lighter weight, more cost-effective model like Luna, that was actually writing Malloy as well. And it was not submitting a SQL statement as its answer, submitting a Malloy query. And then Malloy has a set of compilers,
24:57one for each database that it supports. It supports many, but one of them is DuckDB. So MotherDuck is built on top of DuckDB. We speak the identical dialect and use the same library as a driver. So it was compiling into DuckDB SQL and executing that on MotherDuck and then retrieving the answer and then validating it from there. So it did compile down into SQL.
25:22Related to this question and what we were talking about before a little bit, the DAPSTEP benchmark in that manual contains SQL snippets. So in some ways it's a little bit of a home game for SQL and a bit of an away game from Malloy because the very first step that fable has to do is take some pretty big, pretty gnarly SQL and then convert it into Malloy.
25:43So we actually found that in our initial experimentation, we allowed the context layer to contain SQL and Malloy. And it was hard for the smaller model of Luna to do that on the fly transcription of SQL into Malloy. It was a lot better for the fable model to do that and have the context in Malloy
26:06so that the evaluation could also be done in Malloy. And so we did actually move further away from SQL in our context layer for this Malloy. Yeah, we also, so also Malloy does include or in most semantic layers include a way for you to directly write and execute SQL, right? For something that is, so we had to actually remove that capability as well because what was happening
26:30is when it would get to a complex question, it would basically run the model and look at the compiled SQL and then say, oh, let me just modify the SQL slightly because this is easier for me to reason about than for me to actually manipulate the semantic layer itself directly, which I think is like kind of the core thing here.
26:54We actually were like, wow, it's performing really well. And then Alex and I, of course, looked at the traces and we were like, oh crap, it's like, these models are too smart. They figured out they could just write SQL. And so it was like, I think it was like on the first few times we ran it, it was like 40% of the time or something, it was just writing SQL instead of using the semantic layer. And so we were like, okay, we need to modify our harness to do this.
27:17Speaking about the harness, good question here from Neil. Did you use the Malloy Publisher MCP to find relevant Malloy sources for a given question? Do you want us to start with that, Alex? Sure. We were kind of building this actually, building it on top of some things that we'd already built prior to that Malloy MCP existing. We did collaborate with Lloyd, creator of Malloy,
27:41one of the creators of Malloy, as well as Michael Toy, another co-creator of Malloy and really learned a lot from them. It did influence our approach dramatically. So we looked at how that MCP works. We learned from it and adopted a lot of its techniques. We didn't end up using it directly because we already had another harness that was working effectively for experimentation. So it's a solid option out there. We ended up not using it for our experiment.
28:05Yeah, and I think like, again, part of this is, you know, we're going to talk our own book. MotherDuck has this notion of something called guides. And so we were able to use the MotherDuck guides to hold part of that and then, or our own kind of context lookup tools, right? And then built our own way to execute, you know, execute said Malloy files and all that kind of stuff, right? That's all part of the harness.
28:29By the way, I did link to it in the chat here. If you want to see, if you want to actually see the code and run it yourself, it should actually be fairly easy to adapt to almost any benchmark or to, you know, your own internal evals. So yeah, anything else you want to hit on, you know, in terms of what we learned here, Alex? Yeah, a couple other kind of key metrics. So we really wanted to understand based on, you know,
28:55there's some advantages in terms of a human writing these languages. And one of the advantages of a human writing a semantic layer is it tends to be much more concise when you're writing the individual query. You kind of can abstract the complexity in your core model and then have your queries of the model be simple, right? Like, I want to get- Let's hold on that thought. Let's hold on that thought, because we'll talk about benefits of a semantic layer,
29:18I think, because like there's questions in the chat that are right on that point that we'll get to. I think let's continue talking about kind of what we learned here. Because I think like, I think the other question is like, well, you know, we talked about it being less accurate. We didn't talk about like, is it faster? Is it, you know, more token efficient to use a semantic layer versus just SQL? You wanna talk about that a little bit? Right.
29:41It ended up where it actually used more tokens to work with Malloy. And what that means is that it just takes more time and costs more money or more subscription tokens. So that part of our hypothesis did not pan out in this particular experiment. So that was another interesting finding for us. I think a lot of it is around the model
30:05had to do a couple extra turns to be able to kind of bend Malloy to its understanding of the world, because there's so much SQL in the training data is one of our hypotheses there. So. Yeah, I think that's exactly right. Like there's 50 years of SQL training data in the training. And pick any semantic layer, there's gonna be an order of magnitude less, probably three or four orders of magnitude less.
30:29All of these things lived in private companies, not on the internet. Very, very hard to figure out how they work without ingesting the specific type of documentation. And then that's kind of like the only source that it knows, right? And so like a lot of it also, I mean, I don't know if I looked at the traces specifically, but I think, did we give it the ability to pull up Malloy docs? I think we did.
30:54Or like maybe we just gave it like the overview of the docs. I can't remember. But anyway. Yeah, okay. So like part of that is that like every time you're using, you know, this new semantic layer, you also have to give it, hey, like here's the nuance specifically of how to use this thing, right? That's not in the training set. And I think that that's part of it. So I suspect that like, you know, we're gonna keep monitoring the space
31:16because I suspect that like, you know, in the fullness of time, you know, assuming we also have jobs, you know, they will be much better at writing semantic layer. I'm joking, I'm joking. Not that fullness of time, man. Not the heat death of the universe. Right, right, right. But I think like, you know, I suspect that they will get, like in general, the models will get better at writing and understanding
31:42more arcane languages. And that would obviously include something like Malloy. Let's talk about benefits of the semantic layer. I think there's a, like we got a really nice lead in question here from Mario, which is like, hey, you found the models were not as proficient in Malloy. Like, why did you stick with it? Like, why do this at all? I think this is a really good question. It's a great question.
32:05And I think a lot of ways it comes down to what it feels like to put it into practice. So I'd really love to hear what you folks also have learned putting into practice semantic layers or not. A lot of it to me is around changes. And the only constant in business is change.
32:26And so if you want to make a change to how a metric is defined, what happens then? And so in a semantic layer, it provides a really nice abstraction layer where you can go ahead and make a change to the definition of a metric. And every consumer, the very next time they refresh
32:48will automatically, you know, use that new calculation. So that's a huge advantage where you can keep things standard. I think there's, you know, the, I don't love the word governed, but the word governed really does apply a little bit, right? You can have this source of truth that will update everywhere all at once. And that is very powerful. And it is more and more powerful
33:11the larger and larger your organization is. So that's a key benefit. If you do that in SQL, right? If you change a column in a table, someone goes to run a SQL statement, it's going to say, hey, I've never seen that column before. What are you talking about? And so it's a very different feeling when there's that type of change. So the second thing is that that's, when you want to make an update, you want it to propagate as much as possible.
33:34There's other cases where you want to go ahead and change your underlying model without having to change anything anywhere else either. So that's sort of the other side of the coin, right? Where there's this contract between the people producing the model and the people consuming the model. And that contract is the semantic layer. And as long as that kind of API surface is maintained, anything under the surface can change as much as you'd like without anything
33:59breaking anywhere else. So sort of you get updates when you want them, whenever you need them. And if you don't want things to change, you can keep things constant. So sort of like the, you've got your table with all of your glassware and plates on it and you can pull the tablecloth out from under it and nothing changes, so. Yeah. I think also Alex, you also mentioned earlier, and I'll just circle back on this,
34:21SQL is verbose. Also, so like- Pretty mildly. Yes, pretty, very mildly. I think like a lot of the, one thing that I think semantic layers are great at is very concisely expressing how to do something that is potentially hundreds or thousands
34:46of lines of SQL, right? Especially when you get into things that Looker and Malloy were great at, like symmetric aggregates, right? Percentage of percentage type stuff. Writing those in SQL, when Alex and I were looking at the traces and actually looking at the SQL executed for some of these things, it was like, my God,
35:08this is like 10 lines of a semantic layer and 150 or 1,000 lines of SQL, just like wild, crazy SQL. And so I think like, there's also an advantage of, it uses less tokens from a human perspective, right? Much more interpretable. Yeah, exactly, more interpretable, right?
35:33Now, that being said, like SQL is, because it's so verbose and designed for business users first, it is fairly easy to understand. I think it's all about adding extractions. I think, so Kyle threw this comment in, which I actually highlighted earlier, but I'll throw it up now. I think this is the right, this is totally the, this is the vibe at the moment,
35:57which is basically option one is you use SQL, option two is you start using and injecting all these abstractions. And the models are just like, clearly they're being RL'd to use SQL. So like, just, how much do we fight their instincts? How much will this be true in next generations of models?
36:19I mean, this is why we built eval in the first place, right? So I think like a lot of this is kind of like, hey, like this is where we're at right now. It's possible that OpenAI could drop a model tomorrow or Anthropic could drop a model tomorrow and we would have to rerun this and like we may have a totally different perspective on it. Right? And so I think like, again, we're thinking about how do we make this repeatable so that we can just like help other people who are going through the same stuff, right?
36:44I think that's really important. And I think that's a part of, while we were doing this experiment, OpenAI dropped their 5. 6 model series. That's right. And then they dropped the price of the Luna version of that, their smallest model by, you know, 4X, 5X, right? They dropped the price by 80%. And that actually brought it to be our most effective model in this use case where it was not before.
37:09And so even during the experiment, the optimal model changed and it changed quite a bit. So this is a very, very rapidly evolving space. And I think part of the takeaway here is check the evals. So what do we mean by that? That means like in your business, it makes a lot of sense to have this set up where you can get a sense of how good are the latest generation of models
37:33on answering our business questions. How good is our context layer? How well structured is our semantic layer? So these tests are becoming increasingly important. Exactly. You know, I think that to me, you know, for those of you who don't know, I come from an accounting background. So I'm always like, how do we know it's right? Right, how do we break apart the black box? And so of course I love evals.
37:58So this is very natural work for me to do. But, you know, again, I'm talking my own book here, but I think like the reality here is if you are serious about, you know, figuring out how to empower your users with access to agents and SQL or in your data, you need to figure out what evals to run to. I have to get in my favorite line about evals.
38:22You know, evals are from a certain region of France. Otherwise they're called sparkling data science. Yeah, exactly. So any recovering data scientist like me, there's hope, right? Evals are really doing data science on the performance of AI models in your business with your data. That's right. And I like to say that, you know, data science is just sparkling accounting. So anyway. I love that.
38:47I love that. Yeah. That's not fair to data scientists. So, you know, or accounts probably, but sorry if I offended everyone in this call. All in one shot. Yeah, exactly. Okay, so I think we're gonna go from here, just like deploying in practice is really where I think we're gonna go to. And I think like this actually leads really into a bunch of good questions we have. You know, how do you practically do this stuff, right?
39:13Okay, so let's see. Let's start with this one from Michelle. Have you tried building a domain specific harness as opposed to prompting general purpose frontier agents? I'll start with this one, Alex. We built our own harness. We were using OpenAI, sorry, not OpenAI, OpenRouter, now part of Stripe, to send our prompts. And we had our own custom system prompt
39:37that was built specifically for running this eval. We were not using Clod, we were not using Py, we were not using, you know, any of the other kind of common out-of-the-box harnesses. We built our own. And so I would recommend that like at, you know, once you really are serious about this, I can't imagine that you'd wanna use something general purpose here.
40:00and that means also, you know, you need to think about how you deploy it to your users. You know, that being said, what we've also found is that the system prompts and the affordances built within those kind of more generalized harnesses are pretty dang good at doing data analysis and writing SQL. So I wouldn't say there's been like a ton of drift between those two, but obviously like if you really, really care about a managed curated experience,
40:23you probably do need to manage your own, you know, a little bit more tightly. I don't know if you want to add anything else there, Alex. And there's multiple options in that space. You know, you can do fully custom harness, you can do fully open harness by the book. You can also have your own set of tools in an MCP and that set of tools can do things like, you know, do Malloy
40:43linting or, or do like, you know, read the docs of whichever semantic layer, you know, you're, you're using. So you can also inject that intelligence into an open harness. And so there is a full spectrum of options you can explore there. Yeah, totally, totally right. Let's see here. There's one for you, Alex. Can you get a deterministic trace from Malloy plus
41:06agent? Does it bottom out in SQL execution? Exactly right. You can deterministically compile Malloy into SQL and then see that SQL and, you know, evaluate it and also, you know, understand it. So it absolutely is something you can interpret. You can also do things like parse your Malloy and, you know, tokenize it, you know, as a part of the library. So
41:28we use that in some of our exploration around, you know, there's certain things that are harder to do in Malloy. And we would kind of point it in certain directions of like, hey, use a subquery instead of joining a whole bunch of tables together all in one shot, right? So you could do things like have a skill that you would tokenize it and then see, hey, are you joining a whole bunch of
41:52things together, right? Like there's a lot of introspection that a language like Malloy gives you. It's one of its key strengths. Yeah, totally agree. Okay, this is a great one from Sean. I'm not familiar with Malloy. It sounds like actually writing text-based queries in a specific language has to write and maybe struggles to. How do you think this translates to something like dbt semantic layer through the dbt remote MCP server where it uses tool calls to get
42:16metrics, dimensions, and runs queries? I'll just be totally straight here. We did the exact same thing. Basically, the difference between how you would implement, for example, Malloy in our semantic layer versus dbt semantic layer was functionally identical. The only difference is we also gave it away for questions where they weren't directly answerable by the existing canned
42:35metrics for it to write its own Malloy, right? That is the thing. So like, yes, if you want to constrain it so users can only get answers that are in, you know, already defined, sure. You could give it an MCP and say, all right, you can only use existing combinations of measures.
42:54So, you know, I don't know. Personally, my general experience has been like that doesn't work very well because inevitably everyone wants to slice and dice on some arbitrary set of things that you had not seen before. And everyone's business is different, right? If everyone was the same,
43:13we'd all be running on SAP and we're not, right? And I mean, yeah, exactly. Thank God. So I think like, you know, there's different ways to kind of think about this. But, you know, I think they would all more or less behave the same way and the way you would evaluate
43:33it would be very similar. Yes, I do think that Malloy is on the more flexible end of the spectrum for your semantic model layer choices. And I do think that's a key strength, because if your semantic layer can't answer the question, what do you think happens? Do you think the person doesn't get the answer? Do you think they go ask the data team? Or do you think they
43:55wing it and they write SQL? Well, I know what I would bet on because I know what I would do, right? I'd be winging it. I'd be doing the YOLO SQL in like 10 seconds, right? Yeah, you'd be like, all right, switching to SQL MCP. The table is just figured out, right? So I think that's an organizational question as well, right? Like, yeah, yeah, yeah, I agree. You know, everyone always says like, oh, if you lock it all the way down, it'll always be locked down. It's like, okay, fine.
44:18Whatever system it is has an export to Excel button. And if you ignore that, you know, you do so at your own peril. That's right. So respect the the grassroots respect the the experimentation. And, you know, agents, agents like to hack their way to an answer. And so that's a real, real thing. So it might be better to give them a path to do that safely
44:39rather than constrain them all the way down and give them a jailbreak at the end. Yeah, yeah, yeah. I mean, I think, yeah, it's highly cultural, I think, around how that fits. All right, good question here from Juan. Would it be better to use guides to capture semantic models versus Malloy? When to use each? I'll start. We did some experimentation around this,
44:58too. You know, we found our guides are extremely effective around allowing you to give SQL to your agents and for agents to discover the right parts of context and, you know, progressive disclosure like Alex talked about earlier. Again, I will just emphasize it really the thing that I found
45:18is most important is building a nice relationship in terms of how the context is structured, which is why we give you kind of a way to tie them to references. You know, guides can be tied to references and they also get a path. And so that path, you can think about like a file structure, which lets you say, okay, this is in the accounting domain and it's about month-end close,
45:40right? So therefore it fits in here. And that way your model can say, oh, I see there's something about accounting month-end close. Let me go find all the context in there, right? And that's where the kind of ontology kind of notion or graph, you know, knowledge graph notion fits into this is you can kind of like bootstrap one fairly easily that way. And we found that to be extremely effective. I think there's two things that we discovered through this that like were kind of
46:04ineffective. The first one is agents are pretty bad at search. Well, that's actually not true. They're not, because they're non-deterministic, they often search for different terms for the same context. And so tuning a search engine so that you can say, like, just like put a bunch of documents somewhere and then, you know, find the right ones at the right time is actually
46:28fairly difficult, which I think makes sense intuitively, but like, I don't think we were like, okay, how can we avoid creating Google from first principles or being like, hey, like, let's, okay, in order to actually, you know, use your context, you need to add this other product so that you can do, you know, hybrid search and like it's accurate. And then you're running evals on your search so that you can run evals on your data. So that's the first thing. And the second
46:51thing is that like for context specifically, models are pretty bad at compacting themselves, right? Compared to humans, right? Identifying what things are important. And so like, you know, like Alex talked about earlier, if we just gave like all the context up front and then ask a question and then it has to like compact itself, it often would lose the thing it needed. And so
47:14being able to kind of just like fit things and make it fit into that knowledge graph kind of construct, I think really, really helped. I don't know, Alex, if you want to add more there. I think what we learned is it's not enough to just have the documents there and to say, go grip them. Because a lot of the value in the questions is the relationships. So we used to say
47:35this back at Intel where I used to work, which is all the questions related to one data set have been answered 20 years ago. The value for our company are the relationships between data sets that are novel and the ones we haven't been exploring for 20 years. So the value of this layer is not how to understand the revenue table. It's to understand how the revenue table connects
47:57to the hundreds of other tables that you have in your system. So as a result, that means these linkages, these breadcrumbs, these relationships actually do add a lot of value because that's where the diamond in the rough is. Yeah, absolutely. Absolutely, absolutely.
48:20Let's see here. I'm going to pull up this one. How about structuring like a two-tier system, as in the agent tries to answer the question only using semantic layer. It's not answerable than fallback to SQL, be transparent to the user. And the answer like only use governed metrics versus had to use non-governed analysis. So you just have a feel on trustworthiness of the answer.
48:42Alex, you want to start with that one? I think that's absolutely a reasonable approach. I think in practice, before we forbade SQL in our experiment, because we were experimenting to try and understand how well Malloy worked. So for the sake of our experiment, we excluded that fallback. That's effectively what it would do, is it would try Malloy first, would get close,
49:03it would get back into its context, the SQL result and say, oh, I see what I did wrong. It would make a tweak and it would just resubmit SQL as its answer. That's right. Wrapped in the Malloy syntax to call it. So that fallback is a very natural path for agents. It is a viable option. And I do think that even if you did a fully Malloy solution, there are different
49:24levels of complexity of Malloy. So it is always a good idea to inform your user about how far off the golden path they are. And you can say, hey, your Malloy statement was more than 20 lines, like you're in the wilderness here. You had to do like three joins in your Malloy to get this answer. So maybe you should talk to the dev team. I think that's a very wise approach.
49:48And that's something that agents are not good at by themselves, is assessing how confident they are. Oh, they're so bad at this. Do not ask them this. Right. How confident are you? You'll get basically like a dice roll of like, I don't know, do they like the number 12 right now? I don't know. You'll get the answer like 12%. Yeah. They're really bad at this.
50:08Right. So I think that is another area where we as humans can add value to these systems that we build. And it's heuristics. But it's heuristics that are true for your company. If you are auditing major banks, maybe you have a very strict set of guidelines for this. And guess what? Debate rate has been around. That's a pretty well understood data format
50:32compared to like agentic logs. Okay, cool. Log analytics is like, you know, it's a mess because it's always going to be a mess. So your domain may necessitate the size of your guardrail. Yep. Totally agree. Really good follow up here from Chin. For business context, we usually store in Notion. How do you imagine the semantic layer have some way to ingest this,
50:53or is it the agent's job? I'm going to start. And what I would say is part of the experiment that we did actually was distilling 26 question answer pairs and then the manual, which is kind of a stand in for whatever you'd put in Notion. And then we just gave that to Fable and said, build a semantic layer. So 100% yes. I think like my suspicion is that what you actually want to do
51:16and like, this is wild to say, okay. And this is Alex linked to a blog post he wrote earlier, but it's basically like if you have the question answer pairs that you know are golden and you have some context somewhere, you can basically regenerate your semantic layer at any time. Right. Now, once you have an existing one, you need to improve an existing one with new context.
51:38Right. And we actually explored both of those ideas. And that's one of the things that worked really well in the experiment, actually, is we were able to just not write any semantic layer language at all and just have the model generate all of it and then improve it as the context changed. So I suspect that the agents already can do this. The main thing is just like
51:59you need to have a kind of in the loop eval, like you need to do online evaluation in that case so it can improve itself. Yes. I think a little bit of it is just like a question of when do you do this? But I think that, you know, keeping things just in Notion by themselves inherently means the agents are going to be doing a search operation. And our experiment was finding
52:20that that kind of search was expensive in terms of context and not as effective as it could be of something that was architected and linked and relationships structured. So I do think there's an ingestion process from Notion into some format. We use guides. I think guides are one of those solutions on the market where it's very much a blank canvas. We give you
52:42very, very powerful but very, very fundamental ingredients. We give you markdown and we give you relationships and hierarchy. And you can build something like Wikipedia or you can build something like a tree structure. And you can build a combination of those. So in terms of flexibility, we would probably advocate, especially how fast things are moving, is to aim towards a
53:04tool that offers you as much flexibility as possible. That's right. Notion might be a little bit too flexible. Yeah, I think, I do think it can be used as an input though and consumed. I mean, it has a great MCP. So it has a great MCP. So I think it could totally work. I wouldn't exclude it. All right. Good question here from Michael. I've never heard of Maloian until today. I'm skeptical of semantic layers and I assume our efforts to fight entropy are futile.
53:27That's too bad. That said, how would you design a POC today to test the value of an experiment like yours if you were to do it again? Well, good news. We keep all this up to date in the Mother Duck Labs repo. What I would do if I was designing a game today is I would say, okay, I've learned A, B, and C. Use this reference implementation from Mother Duck and help me design something
53:48that will test my assumptions. We have it there for that exact reason. So it is more of a just a reference. It's just a reference that you can use kind of as the, you can lean on our learnings in terms of building it. And I think that that would be where I will start. And it
54:12should work generically for dbt or cube or anything else. Let me just, I'm just going to link this repo again. Just in here. Yeah, go ahead, Alex. I think a key takeaway that Jacob architected in at the beginning of this, which is being model agnostic. And so I think that's one where choosing
54:36your harness, if you choose Claw as your harness, that locks you in to a certain extent. Whereas the more open harnesses or building your own harness on top of something like an open router keeps your options open. And we saw tremendous value in that, where as new models came out, we were able to pivot to whichever Frontier Lab offers. That's right. That was the best at the
54:57time. So I would absolutely seek openness because of how fast things are changing. Yeah. It is a really great jumping off point. I think I would clone that repo and say, start here. Yeah. I would also add, we didn't really talk about this, but why it's important to be open is that meaning you can choose, you know, kind of generically choose different models to use
55:21is I think we ran the experiment with Opus, I think first. And like, I think a single run was like $500. Something like that. Maybe I'm off or maybe, maybe it was like for a thousand questions, it was $500. And I think we ended up at like four or $5 with Luna Max, Luna 5. 6 Max. And so like,
55:41uh, we were able to kind of hill climb on cost by just switching between models from different providers using open router. And that was very, very valuable to us, right? Because it meant that we were, we kind of were buying optionality into how we were thinking about this. Um, and again, I think, you know, my, what we've learned, right, is as these new models come out, well, we want to be able to know, like, are they better than what we're doing today? Right. And how do we test them?
56:04And so like that, that's part of the reason to build this. Um, we're going to do one more question here and I think we're going to, we're going to run, um, by the way, feel free to, um, uh, uh, feel free to follow up with us in our, in our community Slack, if you can drop a link in there. Um, we'll, we always love talking about this or find us on Twitter or LinkedIn. So I'm
56:23going to, I'm going to grab one more question here. Um, let's see here. Uh, let's see. This is a great question from Claire for teams that already have years of metric logic sitting in look ML or dbt. Does it make sense to migrate? We build out a new semantic layer, let agents read the existing semantic model directly. Where's your line between semantic layer and the logic you,
56:43uh, already wrote in your BI tool. Alex, you want to start? I think it's an amazing question. Um, so there are open source tools out there that will do conversion for you back and forth across multiple different semantic layer languages. And if the look ML language has you locked into a certain ecosystem that you, that you want to choose to use in a different way in, you know,
57:082026, you have the ability to convert that in a way you did not before. And so I would absolutely explore opportunities to convert into the semantic layer that works in all the areas that you want it to work in. So that's the first part. Second part is semantic layers are good context, even if the model doesn't write it directly. So there's really no downside to just including it in a file to
57:28reference. Um, I do think though that if you aren't going to be directly executing that, it might make sense to actually have example SQL statements in context, uh, as opposed to only that, that layer. So we do see advantages to using your, your query history and things
57:46as a part of, of that, um, you know, adding to that context, um, over time, the, the, I think a lot of the challenges you, you want to extract the knowledge and learnings out of your BI tool because your actual may or may not have the best agentic experience. And you might want to be able to just use your existing subscription you have with something
58:08else. Right. So I do think that extracting knowledge out of your BI tool is a good and worthy goal and will pay dividends for a long time. Yeah, I think that's right. Um, I would just add that, like Alex mentioned briefly, you know, query history is a great way to bootstrap your own context layer, especially if you have like, you know, some, someone you really trust to write good queries. Um, it's a great way to kind of like identify, Hey, what have they been looking
58:32at over the last 30 days? Uh, okay. Let's look at these parts. You can use something like SQL glut to break it down, which is great open source library. Um, and then kind of understand what the patterns are. And I found that that is just like, you know, really, really powerful in terms of being able to, you know, start from an empty page. Um, a lot of this stuff is, is like, well, how, you know, you already have all this information. How do you get it into your model? And then,
58:55um, so that you can, you can begin answering these questions. Um, by the way, I do see, I do see a quick note. Oh, sorry, Alex, go ahead. So a quick shout out that there are DuckDB extensions that can read, uh, Power BI, for example. So there are ways that you can reach all the way into these BI tools and, and pull that knowledge out. Yeah. There's lots of things. I just saw a really good question here from Claire or a comment from Claire. I'm not going to pull it up just since we're wrapping up. Like I'm just noting that this, the staleness,
59:19the illness problem is really hard. Totally agree. I mean, you know, I think one thing that I've kind of observed is we're like speed running all of the things in BI. Like we'd start a project, you know, in the old days, maybe like in 2015, I built a data warehouse on SQL server. Um, and, uh, you know, aspirationally it was like, you know, in 36 months, you know, we'll worry about
59:42making sure everyone's on the same page and that we're like keeping metrics up to date. Right. And now we're like doing that in a month where I just, everything just, we're speed running all the way through it. Our users are like, wow, how do I get this MCP in and like ask questions? It's wild. So I think like, you know, um, really good, really good insight there. I think we're all on this journey together. We're running this stuff internally too on our own MotherDuck data
60:06that we're using to make business decisions. And so awesome, awesome, awesome questions from everybody. Really, really appreciate it. We're gonna wrap up now. You know where to find us. Alex and I are very visible on Twitter and LinkedIn. Feel free to hit us up there. Just find our names, type MotherDuck, find us. We'd love to chat publicly there or in private, or you can join our Slack.
60:31Thanks everybody so much. This was a really awesome session today. Pleasure.
