Develop Through Dialogue Keeping Humans in the Loop - Ari Kaplan

This talk emphasizes the importance of keeping humans involved in data-driven decision making across various fields, particularly in sports analytics. The speaker shares personal experiences from his career in major league baseball and Formula 1, illustrating how data and human insight combine to enhance performance. Topics such as predictive analytics, the use of AI in decision-making, and the need for empathetic user interfaces are explored. The discussion also touches on the lessons learned from notable projects, including the creation of chatbots for specific datasets that provide actionable insights. The speaker argues that while technology evolves, human judgment remains critical for interpreting data and fostering innovation.

Uh the theme of this is how humans staying in the loop uh how humans are more important now than ever. But, you know, practicality uh what what does that mean? How do you govern your information, involve yourself and your colleagues in it? But, there are five movies, two of which were directly based on my life experiences. Three were my life experiences, but coincidentally the movie uh had

the same topics. The Moneyball is a story what you're seeing now in cricket and other sports around the world where you could use data and insights to uh make predictions of player performance ticket sales, merchandising but it really changed the way that the sport was run. Um that that's been my life. If you've heard of Caltech it's like IIT I went to Caltech, became called the alumni

of the decade, which was pretty wild. Um but when I was a teenager came up with better ways to analyze players. And the reason that that's important and relevant now is whether you're using math or data is you're trying to isolate where there's value in the information and what is just pure luck. So, you're trying to maintain your customer loyalty or predict your sales you want to

understand what's predictable, what's causal versus correlated. So, that's what I did as a student ended up uh having a crazy experiences creating and leading the Dodgers, the Cubs, the Orioles, the Astros analytics department um but then also working as a professional scout looking at the human attributes. That's where the the Moneyball movie was great is what can you put in metrics with numbers and what can you

not? So, I think everyone here can relate to that. When you software develop, when you run math formulas, what can and cannot you um encapsulate and it you know, you need both of them. Then, after many, many decades, I ended up working with Formula 1. I think more people were fans of Formula 1 my last speech here than baseball, but I worked with McLaren traveling the world

with their race strategy team trying to make use cases, computer vision, weather predictions, and working with the drivers as well. So, Brad Pitt also made a movie. I was on some of the filming, but it was not based directly on me. So, my joke is Brad Pitt, Moneyball, Brad Pitt, Formula 1. 3 years, come back and you'll see Brad Pitt Brad Pitt, the Databricks evangelist movie. But,

not not too much action in in that. And then going over here, um I always encourage when I speak for people aside from their work to do something positive for the world, whether it's giving to charity, doing kindness acts, things like that. For me, aside from my day job at Databricks, I'm president of the investigation of the fate of Raoul Wallenberg and I try to find missing

prisoners of war or people who were prisoners arrested when they shouldn't have been. So, for example, in America, helping someone who was convicted for a murder he didn't commit, served a life sentence, but I helped get him out after 20 years. It's a tragedy. 20 years wrongfully imprisoned. His mother passed away, couldn't get to hug her or anything, but um he's a very famous person. A lot

of people don't know who he was, but he was a Swedish diplomat during World War that rescued 80,000 Hungarian Jewish civilians from the Nazis and their offspring number about 800,000 humans alive today because of him, but tragically the Soviets arrested him and disappeared into the gulag. Um, so I lead the investigation. I was given top clearance access to the Soviet now Russian archives collecting millions of documents

in different languages inputting them, scanning them, using AI to try to you know, make chatbots on on these documents and also making like a prison ERP or prison CRM where every prisoner was in some key prisons who were eyewitnesses, interrogations and so on. So it's not out yet, but it will be out this year. Jamie Lee Curtis who won the Oscar, Brian Cox and Jake Gyllenhaal narrated

a documentary on on the work that I've done. So that's going to be crazy. People don't believe me when I tell them that, but you know, look up on IMDb. That'll be amazing. Bridge of Spies my partner that I work with. I'm going back from Bangalore to Chicago on Sunday. We're going to Mysore tomorrow. Then fly home Sunday. Then on Monday I meet with this guy here.

Not Tom Hanks, but the guy he out portray. It's a great Steven Spielberg Bridge of Spies. He was an American spy that got exchanged. And then Blackberry is a great lesson. I know the world is rapidly changing with AI. During Q&A I'd love to give my thoughts, but I used to work at US Robotics Palm Pilot. So anyone ever heard of or had a Palm Pilot back

in the day. Thank you. It was before the iPhone, before the Blackberry, the first digital assistant. And I started a company, we got venture funding, and I tried to do a partnership with Blackberry when they came out, and they were completely They were so stupid that there was a movie based on how stupid they were. They closed their operating system and said no one except Blackberry could

make software for them. I'm like, "No, you should open it up." So, you know, video game companies and so on could write there. Like, "No, we won't let any company have our software." And then I proposed an idea that would be similar to the Apple Store. I'm like, "Have third-party companies write software on the Blackberry, and you can get 10% um and you make money, and people

have a reason." They're like, "We will never do that." And that's how they ended the meeting. I also like on my company, people would literally kind of similar with money balls, people very resistant. People literally said, I think my phone's over there, um I will never check my email from my mobile device. There's no need to. I'll wait until I get home or to the office. I'm

like, "You're going to be at a grocery store checking your email." And I was basically laughed out of the room that you'll never check an email. So, there those are like four or five movies on my And hopefully you uh you'll learn from all of them. Uh here's Taylor Swift for the Venn diagram, sports fans, Taylor Swift fans, and some people like both. And the reason I

bring her up is what I uh that's her fiance, quarterback uh for the Kansas City Chiefs, won the Super Bowl. If you go to my LinkedIn and you go to my pinned post, I actually I was a sports journalist. Asked him, "Travis, how do you use data? How do you use AI in your football game? And surprisingly, he talked for about 8 minutes on how it helps

him find patterns of his opponent so he could better predict how he plays the game. So even though he's like a muscle person, he was pretty smart and found it helpful. Um they're now a couple. I'm also good friends with her agent Nick Fim since he's the agent to um um Magnus Carlsen. Anyone heard of him? Gukesh, anyone chess playing? My son is a I'm a proud

father. He's 19 years old, but he beat um Magnus Carlsen a couple months ago in chess and uh drew Gukesh and beat all these other people, but um his agent is her agent. And one more side with chess since I'm a a huge fan is you know, in the hallway, I'm not worried about uh AI taking over good software developer jobs, maybe a little for like the

recent college grads, but you know, the big question people are hypothetically in years from now when AI is super incredible, will we all have a job? Which is a good question to ask. I think we all will. Um but with chess, the the worst or just an average engine on my phone can beat my kid, can beat Magnus Carlsen 100% of the time. There's no chance. You

would think that would mean chess playing is dead, but the opposite. The um the chess.com has uh gone up 10 times as much. High school young kids are playing it like never before. Millions of people every day are just on one application, even though a computer will always beat them. So I'm I So the points today then is how humans can stay in the loop. Just looking

at the time, I'm doing good. All right. So with Formula 1, Daniel Riccardo, Lando Norris, I also emphasize being empathetic with people, which means put yourself in their position. If you're writing software and uh it's going to be used on some, you know, I don't know, web store, put yourself in the position of who's going to use it, the response time, the user interface. For me in

sports, it's putting myself in the position of in this case the athlete or our fan base, so I come up with new use cases. Uh how do you do a pit stop where the car comes in, they refuel, the driver drinks from a straw, they uh change the tires, and you uh pull out in under 2 seconds. And McLaren, we set a world record, and we couldn't

have done that without using uh humans to figure it out, but data and AI, even to see where people are standing, the equipment, like what's the bottleneck for air pressure and things like that. And then another use case, coming up with how can you get the driver to improve their depth perception since when you're driving or in my case a passenger in a McLaren in Abu Dhabi,

your head bounces, your depth perception, the thing spins. Realizing that in older drivers, their head isn't as stable, so they weren't able to pass left and right, but they're still able to go straight. And that helped, and I don't want to take any credit, but McLaren's done a very well in the last couple of years. Yeah, I I I'm skipping some slides from the last one, but

yeah, Formula 1, super data intensive, super real time, 80,000 car parts, and then um for aerodynamics, making the wind spe- uh resistance less, the combinations of 80,000 car parts, 80,000 factorial minus one, is more possibilities than the of atoms in the universe. Like in chess, after the first 14 moves, there's something like one sextillion options of the board. Um which is a another compute problem. So, here's

Moneyball partly inspired by my life, not Brad Pitt, but the Jonah Hill the geek that spotted in my life and my experience. uh enough of baseball, this is the cricket audience. So, start bringing up cricket. Um the point of this is you have data or software code that might be just boring or normal. And all this is is a JSON file where you take video and you

mark the XYZ position of 17 limbs on the body and then at a date and timestamp. So, a table of four or database of four columns and they're just boring numbers. They have a lot of meaning to the game, but that in the raw format is uh not helpful to a human. A player can't look at a spreadsheet and uh 60 times a second a dot moving

incrementally a millimeter uh every 30 frames. But what starts to get interesting is uh humans make new features, features like a variable. So, one feature could be this is the leg, this is the degree off of the ground. And that's more informative to a player. How high is my leg parallel at the point my hand is releasing the ball? Um so, that's good information. But you still

need a human like a coach or an athlete to say, is that good or bad? Like how important is that variable with other variables in the formula, how important is it? So, then you can start making features on features or new variables that help your software development or your data science model more. So, the combination of these angles could be called the trunk angle. If your trunk

angle is 35% versus 10%, it may make more of a difference. And um like Jasprit or Jofra, and then you still need the experts to understand in what conditions is it helpful or not. Um and then you can make features on features on features like uh where should you position your players? How do swarms of players like especially in football, how do you have a swarm of

humans, what pattern do they take to be able to pass a ball effectively? And in any industry like in uh retail or consumer products, it's a similar problem of path to purchase. You have 10 different touchpoints with a customer. They see an advertisement, they see a billboard, and then they get a a discount code, 20% off, and then they buy it. Like what paths of sequences end

up with an action. So, now I am going to showcase the first demo. Um this was made just for you all. this is something I don't know if you could see, can make it bigger. CSKMI, anyone going to the game tonight? I'm going to be there. Um two extra tickets if you want to buy now, I'm just kidding. They they come Yeah. And the way you talk

about the X to D of course candidate to appear update at this night. I mean in the first year that I went on All right. That's the first candidate quite most time, but then they share the X to D. Uh would the system need their X to D again? Would when they made the purchase that I think because when we take in the X to D or

uh the human at that point of time out and it up. Yeah, so for the people on the recording once you make a definition or you build like a an expert like system, uh the the humans become important still. So, I'd say yes and no, similar to code development. If once you automate loading, you know, something from GitHub, that part of the process may not be needed

anymore. But, first thing is to have what's called the semantic layer. If you're not involved in that, this is one of the top topics I hear around the world is when you're like developing code, you have um is that a company that um it was uh Sam's Club had over 1 million tables, sales_613, sales_9. The the developer doesn't know what uh how recent the data is. Um

so, may that's the semantic layer. Similar to a coach, you need to define what a quality pass is, what an aggressive reception Once the coaches agree to that and they define it, the coach is no longer needed in the definition of it. But, the they the coaches are needed to relay the information to the players, to watch and see if a player's getting tired. I was with

some of the IPL folks last night for dinner. They said, "If a player's dehydrated, they perform 30% less." So, the coach needs to understand these signs and be more strategic. But, the coach isn't needed for that long process of looking at thousands of videos, looking at their players. Once you define the coach defines it with the developer, which I was for the teams, they they loved it

since it freed up more time for them to be with the players. And so this is an example of more complex features. In basketball, you're dribbling the ball, but you're doing a layup, you're doing an aggressive layup, a dribble, you're doing over and under, you're doing pick and rolls, and so on. So that's a great question. Automation sometimes makes aspects of the job less important, but um

let's them focus on more more things. So my point So this is uh an application we developed. The point isn't necessarily this is cool, fun things with cricket, but to show everyone here, I don't know if um everyone here is every used pick your your code, Claude code, uh co-pilots, uh that's you Some of you are nodding, some of you are afraid to raise your hand, but

if you aren't, you know, start using it. This was like the Vibe code, there's Raplit, Lovable, um uh etc., etc. Um the key thing here is this is based on their own the company's own data, so it's governed, and it's super fast to build more and more complex applications. So all of this, most of the data is real, some of the revenue we made synthetic data, so

the process of collecting this data, making synthetic data, making these dashboards took under 3 hours. And when I ran baseball departments, it it would have taken me like 8 to 9 months, like a whole season to do something this like this. So on the left are different dashboards for different users. One user in this case would be the coach. The defensive positioning and the confidence metric. 90%

confident you should stand here, but 61% here. Is it a pa- What's the best for power play? What's the best for middle? What's the best for death? Up. So, super easy. But So, this is a a nice visual for the coaches. You know, let's look at um for fans. You could pick Ravindra and drill down and see impact of the recent matches, their performance in each of

the venues. I don't know if you can read that, uh but these are the names of the different venues. And then like their statistics. And go through uh any of these players. Uh let's see revenue and fans. So, people in the marketing or sales and ticketing department can better understand but then what's driving the revenue? What are the key components? Um for me, uh my experience is

here, traffic is crazy. So, that could be one big factor. Uh who your opponent is. And then for the fan segmentation, there's six different segments. How do you market to them? But where AI is super helpful is you um I was talking to um CSK. They have 23 million uh Instagram followers. How can you use AI to have a different message for each follower? And then um

something for the data engineers. You have different data pipelines. Which one are complete? Which one is still running? Which one has errors? great insights for that. And then the you could ask open-ended questions, but what makes it unique is it is the team's data. It um It it So, it's more relevant to the team. If you have a general ChatGPT that's trained on our friend Taylor Swift's

lyrics and 2 years ago when I gave a chatbot demonstration, people were blown away. They had never seen it. Now I think everyone uses it and their kids and their grandparents use it. So, you get the idea, but what makes it unique is number one, it's in the terminology of the business. So, in baseball, the word deceptive is great. You're deceiving the batter. But in finance, deceptive

uh is bad. You're a fraudulent. But like here, uh what you're just asking, "What's the forecast?" Normal chatbots are backwards looking or summarizing. You can now do forward-looking research with um more advanced chat technologies. So, it's uh it's creating a linear regression, for example, understanding the variables to go on it and then giving the results. We think uh we're going to forecast and it doesn't just answer

the question, it gives, "Here's the driver, ticket revenue. We think it's going to be 12% above." And then merchandise is also up, um I'll do one more. Recommended playing. This this I like since I love I I don't know cricket well enough. I'm going tonight, I'm going to learn more. You may understand all of this, but this is pretty in-depth. Um you know, here's the defense. Spin-friendly

Chepauk, the model recommends putting Theekshana and Jad Jadeja as the primary spinners. Selection is great confidence. Um let me do another one. Uh you know, field placement emphasizes a slip cordon with two slips and gully, point inside the ring, mid, off, on, and catching positions, etc., etc. That's pretty in-depth. So, that's the point. You're governed on your own chatbots um with your own data. But, to make

it it's it's now easier now than ever. I don't want to sell you Databricks. I'm just bringing this up since it's a a product where you can do these uh it's called Genie, these chatbots, but you have your data. And it's so easy, you just do select and you pick what movie you want. Uh I'm sorry, what data you want. You can select all the data from

different sources. That's the hard part. And now we have a chatbot that's already created. I said 60 seconds was maybe like 3 seconds plus 10 seconds of me typing. And now you have a chatbot, and I'm going to explain again, this is on your own governed data. You get to grant who gets to access what. If this is asking what people's salaries are, I need to I

I don't want anyone here to know what my salary is, but if So, if you ask the chatbot, "What's my salary?" it won't tell you. Um but, if my manager asks or HR asks, it will tell you. So, that that's a type of governance. Uh it gives sample questions um like what is the distribution of movies um and while that's running uh the other thing of keeping

humans in the loop is AI or humans can define your master data management. You can look at a million tables and spend the rest of your life defining what data is in the table or you and or you can use AI to accelerate it. So, here's what we we uh what the AI uh the GenAI suggests is a description for it. Then you can have a human

say I accept it. Like, you know, here's more details. I can accept it. I can edit it. So, now the chatbot remembers that's the definition. So, anyone in the future who asks a similar question with a similar wording will get the insights. You can also do specific instructions like for me, I know India time zone is 30-minute offset of America. Then the rest of the chatbot, it'll

realize it. Or MCA stands for my company abbreviation. When a user asks for a performance, they are interested in product's revenue. That's what performance means. So, now this is humans in the loop making these definitions. And over time it gets smarter and better and realizes usage patterns. You know, you can look at the not just the table using AI to describe it, but for each each column,

you can make definitions and so on. So, a long time ago it already answered distribution of movie. It didn't just answer the question. It provided guidance. It made charts and graphs that it thought was appropriate. I didn't ask for it, but you know, even had how many minutes the median and average was. So, so those are uh ju- just some common basic things. And then on the

technical side, you can monitor, do benchmarks, which answers are better than others, and have humans continually uh improve upon it. So, I already it seems like I just started 3 minutes ago, but I already got at 5-minute warning. so maybe I'll do a little more and then I'll let people go and people who want to stay, I can answer questions. any final points? Yes, so we're humans

will continue to go is uh it's incredible. You're going to be able not just do thumbs up and thumbs down, but here's one of a car dealer for direct they allow the users just like a survey to say accurate, inaccurate, lack of response, non-technical people get involved. That's the other good thing is this democratizes all your software development work, all your data work, all your AI work,

so non-technical people could use it. Um and you as the software developer are helping guide things along, making sure it's not hallucinating. So that gets better and better. That that's a Ford direct. We also are starting to see the um AI usage for making uh it's called hyper-personalization. Uh company on Instagram that has uh cook sells cookies in different cities, different types of cookies, different types of

milk and pricing. They realize by looking at whatever trend and data they have that it'll make a cookie of their favorite type and the image of the city behind it. Or they like coffee and scones. So it makes an image that's interesting just to You do that across millions of people and they find that the lift in sales is uh pretty significant. Here's France and the cookie.

So this is this is what uh before we came in we started talking about different levels. There's a general chatbot where it tends to hallucinate or it's not really controlled, it doesn't know the terminology. There's the chatbot that I just showed, which is on your own data governed, and then there's a tightly controlled chatbot. Uh the example at Databricks, we have analyst firms, analyst meaning Forrester, Gartner,

IDC, companies that look at trends in the industry. And we also have customers that do a request for a proposal. And we want the customers to be able to ask, you know, is Databricks compliant with the European, you know, whatever healthcare uh guidance? Or give me customers that are in uh Bangalore in the oil and gas industry? And we don't want the chatbot to give away information

that's wrong, so we limit it. Sales uh customers and analyst firms can only ask 200 types of questions that we already prepared in advance. They can ask the question with different words, but if they ask a question above and beyond it, it'll say something like, you know, I'm you know, that's confidential, the names of our customers in Bangalore in oil. It'll tell you I'm not at liberty

to say. So, that is one more area of chatbots. And one other example, Fox Sports is a major sporting website and broadcast. Um and they use you know, in this case happened to be Databricks, but they use the similar tightly controlled chatbot. So, they let their fans ask questions, what is the schedule of this I IPL team? And it'll give the answer, it'll give insights, what's the

probability of who's going to win? CSK Mumbai last night. Uh where the players? Here's a link to buy tickets. Good luck, it's sold out. Um that's the highly curated chatbot. Let me um just end then. Do you want more of these? They're free. I co-authored two things. One is gun-free, not selling anything. Data Intelligence Platform for Dummies. If you want to learn more. And then um I

made a like a quick 20-30 minutes of everything data and AI, which really expands upon what I've said. So, I got the time out sign, but there's no one else in the room. So, I'll I'll leave this up here and everyone watching the recording or live stream, thank you for tuning in. So, with that, I'll let you clap and then I'll take questions. So, I'm super excited

that you were able to be here and stay Friday afternoon to be with me. All right, thank you. >> [music]