Episode Transcript
[00:00:00] Andrew Lebbos: So not only do we use AI, we also contribute to AI, and a lot of our partners are building AI systems in their brokerages and in their platforms, and we're powering those systems with our data.
[00:00:14] Elizabeth Gearhart: When people are building their systems now for their companies, instead of adding AI to what they already have, they're building them around AI.
[00:00:22] Richard Gearhart: It's obviously pretty important that if you're providing financial data to an LLM that the data is correct. And one of the challenges with AI is sometimes that it can make mistakes.
[00:00:39] Elizabeth Gearhart: I have to agree with you. Right now I'm trying to automate a process that I have. It's actually for this podcast itself.
[00:00:46] Andrew Lebbos: We use AI every single day to develop for all of our processes for communication.
We also are a provider to some of the AI companies.
[00:00:56] Elizabeth Gearhart: This is Real AI Use Cases Business Owners Roundtable. Here's how real companies are using AI right now.
Welcome to Real AI Use Cases Business Owners Roundtable. We have Andrew Levos here with us. He's in the financial industry. He's doing something really interesting with AI.
[00:01:13] Andrew Lebbos: Yeah, absolutely. We use AI every single day to develop all of our processes for communication. We also are a provider to some of the AI companies. We provide our data to Perplexity to help with their answer engine. If you go on Perplexity and ask the earnings results for SpaceX yesterday, it will return some Benzinga data that we produced in house. So not only do we use AI, we also contribute to AI, and a lot of our partners are building AI systems in their brokerages and their platforms, and we're powering those systems with our data.
[00:01:51] Richard Gearhart: So, Andrew, what is your role in the company? Are you the founder or do you have a team that's helping you with this?
[00:01:59] Andrew Lebbos: So I have been at Benzinga for seven years. I am not a founder, but I'm one of the longest tenured folks there. I run the licensing department. Benzinga. Most people know us as a financial media company. They see our news on their trading platforms. But we also have a data division that we license to brokerages, and that's the department that I run.
[00:02:20] Elizabeth Gearhart: I listen to this podcast when I can. It's called the AI Daily Brief. Do you ever listen to that one?
[00:02:25] Andrew Lebbos: No, but I think they have a newsletter, right? I think I'm subscribed to the newsletter.
[00:02:28] Elizabeth Gearhart: One thing he said a few weeks ago was that when people are building their systems now for their companies, instead of adding AI to what they already have, they. They're building them around AI. Are you finding that with your clients
[00:02:41] Andrew Lebbos: for the Newer startups that we're onboarding. Yes, for sure. I think that there's definitely a benefit of building today where integrating into a legacy platform is a lot harder than building from scratch.
[00:02:56] Elizabeth Gearhart: So what do you think is the most important use of AI today in a business?
[00:03:00] Andrew Lebbos: I think it depends on the business.
Whatever process is most tedious or taxing for the business, if it can be automated with AI, that's the most important use of it. Yeah. What do you think is the most important use of AI in a business?
[00:03:18] Elizabeth Gearhart: I have to agree with you. Right now, I'm trying to automate a process that I have. It's actually for this podcast itself. We have an assistant who generates a YouTube video from this, but she also gives me the audio file and also the transcript and everything. And then I take that and I put it on our podcast distribution platform and generate the COVID that. So all the steps that I do and then putting everything on a website, I'm trying to figure out how to automate that process.
And Perplexity has been pretty amazing lately. I've been using Perplexity a lot. My friend said Fable 5 with anthropic is really good too. So I have to fabric figure out which one and maybe I'll try just dabble with both of them and see which one gives me better results.
[00:04:05] Richard Gearhart: Certainly that's ultra complex.
So it's pretty interesting in terms of accuracy of the data that you're providing. It's obviously pretty important that if you're providing financial data to an LLM that the data is correct. And one of the challenges with AI is sometimes that. That it can make mistakes. So what kind of steps do you take to verify the accuracy of the data?
[00:04:36] Andrew Lebbos: Big investment of time and resources on our end is qa. Like, QA is half of our job.
The aggregation and sourcing the data from the primary sources is half. And then bringing it in, organizing it, ensuring that it's accurate, is the other half depending on the data set. We have automated systems that run to ensure there's no outliers. We have some human qa. We have humans actually looking at the output before we send it. We have some products that actually are built on AI. And before we publish, the bulls, say bearsay. Specifically, we have a human review it to make sure there's no hallucinations. QA is a huge part of our business, and it's one of our differentiators with our clients is that, like, we take it extremely serious.
[00:05:28] Richard Gearhart: What are some of the steps involved in doing that?
[00:05:30] Andrew Lebbos: One example is Government trades. When someone in the Senate or in the House make a trade, they have to file it. And the way that they file it is not standardized. Some of these folks are hand and spilling their coffee on these sheets.
And so we built an OCR system in house and then we built a QA on top of it. Years ago, before AI and before technology was there, we would go one by one through the filings and tried to do it with our eyes.
And human error dangerous in financial data. And consider too, our data is live on brokerage platforms. Like you see our news on Fidelity, you see our news on JP Morgan Chase.
Those platforms have no space for error. And if we miss, we hear about it right away.
So we've really doubled down on our qa and it's helped with the invent of AI. And so we've been winning a lot of those partnerships.
[00:06:37] Elizabeth Gearhart: How many large language models do you. Do you personally use? Do you have a favorite one?
[00:06:42] Andrew Lebbos: I primarily use Claude. I use Claude Computer. It's extremely helpful the majority of the time. I also use the web app still, like old school. I use Perplexity. I used Perplexity Computer. I switched over. I use genspark for decks. I'm not sure if you guys have tried that one. It's similar to Perplexity, but they're specifically for deck making.
Sometimes I just turn my record on and I just speak out loud what I want the deck to look like, attach a couple things, and then in 10 minutes, there's a deck made. I think that one is the coolest use case because the first time I used it, I did a deck for a pretty big partnership.
I was like, there were a couple of points of contention that I couldn't figure out what direction I should go.
And so not only did it build a deck that would have taken me 40 hours, it built it beautifully. And it also answered the questions that were lacking. Like it filled in the gaps that I would have otherwise had to figure out. It's pretty incredible.
[00:07:47] Richard Gearhart: Yeah, I was just going to say, in your company, what are the policies around individual employees using AI for their work?
Are there any guardrails or guidelines that management suggests, or is it pretty much left to the discretion of the individual employee?
[00:08:07] Andrew Lebbos: It's a great question.
So our company, and I'm not quoting this out of our handbook, I'm sure there's like a well written, well thought out way to describe it. But in short, we all work out of company instances.
So we have enterprise agreements with all these AI companies. So, like, we're not just going to an open source ChatGPT and uploading all of our personal or our company data.
[00:08:36] Elizabeth Gearhart: Nobody should do that. You should always use an enterprise.
[00:08:40] Andrew Lebbos: Yeah, especially when it comes to like your business data and information.
So we have an enterprise relationship with a lot of these different LLMs and other models.
[00:08:51] Richard Gearhart: But then when it comes to day to day usage, it's up to the team member to decide. I think I'll ask Claude enterprise version about this, see what they say. And then ultimately if they present that information or make decisions based on it, then they're held accountable for how they use it and how it's implemented. So is there a wide spectrum of team member usage? Is there a pretty wide spectrum of how people use it or. And do you ever talk about how you use it?
[00:09:22] Andrew Lebbos: The team uses it extensively. I think that our engineers are best at it just because they're tech forward. And I know that we've amplified the amount of tickets that we can put out and the products.
The time to MVP is so much faster, like exponentially faster than before AI.
So we're pumping out products, fixes are much faster, tickets are getting knocked out.
I think that our engineers do an incredible job with AI, like our analytics engineering team, our data engineering team. And so it's a wide range, right? And there are controls. So like we have a news desk, right? We have a 50 person news desk. We write everything by hand and we will continue to do that because the art of editorial is the connections that these writers have.
But we use AI to help find sources for the news desk. And so whereas the engineering team may be producing actual code with AI, newsdesk is using it as a resource to point it in the right direction. So every role at the company, it uses it in a different way.
[00:10:25] Elizabeth Gearhart: I had another question for you. So you said you were letting Perplexity train on your data, is that right?
[00:10:31] Andrew Lebbos: No. So Perplexity is a client of ours. Imagine if you went to Perplexity and you said, hey, what were the results of Tesla's earnings this quarter?
Perplexity is going to pull from specific data vendors instead of just scrape the web to avoid for hallucination and have a more targeted response. And we're one of those data vendors and so earnings is just one example. They take a few of our calendars, they take our news feed and they're really good about attribution, they're very nice to work with and they're a very upfront, honest AI company and so they're using us as a reputable source to answer questions in their engine.
[00:11:13] Elizabeth Gearhart: And they're paying you for that?
[00:11:14] Andrew Lebbos: Yes.
[00:11:15] Elizabeth Gearhart: That's sweet.
[00:11:16] Andrew Lebbos: That's our whole business, right? Like, our whole business is to license our data to a partner. Right. So if you see our news or information on whatever your 401k provider is, there's probably Benzynga data on that platform, and that platform pays us a monthly fee. Like, instead of just displaying on their platform, these AI companies are using it to answer questions. But it's the same premise. It's like, we produce the data, you use it, you're paying us for the production and delivery of the data.
[00:11:48] Richard Gearhart: But not every bit of data is paid for by perplexity. Right. So presumably your data is behind some sort of paywall that they couldn't access without your license. Right. But then there's also other data that could be available free of charge, just based on any number of sources.
What would be the business model for somebody who has a valuable data set and wants to license it to an LLM? What are the steps that they would take to make that happen?
[00:12:29] Andrew Lebbos: Every year we host this event and it's fintech Awards and Deal Day, and every year people come out of the woodwork with interesting data that they might have that they're looking to sell. Ten years ago they wanted to sell it to the hedge funds, and then five years ago they wanted to sell to the brokerages, and now they want to sell to the LLMs.
And the biggest barrier to entry for anyone who has data that they want to sell is the consistency of the data and the delivery. They need something that is like daily output, something structured. That's how the data has to look in order for not only LLMs but anyone to buy it. And so that is the first and hardest part, I guess another barrier to entry that I've seen is some people come up and they have this data, but they're not like a legitimate, well, reputable company. People try to enter the space and they're like, we have this data set that's going to help you trade and the foundation of the business. And the data set needs to be of a certain level for people to even consider purchasing it. And I assume that's for any service.
[00:13:41] Elizabeth Gearhart: So you have to have the authority and you have to have an actual research system set up if you want to sell data. But I think Benzinga gets to sell theirs because you have the authority. You've been around for a long time, you've proven yourselves and your data's proven out because people are using It. So I do have a question about this whole kind of accuracy thing. I was talking to Gemini one night and I was like, how can I show up? How can we be the answer? And it said Gemini invested in Reddit and is training on Reddit. There's no verification of the answers there. Anybody can say anything. Have you looked into that at all since you guys are using some of this data?
[00:14:22] Andrew Lebbos: I don't know that I am.
I have the credentials to answer on the specific application of Reddit within these platforms. I know that it uses it in a question answer fashion and I know that Reddit is like very human.
Very. There's a lot of human error on Reddit and there's a lot of bias.
But I also know that they use it for understanding of how humans talk and speak. From a training perspective, which that I think that is pretty cool. But from an accurate perspective, accuracy perspective, I'm not really sure it's honestly something I should look into because the reason why perplexities and other LLMs that some most of them are under NBA. So I can't really say the specific ones that are using our data. But the reason they approach us is because like financial information is cut and dry. There's a right answer and the rest of the answers are wrong. Like there's not going to be ambiguity around analyst ratings for a stock. The analysts gave these numbers, you add them together, you divide them by the amount and that is a consensus price target. Right. It's pretty straightforward. Hence, in providing results for financial questions, you need really sound information. It can't be ambiguous from 10 Reddit threads adding a bunch of things and trying to deduce an answer. If, if you're asking for a recipe on pizzas, you could ask Reddit and you could ask, you could pull the thousand pizza threads and see what is most often said. But who knows? Perplexity might be buying data from cookinghealth.com or like whatever the industry leader is in the cooking space too. I have no idea. But what I know for finances that I think they realize how important accuracy is and input equals output.
[00:16:18] Elizabeth Gearhart: So do you think that shifting then that they're trying to get more accurate?
I know for financial they have been. But in general, do you think they're trying to get more accurate?
[00:16:27] Andrew Lebbos: I think that's the goal of course and I think that they're getting there as well. Like I think that every time I use, not every time I use, but if I look back every quarter, the models are so much better every quarter it's Scary how fast they're getting so good.
[00:16:43] Elizabeth Gearhart: Have you used Fable 5?
[00:16:45] Andrew Lebbos: Yeah, there was a free. They had free access for a little. A part of June. I don't remember when in June.
And Fable is incredible. I was blown away. I helped. I used it to organize some things, to create lists. I used it for a competitive comparison. And it's just. You just give it a prompt and then you walk away and come back and you have like, literally a word, like a word doc that you could present to a client.
[00:17:11] Elizabeth Gearhart: Do you think it's better than Perplexity? Have you tried Perplexity for the same task?
[00:17:17] Andrew Lebbos: That I can't give one or the other. But I know Fable burns a ton of credits.
[00:17:22] Elizabeth Gearhart: Oh, yeah. Because I wanted to ask Fable. I asked him each the same question. I had to buy credits from Fable where Perplexity. My subscription worked. And I thought I actually got the better answer from Perplexity, to be honest with you. But people swear by Fable, so I'm gonna keep experimenting, I think.
[00:17:40] Andrew Lebbos: Yeah, I think it's good to test all of them. Keep pushing it. Like, the more we use it, the better it'll get. Especially for your use case. Right? Like your instance.
[00:17:47] Elizabeth Gearhart: Yeah. And I use Gemini too, because we do a lot with video.
And then I use chat because I like the images that it generates. And I am using mana, so it takes a lot of time to compare all of them with the same prompt. But I think in some cases it's worth it for the double and triple checking. This has been a really interesting conversation. It's really comforting to know that there's actual real data that's being used to train these models.
It's not all just the wild west, for sure.
[00:18:13] Andrew Lebbos: No, we agree. It's like our goal at Benzinga is to democratize finance and make it easy for everyone to invest. And we think AI is a great tool that enables personalized experiences for any investor. And we're happy to be providing strong data to power these models, honestly, so do they.
[00:18:31] Elizabeth Gearhart: Somebody wanted to follow up on this. Could they follow up with you?
[00:18:34] Andrew Lebbos: Appreciate that. If you. I would love if you visited Benzinga.com APIs or you can just add me on LinkedIn too. Andrew Lubos on LinkedIn. I'm always on there.
[00:18:43] Elizabeth Gearhart: Okay, great. Thank you, Andrew. It's been a pleasure.
[00:18:46] Andrew Lebbos: You have been listening to real AI use cases Business Owners Roundtable with hosts Elizabeth Gearhart, marketing expert and PhD researcher using AI every day. And Richard Gearhart, entrepreneur, business owner and intellectual property attorney specializing in innovation.
We hope you found this valuable. Join us again for more stories. Because the future of business is driven by AI.
