Hands-On AI 4 Transcript

Please be advised that this transcript is AI-generated and may not be word-for-word. Time codes refer to the approximate times in the ad-free version of the show.
 

Mikah Sargent [00:00:00]:
I have been looking forward to this. There is a feature that I use with many of the AI services, and it's one that I often ask other people who use AI if they have used. It's a research mode, and it's something that AI can do well. You ask a question of your AI system, and instead of answering in just a few seconds, which is what you normally would want, it's a little bit agentic. It goes off And works and does so for minutes instead of seconds, and then afterward hands you a report with sources. So today I'm going to show you research mode by asking all 4 of the systems that we'll be using today the same difficult question. And in order to compare it against a control, I will ask a regular chat with web search the same question. And by the way, I'm keeping score, so we will have a result at the end.

Mikah Sargent [00:01:07]:
We'll look at how long it took, where it looked, what it missed, and then how much I would have to kind of clean things up afterward before I'd trust it. I'm Micah Sargent, and this is Hands on AI.

TWiT.tv [00:01:24]:
Podcasts you love.

TWiT.tv [00:01:26]:
From people you trust. This is TWiT.

Mikah Sargent [00:01:37]:
Last week we took a look at an agent, Muse in that case, going and doing things for me, making plans, using a tool, checking the results, and then going around and around and around again. And if you were there for that episode, then you would have learned about agents and how they loop. The research loop and why I compared it to an agent at the beginning is because It's essentially the same thing. Okay, the deep research tool is that loop, but applied to a specific question. So first, the agent will go in and will make a plan. So it will say, hey, this is what I think you're looking for. And they will occasionally show you an actual plan, which you'll get a chance to see as well. And then— We'll even let you edit that plan before the start.

Mikah Sargent [00:02:33]:
From there, it goes out, it searches the web, and it reads the web. Searches and reads, searches and reads. Then it takes what it's actually found by going and doing that search and by reading that content and then finds the next lead. So it needs to get the basics, and then from there it can figure out what research is still required to figure out the rest of the answer that you're looking for. Then once that's done, it finds that answer and brings it all together with sources. And that's the big thing. It is all about the sourcing of it. And it's the why I like to use the research tool.

Mikah Sargent [00:03:18]:
So where you have, you know, regular search for a lot of these tools, that's built for quick answers, but deep research Is you giving the tool the time. You're saying, hey, go out, go spend some time kind of getting my answer, but also following leads and then bringing those sources all together. And there's something that you need to keep in mind when it comes to this, of course, which is that deep research can use more of whatever usage allowance you have. So some apps will give research. You'll need to check this out depending on what you have. Some are going to give research its own limit. Claude, though, will count against your regular usage. So you want to kind of, before you go out and do it, or perhaps after you try it out, get a good idea of how much of your usage credits it's actually taking up.

Mikah Sargent [00:04:12]:
Now, OpenAI actually has some advice for using deep research, as it is called. The rule of thumb is use search for quick facts and save deep research for depth And thoroughness. So if you're looking for more than just the basics, that is where deep research can come into play. So I mentioned 4 different AI systems, AI apps, AI chatbots, whatever you want to call them. Regardless, we're using 4 different tools. So who are the contestants in our testing today? Well, ChatGPT. ChatGPT calls it deep research. It's in the plus menu in the ChatGPT box.

Mikah Sargent [00:04:55]:
You can also type Now, free accounts will get a limited number of runs. Paid plans get more runs. Of course, it just depends on how, uh, how much they have to do and how you're using them. Gemini also, Deep Research. Uh, it is a plus menu in the Gemini box. Anyone signed in who is also 18 or older can use it. There is a daily limit to it. And then of course, paid plans get higher limits.

Mikah Sargent [00:05:24]:
Claude calls it Research, and it is a switch in the plus menu. It needs to have web search turned on in order to work, and it's only on paid Claude plans. And lastly but not leastly, Perplexity, also called research mode. You pick research in the mode selector, and then Perplexity's own help pages. This is this is something that I ran into. Perplexity's own help pages say that free accounts get one research run a month. However. I tried it with a free account and it wouldn't let me run it, so I am using a paid plan for today's testing.

Mikah Sargent [00:06:03]:
It may work if you have a free plan, it just didn't work on mine. So each of these, these actual systems are all running on paid plans. And as I mentioned, we also have The control, which is that for ChatGPT, we did it with web search, but we also did it with deep research turned off. Okay? So ChatGPT with just web search, no deep research turned on, is our way of testing it just to see if we can figure out if deep research, which is meant to take a little bit longer. Is actually worth the wait? Now, I had to come up with a question that I felt was going to be difficult enough and involved enough to be able to require deep research in the first place. And so the question needs to give a thorough answer and give that AI agent something to do. And then the most important part for me is that I need to be able to give to— I need to be able to have it have facts that it hands me back that I can actually check. So I thought about how I could come up with a question that would work, that would also be something that I could independently check.

Mikah Sargent [00:07:38]:
What? Actually do the research myself? And I was able to. The way that I did this was I made up a house, a house for a friend, not my house. And so I'm Asking for a friend who lives in Sacramento. And, uh, just so you know, whenever I show you this prompt, SMUD, S-M-U-D, is Sacramento's electric utility. So what was the prompt? Let's take a look. The prompt that I used across the board was this: My friend's Sacramento house. 1,900 square feet, built in 1985, SMUD electric. Has an 18-year-old gas furnace, central air, and the original ducts.

Mikah Sargent [00:08:25]:
Should they switch to a heat pump in the next year? Give me a recommendation, a cost range, the rebates and tax credits they can get now, the risks, and questions for installers. Cite your sources. Now, you don't technically have to say the cite your sources thing at the end because in almost every case, they're going to, the AI agents will cite their sources. But I just wanted to make absolutely certain while we were going through this. And of course, make sure that we're asking the same question across the board. So that question is the one we went with because the rebate part is local to that area. It also changes year to year, and there's some stuff out there floating around that is semi-accurate, but it's semi, semi-accurate because it went out of date this year. So you gotta really dig in to get the answer correct.

Mikah Sargent [00:09:30]:
And then, as I mentioned, I went through and checked out the rebate and tax credit rules myself on the agencies' and the utilities' own pages. So that way I've got an answer key that I can look at and compare against what the research, the deep research tools and the research tools came up with. So no, it doesn't settle what this house should install or what it'll cost. That's something that will be determined with the results that come in. So that's something that we can look forward to. Let's get into— I know this is a lot of setup, but we gotta talk about how we can even score these things. So how are we scoring it to determine if deep research is worth it across these different tools? Well, I've got a scorecard. First and foremost, we are Taking a look at these 4 things, and of course the control gets scored on them as well.

Mikah Sargent [00:10:36]:
The time that it takes to send for the full report to come in. So this is send to full report time. So from the moment I hit send to the finished report, and then if a tool stops to ask me something or wants a plan approved, then that becomes part of the process as well. Unique pages cited. So sources, of course. How many different pages did the report actually cite? And then what kind? Were they valuable? Was it the agency or the utility itself? Did they go to blog posts that summarized them? And of course, more isn't always better. Hopping around the web and finding a bunch of sources that don't matter, A, uses up your usage credits, and B, may provide inaccurate information. And then, of course, checked against the key for omissions, right? Were there things that were left out? Was it right? Was it right with the right conditions baked in? Was it wrong? Was it completely missing? That's where the answer key comes in.

Mikah Sargent [00:11:38]:
And then last but not least is cleanup. If I were to send this to my Sacramento friend, How much would I have to cut out of it or fix it or need to double-check? Or I should say triple-check first, because it's always worth a double-check. Then we went into the system and we did these runs. After prompting, some of these systems may ask a clarifying question or 2, and then afterwards show you a research plan that you can edit before it starts. Gemini typically will show you a plan first. You get this little edit button so that you can make a change and There's a timer running on a lot of these. You may need to click start research before the plan will actually kick into gear. Perplexity's really good at asking clarifying questions if the requests are too broad.

Mikah Sargent [00:12:32]:
And then it also will, while the process is running, show you some of the sources that it's reading as it goes. Claude is also quick at collecting sources along the way. And if you have the Claude app, you can kind of keep track of it on a different device as well. As for the control, well, that searched and finished in 2 minutes and 34 seconds and did not have any initial questions for me. So the quick answer first, the control group. This is ChatGPT's answer with web search. I asked at the top the same prompt with web search, and in 2 minutes and 34 seconds, seconds, this is the answer that it provided. It started with a nice recommendation there and said, yes, plan the replacement within 12 months.

Mikah Sargent [00:13:24]:
It goes on to talk about what you should and shouldn't do. It even talks about a planning budget and other information like the rebate that you might get. And it did cover rebates and tax credits. It did so in this full table. It then went on to provide even more information and It did a relatively good job in 2 minutes and 34 seconds. Honestly, I was impressed with some of the symbols that it added. It made it very easy to read. But let's talk about how well it did.

Mikah Sargent [00:14:00]:
We need to grade it against the actual scorecard and of course the answer key that we had as well. So let's take a look at that. Moving along to our Controls answer. As far as the score goes, well, first and foremost, remember we're measuring time. 2 minutes, 34 seconds. Sources, 12. 10 of them were official: SMUD, the IRS, Tech Clean California, Energy Star, the California Energy Commission. And then the other 2 were a couple of cost guides that helped it to determine the pricing for the install.

Mikah Sargent [00:14:40]:
And then we have to ask, well, does spending longer on research buy a better answer? Uh, that's going to depend on how things shake out with our other reports. So let's take a look at those. Interestingly, ChatGPT being the tool for deep research and the tool for the control. Its deep research plan took about 4 minutes. It had 7 sources: SMUD, the IRS, and tech, plus a contractor's blog and a cost guide. Its answer was yes, but it said so, yes with conditions. And what about cleanup? Well, it was going to take a lot of cleanup because its timeline calls October 2026 to October 2027, and 18 to 24 month project. So yeah, I'm gonna say no, but remember, we're gonna get to the key in a moment as well.

Mikah Sargent [00:15:52]:
As for Claude, Claude took about 5 minutes. It had 50 sources and about half of them were from SMUD, the IRS, PG&E, tech, and the CEC, about half from HVAC contractors, vendors, and blogs. There was one smud page that it cited 3 times. It cited it in English, it cited it in Tagalog, and it cited it in Chinese. Its answer was yes. And as far as cleanup, even though it had a lot of sources, moderate cleanup. It was mostly just cutting out extra cruft. It was by far the longest.

Mikah Sargent [00:16:27]:
So in that way, you know, my friend would probably only need about a third of the report. But I mean, there's a difference between getting things completely wrong, like an 18 to 24 month project that's actually only 12 months. And then this, which was just, hey, I gave you more information than you actually needed. What about Perplexity? Well, Perplexity took about 6 minutes. It had 17 sources, and every one of them was official: SMUD, Energy Star, the Energy Department, the IRS, the CEC, and congress.gov. It also said yes, And said to do so as a planned project. As far as cleanup, honestly pretty light. I'd trim maybe its 28 installer questions that it suggested to about 8 instead, because about only about 8 of them mattered to the project, and you don't want to annoy the person who's doing the project either.

Mikah Sargent [00:17:25]:
And then Gemini, which took the longest, about 7 minutes, 40 sources listed. and one of them was a dead link. Only 11 of the sources from the 40 were official. The rest were contractors and cost calculators and vendor blogs. And there were some blogs and pieces about Los Angeles, Mission Viejo, and even Austin, Texas. As far as its answer, it said yes, strongly. What about cleanup? Oh, buddy, it was heavy. Its savings math is down to the penny.

Mikah Sargent [00:17:57]:
Which is kind of interesting. $892.77 a year. But of course, as I said, it cites the dead link. If you zoned out a little bit during all of that detail, a quick recap when it comes to these sources and times. ChatGPT, 4 minutes, 7 sources. Claude, 5 minutes, 50 sources. Perplexity, 6 minutes, 17 sources. And Gemini, 7 minutes and 40.

Mikah Sargent [00:18:23]:
Sources. By the way, the control did it in 2.5. Let's talk about the answer key, the true full official guide. First and foremost, straight from the source, I checked all 3 of these as I recorded this. The federal heat pump tax credit can reduce your bill, your tax bill, by 30% of qualifying costs up to $2,000. But it isn't available for a heat pump put into service after December 31st, 2025. That's an important factor. Because ChatGPT's deep research says the credit expired, then tells my friend to claim 30% federal credit on the remaining cost anyway.

Mikah Sargent [00:19:11]:
Everyone else, including the control, which was also ChatGPT, got it right. Next, California H-E-E-H-R-A. HEERAW? Sure. California's HEERAW program offered income-qualified households up to $8,000 toward a heat pump. Its single-family funds have been fully reserved since February 24th, 2026. But don't count on that rebate for a new project. Claude, Perplexity, and the Control had it with the date. Gemini got the conclusion, Without the date, ChatGPT's deep research knew it was full but still called $8,000 possible and also got the income limits wrong.

Mikah Sargent [00:19:57]:
SMUD, variable, and 2-stage. For an eligible SMUD customer switching from gas, a qualifying variable-speed heat pump can get a $3,000 rebate or $2,000 for a qualifying 2-stage system. And lastly, SMUD, once again, its conditions say that a contractor in SMUD's network has to submit it, and funding, of course, has to be available. Now, all 5 of these systems found the amounts. Claude, Perplexity, and the Control had both conditions. Gemini called the rebate active and funded. ChatGPT's Deep Research skipped the funding catch and told my friend to reserve the rebate first. Which is a step that SMUD doesn't have.

Mikah Sargent [00:20:44]:
A report that says if you qualify or a report that says if funding's still there, I went ahead and gave those credit because it's careful and it's also not leaving something out, right? That's, that's choosing to be careful instead. And so in that case, that's what we want are the careful wordings. Also 2 separate, did it think of it checks? Did it bring up whether the electrical panel can handle a heat pump? Did it suggest getting more than one quote? Guess what? All 5 did both. 3 for 3. Claude, Perplexity, and that 2.5-minute control. The one that counted money my friend can't get? That was ChatGPT's Deep Research. When OpenAI launched Deep Research in February of 2025, it warned that it could invent facts. It could confuse reliable sources with rumors, and it could sound more certain than it should.

Mikah Sargent [00:21:42]:
So that's why we have to check these reports. One more test, and I think it's the most important habit in this whole episode, like the thing that you should know about this whenever it comes to it. Make sure that you check the citations that are included in the report and then click on them and make sure that they go to a page. Just do this once or twice in any report that you do. Go to the page and read what it says and see if it matches what the report says. You're already reducing your time to do this by having it make your report, right? So Taking a little bit of extra time to check those cited sources is great. Does the page support the claim? Is the rule still enforced at the agency or someone has changed? You know, like if the agency has changed its ruling, then that's something to bear in mind. With ChatGPT's deep research mode, yes, once again, SMUD's own post from October 1st, 2025, it says it It really does say to stack the federal credit expiring December 31st, 2025.

Mikah Sargent [00:23:07]:
So that was a year-old page that was being read as current. For Gemini, behind the federal credit, there's an AC Sellers blog, not the IRS. So they weren't quoting the IRS. It wasn't quoting the IRS directly. And then behind that savings math was that link that I talked about that went nowhere. As far as Claude and Perplexity, SMUD's rebate page was where it went. It checks out. Claude's climate claim cites an HVAC contractor's blog.

Mikah Sargent [00:23:43]:
And as far as the control goes, behind the federal credit is the IRS's own FAQ. And that, of course, is the same page that I used for the answer key. So maybe this time giving the agent too much time resulted in something that was less accurate, especially again using ChatGPT without deep research as the control. A citation gives you something to check, but it isn't a guarantee. So is it worth the wait? The control took 2.5 minutes and got all 3 money facts right. 2 of the 4 deep research tools also got things right. Now, one missed SMUD's catch, and then one counted a credit that is no longer around. So more time didn't always mean better results.

Mikah Sargent [00:24:40]:
The fastest deep research report did have the biggest mistake. And unfortunately, the slowest deep research report did need heavy cleanup. As far as what report I would send to my friend, it would probably be the Claude report with a little bit of cleanup done, with the ChatGPT non-deep research being sent as the secondary option. Before you trust a report, something to end on, some advice for you. Before you trust a report, make sure that you give it the details that only you know, okay? For these kinds of reports, where you live, what you've got, what you've already ruled out. If you're doing these reports, I mean, even if it's about I don't know, buying the right filament for your next 3D printer project. It needs to know exactly what you have, not just broad terms. And then maybe even letting it know where you live would be helpful because humidity makes a difference whenever it comes to filaments and what you're trying to make.

Mikah Sargent [00:25:54]:
Then always review the plan whenever it's offered. Okay, if it offers the plan, read it before you approve it. I I have had a habit of skipping over this in the past and saying, yeah, yeah, yeah, research, research. And then I come to find out that the end report is not good and not what I was looking for. So definitely read that plan before you do so. Then make sure when it comes to rules and laws, verify it, check it. It's already doing half the work by, in theory, citing those sources that you can go and check. So anything current or local to yourself, worth looking into.

Mikah Sargent [00:26:32]:
Rules and rebates, they change fast. And if you are using some AI assistance, they may not be using the most recent version, version of a webpage. They may instead be using an older version of a webpage. And of course, open those sources behind the key claims. Check it all. Always check it. As for homework, if you'd like some, I think now's a great time to take a question that you've been meaning to look into and then run it through a deep research tool. Pick one, it doesn't have to be all of them.

Mikah Sargent [00:27:08]:
And then also do it through a regular chat and see if for you the difference was worth the wait. If you have AI questions, I've got AI answers. Send them to hands on ai@twit.tv and you just might hear your answer on the show. Hit subscribe or follow wherever you're watching or listening, or head to twit.tv/HOI. And if you want it ad-free, well, that's Club Twit, twit.tv/ClubTwit. I'm Micah Sargent. It's my pleasure to bring you this show every week. Don't just wonder about AI, get your hands on it.

Mikah Sargent [00:27:44]:
I'll see you next week. Bye.