Hands-On Apple 244 transcript
Please be advised that this transcript is AI-generated and may not be word-for-word. Time codes refer to the approximate times in the ad-free version of the show.
Mikah Sargent [00:00:00]:
If you've ever tried to show someone live text and had nothing happen, well, you probably didn't do anything wrong. You just tapped a half second too early. These features are genuinely great. They're genuinely fussy, though, and that is the problem. And almost all of that fussiness comes down to a handful of things you can control. So let's make them reliable. Stay tuned. Podcasts you love.
Mikah Sargent [00:00:26]:
From people you trust. This is TWiT. Welcome back to Hands-On Apple. I am Micah Sargent, and today we are taking a look at Live Text and Visual Lookup on iOS. So let's start with kind of understanding which is which. There are 3 features, and you gotta know which one you want because they do kind of get combined together, and using the wrong one at the wrong time is the reason why some sometimes you don't see it working as you expect. So what are the 3? Well, the first one is Live Text, and it's essentially on-device optical character recognition. So it'll turn text in a photo or screenshot or in the camera viewfinder into selectable text.
Mikah Sargent [00:01:20]:
The cool thing is it works offline. It's been available since iOS 15 and just requires an A12 Bionic or newer. So that's about iPhone XS area or newer. later. Now, Visual Lookup is a tool used to identify subjects. So think plants and animals and landmarks and artwork and laundry symbols and car dashboards and lights and food, all that kind of stuff. It requires an internet connection, of course, because it does do the lookup that happens on the server side. And then Visual Intelligence.
Mikah Sargent [00:01:53]:
This is a newer Apple intelligence feature that is part of the camera on newer phones. And in iOS 27, iOS 26, it moved beyond that camera into the screenshot interface as well. So you can do highlight to search and you can also use it to ask questions. So let's talk about how we make Live Text work. The most important thing is, let's head over to iOS so we can look at this. Important thing to do is make sure you have it turned on. So you tap on Settings, You tap on General, you scroll down to Language and Region, and you make sure that Live Text is turned on. Once that's turned on, then you can make use of it.
Mikah Sargent [00:02:43]:
It of course only works for supported languages, and it does that based on the iPhone language that you have. So if you don't have English set as your language, if you have, you know, German set as your language, then trying to use Live Text with the English language is not going to work. So make sure that you have this going and that you, of course, have it toggled on. There's a way to kind of break things down, right? When you're trying to think of how each of these features works so that you don't get them confused. If it's text on the thing, that's Live Text. If you're trying to take— if you're trying to deal with the thing itself, that's Visual Lookup. If it's something that's on your screen that you want to ask about, that is Visual Intelligence. So how do we do Live Text? Well, one of the ways that we can make use of Live Text is in the Photos app.
Mikah Sargent [00:03:36]:
So let's head into the Photos app and look for something that has some text in it. Um, here we go. So this has some text in it, and the first thing that you want to do is look for some, uh, a small symbol in the bottom right-hand corner. This is the live text button. And when it's there, and it takes about half a second because the photo needs to sort of fully load and it needs to recognize that there's text. When we tap on this, it actually highlights all of the different text in the photo. And you can then, you can choose any of that. Now, the cool thing is that you can also just long press directly on the photo and that will also allow you to select it.
Mikah Sargent [00:04:27]:
You can take it and move to select more with those grab points, and then you can interact with it. Copy, select all, look up, translate, all that stuff. When it comes to phone numbers, email addresses, URLs, addresses like physical addresses, dates, and then tracking numbers, those don't actually need to have live text, the button, or any of the taps or holds. Those automatically get recognized in a photograph and are then made available for you to tap on. You'll see an underline with them. So if you want to call a number on the fly, you can tap on that number and you don't have to wait for that Live Text button to activate. It also lives in the camera app. So if we launch the camera app here, you'll note that shortly after I hold my camera over this photo or this live item here, tapping on Live Text shows us the different portions of the text on the screen.
Mikah Sargent [00:05:27]:
And we can then select, for example, boarding pass. Isn't that cool? Mm-hmm. And so when you do this, tapping on it freezes that portion. And so you're not having to kind of work with a shaky hand trying to figure out, you know, what text you're trying to select. And so we can see we've got a lot of text here. So I can highlight Titanic, but I've also got an address down here at the bottom, which is really nice as well. All right there too. Let's move out of that.
Mikah Sargent [00:06:02]:
You can also scan text into a field. So let's open the Notes app and we'll start a new note and we will tap and we will choose to autofill and then we choose scan text. Now what's cool is it will actually begin to select this text for us based on what we have, and then you can hit insert and it will put that text there. You can also select some of the text and remove some of the text simply by using your selection points. So bringing up that text using the live text button in the bottom right-hand corner will bring up a smaller portion that is again, that freeze frame, and then you can move your finger along and select it as well. Really nice. And it's a great way if you have a serial number, for example, if you need to get a serial number somewhere else, especially say you buy a new, I don't know, a new stove and you are registering it online. What's something that always asks for? The serial number.
Mikah Sargent [00:07:15]:
Being able to hold your phone over that serial number, use the Live Text button and get it to load is really nice. It works in Preview and Photos and Safari and Quick Look on the Mac. So in any of those, you can drag and select over the image text. And if nothing selects, you can always make sure that you do have text selection as an option there. And in Safari, Control-clicking an image lets you copy text as well. So all across the board, across all of Apple's platforms, it has been set up for you to be able to make those Live Text selections. Now, that is Live Text. Let's talk about Visual Lookup because I think this is one of my favorite points.
Mikah Sargent [00:08:03]:
If we open up the Photos app and we find a photo like this photo of a cat, you'll note that in the bottom middle, the I button, my favorite button, turns into an eye with a little sparkle that tells you that it has something for you. If you tap on it, it will detect that a cat is in the photo in this case, and then tapping on Look Up Cat will show you different cats, and it will also try to assume what cat is actually in the photo. So it's not just looking up the word cat in general, which was my first thought when this feature hit. And it said, Look Up Dog, and I said, um, Well, I know what a dog is, but no, it's saying I have recognized that there is a dog in this photo and I can look up that dog. This says look up graylag goose and tells us a little bit more about the goose that it thinks is in this photograph. And then let's do another one here. These are all stock photos that I've pulled from a site. So, wow, this detected that there was a lizard.
Mikah Sargent [00:09:15]:
An iguana. And it even shows in newer versions of iOS, it will have a little icon at the bottom of what it expects is actually in the photograph. So if it's a dog, it will show a dog. If it's a cat, you'll note that a cat symbol appears. Here we have horses. It can't quite tell. And so it just showed a general paw. And then let's do one more.
Mikah Sargent [00:09:45]:
We've got a frog here. It does think it's some sort of lizard or amphibian. In this case, it's perhaps a spring peeper. Cool. Really like visual lookup. You can also use it for laundry symbols. So if you take a photo of your, of your care tag on your shirt, then Make sure that, you know, that's most of what the photograph is. When you tap on the eye icon after it detects something, it will tell you what those actual symbols mean and helps you to figure out what is going on.
Mikah Sargent [00:10:26]:
Also, it works for many of a, many of the car dashboard lights as well. So if a light ever pops up and you're not sure what it means, taking a photo of it and using visual lookup can help. Um, It's great for plants and animals. So full plant shot with visible leaf structure is going to give you the ability to figure out what the photo is. And again, what we love is that Siri intelligence down there at the bottom that when we look it up shows us what type of plants it is and even shows you similar web images. You can also tap show more or show less to get even more information about it. This works in Photos, in Safari, in Messages, messages in Mail and Quick Look. So it's not necessary to actually save the image first.
Mikah Sargent [00:11:16]:
If you get more information about it, you simply long press the subject and then you can get more information as well. Now, here's the thing. One of the biggest issues that comes with these features is that occasionally they can be a little finicky. And that's kind of The part that I really want to cover today. Live text and that sparkly eye, and then also doing especially the photo select or the text selection. They need some time. So here are some suggestions for you to make those work. Both the live text button and the sparkly eye populate after the image renders.
Mikah Sargent [00:11:58]:
So tapping instantly is not going to make it work. hitting that eye button immediately is not going to make it work. On an iCloud library, these low-res placeholders will load first, and then they have kind of too little detail to scan immediately. This is for the system to scan immediately and determine what's in it. So this lets you download the image fully, and then you're able to interact with it. And look, connection does matter for some of these features. Live Text is, of course, on device. It does work in airplane mode, but Visual Lookup needs the network to actually work.
Mikah Sargent [00:12:32]:
So So if you don't have great Wi-Fi or if there's something going on that's got your connection not figured out, that might be why. Low power mode and low data mode will both make it worse because it'll take longer for this feature to kick in. And when it comes to using live text, especially being able to interact with the text, Keep in mind that you're taking the photo for OCR. You're not taking the photo for a good photo. So make sure that the text is upright. You know, you don't want it to be at a distance and glare can be a big issue there. Fill the frame with the text as much as possible, but also don't get too close because then the lens can't lock onto the text and maybe even pinch to zoom before you select small text because it'll make those grab points a lot bigger. So it's easier for you to select the parts that you want to.
Mikah Sargent [00:13:29]:
Visual lookup does get a little bit indecisive when it comes to a cluttered frame. So make sure that you crop, if you're specifically looking to use visual lookup, duplicate the photo, crop it to the part that you're actually trying to look up and then use it that way. And then live text too on a busy sign. If you've got all this text on the screen, no, you just want this little part, obviously. So very easy to do. You know, that is going to make it so that your Live Photos work. Now, your Live Text works and so does your Visual Lookup. Now, when it comes to Visual Intelligence, this is a feature that frankly is still in flux and is something that requires newer model phones.
Mikah Sargent [00:14:13]:
So we will be taking a bigger look at Visual Intelligence in future episodes of Hands-On Apple. I end with homework once again. Consider finding, you know, 3 things around the house, a paper receipt, plant, a clothing care tag, a dog, a cat, and then run each one through the feature that you're trying to use. Try to scan that. I love, again, the scan text feature. The next time you've got a serial number or something that you need to grab, a model number, use scan text to get it to populate where you need it to. That way, once you practice it a little bit, it starts to be built into your muscle memory and you'll remember to reach for it. for it next time.
Mikah Sargent [00:14:54]:
Folks, I want to thank you so much for taking the time to join us today on Hands On Apple. It's always a pleasure to bring you this show. If you have questions for me, you can email me hoa@twit.tv or mike@twit.tv. Both will get to me. And I will catch you again next week for another episode of Hands On Apple. Bye-bye. Bye.
