There’s a ton of cleverly set up cameras and sensors in the expensive Apple headset. That makes the device quite adept at catching this small pinch action from all sorts of angles. There’s no need for you to hold your hand up in front of you in some weird way for the device to detect it.
There’s really something to this whole eye and hand tracking thing. After just a couple of hours of using the Vision Pro, I switched to my computer and was initially surprised that things don’t just light up when I look at them. I started missing the natural way of interacting with digital objects. On several occasions, I instinctively looked at a browser tab or an app icon and tried pinching to tap on it, to no avail.
– PhoneArena review of the Apple Vision Pro, 25 March 2024.
That’s why now, these Apple Vision Pro gestures might expand to control more of Apple’s devices, like the iPhone, the iPad, and some Mac computers.
There’s a new patent application by Apple, seen by AppleInsider, called “Devices, Methods, And Graphical User Interfaces For Using A Cursor To Interact With Three-Dimensional Environments”. This one builds on a similar patent from 2009 but is now strengthened by the success of the Vision Pro‘s gestures, of course.
The application includes drawings, some showing parts of the Vision Pro headset and how it could recognize gestures and send them to other devices like a Mac. It also features images of people using gestures with devices like an iPad or an old desktop computer, even without wearing a headset.
The patent talks about devices that offer computer-generated experiences, such as virtual and mixed reality through displays. Since iPads and iPhones already have Face ID sensors that scan the whole face, it’s easy to imagine them detecting hand movements in front of them.
Apple already uses Face ID sensors for more than just unlocking devices. With iOS 18, you can navigate your iPhone just by looking at it with eye tracking. This new patent focuses on how to accurately recognize gestures and avoid mistakes, like ignoring a gesture when someone is just scratching their nose.
The patent describes how a gesture can, for instance, move a cursor based on the hand’s movement. If the gesture doesn’t meet certain criteria, the system will ignore it. Patents are often vague on specific uses, and this one repeatedly describes different ways gestures could be used, from moving a cursor to moving windows.