From the Keyboard to the Bare Hand

For nearly forty years, the keyboard and mouse were the central means of human-computer interaction, complemented by the graphical user interface with windows, icons, and pointers. This so-called WIMP logic shaped practically all computer use, from the first graphical interfaces at Xerox PARC in the 1970s to the modern desktop computer.

With the arrival of touch-sensitive screens, this logic began to visibly shift for the first time. The field has since spoken of natural user interfaces, meaning forms of interaction such as touch, voice, and gesture that come naturally to humans, rather than being controlled indirectly through an extra device.^[3] The shift from the mouse click to the finger tap on a touchscreen was, in that sense, only the first visible step of a considerably larger development.

The next step in this development leads consistently further away from the device and toward the bare hand itself. Expert analyses of the history of graphical user interfaces describe this transition as the logical continuation of the touchscreen, one in which the hand no longer meets a glass surface but acts freely in space, with all the benefits of immediacy, but also with new demands on precision and reliability.^[4]

From the keyboard through the touchscreen to the bare hand in space.

^{Infographic: the development line of human-computer interaction from WIMP to natural hand control | Graphic: © Ulrich Buckenlei | XR Stager Online Magazine | VISORIC GmbH}

This shift, from indirect intermediary devices to direct hand control, fundamentally changes how people could handle digital content in the future. Instead of first translating a movement through a mouse or a touchscreen, the hand itself becomes the input device, with no additional hardware in the hand at all.

For glasses or a headset to reliably use a bare hand as an input device, though, they must first be able to technically recognize and track that hand with precision. Exactly how that works is shown in the next chapter.

• Keyboard and mouse shaped nearly forty years of computer use.
• Touchscreens ushered in the shift to natural user interfaces.
• The next step leads from the glass surface to the free hand.
• Free hand control demands new precision and reliability.
• The hand itself becomes the input device, with no extra hardware.

This makes clear that the shift to hand control is no sudden invention, but the logical next step of a decades-long development. What makes this step technically possible at all today is a combination of sensors and Artificial Intelligence that has only become practical for everyday use in the last few years.

How a Pair of Glasses Recognizes a Hand at All

That a pair of glasses or a headset can reliably recognize a hand with no extra device at all rests on the interplay of several cameras, invisible infrared light, and a trained AI model, which only together produce the actual result.

Around the edge of the device sit several small cameras at once, facing forward, sideways, and downward, so the hand is captured from multiple angles simultaneously. Invisible infrared lights additionally illuminate the field of view, which is why recognition also works in poor lighting, since it doesn’t depend on normal, visible light. From the slightly different images captured by the individual cameras, the system calculates depth, much the way the human brain automatically computes distance and spatial relationships from two eyes.^[5]

Onto this depth data, a trained AI places a digital skeleton of roughly 20 to 25 joint points per hand, similar to a stick figure made of knuckle and fingertip positions. The model was previously trained on a very large number of example images of hands in the widest range of positions, and can therefore still estimate the correct hand posture even when one finger is partially hidden behind another.^[6] A dedicated chip built specifically for this purpose processes this camera stream continuously, on current headsets within a few milliseconds, which is why recognition feels practically instantaneous to the person wearing the device.

It is exactly this interplay of multiple cameras, infrared light, and a trained recognition model that explains why a simple pinch works so reliably today, with no glove, no controller, and no visible markers on the hand itself.

Multiple cameras plus infrared light plus AI produce hand recognition.

^{Infographic: from camera image through depth calculation to the digital hand skeleton | Graphic: © Ulrich Buckenlei | XR Stager Online Magazine | VISORIC GmbH}

Notably, none of these three components would be sufficient on its own. Cameras without infrared light would fail in poor lighting, infrared light without multiple viewing angles could not compute reliable depth, and even perfect depth data would be nothing more than a meaningless point cloud without a trained model. Only all three together produce a hand recognition system that feels, for the user, like magic, even though it rests on perfectly explainable technology.

This recognition, however, is only half the story. How a recognized skeleton turns into an actually reliable, click-like input is shown in the next chapter.

• Several cameras capture the hand from different viewing angles.
• Invisible infrared light enables recognition even in poor lighting.
• A trained AI model places roughly 20 to 25 joint points.
• A dedicated chip processes the camera stream within milliseconds.
• Only all three components together yield reliable recognition.

For users, this means in practice that they never have to think about any of these technical layers, they only see the result: a hand that feels as if it is interacting directly with virtual objects. This invisibility of the underlying technology is exactly the real benchmark for good hand recognition.

When a Pinch Itself Becomes a Click

Above pure hand recognition lies a second, often overlooked layer: turning a recognized gesture into a reliable, repeatable input signal, comparable to a mouse click.

In the everyday work of developers building software for such headsets, this becomes tangible: an application that lets users grab, place, and combine virtual objects must trigger reliably on every single pinch gesture, otherwise the entire interaction immediately feels imprecise and frustrating. This is not a visual effect, but the result of a recognition system that must register precisely when thumb and index finger actually touch, and when they don’t.

Scientific studies on the accuracy of hand-tracking systems provide concrete figures here. A methodological study assessing VR hand-tracking systems using the Meta Quest 2 as a case found an average fingertip position error of 1.1 centimeters, an average finger joint angle error of 9.6 degrees, and an average delay of 45 milliseconds, values already precise enough for most everyday interactions.^[7]

This difference between mere recognition and genuinely reliable input is decisive. Comparative studies between hand tracking and classic controllers show that a grab or pinch gesture fundamentally requires more complex hand movements than simply pressing a button that already sits right under the finger, which is why reliably recognizing exactly this gesture is the real technical milestone, not hand recognition as such.^[8]

Only precise recognition turns a pinch into a reliable click.

^{Infographic: from the recognized gesture to a measurably precise, click-like input | Graphic: © Ulrich Buckenlei | XR Stager Online Magazine | VISORIC GmbH}

This precision also changes what hand tracking is actually good for in the first place. An imprecise recognition system might suffice for a broad gesture like a wave, but for precisely placing a 3D object in space, it takes centimeter-level rather than merely rough directional accuracy.

For a pinch to genuinely work like a click, though, this precision must be reliable not just in the lab, but also in the everyday use of a still-immature prototype. Exactly where this limit currently sits is shown in the next chapter.

• Reliable gesture recognition is what turns a pinch into input.
• A study measures 1.1 centimeters of fingertip position error.
• The measured angle error averages 9.6 degrees.
• Average delay comes in at around 45 milliseconds.
• Reliable gesture recognition is the real technical milestone.

These figures show that hand tracking is no longer an experimental niche technique today, but a scientifically measured, everyday-ready input method. Which is exactly why it’s worth looking at what current software still doesn’t deliver.

What Hand-Tracking Software Isn’t Yet

As convincing as individual demos may look, most of this software still runs on comparatively bulky developer headsets, not on finished, everyday-ready products published in an app store. This framing matters for assessing the technology realistically.

Many native applications for current headset platforms are still in the prototype stage and are only gradually published as standalone apps, often after an earlier version has already been available on a different platform.^[9] Whether a specific version shown in a demo has already entered closed testing, and exactly when a given recording was made, often cannot be independently verified for such developer projects.

The oft-described everyday glasses with full AI support don’t exist yet in this form, either. Serious developers usually state this themselves clearly as a direction, not as a finished product, an important difference from a marketing claim that would suggest an already available state.

Even pure hand-tracking prototypes run into a further, often overlooked limit: comparative studies between hand tracking and classic controllers show that tasks requiring more precision than a hand in mid-air can offer are, in practice, still frequently combined with small physical aids, a useful reminder for anyone picturing a fully screen- and device-free future.^[10]

Prototype, not everyday device: where the technology stands today.

^{Infographic: the current state of the technology between demonstrated principle and open technical implementation | Graphic: © Ulrich Buckenlei | XR Stager Online Magazine | VISORIC GmbH}

These limitations are not a reason to downplay the technology, they are a reason to place it correctly. A single, convincing demo proves that a principle works, it does not yet prove that it has become a mature, everyday-ready product.

Anyone who knows this limit can assess the technology realistically and recognize where hand-based interfaces can already be used productively today. Which industries could already benefit from this is shown in the next chapter.

• Many native headset apps are still in the prototype stage.
• Earlier versions often ran on other platforms first.
• The described everyday glasses with AI don’t exist yet.
• Serious developers themselves call it a direction, not a product.
• Some tasks still require physical aids.

Anyone who knows these limits also understands why the real value of these prototypes lies not in their technical completeness, but in making an otherwise abstract principle tangible. This exact principle can already be applied to entirely different fields today.

From the Developer Environment to Enterprise Applications

Detach the principle from a single design tool, and a field of application emerges that reaches far beyond creative software, everywhere people need to interact with complex, spatial content without being able to free their hands from a physical task.

Mixed-reality systems with hand tracking are already used today in education, healthcare, and industrial training, scientific evaluations of fine motor hand-movement recognition using devices such as the HoloLens 2 show concretely how precise hand tracking can support medical training scenarios and technical instruction, with no need for learners to hold a controller at all.^[11]

In remote assistance, a technician could rotate a virtual component with bare hands and show it to a colleague remotely, while both hands stay free for the actual tool. In retail, products could be configured virtually and placed in space without customers ever having to operate a controller, a low-barrier entry point that, according to current market overviews, ranks among the most frequently cited use cases for smart glasses.^[12]

One principle, many industries beyond the design tool.

^{Infographic: application fields of hand-based mixed-reality interaction beyond the original example | Graphic: © Ulrich Buckenlei | XR Stager Online Magazine | VISORIC GmbH}

A central advantage runs through all of these applications: wherever hands are already needed for a physical task, a short gesture replaces the detour through an extra device that would first have to be found, held, and put down again. For companies with many recurring, hand-intensive workflows, that is a substantial practical advantage.

This range of possible applications raises an obvious question: if a single prototype can already demonstrate this principle so convincingly, how close is a version that doesn’t just recognize but actually understands? That is exactly what the next chapter addresses.

• Hand tracking already supports medical and technical training.
• Remote assistance benefits from free hands for tool and gesture.
• Retail uses virtual product configuration without a controller.
• Gestures replace the detour through a separate extra device.
• Especially valuable for recurring, hand-intensive workflows.

The more industries take an interest in this principle, the more important the question becomes of how much a system understands on its own, rather than merely recognizing positions. That is exactly where the next stage of development begins.

When AI Ushers in the Next Stage

Today, every hand places every virtual building block one at a time, step by step, exactly as current hand-tracking prototypes consistently show. What today is still manual, individual work is already hinted at by current research as an automatable, AI-driven next step.

Research on generative 3D AI already shows how a mere description or a single photo can turn directly into a finished 3D object, with no hand needing to manually model every single detail.^[13] Applied to typical hand-tracking applications, that would mean: the hand no longer places every single building block, but roughly describes or sketches what should emerge, and the AI takes care of the rest of the spatial detail work.

That this direction is no fringe phenomenon is already shown by current consumer devices: current mixed-reality products such as Samsung’s Galaxy XR integrate AI assistants such as Google Gemini directly into the glasses, systems that not only recognize what an object is, but also what it means in its given context.^[14] Hand tracking thus lays the first building block, Artificial Intelligence will likely lay the next thousand.

From manual placement to AI-driven object generation.

^{Infographic: development line from hand-controlled individual placement to AI-driven generation | Graphic: © Ulrich Buckenlei | XR Stager Online Magazine | VISORIC GmbH}

For users, the effect would be substantial. Instead of a system that merely shows where the hand currently is, one would emerge that additionally understands what the person actually wants to achieve, and fills in the necessary spatial details on its own, with no need for every single gesture to be manually performed.

This step, from pure hand recognition to contextual AI support, is technically demanding but by no means science fiction. It builds directly on already established hand-tracking technology and shows where the entire mixed-reality industry is currently heading. How quickly these technologies are already spreading into everyday use is shown in the next chapter.

• Generative 3D AI creates objects from a description or photo.
• The hand could soon sketch instead of placing every detail.
• Samsung’s Galaxy XR integrates Google Gemini directly into the glasses.
• AI increasingly understands context, not just object recognition.
• Hand tracking lays the first block, AI follows at scale.

This integration is also why the field is developing so quickly right now. What used to be an elaborate research prototype is increasingly becoming a feature built directly into commercially available devices.

From Niche Headset to Everyday Tool

Just a few years ago, mixed-reality headsets were heavy, expensive, and limited to a small group of enthusiasts and professionals. Current market analyses show that this picture is already changing significantly by 2026.

A pair of everyday glasses simply has no room for buttons, a mouse, or a separate controller. Analysts at IDC therefore expect rapid growth for lightweight display glasses, at roughly 42 percent annual growth through 2030, alongside falling prices, with the hand becoming the central input method, much as swiping and tapping did for touchscreens.^[15]

This shift follows a pattern already observed in earlier interface transitions: new input methods first establish themselves in niches before falling costs and growing reliability carry them into the mass market, a pattern that repeated itself from the mouse through the touchscreen to voice control, and is now continuing with hand control.^[16]

From heavy specialist gear to lightweight everyday glasses.

^{Infographic: the development of mixed-reality headsets from niche product to mass market | Graphic: © Ulrich Buckenlei | XR Stager Online Magazine | VISORIC GmbH}

For companies, this development means a noticeably lower barrier to entry into a demanding technology. Rather than waiting for expensive, bulky specialist devices, hand-based interfaces can already be tested today on available consumer headsets, with falling prices and growing software support.

At the same time, this development shows how progress in mixed-reality interfaces unfolds overall: not through a single technological breakthrough, but through falling costs, growing reliability, and the increasing spread of already existing capabilities. How this development fits into the bigger picture of the spatial computing ecosystem is shown in the final chapter.

• Mixed-reality headsets were long heavy, expensive niche products.
• IDC expects roughly 42 percent annual growth through 2030.
• Falling prices make the hand the central input method.
• New input methods always follow the same adoption pattern.
• Consumer headsets already make entry accessible today.

This market shift is more than a pure cost question, it also changes who is even in a position to work with and plan around this technology. That is exactly the point that leads to the final, larger question of this article.

The Hand as an Interface Within the Larger Spatial Computing Ecosystem

Zoom out from the single example, and a principle emerges that matches exactly the core of every professional spatial computing system, just demonstrated here on a single voxel editor.

The preceding chapters have shown how a pair of glasses technically recognizes a hand, how a gesture becomes a reliable input, where the limits of current prototypes lie, and where development through AI-driven automation is headed. Together, these building blocks form a principle that reaches far beyond a single design prototype.

The difference between a single developer prototype and a professional digital twin lies not in the underlying principle, but in scale and structure: both permanently connect real, physical movement to digital information, and only that connection makes it genuinely usable.^[17] Current industry analyses confirm that exactly this connection between sensors, AI, and spatial representation is gaining importance across industries in 2026, from developer prototypes to industrial digital twins.^[18]

Design tool, factory, or clinic: the same interface, the same hand.

^{Infographic: hand-based interfaces as a building block of the larger spatial computing and digital twin ecosystem | Graphic: © Ulrich Buckenlei | XR Stager Online Magazine | VISORIC GmbH}

For companies investing in spatial computing and digital twins, a clear strategic consideration follows from this. If even a movement as fine and complex as a finger pinch can be reliably recognized and evaluated in real time, the question for every company becomes which part of its own, often considerably coarser operations is still controlled through cumbersome intermediary devices rather than direct, natural interaction.

Research and early broad adoption show that the individual building blocks of this capability are already technically mature and usable today. The path from an impressive individual prototype to everyday, company-wide practice is therefore, for companies that act now, considerably shorter than it appears at first glance.

• Such developer prototypes follow the same principle as professional twins.
• The difference lies in scale and structure, not the principle.
• Sensors, AI, and spatial representation are converging across industries.
• Direct hand interaction increasingly replaces cumbersome intermediary devices.
• The path to company-wide practice is now much shorter.

This closes the circle of this article. What begins with the desire to shape digital objects by hand movement instead of by controller evolves into a foundational principle for the next generation of mixed-reality interfaces, far beyond a single developer project. Just how convincing this principle already looks in practice is shown in the video below.

When a Hand Becomes a Tool in Space

The previous chapters have shown how hand tracking works technically, how precise it already is today, and where the limits of current prototypes lie. Just how convincing this principle already looks is most striking when looking directly at the original clip itself.

Embedded below is the original clip by Oleg Frolov, in which he tests his visionOS version of BoxelXR, a spatial voxel editor he is building with RealityKit. With simple hand gestures, he grabs, places, and combines virtual blocks directly in physical space, while hand tracking translates his movements into precise 3D input and spatial computing keeps the digital objects anchored around him.^[2]

This moment, translating bare hand movements into precise, digital 3D actions, illustrates in an instant what the previous chapters explained technically: Artificial Intelligence can take this interaction even further in the future by helping systems understand not just hands, but also objects and spatial context, the actual bigger shift behind it: smart glasses are evolving from pure information displays into systems that understand how we interact with our environment.

^{Video: hand-tracking demo of BoxelXR on visionOS, voxel objects are shaped directly in physical space via pinch gesture | Visuals by original creator Oleg Frolov (@olegdesignfrolov / @Volorf) | Analysis, script, editing, and video production: © Ulrich Buckenlei | XR Stager Online Magazine | VISORIC GmbH}

The video makes clear that hand-based mixed-reality interaction is no distant future concept, but already works today on available headsets, with no controller and no extra hardware in the hand at all. For companies thinking about their own space-based interfaces for training, maintenance, or design, this example shows just how close a practical implementation already is.

At the same time, the video reveals the decisive conceptual difference: it’s not the individual gesture that’s remarkable, but the fact that the hand itself becomes a full, precise interface, with no intermediary device at all. That is exactly what turns a developer demo clip into a reliable principle for the future.

• The video shows the original BoxelXR hand-tracking demo.
• Voxel blocks are shaped directly in space via pinch.
• Hand tracking translates movement into precise 3D input.
• The technology already works without a controller or extra hardware.
• Decisive is the hand as a full-fledged interface.

This example makes tangible where mixed-reality interfaces are heading: from an impressive developer demo to a reliable, everyday-ready interface for everything people will one day do with their hands in space.

From Idea to Your Own Spatial Interface

A reliable, hand-based interface doesn’t come from a single app, but from the thoughtful interplay of sensors, AI-driven gesture recognition, and a platform that makes this interaction permanently usable, exactly the combination at the core of VISORIC’s work.

The expert team at VISORIC GmbH in Munich combines over 15 years of experience in 3D, AI, and XR with hands-on experience in spatial computing, real-time 3D, and digital twins, exactly the building blocks a reliable spatial interface requires, whether it’s about training, remote assistance, product configuration, or an entire network of facilities. VISORIC builds the technical bridge from the first idea to a permanently usable, hand-based application, tailored to a company’s actual requirements.

15 years of experience in 3D, AI, and XR: the VISORIC expert team from Munich.

^{Image: © Ulrich Buckenlei | XR Stager Online Magazine | VISORIC GmbH}

A well-thought-out pilot project, a single training session, a single application, can often be realized considerably faster and more cost-effectively than many companies expect. VISORIC accompanies this path from the first concept through the technical implementation strategy to a fully operational, permanently usable spatial interface.

• Design of hand-based interfaces for training and maintenance.
• Integration of hand tracking, AI analytics, and spatial computing.
• From pilot application to company-wide spatial interface.

That is exactly the right starting point for a conversation: not the big, company-wide vision, but a clearly scoped, quickly implementable first step that shows how the principle translates to your own operational reality.

Do you want to control complex workflows, training, or facilities in the future through direct, hand-based interaction instead of cumbersome intermediary devices?

Talk to the VISORIC expert team in Munich about spatial computing, real-time 3D, and modern mixed-reality platforms. Together, we’ll turn your requirements into a precise, permanently usable spatial interface, with a tangible advantage in speed, intuitiveness, and user acceptance.

Contact:

Email: info@visoric.com
Phone: +49 89 21552678

Sources and References

1. Natural User Interfaces: Intuitive by Nature: The Evolution of Natural User Interfaces. FasterCapital.
2. Oleg Frolov (@olegdesignfrolov / @Volorf). Original video and post on BoxelXR for visionOS, Instagram and LinkedIn, 2026.

3. Don Norman. Natural User Interfaces Are Not Natural. jnd.org.
4. Jakob Nielsen. History of the Graphical User Interface: The Rise (and Fall?) of WIMP Design. uxtigers.com.

5. Apple. Official Vision Pro product page, camera, infrared, and R1 chip specifications, apple.com/apple-vision-pro.
6. PetaPixel. How Apple Vision Pro’s Infrared Eye-Tracking Technology Works.

7. Abdlkarim et al. A Methodological Framework to Assess the Accuracy of Virtual Reality Hand-Tracking Systems: A Case Study with the Meta Quest 2. Behavior Research Methods, 2023.
8. Encora. Beyond Controllers: Apple’s Vision Pro Brings Hand Gestures and Eye Tracking to Virtual Worlds.

9. Oleg Frolov. BoxelXR project page, boxelxr.com, and Meta Quest Store listing for BoxelXR.
10. Comparison of Hand Tracking-Based and Controller-Based Interaction in a Consumer Virtual Reality Game. Virtual Reality, Springer Nature, 2025.

11. Evaluation of HoloLens 2 for Hand Tracking and Kinematic Features Assessment. Virtual Worlds, 2025.
12. Treeview. Smart Glasses: The Complete Guide for 2026. treeview.studio.

13. SIGGRAPH 2025. 3D Generative AI Transforms How We Create, Design, Interact With Digital Content.
14. Framesixty. Mixed Reality Products in 2026. framesixty.com.

15. IDC. Smart Glasses Market 2026: XR Is Rewriting the Rules. idc.com.
16. Evolution of User Interfaces: From CLI to Immersive Tech. tusharbisht.com.

17. ISO 23247. Reference framework for digital twins, general definition and core principle.
18. Spatial Computing Industry Research Report 2026. GlobeNewswire, February 2026.

19. VISORIC case studies in digital twins, real-time 3D, and spatial computing.
20. XR Stager platform for real-time 3D, digital twins, Knowledge AI, and industrial spatial computing applications.