When looking at molecular structures, and especially when trying to explore their conformations and interactions interactively, computer systems based in flat screens and keyboard-mouse combinations are extremely limiting. Such interfaces feel unintuitive given the poor coupling of 2D inputs and outputs to the actually 3D nature of molecular structure. They are also clunky to use due to the abundance of menus and commands. They offer virtually no way for multiple users to collaborate concurrently other than by sharing video feed of the screen. Even a single user is limited in that s/he can act on the systems at single points.

Certainly, in the light of all the recent advances in artificial intelligence, speech recognition, virtual and augmented realities, and other technologies, regular computer graphics software seems, at the very least, antiquated.

Now, imagine being able to manipulate and explore molecular systems in an immersive environment, using your two hands in three dimensions to move and deform molecules, and commands spoken in natural language to control the environment.

This might sound like science fiction, but it isn’t anymore. A team of researchers next door from Nexco at EPFL have developed a novel tool called HandMol, which enables impressively immersive and intuitive molecular graphics and modeling. In its most advanced mode, running in high-end headsets with hand tracking capabilities such as Meta’s Oculus Quest 2 and its successors the Quest 3 and Quest Pro, Apple’s Vision Pro (and several other headsets released in the last 5 years), HandMol allows users to view molecules in real 3D and to interact with them by using their hands to move, break, create and alter the structures of molecules and multimolecular systems.

In the following example video, two users build some simple molecules together by bringing atoms together while a specialized AI system trained on DFT calculations handles the physics in order to produce realistic bond lengths and angles (and energies, not shown but available for further calculations):

In the next screenshot, here’s one of us from Nexco with privileged access to the tool, exploring in augmented reality how imatinib, a cancer growth blocker known commercially as Gleevec, docks to its tyrosine kinase inhibitor as seen in an X-ray structure:

Note in the above screenshot that using your hands is not only natural to move atoms in real 3D without any 2D-to-3D projection as regular molecular graphics programs do, but also even to simply and naturally point at elements of the structure, essential when discussing among peers or instructing in an educational or training session.

Indeed, by using their hands natural inside HandMol users can change the conformations of molecules, move them around in space, build or break bonds, make measurements, control menus, etc. Here, for example, a user is measuring the distance between two atoms simply by pointing at them with his index fingers:

Integration of six modern technologies

At its core, HandMol combines 6 modern computer technologies, all in web-based flavors that allow it to exist as a stand alone app that runs directly as regular web pages inside regular for web browsers. These technologies are WebXR for visualization in virtual reality (VR) or augmented reality (AR), hand tracking via the infrared cameras built into modern VR headsets, speech recognition assisted by one of OpenAI’s large language models, molecular mechanics not only with regular forcefields but also with machine learning-based forcefields trained on quantum simulations, and WebRTC for collaborative capabilities.

Testing HandMol for free

The best of all this, is that you can try HandMol out now. Even if you don’t have a VR headset!

In its current prototype version, HandMol is free to use. And you don’t even need a VR headset to at least see it at work, because it runs in the web browsers of all modern devices: from virtual reality headsets where HandMol exposes the ability to manipulate molecules with your bare hands while seeing them as if they were right there in your space, to smartphones that allow deeply immersive graphics via $5 cardboard goggles using the VR mode or at zero cost without any goggle at all if in AR mode. The following picture summarizes the smartphone-based VR and AR models:

Connecting multiple HandMol sessions

By connecting two instances of HandMol running in two VR headsets users can act directly with all four 4 hands simultaneously on the same molecular system, allowing collaboration like never before, as shown in the first video of this post.

Likewise, by connecting HandMol instances in headsets and computers, users can easily ask for commands to move files, activate features, etc. in spoken form, with the versatility enabled by modern large language models coupled to speech recognition systems. This akes the learning curve faster, as the user doesn’t need to remember the locations of menus or any scripting syntax.

Mechanics

HandMol brings built-in capabilities for molecular mechanics that allow it to treat different types of molecular structures, from proteins and nucleic acids handled via force fields from the Amber of Charmm families to machine learning-assisted forcefields trained on DFT data that allow ab initio but extremely fast calculations including bond breaking and formation.

In parallel, an efficient code for rigid body mechanics provides a fluid visualization of molecular motions and hand-induced displacements and deformations.

The Future is Here, and we at Nexco Follow it Closely!

Fascinated with HandMol and with the exciting future it inaugurates, we at Nexco envision applications in various domains such as chemistry, molecular biology, structural biology, biochemistry and pharmacology; for work, training, education, engaging with funders and clients, etc.

What else is within reach of modern XR technology?

It is an interesting exercise to perceive how all these technologies have been around for sometime already, but required a group of people to put them together to work. And then, it is also interesting to think about what other features can current technologies provide to these kinds of tools — either immersive tools for molecular graphics or for other applications.

Inspecting the current state of various technologies, it is clear that they are quite ripe to allow programmers to add tools to select, move, rotate, scale, measure, and color molecules, as well as for creating and editing molecular bonds in real time, all using simple and natural hand motions and voice instructions and assisted by realistic physics. Likewise, apps could provide real-time feedback and guidance by highlighting atoms, displaying interactive labels, playing sounds, etc. to enhance the user experience and situational awareness.

Thanks to the power of AI-based speech recognition, language models, the WebXR technology itself, and modern AI-assisted methods to compute mechanics, it is all just about setting one self to code, to make science fiction a reality.

References and Related Entries From our Blog

If you are interested in learning more about HandMol, you can read the full preprint describing it here:

HandMol: Coupling WebXR, AI and HCI technologies for Immersive, Natural, Collaborative and Inclusive Molecular Modeling

And here’s a previous blog post by us, reporting a related tool from the same EPFL lab that does not deal with atoms but with 3D objects of broader nature allowing users to display and manipulate density maps from QM calculations, Cryo-electron microscopy or tomography data, complex visualizations, etc.

New free multiuser virtual reality tool allows easier 3D presentations and discussions in chemistry and biology