Interaction Understanding: The Missing Layer for Physical AI

The physical world is not a collection of isolated objects. It is a stream of interactions.

A customer operating a vending machine.
A guest occupying a table.
An employee handling food.
A container left on the floor.
A person looking at a menu board.
A car entering another vehicle’s blind spot.
A robot aligning a tool with an object.

In each case, the important signal is not the presence of a single object. The value comes from understanding how people, objects, places, and processes relate to each other over time.

This is why we believe the next major layer for Physical AI is interaction understanding.

Not just object detection.
Not just image captioning.
Not just visual question answering.

But real-time recognition of what is happening in the physical world:

Who is interacting with what?
Where is that interaction happening?
What state is the interaction in?
How is it changing?
How confident is the system?
What should happen next?

At Palona, we believe Physical AI needs interaction understanding that is affordable, real-time, calibrated, and grounded in operational reality.

From objects to interactions

Today’s AI vision systems are impressive. They can detect objects, segment images, describe scenes, and answer questions about visual content.

But without interaction understanding, many multimodal and robotics systems remain instruction-reactive rather than event-proactive.

A robot or embodied model may be able to answer questions about a scene, follow a human instruction, or plan a motion after being told what to do. That is useful, but it is not enough for many real-world environments. Restaurants, stores, kiosks, factories, and autonomous systems cannot wait for a human to explicitly ask, “Is something wrong?” or “Should you respond now?”

Physical AI needs to continuously recognize meaningful interaction states on its own: a customer is stuck at a vending machine, a guest has occupied a table, an employee is handling food, a container has fallen onto the floor, a line is forming, a robot’s tool is misaligned, or a person has entered a risky spatial relationship with a vehicle.

This is the difference between understanding a scene of static objects and understanding a live environment.

Static scene understanding asks: What is here?
Instruction-following asks: What did the human tell me to do?
Interaction understanding asks: What is happening now, and does it require a response?

Without interaction understanding, Physical AI remains reactive. With interaction understanding, it can become proactive.

Why current models are not enough

Large vision-language models are very useful, but they are not yet enough for many deployed physical-world systems.

First, many are not fast enough. A model that takes several seconds to reason over a scene may be useful for offline analysis, but restaurants, stores, robots, vehicles, and kiosks need systems that can react in real time.

Second, many are not affordable enough. Physical AI often requires dense deployment across cameras, locations, devices, and edge environments. If each inference requires a very large model or expensive hardware, many real-world applications become economically infeasible.

Third, many are not calibrated enough. In physical environments, a system must know when it is confident and when it is uncertain. “The customer is operating the kiosk with high confidence” and “the customer may be operating the kiosk, but the view is partially occluded” should trigger different downstream behavior.

Fourth, many models are not grounded in operational state. A caption such as “a person is standing near a counter” may be correct but still not useful. A business or robot needs to know whether the person is ordering, waiting, being served, blocking traffic, or needing assistance.

This is the gap between visual description and interaction understanding.

Physical AI needs models that can transform raw perception into structured interaction state.

Interaction understanding requires space, time, and context

Interaction is broader than physical contact.

A customer pressing a vending-machine button is interacting with the machine. But so is a customer standing in front of it, looking at the screen, hesitating, and reaching toward the payment panel.

A guest sitting at a table is interacting with a place. A person waiting in line is interacting with a process. A bag blocking a walkway is interacting with the environment by creating an operational constraint. A driver attending to a pedestrian is interacting with a dynamic safety context.

So interaction understanding requires several layers:

Objects and actors: people, tools, food containers, tables, counters, vehicles, robots, machines.
Spatial relations: near, above, below, inside, in front of, behind, aligned with, blocking, occluding.
Temporal relations: approaching, leaving, waiting, moving together, stopping, repeating, transitioning.
Semantic context: ordering, serving, cleaning, operating, occupying, browsing, preparing, delivering.
Uncertainty: whether the system knows enough to act or should fall back to a safer behavior.

The core unit is not the object. The core unit is the interaction state.

Restaurants as a real-world learning environment

Restaurants are one of the best starting points for interaction understanding.

They are dynamic physical environments. People move, wait, order, sit, leave, pick up food, look for help, and interact with employees, tables, counters, kiosks, menu boards, pickup shelves, and food containers. Operational state changes every few seconds.

At the same time, restaurants are practical deployment environments. Palona can deliver immediate value to customers while learning from real operational data. That is an important difference from purely offline research datasets.

Restaurants also sit in a useful middle ground for Physical AI. They are much more complex than static image benchmarks, but they do not require the extreme precision of robot manipulation or the safety-critical guarantees of self-driving cars. This makes them a powerful proving ground: rich enough to expose real-world complexity, but practical enough to deploy, learn, and improve quickly.

The data advantage matters.

One of the hardest problems in Physical AI is that the real world does not come with clean labels. Lighting changes. Camera angles vary. People occlude each other. Objects are moved, stacked, dropped, or hidden. Different locations have different layouts and workflows. Human labeling is expensive, and many of the most important states are temporal rather than visible in a single frame.

Palona’s advantage is that customer deployments can create a data flywheel. We can provide value in real environments while learning from the repeated patterns, temporal transitions, edge cases, and operational feedback that those environments produce.

The learning is not limited to restaurants.

The same interaction-understanding layer can generalize to many physical-world applications: vending machines, retail stores, warehouses, service kiosks, food production, robotics, autonomous vehicles, smart buildings, and industrial environments.

The domain changes. The objects change. The risk profile changes. But the core problem remains the same:

Understand how people, objects, places, and processes interact over time.

GRIT: Geometry-Reinforced Interaction Training

To build toward this, Palona is developing a core technology we call GRIT: Geometry-Reinforced Interaction Training.

The idea is simple at a high level: interaction understanding should not rely only on language supervision or human labels. The physical world contains structure. Geometry, motion, temporal continuity, object persistence, repeated workflows, and multi-view consistency all provide learning signals.

GRIT uses these signals to help convert unlabeled operational data into supervision for interaction-state models.

Rather than treating foundation models as the final product, we use them as part of a learning system. Segmentation, depth, temporal tracking, geometric consistency, and frontier multimodal reasoning can help bootstrap and validate interaction labels. High-confidence agreement becomes training signal. Disagreement becomes a source of active learning and model improvement.

The goal is not merely to describe a scene. The goal is to train efficient models that can recognize interaction states in real time, at deployment cost, with calibrated uncertainty.

This is especially important because many valuable interaction states are not labeled directly. A customer “operating a vending machine,” an employee “handling fish,” a guest “occupying a table,” or a person “looking at a menu board” are states that emerge from spatial, temporal, and semantic cues together.

GRIT is our path toward learning those states from the real world.

Where Palona fits in the Physical AI ecosystem

The broader AI ecosystem is moving toward spatial intelligence, world models, and embodied reasoning.

World Labs is building frontier models for spatial intelligence that can perceive, generate, reason, and interact with the 3D world. Google DeepMind’s Gemini Robotics work emphasizes visual and spatial understanding, task planning, and success detection for robots. Meta’s V-JEPA 2 explores self-supervised video models that support understanding, prediction, and robot planning. NVIDIA Cosmos is positioned as a world-foundation-model platform for Physical AI, including robotics and autonomous vehicles. These are strong signs that the field is moving beyond static perception and toward AI systems that understand and act in the physical world.

Palona’s focus is complementary. Instead of focusing on scene understanding, instruction following, or action generation after a task has already been specified, Palona focuses on the layer before that: continuously recognizing interaction state in the real world.

If world models ask, “What could happen next?” interaction understanding asks, “What is happening now?”

If robotics models need to plan, interaction understanding provides the state they need to plan from. If autonomous systems need to reason about risk, interaction understanding helps structure the relations among agents, objects, places, and time. If multimodal models need grounding, interaction understanding can provide real-world validation signals. If simulation systems need realistic scenarios, interaction understanding can help extract those scenarios from operational data.

Our vision is that interaction understanding becomes a cornerstone layer for many more ambitious Physical AI systems.

It can serve as:

A state layer for robots and embodied agents.
A validation layer for multimodal and world models.
A data-generation layer for training and simulation.
A real-time operational layer for businesses.
A bridge between perception, planning, and action.

This is why we think interaction understanding is not a narrow restaurant problem. It is one of the missing abstractions for Physical AI.

The problem we are working on

Our thesis is simple:

Physical AI should not stop at detecting objects. It must understand interactions.

The real world is made of people, objects, places, and processes changing together over time. To act intelligently, AI systems need to recognize those interaction states accurately, affordably, in real time, and with calibrated uncertainty.

At Palona, restaurants give us a practical place to build this capability while already delivering value to customers. GRIT gives us a path to learn from real operational data without depending entirely on expensive human labels. And the broader Physical AI ecosystem gives this work a much larger horizon.

Interaction understanding can help a restaurant know when a customer needs service.
It can help a vending machine know when someone is operating it.
It can help a robot know how a tool relates to an object.
It can help an autonomous system reason about surrounding agents.
It can help world models and multimodal systems ground their predictions in real physical state.

A single object is rarely the unit of value.

The value is in the interaction.

If you are working on spatial intelligence, robotics, autonomous systems, multimodal models, self-supervised video learning, or world models, we would love to talk.

And if you are a PhD student excited by the question of how AI should learn interaction state from the physical world, we are especially interested in hearing from you.