Ask today's VLM, "Is there a knife on the floor?" and it may answer correctly. But it will not independently tell you that a baby is crawling toward it.

That gap—between answering a question and recognizing that something matters—is the core problem.

Current VLMs are mostly reactive. A human decides what to ask, when to ask it, and which parts of the world deserve attention — the human is still responsible for defining the question and, implicitly, the space of possible concerns.

A proactive model must do something different. It must continuously observe how a scene is evolving, predict what may happen next, determine whether any of those futures matter, and decide whether it should speak or act—without waiting for a predefined question.

The question is what such a system would actually look like.

Why repeated prompting is not proactivity

A common approximation is to run a fixed prompt every few seconds:

Is anything dangerous happening?

This can be useful, but it has a fundamental limitation: the model is still operating inside a human-defined task. The prompt determines what the model searches for and how it interprets the scene.

Adding more prompts expands coverage, but it does not remove the underlying constraint. The system still depends on someone anticipating the relevant situations in advance.

This resembles the limitation of fixed-vocabulary object detectors before open-vocabulary models: no matter how many categories are enumerated, reality contains a longer tail.

A genuinely proactive system reverses the direction of interaction:

model → human, rather than human → model.

The model should not merely answer questions about the world. It should decide when the world has produced something worth bringing to a person's attention.

What proactivity requires

At a high level, a proactive VLM needs four capabilities.

1. Maintain a predictive state of the scene

The system needs more than a representation of the current frame.

It should maintain a state that captures objects, people, relationships, motion, history, and possible intent: a person walking toward a door, a pan heating on the stove, or a baby moving across the floor toward a knife.

This state should represent not only what exists now, but also how the scene is changing and where it may go next.

In other words, the model needs a world state that can be rolled forward — a world model in the sense of blogs 01 and 02.

2. Predict multiple possible futures

The future is not deterministic. From the same scene, many trajectories may be possible.

The person may open the door, stop to answer a phone, or turn around. The baby may continue toward the knife, become distracted, or be picked up by an adult.

The model should therefore predict a distribution over future trajectories rather than a single next state.

One possible implementation is to model the latent dynamics continuously — the Neural ODE move (Chen et al., 2018):

dz/dt = f(z, t)

The current state z evolves according to learned dynamics, producing a distribution over possible future trajectories.

I have previously worked on related problems through Mixed-Effects Neural ODEs for longitudinal dynamics and variational sampling of temporal trajectories. The same general machinery can be used here: represent uncertainty not as one predicted future, but as a set of plausible futures and their probabilities.

3. Estimate importance, not only probability

An unusual trajectory is not automatically important.

A shadow moving across the floor may be statistically rare but irrelevant. A child approaching a staircase may be completely common but still require attention.

This means that pure anomaly detection is insufficient. The system must evaluate at least two separate properties:

  • How plausible is this future?
  • How consequential is this future?

Probability tells us what is likely to happen. It does not tell us whether the outcome matters.

A proactive system therefore needs some estimate of risk, utility, goal relevance, or expected consequence. A simplified decision score might look like:

intervention value = probability × consequence × relevance

This score would also depend on uncertainty, urgency, the model's confidence that intervention would help, and the cost of interrupting the user.

The baby crawling toward a knife matters not because the trajectory is unusual, but because one plausible continuation has a severe outcome and there is still time to prevent it.

4. Decide whether to speak or act

Once the model predicts a meaningful future, it must decide what to do.

Possible decisions include:

  • remain silent;
  • continue observing;
  • ask a clarifying question;
  • warn someone;
  • redirect a robot's attention;
  • physically intervene.

This is the actual point at which the system becomes proactive.

The model is no longer answering, "Is there a knife?" It is reasoning:

The baby is moving toward the knife. Contact is plausible within several seconds. No adult appears to be responding. A warning now has high expected value.

Then it speaks without being asked.

A minimum viable architecture

A minimal proactive VLM could contain the following components:

Architecture of a minimum-viable proactive VLM: a multimodal encoder builds a persistent scene state, a predictive dynamics model rolls it forward into a future trajectory sampler, a consequence and relevance evaluator judges which futures matter, and an action policy decides whether to stay silent, observe, warn, or intervene.
The minimum viable proactive VLM — consequence and relevance gate the action policy, not raw likelihood alone.

Note where the language model sits: last, after the action policy has already decided to speak. It is not the intelligence at the center of the system — it is the interface used to explain a decision that was already made.

The difficult part is not generating the warning. The difficult part is deciding that a warning should exist.

The hard design problems

Problem Why it's hard
Calibrate interruption Interrupt too often and people start ignoring it; wait for near certainty and it responds too late. The right threshold depends on the environment, the user, the severity of the outcome, and how much time is left to act — water on the floor means something different in an empty warehouse, a home with a toddler, and a hospital corridor. Calibration can't be global; it has to be contextual, and eventually personalized.
Model consequence Likelihood alone can't distinguish a harmless surprise from a dangerous but familiar event. The system needs an actual model of physical and social consequence: what may be damaged, who may be harmed, which human goal may fail, whether intervention could still change the outcome, what a false alarm costs. That needs learned priors, environment-specific goals, human feedback, episodic memory, and explicit constraints.
Set an editorial policy Most observations — even unusual ones — aren't worth communicating. The model needs an internal sense of what deserves attention, closer to an editor deciding what's newsworthy than a captioner describing everything in a frame. The question isn't "What is happening?" — it's "What is happening that this person should know about now?"
Maintain memory Understanding what's unusual, relevant, or unresolved requires remembering the past — whether the front door is normally left open, whether the person was already warned, whether the stove was turned on intentionally, whether someone is expected to arrive. Without memory the system reinterprets the world from scratch every time; it may detect events, but it can never maintain responsibility for them.

A deeper framing

This whole design is a special case of predictive processing:

A model continuously predicts how the world will evolve. Attention, communication, and action occur when the predicted futures contain something important enough to justify intervention.

Speaking is just one action available to a system built this way — not a separate feature bolted onto a predictor, but a direct consequence of predicting well and weighing consequence, as laid out above.

Failure modes

Failure mode What goes wrong
Alert fatigue Everyday environments contain enormous variation — lighting changes, objects move, people behave inconsistently, sensors fail. A naive system finds endless reasons to interrupt, and once the false-alert rate gets high, the product stops functioning regardless of its benchmark accuracy. This is a decision-policy problem as much as a perception problem.
Distribution drift The definition of normal changes over time: a baby enters the household, furniture moves, a person develops a new routine, seasons change lighting. The system must adapt without immediately absorbing dangerous behavior into its baseline — learning continuously, but not so aggressively that repeated risk becomes accepted as normal.
Missing representations The system depends entirely on what its representation can perceive. When the encoder doesn't represent a relevant object, relationship, or social cue, the dynamics model can't predict its consequences — the event isn't anomalous, it's invisible. No policy, however sophisticated, can recover information the representation discarded.
Mistaking rarity for importance Rare events are often harmless, and common events can be dangerous. Treating low likelihood as the primary trigger produces the wrong behavior — probability has to stay separate from consequence, relevance, urgency, and actionability.
Gradual normalization A system that continuously updates its baseline can be manipulated, or can simply drift, into accepting worsening conditions. Unsafe behavior introduced slowly can become statistically normal, so proactive systems need stable safety priors and constraints that observation alone can't overwrite.
Unnecessary intervention Even a correct prediction doesn't guarantee a warning helps. Someone may already understand the situation, another person may already be responding, a warning may distract at the wrong moment, physical intervention may create more risk than it prevents. The system has to predict not just the world, but the likely effect of its own actions.

The thesis

A proactive VLM is not simply a VLM that runs more often or receives a broader prompt — it has a different job. A reactive model answers questions about the current world; a proactive model maintains a persistent understanding of the scene, predicts multiple possible futures, evaluates their consequences, and decides whether speaking or acting would improve the outcome. Its core loop is:

observe → maintain state → predict futures → evaluate consequences → choose whether to intervene

The difficult problem is not generating a description of what the camera sees. It is deciding which possible future matters, whether the model should interrupt, and what action would actually help — the shift from perception to agency, and from a VLM that answers questions to a system that understands when a question should never have needed to be asked.