Publications

We introduce GHADAR, the task of per-person, open-set attribute and activity description in multi-person videos, together with AVA-Captions — the first large-scale grounded dataset of this kind, built by extending AVA-Actions with VLM-generated captions and identity-aware deduplication. We propose CAMP (Constrained Attention Masking-based Pretraining), a two-stage VLM training strategy that explicitly leverages grounding through attention-mask constraints and outperforms state-of-the-art VLMs, alongside a VLM-driven evaluation framework that compares video and prediction at the concept level rather than via n-gram or embedding metrics.

A deterministic temporal process can be determined by its trajectory, an element in the product space of an initial condition and a transition function. We introduce a mechanism to learn the distribution of trajectories by parameterizing the transition function explicitly as an element in a function space. The framework allows efficient synthesis of novel trajectories while directly providing tools for inference — uncertainty estimation, likelihood evaluation, and out-of-distribution detection for abnormal trajectories.

Quantitative descriptions of confidence intervals and uncertainties are needed in many vision and machine learning applications. We show how revisiting results from Random Field Theory, paired with deep neural network tools, leads to efficient frameworks that provide hypothesis-testing capabilities for uncertainty maps from models used in many vision tasks.

We propose Warping Neural ODE, a framework for generating smooth animations (video frame interpolation) in a continuous manner given two far-apart frames. By using continuous spatial transformation of the image based on a vector field derived from a system of differential equations, the method achieves smooth, realistic animations with infinitely small time steps between frames.

Recent multimodal models such as ViLBERT and UNITER show impressive performance on vision-language tasks but remain sensitive to subtle variations in input. We propose a novel adversarial objective that incorporates the distribution of possible linguistic variations, and a VQA-specific mixup technique leveraging object replacement, demonstrating improved model robustness on benchmark datasets.

A Bayesian Neural Network approach for the cyber-defense domain, designed to handle sparse, class-imbalanced, and size-limited datasets. The method serves as a human-in-the-loop alarming system for detecting potential ransomware attacks.

We propose an ordered time series classification framework robust against missing classes in the training data. It relies on a newly proposed ordinal-quadruplet loss that preserves the ordinal relation among labels, and a testing procedure that exploits order preservation in the latent representation — significantly improving prediction of missing labels even when 40% of classes are absent during training.

Panel data — longitudinal measurements of the same set of entities over multiple time points — is common in studies of early childhood development and disease modeling. We propose ME-NODE, a probabilistic model that incorporates fixed and random mixed effects, derived using smooth approximations of SDEs via the Wong-Zakai theorem. We demonstrate its utility on tasks ranging from toy datasets to real longitudinal 3D imaging data from an Alzheimer’s disease study.

Moving beyond simple Gaussian formulations in deep uncertainty estimation requires Monte Carlo sampling, which scales poorly as data and model dimensions grow. We construct a framework to describe the computation graphs of these estimates and identify probability families where the graph size is independent of — or only weakly dependent on — the number of MC samples, enabling far more iterations with gains in confident accuracy, training stability, memory, and time.