Predictive Representation Learning at the Right Granularity
From JEPA-style regularization to variable-granularity concepts in language models
My initial work showed that JEPA-style prediction can be used as a regularizer alongside a primary learning objective, removing the need for a moving-average target network. This led to a broader question: rather than predicting between fixed views or individual tokens, can models learn the appropriate granularity of abstraction? Subsequent work explores randomized fractional views, boundary bottlenecks, and continuous concepts as mechanisms for concentrating information into representations of meaningful structure.
From structural regularization to semantic inductive bias
Most conventional regularization constrains the structure of a model: parameter magnitude, activation geometry, smoothness, sparsity, or information flow. But the representations we ultimately care about are semantic. This raises a more fundamental question: can regularization act directly on how meaning is organized?
JEPA-style learning offers one route by encouraging representations of related views to remain predictively self-consistent. Our initial work recast this principle as a semantic regularizer: an inductive bias that may help organize representations around meaning rather than structure alone.
Our broader hypothesis is that semantic constraints may be particularly valuable in language and other high-dimensional modalities, where the meaningful units of representation are not given in advance. Stochastic gradient descent can fit the training objective, but it may not reliably discover the right abstraction boundaries or information bottlenecks on its own. Semantic-aware regularization may provide the small inductive bias—the “nudge”—needed for meaningful structure to emerge.
Publications
Semantic Tube Prediction: Beating LLM Data Efficiency with JEPA
Relaxes LLM-JEPA's two-view requirement through randomized fractional views and introduces a JEPA-style regularizer that improves signal-to-noise ratio and preserves diversity during LLM fine-tuning.
PaperLLM-JEPA: Large Language Models Meet Joint Embedding Predictive Architectures
First successful integration of JEPA with LLMs, establishing JEPA as a practical regularizer for language models while eliminating the moving-average mechanism traditionally required to stabilize JEPA training.
PaperCurrent Direction
Developing a unified account of perception and generation as prediction at different levels of abstraction.