Sensing & Reasoning Lab · Rutgers University
Jorge Ortiz

Jorge Ortiz

Sensing & Reasoning

Building perception-to-decision systems with calibrated competence.

How should an AI system decide when evidence is insufficient?

Associate Professor, Rutgers Full Member, CS Graduate Faculty Director, Sensing & Reasoning Lab
Rutgers Site Director, CRAIG (NSF IUCRC) Rutgers Site Lead, CS3 (NSF ERC) Research Analyst, New York Yankees

About

I build AI systems for safety-critical environments: urban infrastructure, clinical spaces, human-machine systems, and sports. I pair perception and reasoning with explicit competency boundaries, the conditions under which the system should and should not drive action. The systems I work on maintain a calibrated sense of when their inferences are reliable and when they are not, so that high-stakes decisions can be deferred or handed off instead of guessed.

Research architecture: Perception and structure extraction feed into competency boundaries, then human-in-the-loop decision with justified output.

I study this in three domains: urban AI (causal discovery, privacy-preserving sensing), social reasoning in human–machine systems (robotics, driver attention), and multimodal health sensing (behavioral biomarkers, spatial intelligence). Each setting surfaces different ways systems can fail, and what it takes to know when to trust them.

I direct the Sensing & Reasoning Lab at Rutgers, serve as Rutgers Site Director for CRAIG (NSF IUCRC on Responsible AI & Governance) and Rutgers Site Lead for CS3 (NSF ERC for Smart Streetscapes). I am also a full member of the Rutgers Computer Science graduate faculty. Before Rutgers I was at IBM Research and several startups. MS and PhD from UC Berkeley, BS from MIT.

External engagement: Research Analyst for the New York Yankees, computer vision and biomechanics from video for player development and performance.

Research Program

We build systems that can say when their conclusions hold, in cities and health settings where you can’t afford to guess.

Urban AI

Urban environments generate a lot of sensor data, but data alone can’t tell correlation from causation. We work on causal discovery and privacy-preserving sensing so decision-makers can predict what happens under interventions.

TeLLMe does causal discovery from urban video with self-confidence assessment; we’ve run it on real dashcam data with a causal-card interface for policy-style questions. CityOS builds on a formal impossibility result (privacy and correct counting fail for the same reason, ambiguity) and is piloted for parking, assistive crossing, and crowd-density at CS3 testbeds in NYC, Rutgers, and West Palm Beach.

Social Reasoning in Human–Machine Systems

How can robots read social context when that takes common sense we don’t know how to program? And when are those inferences trustworthy enough to act on?

We work on VLM-based social HRI (nonverbal cues), SoNNET for bite timing in robot-assisted feeding, including group dining (CoRL 2022), and Project Paz for driver attention and when to interrupt. Collaboration with Cornell on caregiving robotics; published at CoRL and IMWUT.

Multimodal Spatial Reasoning

Instrumented spaces give you many modalities at once. The hard part is what a space affords and how to handle distribution shift, same kind of event, different readings across time and place.

DFGauss (NeurIPS 2025) does 3D occupancy prediction; Maestro is our 18-channel multisensor platform; CAMERA (NIH BRAIN Initiative, $5M) targets behavioral biomarkers for anxiety. Deployed across multiple campus buildings.

Centers & Initiatives

Different scales, governance, city infrastructure, the lab, and applied work.

NSF IUCRC · GOVERNANCE

CRAIG: Responsible AI & Governance

Role: Rutgers Site Director

When should AI systems act? We develop theoretical foundations and evaluation pipelines for systems that know their boundaries.

NSF ERC · URBAN SCALE

CS3: Smart Streetscapes

Role: Rutgers Site Lead

City-scale sensing and causal reasoning for pedestrian safety. We deploy and evaluate urban AI in the real world.

RESEARCH LAB · CORE

Sensing & Reasoning Lab

Role: Director

Perception, causal inference, and self-assessment, the core work that feeds the three pillars above.

APPLIED · SPORTS

New York Yankees

Role: Research Analyst

Computer vision and biomechanics from video for player development, strategy, and performance, systems that have to signal when they’re uncertain.

Selected Publications

Full list on Google Scholar.

Future Directions

I’m working toward systems that discover causal, social, and spatial structure, know where that structure applies, and can say so clearly to the people and organizations that rely on them. That means formal ways to represent when conclusions hold across tasks, methods for systems to learn their own boundaries from experience and feedback, and interfaces that make those boundaries interpretable.

This vision drives the next 5–10 years of my work through CRAIG (governance), CS3 (urban deployment), and NIH CAMERA (health sensing), three funded initiatives where knowing when to trust the system is central.

News

Apr '26
Rutgers CS Graduate Faculty: Approved as a full member of the Computer Science graduate faculty.
Apr '26
TraceFix accepted at ACM CAIS 2026: "Repairing Agent Coordination Protocols with TLA+ Counterexamples" with Xia, Li, and Ehsan.
Apr '26
Rutgers CS Colloquium: Invited talk, "Toward Trustworthy Agentic Systems for Smart Spaces."
Jan '26
2025
Appointed Rutgers Site Director for CRAIG (NSF IUCRC on Responsible AI & Governance).
2025
Rutgers Site Lead for CS3 (NSF ERC for Smart Streetscapes).
Sep '25
Three NeurIPS 2025 papers accepted: one main conference (DFGauss), two workshops (TellMe Why, PolicyGrid).

Media & Press

WALL STREET JOURNAL

The Overestimation of Artificial Superintelligence

Interviewed about AI capabilities and the hype around superintelligence

NEW SCIENTIST

Robot that learns social cues could feed people with tetraplegia

Coverage of our CoRL 2022 work on bite timing prediction for robot-assisted feeding in group dining settings

July 2022

Teaching

Selected courses

ECE 252: Programming Methodology I

Introduction to C and C++ (100–170 students, taught six times). Covers language expressivity, OS memory models with a pictorial representation I devised, and object-oriented design: students develop OO strategies using only C data structures. I work backward from practice to theory, framing recursion through web crawlers before presenting the generalization.

ECE 532: Multimodal Learning for Sensing Systems

Multimodal learning across audio, video, time series, and text for sensing systems. Students work on real data and deployed settings; capstone teams build systems that run in the lab and in the field.

Contact