Sign up to access an inspiration library of 30,000+ posts
Jim Fan's Recent LinkedIn Posts
Jim Fan
@drjimfan
NVIDIA Director of AI & Distinguished Scientist. Co-Lead of Project GR00T (Humanoid Robotics) & GEAR Lab. Stanford Ph.D. OpenAI's first intern. Solving Physical AGI, one motor at a time.
Posts
Jim Fan
Tech & AI
9mo
BEHAVIOR challenge: a new robotics leaderboard just dropped! It's bringing back the vibe of ImageNet’s early days. There was something deeply satisfying about ImageNet: a well curated training set; a clearly defined testing protocol; a competition that rallied the best researchers; and a leaderboard that spawned ResNets and ViTs, and ultimately changed the field for good.
The LLM world is similar. No matter how much OpenAI, Anthropic, and xAI disagree, they at least agree on one thing: benchmarking. MMLU, HLE, SWEBench - you can’t make progress until you are able to measure it.
Robotics still doesn’t have such a rallying call. No one agrees on anything: hardware, task, scoring, simulation engine, or real world environment. Everyone is SOTA, by definition, on the benchmark they define on the fly for each paper.
From the maker of ImageNet - BEHAVIOR takes a stab at the daunting challenge of unifying robotics benchmarking on a reproducible physics engine (Isaac Sim). The project started before I graduated from Stanford Vision Lab, and I witnessed up close the many years of dedication and entire PhD careers it took to bring to life. I hope BEHAVIOR is either the hill-climbing signal we need, or the spark that finally gets us talking about how to measure real progress as a field.
https://lnkd.in/gcJKKHXR
1.8K
Jim Fan
Tech & AI
9mo
BEHAVIOR challenge: a new robotics leaderboard just dropped! It's bringing back the vibe of ImageNet’s early days. There was something deeply satisfying about ImageNet: a well curated training set; a clearly defined testing protocol; a competition that rallied the best researchers; and a leaderboard that spawned ResNets and ViTs, and ultimately changed the field for good.
The LLM world is similar. No matter how much OpenAI, Anthropic, and xAI disagree, they at least agree on one thing: benchmarking. MMLU, HLE, SWEBench - you can’t make progress until you are able to measure it.
Robotics still doesn’t have such a rallying call. No one agrees on anything: hardware, task, scoring, simulation engine, or real world environment. Everyone is SOTA, by definition, on the benchmark they define on the fly for each paper.
From the maker of ImageNet - BEHAVIOR takes a stab at the daunting challenge of unifying robotics benchmarking on a reproducible physics engine (Isaac Sim). The project started before I graduated from Stanford Vision Lab, and I witnessed up close the many years of dedication and entire PhD careers it took to bring to life. I hope BEHAVIOR is either the hill-climbing signal we need, or the spark that finally gets us talking about how to measure real progress as a field.
https://lnkd.in/gcJKKHXR
1.8K
Jim Fan
Tech & AI
9mo
BEHAVIOR challenge: a new robotics leaderboard just dropped! It's bringing back the vibe of ImageNet’s early days. There was something deeply satisfying about ImageNet: a well curated training set; a clearly defined testing protocol; a competition that rallied the best researchers; and a leaderboard that spawned ResNets and ViTs, and ultimately changed the field for good.
The LLM world is similar. No matter how much OpenAI, Anthropic, and xAI disagree, they at least agree on one thing: benchmarking. MMLU, HLE, SWEBench - you can’t make progress until you are able to measure it.
Robotics still doesn’t have such a rallying call. No one agrees on anything: hardware, task, scoring, simulation engine, or real world environment. Everyone is SOTA, by definition, on the benchmark they define on the fly for each paper.
From the maker of ImageNet - BEHAVIOR takes a stab at the daunting challenge of unifying robotics benchmarking on a reproducible physics engine (Isaac Sim). The project started before I graduated from Stanford Vision Lab, and I witnessed up close the many years of dedication and entire PhD careers it took to bring to life. I hope BEHAVIOR is either the hill-climbing signal we need, or the spark that finally gets us talking about how to measure real progress as a field.
https://lnkd.in/gcJKKHXR
1.8K