On the industry panel at NeurIPS

Kenneth Tran joined the industry panel at Physical Reasoning and Inductive Biases for the Real World, a workshop at the NeurIPS machine learning conference on 14 December 2021. The workshop set out to put machine learning researchers in the same room as people from the physical sciences, cognitive and developmental psychology, and robotics, and to ask what it would take for the work to survive contact with the real world.

He sat alongside Aleksandra Faust, Hiro Ono and Michael Roberts. The recording is on SlidesLive.

The argument he made

The panel was asked how academic research becomes applicable to real problems. Tran’s answer was about benchmarks, and it is the reason the panel is still worth linking to:

ML benefited from benchmarks initially, but the progress can be slowed down by benchmarks as well; if there is not enough emphasis on the interdisciplinary research, the focus can move away from research directions that can handle realistic data.

Why this is the company’s position too

That is not an abstract point for us. A greenhouse or a dryer does not resemble a benchmark. The inputs move, the sensors drift, the actuators are slow, and the thing you are controlling responds over hours rather than steps. A model tuned to win on a clean dataset has learned something that does not transfer.

It is why our models are grounded in how heat, mass and machinery actually behave rather than fitted to last quarter’s numbers, and why the results we publish come from live plants and open competition rather than from a leaderboard.