Better On Average™

I went to an incredible talk this week by a renowned optimizer in AI spaces.

He’s worked on everything from power grids to e-commerce shipping logistics, and developed a lot of the Industry Standard Technology sold to governments and corporations by giants like IBM. He’s been around for a while, and genuinely shaped the field.

I knew I had to see his talk when I read the first sentence of his abstract, because I have no idea what this means:

The fusion of AI with optimization and control has the potential to deliver outcomes that are beyond the realm of these technologies when applied independently on complex engineering applications.

Beyond the realm of what technologies—optimization and control? Are those technologies?? Maybe that’s what he meant, or at least, that’s my best guess after attending his talk.

Following a colorful early slide called “The Beauty of Optimization,” he mentioned that optimization often comes with a cost. Here I’m leaning forward, scribbling away in my notebook: What tradeoffs are we gonna spend the next hour exploring?

So here it is, the big cost:

Optimization is too slow!

I cursed loudly enough someone turned around. Sorry, I’m still an amateur anthropologist.

Here’s what’s going on: It takes compute time to find an optimal solution. And in real-time applications, we need to get answers faster—sometimes in seconds, sometimes minutes, and sometimes, as he noted in Q&A, ‘real-time’ can mean in a week(?!)

After all, as he said over a reappearing slide of Planet Earth taken from space three different times, “We are living in The Real World.” I try not to be mean on this blog, but sir, I do not know why you’re telling me this.

As you can imagine, I filled pages and pages of my notebook with quotes and concerns to follow up on. But today I wanna focus on one corner of his talk that I keep thinking about.

What he’s attempting is actually a very interesting technical challenge: He’s trying to train proxies for optimal solutions. You’ve created a huge model to optimize X, but it’s way too slow. So now you train a second, much smaller model to try to guess what the huge model would say. If you do it right, the smaller model will do a decent job way faster.

The details are really interesting. These proxy models can generate blatantly unrealistic solutions, which you then try to project back within your realistic boundary conditions. Sorry computer, we can’t utilize minus 5 generators right now. Did you mean 0? I’m oversimplifying, and he’s got some genuinely sophisticated math here, but there really is plenty to worry about, even if the option to add “repair layers” to a model is intriguing and might work great in some applications.

So the central question of the talk becomes, “How good is my proxy?” And he has two ways of measuring this:

• One is by looking at the average error. (For the nerds, he’s comparing Normalized Mean Absolute Errors.) If the big model’s off by 1.0% on average, and the proxy’s off by 1.2% on average, our “optimality gap” is 0.2%. Make sense? We’re just subtracting one average from another.

• But we’re also considering “runtime reduction”: How much faster the proxy computes. If the full model takes 30 minutes to run, and the proxy takes 3, we’re celebrating a runtime reduction of 90%. Make sense? We’re just finding the percentage decrease in runtime.

So that’s our trade off: You want a low optimality gap and a high runtime reduction.

Alright, so here’s the case I want to think about.

This guy helps cities optimize “on-demand multimodal public transit” which now I need to explain what that means. It turns out, if people have to walk a quarter mile to get to the metro station, 90% of folks won’t do it. His solution is to use AI to get shuttle busses to pick you up efficiently, drop you off at the metro station, and then pick you up again on the other side. It’s on-demand. It’s multimodal. It’s public transit, baby!

His ultimate goal is to help Atlanta, a huge, sprawling highway city with god-awful public transit and incredibly congested traffic. When I lived there for a summer, I remember having to walk over a mile each way in 95 degree heat to shop at the nearest grocery store. Food deserts are real, but I guess we’re too busy sending billions in weapons to Israel to invest in our own infrastructure. Wait, how’d this sidebar on how we structure public spaces turn political all of a sudden?!

Anyway, he builds a proof-of-concept model for Manhattan—which seems like his worst case scenario, given how great their metro system is—but his model works even better. Folks will only have to wait 3.6 minutes for a shuttle on average, which is much better than Uber or Lyft, and he claims that on average, total travel time is cut in half—wait, actually he said it’s “reduced by a factor of 2,” which I just think is funny.

That’s right folks, we’re doing Better On Average™!

So he asks one of his students to produce a visualization to show off what his proposed solution will look like. If we light up a map of Manhattan with 2,000 little shuttle busses, where are they all going? (NYC has 13,587 taxis so he’s happy that his transit solution requires way fewer…busses? But wait, there are five boroughs! I’m confused, too.)

So his student creates a simulation and it looks terrible.

But this clickbaity image “Sunset Over Manhattan” looks great.

Here’s the problem: Manhattan’s long and thin. So the shuttles in the middle end up doing all the work! The ones at either end of the island mostly just…sit around, rarely being summoned to do anything. So our expert had completely missed vehicle utilization until he saw a little map where most of the blue dots didn’t move.

So sure, his modified solution improves on vehicle utilization, blah blah blah. But uh…does your student’s simulation miss anything else, or embed some other weird assumptions? How does it simulate demand? Are you ready for, I don’t know, rush hour? What else aren’t you seeing yet?

You’ve seen me complain about averages before, and just how much they can hide from us. But I’m also worried about there being domain-independent optimization experts who go from one application to another, pull out a suite of Machine Learning methods to account for whatever metrics a room of consultants told them were important, and then design another Castle in the Clouds that works really well as a simulation.

I don’t think these methods are useless! I don’t think this guy is a fraud! But I do think we’re using these methods overconfidently, without adequately understanding how they might break.

Once again, I am double dog daring everyone in AI to read just one book from the humanities, and to make it Seeing Like a State. (And if a whole book seems too long, that’s why I’m workshopping Board Game Ethics, so please, drop me a line!)

And hey, if you end up reading two things that’s an increase by a factor of 2.