You are totally right. I looked into it a bit, the mistake I made was assuming that we’re more efficient at extracting oxygen from the air than we actually are. A held breath contains quite a lot of usable oxygen, which we can extract over minutes of time. Breathing in nitrogen would rapidly replace that still fairly-oxygenated air with pure nitrogen, and evidently our blood doesn’t carry more than a few seconds worth of oxygen.

Thanks for the gentle correction!