Modern infrastructure will collapse in very extreme climates.

I mean, the existing infrastructure will be stressed, sure. But it's not like it hits 60C and all the roads and bridges instantly crumble. The question is if you can afford to fix your infrastructure when it does degrade, or if it becomes a losing battle. We forget but we're so far above subsistence at this point; for most of human history almost everyone worked at producing food, and now it's maybe 2% in the West. I think it's very likely that we'll keep something resembling civilisation going, if maybe a poorer version.

The bigger concern is if we decide to nuke ourselves fighting over who gets a slightly larger slice of the shrinking pie.

Alternate forms of energy won’t be possible for future tribal man without the steppingstone that is oil.

Nah. Steam engines can run on biomass too, and hydro/wind is just magnets and mechanical parts. Once you have enough industry to make your first solar panels, it's off to the races. I've spent an unreasonable amount of time researching exactly this. Fossil fuel makes it all a lot cheaper and easier to scale but you can do without from an engineering perspective.