That's a good take. I agree that consequences shouldn't happen to spite the players. They made their decisions and had the enjoyment they sought, and as DMs, we're only here to facilitate that.
I think consequences are there to add a sense of scope. It's not a "you didn't do this, so here's your punishment." It's "let's take a look at how your active choices affected the world and figure out how to play in that space together." Maybe 'consequence' wasn't the best word for this discussion.
Let me put this to you – has there been an instance where you revisited an area or plotline that the players diverted from? If so, how did you do it?
100% yes. I believe my players have a much better time when they "find the plot." There are narrative lines to pull, but the key is to allow the players to find these lines in what they believe to be important. I can create as many characters and plot devices as I like, but many times, the players' actions will necessitate a whole new character or route.
This happened in a recent session. I had a location with named NPCs who could help the party escape from a city where they were wanted. Even after all the info on these characters and coming so close (literally one room away), the players decided they wanted to find someone else. Now Patrick Seaworth, an arm-wrestling legend, exists in the world.