There is weight and then there is weight. It's one thing to have a very heavy vintage bike and add weight to that. A few ounces here and there can add up, and if you are competing its a big deal. Situation makes a difference too. Would you carry a Park tool set with you on a 5 mile fun ride near the house?? Probably not. On a 50 mile weekend ride? Still just an unlikely or maybe. On a 500 mile tour? In my case probably.
I don't really notice that much difference when I add my camera gear to the bike. My camera gear isn't typical. Instead of 24 ounces or so, its closer to 24 lbs. I can't honestly say that you would get a better work out, but if it serves a function or makes you more comfortable, that alone would be justification and worth the weight to go ahead and add it.
Well my rack is seat post mounted w/custom supports,so the whole rig is quick release;ie I can get rid of some weight quick if I just want to ride...
But as I frequently ride for fun or commuting,and I just like having my wallet,money,cell,and if Im going to stop somewhere I need a lock...and if Im stopping with money,Im buying something and need to be able to take it home,so kind of a snowball effect I guess.
But that being said Im a runner 1st and foremost,Im also a bit of a healthnut and cardio nut so its not that big of a deal for me to pop of the rack,ride with the bare minimum on a group ride and more than keep-up with the roadies who are just weekend warriors

As you suggested about other riders "engines",my engine is efficient,but Ill admit when Im mid pack,on a group ride full of road bikes,and Im on a hybrid...I think I kinda piss some people off:thumbsup: