I was the first in my workplace to take paternity leave when it became law in my state. I didn't take it in one chunk, but used every single day I was entitled to. I got many similar comments as you from older guys, and I believe they came from a place of jealousy at worst and self-rationalization at best, since those people weren't afforded the same rights when they had kids.
Pay then no attention, the first few months as a dad to a new kid are some of the most important and precious moments you'll ever have, and if you miss them you will never ever get a do over. Take every second you can without an ounce of shame.
You may also find yourself setting an example, as I noticed none of the new Dads in my workplace after me had any reservations about taking their full leave, and I work in construction with some needlessly macho guys.