Modern heat pumps have an energy efficiency of about 300 - 400% so they aren't nearly as expensive to run as most people think. If you live somewhere where it rarely gets below freezing they're a definite winner for heating. Hell, we've even got plenty of people installing them up in MN where they're far less efficient in the winter. You would probably still pay just slightly more than for gas heating but a heat pump also removes the need for a furnace entirely in warmer areas so you no longer have furnace maintenance costs to worry about. So if you do wind up needing a new furnace or any major furnace repairs then definitely go with the heat pump.

As far as water heaters go you can also get heat pump water heaters which are far more efficient than electric and have the added benefit of cooling your house when they run (because they pull the heat from the ambient indoor air). The only problems with those are that all the ones that I can find are absurdly expensive compared to a normal gas or electric water heater and the maintenance on them is far more complicated than on a normal water heater.