I personally dislike the rubber bumpers. It's not that they're particularly ugly, it just reminds me that in 1975, the sporty and classic looking MG disappeared.
Apart from a major weight savings, removing the bumpers uncovers the original Spridget. As mentioned by Bill Young above, you do have to fill in the gaping holes and figure out a reasonable alternative for protection - ie either chrome or custom bumpers, unless you're happy to kiss someone else's grill sometime. In my avatar you can see the hole I have to fill-in below the headlights. I bought some round LED lights to replace the turn signals, so once the spot is welded over, I will drill it out to mount them.
I fabricated some steel bar and flat stock bumpers that mount to the rubber bumper locations in the grill and rear. They are strong enough to support the car using a jack. I will powder coat them gloss black to match the new paint. They look good IMO, and will give me some protection from rear enders. They're also a fraction of the weight of the originals.
Chrome will always look good, but you'll have a lot more fabricating to do to fit them.