I adore Silvia -- it is one of my favorite names. I don't think it's too old ladyish at all. I'd love to see someone use it!
Silvia is gorgeous! I personally prefer the more traditional Sylvia spelling--it seems more silvery and balanced and sweet to me--but if Silvia is what you love, by all means, use it! It's more classic than old-lady, imo, and it would be so refreshing to meet a little Silvia these days! I've always been partial to Sylvia Claire/Silvia Claire as a combo--I think that'd be fab. Something like Silvia Felicity or Silvia Penelope would be lovely, too, imo.
Silvia is NOT an old lady name...it's gorgeous and classy and sweet! It'll fit right in today, but stand out just enough to make it feel special. It's a wonderful choice, made even better with it's family connection for you.