My first thought was Piper, but I see you've already thought of that one. How about Margeurite, as in The Scarlet Pimpernel, possibly with Margo as a nn? It's definitely uncommon; is feminine but has a lot of the same strong sounds as Harper; and nn Margo is cool, on trend, and less frilly.
Another idea: Fia? That's a recent love of mine -- only my hubby and I are the reverse of you two; he wants a long frilly name and for Fia to be a nickname. Some others I love but DH seems to think aren't frilly enough are Willow, Gwen/Gwenyth/Gwenllian/Gwendolyn/Gwenora, Tove (I believe this is pronounced toe-veh), and Cora (Last of the Mohicans) -- maybe one of those will strike your fancy?
I LOVE Zinnia and Nerissa from your original list, Pippa from his is cute, and from the compromise list I like Saffron, Piper, Maren, Wren & Romilly.
And also, I have to say: EEEEEEEK!!! ARE THOSE LAMANCHA BABIES I SEE IN YOUR AVATAR?!
From your list I like Vesper, Maren, and Willa best. Least favorites are Journey (makes me think of names like Destiny, like someone else mentioned) and Rue (because of the Hunger Games). Flannery sounds like a surname name. I don't think of Piper as unusual, I know 3 people with Pipers that are 2 and under. Tansy, Paisley, and Poppy sound "frilly" and nowhere near unisex. I don't really have thoughts on the others.
Favorite suggestions from everyone else's lists are:
Fern (lots of nature names on your list so I feel like it fits right in!)
Briony (prefer this spelling to Bryony)
I like a lot from your original list! Maybe you talk him into Genevieve nickname Evie, so beautiful. Zinnia is great too. Maren is my favorite from the compromise list. I'm think of using it myself for a middle name.
I would avoid names that are too "wordy". It opens the door for a lot of assumptions and judgements, why saddle your child with that? Yes some names that are technically words may work, like colors, but imo some word names just make one think of nothing but the object, and that takes away from the person.
I like- Tamsin, Maren, Lilac, Willa
Flannery- I love Flannery O'Connor and I'm guessing this is in honor of her, but I just don't like the sound. However, it may be perfect for you. It would be unisex, non frou frou, and literary. If I knew someone named Flannery, it could grow on me.
My suggestions- Vita, Enid, Lyra, Vera, Beatrix- nn Bea, Arden- Shakespeare connection, Maeve- associated with fairy Queen Mab, another Shakespeare connection, and I second Bylthe.
From your mutual list, I like Maren, Flannery, Willa and Lark. Sorry if some of these are a bit surnamey, I'm not quite sure if they qualify, but my suggestions would be:
Saffron - Maaaaayyyybeeee
Vesper - Cool idea, but not a great name, IMO.
Journey - No
Tamsyn/Tamsin - I actually like this one a lot.
Sparrow - Meh.
Maren - Great name!
Piper - Cute, but won't grow well, IMO.
Poppy - Ditto
Flannery - No
Paisley - Nah.
Tempest - Asking for trouble.
Willa - Love it!
Lilac - Has potential
Plum - Nah.
Lark - Has potential
Wren - I like it.
Romilly - Not terrible, but NMS.
Rue - Too Hunger Games
Tansy - Cute, but won't grow well.
[Saffron - quite like it, makes me think of Ab Fab though
Vesper - Ok. What about Esther or Estar
Journey - Not my style, maybe MN?
Tamsyn/Tamsin - Love! But dislike the nn Tammy. I think Tansy could work as a nn for Tamsin
Sparrow - It's daring.
Maren - Ok
Piper - Meh
Poppy - Ok
Flannery - NMS
Paisley - Ok
Tempest - NMS.
Willa - like
Lilac - Like
Plum - Prefur as whimsical middle spot
Lark - Like
Wren - Like
Romilly - Like
Rue - Ok, very Hunger Games
Tansy - Love. It's on my list
I like suggestions of Kestral and Linnet
Other options (sorry for repeats)
Pania (Maori mythical woman, pa-NEE-ah)