I guess it all depends. It's not something we would probably ever consider, but then, we've had a lot of fun doing things we wouldn't ever consider
I'll say that my biggest concern with our swinging activities has always been the disruption that could result from a side relationship gone bad, and try as we might, basically every partner we've both had outside our relationship has eventually wanted more from us than we've been willing to give. Doing what you're proposing gives someone a lot of power over you.
But no one can really speak to your specific situation without knowing more about it, because it just all depends.
