Should I continue the fantasy I have with my wife having sex with other men?

My wife and I often fantasize about her having sex with other men. Actually, it's more I fantasize and she goes along. Usually, this results in some really really hot sex which makes her very satisfied. The problem I have is that for a little while after, I feel bad about it and just dont want to go there. But then, it always comes back. She assures me that it really doesnt matter to her one way or the other and I should just not think about that if it bothers me any. On the other hand, I feel there's a pretty good chance she would go along if I wanted to make it happen for real. It is one of the greatest turn ons for me, but then would have a sting after the fact I am pretty sure. I have read all the stuff about why men fantasize about this and I can relate to some of them but not all. I can also relate to the reasons a guy would NOT want to do it. I am just afraid that taking it any farther would hurt bad, but I just cant understand why I keep thinking about it. ????