The world's a mean place. It's unfair, then it's fair. It's hateful, then it's loving. It's a very peculiar place on philosophical and metaphysical and religious levels.
To be intimate with a married man, when my own father cheated on my mother, is not something I could forgive. I could not look at myself in the morning if I did that. I wouldn't be attracted to a man who would cheat on his wife.
If the record was picked up by Dot Records, I would imagine that they would have wanted both sides of the record to be something by Lou alone which would account for the dropping of 'So Blue'.