You "went home for a few weeks"? If you are married, your home should be with him.
I've been married for 37 years and my wife has never visited some where other than our home for a "few weeks" since we've been married.
If you and your husband were having problems, you should have stayed with him and not deserted him "for a few weeks". He shouldn't be having conversations with anyone of the opposite sex of a sexual nature, but you are equally wrong.
Get the book, "The Proper Care and Feeding of Husbands" and follow its suggestions. Its written about 75% for women and 25% for men. After you read it, give it to your husband to read. Watch the movie, "Fireproof" with your husband. Get the companion manual, "The Love Dare",also get one for your husband and you and he should follow its instructions. (BTW, I'm on day 13,000 plus).
His co-worker "bragging" about being a "home wrecker" probably has a great deal to do with the conversation you had with her. I wasn't at his work to hear her tone of voice, but I've worked in different offices and that kind of talk wouldn't have been tolerated in any of the offices I worked in. Unless you went to his work, how did you find out she was "bragging about being a home wrecker"? Some important facts seem to have been left out of your story.
Step back, cool down and get some marriage counselling for the both of you. Get the books and movie and be pro-active rather than re-active.
Good luck to you and yours.