V.B.
I think you should tell your husband how you feel and leave it up to him to speak to his brother. You should also discuss with your husband if he thinks it will be a good idea you be there during the conversation. I say this because I don't know how your brother-in-law is.