Should i quit going to visit him?

Why would a 44 year old married man want to "hang out" with a 23 year old single woman? How would his wife feel about that? You would think that in 2 years, he could have mentioned that he's married. You said that you have feelings for him, so I think that staying in contact with this man has the potential to lead to a whole bunch of problems and I'm sure that in the back of your mind, you know that too. For your own well-being, stop visiting him. If he does call you, be honest with him and end it at that.