If someone adds me as their contact (but not vice versa), can they see my private questions/answers?

Today in my inbox (in my other account), I got a message saying that a user added me as their contact, but when I click on their profile it says their Q's and A's are private. This user is a troll and I think he added me as a contact, then somehow rescinded the offer. But can he see my private Q's and A's and contacts list after he adds me even without me adding him back? If so I think he purposely did this to spy on my questions and contacts. Is there anything I can do to stop it?