Social Security Issues: How to Stop Serving as a Representative Payee

I'm in a confusing and frustrating position. I was a payee for my ex boyfriend. We broke up in March. I no longer wanted to be his payee or seek future contact with him. I informed him and his immediate family members that had been his previous payees that I no longer wanted to be his payee. His family informed me to not worry and that they would take care of it. I tried calling the social security office and was sent to voicemail. I left an informative voicemail detailing that I no longer wanted to be a payee any longer. I never received a call back. I informed what I thought was my former boyfriend of the call and his family. They again told me that they would take it from there. I received numerous calls and messages about his direct express card going missing. He said that I was still listed as his payee and needed to send for an emergency direct express card. I found the form on the social security website and had it sent to him. To try to make this short, I received a message today stating that he never received the card, hasn't received benefits since May and for me to call the social security office again or there would be serious consequences. What should I do to take care of this? I want this to be taken care of already and for a new payee to be named. I thought my name had been taken off in March and this is still continuing. I would be more than happy to help with getting his card issued and don't want to be looked at as mistreating him by not making sure he received his benefits while I was unaware that I was still his payee. Thank you for your time.

- - - Updated - - -

Also I thought that his Aunt had become his payee. She contacted me on July 1st saying that some paperwork was messed or lost. She said she was taking care of everything and to not worry. I took this with ease thinking that I could finally put my mind as rest knowing this was behind me.

Social Security Issues: How to Stop Serving as a Representative Payee