The ultimate problem with having a sugar daddy is the simple idea that you're using someone. Not based on principle, but this is problematic because one person can generally only give so much and this is where issues arise. Eventually, "taking care of someone" just isn't enough. Often times the relationship becomes more demanding physically or emotionally whether on purpose or accident.
Then there's the whole having a relationship but not really having a relationship with them. It can affect your life in areas that you wouldn't expect. Current relationships, friendships, even work. I've known a SD who demanded a girl to drop what she was doing and come to him at any moment. Typically, you set the rules and boundaries in the first chat with a potential SD so this may or may not be an issue.
Another major downside is that you may or may not even like the person you're exchanging services with. I think this is the worst part for me. I rather enjoy having sex with people I like and find attractive but some people can just shut it off.