I hate people management?

Just say you wanted a change of pace and really prefer working with the public. Be honest. Don't say you hate being a team leader but just say you really prefer working with external customers and do best in that environment. Put a positive spin on your last job (I work with great people but my passion is working with the public) or whatever.. good luck. I have been in your shoes!