Do I have to stick to one particular pantheon for my worship? What does that say about me if I don't?

I seem to find myself drawn to Goddesses and Gods from a multitude of pantheons. How do Pagans decide which pantheon they want to worship? Do most Pagans even settle on one anyway?