rub your eyes a lot for whatever reason. Contacts are made to sit on the eye, but rubbing can pop them out. Not a great idea to wear them swimming.
You could also visit an optometrist and get their opinion on whether you'd be a suitable candidate, but I'd say the vast majority of people contacts work just fine.