Even if you keep hold of your baby, certain states and countries require one to be worn by law if you’re on a boat. Even if you aren’t, then it’s still sensible to ensure that your infant has an effective form of water protection.
Should your footing become compromised and you lose control of your baby in the water, it only takes a few seconds for them to ingest enough water to drown. For that reason, a jacket should always be worn.
Realistically, the only time it’s safe to take them off is when your infant is relaxing in their baby beach tent or indoors, completely away from the water.