First, the vast majority of species depend on camouflage at least somewhat, since there are very few species that are neither predator nor prey. Also, albinism prevents your skin from properly protecting you from the sun. So even then, it is selected against. As other have pointed out though, caves don’t have either of these forces at play—the darkness makes visual camouflage irrelevant and there is no UV light. So there, as you predicted, most species are albino.