Natural selection isn't the only mechanism that changes a population's genetic makeup over time. Genetic drift, purely random chance in which individuals happen to survive and reproduce in any given generation, can shift a population's allele frequencies with no relation to fitness whatsoever, and gene flow, individuals moving between separate populations and interbreeding, can introduce or remove genetic variation regardless of whether that variation actually helps anyone survive.
Genetic drift is strongest, and most consequential, in small populations
Genetic drift is a random sampling effect, purely by chance, some individuals happen to survive and reproduce while others don't, regardless of any genetic advantage, and that random sampling shifts a population's allele frequencies over generations in a way that has nothing to do with which alleles were actually beneficial. Its effect is strongest in small populations, where a single generation's random outcome can shift allele frequencies dramatically, the founder effect, when a small group establishes a new, genetically unrepresentative population, and the bottleneck effect, when a population's size crashes and only a random genetic subset survives, are both specific drift scenarios with genuinely lasting genetic consequences.
Gene flow moves genetic variation between populations, independent of local fitness
Gene flow happens whenever individuals move from one population into another and interbreed, carrying their own alleles into a new gene pool regardless of whether those alleles are actually advantageous in the new environment. Its overall effect tends to homogenise genetic differences that would otherwise build up between separated populations, which can directly work against local adaptation, an allele well suited to one specific environment can get diluted or swamped by gene flow bringing in alleles from a population adapted to different conditions entirely.
What we're still unsure about
That genetic drift and gene flow are genuine mechanisms of evolutionary change distinct from natural selection is well established, uncontroversial population genetics. What's more genuinely difficult is that separating drift's effect from selection's effect on any specific gene in a real, wild population is a hard empirical problem, since both processes can leave broadly similar statistical signatures in genetic data, and population geneticists continue to actively debate, under what's usually called the neutral theory of molecular evolution, exactly how much of the genetic variation observed in real species is actually adaptive, driven by selection, versus effectively neutral, driven mainly by drift, a debate that remains genuinely open rather than settled.
This sits inside Mechanisms of Evolution (Genetic Drift, Gene Flow), one of seven topics in Evolution, one of six domains in Biology, one of seventeen subjects the app can quiz you on.