In this study, we seek to understand and to correlate the genetic patterns observed in the population of the island of Ibiza in the Western Mediterranean basin with past events. Genome-wide genotypes of 189 samples representing 13 of 17 regions in Spain have been analyzed, in addition to 105 samples from the Levant, 157 samples from North Africa, and one ancient sample from the Phoenician Cas Molí site in Ibiza. Before the Catalans conquered the island in 1235 CE, Ibiza (Eivissa) had already been influenced by several cultures, starting with the Phoenicians, then the Carthaginians, followed by the Umayyads. The impact of these various cultures on the genetic structure of the islanders is still unexplored. Our results show a clear distinction between Ibiza and the rest of Spain. To investigate whether this was due to the Phoenician colonization or to more recent events, we compared the genomes of current Ibizans to that of an ancient Phoenician sample from Ibiza and to both modern Levantine and North African genomes. We did not identify any trace of Phoenician ancestry in the current Ibizans. Interestingly, the analysis of runs of homozygosity and changes in the effective population size through time support the idea that drift has shaped the genetic structure of current Ibizans. In addition to the small carrying capacity of the island, Ibiza experienced a series of dramatic demographic changes due to several instances of famine, war, malaria and plague that could have significantly contributed to its current genetic differentiation.