Why did insects win?
The question, to me, is not peregrine. Arthropoda include a lot of moderately to robustly successful terrestrial groups: Arachnida and the classes below subphylum Myriapoda (millipedes, centipedes and siblings) among all others.
Yet, imagine a world without millipedes: would you really notice? Imagine a world without arachnids: there would be no critters weaving webs in your ceilings, and perhaps some odd bug would take their place, but you would probably notice the difference only slightly with a careful eye. Imagine now a world without insects. It would be a world without without flies, without mosquitoes in the summer and without ants, a world without butterflies and bees: therefore a world almost devoid of flowers (and of most arachnids as well: what does a spider eat, after all?). It would be an immensely different, much emptier world.
In general, insects diversified enormously, comparable perhaps only to how crustaceans diversified (And there are theories for which insects actually evolved from crustaceans). But why only them? The arachnid behavioural and morphological diversity is relatively low compared to the insect one. Other classes, like the Diplopoda (millipedes), are probably less diverse than all Dipterans -a single order of insects. There are insects that live their whole life on the surface of the ocean. No arachnid does that, nor any myriapod I am aware of. Insects as a whole eat basically anything, but there is only one known specie of vegetarian spider and almost no other group of Arachnida include plant eating members. Why?
Of course, the first immediate answer is that there are several unique features of insects that put them at advantage. The first two coming to mind are flight and metamorphosis. But that only brings more fundamental questions: why, for example, did only insects develop flight? Why spiders don’t fly? Why don’t we have fearsome, buzzing spiders hunting their preys in the air? Why don’t we have gliding crabs, or hovering millipedes? In Vertebrata, flight evolved independently at least three times (in flying reptiles like pterosaurs, in birds, in bats). Why this happened only once in Arthropoda?
One answer could be that it is just an historical accident. Insects happened by chance to develop wings (and the evolution of insect wings is a tricky subject: the oldest insect known to us shows features common to already flying insects), while other groups didn’t, and that’s it. However I think it’s not only wings. After all, the lifestyle of termites doesn’t require flight (except for mating), but we don’t have termite equivalents in other groups of Arthropoda (there is some weakly social group of mites, but it really does not compare with the eusociality evolution in insects). And also, it doesn’t mean that other groups couldn’t have invented wings later. One could say that since insects did, the niche was closed, but this doesn’t seem a compelling argument to me: there is almost nothing a bat can do that a bird can’t, yet bats evolved despite competition from birds.
The question of flight is tricky because other arthropods do a lot of things that seem to conduct quite naturally to flight. Spiders, for example, jump a lot. This incredibly beautiful spider displays a wide “flap” as a peacock, during courtship. There is an ongoing meme that it also uses the flap to glide in the air. It seems it is false in this specific case, but there is really no apparent, compelling reason for such an adaptation not to happen. More strikingly, many spiders “balloon”, gliding for hundreds of kilometers attached to silk kites.
My first guess is that insects won because they were genetically more morphologically plastic -they have a body plan genetic development plan that is more malleable to evolution. They diversified because because they were intrinsically evolvable: to develop new forms of insects, so to say, is “easy”, while developing really new forms of arachnids is “hard”, and developing new forms of diplopods is “harder” still. Of course this is little more than a tautology if not rooted with a molecular biology explanation: it is really the insect genome more prone to easy morphological plasticity? If yes, how? I hope to have a look at the issue and see if there is anything on the subject. Work in progress on the main site.