Two races are specifically referred to in the Ainulindalë as Eru’s special creation: “Now the Children of Ilùvatar are Elves and Men, the Firstborn and the Followers.” This raises an obvious question: What about hobbits? Or dwarves? Or orcs? Do they have any role in Tolkien’s creation myth whatsoever? Perhaps a better way to phrase the question is "Why are other beings not included in the Ainulindalë?"
The obvious reason here seems problematic. It’s difficult to believe that Tolkien did not intend for hobbits, in particular, to be Children of Ilùvatar, in light of the role they play in his mythos. It also seems racist to introduce a spiritual distinction that only certain races have access to. I hope that exploring the various explanations for this issue may be enlightening.
One possible explanation is the myth’s tellers’ biases. In the original extant versions (labelled B, C, C*, and D by Christopher Tolkien), either Rùmil (B) or Pengoloð (C-D) speaks the myth to Ælfwine. In versions C-D, Pengoloð tells the myth that Rumil had previously written. Notably, both of the tellers are elves, and the myth’s receiver is a man. This means that the two races mentioned in the myth match with the two races mentioned as the Children of Ilùvatar. So one possible reason for the exclusion of other races is that its tellers included themselves, but not others.
That, of course, has several possible meanings. One: The elves were racist and purposely left out other races. Two: The elves (as transmitters of the myth) were simply careless and did not insert other races (except men) into their myth. Three: The Valar, who gave them the myth, gave it to them with an elvish focus, and the elves were loath to change it. Four: The elves didn’t know about other races. This option actually seems possible for the B version, but not for the others, since Rùmil had dwelt in Tirion since the Eldar first traveled to Valinor. Therefore, he may not have known of the existence of hobbits or dwarves or other creatures. Pengoloð, though, as a survivor of the sack of Tuor’s city, would certainly have known of the existence of Melkor’s creatures, and may well have known of the existence of dwarves.
However, this explanation as a whole has one glaring difficulty: The Silmarillion as we have it today is supposed to be Bilbo’s—a hobbit—translation of Rùmil’s works into Westron. Even if Rùmil did not know of the existence of hobbits, Bilbo certainly did, and had to choose not to change or annotate the myth.
Another possible explanation is satisfying for two subsets of the races of Middle-Earth. Dwarves were not actively created by Ilùvatar, but by Aulë. Likewise, orcs were created by Melkor, and not by Ilùvatar. Both of these races were created by Valar—but how? It seems impossible that the dwarves, in particular, could be mere automatons or golems; they act as though they have free will. But free will, like Being itself, is a gift from Ilùvatar. The Silmarillion states that Melkor “had gone often alone into the void places seeking the Imperishable Flame; for desire grew hot within him to bring into Being things of his own…” The Imperishable Flame seems to be crucial to creating Being, especially since Ilùvatar possessed it when he created the Music, but he placed it at the center of Arda when he first called it into existence. Is it possible that both Aulë and Melkor, as two of the most cunning Valar, were allowed to find part of this Flame? This seems especially plausible since Aulë was concerned with the earth and Melkor hid in caves. If it was within the earth itself, they may well have found it. Then the dwarves (and possibly the orcs) would be sub-creations of the Valar, imbued with life through Ilùvatar’s gift to them. They would not strictly be Children of Ilùvatar. Gimli’s voyage to Valinor, then, could be seen as the adoption of his race.
But what about hobbits? In Splintered Light, Verlyn Flieger points out that Tolkien compares them with men. That comparison, though, seems to be made because both races have free will and both have the Gift of Men—death. Dwarves, as a sub-creation of the Valar, also have these two gifts—they too are like men.
The final possibility is that the hobbits are also sub-created beings. Like dwarves, they are given the gifts of man, but are not quite men. They are lesser in both size and number, and they have a ‘racial personality’ that is much more strongly marked than that of men—dwarves are greedy diggers, hobbits are silly drinkers. (Of course, these stereotypes are proven wrong.) Men are marked by nationality rather than by being men. These similarities may indicate that both dwarves and hobbits were sub-created races. This brings up the question of racism again—are these sub-created races less than the races created by Ilùvatar himself? I have no good answer except that their fate, like the fate of men, remains unknown to any but Ilùvatar himself.
I do, however, have a random theory for your amusement.What if Tom Bombadil, like Yavanna, is a Valar who communes with nature? What if his power over the Ring and over nature—power that even Gandalf does not quite possess—indicates that he is even more powerful than Elrond suspects? If Tom Bombadil, as he is described in the books, were to create a race of beings, I believe he would create one very much like hobbits.