Things that happened thousands of years ago have direct consequences that the characters need to deal with, and there are people around who were actually alive back then, mixing with the mortals. Furthermore, we (and the hobbits) are told much less than everything about the ancient people and events - the critical bits, of course, and there are allusions to many other things, but one ends up feeling there are many other stories that could be told, which I think helps make the ones that are told feel more real.One thing that did it for me in Tolkien's world is the sidereal feel of other 'ages' that were indeed ages and ages ago, a mythic era of which only names and fragments survive. (Never did figure out what or where Gondolin was just from LotR.) This half-knowledge gives a sense of complexity, or possibly just complication, to the background-- that it is indeed a huge organic history, same as our earth history, and not, as so often in fantasy, a tidy/ sketchy construct against which the plot plays out. In world building, messiness = versimilitude.
And then the brain-spin fact that there are people you meet now who were alive then and part of those ancient historic events. Not as Immortals (Taoist or Highlander) but just because that's how they live their lives. This guy who's the current Minister for Urban Affairs was a counsellor in the court of King Wen of Zhou and present at the fall of Shang. Oh. OK.
I don't know how a writer can arrive at that complexity, short of having their own world worked out over decades like Tolkien, which few people do; and I must wonder whether Middle earth's sense of authenticity comes from the fact that Tolkien was more interested in the languages than the story. That certainly helps avoid neat and tidy and 'all background detail must be tied to the main plot.'