Hagoromo and Hamura

How did the Kaguya clan come in, if its implied that Uchiha, Uzumaki and Senju came from Hagoromo's sons (makes sense as they have Hagoromos many abilities) and Hyuga and Otsutsuki from Hamura (also makes sense), but the Kaguyaa clan doesn't have much connection with the brothers, but only have the looks and the bones thing that only Kaguya had, so does this mean Kaguya had another kid that we don't know about or am I missing something?

I read the article where they say they are Kaguya's descendants, so that's not my question, so please do not misinterpret on that.