The split in the family comes when Jehovah warns Nephi to flee. He takes people with him. Some people stay. How many people are we talking about?
When Nephi identifies names, we have families with their family head. Thus, there is Nephi, Zoram, Sam, Jacob, and Joseph, and associated families. Nephi mentions sisters. This is the first that we know that he has sisters, and there are at least two. That leaves Laman and Lemuel and the sons of Ishmael with their families. Not knowing who might not be named in addition to Nephi’s sisters, that leaves perhaps twelve to fifteen adults who will form the foundation of the Nephites (if Jacob, Joseph, and the sisters were of adult age at the time). There are perhaps eight to ten adults who found the Lamanites. That isn’t very many people. Were they alone in a New World, not knowing the local flora and fauna, their long-term survival would be suspect.
The most important hint is that after naming all of the people that we might expect to go or stay, Nephi adds that he would also take “all those who would go with me.” That unnumbered set of people is also not only unnamed, but otherwise unmentioned. What Nephi never clearly states is that they did not arrive in a vacant world. If they arrived, as assumed, on the Pacific coast of modern Guatemala, archaeology knows that there were communities already living the foothills.
The rapid expansion of a single family into two smaller pieces that rapidly begin to build a city (in the case of the Nephites), suggests that Lehi’s family was met by these New World inhabitants and had already merged into one of their villages. Thus, the definition of “all those would go” with Nephi would be some of those native populations with whom they had already merged.