The destruction has been of the faithful. Moroni is faithful and remains. The three who “did tarry” remain. Moroni knows this first hand, and knows that they are not visible among the people.
This statement that the three ministered to Mormon and Moroni confirms what we would suspect, which is that they spent that time together that they could before Mormon was captured and put to death.