Instead of the power of Alma’s prayers turning the hearts of the people of Ammonihah to God, they clearly remain in their current state. In addition, they declare that they have no reason to listen to Alma. Since Alma gave up the judgment seat, he has no political authority. As they declare that “we are not of thy church,” he has no religious authority over them.
This question of authority to preach will be subtly reinforced later in this event. Alma will preach with Amulek, and, thus, there will be two witnesses as required by the law of Moses. While Mormon will not make that law of witnesses completely obvious, he sets it up by making certain to include the problem of why Alma would have authority to preach by including these statements from the people of Ammonihah.