This is the context that allows us to understand why Alma continues with this explanation. He has brought up the two worse sins, and now he explains why they are worse. They are worse because one is unpardonable and one is difficult to obtain forgiveness. The explanation of these two categories highlights the sins which make it difficult to receive forgiveness because of the sin. By implication, the next worse will be a sin from which one refuses to repent. Apostasy would fit that definition, and consorting with harlots would be only a part of the greater sin. Were there no apostasy, the consorting with harlots would be a forgivable sin, should the person come to a realization of the sin and desire repentance.
We should understand that Alma is contrasting the unpardonable, and the unforgivable against that which is pardonable and forgivable:
“Persons guilty of unchastity may receive forgiveness through full repentance.”(Encyclopedia of Mormonism, 1-4 vols., edited by Daniel H. Ludlow (New York: Macmillan, 1992), 266.)
“Serious as is the sin of fornication, there is forgiveness upon condition of total repentance. “ (Spencer W. Kimball, Faith Precedes the Miracle [Salt Lake City: Deseret Book Co., 1972], 172.)
The critical difference is in the availability of forgiveness. Consorting with a harlot is certainly forgivable, but only upon complete and sincere repentance, a process that would be difficult to achieve while in a state of apostasy from the God who could forgive.