At the beginning of the Book of Mormon, in 1 Nephi 1:2, Nephi explains that the language he was using on his plates was “the language of my father, which consists of the learning of the Jews and the language of the Egyptians.” In Mosiah 1, King Benjamin required his sons, to learn Egyptian and Hebrew, “all the language of his fathers” (verse 2), so they could read both the prophecies on the plates from Jerusalem (verse 2), and the writings on the plates of Nephi.
Here, Moroni explained that the language used was “reformed Egyptian, being handed down and altered by us, according to our manner of speech.” He also explained that they would have needed larger plates if they had been writing in Hebrew, which also had been altered somewhat.
We do not know for sure, but some researchers believe that Mormon and Moroni probably wrote in the Hebrew language, except they wrote the sounds using Egyptian characters or modified Reformed Egyptian characters which may have been some kind of shorthand at this time.
People might wonder if one alphabetic-based script has ever been used to write the sounds of another language. The answer is yes. In modern times, a person can find on the web the Hebrew or Greek words of the Bible transliterated into English letters, not as translations but so that the reader can pronounce out loud the sounds of those ancient words.
Did this kind of thing ever happen in the ancient world? Yes, it did. The text of a Hebrew psalm was found years ago in Egypt, written out phonetically using Egyptian characters. Another example comes from Gubbio in central Italy. The people who lived in what is now called Gubbio were Umbrians, and they spoke a dialect of the Umbrian language. They apparently had no written language, as far as we know, but wanted to record their laws, especially their religious laws on conducting public festivals in worshiping their gods. This was probably in the third Century BC. So they went to the Etruscans, who were master metallurgists, knowing how to make metal artifacts and plates of all kinds. The Umbrians apparently stood there and spoke in their language what they wanted this legal posting to say; however, the Etruscan platemaker knew only how to write in Etruscan script. So these Umbrian public city ordinances and texts were recorded, on plates of brass, in Umbrian but using Etruscan letters.
When these Umbrian plates were discovered in the seventeenth century, no one was able to read them. The people who understood Etruscan studied them and gave up, pronouncing them as nonsense. Eventually, however, someone who knew Umbrian was there while another was reading the lines aloud in an attempt to make sense of the sounds. He recognized it as Umbrian, and that solved the puzzle. Similarly, Ugaritic texts in a Northwest Semitic language are written using cuneiform symbols. There were many languages, but not that many writing systems, so this kind of thing happened more often than people might think.
Book of Mormon Central, “Why Did Mormon and Moroni Write in Reformed Egyptian? (Mormon 9:32), KnoWhy 513 (May 2, 2019).
John Gee, “Two Notes on Egyptian Script,” Journal of Book of Mormon Studies 5, no. 1 (1996): 162–176.
Stephen D. Ricks, “Semitic Texts Written in Egyptian Characters,” in Pressing Forward with the Book of Mormon: The FARMS Updates of the 1990s, edited by John W. Welch and Melvin J. Thorne (Provo, UT: FARMS, 1999), 237–243.
On the history and contents of the seven Iguvine tablets, see generally https://en.wikipedia.org/wiki/Iguvine_Tablets