II Nephi continues the saga of Nephi, the main character of the first two books of the Book of Mormon. He is constantly at conflict with his brother Laman. The sons of Nephi are called Nephites, while the sons of Laman are called the Lamanites. In this text, the Lamanites were cursed for their unbelief. Let's check it out.
"And he (God) had caused the cursing to come upon them (the Lamanites), yea, even a sore cursing, because of their iniquity. For behold, they had hardened their hearts against him, that they had become like unto a flint; wherefore, as they were white, and exceedingly fair and delightsome, that they might not be enticing unto my people the Lord God did cause a skin of blackness to come upon them."
-- II Nephi 5:21
The Mormons really believe that Black people derived from this manner. They really believe that blackness of skin is a curse from God. If this was the case, then everyone would be Black because everyone has been in unbelief during some part of their lives. Look at the association made in this verse that Black skin is not "enticing." That sort of explains why there are very few people of color that choose to become a Mormon.