The Smoke Monster Was Created by the Protector, and Commandeered by MiB
One of the island's earlier (pre-Jacob) protectors created the "smoke monster" to assist in the processes of guiding, judging, and protecting those who come to reside on the island. For most or all of its history it was a tool used by the protector of the island (as illustrated by Mother when she used it to destroy the MiB's village) to guard the source of the island's energy (The afterlife - The source of life, death, and rebirth). However, the Man in Black, because of his special communion with the island and the dead, was able to take advantage of Mother's prohibition of violence between him and Jacob to "cheat" death and commandeer the smoke monster for himself. Despite having made this transformation, he still remains partially subject to the Protector's will and thus cannot leave the island without his/her absolution or, failing that, termination of the monster's "appointment" via deactivation of the fountain at the heart of the island.
The Smoke Monster Protects The Island
Through some unknown means, the island was given or created itself a protector to guard it from outsiders and other possible threats. This protector became popularly known as the smoke monster. The smoke monster's origin is still shrouded in mystery. However, we do know the smoke monster's first appearance was when it appeared to the man in black (as his mother) and convinced him to leave the island.
Through some unknown event, the smoke monster was trapped for some time, until it was accidentally released by Jacob when he threw his dead brother down the secret cavern. This freed the smoke monster from the cavern and it began to protect the island again, keeping outsiders from finding or entering the island. However, when individuals did find the island, it protected the island by taking the form of dead people to confuse those trespassing.
At some point, the smoke monster entered into a "game" with Jacob to determine the fate of the island.
The Monster represents the past
The monster is a metaphor for people's troubled pasts. The more morally questionable actions a person has committed in the past the larger the monster will be for them. When you run from these past events, they will eventually catch up to you. But if you face your problems or the Monster and come to terms with them, they seem less daunting and scary. It is not necessarily the monsters purpose to make you face your past, it simply derives it's power through these terms.
Mr Eko stares the monster down and it slowly shrinks and then runs off. When he is drinking from the stream, the monster sneaks up on him, but he quickly turns and faces it and it slips away. The monster had never been seen so large as when it attacked Keamy and his men, who had committed many immoral acts as mercenaries.
It manifests as whole, dead bodies
The Monster can manifest itself as people whose dead bodies remained whole. That's why the Others burn their dead and cast them out to sea. It successfully took the form of Alex, even though Richard buried her. It usually takes the form of those who died on the Island, which explains why Emily Linus's appearance surprised Richard, who's earlier only seen it appear as his own wife.
It is electromagnetism
The Monster is the manifestation of the high electromagnetic fields the island produces. High electromagnetic fields are known to cause hallucinations and "fear cages" that can enclose on peoples psyches. DHARMA has been studying electromagnetism and possibly made the connection that the Monster is made up high electromagnetic fields, so they built the fence around the barracks to protect themselves.
The spectral analysis of the sound made by the sonic barrier, one of the frequency bands corresponds to 972 Hz, which is the average resonant frequency of the human skull. The barrier could be a means of not allowing the Monster to "get into the mind" or "skull" of its target.
The leader summons the monster
Only the Island's leader can summon the Monster.
It drags people to EM pockets
The Smoke Monster dragged people into Cerberus Vents (which all lead to a pocket of electromagnetic energy, like all the wells on the Island do) because it was trying to use them to "unplug" the energy pockets there. Similar to what he wanted to do at the Source, which was the strongest of all the EM pockets. Since Smokey had no way to access the Source back then it had to try to make do with the "weaker" pockets to accomplish its goal of leaving the Island. Didn't work, though.
The reference "a mouse does not rely on just one hole" that is on the blast door map (assuming that 'Cerberus' is the smoke monster) is referring to the Cerberus Vents, or CV I-CV IV. These include the hole that the monster almost dragged Locke into.
There Are Two Monsters/It appears differently to different people
As referenced when Locke and Eko first encounter the monster, the monster seems to appear as light to those two. They describe it as "beautiful" and "a great light" yet obviously it appears as dark smoke to everyone else. So perhaps this is Jacob as an unMonster scouting candidates on 815 or Locke and Eko are somehow inherently different. Q: If it is the latter case, what is it about Locke and Eko that makes the monster appear differently to them?
COMMENTARY S01 DISC1 E04: Terry O'Q. (when talking about the first smoke monster encounter) "-I was told to imagine it was the most beautiful thing I had ever seen."
A: Locke and Eko both came to the Island without fear, either losing it when they arrived (Locke) or conquering it before they arrived through horrific life experiences (Eko). This lack of instinctual fear was potentially problematic for the plans of the smoke monster. Smoke Monster knew that both men saw the Island as a place of renewal, beauty, and miracles, so it appears as something wonderful once to each man to gain confidence and assurances, to show the Island is a place of magic. The Smoke Monster was running the con from the start, it knows how to manifest itself to others so-as to get what it wants.
There has always been an evil on the island - The smoke monster is the evil.
It seems that the island has always housed both the light and the evil... The evil judges the people who come to the island - per whoever is in charge (most of which occurred during Jacob's time as protector). If the person has not yet been influenced by Jacob (ie. touched by him, or chosen as a candidate) then the monster can kill them. In Ab Aeterno, the smoke monster spared Ricardo because when it looked into him (absorbing or seeing his memories) it saw what was trully in Ricardo's heart - in this case, a man very dedicated to his religion, to the idea of a god, a devil, heaven and hell. The smoke monster saw a way to use Ricardo - to kill Jacob - thus freeing himself. MiB spoke of a devil (saying it was Jacob) however this was all just words he was using to manipulate and persuade Ricardo into taking action against Jacob -- which if successful would allow him to leave the island and spread darkness, corruption and malice into the hearts of humanity - for if he was to leave, it is suggested that evil would spread.
There is also a sense of intimacy between the smoke monster (MiB) and Jacob - they say good morning to each other - and seem to both be at ease with each other's company - we learn they are siblings in Across the Sea, where Jacob makes the mistake of hurting, possibly killing his brother - which results in him entering the heart of the island where he emerges as the smoke monster. It seems that this was Jacob's ultimate mistake, he disobeyed the rules set out by their mother, and harmed his brother. As protector this allowed the smoke monster (the evil) to claim his brother's soul, body and memories - he became a physical being - one who could contend with Jacob now - who could physically sway/manipulate people - not quite to the extent that Jacob has (touching them and "gifting" them) but he could now speak with his own words and interact with people (handing Ricardo the knife), instead of just showing them visions based on the memories of their loved ones or grabbing them and tossing them as a smoke monster... The evil now had a form and means to alter its own confinements to the island. Jacob probably realized it was his mistake that allowed this, and now it was up to him to "play this game" with the evil, to keep it contained - after all - Jacob and his brother played a lot of games - this was probably the way Jacob deemed necessary to contend with the evil (smoke monster) which now possessed his brother.
It would also be the reason why the smoke monster only had the one form until Locke came along - perhaps when Ben killed Locke it was a similar concept - when Locke's body returned to the island, the smoke monster was able to possess his soul, body and memories - thus it gave up the previous form of being MiB to take on Locke's form - which the importance behind that choice, was probably due to the fact that John Locke was the most suitable candidate to take over Jacob's position as protector - so when Locke died, Jacob had lost his most precious candidate and the smoke monster knew that this form would prove more effective against Jacob. Sure enough, Locke (smoke monster) was able to persuade Ben Lynus into finally killing Jacob - now all that remained was ensuring that no other protector would come about to keep him contained - and so the events of season 6 was all the smoke monster (the evil) ensuring it would be able to finally leave...
Something else to note: everything that smoke monster (Locke) says in season 6 - about how there is no evil, how the island isn't special and how Jacob was a nobody (or that he lied, etc...) was all falsehoods to persuade and manipulate. The smoke monster is all about lies, deceit and evil - it is the closest entity to the devil - therefore it will say and do anything to taint the hearts and souls of those it comes in contact with - the only thing that can counter it, is a strong will and belief in Jacob, the island, and what he was doing.