So I got this theory. It's quite long so I hope you can stay with me, it's worth it. I'm gonna start by goign thru the complete story we were presented from the MiB's point of view.
The MiB wanted to find a loophole in order to kill Jacob for a very long time. The scene in which we see the two of them talk implies that MiB has been trying to find a loophole for some time already, there seems to be a deep grudge between them and they both seem to be sick of it. Interesting about this scene is: Jacob brings people to the island again and MiB instantly starts talking about the loophole. Subtext could be "You keep doing that and sooner or later im going to find someone among those you bring here whos gonna kill you for me" (stay tuned for more arguments for that). Later we find out that MiB cannot leave the island.
Then 815 happens on Jacobs behalf, he planned this. He touched Jack and so on and made the thing happen. The smoke monster comes to the survivors to check them out. Who did Jacob bring to the island this time and more importantly - why? Anyone suitable for my loophole? At this point he must know about the candidates on the cliffside cave wall already and here I'm gonna take a big assumption (stay with me for a while to see where I'm going with this): he does know that Frank Lapidus is the "biggest" candidate Jacob has. Now Frank Lapidus is supposed to be the pilot of the plane (we do learn that in season 4) and MiB somehow knows this - so he kills him (him being Seth Norris, the actual pilot of flight 815. He is being killed in the very first episode). Since Jack, Kate and Charlie are present he doesn't have the time to "judge" Lapidus as he did with Eko later (stay tuned for more about Eko too) - or he doesn't need to because Jacob gave him a clue that this Frank Lapidus is going to be the plane's pilot. Jacob and MiB do talk from time to time, we know that.
Maybe Jacob even crossed out Lapidus on the cliffside wall to give MiB some confidence here. So MiB sees himself in a good position - maybe he can even find a loophole now that with these people Jacob's plan already went wrong. So he keeps checking them out (the most encounters with the monster do appear in the beginning of the series) and starts planning. The next thing we see of the monster is with John Locke. John claims he looked into the eye of the island and that he saw a bright light. Now here are two theories:
1. this is MiB checking out Locke and he somehow can also be bright light instead of smoke 2. this is Jacob in his "lightform", there to give John Locke some faith. John dces need some of that, he had a miserable life and he needs to be pushed to having faith in the island. Jacob healed him several times and with this apparition he gives John the rest, from then on John is a believer.
I'm gonna stick with version 2 - again, stay tuned. A few days later Boone is high on some drugs Locke gave him. MiB uses this moment to go ahead and check Boone who sees it and dreams some weird stuff about the monster killing Shannon. The next apparition of the smoke monster is when some survivors join up with Rousseau in order to get some dynamite to blow up the hatch. The MiB is very agressive here, chasing the group (being Jack, Locke, Hurley, Kate, Arzt and Rousseau). But it doesn't harm them. We do know it wouldnt have a problems kill them all, we saw it kill groups of people in the statue (this being season 6). It looks like the MiB wants to scare them off, either it doesn't want them to know about the Black Rock, it doesnt want them to get into the hatch or (and this is what I believe): they got close to something sacred to the MiB which is located in the Dark Territory. MiB only scared em off until they ran a good deal through the jungle.
The next big thing is Mr. Eko. MiB appears several times to Eko, until that famous scene comes where it comes up to Eko and stands still for a few seconds. The MiB still on his course of checking the survivors out and planning his loophole scares them a bit and is impressed by Eko's courage. He sticks with Eko for a while and tests him, he takes the form of Yemi to even ask Eko some questions. He thinks maybe Eko is the one he can perform his loophole with - meaning that MiB can get Eko to kill Jacob. He appears to him as Yemi to be able to get some answers from Eko. He makes Eko confess. Then he kills Eko. Eko must revealed to him in his confession that he would ultimately be on Jacob's side and can never be conviced to kill Jacob.
Then MiB came to Nikki and Paulo as Medusa Spider. Again, judgement. If all those spiders had poisoned them they would have been instantly dead, not only paralysed for a few hours. I think only one of the spiders was real, MiB just transformed into some to see what's happening there first hand. Nikki and Paulo did have some nasty stuff going on and were doing nasty things just for those diamonds. MiB saw that, and thought maybe he could get one of them to kill Jacob. But they got barried alive.
The next thing is when Juliet and Kate are handcuffed together in the jungle. We do have three apparitions of the monster here. Two as black smoke, one as bright light. At first its black smoke chasing them, then when they are hiding we only see their faces with bright light, and then again they are chased until they are safe behind the sonar fence. Ultimately, this whole thing was staged by Ben and Juliet in order to gain Kates trust. However, just handcuffing yourself together with someone doesnt gain his trust. In order for trust to come fast you have got to face a life threatening situation. This whole thing was staged, including the apparition of the monster. In this case it was Jacob. I already stated Jacob having a bright light form. Again the MiB could very easily just have killed the two of them if he wanted - in this case it wasn't even the MiB. Jacob must be able to appear as black smoke as well. This leads to Jacob workign closely together with Ben. All encounters and situations in which we were led to believe that this was not the case must have been staged, too. Later about that.
Jacob having a smoke form too and them working together corresponds with what happens next in season 4: mercenaries threaten Ben's life, he somehow calls the black smoke and it kills them all. I believe the thing in Ben's secret room is his way of communicating with Jacob. This corresponds with them working together and enables them to stage all other encounters where they are seen together.
Then already we come to MiB staging as John Locke. Now this must be seen with a lot of background story. Ben (and Jacob) made it clear to the alive Locke that Ben didn't realy have contact with Jacob. The visit at the cabin for example was staged, the apparition of Walt was Jacob and so on. Not only that: to Locke, Ben looked weak and vulnerable. All the events with the two of them in season 3 suggested that. Then Locke dies and his body comes to the island, enabling MiB to take his form (stay tuned here) and giving him access to Lockes memories. This makes him see that Ben is vulnerable and could be turned against Jacob. Ben, the one man he believed was 100% loyal to Jacob. So he convinces Ben to kill Jacob, believing to finally (and conincidentaly really) having found his loophole. This whole thing was staged by Ben and Jacob, from the events in season 3 on (including the tumor Ben has, it gived MiB very strong arguments to later convince Ben!). This corresponds with Ben killing Locke and making sure that his body is on Ajira 316 - the whole thing about it being a substitute for Jack's dead father is just lies to convince Jack to get the dead Locke on the plane.
Now lets tie up some loose ends:
Why would Ben and Jacob do that? To ultimately get rid of the MiB. Jacob and MiB have been fighting for so long, Jacob obviously is sick of it when they're talking in the S5 finale. This is why he plans a big plan in order to get rid of him. Yes, this includes his death. But: is he really dead? He was able to speak to Hurley. Also he has got Frank Lapidus in place as a secret candidate to take over his role, the other (only so-called, as I believe) "candidates" may just have a big role in the now following "taking down" of the MiB.
Bringing dead bodies to the island -Yemi died before that plane crashed on the island by a gunshot (we see this is The 23rd Psalm) -Jacob had Jack on 815 so that Christian would be brought to the island. Christians apparitions are done by Jacob. He simply hid it so that MiB couldnt find and learn to resemble it. All the actions Christian took on the island were somehow supporting Jacobs plan as stated above. This supports the idea of Jacob and MiB having the same powers, with the only difference that MiB cannot leave the island and therefor bring ppl here. -The Oceanic 6 were neccessary to get Locke killed off the island and then brought back to it, all this after Ben was able to implant Locke with the idea that Ben is weak enough to be convinced to kill Jacob. Thats all there is to the oceanic 6, at least so far. -Then there is Alex Rousseau. Ben is alone in the chamber, black smoke comes out. We do not see MiB (as John Locke) in this scene, only know that he is not far away. Alex was both born and died on the island, being the only one next to Ethan Rom. So maybe people must have come to and gone out of their life in the same place, meaning on or off the island in order to be usable by Jacob and MiB.
Frank Lapidus Frank was, as stated before, supposed to be the pilot of 815. But he wasn't, this was planned by Jacob to give MiB confidence, tricking him into thinking that now the time for the loophole might have come. Ultimately however he comes to the island with Ajira 316. Ilana states after Ajira 316 that he 'may' be a candidate. Ilana is not 100% filled in (as Ben is) - Frank Lapidus is the ace up Jacob's sleeve, the one thing MiB does not know about yet (MiB does know about all the other candidates).
LA X The side universe is where our characters are all ultimately going to end up. Jacob is the good guy and in order to fulfill his plan he had to put these people thru a lot. They died, they fought, they starved, they were put into cages and so on. Jacob doesn't want all this to turn out this way which is why he made the survivors travel in time and create the side universe. It's where they are going to end up after they are dead in the original timeline: in a world without the terrible events on the island. This is why Juliet's last words were "It worked" - she was dieing, at that time she was somewhere inbetween the universes. As a little addition Jacob made their lives a bit better with the losties helping each other: Claire gives Kate her credit card, Kate and Ethan help Claire deliver the baby, Locke gets a job from Hurley and Rose and so on. This is Jacobs way of making things right. Maybe he himself is in the side universe too. The island being underwater and the MiB being trapped on the island speaks for itself. I do doubt that tho, but it would be cool :)
Pragnancies With everything planned out to be right in LA X, nobody could be born on the island because his/her spirit wouldnt be able to move over to the side universe with the exception of those who would have been born anyway, in other words were conceived off the island or for reasons unrelated to it (like Ethan and Aaron). This is a rule by Jacob, he rather killed the mothers knowing they will be in the side universe anyways.
-Hurley being able to talk to the dead suggests the existance of spirits. The whispers are spirits floating around and Jacob or MiB nearby make them possible to be heard. -The numbers. I do hope it's not going to come down to "everything was planned by Jacob" and "Jacob has a thing for numbers" but they will get a meaning for themselves. By now its impossible to say what that is tho.
Of course I dont have everything covered, this already is a wall of text as it is. But its all falling into places.