We can see some kind of "wave" twice at the end of season 1 just before the raft is destroyed by the Others. The first time we can see it is when the radar is beeping, the second time a few seconds before the Others' boat lights come up (and the wave is NOT where the boat comes up). This wave could have been done by the top of the sub (which could have been there in case of resistance from Michael, Sawyer and Jin). In "Homecoming", there is a shot out to sea where sinister music is played and the surface of the water is broken by the submarine.
The sub, like the helicopter, experiences turbulence
The "roughness" of approaching the Island in the sub is similar to the turbulence experienced by Frank, Sayid, and Desmond when they left the Island via helicopter. These are spacetime ripples. If on a surface vessel would likely be prone to Bermuda Triangle-like issues of rogue waves and white water in the open ocean. The sub would be immune to capsizing even if it encountered spacetime turbulence underwater.
The submarine is not used for travel to and from the Island
Travel to and from the Island is by whatever method is enabled by the Island. Think, in general terms of teleportation booths. The Island booth is at the Looking Glass and people who are not allowed to know how inbound travel works are sedated, transported, placed on the submarine, and moved to the dock. When they arrived, they are awakened and led out of the boat. Outbound travel works in reverse, starting with being sedated after boarding the submarine.
This is how Tom Friendly travelled to the mainland, presumably after the sub was destroyed, in Meet Kevin Johnson (if it was in fact destroyed). In said episode, he answers Michael's question regarding transport to and from the island saying that "some of us can (come and go)". After the sub is no longer available, only people who are allowed to know about the real method of transport are able to leave/return to the island.
Tom Friendly returned to the island, after meeting Michael, before the submarine was blown up. The episode was shown in season 4, but was a flashback of Michaels story after the others let him go. Tom actually left while Kate and Sawyer were locked in the cages at beginning of season 3. Further proof, pay close attention to the beginning episodes of season 3, Tom isn't present for a short time. He is then back wearing a tight black shirt. The same shirt he wore off island in Michaels flashback. He wore the shirt on his trip back to the island. As he stated, they're only allowed to leave for a short time.
The boat given to Michael obviously is used to come and go, and given that Rousseau's boat found the island by following the radio beacon, AND Desmond's sailboat AND the Black Rock, AND the US Army nuclear weapons testers, all found the Island without guidance, it therefore stands to reason that it is possible to reach the island by other means. The submarine is primarily used in order to hide the fact that the Dharma Project, and later the Others, were using this particular island, from world naval powers and other islands in the area (the island likely has a major taboo in local polynesian legend as haunted and evil which keeps locals away).