Re: Have mammals that have adapted to the sea gone back to the land?

Cetaceans (dolphins and whales) probably began their evolutionary divergence when creodonts (ancestral mammals) started to colonize niches (abandoned by plesiosaurs, ichtyosaurs and other reptiles) in marshes and coastal waters near what is now the Mediterranean Sea and Arabian Gulf. This was about 58 million years ago, during the late Paleocene. The first recognizable cetaceans are from the middle Eocene, around 50 million years ago. These animals formed the suborder Archaeoceti, most of which went extinct by the end of the Oligocene, 26 million years ago.

Ancestors of the seals, sea lions and walruses (pinnipeds) diverged from dog-like carnivores around the same time--the early Miocene/late Oligocene, ~25 million years ago. Obviously, these animals entered the sea much more recently than the cetaceans' ancestors.

Sea otters are even more recent. I couldn't find a date for their divergence from other weasels (family Mustelidae), but I did find a date by which "recognizable" otters entered South America from North America--about 2.5 million years ago. Even if they diverged from other mustelids much before then, otters are still the most recent mammals to enter the sea. (Actually, there are some other aquatic mammals you didn't mention, but I don't think any of them go into salt water. They include nutria, which are rodents, and water mongooses.)

The cetaceans are the only ones of these sea mammals that do not regularly come ashore. Pinnipeds must breed on land, and although sea otters do spend most of their time in the water, they are perfectly capable of locomoting on land.

Your question regarding competition for niches on land and in the sea is very interesting. I can't really answer it directly. However, I think that the reason mammals that are adapted to the sea would not come back to land may just be that it would require a lot of physical and physiological modifications. If you think about how awkward a seal is on land, and imagine it becoming more terrestrial, it is easy to see that it would be at a severe disadvantage in terms of finding food and escaping predation. A dolphin would be in even more of a predicament. (I should note, however, that there is a species called the Indo-Pacific humpbacked dolphin which does come into shallow waters readily and can move from pool to pool across mudbanks by using its flippers to drag its body along.)

Some of the adaptations mammals have evolved to increase their fitness in the sea include streamlining of the body for more efficient swimming (specifically, loss or modification of the hindlimbs, except in otters, and loss of external ears, except in otters and "eared" seals); insulation to stay warm in the sea without fur (this mainly refers to cetaceans, although pinnipeds have a good layer of blubber too); specialized muscles to power the tail for movement through the water; specialized myoglobin (a form of hemoglobin found in muscle tissue) that stores oxygen extremely well so the animals don't need to come to the surface of the water to breathe so often; and in certain whales, baleen plates instead of teeth, for straining water through the mouth to capture their food (krill).

While some of these adaptations to the sea would not necessarily hurt a terrestrial animal, many would (e.g., how would a baleen whale, with no hindlimbs and very little pelvis to speak of, and with a feeding apparatus designed for straining food out of the water, adapt to moving around on land again?) Obviously, evolution takes time, and given enough time, a whale could conceivably evolve into anything, but it seems like it would be very difficult. (I can't help thinking that it would be easier for sea mammals to just go ahead and lose their front legs too, if they were returning to the land, and go on evolving as some sort of mammalian serpents. This seems like a good premise for a science fiction story. If you write it, please send me a copy.)

This answer necessitates the assumption that adapting to the sea environment was much easier. Again, I don't know for a fact that that is correct. As a terrestrial mammal who enjoys swimming, I may be biased in my opinion about this. It does seem that any organism with enough fat to float reasonably well (like me) could make a go of it in the sea (and survive as a species long enough to adapt to the sea quite well) and that this would be much easier than going the other way. What do you think?