[I wrote part of my master's thesis on Mineral King, so I have some knowledge of the subject. Walt was never around long enough to give up on Mineral King, as he died in 1966 and the project wasn't abandoned until the late 1970s.
As John Harper's extraordinary book makes clear, a real problem to the proposal was the lack of an all-weather highway to get up there. Disney did not want to pay for it and was unable to get the state or Federal government to pay for its construction. (As governor, Reagan actually stopped California from funding it.) There were also problems with safety--a snow surveyor working for Disney was killed by an avalanche in an area which experienced them regularly (and right where Disney was planning to build its guest facilities).
Didier, although Harper worked for the Sierra Club, the book is very balanced and a really detailed chronicling of the story--surely the best history of Mineral King's possible development as a ski resort that exists.