Subsequently, he expanded it to accommodate himself, his wife and four children. For many years, they were the only residents of what is now Hillarys, and the nearest store, at North Beach to the south, was a 5 km walk away. During the Second World War, the Australian Army used the area, and named it "Hillary's Beach" on their maps. Bert Hillarys died in 1957 and the shack and 14 others nearby were destroyed in a fierce storm in 1964.