From holiday palace to retreat house