Even for a historically significant, large-scale institution like the Institute for Advanced Study in Princeton, having a world-renowned theoretical physicist like Edward Witten is of paramount importance.
Not to mention, he is the only theoretical physicist in the world to have ever won a Fields Medal.
It can be said that wherever such a towering figure resides, the academic prestige and international reputation of that place are vastly elevated.
To put it bluntly, even in Princeton's current physics department, several top scientists are there because of Edward Witten. Additionally, many donors are willing to fund the Institute for Advanced Study precisely because of his name.
If Edward Witten were to just walk away like this, the resulting issues would undoubtedly leave the director scrambling to pieces.
Roth Dugan is also a director, so he can empathize with such feelings.