The Refusal to Change
Protagonists grow and change, learn and develop: the story is the soil in which they do this. They are the part of the tree which is above the ground, if you like, visibly getting bigger and more interesting as time goes on.
Antagonists refuse to change. They are sometimes given opportunities to learn and develop, but it is their refusal to do so which makes them interesting. They are the part of the tree which is below the ground, invisibly sending roots deeper and deeper and denying themselves any kind of movement.
As readers familiar with the secret language of fiction (whether we know it or not) we expect to see the main character change, while his or her adversary does not. We don’t know how it will all happen, but we know that it will - if the story is to succeed, that is.
We also expect there to be, in most stories, a character who is older and wiser and who effectively doesn’t need to change. The next thing that happens to them is that they die or disappear for large sections of the tale - which often happens in the course of the story (think of Gandalf, Obi-Wan, Yoda, Dumbledore, Merlin, Atticus in To Kill a Mockingbird, and many others) only to re-appear later in some form.
Similarly, we look for a comic character or sidekick in many stories. This figure ‘saves the day’ at some point, or instigates a positive action, and usually takes over from the protagonist towards the end of the story. Think of Samwise Gamgee or R2-D2 or Ron Weasley or Dill in To Kill a Mockingbird. There’s also the Porter in Macbeth, or the gravedigger in Hamlet, or the Fool in King Lear.
It’s that antagonist’s refusal to change, though, to which many stories hold. If the bad guy were to change, were to suddenly see reason or compromise or recognise a folly in themselves, the plot would generally crumble and the protagonist seem less sympathetic. If Sauron had become apathetic and declined to search for his Ring, if Darth Vader had recognised the good in himself earlier on, our attention would suddenly switch from whoever we thought was the protagonist to this new, complex figure who was wrestling with inner demons - suddenly he or she would be the one with the character ‘gap’ or vacuum which had the most pull on us. We see a glimmer of this in Tragedies where the protagonist is in the process, usually, of becoming the antagonist: they have doubts, they question themselves morally, and then fail and go on putting down deeper roots, refusing to grow upward, and so perish at the end.