In successful stories, there is a balance between Idea and Image, Reason and Imagination, Theme and Archetypes (or the symbolic representations of themes), Meaning and Form, Purpose and Shape, the Masculine and the Feminine.
In good stories, this balance is threatened by the antagonist. The threat represented by this character creates a core vacuum which causes motion or action. This is called the Plot. It’s not often discussed in this way, but a plot in a story is a mechanical device designed to acquire and hold enough of a reader’s attention to bring about the annihilation of the core vacuum above and to restore balance. In other words, a story needs a series of events in a particular sequence which draws the reader along, never letting go until the reader comes face to face with the fundamental ‘hole’ or problem which has driven the whole thing from the beginning.
We find that in most successful stories the protagonist and the antagonist are linked in some way - Luke Skywalker is the son of Darth Vader, Frodo is connected to Sauron through the Ring, Morgana Le Fey is the sister of Arthur, Harry Potter’s scar connects him to Voldemort, and so on. This isn’t just coincidence or something to do with Jung: this is a very precise mechanism which must arise if a story is being told properly and fully. Protagonists must be pulled towards antagonists, just as a magnet pulls iron. The ‘hole’ created by the antagonist needs to be filled by the protagonist and what he or she brings.
A good story is at its most basic about two poles, the coming together of which is the story.
The core vacuum of a story is embodied in the antagonist, and is a representation of an absence of one of the things listed above:
In stories, there must be too much or too little Idea, too much or too little Image, too much or too little Reason, too much or too little Imagination, and so on.
The degree of these imbalances is represented by Archetypes. There are seven of these archetypal figures in most successful stories:
Complete imbalance i.e. one part totally missing, is represented by the antagonist.
An almost complete imbalance is represented by the shadow protagonist, a figure who in many ways is similar to the protagonist but who has wandered into the grip or influence of the antagonist. Think of Gollum, or Mordred; picture Orlick or Bentley Drummle in Great Expectations. Darth Vader himself is a shadow protagonist when the Star Wars trilogy is viewed as a whole, like yet unlike Luke his son.
A dichotomy occurs when a balance is unstable. This is most commonly represented in fiction by a female character, who flips between light and darkness or who is duplicitous for at least part of the story. Lady Macbeth, Galadriel, Guinevere, Miss Havisham - there are almost as many examples as there are stories. This ‘flipping’ has nothing to do with gender in real life, it is simply a convention in fiction.
A story’s protagonist represents the point of dangerous imbalance which is intended to grip the attention of the reader. Thus Frodo is always on the edge of succumbing to the Ring, Luke continually feels the power of the Dark Side, Pip is perpetually infatuated by Estella, and so forth. It is precisely that risk, that tipping of what might have been a balance, which engages us as readers or viewers for the duration of the tale.
But that is not all. Most good stories also have a character who has been imbalanced and, in the course of the tale, emerges from that into a sane and reasoned mix: this is the Warrior - Han Solo, Aragorn, Lancelot, Sirius Black, Darcy, are a few examples in a cast of thousands.
Two more archetypes appear in most successful fiction: the comic figure, who is so at ease with himself or herself that little in the events of the story throw that balance off; and the Wise Figure, the almost omniscient character who acts as a guide to the protagonist. Comic figures abound, from Sam Gamgee to Herbert Pocket, as do wise old men or women, like Gandalf, Obi-wan, or the grandmother in Moana.
Thus these characters and their role in stories is not a historical accident or even a cultural meme: it is a mechanical necessity.
If a good story is about two poles coming together, these archetypal figures are a major part of representing that on the page. Each demonstrates to some degree something to do with balance, and it is the dance of these figures which gives us the story and the eventual outcome.
You can read more about this in the book How Stories Really Work.