Each SOUL Mode corresponds to one of the SOULS of the Eight Humans, excluding the one used in Mad Mew Mew's fight.
While the SOUL is red, it is in Normal Mode and behaves normally. It can move in any direction at a constant speed, and can slow down by holding [X] or Shift. The SOUL's heart sprouts legs when it runs away from a battle.
YOU'RE BLUE NOW. THAT'S MY ATTACK!
When blue, the SOUL is in Jump Mode and affected by gravity, much like a side-scrolling/platformer perspective. This SOUL color is used by both Papyrus and Sans in battle and is the first time the protagonist's SOUL changes color. Sans appears to have much more control over this, being able to force gravity in all four directions rather than just straight down.
Pressing a direction opposite to the force of gravity causes the SOUL to "jump" in that direction. The height of the jump is dependent on how long the direction up is held until the maximum height is reached, which can depend on the attack. In fact, near the end of Papyrus's final attack, the max height is so great that holding the jump can cause the SOUL to rise beyond the normal height of the Bullet Board, which extends upward to make room.
As long as you're GREEN you CAN'T ESCAPE! Unless you learn to face danger head-on…You won't last a SECOND against ME!
While green, the SOUL is in Shield Mode and cannot move, but wields a shield-like spear from Undyne that protects the protagonist in whichever direction is pressed. Instead of dodging a pattern, the protagonist must block arrows coming in from all four directions using this spear. This SOUL color is used by Undyne, who switches the SOUL between green and red using her spear. It is impossible to flee in this mode. Undyne gave the protagonist the spear in an attempt to force them to face danger head-on.
... I think purple is a better look on you! Ahuhuhu~
When purple, the SOUL is in Trap Mode and can only move left and right on a line and switch between lines using up and down. Spiders move on these lines that the protagonist has to dodge, acting as a web. When in this mode, the SOUL cannot slow down by holding [X] or Shift. This SOUL color is used by Muffet after she fills the Bullet Board with purple liquid on her first turn.
Poison/KARMA is a bright magenta instead of a dark one, and it is unclear if it is related to the purple SOUL mode.
You see that yellow button...? [...] Now press [Z]!!!
Alphys introducing the protagonist to yellow mode
While Yellow, the SOUL is in Shooter Mode and is flipped upside down like a monster's SOUL, so its heart's point is directed toward the foe. Pressing [Z] shoots a projectile that can destroy certain oncoming projectiles or set off bombs, as well as damage the enemy in certain situations. This SOUL color activates after the protagonist presses a button on the phone Alphys gives them and is only used against Mettaton and Mettaton EX as well as the Lost Soul variation of Alphys.
On the ceiling above the door, there's an image of two analog sticks being pushed apart.
Flavor text upon inspecting the door to Mad Mew Mew's room after being unlocked
While Blue/Red, the SOUL is split into red and blue halves. When in this state, the left analog stick moves the cyan half and the right analog stick moves the red half. Depending on the attack, the halves can either move linearly left or right or in four directions (up, down, left, right). This SOUL color is used exclusively by Mad Mew Mew in the Nintendo Switch version of Undertale.
Unlike all of the other SOUL modes, this color is used to represent the signature neon blue and neon red colors of the Nintendo Switch's Joy-Con controllers.