Voltage spikes from line inductance, voltage drop-outs from line resistance. Basically you have little reservoirs of charge scattered all around the board (current flow isn't instantaneous in a real circuit).

It helps to always think of current draw in a compete loop, out the "top" of the capacitor, through your IC, and back into the ground side (this isn't necessarily what's happening physically). Shorter loop means less inductance, shorter traces less resistance.

Smoothing is part of the story: but the important question is what is causing the roughness? Switch mode power supplies have inherent output ripple that can be filtered, but that’s distinct from transient variations in the load. Decoupling capacitors are used to provide a low impedance path at high frequencies i.e. fighting inductance.