Notice that the song is played in 4/4 time, but danced in 5/4 time!
Should be noted that this isn’t a true “5 over 4” polyrhythm, as the band and dancers are sharing a tempo (i.e. the quarter note is the same). They’re just phrasing things so that the band lands on the shared 1 every fifth bar of their 4/4, while the dancers land on the shared 1 every fourth bar of their 5/4.
A true 5 over 4 between band and dancers would have everybody landing on the shared 1 *every* bar. That means they’d be playing two different tempos (i.e. the quarter note would *not* be equal); four of the band’s quarter notes would equal five of the dancer’s quarter notes.
For fun, watch Mike Portnoy demonstrate the same phrasing principles as the Astaire bit, but on plutonium.