You should not get any whooshes with fuse 19 out, This is the fuse that forwards the 'battery' to operate the solenoid.
Consider the possibility that someone has doctered the wiring to make the 4x4 light look like it's operating normally.
With fuse 19 out the 4x4 light should be on permanently, there is no condition where the light goes out when the fuse is out.
Have a look at the functional testing section of the 'Workshop Manual'. and Observer
4x4 Functional Testing