I found some interesting new things about invalid instrument numbers while trying to fix libxmp's Fasttracker 2 default volume implementation. I've attached the new libxmp test module for these quirks, but to summarize the suspected OpenMPT bugs:

1) When a channel currently has an invalid instrument active, instrument numbers should not apply default volume/pan or reset the envelope positions. Row 4 of the example (rows 4-5, 20-21 of the module) plays too loud in OpenMPT, presumably because it reset the envelope positions. This also causes row A of the example below (rows 10-11 of the module) to play when it should be silent.

2) When activating an invalid instrument cuts the channel, it should also zero the channel volume—the volume number by row 4 (rows 4, 20 of the module) is actually necessary for the note to be heard. Rows 28-29 in the module should be silent, but they can be heard in OpenMPT.

0| C-5 01 --    play regular note
1| --- 7f --    invalid instrument sets default volume, resets envelope
2| C-5 -- --    activates invalid instrument; cut channel; no envelope update
3| --- 01 --    no default volume, no envelope reset/update
4| C-6 -- 40    ONLY plays with a volume effect, thus row 2 also zeroed volume.
                Starts on tick (speed) of envelope, thus row 1 reset the
                envelope positions and rows 2/3 did not update them.

8| C-5 7f --    play invalid note; cut, zero volume, no envelope reset/update
9| --- 01 --    no default volume, no envelope reset/update
A| C-6 -- 40    inaudible, because the envelope did not reset. If the envelope
                is extended, however, this does play.