Another issue cropped up I just found.

This resaved .MOD file from FT2 is detected as Protracker-compatible, but it utilizes some arpeggios that go out of range and get silenced.

Trying again. I don't think it lets me upload files, so here is a link just in case

ProTracker mode heuristics cover more than just out-of-range notes. The differences you observed are the loop length bytes, they mustn't be 0 even if the sample has no loop. Some PC trackers (e.g. older ModPlug versions) would write a loop length of 0 and thus can be easily identified as not being ProTracker. However, if you re-save the file in a different tracker, and that tracker conforms to ProTracker limitations (i.e. writes a loop length of 1), then it's only natural that OpenMPT detects it as a ProTracker-compatible file. Since the absolute majority of 4-channel MODs was written in ProTracker, it's generally preferred to detect MODs as ProTracker MODs even if they were written in a different tracker.

I'll see if there are any other details in that MOD file that could make it detectable as a non-ProTracker MOD (arpeggio overflow is not one of them, this effect was deliberately used in some ProTracker MODs), but I doubt I will find something. You will have to turn of ProTracker mode in the song properties manually.

Hmm, it does seem to play back fine in XMPlay regardless of the loop length being 0 or 1. Are there ProTracker files with arpeggio overflow that break XMPlay? I did not even know that was a thing. Very interesting!

XMPlay doesn't use extensive heuristics like OpenMPT does (at least not in this particular case), and does not emulate the arpeggio bug. To be clear, only ProTracker itself (and maybe some other similar trackers on Amiga) will have problems with the loop length of 0. PC-based trackers will virtually never have an issue with this because their software mixers work very differently. But as ProTracker is so popular (and popularized the format), it's always a good idea to use the loop length of 1, even on PC.

Technical background information: ProTracker doesn't have a concept of samples not having a loop. All samples have a loop, but if this loop starts at 0, the sample will first play through completely, and then just the loop part will be played. So typically the first two sampling points of a sample are silenced and the loop is set to cover those two samples, which is effectively just a silent loop going on until the next note is played. If you now set this loop to a length of 0, it breaks ProTracker.

It seems FT2 never actually saved 1 there originally, it saved 0 there - and this was 'fixed' in the FT2-clone version. Perhaps MOD files saved in FT2-clone should have kept a loop length of 0 since that distinction makes it easier to tell that those MODs weren't made in a PT-compliant tracker.

My thoughts:

• Is a 0 loop length really that big a problem on Amiga trackers? PT2-clone seems to treat 0 and 1 the same as far as my limited tests go. Shouldn't replayers be patched for this? IMO starting at 1 is a bit inelegant. I encountered a similar issue with AHX, a hacked decay value of 0 crashes HivelyTracker.
• Did many PT-compliant MOD tunes use non-standard arpeggios as a feature, any examples? I didn't think it was common, but I could be wrong.

Welp, hopefully some clever logic can handle this condition, but for now I think I will resave these FT2 mods in the DOS version rather than the remake.

Perhaps MOD files saved in FT2-clone should have kept a loop length of 0 since that distinction makes it easier to tell that those MODs weren't made in a PT-compliant tracker.

I doubt 8bitbubsy will do that as he cares that his files are playable with ProTracker (he also wrote the ProTracker clone similar to the FT2 clone).

Is a 0 loop length really that big a problem on Amiga trackers?

From what I remember it will lead to various crashes or hangs when you try to edit or save the file. The PT Clone obviously doesn't have this problem because bubsy is aware that this problem exists. It automatically corrects such files when it loads them.

Shouldn't replayers be patched for this? IMO starting at 1 is a bit inelegant

You cannot go back in time and fix software that hasn't been updated in 30 years. You may call it inelegant, but it is simply a de-facto standard. It's how the MOD format works. You cannot change it anymore. HivelyTracker is an actively developed tracker. ProTracker isn't. But people keep using it, and it's the most relevant Amiga tracker to this day.

Either way - the main problem here is that it MOD files contain very little information that helps identifying the program they were made with. Changing one program to write them out in a different way won't help at all in this situation. We can try improving heuristics whenever it's possible, but most often it isn't, and manual tweaking of the compatibility flags will be required. From an initial look at these files, it looks like this will be the case here as well.

https://demozoo.org/music/105630/ - Hot Buttered by Reed and Tempest

is this not a valid protracker file? it is detected as 'screamtracker' and has PT mode disabled whereas tempest/reed's stuff were done on Amiga.

is this not a valid protracker file?

No, there's C-7 in the last pattern. Reed used FT2 at that point. It was probably done in FT2 but as said, the MOD format doesn't have many possibilities to identify a specific tracker a file was made with. However, someones these heuristics are also thrown off if a file is re-saved in different trackers one after another, which is what might have happened here and thus causes the ScreamTracker identification. But it's most definitely not saved with ProTracker.

Anyway, all of this is going off-topic. The original question - whether the heuristics can be improved or not - can at this point be answered with "no" for those specific files. The MOD format has a messy history, and not every problem it has can be fixed by a heuristic. But the tracker gives you the option to fix it manually.