I guess it is worth discussing if this should only generate a warning or fail hard; at least in the context of OpenMPT I think it can still be desirable to load huge modules in a low-memory situation while receiving a warning from the application. For example, think of a slightly damaged file where the "sample length" field of a single sample was destroyed due to a transfer error, and now claims that this one sample is very huge. For the sake of rescuing parts of such a file, we'd still want to be able to load it.

For the specific case of sample data, yes, it might make sense to handle that explicitly and report a warning.

However this probably requires further refactoring compared to just failing to load completely. For libopenmpt, such behaviour would need to be opt-in (i.e. the API user can explicitly set an option to skip out-of-memory sample data after trying first without having that option set).

The only sane way to implement this, is to support setting a maximum allocation size limit upfront before loading, and still failing hard for any allocation failure.

Otherwise, minor allocations that happen after a huge one will still cause a failure to load anyway, without any chance of loading this particular file at all. The only way to handle this is to provide a limit for (in particular) sample data and plugin data, before starting the loading process, so that huge problematic allocations will not happen in the first place.

Converting allocation failure to a warning is the wrong thing to do, even for this use case.

Also, there are tons of other places in the various module formats that might cause large allocations (ultimately, any length field that is 32 bits could have the same issue, iff the allocation is not additionally limited to remaining file data). [1]

Which is a very important point: Iff the file is actually huge, we should better try to allocate the huge requested buffer, and if it is not, we should only allocate as much as could sanely be needed, and if this is not known upfront, we should be just growing a buffer on demand. I fail to see how converting to warning would be required to handle broken files.

This leaves leaves the allocation size limitation as only an additional feature which could maybe useful for very memory-constrained situations in the tracker. I do not see any use for it in libopenmpt any more. In particular, I do not think we should block/delay proper out-of-memory handling because this potential feature.

[1] Rogue unlimited allocations based on a 32bit length field in a module file can easily be found with fuzzing (in combination with allocator instrumentation if needed).

It might make sense to implement proper out-of-memory handling in a development branch instead of directly in trunk.