View Issue Details

IDProjectCategoryView StatusLast Update
0000828OpenMPTFile Format Supportpublic2019-03-16 13:56
ReporterMonsterovich Assigned To 
PrioritynormalSeverityminorReproducibilitysometimes
Status closedResolutionwon't fix 
Product VersionOpenMPT 1.26.02.* (old testing) 
Summary0000828: itogg format support
Description

https://github.com/Monsterovich/itogg

OpenMPT just should decompress itogg modules to normal .it (.mptm) modules.

TagsNo tags attached.
Has the bug occurred in previous versions?
Tested code revision (in case you know it)

Activities

Saga Musix

Saga Musix

2016-07-07 14:58

administrator   ~0002498

As discussed before on IRC, there is already Modipulate's ITQ format with ogg-compressed samples: https://github.com/MrEricSir/Modipulate/blob/master/src/modipulate/libopenmpt-forked/soundlib/Load_itq.cpp
I'm not a fan of hacking the IT format even further, but if I had to choose between the two formats, I'd rather add ITQ support since it has fewer modifications (header and sample header signatures are changed). Why does your format need an extra header behind the MPTM header, which by the way will make loading logic for the MPTM a lot more complicated?

And even IF we were going to implement this, I'd rather implement the additional changes to the IT format myself rather than relying on an external library which essentially just duplicates code that already exists. I don't consider a DIA file to be sufficient documentation for a file format, though.

Monsterovich

Monsterovich

2016-07-07 18:16

reporter   ~0002499

Last edited: 2016-07-07 18:18

there is already Modipulate's ITQ format with ogg-compressed samples:

I can use my code anywhere, because it's very SIMPLE to do that: https://github.com/Monsterovich/itogg/blob/master/itogg.h There are still a lot of projects/players that not use libopenmpt, but they use libmodplug, libbass, libdumb, libfmodex... What program can covnert to ITQ?

Why does your format need an extra header behind the MPTM header

  1. I don't change impulse tracker sample headers for many reasons.
  2. I have some useful checks for garbage samples that may appear in .itogg modules.

which by the way will make loading logic for the MPTM a lot more complicated?

It won't.

Saga Musix

Saga Musix

2016-07-07 18:19

administrator   ~0002500

<blockquote>What program can covnert to ITQ?</blockquote>
So your only argument against ITQ is that your own program does not support it?
You can write a (far simpler) converter for ITQ just like you could for itOgg.

<blockquote>1. I don't change impulse tracker sample headers for many reasons.</blockquote>
I'm curious about those reasons, because I think it's a particularly bad idea.

Monsterovich

Monsterovich

2016-07-07 18:37

reporter   ~0002501

So your only argument against ITQ is that your own program does not support it?

I can't use ITQ outside libopenmpt anyway. I've already mentioned that.

You can write a (far simpler) converter for ITQ just like you could for itOgg.

Without provided converter this format is useless.

I'm curious about those reasons, because I think it's a particularly bad idea.

Why?

manx

manx

2016-07-07 18:42

administrator   ~0002502

Frankly, I do see neither a reason to add a whole new set of file structures and headers like ITOGG, nor to extend the on-disk structure of the IT sample header like ITQ does. As far as I can see, there are still bits left in the sample header (OpenMPT used one of those to signal external sample in MPTM), which could easily be used to signal OggVorbis encoding (or general compressed encoding, detected using file magics stored in the sample data). As IT already has support for variable-length encoded data in case of IT sample compression, the difference for stored bytes and sample length could be handled just the same way as there.
ITOGG really does complicate the file parsing as you have to parse the whole file in order to actually SEE that you are actually dealing with ITOGG at all. ITQ makes that part really simple by using a distinct file magic instead of amending to an already wide-spread format in confusing ways (which, sadly, OpenMPT has done in the past, and which is still causing us all kinds of parsing and saving trouble on and on again (this is by far the part that most commonly breaks accidentally in the test suite when other minor improvements to the format are done)).
(Just my personal opinion on the matter).

Saga Musix

Saga Musix

2016-07-07 18:48

administrator   ~0002503

Last edited: 2016-07-07 18:50

<blockquote>I can't use ITQ outside libopenmpt anyway. I've already mentioned that.</blockquote>

Yes you can. Just from a quick look (I was never involved with this format), ITQ's only differences to IT are:

  • Magic bytes are ITQM rather than IMPM
  • Magic bytes of a compressed sample are ITQS rather than IMPS. The sample header is extended by two fields, a 32-bit field contaning the compressed length and an 8-bit field containing the codec, which is currently unused as it seems

It cannot be any simpler than this, apart from using existing bits that are still left in the sample header, as manx states. The things you add to the format make it more complicated to parse.

<blockquote>Without provided converter this format is useless.</blockquote>
You can either search for the original ITQ encoder or write your own. Just look at the informal format description above, and you should realize that working from your existing itOgg encoder, it should be little work to extend it to write out ITQ files instead. After all, all you need to do is changing some magic bytes and inserting your vorbis-compressed samples anywhere in the file (preferrably in the same place as PCM-compressed samples, of course).

<blockquote>I'm curious about those reasons, because I think it's a particularly bad idea.

Why?</blockquote>
Because your format contains redundant information and generally no longer follows the way IT files are structured (see also what manx said).

manx

manx

2016-07-08 07:02

administrator   ~0002504

Looking again, I see you are in fact using different magic bytes (using color coding to code only SOME instead of all differences made that very easy to miss in your diagram). All other concerns still apply.

Monsterovich

Monsterovich

2016-08-08 16:23

reporter   ~0002569

Because your format contains redundant information

This information is not redundant, it also simplifies the decoding, because I can decode my format in three steps while ITQ changes a lot of sample data. Plus, I use this information to check for modules with garbage samples.

no longer follows the way IT files are structured

I wasn't planned to do an IT-only structure. What if I want to support other module formats like .xm?

manx

manx

2016-08-08 16:55

administrator   ~0002572

This information is not redundant, it also simplifies the decoding, because I can decode my format in three steps while ITQ changes a lot of sample data. Plus, I use this information to check for modules with garbage samples.

If you can use the information to detect some corruption, it is in fact redundant.
And precisely this kind of redundant information is the root cause of various exploitable file format parsing bugs in software, because different software tends to react differently to inconsistent data.
Also, the by far most common cause for "corrupted" file format data is not data being randomly corrupted due to transfer errors in the wild, but software misunderstanding how it should write such redundant data.
This fact alone is reason enough to not use your format, it's just bad design IMHO.

I wasn't planned to do an IT-only structure. What if I want to support other module formats like .xm?

XM works completely differently and I frankly cannot see how to apply the ITOGG approach to XM at all. For example, your "ISGG" header feels completely alien (even more so than in IT) in XM (as do the OpenMPT extensions, which were a bad design in the first place, in particular in XM). You would need to invent yet another mechanism.

Also, there are already 4 variants of compressed lossy samples for XM and/or IT:
Un4seen MO3 (with 2 (and a half) different implementations), ITQ, OXM, OpenMPT MPTM with external samples. (There is also a hack for minifmod to use MP3 compressed samples which I did myself in the past, for the sole purpose of using it in 64k intros. I think I did spread that code to at least some people. I think there are/were also several other minifmod hacks in similar fashion).
And there is also lossless IT sample compression, which has widespread support already.

Does the world really need yet another, incompatible, method?

Issue History

Date Modified Username Field Change
2016-07-07 14:32 Monsterovich New Issue
2016-07-07 14:58 Saga Musix Note Added: 0002498
2016-07-07 18:16 Monsterovich Note Added: 0002499
2016-07-07 18:18 Monsterovich Note Edited: 0002499
2016-07-07 18:19 Saga Musix Note Added: 0002500
2016-07-07 18:37 Monsterovich Note Added: 0002501
2016-07-07 18:42 manx Note Added: 0002502
2016-07-07 18:48 Saga Musix Note Added: 0002503
2016-07-07 18:50 Saga Musix Note Edited: 0002503
2016-07-08 07:02 manx Note Added: 0002504
2016-08-08 16:23 Monsterovich Note Added: 0002569
2016-08-08 16:55 manx Note Added: 0002572
2016-08-11 12:43 Monsterovich Steps to Reproduce Updated
2019-03-16 13:56 manx Status new => closed
2019-03-16 13:56 manx Resolution open => won't fix