View Issue Details

IDProjectCategoryView StatusLast Update
0000575OpenMPTlibopenmptpublic2019-09-24 12:11
Reportermanx Assigned Tomanx  
PrioritynormalSeverityminorReproducibilityalways
Status resolvedResolutionfixed 
Target VersionOpenMPT 1.29.01.00 / libopenmpt 0.5.0 (upgrade first)Fixed in VersionOpenMPT 1.29.01.00 / libopenmpt 0.5.0 (upgrade first) 
Summary0000575: Inefficient FILE I/O
Description

Inefficient FILE I/O (especially in openmpt::could_open_propability):
libopenmpt currently pre-caches the whole file in more situations than theoretically necessary.
This is caused mainly by calls to FileReader::GetRawData() or calls to FileReader::GetLength() (for unseekable streams).

Additional Information

As of r4875, the status is as follows:

  1. GetRawData() always precaches the whole file. Theoretically, only from the current position would be necessary. Ultimately, GetRawData() will always require a view to almost the complete file data as contiguous memory. Many GetRawData() uses have been removed in r6102 and related revisions.
  2. Even seekable streams are currently completely cached while reading.
    This results in twice the filesize of memory consumption: 1. the cache and 2. the parsed file. This hurts platforms with low memory.
    Somewhere around r6102, this was also improved.
    Status as of r6641:
  3. GetRawData() still always pre-caches the whole file. However, it is only used rarely now.
TagsNo tags attached.
Has the bug occurred in previous versions?
Tested code revision (in case you know it)

Relationships

related to 0000914 resolvedmanx split module format probing into separate TestFOO() static member functions 

Activities

manx

manx

2016-03-18 15:29

administrator   ~0002292

As of r6140, almost all cases relevant to libopenmpt in practice have been converted to not use GetRawData() anymore.
The remaining offenders are: MMCMP, XPK, PP20 depackers and MDL, MED loaders (which need a full loader rewrite anyway).

Saga Musix

Saga Musix

2016-07-17 18:06

administrator   ~0002514

As of r6637, the MDL loader is now also based on FileReader.

manx

manx

2019-07-01 07:50

administrator   ~0003965

As of r11660, r11661 , std::istream is no longer slow.

Issue History

Date Modified Username Field Change
2014-08-24 07:45 manx New Issue
2014-08-24 07:45 manx Status new => assigned
2014-08-24 07:45 manx Assigned To => manx
2015-03-12 10:25 manx Summary Inefficient FILE I/O, especially in openmpt::could_open_propability => Inefficient FILE I/O (especially in openmpt::could_open_propability)
2015-03-12 10:25 manx Additional Information Updated
2015-03-12 15:00 manx Summary Inefficient FILE I/O (especially in openmpt::could_open_propability) => Inefficient FILE I/O
2015-03-12 15:00 manx Description Updated
2016-03-15 12:06 manx Additional Information Updated
2016-03-18 15:29 manx Note Added: 0002292
2016-03-18 15:30 manx Assigned To manx =>
2016-03-18 15:30 manx Assigned To => manx
2016-03-18 15:30 manx Status assigned => confirmed
2016-03-18 15:30 manx Assigned To manx =>
2016-07-17 18:06 Saga Musix Note Added: 0002514
2016-07-17 18:42 manx Additional Information Updated
2017-09-17 09:34 manx Relationship added related to 0000914
2019-07-01 07:50 manx Note Added: 0003965
2019-09-24 12:10 manx Target Version OpenMPT 1.?? (long term goals) => OpenMPT 1.29.01.00 / libopenmpt 0.5.0 (upgrade first)
2019-09-24 12:10 manx Assigned To => manx
2019-09-24 12:10 manx Status confirmed => resolved
2019-09-24 12:10 manx Resolution open => fixed
2019-09-24 12:10 manx Fixed in Version => OpenMPT 1.29.01.00 / libopenmpt 0.5.0 (upgrade first)