View Issue Details

IDProjectCategoryView StatusLast Update
0000575OpenMPT[All Projects] libopenmptpublic2017-09-17 09:34
ReportermanxAssigned To 
PrioritynormalSeverityminorReproducibilityalways
Status confirmedResolutionopen 
Product Version 
Target VersionOpenMPT 1.?? (long term goals)Fixed in Version 
Summary0000575: Inefficient FILE I/O
Description

Inefficient FILE I/O <s>(especially in openmpt::could_open_propability)</s>:
libopenmpt currently pre-caches the whole file in more situations than theoretically necessary.
This is caused mainly by calls to FileReader::GetRawData() or calls to FileReader::GetLength() (for unseekable streams).

Additional Information

As of r4875, the status is as follows:

  1. <del>GetRawData() always precaches the whole file. Theoretically, only from the current position would be necessary. Ultimately, GetRawData() will always require a view to almost the complete file data as contiguous memory.</del> Many GetRawData() uses have been removed in r6102 and related revisions.
  2. <del>Even seekable streams are currently completely cached while reading.
    This results in twice the filesize of memory consumption: 1. the cache and 2. the parsed file. This hurts platforms with low memory.</del> Somewhere around r6102, this was also improved.
    Status as of r6641:
  3. GetRawData() still always pre-caches the whole file. However, it is only used rarely now.
TagsNo tags attached.
Has the bug occurred in previous versions?
Tested code revision (in case you know it)

Relationships

related to 0000914 resolvedmanx split module format probing into separate TestFOO() static member functions 

Activities

manx

manx

2016-03-18 15:29

administrator   ~0002292

As of r6140, almost all cases relevant to libopenmpt in practice have been converted to not use GetRawData() anymore.
The remaining offenders are: MMCMP, XPK, PP20 depackers and MDL, MED loaders (which need a full loader rewrite anyway).

Saga Musix

Saga Musix

2016-07-17 18:06

administrator   ~0002514

As of r6637, the MDL loader is now also based on FileReader.

Issue History

Date Modified Username Field Change
2014-08-24 07:45 manx New Issue
2014-08-24 07:45 manx Status new => assigned
2014-08-24 07:45 manx Assigned To => manx
2015-03-12 10:25 manx Summary Inefficient FILE I/O, especially in openmpt::could_open_propability => Inefficient FILE I/O <s>(especially in openmpt::could_open_propability)</s>
2015-03-12 10:25 manx Additional Information Updated View Revisions
2015-03-12 15:00 manx Summary Inefficient FILE I/O <s>(especially in openmpt::could_open_propability)</s> => Inefficient FILE I/O
2015-03-12 15:00 manx Description Updated View Revisions
2016-03-15 12:06 manx Additional Information Updated View Revisions
2016-03-18 15:29 manx Note Added: 0002292
2016-03-18 15:30 manx Assigned To manx =>
2016-03-18 15:30 manx Assigned To => manx
2016-03-18 15:30 manx Status assigned => confirmed
2016-03-18 15:30 manx Assigned To manx =>
2016-07-17 18:06 Saga Musix Note Added: 0002514
2016-07-17 18:42 manx Additional Information Updated View Revisions
2017-09-17 09:34 manx Relationship added related to 0000914