View Issue Details

IDProjectCategoryView StatusLast Update
0001559OpenMPTGeneralpublic2025-12-12 19:57
ReporterSaga Musix Assigned Tomanx  
PrioritynormalSeverityminorReproducibilityN/A
Status assignedResolutionopen 
Target VersionOpenMPT 1.33 / libopenmpt 0.9 (goals) 
Summary0001559: Don't use WinAPI functions for INI reading/writing
Description

Currently OpenMPT uses ReadPrivateProfile* / WritePrivateProfile*functions provided by Windows for reading and writing its settings files. This has several issues

  • It's relatively slow, which is why we only write setting that we know have changed
  • It requires hacks to enforce writing unicode files

As a consequence of the first point, it's possible that one OpenMPT instance takes a long time to shut down, and a newly launched instance may not see the settings as saved by the previous instance. One particularly worrying example is the following scenario:

  1. Fresh OpenMPT installation, user sees the Welcome dialog and unchecks automatic update checks
  2. User closes OpenMPT. This is the first time the INI file is written, so it takes quite a while to write out all settings.
  3. User immediately reopens OpenMPT. It may now try to search for updates because the old instance might not have finished writing the settings yet.

A custom INI implementation should avoid this scenario. While the file is being written, other OpenMPT instances should wait until the file is no longer locked, rather than reading incomplete settings.

TagsNo tags attached.
Has the bug occurred in previous versions?
Tested code revision (in case you know it)

Activities

manx

manx

2025-08-16 16:27

administrator   ~0006447

A custom INI implementation should avoid this scenario. While the file is being written, other OpenMPT instances should wait until the file is no longer locked, rather than reading incomplete settings.

I do not think this would be necessary. Atomically writing the file is the easier approach, because we do not (and do not want to, and cannot) guarantee settings consistency across different instances running concurrently anyway. Writing the file in 1 go should also be plenty fast enough to avoid the mentioned race condition. If the second instance is started while the first one is still actively running, there is not much we can do anyway, except for maybe flushing the configuration file after the welcome dialog (which would probably be a good idea even with the old implementation).

There are a couple of open questions regarding our own implementation:

  1. Do we want to handle binary data in string data and key names properly? Windows INI parsers do random confusing and undocumented things here, mostly stripping unwanted characters and trimming the result.
  2. Do we want to preserve ordering of the INI file as read from disk, or do we want to canonicalize/sort what we write out?
  3. Should the format be strictly backward compatible with the Windows INI parser, mimicing all its undocumented quirks?
  4. How to handle duplicate entries? (I did not look into how the Windows INI parser handles this)

The INI implementation that I have in an older code base answers these questions as:

  1. escape both key and value data
  2. strictly sorted output, preserving comments to the respective entry on a best effort basis
  3. no, because of 1 and 2
  4. I have not looked into that in detail yet.

In case the answer to 3 is "no", we should use a different file name in order to not break downgrades completely. "OpenMPT.ini" would make sense here.

Saga Musix

Saga Musix

2025-08-30 13:00

administrator   ~0006454

While strict backwards compatibility would be nice to have, I think it should not delay this effort. I think under normal circumstances 3 should not matter at all, only when a user hand-edited the file. As we lost the ability to automatically back up the current OpenMPT configuration during upgrades, using a new settings name file would probably the safest choice.

manx

manx

2025-10-18 17:35

administrator   ~0006514

[Imp] Settings: Write settings in batches of whole sections with WritePrivateProfileSection. This significantly speeds up writing initial settings on the first run from ~50ms to ~5ms on an NVMe SSD.
[Reg] Settings: Comments in INI files are now always deleted. This is a limitation of GetPrivateProfileSection/WritePrivateProfileSection.
[Reg] Settings: Settings inside each section are now ordered alphabetically in the INI file instead of retaining previous ordering. This could be changed.
[Mod] Settings: Always convert all INI files to Unicode when on Windows XP or later and using a UNICODE build. We cannot sensible do this lazily any more, because that would imply testing each individual setting for roundtrip invariance, which would invalidate the whole speedup.

This mainly happened by accident. Using GetPrivateProfileSection would be required anyway for converting from the existing Win32 INI functions to a custom implementation, because we cannot rely on all settings being read eagerly into out internal data structures (i.e. currently not selected Sound Devices). Adding the writing was only a small extra step.

There is another potential problem that I noticed while working on this: We are passing the INI file name to MFC via m_pszProfileName. If we would implement our own incompatible INI format, this would require passing a separate file to MFC. I am not sure what MFC is currently using this file for.

speedup-ini-settings-v1.patch (7,831 bytes)   
Index: mptrack/Settings.cpp
===================================================================
--- mptrack/Settings.cpp	(revision 24333)
+++ mptrack/Settings.cpp	(working copy)
@@ -140,6 +140,16 @@
 	backend->RemoveSection(section);
 }
 
+bool SettingsContainer::BackendsCanWriteMultipleSettings() const
+{
+	return backend->CanWriteMultipleSettings();
+}
+
+void SettingsContainer::BackendsWriteMultipleSettings(const std::map<SettingPath, SettingValue> &settings)
+{
+	backend->WriteMultipleSettings(settings);
+}
+
 SettingValue SettingsContainer::ReadSetting(const SettingPath &path, const SettingValue &def) const
 {
 	ASSERT(theApp.InGuiThread());
@@ -242,6 +252,20 @@
 {
 	ASSERT(theApp.InGuiThread());
 	ASSERT(!CMainFrame::GetMainFrame() || (CMainFrame::GetMainFrame() && !CMainFrame::GetMainFrame()->InNotifyHandler())); // This is a slow path, use CachedSetting for stuff that is accessed in notify handler.
+	if(BackendsCanWriteMultipleSettings())
+	{
+		std::map<SettingPath, SettingValue> settings;
+		for(auto &[path, value] : map)
+		{
+			if(value.IsDirty())
+			{
+				settings.insert(std::make_pair(path, value.GetRefValue()));
+				value.Clean();
+			}
+		}
+		BackendsWriteMultipleSettings(settings);
+		return;
+	}
 	for(auto &[path, value] : map)
 	{
 		if(value.IsDirty())
@@ -407,7 +431,12 @@
 IniFileSettingsBackend::IniFileSettingsBackend(const mpt::PathString &filename)
 	: filename(filename)
 {
-	return;
+#if defined(UNICODE)
+	if(mpt::osinfo::windows::Version::Current().IsAtLeast(mpt::osinfo::windows::Version::WinXP))
+	{
+		ConvertToUnicode();
+	}
+#endif
 }
 
 IniFileSettingsBackend::~IniFileSettingsBackend()
@@ -493,10 +522,143 @@
 	RemoveSectionRaw(section);
 }
 
+std::set<mpt::winstring> IniFileSettingsBackend::ReadSections() const
+{
+	std::set<mpt::winstring> result;
+	const std::vector<TCHAR> sectionsstr = [&]()
+		{
+			std::vector<TCHAR> buf;
+			buf.resize(1024);
+			while(true)
+			{
+				DWORD bufused = ::GetPrivateProfileSectionNames(buf.data(), mpt::saturate_cast<DWORD>(buf.size()), filename.AsNative().c_str());
+				if(bufused >= (buf.size() - 2))
+				{
+					buf.resize(mpt::exponential_grow(buf.size()));
+					continue;
+				}
+				buf.resize(bufused);
+				break;
+			};
+			return buf;
+		}();
+	const std::vector<mpt::winstring> sections = mpt::split(mpt::winstring(sectionsstr.data(), sectionsstr.size()), mpt::winstring(_T("\0"), 1));
+	for(const auto &section : sections)
+	{
+		result.insert(section);
+	}
+	return result;
+}
 
+std::map<mpt::winstring, mpt::winstring> IniFileSettingsBackend::ReadSection(const mpt::winstring &section) const
+{
+	std::map<mpt::winstring, mpt::winstring> result;
+	const std::vector<TCHAR> keyvalsstr = [&]()
+		{
+			std::vector<TCHAR> buf;
+			buf.resize(1024);
+			while(true)
+			{
+				DWORD bufused = ::GetPrivateProfileSection(mpt::ToWin(section).c_str(), buf.data(), mpt::saturate_cast<DWORD>(buf.size()), filename.AsNative().c_str());
+				if(bufused >= (buf.size() - 2))
+				{
+					buf.resize(mpt::exponential_grow(buf.size()));
+					continue;
+				}
+				buf.resize(bufused);
+				break;
+			};
+			return buf;
+		}();
+	const std::vector<mpt::winstring> keyvals = mpt::split(mpt::winstring(keyvalsstr.data(), keyvalsstr.size()), mpt::winstring(_T("\0"), 1));
+	for(const auto &keyval : keyvals)
+	{
+		const auto equalpos = keyval.find(_T("="));
+		if(equalpos == mpt::winstring::npos)
+		{
+			continue;
+		}
+		if(equalpos == 0)
+		{
+			continue;
+		}
+		result.insert(std::make_pair(keyval.substr(0, equalpos), keyval.substr(equalpos + 1)));
+	}
+	return result;
+}
 
+mpt::winstring IniFileSettingsBackend::FormatValueAsIni(const SettingValue &value)
+{
+	switch(value.GetType())
+	{
+		case SettingTypeBool:
+			return mpt::tfmt::val(value.as<bool>());
+			break;
+		case SettingTypeInt:
+			return mpt::tfmt::val(value.as<int32>());
+			break;
+		case SettingTypeFloat:
+			return mpt::tfmt::val(value.as<double>());
+			break;
+		case SettingTypeString:
+			return mpt::ToWin(value.as<mpt::ustring>());
+			break;
+		case SettingTypeBinary:
+			{
+				std::vector<std::byte> data = value.as<std::vector<std::byte>>();
+				uint8 checksum = 0;
+				for(const std::byte x : data) {
+					checksum += mpt::byte_cast<uint8>(x);
+				}
+				return mpt::ToWin(mpt::encode_hex(mpt::as_span(data)) + mpt::ufmt::HEX0<2>(checksum));
+			}
+			break;
+		case SettingTypeNone:
+		default:
+			return mpt::ustring();
+			break;
+	}
+}
 
+void IniFileSettingsBackend::WriteSection(const mpt::winstring &section, const std::map<mpt::winstring, mpt::winstring> keyvalues)
+{
+	mpt::winstring keyvals;
+	for(const auto &[key, val] : keyvalues)
+	{
+		keyvals.append(key);
+		keyvals.append(_T("="));
+		keyvals.append(val);
+		keyvals.append(mpt::winstring(_T("\0"), 1));
+	}
+	keyvals.append(mpt::winstring(_T("\0"), 1));
+	::WritePrivateProfileSection(section.c_str(), keyvals.c_str(), filename.AsNative().c_str());
+}
 
+void IniFileSettingsBackend::WriteMultipleSettings(const std::map<SettingPath, SettingValue> &settings)
+{
+	std::map<mpt::ustring, std::map<SettingPath, SettingValue>> sectionssettings;
+	for(const auto &[path, value] : settings)
+	{
+		sectionssettings[path.GetRefSection()][path] = value;
+	}
+	for(const auto &[section, sectionsettings] : sectionssettings)
+	{
+		std::map<mpt::winstring, mpt::winstring> workingsectionsettings = ReadSection(mpt::ToWin(section));
+		for(const auto &[path, value] : sectionsettings)
+		{
+			workingsectionsettings[mpt::ToWin(path.GetRefKey())] = FormatValueAsIni(value);
+		}
+		WriteSection(mpt::ToWin(section), workingsectionsettings);
+	}
+}
+
+bool IniFileSettingsBackend::CanWriteMultipleSettings() const
+{
+	return true;
+}
+
+
+
 IniFileSettingsContainer::IniFileSettingsContainer(const mpt::PathString &filename)
 	: IniFileSettingsBackend(filename)
 	, SettingsContainer(this)
Index: mptrack/Settings.h
===================================================================
--- mptrack/Settings.h	(revision 24333)
+++ mptrack/Settings.h	(working copy)
@@ -408,6 +408,8 @@
 	virtual void WriteSetting(const SettingPath &path, const SettingValue &val) = 0;
 	virtual void RemoveSetting(const SettingPath &path) = 0;
 	virtual void RemoveSection(const mpt::ustring &section) = 0;
+	virtual bool CanWriteMultipleSettings() const = 0;
+	virtual void WriteMultipleSettings(const std::map<SettingPath, SettingValue> &settings) = 0;
 protected:
 	virtual ~ISettingsBackend() = default;
 };
@@ -448,6 +450,8 @@
 	void BackendsWriteSetting(const SettingPath &path, const SettingValue &val);
 	void BackendsRemoveSetting(const SettingPath &path);
 	void BackendsRemoveSection(const mpt::ustring &section);
+	bool BackendsCanWriteMultipleSettings() const;
+	void BackendsWriteMultipleSettings(const std::map<SettingPath, SettingValue> &settings);
 	void NotifyListeners(const SettingPath &path);
 	SettingValue ReadSetting(const SettingPath &path, const SettingValue &def) const;
 	bool IsDefaultSetting(const SettingPath &path) const;
@@ -711,6 +715,14 @@
 	virtual void WriteSetting(const SettingPath &path, const SettingValue &val) override;
 	virtual void RemoveSetting(const SettingPath &path) override;
 	virtual void RemoveSection(const mpt::ustring &section) override;
+private:
+	std::set<mpt::winstring> ReadSections() const;
+	std::map<mpt::winstring, mpt::winstring> ReadSection(const mpt::winstring &section) const;
+	static mpt::winstring FormatValueAsIni(const SettingValue &value);
+	void WriteSection(const mpt::winstring &section, const std::map<mpt::winstring, mpt::winstring> keyvalues);
+public:
+	virtual bool CanWriteMultipleSettings() const override;
+	virtual void WriteMultipleSettings(const std::map<SettingPath, SettingValue> &settings) override;
 	const mpt::PathString& GetFilename() const { return filename; }
 };
 
speedup-ini-settings-v1.patch (7,831 bytes)   
Saga Musix

Saga Musix

2025-10-19 18:35

administrator   ~0006515

There is another potential problem that I noticed while working on this: We are passing the INI file name to MFC via m_pszProfileName. If we would implement our own incompatible INI format, this would require passing a separate file to MFC. I am not sure what MFC is currently using this file for.

This is for storing the position and visibility of toolbars. It would probably be possible to serialize them outselves by overwriting some functions, then MFC would not be involved at all anymore.

manx

manx

2025-10-25 18:52

administrator   ~0006517

speedup-ini-settings-v3-wip.patch (17,925 bytes)   
Index: mptrack/Settings.cpp
===================================================================
--- mptrack/Settings.cpp	(revision 24343)
+++ mptrack/Settings.cpp	(working copy)
@@ -140,6 +140,16 @@
 	backend->RemoveSection(section);
 }
 
+bool SettingsContainer::BackendsCanWriteMultipleSettings() const
+{
+	return backend->CanBatchedSettings();
+}
+
+void SettingsContainer::BackendsWriteMultipleSettings(const std::map<SettingPath, SettingValue> &settings)
+{
+	backend->WriteMultipleSettings(settings);
+}
+
 SettingValue SettingsContainer::ReadSetting(const SettingPath &path, const SettingValue &def) const
 {
 	ASSERT(theApp.InGuiThread());
@@ -242,6 +252,26 @@
 {
 	ASSERT(theApp.InGuiThread());
 	ASSERT(!CMainFrame::GetMainFrame() || (CMainFrame::GetMainFrame() && !CMainFrame::GetMainFrame()->InNotifyHandler())); // This is a slow path, use CachedSetting for stuff that is accessed in notify handler.
+	if(BackendsCanWriteMultipleSettings())
+	{
+		std::map<SettingPath, SettingValue> settings;
+		for(auto &[path, value] : map)
+		{
+			if(value.IsDirty())
+			{
+				settings.insert(std::make_pair(path, value.GetRefValue()));
+			}
+		}
+		BackendsWriteMultipleSettings(settings);
+		for(auto &[path, value] : map)
+		{
+			if(value.IsDirty())
+			{
+				value.Clean();
+			}
+		}
+		return;
+	}
 	for(auto &[path, value] : map)
 	{
 		if(value.IsDirty())
@@ -405,11 +435,22 @@
 
 
 IniFileSettingsBackend::IniFileSettingsBackend(const mpt::PathString &filename)
-	: filename(filename)
+	: IniFileSettingsBackend(filename, CachePolicy::Cached)
 {
 	return;
 }
 
+IniFileSettingsBackend::IniFileSettingsBackend(const mpt::PathString &filename, CachePolicy cachePolicy)
+	: filename(filename)
+{
+	OPENMPT_PROFILE_FUNCTION(Profiler::Settings);
+	if(cachePolicy == CachePolicy::Cached)
+	{
+		ConvertToUnicode();
+		cache = ReadAllSectionsRaw();
+	}
+}
+
 IniFileSettingsBackend::~IniFileSettingsBackend()
 {
 	return;
@@ -469,14 +510,41 @@
 SettingValue IniFileSettingsBackend::ReadSetting(const SettingPath &path, const SettingValue &def) const
 {
 	OPENMPT_PROFILE_FUNCTION(Profiler::Settings);
-	switch(def.GetType())
+	if(cache.has_value())
 	{
-	case SettingTypeBool: return SettingValue(ReadSettingRaw(path, def.as<bool>()), def.GetTypeTag()); break;
-	case SettingTypeInt: return SettingValue(ReadSettingRaw(path, def.as<int32>()), def.GetTypeTag()); break;
-	case SettingTypeFloat: return SettingValue(ReadSettingRaw(path, def.as<double>()), def.GetTypeTag()); break;
-	case SettingTypeString: return SettingValue(ReadSettingRaw(path, def.as<mpt::ustring>()), def.GetTypeTag()); break;
-	case SettingTypeBinary: return SettingValue(ReadSettingRaw(path, def.as<std::vector<std::byte> >()), def.GetTypeTag()); break;
-	default: return SettingValue(); break;
+		const auto sectionit = (*cache).find(mpt::ToWin(path.GetRefSection()));
+		if(sectionit != (*cache).end())
+		{
+			const std::map<mpt::winstring, std::optional<mpt::winstring>> &section = sectionit->second;
+			const auto it = section.find(mpt::ToWin(path.GetRefKey()));
+			if(it != section.end())
+			{
+				if(it->second.has_value())
+				{
+					return ParseValueFromIni(it->second.value(), def);
+				} else
+				{
+					return def;
+				}
+			} else
+			{
+				return def;
+			}
+		} else
+		{
+			return def;
+		}
+	} else
+	{
+		switch(def.GetType())
+		{
+		case SettingTypeBool: return SettingValue(ReadSettingRaw(path, def.as<bool>()), def.GetTypeTag()); break;
+		case SettingTypeInt: return SettingValue(ReadSettingRaw(path, def.as<int32>()), def.GetTypeTag()); break;
+		case SettingTypeFloat: return SettingValue(ReadSettingRaw(path, def.as<double>()), def.GetTypeTag()); break;
+		case SettingTypeString: return SettingValue(ReadSettingRaw(path, def.as<mpt::ustring>()), def.GetTypeTag()); break;
+		case SettingTypeBinary: return SettingValue(ReadSettingRaw(path, def.as<std::vector<std::byte> >()), def.GetTypeTag()); break;
+		default: return SettingValue(); break;
+		}
 	}
 }
 
@@ -493,12 +561,22 @@
 	case SettingTypeBinary: WriteSettingRaw(path, val.as<std::vector<std::byte> >()); break;
 	default: break;
 	}
+	if(cache.has_value())
+	{
+		(*cache)[mpt::ToWin(path.GetRefSection())][mpt::ToWin(path.GetRefKey())] = FormatValueAsIni(val);
+	}
 }
 
 void IniFileSettingsBackend::RemoveSetting(const SettingPath &path)
 {
 	OPENMPT_PROFILE_FUNCTION(Profiler::Settings);
-	RemoveSettingRaw(path);
+	if(cache.has_value())
+	{
+		(*cache)[mpt::ToWin(path.GetRefSection())][mpt::ToWin(path.GetRefKey())] = std::nullopt;
+	} else
+	{
+		RemoveSettingRaw(path);
+	}
 }
 
 void IniFileSettingsBackend::RemoveSection(const mpt::ustring &section)
@@ -505,19 +583,255 @@
 {
 	OPENMPT_PROFILE_FUNCTION(Profiler::Settings);
 	RemoveSectionRaw(section);
+	if(cache.has_value())
+	{
+		const auto it = (*cache).find(mpt::ToWin(section));
+		if(it != (*cache).end())
+		{
+			(*cache).erase(it);
+		}
+	}
 }
 
+std::set<mpt::winstring> IniFileSettingsBackend::ReadSectionNamesRaw() const
+{
+	std::set<mpt::winstring> result;
+	const std::vector<TCHAR> sectionsstr = [&]()
+		{
+			std::vector<TCHAR> buf;
+			buf.resize(mpt::IO::BUFFERSIZE_SMALL);
+			while(true)
+			{
+				DWORD bufused = ::GetPrivateProfileSectionNames(buf.data(), mpt::saturate_cast<DWORD>(buf.size()), filename.AsNative().c_str());
+				if(bufused >= (buf.size() - 2))
+				{
+					std::size_t newsize = mpt::exponential_grow(buf.size());
+					buf.resize(0);
+					buf.resize(newsize);
+					continue;
+				}
+				if(bufused >= 1)
+				{
+					bufused -= 1;  // terminating \0
+				}
+				buf.resize(bufused);
+				break;
+			};
+			return buf;
+		}();
+	const std::vector<mpt::winstring> sections = mpt::split(mpt::winstring(sectionsstr.data(), sectionsstr.size()), mpt::winstring(_T("\0"), 1));
+	for(const auto &section : sections)
+	{
+		result.insert(section);
+	}
+	return result;
+}
 
+std::map<mpt::winstring, std::optional<mpt::winstring>> IniFileSettingsBackend::ReadNamedSectionRaw(const mpt::winstring &section) const
+{
+	std::map<mpt::winstring, std::optional<mpt::winstring>> result;
+	const std::vector<TCHAR> keyvalsstr = [&]()
+		{
+			std::vector<TCHAR> buf;
+			buf.resize(mpt::IO::BUFFERSIZE_SMALL);
+			while(true)
+			{
+				DWORD bufused = ::GetPrivateProfileSection(mpt::ToWin(section).c_str(), buf.data(), mpt::saturate_cast<DWORD>(buf.size()), filename.AsNative().c_str());
+				if(bufused >= (buf.size() - 2))
+				{
+					std::size_t newsize = mpt::exponential_grow(buf.size());
+					buf.resize(0);
+					buf.resize(newsize);
+					continue;
+				}
+				if(bufused >= 1)
+				{
+					bufused -= 1;  // terminating \0
+				}
+				buf.resize(bufused);
+				break;
+			};
+			return buf;
+		}();
+	const std::vector<mpt::winstring> keyvals = mpt::split(mpt::winstring(keyvalsstr.data(), keyvalsstr.size()), mpt::winstring(_T("\0"), 1));
+	for(const auto &keyval : keyvals)
+	{
+		const auto equalpos = keyval.find(_T("="));
+		if(equalpos == mpt::winstring::npos)
+		{
+			continue;
+		}
+		if(equalpos == 0)
+		{
+			continue;
+		}
+		result.insert(std::make_pair(keyval.substr(0, equalpos), std::make_optional(keyval.substr(equalpos + 1))));
+	}
+	return result;
+}
 
+std::map<mpt::winstring, std::map<mpt::winstring, std::optional<mpt::winstring>>> IniFileSettingsBackend::ReadAllSectionsRaw() const
+{
+	std::map<mpt::winstring, std::map<mpt::winstring, std::optional<mpt::winstring>>> result;
+	const std::set<mpt::winstring> sectionnames = ReadSectionNamesRaw();
+	for(const mpt::winstring &sectionname : sectionnames)
+	{
+		result.insert(std::make_pair(sectionname, ReadNamedSectionRaw(sectionname)));
+	}
+	return result;
+}
 
+mpt::winstring IniFileSettingsBackend::FormatValueAsIni(const SettingValue &value)
+{
+	switch(value.GetType())
+	{
+		case SettingTypeBool:
+			return mpt::tfmt::val(value.as<bool>());
+			break;
+		case SettingTypeInt:
+			return mpt::tfmt::val(value.as<int32>());
+			break;
+		case SettingTypeFloat:
+			return mpt::tfmt::val(value.as<double>());
+			break;
+		case SettingTypeString:
+			return mpt::ToWin(value.as<mpt::ustring>());
+			break;
+		case SettingTypeBinary:
+			{
+				std::vector<std::byte> data = value.as<std::vector<std::byte>>();
+				uint8 checksum = 0;
+				for(const std::byte x : data) {
+					checksum += mpt::byte_cast<uint8>(x);
+				}
+				return mpt::ToWin(mpt::encode_hex(mpt::as_span(data)) + mpt::ufmt::HEX0<2>(checksum));
+			}
+			break;
+		case SettingTypeNone:
+		default:
+			return mpt::winstring();
+			break;
+	}
+}
 
+
+SettingValue IniFileSettingsBackend::ParseValueFromIni(const mpt::winstring &str, const SettingValue &def)
+{
+	switch(def.GetType())
+	{
+		case SettingTypeBool:
+			return SettingValue(mpt::parse_or<bool>(mpt::trim(str), def.as<bool>()), def.GetTypeTag());
+			break;
+		case SettingTypeInt:
+			return SettingValue(mpt::parse_or<int32>(mpt::trim(str), def.as<int32>()), def.GetTypeTag());
+			break;
+		case SettingTypeFloat:
+			return SettingValue(mpt::parse_or<double>(mpt::trim(str), def.as<double>()), def.GetTypeTag());
+			break;
+		case SettingTypeString:
+			return SettingValue(mpt::ToUnicode(str), def.GetTypeTag());
+			break;
+		case SettingTypeBinary:
+			{
+				mpt::ustring ustr = mpt::trim(mpt::ToUnicode(str));
+				if((ustr.length() % 2) != 0)
+				{
+					return SettingValue(def.as<std::vector<std::byte>>(), def.GetTypeTag());
+				}
+				std::vector<std::byte> data = mpt::decode_hex(ustr);
+				if(data.size() < 1)
+				{
+					return SettingValue(def.as<std::vector<std::byte>>(), def.GetTypeTag());
+				}
+				const uint8 storedchecksum = mpt::byte_cast<uint8>(data[data.size() - 1]);
+				data.resize(data.size() - 1);
+				uint8 calculatedchecksum = 0;
+				for(const std::byte x : data) {
+					calculatedchecksum += mpt::byte_cast<uint8>(x);
+				}
+				if(calculatedchecksum != storedchecksum)
+				{
+					return SettingValue(def.as<std::vector<std::byte>>(), def.GetTypeTag());
+				}
+				return SettingValue(data, def.GetTypeTag());
+			}
+			break;
+		default:
+			return SettingValue();
+			break;
+	}
+}
+
+void IniFileSettingsBackend::WriteSectionRaw(const mpt::winstring &section, const std::map<mpt::winstring, std::optional<mpt::winstring>> &keyvalues)
+{
+	mpt::winstring keyvals;
+	for(const auto &[key, val] : keyvalues)
+	{
+		if(val.has_value())
+		{
+			keyvals.append(key);
+			keyvals.append(_T("="));
+			keyvals.append(val.value());
+			keyvals.append(mpt::winstring(_T("\0"), 1));
+		}
+	}
+	keyvals.append(mpt::winstring(_T("\0"), 1));
+	::WritePrivateProfileSection(section.c_str(), keyvals.c_str(), filename.AsNative().c_str());
+}
+
+void IniFileSettingsBackend::WriteMultipleSettings(const std::map<SettingPath, SettingValue> &settings)
+{
+	OPENMPT_PROFILE_FUNCTION(Profiler::Settings);
+	std::map<mpt::ustring, std::map<SettingPath, SettingValue>> sectionssettings;
+	for(const auto &[path, value] : settings)
+	{
+		sectionssettings[path.GetRefSection()][path] = value;
+	}
+	for(const auto &[section, sectionsettings] : sectionssettings)
+	{
+		// we need to re-read the section before writing it out in order to not overwrite settings written by MFC
+		std::map<mpt::winstring, std::optional<mpt::winstring>> workingsectionsettings = ReadNamedSectionRaw(mpt::ToWin(section));
+		// apply deleted settings from cache
+		for(const auto &[key, value] : (*cache)[mpt::ToWin(section)])
+		{
+			if(!value.has_value())
+			{
+				workingsectionsettings[key] = std::nullopt;
+			}
+		}
+		for(const auto &[path, value] : sectionsettings)
+		{
+			workingsectionsettings[mpt::ToWin(path.GetRefKey())] = FormatValueAsIni(value);
+		}
+		WriteSectionRaw(mpt::ToWin(section), workingsectionsettings);
+		if(cache.has_value())
+		{
+			(*cache)[mpt::ToWin(section)] = std::move(workingsectionsettings);
+		}
+	}
+}
+
+bool IniFileSettingsBackend::CanBatchedSettings() const
+{
+	return cache.has_value();
+}
+
+
+
 IniFileSettingsContainer::IniFileSettingsContainer(const mpt::PathString &filename)
-	: IniFileSettingsBackend(filename)
+	: IniFileSettingsBackend(filename, CachePolicy::Cached)
 	, SettingsContainer(this)
 {
 	return;
 }
 
+IniFileSettingsContainer::IniFileSettingsContainer(const mpt::PathString &filename, CachePolicy cachePolicy)
+	: IniFileSettingsBackend(filename, cachePolicy)
+	, SettingsContainer(this)
+{
+	return;
+}
+
 IniFileSettingsContainer::~IniFileSettingsContainer()
 {
 	return;
Index: mptrack/Settings.h
===================================================================
--- mptrack/Settings.h	(revision 24343)
+++ mptrack/Settings.h	(working copy)
@@ -408,6 +408,8 @@
 	virtual void WriteSetting(const SettingPath &path, const SettingValue &val) = 0;
 	virtual void RemoveSetting(const SettingPath &path) = 0;
 	virtual void RemoveSection(const mpt::ustring &section) = 0;
+	virtual bool CanBatchedSettings() const = 0;
+	virtual void WriteMultipleSettings(const std::map<SettingPath, SettingValue> &settings) = 0;
 protected:
 	virtual ~ISettingsBackend() = default;
 };
@@ -448,6 +450,8 @@
 	void BackendsWriteSetting(const SettingPath &path, const SettingValue &val);
 	void BackendsRemoveSetting(const SettingPath &path);
 	void BackendsRemoveSection(const mpt::ustring &section);
+	bool BackendsCanWriteMultipleSettings() const;
+	void BackendsWriteMultipleSettings(const std::map<SettingPath, SettingValue> &settings);
 	void NotifyListeners(const SettingPath &path);
 	SettingValue ReadSetting(const SettingPath &path, const SettingValue &def) const;
 	bool IsDefaultSetting(const SettingPath &path) const;
@@ -684,11 +688,26 @@
 };
 
 
+enum class CachePolicy
+{
+	Direct,
+	Cached,
+};
+
 class IniFileSettingsBackend : public ISettingsBackend
 {
+public:
 private:
 	const mpt::PathString filename;
+	std::optional<std::map<mpt::winstring, std::map<mpt::winstring, std::optional<mpt::winstring>>>> cache;
+#ifdef ENABLE_TESTS
+public:
+#else
 private:
+#endif
+	std::set<mpt::winstring> ReadSectionNamesRaw() const;
+	std::map<mpt::winstring, std::optional<mpt::winstring>> ReadNamedSectionRaw(const mpt::winstring &section) const;
+	std::map<mpt::winstring, std::map<mpt::winstring, std::optional<mpt::winstring>>> ReadAllSectionsRaw() const;
 	std::vector<std::byte> ReadSettingRaw(const SettingPath &path, const std::vector<std::byte> &def) const;
 	mpt::ustring ReadSettingRaw(const SettingPath &path, const mpt::ustring &def) const;
 	double ReadSettingRaw(const SettingPath &path, double def) const;
@@ -699,12 +718,17 @@
 	void WriteSettingRaw(const SettingPath &path, double val);
 	void WriteSettingRaw(const SettingPath &path, int32 val);
 	void WriteSettingRaw(const SettingPath &path, bool val);
+	void WriteSectionRaw(const mpt::winstring &section, const std::map<mpt::winstring, std::optional<mpt::winstring>> &keyvalues);
 	void RemoveSettingRaw(const SettingPath &path);
 	void RemoveSectionRaw(const mpt::ustring &section);
+private:
+	static mpt::winstring FormatValueAsIni(const SettingValue &value);
+	static SettingValue ParseValueFromIni(const mpt::winstring &str, const SettingValue &def);
 	static mpt::winstring GetSection(const SettingPath &path);
 	static mpt::winstring GetKey(const SettingPath &path);
 public:
-	IniFileSettingsBackend(const mpt::PathString &filename);
+	[[deprecated]] IniFileSettingsBackend(const mpt::PathString &filename);
+	IniFileSettingsBackend(const mpt::PathString &filename, CachePolicy cachePolicy);
 	~IniFileSettingsBackend() override;
 	void ConvertToUnicode(const mpt::ustring &backupTag = mpt::ustring());
 	virtual SettingValue ReadSetting(const SettingPath &path, const SettingValue &def) const override;
@@ -711,6 +735,9 @@
 	virtual void WriteSetting(const SettingPath &path, const SettingValue &val) override;
 	virtual void RemoveSetting(const SettingPath &path) override;
 	virtual void RemoveSection(const mpt::ustring &section) override;
+	virtual bool CanBatchedSettings() const override;
+	virtual void WriteMultipleSettings(const std::map<SettingPath, SettingValue> &settings) override;
+public:
 	const mpt::PathString& GetFilename() const { return filename; }
 };
 
@@ -717,7 +744,8 @@
 class IniFileSettingsContainer : private IniFileSettingsBackend, public SettingsContainer
 {
 public:
-	IniFileSettingsContainer(const mpt::PathString &filename);
+	[[deprecated]] IniFileSettingsContainer(const mpt::PathString &filename);
+	IniFileSettingsContainer(const mpt::PathString &filename, CachePolicy cachePolicy);
 	~IniFileSettingsContainer() override;
 };
 
Index: test/test.cpp
===================================================================
--- test/test.cpp	(revision 24343)
+++ test/test.cpp	(working copy)
@@ -2737,9 +2737,35 @@
 			VERIFY_EQUAL(inifile.ReadSetting(SettingPath{U_("S1"), U_("bar1")}, U_("empty")).as<mpt::ustring>(), U_("a"));
 			VERIFY_EQUAL(inifile.ReadSetting(SettingPath{U_("S1"), U_("bar2")}, U_("empty")).as<mpt::ustring>(), U_("empty"));
 		}
+		{
+			IniFileSettingsBackend inifile{filename};
+			const std::set<mpt::winstring> a = inifile.ReadSectionNamesRaw();
+			for(const mpt::winstring & s : a)
+			{
+				std::map<mpt::winstring, std::optional<mpt::winstring>> b = inifile.ReadNamedSectionRaw(s);
+			}
+		}
 		DeleteFile(mpt::support_long_path(filename.AsNative()).c_str());
 	}
 
+	{
+		const mpt::PathString filename = theApp.GetConfigPath() + P_("test.ini");
+		std::vector<std::byte> data;
+		for(std::size_t i = 0; i < 10; ++i)
+		{
+			data.push_back(mpt::byte_cast<std::byte>(static_cast<uint8>(i)));
+		}
+		{
+			IniFileSettingsBackend inifile{filename, CachePolicy::Cached};
+			inifile.WriteMultipleSettings({{SettingPath(U_("Test"), U_("Data")), data}});
+		}
+		{
+			IniFileSettingsBackend inifile{filename, CachePolicy::Cached};
+			VERIFY_EQUAL(inifile.ReadSetting(SettingPath(U_("Test"), U_("Data")), SettingValue(data)).as<std::vector<std::byte>>(), data);
+		}
+		DeleteFile(mpt::support_long_path(filename.AsNative()).c_str());
+	}
+
 #endif // MODPLUG_TRACKER
 
 }
manx

manx

2025-12-11 19:57

administrator   ~0006533

speedup-ini-settings-v6-wip.patch (20,093 bytes)   
Index: common/Profiler.h
===================================================================
--- common/Profiler.h	(revision 24593)
+++ common/Profiler.h	(working copy)
@@ -54,7 +54,7 @@
 
 #if defined(MODPLUG_TRACKER)
 
-//#define USE_PROFILER
+#define USE_PROFILER
 
 #endif
 
Index: mptrack/Mptrack.cpp
===================================================================
--- mptrack/Mptrack.cpp	(revision 24593)
+++ mptrack/Mptrack.cpp	(working copy)
@@ -932,7 +932,7 @@
 
 const mpt::PathString &CTrackApp::GetSongSettingsFilename() const
 {
-	return m_pSongSettingsIniFile->GetFilename();
+	return m_pSongSettingsIniFile->Filename();
 }
 
 
Index: mptrack/SettingsIni.cpp
===================================================================
--- mptrack/SettingsIni.cpp	(revision 24593)
+++ mptrack/SettingsIni.cpp	(working copy)
@@ -13,6 +13,7 @@
 
 #include "SettingsIni.h"
 
+#include "mpt/binary/hex.hpp"
 #include "mpt/io_file/fstream.hpp"
 #include "mpt/io_file/outputfile.hpp"
 #include "mpt/parse/parse.hpp"
@@ -22,11 +23,33 @@
 #include "../common/mptStringBuffer.h"
 
 #include <algorithm>
+#include <utility>
 
 
+
 OPENMPT_NAMESPACE_BEGIN
 
 
+
+WindowsIniFileBase::WindowsIniFileBase(mpt::PathString filename)
+	: filename(std::move(filename))
+	, file(filename)
+{
+	return;
+}
+
+const mpt::PathString &WindowsIniFileBase::Filename() const
+{
+	return filename;
+}
+
+mpt::PathString WindowsIniFileBase::GetFilename() const
+{
+	return filename;
+}
+
+
+
 std::vector<std::byte> ImmediateWindowsIniFileSettingsBackend::ReadSettingRaw(const SettingPath &path, const std::vector<std::byte> &def) const
 {
 	std::vector<std::byte> result = def;
@@ -126,12 +149,17 @@
 	::WritePrivateProfileString(mpt::ToWin(section).c_str(), NULL, NULL, filename.AsNative().c_str());
 }
 
+void CachedBatchedWindowsIniFileSettingsBackend::RemoveSectionRaw(const mpt::ustring &section)
+{
+	::WritePrivateProfileString(mpt::ToWin(section).c_str(), NULL, NULL, filename.AsNative().c_str());
+}
 
-mpt::winstring ImmediateWindowsIniFileSettingsBackend::GetSection(const SettingPath &path)
+
+mpt::winstring IniFileHelpers::GetSection(const SettingPath &path)
 {
 	return mpt::ToWin(path.GetSection());
 }
-mpt::winstring ImmediateWindowsIniFileSettingsBackend::GetKey(const SettingPath &path)
+mpt::winstring IniFileHelpers::GetKey(const SettingPath &path)
 {
 	return mpt::ToWin(path.GetKey());
 }
@@ -138,18 +166,36 @@
 
 
 
-ImmediateWindowsIniFileSettingsBackend::ImmediateWindowsIniFileSettingsBackend(const mpt::PathString &filename)
-	: filename(filename)
-	, file(filename)
+ImmediateWindowsIniFileSettingsBackend::ImmediateWindowsIniFileSettingsBackend(mpt::PathString filename)
+	: WindowsIniFileBase(std::move(filename))
 {
+	OPENMPT_PROFILE_FUNCTION(Profiler::Settings);
 	return;
 }
 
+CachedBatchedWindowsIniFileSettingsBackend::CachedBatchedWindowsIniFileSettingsBackend(mpt::PathString filename)
+	: WindowsIniFileBase(std::move(filename))
+{
+	OPENMPT_PROFILE_FUNCTION(Profiler::Settings);
+	ConvertToUnicode();
+	cache = ReadAllSectionsRaw();
+}
+
+void CachedBatchedWindowsIniFileSettingsBackend::InvalidateCache()
+{
+	cache = ReadAllSectionsRaw();
+}
+
 ImmediateWindowsIniFileSettingsBackend::~ImmediateWindowsIniFileSettingsBackend()
 {
 	return;
 }
 
+CachedBatchedWindowsIniFileSettingsBackend::~CachedBatchedWindowsIniFileSettingsBackend()
+{
+	return;
+}
+
 static void WriteFileUTF16LE(mpt::IO::atomic_shared_file_ref &file, const std::wstring &str)
 {
 	static_assert(sizeof(wchar_t) == 2);
@@ -160,7 +206,7 @@
 	file.write(mpt::byte_cast<mpt::const_byte_span>(mpt::as_span(inifile.str())));
 }
 
-void ImmediateWindowsIniFileSettingsBackend::ConvertToUnicode(const mpt::ustring &backupTag)
+void WindowsIniFileBase::ConvertToUnicode(const mpt::ustring &backupTag)
 {
 	// Force ini file to be encoded in UTF16.
 	// This causes WINAPI ini file functions to keep it in UTF16 encoding
@@ -209,6 +255,39 @@
 	}
 }
 
+SettingValue CachedBatchedWindowsIniFileSettingsBackend::ReadSetting(const SettingPath &path, const SettingValue &def) const
+{
+	OPENMPT_PROFILE_FUNCTION(Profiler::Settings);
+	const auto sectionit = cache.find(path.GetRefSection());
+	if(sectionit != cache.end())
+	{
+		if(sectionit->second.has_value())
+		{
+			const std::map<mpt::ustring, std::optional<mpt::ustring>> &section = sectionit->second.value();
+			const auto it = section.find(path.GetRefKey());
+			if(it != section.end())
+			{
+				if(it->second.has_value())
+				{
+					return ParseValueFromIni(it->second.value(), def);
+				} else
+				{
+					return def;
+				}
+			} else
+			{
+				return def;
+			}
+		} else
+		{
+			return def;
+		}
+	} else
+	{
+		return def;
+	}
+}
+
 void ImmediateWindowsIniFileSettingsBackend::WriteSetting(const SettingPath &path, const SettingValue &val)
 {
 	OPENMPT_PROFILE_FUNCTION(Profiler::Settings);
@@ -236,22 +315,271 @@
 	RemoveSectionRaw(section);
 }
 
+std::set<mpt::ustring> CachedBatchedWindowsIniFileSettingsBackend::ReadSectionNamesRaw() const
+{
+	std::set<mpt::ustring> result;
+	const std::vector<TCHAR> sectionsstr = [&]()
+		{
+			std::vector<TCHAR> buf;
+			buf.resize(mpt::IO::BUFFERSIZE_SMALL);
+			while(true)
+			{
+				DWORD bufused = ::GetPrivateProfileSectionNames(buf.data(), mpt::saturate_cast<DWORD>(buf.size()), filename.AsNative().c_str());
+				if(bufused >= (buf.size() - 2))
+				{
+					std::size_t newsize = mpt::exponential_grow(buf.size());
+					buf.resize(0);
+					buf.resize(newsize);
+					continue;
+				}
+				if(bufused >= 1)
+				{
+					bufused -= 1;  // terminating \0
+				}
+				buf.resize(bufused);
+				break;
+			};
+			return buf;
+		}();
+	const std::vector<mpt::winstring> sections = mpt::split(mpt::winstring(sectionsstr.data(), sectionsstr.size()), mpt::winstring(_T("\0"), 1));
+	for(const auto &section : sections)
+	{
+		result.insert(mpt::ToUnicode(section));
+	}
+	return result;
+}
 
+std::map<mpt::ustring, std::optional<mpt::ustring>> CachedBatchedWindowsIniFileSettingsBackend::ReadNamedSectionRaw(const mpt::ustring &section) const
+{
+	std::map<mpt::ustring, std::optional<mpt::ustring>> result;
+	const std::vector<TCHAR> keyvalsstr = [&]()
+		{
+			std::vector<TCHAR> buf;
+			buf.resize(mpt::IO::BUFFERSIZE_SMALL);
+			while(true)
+			{
+				DWORD bufused = ::GetPrivateProfileSection(mpt::ToWin(section).c_str(), buf.data(), mpt::saturate_cast<DWORD>(buf.size()), filename.AsNative().c_str());
+				if(bufused >= (buf.size() - 2))
+				{
+					std::size_t newsize = mpt::exponential_grow(buf.size());
+					buf.resize(0);
+					buf.resize(newsize);
+					continue;
+				}
+				if(bufused >= 1)
+				{
+					bufused -= 1;  // terminating \0
+				}
+				buf.resize(bufused);
+				break;
+			};
+			return buf;
+		}();
+	const std::vector<mpt::winstring> keyvals = mpt::split(mpt::winstring(keyvalsstr.data(), keyvalsstr.size()), mpt::winstring(_T("\0"), 1));
+	for(const auto &keyval : keyvals)
+	{
+		const auto equalpos = keyval.find(_T("="));
+		if(equalpos == mpt::winstring::npos)
+		{
+			continue;
+		}
+		if(equalpos == 0)
+		{
+			continue;
+		}
+		result.insert(std::make_pair(mpt::ToUnicode(keyval.substr(0, equalpos)), std::make_optional(mpt::ToUnicode(keyval.substr(equalpos + 1)))));
+	}
+	return result;
+}
 
+std::map<mpt::ustring, std::optional<std::map<mpt::ustring, std::optional<mpt::ustring>>>> CachedBatchedWindowsIniFileSettingsBackend::ReadAllSectionsRaw() const
+{
+	std::map<mpt::ustring, std::optional<std::map<mpt::ustring, std::optional<mpt::ustring>>>> result;
+	const std::set<mpt::ustring> sectionnames = ReadSectionNamesRaw();
+	for(const mpt::ustring &sectionname : sectionnames)
+	{
+		result.insert(std::make_pair(sectionname, std::make_optional(ReadNamedSectionRaw(sectionname))));
+	}
+	return result;
+}
 
+mpt::ustring IniFileHelpers::FormatValueAsIni(const SettingValue &value)
+{
+	switch(value.GetType())
+	{
+		case SettingTypeBool:
+			return mpt::ufmt::val(value.as<bool>());
+			break;
+		case SettingTypeInt:
+			return mpt::ufmt::val(value.as<int32>());
+			break;
+		case SettingTypeFloat:
+			return mpt::ufmt::val(value.as<double>());
+			break;
+		case SettingTypeString:
+			return value.as<mpt::ustring>();
+			break;
+		case SettingTypeBinary:
+			{
+				std::vector<std::byte> data = value.as<std::vector<std::byte>>();
+				uint8 checksum = 0;
+				for(const std::byte x : data) {
+					checksum += mpt::byte_cast<uint8>(x);
+				}
+				return mpt::encode_hex(mpt::as_span(data)) + mpt::ufmt::HEX0<2>(checksum);
+			}
+			break;
+		case SettingTypeNone:
+		default:
+			return mpt::ustring();
+			break;
+	}
+}
 
-IniFileSettingsContainer::IniFileSettingsContainer(const mpt::PathString &filename)
-	: ImmediateWindowsIniFileSettingsBackend(filename)
-	, SettingsContainer(this)
+
+SettingValue IniFileHelpers::ParseValueFromIni(const mpt::ustring &str, const SettingValue &def)
 {
+	switch(def.GetType())
+	{
+		case SettingTypeBool:
+			return SettingValue(mpt::parse_or<bool>(mpt::trim(str), def.as<bool>()), def.GetTypeTag());
+			break;
+		case SettingTypeInt:
+			return SettingValue(mpt::parse_or<int32>(mpt::trim(str), def.as<int32>()), def.GetTypeTag());
+			break;
+		case SettingTypeFloat:
+			return SettingValue(mpt::parse_or<double>(mpt::trim(str), def.as<double>()), def.GetTypeTag());
+			break;
+		case SettingTypeString:
+			return SettingValue(str, def.GetTypeTag());
+			break;
+		case SettingTypeBinary:
+			{
+				if((str.length() % 2) != 0)
+				{
+					return SettingValue(def.as<std::vector<std::byte>>(), def.GetTypeTag());
+				}
+				std::vector<std::byte> data = mpt::decode_hex(str);
+				if(data.size() < 1)
+				{
+					return SettingValue(def.as<std::vector<std::byte>>(), def.GetTypeTag());
+				}
+				const uint8 storedchecksum = mpt::byte_cast<uint8>(data[data.size() - 1]);
+				data.resize(data.size() - 1);
+				uint8 calculatedchecksum = 0;
+				for(const std::byte x : data) {
+					calculatedchecksum += mpt::byte_cast<uint8>(x);
+				}
+				if(calculatedchecksum != storedchecksum)
+				{
+					return SettingValue(def.as<std::vector<std::byte>>(), def.GetTypeTag());
+				}
+				return SettingValue(data, def.GetTypeTag());
+			}
+			break;
+		default:
+			return SettingValue();
+			break;
+	}
+}
+
+void CachedBatchedWindowsIniFileSettingsBackend::WriteSectionRaw(const mpt::ustring &section, const std::map<mpt::ustring, std::optional<mpt::ustring>> &keyvalues)
+{
+	OPENMPT_PROFILE_FUNCTION(Profiler::Settings);
+	mpt::winstring keyvals;
+	for(const auto &[key, val] : keyvalues)
+	{
+		if(val.has_value())
+		{
+			keyvals.append(mpt::ToWin(key));
+			keyvals.append(_T("="));
+			keyvals.append(mpt::ToWin(val.value()));
+			keyvals.append(mpt::winstring(_T("\0"), 1));
+		}
+	}
+	keyvals.append(mpt::winstring(_T("\0"), 1));
+	::WritePrivateProfileSection(mpt::ToWin(section).c_str(), keyvals.c_str(), filename.AsNative().c_str());
+}
+
+void CachedBatchedWindowsIniFileSettingsBackend::WriteRemovedSections(const std::set<mpt::ustring> &removeSections)
+{
+	OPENMPT_PROFILE_FUNCTION(Profiler::Settings);
+	for(const auto &section : removeSections)
+	{
+		const auto it = cache.find(section);
+		if(it != cache.end())
+		{
+			it->second = std::nullopt;
+		}
+	}
+	for(const auto &[section, keyvalues] : cache)
+	{
+		if(!keyvalues.has_value())
+		{
+			RemoveSectionRaw(section);
+		}
+	}
+	for(const auto &section : removeSections)
+	{
+		cache.erase(section);
+	}
+}
+
+void CachedBatchedWindowsIniFileSettingsBackend::WriteMultipleSettings(const std::map<SettingPath, std::optional<SettingValue>> &settings)
+{
+	OPENMPT_PROFILE_FUNCTION(Profiler::Settings);
+	std::map<mpt::ustring, std::map<SettingPath, std::optional<SettingValue>>> sectionssettings;
+	for(const auto &[path, value] : settings)
+	{
+		sectionssettings[path.GetRefSection()][path] = value;
+	}
+	for(const auto &[section, sectionsettings] : sectionssettings)
+	{
+		if(!cache[section].has_value())
+		{
+			cache[section].emplace();
+		}
+		std::map<mpt::ustring, std::optional<mpt::ustring>> &workingsectionsettings = cache[section].value();
+		for(const auto &[path, value] : sectionsettings)
+		{
+			if(value.has_value())
+			{
+				workingsectionsettings[path.GetRefKey()] = FormatValueAsIni(value.value());
+			} else
+			{
+				workingsectionsettings.erase(path.GetRefKey());
+			}
+		}
+		WriteSectionRaw(section, workingsectionsettings);
+	}
+}
+
+
+
+IniFileSettingsContainer::IniFileSettingsContainer(mpt::PathString filename)
+	: ImmediateWindowsIniFileSettingsBackend(std::move(filename))
+	, SettingsContainer(static_cast<ISettingsBackendFlavour<SettingsBatching::Single>*>(this))
+{
 	return;
 }
 
+BatchedIniFileSettingsContainer::BatchedIniFileSettingsContainer(mpt::PathString filename)
+	: CachedBatchedWindowsIniFileSettingsBackend(std::move(filename))
+	, SettingsContainer(static_cast<ISettingsBackendFlavour<SettingsBatching::Section>*>(this))
+{
+	return;
+}
+
 IniFileSettingsContainer::~IniFileSettingsContainer()
 {
 	return;
 }
 
+BatchedIniFileSettingsContainer::~BatchedIniFileSettingsContainer()
+{
+	return;
+}
 
 
+
 OPENMPT_NAMESPACE_END
Index: mptrack/SettingsIni.h
===================================================================
--- mptrack/SettingsIni.h	(revision 24593)
+++ mptrack/SettingsIni.h	(working copy)
@@ -13,12 +13,12 @@
 
 #include "openmpt/all/BuildSettings.hpp"
 
-
 #include "Settings.h"
 
 #include "mpt/io_file_atomic/atomic_file.hpp"
 
 #include <map>
+#include <optional>
 #include <set>
 #include <vector>
 
@@ -25,49 +25,112 @@
 
 OPENMPT_NAMESPACE_BEGIN
 
+
+// Version 1: ANSI
+// Version 2: UTF-16LE with BOM
+// Version 3: UTF8 with BOM (not supported by Windows)
+// Version 4: UTF8 without BOM and explicit version
+
+
+class IniFileHelpers
+{
+protected:
+	static mpt::ustring FormatValueAsIni(const SettingValue &value);
+	static SettingValue ParseValueFromIni(const mpt::ustring &str, const SettingValue &def);
+	static mpt::winstring GetSection(const SettingPath &path);
+	static mpt::winstring GetKey(const SettingPath &path);
+};
+
+
+class WindowsIniFileBase
+{
+protected:
+	const mpt::PathString filename;
+	mpt::IO::atomic_shared_file_ref file;
+protected:
+	WindowsIniFileBase(mpt::PathString filename);
+	~WindowsIniFileBase() = default;
+public:
+	void ConvertToUnicode(const mpt::ustring &backupTag = mpt::ustring());
+public:
+	const mpt::PathString &Filename() const;
+	mpt::PathString GetFilename() const;
+};
+
+
 class ImmediateWindowsIniFileSettingsBackend
 	: virtual public ISettingsBackend
 	, public ISettingsBackendFlavour<SettingsBatching::Single>
+	, public WindowsIniFileBase
+	, protected IniFileHelpers
 {
 private:
-	const mpt::PathString filename;
-	mpt::IO::atomic_shared_file_ref file;
-private:
 	std::vector<std::byte> ReadSettingRaw(const SettingPath &path, const std::vector<std::byte> &def) const;
 	mpt::ustring ReadSettingRaw(const SettingPath &path, const mpt::ustring &def) const;
 	double ReadSettingRaw(const SettingPath &path, double def) const;
 	int32 ReadSettingRaw(const SettingPath &path, int32 def) const;
 	bool ReadSettingRaw(const SettingPath &path, bool def) const;
+	void RemoveSectionRaw(const mpt::ustring &section);
+	void RemoveSettingRaw(const SettingPath &path);
 	void WriteSettingRaw(const SettingPath &path, const std::vector<std::byte> &val);
 	void WriteSettingRaw(const SettingPath &path, const mpt::ustring &val);
 	void WriteSettingRaw(const SettingPath &path, double val);
 	void WriteSettingRaw(const SettingPath &path, int32 val);
 	void WriteSettingRaw(const SettingPath &path, bool val);
-	void RemoveSettingRaw(const SettingPath &path);
-	void RemoveSectionRaw(const mpt::ustring &section);
-	static mpt::winstring GetSection(const SettingPath &path);
-	static mpt::winstring GetKey(const SettingPath &path);
 public:
-	ImmediateWindowsIniFileSettingsBackend(const mpt::PathString &filename);
+	ImmediateWindowsIniFileSettingsBackend(mpt::PathString filename);
 	~ImmediateWindowsIniFileSettingsBackend() override;
-	void ConvertToUnicode(const mpt::ustring &backupTag = mpt::ustring());
+public:
 	virtual SettingValue ReadSetting(const SettingPath &path, const SettingValue &def) const override;
+	virtual void RemoveSection(const mpt::ustring &section) override;
+	virtual void RemoveSetting(const SettingPath &path) override;
 	virtual void WriteSetting(const SettingPath &path, const SettingValue &val) override;
-	virtual void RemoveSetting(const SettingPath &path) override;
-	virtual void RemoveSection(const mpt::ustring &section) override;
-	const mpt::PathString& GetFilename() const { return filename; }
 };
 
-#ifndef IniFileSettingsBackend
+
+class CachedBatchedWindowsIniFileSettingsBackend
+	: virtual public ISettingsBackend
+	, public ISettingsBackendFlavour<SettingsBatching::Section>
+	, public WindowsIniFileBase
+	, protected IniFileHelpers
+{
+private:
+	std::map<mpt::ustring, std::optional<std::map<mpt::ustring, std::optional<mpt::ustring>>>> cache;
+private:
+	std::set<mpt::ustring> ReadSectionNamesRaw() const;
+	std::map<mpt::ustring, std::optional<mpt::ustring>> ReadNamedSectionRaw(const mpt::ustring &section) const;
+	std::map<mpt::ustring, std::optional<std::map<mpt::ustring, std::optional<mpt::ustring>>>> ReadAllSectionsRaw() const;
+	void RemoveSectionRaw(const mpt::ustring &section);
+	void WriteSectionRaw(const mpt::ustring &section, const std::map<mpt::ustring, std::optional<mpt::ustring>> &keyvalues);
+public:
+	CachedBatchedWindowsIniFileSettingsBackend(mpt::PathString filename);
+	~CachedBatchedWindowsIniFileSettingsBackend() override;
+public:
+	virtual void InvalidateCache() override;
+	virtual SettingValue ReadSetting(const SettingPath &path, const SettingValue &def) const override;
+	virtual void WriteRemovedSections(const std::set<mpt::ustring> &removeSections) override;
+	virtual void WriteMultipleSettings(const std::map<SettingPath, std::optional<SettingValue>> &settings) override;
+};
+
+
 #define IniFileSettingsBackend ImmediateWindowsIniFileSettingsBackend
-#endif
 
-class IniFileSettingsContainer : private IniFileSettingsBackend, public SettingsContainer
+
+class IniFileSettingsContainer : private ImmediateWindowsIniFileSettingsBackend, public SettingsContainer
 {
 public:
-	IniFileSettingsContainer(const mpt::PathString &filename);
+	IniFileSettingsContainer(mpt::PathString filename);
 	~IniFileSettingsContainer() override;
 };
 
 
+class BatchedIniFileSettingsContainer : private CachedBatchedWindowsIniFileSettingsBackend, public SettingsContainer
+{
+public:
+	BatchedIniFileSettingsContainer(mpt::PathString filename);
+	~BatchedIniFileSettingsContainer() override;
+};
+
+
+
 OPENMPT_NAMESPACE_END
Index: test/test.cpp
===================================================================
--- test/test.cpp	(revision 24593)
+++ test/test.cpp	(working copy)
@@ -2741,6 +2741,14 @@
 			VERIFY_EQUAL(inifile.ReadSetting(SettingPath{U_("S1"), U_("bar1")}, U_("empty")).as<mpt::ustring>(), U_("a"));
 			VERIFY_EQUAL(inifile.ReadSetting(SettingPath{U_("S1"), U_("bar2")}, U_("empty")).as<mpt::ustring>(), U_("empty"));
 		}
+		{
+			IniFileSettingsBackend inifile{filename};
+			const std::set<mpt::winstring> a = inifile.ReadSectionNamesRaw();
+			for(const mpt::winstring & s : a)
+			{
+				std::map<mpt::winstring, std::optional<mpt::winstring>> b = inifile.ReadNamedSectionRaw(s);
+			}
+		}
 		DeleteFile(mpt::support_long_path(filename.AsNative()).c_str());
 	}
 
@@ -2801,6 +2809,23 @@
 	}
 #endif
 
+	{
+		std::vector<std::byte> data;
+		for(std::size_t i = 0; i < 10; ++i)
+		{
+			data.push_back(mpt::byte_cast<std::byte>(static_cast<uint8>(i)));
+		}
+		{
+			IniFileSettingsBackend inifile{filename, CachePolicy::Cached};
+			inifile.WriteMultipleSettings({{SettingPath(U_("Test"), U_("Data")), data}});
+		}
+		{
+			IniFileSettingsBackend inifile{filename, CachePolicy::Cached};
+			VERIFY_EQUAL(inifile.ReadSetting(SettingPath(U_("Test"), U_("Data")), SettingValue(data)).as<std::vector<std::byte>>(), data);
+		}
+		DeleteFile(mpt::support_long_path(filename.AsNative()).c_str());
+	}
+
 #endif // MODPLUG_TRACKER
 
 }
manx

manx

2025-12-12 19:31

administrator   ~0006534

Full "read all settings, write all settings" cycle is down from ~1300ms to ~10ms with the attached patch.

speedup-ini-settings-v8-wip.patch (38,324 bytes)   
Index: mptrack/Mptrack.h
===================================================================
--- mptrack/Mptrack.h	(revision 24610)
+++ mptrack/Mptrack.h	(working copy)
@@ -35,6 +35,7 @@
 class TrackerSettings;
 class ImmediateWindowsIniFileSettingsBackend;
 class BatchedWindowsIniFileSettingsBackend;
+class CachedIniFileSettingsBackend;
 template <typename Backend> class FileSettingsContainer;
 #ifndef IniFileSettingsBackend
 #define IniFileSettingsBackend ImmediateWindowsIniFileSettingsBackend
Index: mptrack/SettingsIni.cpp
===================================================================
--- mptrack/SettingsIni.cpp	(revision 24610)
+++ mptrack/SettingsIni.cpp	(working copy)
@@ -13,25 +13,44 @@
 
 #include "SettingsIni.h"
 
+#include "mpt/base/integer.hpp"
 #include "mpt/binary/hex.hpp"
 #include "mpt/io_file/fstream.hpp"
 #include "mpt/io_file/outputfile.hpp"
 #include "mpt/parse/parse.hpp"
 
+#include "mpt/base/bit.hpp"
+#include "mpt/io_read/filecursor.hpp"
+#include "mpt/io_read/filecursor_memory.hpp"
+#include "mpt/io_read/filereader.hpp"
+#include "mpt/format/message_macros.hpp"
+#include "mpt/string/types.hpp"
+#include "mpt/string_transcode/transcode.hpp"
+
 #include "../common/misc_util.h"
 #include "../common/mptFileIO.h"
 #include "../common/mptStringBuffer.h"
 
 #include <algorithm>
+#include <list>
+#include <map>
+#include <optional>
+#include <set>
+#include <string>
+#include <string_view>
 #include <utility>
+#include <vector>
 
+#include <cstring>
 
 
+
+
 OPENMPT_NAMESPACE_BEGIN
 
 
 
-WindowsIniFileBase::WindowsIniFileBase(mpt::PathString filename_)
+IniFileBase::IniFileBase(mpt::PathString filename_)
 	: filename(std::move(filename_))
 	, file(filename)
 {
@@ -38,6 +57,24 @@
 	return;
 }
 
+const mpt::PathString &IniFileBase::Filename() const
+{
+	return filename;
+}
+
+mpt::PathString IniFileBase::GetFilename() const
+{
+	return filename;
+}
+
+
+
+WindowsIniFileBase::WindowsIniFileBase(mpt::PathString filename_)
+	: IniFileBase(std::move(filename_))
+{
+	return;
+}
+
 void WindowsIniFileBase::ConvertToUnicode(const mpt::ustring &backupTag)
 {
 	// Force ini file to be encoded in UTF16.
@@ -80,18 +117,86 @@
 #endif
 }
 
-const mpt::PathString &WindowsIniFileBase::Filename() const
+
+
+IniProbeResult IniFileHelpers::Probe(mpt::const_byte_span data)
 {
-	return filename;
+	IniProbeResult result;
+	const std::array<std::byte, 4> bom_utf32be = {mpt::byte_cast<std::byte>(uint8{0x00}), mpt::byte_cast<std::byte>(uint8{0x00}), mpt::byte_cast<std::byte>(uint8{0xfe}), mpt::byte_cast<std::byte>(uint8{0xff})};
+	const std::array<std::byte, 4> bom_utf32le = {mpt::byte_cast<std::byte>(uint8{0xff}), mpt::byte_cast<std::byte>(uint8{0xfe}), mpt::byte_cast<std::byte>(uint8{0x00}), mpt::byte_cast<std::byte>(uint8{0x00})};
+	const std::array<std::byte, 2> bom_utf16be = {mpt::byte_cast<std::byte>(uint8{0xfe}), mpt::byte_cast<std::byte>(uint8{0xff})};
+	const std::array<std::byte, 2> bom_utf16le = {mpt::byte_cast<std::byte>(uint8{0xff}), mpt::byte_cast<std::byte>(uint8{0xfe})};
+	const std::array<std::byte, 3> bom_utf8 = {mpt::byte_cast<std::byte>(uint8{0xEF}), mpt::byte_cast<std::byte>(uint8{0xBB}), mpt::byte_cast<std::byte>(uint8{0xBF})};
+	if((data.size() >= bom_utf32be.size()) && (std::memcmp(data.data(), bom_utf32be.data(), bom_utf32be.size()) == 0))
+	{
+		result.encoding = IniEncoding::UTF32BE;
+		result.data_offset = 4;
+	} else if((data.size() >= bom_utf32le.size()) && (std::memcmp(data.data(), bom_utf32le.data(), bom_utf32le.size()) == 0))
+	{
+		result.encoding = IniEncoding::UTF32LE;
+		result.data_offset = 4;
+	} else if((data.size() >= bom_utf16be.size()) && (std::memcmp(data.data(), bom_utf16be.data(), bom_utf16be.size()) == 0))
+	{
+		result.encoding = IniEncoding::UTF16BE;
+		result.data_offset = 2;
+	} else if((data.size() >= bom_utf16le.size()) && (std::memcmp(data.data(), bom_utf16le.data(), bom_utf16le.size()) == 0))
+	{
+		result.encoding = IniEncoding::UTF16LE;
+		result.data_offset = 2;
+	} else if((data.size() >= bom_utf8.size()) && (std::memcmp(data.data(), bom_utf8.data(), bom_utf8.size()) == 0))
+	{
+		result.encoding = IniEncoding::UTF8;
+		result.data_offset = 3;
+	} else
+	{
+		result.encoding = IniEncoding::ANSI;
+		result.data_offset = 0;
+	}
+
+	return result;
 }
 
-mpt::PathString WindowsIniFileBase::GetFilename() const
+std::list<std::pair<SettingPath, SettingValue>> IniFileHelpers::CreateIniHeader(IniVersion version)
 {
-	return filename;
+	std::list<std::pair<SettingPath, SettingValue>> result;
+	switch(version.major)
+	{
+		case 1:
+			result.emplace_back(std::make_pair(SettingPath{U_("!Type"), U_("!Format")}, U_("com.microsoft.fileformat.ini")));
+			result.emplace_back(std::make_pair(SettingPath{U_("!Type"), U_("!Encoding")}, U_("ANSI")));
+			result.emplace_back(std::make_pair(SettingPath{U_("!Type"), U_("!CaseSensitive")}, static_cast<int32>(0)));
+			result.emplace_back(std::make_pair(SettingPath{U_("!Type"), U_("!VersionMajor")}, static_cast<int32>(version.major)));
+			result.emplace_back(std::make_pair(SettingPath{U_("!Type"), U_("!VersionMinor")}, static_cast<int32>(version.minor)));
+			result.emplace_back(std::make_pair(SettingPath{U_("!Type"), U_("!VersionPatch")}, static_cast<int32>(version.patch)));
+			break;
+		case 2:
+			result.emplace_back(std::make_pair(SettingPath{U_("!Type"), U_("!Format")}, U_("com.microsoft.fileformat.ini")));
+			result.emplace_back(std::make_pair(SettingPath{U_("!Type"), U_("!Encoding")}, U_("UTF-16LE")));
+			result.emplace_back(std::make_pair(SettingPath{U_("!Type"), U_("!CaseSensitive")}, static_cast<int32>(0)));
+			result.emplace_back(std::make_pair(SettingPath{U_("!Type"), U_("!VersionMajor")}, static_cast<int32>(version.major)));
+			result.emplace_back(std::make_pair(SettingPath{U_("!Type"), U_("!VersionMinor")}, static_cast<int32>(version.minor)));
+			result.emplace_back(std::make_pair(SettingPath{U_("!Type"), U_("!VersionPatch")}, static_cast<int32>(version.patch)));
+			break;
+		case 3:
+			result.emplace_back(std::make_pair(SettingPath{U_("!Type"), U_("!Format")}, U_("org.winehq.fileformat.ini")));
+			result.emplace_back(std::make_pair(SettingPath{U_("!Type"), U_("!Encoding")}, U_("UTF-8")));
+			result.emplace_back(std::make_pair(SettingPath{U_("!Type"), U_("!CaseSensitive")}, static_cast<int32>(0)));
+			result.emplace_back(std::make_pair(SettingPath{U_("!Type"), U_("!VersionMajor")}, static_cast<int32>(version.major)));
+			result.emplace_back(std::make_pair(SettingPath{U_("!Type"), U_("!VersionMinor")}, static_cast<int32>(version.minor)));
+			result.emplace_back(std::make_pair(SettingPath{U_("!Type"), U_("!VersionPatch")}, static_cast<int32>(version.patch)));
+			break;
+		case 4:
+			result.emplace_back(std::make_pair(SettingPath{U_("!Type"), U_("!Format")}, U_("org.openmpt.fileformat.ini")));
+			result.emplace_back(std::make_pair(SettingPath{U_("!Type"), U_("!Encoding")}, U_("UTF-8")));
+			result.emplace_back(std::make_pair(SettingPath{U_("!Type"), U_("!CaseSensitive")}, static_cast<int32>(1)));
+			result.emplace_back(std::make_pair(SettingPath{U_("!Type"), U_("!VersionMajor")}, static_cast<int32>(version.major)));
+			result.emplace_back(std::make_pair(SettingPath{U_("!Type"), U_("!VersionMinor")}, static_cast<int32>(version.minor)));
+			result.emplace_back(std::make_pair(SettingPath{U_("!Type"), U_("!VersionPatch")}, static_cast<int32>(version.patch)));
+			break;
+	}
+	return result;
 }
 
-
-
 mpt::winstring IniFileHelpers::GetSection(const SettingPath &path)
 {
 	return mpt::ToWin(path.GetSection());
@@ -558,4 +663,695 @@
 
 
 
+static inline bool is_hex(mpt::uchar c)
+{
+	return (MPT_UCHAR('0') <= c && c <= MPT_UCHAR('9')) || (MPT_UCHAR('a') <= c && c <= MPT_UCHAR('f')) || (MPT_UCHAR('A') <= c && c <= MPT_UCHAR('F'));
+}
+
+static mpt::ustring escape(mpt::ustring_view text)
+{
+	mpt::ustring result;
+	const std::size_t len = text.length();
+	result.reserve(len);
+	for(std::size_t pos = 0; pos < len; ++pos)
+	{
+		const mpt::uchar c = text[pos];
+		if((c == MPT_UCHAR('^')) || (c == MPT_UCHAR(';')) || (c == MPT_UCHAR('[')) || (c == MPT_UCHAR(']')) || (c == MPT_UCHAR('=')) || (c == MPT_UCHAR('\"')))
+		{
+			result.push_back(MPT_UCHAR('^'));
+			result.push_back(c);
+		} else if(mpt::char_value(c) < 32)
+		{
+			result.push_back(MPT_UCHAR('^'));
+			result.push_back(MPT_UCHAR('x'));
+			result.append(mpt::ufmt::hex0<2>(mpt::char_value(c)));
+		} else if((0x80 <= mpt::char_value(c)) && (mpt::char_value(c) <= 0x9f))
+		{
+			result.push_back(MPT_UCHAR('^'));
+			result.push_back(MPT_UCHAR('x'));
+			result.append(mpt::ufmt::hex0<2>(mpt::char_value(c)));
+		} else
+		{
+			result.push_back(c);
+		}
+	}
+	return result;
+}
+
+static mpt::ustring unescape(mpt::ustring_view text)
+{
+	mpt::ustring result;
+	const std::size_t len = text.length();
+	result.reserve(len);
+	for(std::size_t pos = 0; pos < len; ++pos)
+	{
+		std::size_t rem = len - pos;
+		if(text[pos] == MPT_UCHAR('^'))
+		{
+			if((rem >= 2) && (text[pos+1] == MPT_UCHAR('^')))
+			{
+				result.push_back(MPT_UCHAR('^'));
+				pos += 1;
+			} else if((rem >= 2) && (text[pos+1] == MPT_UCHAR(';')))
+			{
+				result.push_back(MPT_UCHAR(';'));
+				pos += 1;
+			} else if((rem >= 2) && (text[pos+1] == MPT_UCHAR('[')))
+			{
+				result.push_back(MPT_UCHAR('['));
+				pos += 1;
+			} else if((rem >= 2) && (text[pos+1] == MPT_UCHAR(']')))
+			{
+				result.push_back(MPT_UCHAR(']'));
+				pos += 1;
+			} else if((rem >= 2) && (text[pos+1] == MPT_UCHAR('=')))
+			{
+				result.push_back(MPT_UCHAR('='));
+				pos += 1;
+			} else if((rem >= 2) && (text[pos+1] == MPT_UCHAR('\"')))
+			{
+				result.push_back(MPT_UCHAR('\"'));
+				pos += 1;
+			} else if((rem >= 4) && (text[pos+1] == MPT_UCHAR('x')) && is_hex(text[pos+2]) && is_hex(text[pos+3]))
+			{
+				result.push_back(static_cast<mpt::uchar>(mpt::parse_hex<uint8>(mpt::ustring{text.substr(pos + 2, 2)})));
+				pos += 3;
+			} else if((rem >= 6) && (text[pos+1] == MPT_UCHAR('u')) && is_hex(text[pos+2]) && is_hex(text[pos+3]) && is_hex(text[pos+4]) && is_hex(text[pos+5]))
+			{
+				result.append(mpt::transcode<mpt::ustring>(std::u16string(1, static_cast<char16_t>(mpt::parse_hex<uint16>(mpt::ustring{text.substr(pos + 2, 4)})))));
+				pos += 5;
+			} else if((rem >= 10) && (text[pos+1] == MPT_UCHAR('U')) && is_hex(text[pos+2]) && is_hex(text[pos+3]) && is_hex(text[pos+4]) && is_hex(text[pos+5]) && is_hex(text[pos+6]) && is_hex(text[pos+7]) && is_hex(text[pos+8]) && is_hex(text[pos+9]))
+			{
+				result.append(mpt::transcode<mpt::ustring>(std::u32string(1, static_cast<char32_t>(mpt::parse_hex<uint32>(mpt::ustring{text.substr(pos + 2, 8)})))));
+				pos += 9;
+			} else
+			{
+				result.push_back(text[pos]);
+			}
+		} else
+		{
+			result.push_back(text[pos]);
+		}
+	}
+	return result;
+}
+
+static inline std::size_t find_unescaped_first(mpt::ustring_view text, mpt::uchar c)
+{
+	const std::size_t len = text.length();
+	for(std::size_t pos = 0; pos < len; ++pos)
+	{
+		if(text[pos] == c)
+		{
+			return pos;
+		}
+		std::size_t rem = len - pos;
+		if(text[pos] == MPT_UCHAR('^'))
+		{
+			if((rem >= 2) && (text[pos+1] == MPT_UCHAR('^')))
+			{
+				pos += 1;
+			} else if((rem >= 2) && (text[pos+1] == MPT_UCHAR(';')))
+			{
+				pos += 1;
+			} else if((rem >= 2) && (text[pos+1] == MPT_UCHAR('[')))
+			{
+				pos += 1;
+			} else if((rem >= 2) && (text[pos+1] == MPT_UCHAR(']')))
+			{
+				pos += 1;
+			} else if((rem >= 2) && (text[pos+1] == MPT_UCHAR('=')))
+			{
+				pos += 1;
+			} else if((rem >= 2) && (text[pos+1] == MPT_UCHAR('\"')))
+			{
+				pos += 1;
+			} else if((rem >= 4) && (text[pos+1] == MPT_UCHAR('x')) && is_hex(text[pos+2]) && is_hex(text[pos+3]))
+			{
+				pos += 3;
+			} else if((rem >= 6) && (text[pos+1] == MPT_UCHAR('u')) && is_hex(text[pos+2]) && is_hex(text[pos+3]) && is_hex(text[pos+4]) && is_hex(text[pos+5]))
+			{
+				pos += 5;
+			} else if((rem >= 10) && (text[pos+1] == MPT_UCHAR('U')) && is_hex(text[pos+2]) && is_hex(text[pos+3]) && is_hex(text[pos+4]) && is_hex(text[pos+5]) && is_hex(text[pos+6]) && is_hex(text[pos+7]) && is_hex(text[pos+8]) && is_hex(text[pos+9]))
+			{
+				pos += 9;
+			}
+		}
+	}
+	return mpt::ustring_view::npos;
+}
+
+static inline std::size_t find_unescaped_last(mpt::ustring_view text, mpt::uchar c)
+{
+	std::size_t result = mpt::ustring_view::npos;
+	const std::size_t len = text.length();
+	for(std::size_t pos = 0; pos < len; ++pos)
+	{
+		if(text[pos] == c)
+		{
+			result = pos;
+		}
+		std::size_t rem = len - pos;
+		if(text[pos] == MPT_UCHAR('^'))
+		{
+			if((rem >= 2) && (text[pos+1] == MPT_UCHAR('^')))
+			{
+				pos += 1;
+			} else if((rem >= 2) && (text[pos+1] == MPT_UCHAR(';')))
+			{
+				pos += 1;
+			} else if((rem >= 2) && (text[pos+1] == MPT_UCHAR('[')))
+			{
+				pos += 1;
+			} else if((rem >= 2) && (text[pos+1] == MPT_UCHAR(']')))
+			{
+				pos += 1;
+			} else if((rem >= 2) && (text[pos+1] == MPT_UCHAR('=')))
+			{
+				pos += 1;
+			} else if((rem >= 2) && (text[pos+1] == MPT_UCHAR('\"')))
+			{
+				pos += 1;
+			} else if((rem >= 4) && (text[pos+1] == MPT_UCHAR('x')) && is_hex(text[pos+2]) && is_hex(text[pos+3]))
+			{
+				pos += 3;
+			} else if((rem >= 6) && (text[pos+1] == MPT_UCHAR('u')) && is_hex(text[pos+2]) && is_hex(text[pos+3]) && is_hex(text[pos+4]) && is_hex(text[pos+5]))
+			{
+				pos += 5;
+			} else if((rem >= 10) && (text[pos+1] == MPT_UCHAR('U')) && is_hex(text[pos+2]) && is_hex(text[pos+3]) && is_hex(text[pos+4]) && is_hex(text[pos+5]) && is_hex(text[pos+6]) && is_hex(text[pos+7]) && is_hex(text[pos+8]) && is_hex(text[pos+9]))
+			{
+				pos += 9;
+			}
+		}
+	}
+	return result;
+}
+
+static std::vector<mpt::ustring> split_any_line_ending(mpt::ustring_view text)
+{
+	std::vector<mpt::ustring> result;
+	const std::size_t len = text.length();
+	std::size_t beg = 0;
+	for(std::size_t pos = 0; pos < len; ++pos)
+	{
+		std::size_t rem = len - pos;
+		if((rem >= 2) && (text[pos+0] == MPT_UCHAR('\r')) && (text[pos+1] == MPT_UCHAR('\n')))
+		{
+			result.emplace_back(text.substr(beg, pos - beg));
+			pos += 2;
+			beg = pos;
+		} else if((rem >=1) && text[pos+0] == MPT_UCHAR('\n'))
+		{
+			result.emplace_back(text.substr(beg, pos - beg));
+			pos += 1;
+			beg = pos;
+		} else if((rem >=1) && text[pos+0] == MPT_UCHAR('\r'))
+		{
+			result.emplace_back(text.substr(beg, pos - beg));
+			pos += 1;
+			beg = pos;
+		}
+	}
+	if(!result.empty() || ((len - beg) > 0))
+	{
+		result.emplace_back(text.substr(beg, len - beg));
+	}
+	return result;
+}
+
+static inline mpt::ustring_view trim_whitespace(mpt::ustring_view text)
+{
+	mpt::ustring_view::size_type beg_pos = text.find_first_not_of(MPT_ULITERAL(" \t\r\n"));
+	if(beg_pos != mpt::ustring_view::npos)
+	{
+		text = text.substr(beg_pos);
+	} else
+	{
+		text = text.substr(text.length(), 0);
+	}
+	mpt::ustring_view::size_type end_pos = text.find_last_not_of(MPT_ULITERAL(" \t\r\n"));
+	if(end_pos != mpt::ustring_view::npos)
+	{
+		text = text.substr(0, end_pos + 1);
+	} else
+	{
+		text = text.substr(0, 0);
+	}
+	return text;
+}
+
+static inline mpt::ustring_view remove_quotes(mpt::ustring_view text)
+{
+	if(text.length() <= 1)
+	{
+		return text;
+	}
+	mpt::ustring_view::size_type beg_quote = find_unescaped_first(text, MPT_UCHAR('\"'));
+	mpt::ustring_view::size_type end_quote = find_unescaped_last(text, MPT_UCHAR('\"'));
+	if((beg_quote == 0) && (end_quote == (text.length() - 1)))
+	{
+		return text.substr(1, text.length() - 2);
+	}
+	return text;
+}
+
+void CachedIniFileSettingsBackend::ReadFileIntoCache()
+{
+	cache.clear();
+	const std::vector<std::byte> filedataraw = file.read();
+	IniProbeResult probe = Probe(mpt::as_span(filedataraw));
+	const mpt::span filedata = mpt::as_span(filedataraw).subspan(probe.data_offset);
+	mpt::ustring filetext;
+	switch(probe.encoding)
+	{
+		case IniEncoding::UTF32BE:
+			{
+				if((filedata.size() % sizeof(char32_t)) != 0)
+				{
+					MPT_LOG_GLOBAL(LogError, "Settings", MPT_UFORMAT_MESSAGE("{}: {}")(filename, U_("UTF32 encoding detected, but file size is not a multiple of 4.")));
+				}
+				MPT_MAYBE_CONSTANT_IF(mpt::endian_is_big())
+				{
+					std::u32string utf32data;
+					std::size_t count = filedata.size() / sizeof(char32_t);
+					utf32data.resize(count);
+					std::memcpy(utf32data.data(), filedata.data(), count * sizeof(char32_t));
+					filetext = mpt::transcode<mpt::ustring>(utf32data);
+				} else
+				{
+					auto fc = mpt::IO::make_FileCursor<mpt::PathString>(filedata);
+					std::u32string utf32data;
+					utf32data.reserve(fc.GetLength() / sizeof(char32_t));
+					while(!fc.EndOfFile())
+					{
+						utf32data.push_back(static_cast<char32_t>(mpt::IO::FileReader::ReadInt32BE(fc)));
+					}
+					filetext = mpt::transcode<mpt::ustring>(utf32data);
+				}
+			}
+			break;
+		case IniEncoding::UTF32LE:
+			{
+				if((filedata.size() % sizeof(char32_t)) != 0)
+				{
+					MPT_LOG_GLOBAL(LogError, "Settings", MPT_UFORMAT_MESSAGE("{}: {}")(filename, U_("UTF32 encoding detected, but file size is not a multiple of 4.")));
+				}
+				MPT_MAYBE_CONSTANT_IF(mpt::endian_is_little())
+				{
+					std::u32string utf32data;
+					std::size_t count = filedata.size() / sizeof(char32_t);
+					utf32data.resize(count);
+					std::memcpy(utf32data.data(), filedata.data(), count * sizeof(char32_t));
+					filetext = mpt::transcode<mpt::ustring>(utf32data);
+				} else
+				{
+					auto fc = mpt::IO::make_FileCursor<mpt::PathString>(filedata);
+					std::u32string utf32data;
+					utf32data.reserve(fc.GetLength() / sizeof(char32_t));
+					while(!fc.EndOfFile())
+					{
+						utf32data.push_back(static_cast<char32_t>(mpt::IO::FileReader::ReadInt32LE(fc)));
+					}
+					filetext = mpt::transcode<mpt::ustring>(utf32data);
+				}
+			}
+			break;
+		case IniEncoding::UTF16BE:
+			{
+				if((filedata.size() % sizeof(char16_t)) != 0)
+				{
+					MPT_LOG_GLOBAL(LogError, "Settings", MPT_UFORMAT_MESSAGE("{}: {}")(filename, U_("UTF16 encoding detected, but file size is not a multiple of 2.")));
+				}
+				MPT_MAYBE_CONSTANT_IF(mpt::endian_is_big())
+				{
+					std::u16string utf16data;
+					std::size_t count = filedata.size() / sizeof(char16_t);
+					utf16data.resize(count);
+					std::memcpy(utf16data.data(), filedata.data(), count * sizeof(char16_t));
+					filetext = mpt::transcode<mpt::ustring>(utf16data);
+				} else
+				{
+					auto fc = mpt::IO::make_FileCursor<mpt::PathString>(filedata);
+					std::u16string utf16data;
+					utf16data.reserve(fc.GetLength() / sizeof(char16_t));
+					while(!fc.EndOfFile())
+					{
+						utf16data.push_back(static_cast<char16_t>(mpt::IO::FileReader::ReadInt16BE(fc)));
+					}
+					filetext = mpt::transcode<mpt::ustring>(utf16data);
+				}
+			}
+			break;
+		case IniEncoding::UTF16LE:
+			{
+				if((filedata.size() % sizeof(char16_t)) != 0)
+				{
+					MPT_LOG_GLOBAL(LogError, "Settings", MPT_UFORMAT_MESSAGE("{}: {}")(filename, U_("UTF16 encoding detected, but file size is not a multiple of 2.")));
+				}
+				MPT_MAYBE_CONSTANT_IF(MPT_OS_WINDOWS)
+				{
+					std::wstring utf16data;
+					std::size_t count = filedata.size() / sizeof(wchar_t);
+					utf16data.resize(count);
+					std::memcpy(utf16data.data(), filedata.data(), count * sizeof(wchar_t));
+					filetext = mpt::transcode<mpt::ustring>(utf16data);
+				} else MPT_MAYBE_CONSTANT_IF(mpt::endian_is_little())
+				{
+					std::u16string utf16data;
+					std::size_t count = filedata.size() / sizeof(char16_t);
+					utf16data.resize(count);
+					std::memcpy(utf16data.data(), filedata.data(), count * sizeof(char16_t));
+					filetext = mpt::transcode<mpt::ustring>(utf16data);
+				} else
+				{
+					auto fc = mpt::IO::make_FileCursor<mpt::PathString>(filedata);
+					std::u16string utf16data;
+					utf16data.reserve(fc.GetLength() / sizeof(char16_t));
+					while(!fc.EndOfFile())
+					{
+						utf16data.push_back(static_cast<char16_t>(mpt::IO::FileReader::ReadInt16LE(fc)));
+					}
+					filetext = mpt::transcode<mpt::ustring>(utf16data);
+				}
+			}
+			break;
+		case IniEncoding::UTF8:
+			filetext = mpt::transcode<mpt::ustring>(mpt::common_encoding::utf8, mpt::buffer_cast<std::string>(filedata));
+			break;
+		case IniEncoding::ANSI:
+			filetext = mpt::transcode<mpt::ustring>(mpt::logical_encoding::locale, mpt::buffer_cast<std::string>(filedata));
+			break;
+	}
+#if 0
+	mpt::ustring lineending;
+	switch(probe.lineending)
+	{
+		case IniLineEnding::CRLF:
+			lineending = MPT_USTRING("\r\n");
+			break;
+		case IniLineEnding::LF:
+			lineending = MPT_USTRING("\n");
+			break;
+		case IniLineEnding::CR:
+			lineending = MPT_USTRING("\r");
+			break;
+	}
+	const std::vector<mpt::ustring> lines = mpt::split<mpt::ustring>(filetext, lineending);
+#else
+	const std::vector<mpt::ustring> lines = split_any_line_ending(filetext);
+#endif
+	mpt::ustring current_section;
+	mpt::ustring last_key;
+	bool empty_line_after_last_key = false;
+	std::vector<mpt::ustring> running_comments;
+	std::size_t line_number = 0;
+	bool last_line_empty = false;
+	auto store_previous_comments = [&]()
+		{
+			if(!empty_line_after_last_key && (running_comments.size() > 0))
+			{
+				comments[std::make_pair(current_section, last_key)].after = std::move(running_comments);
+				running_comments = {};
+			}
+		};
+	for(const auto &raw_line : lines)
+	{
+		line_number += 1;
+		last_line_empty = (raw_line.length() == 0);
+		mpt::ustring_view line = trim_whitespace(raw_line);
+		std::size_t line_len = line.length();
+		if(line_len == 0)
+		{ // empty line
+			store_previous_comments();
+			empty_line_after_last_key = true;
+			continue;
+		}
+		if(find_unescaped_first(line, MPT_UCHAR(';')) == 0)
+		{ // comment
+			running_comments.push_back(mpt::ustring{trim_whitespace(line.substr(1))});
+			continue;
+		}
+		if(find_unescaped_first(line, MPT_UCHAR('[')) == 0)
+		{ // section start
+			mpt::ustring_view::size_type opening_pos = 0;
+			mpt::ustring_view::size_type closing_pos = find_unescaped_first(line, MPT_UCHAR(']'));
+			if(closing_pos == mpt::ustring_view::npos)
+			{
+				store_previous_comments();
+				empty_line_after_last_key = true;
+				MPT_LOG_GLOBAL(LogError, "Settings", MPT_UFORMAT_MESSAGE("{}({}): No closing bracket in section header. ('{}').")(filename, line_number, raw_line));
+				continue;
+			}
+			mpt::ustring_view remainder = line.substr(closing_pos + 1);
+			if(trim_whitespace(remainder).length() > 0)
+			{
+				MPT_LOG_GLOBAL(LogWarning, "Settings", MPT_UFORMAT_MESSAGE("{}({}): Junk after section header. ('{}').")(filename, line_number, raw_line));
+			}
+			mpt::ustring_view escaped_section = trim_whitespace(line.substr(opening_pos + 1, closing_pos - opening_pos - 1));
+			mpt::ustring section = unescape(escaped_section);
+			if(cache.find(section) != cache.end())
+			{
+				store_previous_comments();
+				empty_line_after_last_key = true;
+				current_section.clear();
+				MPT_LOG_GLOBAL(LogWarning, "Settings", MPT_UFORMAT_MESSAGE("{}({}): Ignoring duplicate section '{}'. ('{}').")(filename, line_number, escape(section), raw_line));
+				continue;
+			}
+			MPT_LOG_GLOBAL(LogInformation, "Settings", MPT_UFORMAT_MESSAGE("{}({}): [{}]")(filename, line_number, escape(section)));
+			comments[std::make_pair(section, mpt::ustring{})].before = std::move(running_comments);
+			running_comments = {};
+			cache[section].emplace();
+			current_section = std::move(section);
+			last_key.clear();
+			empty_line_after_last_key = false;
+			continue;
+		}
+		mpt::ustring_view::size_type equal_pos = find_unescaped_first(line, MPT_UCHAR('='));
+		if(equal_pos == mpt::ustring_view::npos)
+		{ // syntax error
+			store_previous_comments();
+			empty_line_after_last_key = true;
+			MPT_LOG_GLOBAL(LogError, "Settings", MPT_UFORMAT_MESSAGE("{}({}): Syntax error: Invalid token. ('{}').")(filename, line_number, raw_line));
+			continue;
+		}
+		// key value
+		if(current_section.empty())
+		{
+			store_previous_comments();
+			empty_line_after_last_key = true;
+			MPT_LOG_GLOBAL(LogWarning, "Settings", MPT_UFORMAT_MESSAGE("{}({}): Ignoring key-value without section. ('{}').")(filename, line_number, raw_line));
+			continue;
+		}
+		mpt::ustring_view escaped_key = trim_whitespace(line.substr(0, equal_pos));
+		mpt::ustring_view escaped_val = remove_quotes(trim_whitespace(line.substr(equal_pos + 1)));
+		if(escaped_key.empty())
+		{
+			store_previous_comments();
+			empty_line_after_last_key = true;
+			MPT_LOG_GLOBAL(LogError, "Settings", MPT_UFORMAT_MESSAGE("{}({}): Syntax error: Empty key. ('{}').")(filename, line_number, raw_line));
+			continue;
+		}
+		mpt::ustring key = unescape(escaped_key);
+		mpt::ustring val = unescape(escaped_val);
+		if(cache[current_section].value().find(key) != cache[current_section].value().end())
+		{
+			store_previous_comments();
+			empty_line_after_last_key = true;
+			MPT_LOG_GLOBAL(LogWarning, "Settings", MPT_UFORMAT_MESSAGE("{}({}): Ignoring duplicate key '{}'. ('{}').")(filename, line_number, escape(key), raw_line));
+			continue;
+		}
+		MPT_LOG_GLOBAL(LogInformation, "Settings", MPT_UFORMAT_MESSAGE("{}({}): {}={}")(filename, line_number, escape(key), escape(val)));
+		comments[std::make_pair(current_section, key)].before = std::move(running_comments);
+		running_comments = {};
+		cache[current_section].value()[key] = std::make_optional(std::move(val));
+		last_key = std::move(key);
+		empty_line_after_last_key = false;
+	}
+	store_previous_comments();
+	if(!last_line_empty)
+	{
+		MPT_LOG_GLOBAL(LogNotification, "Settings", MPT_UFORMAT_MESSAGE("{}({}): {}")(filename, line_number, U_("No newline after last line.")));
+	}
+}
+
+void CachedIniFileSettingsBackend::MergeSettingsIntoCache(const std::set<mpt::ustring> &removeSections, const std::map<SettingPath, std::optional<SettingValue>> &settings)
+{
+	for(const auto &section : removeSections)
+	{
+		cache.erase(section);
+	}
+	std::map<mpt::ustring, std::map<SettingPath, std::optional<SettingValue>>> sectionssettings;
+	for(const auto &[path, value] : settings)
+	{
+		sectionssettings[path.GetRefSection()][path] = value;
+	}
+	for(const auto &[section, sectionsettings] : sectionssettings)
+	{
+		if(!cache[section].has_value())
+		{
+			cache[section].emplace();
+		}
+		std::map<mpt::ustring, std::optional<mpt::ustring>> &workingsectionsettings = cache[section].value();
+		for(const auto &[path, value] : sectionsettings)
+		{
+			if(value.has_value())
+			{
+				workingsectionsettings[path.GetRefKey()] = FormatValueAsIni(value.value());
+			} else
+			{
+				workingsectionsettings.erase(path.GetRefKey());
+			}
+		}
+	}
+}
+
+void CachedIniFileSettingsBackend::WriteCacheIntoFile()
+{
+	mpt::ustring filetext;
+	const std::list<std::pair<SettingPath, SettingValue>> header = CreateIniHeader(IniVersion{4, 0, 0});
+	for(const auto &path : header)
+	{
+		cache.erase(path.first.GetRefSection());
+	}
+	const mpt::ustring newline = MPT_USTRING("\r\n");
+	mpt::ustring last_section;
+	for(const auto &[path, value] : header)
+	{
+		if(path.GetRefSection() != last_section)
+		{
+			last_section = path.GetRefSection();
+			filetext += MPT_UFORMAT_MESSAGE("[{}]{}")(escape(path.GetRefSection()), newline);
+		}
+		filetext += MPT_UFORMAT_MESSAGE("{}={}{}")(escape(path.GetRefKey()), escape(FormatValueAsIni(value)), newline);
+	}
+	for(const auto &[section, sectionsettings] : cache)
+	{
+		filetext += newline;
+		auto sectioncomments = comments.find(std::make_pair(section, mpt::ustring{}));
+		if(sectioncomments != comments.end())
+		{
+			for(const auto &comment : sectioncomments->second.before)
+			{
+				filetext += MPT_UFORMAT_MESSAGE(";{}{}")(comment, newline);
+			}
+		}
+		filetext += MPT_UFORMAT_MESSAGE("[{}]{}")(escape(section), newline);
+		if(sectioncomments != comments.end())
+		{
+			for(const auto &comment : sectioncomments->second.after)
+			{
+				filetext += MPT_UFORMAT_MESSAGE(";{}{}")(comment, newline);
+			}
+			if(sectioncomments->second.after.size() > 0)
+			{
+				filetext += newline;
+			}
+		}
+		if(sectionsettings.has_value())
+		{
+			for(const auto &[key, value] : sectionsettings.value())
+			{
+				if(value.has_value())
+				{
+					auto keycomments = comments.find(std::make_pair(section, key));
+					if(keycomments != comments.end())
+					{
+						for(const auto &comment : keycomments->second.before)
+						{
+							filetext += MPT_UFORMAT_MESSAGE(";{}{}")(comment, newline);
+						}
+					}
+					const mpt::ustring escaped_value = escape(value.value());
+					if(escaped_value != trim_whitespace(escaped_value))
+					{
+						filetext += MPT_UFORMAT_MESSAGE("{}=\"{}\"{}")(escape(key), escaped_value, newline);
+					} else
+					{
+						filetext += MPT_UFORMAT_MESSAGE("{}={}{}")(escape(key), escaped_value, newline);
+					}
+					if(keycomments != comments.end())
+					{
+						for(const auto &comment : keycomments->second.after)
+						{
+							filetext += MPT_UFORMAT_MESSAGE(";{}{}")(comment, newline);
+						}
+						if(keycomments->second.after.size() > 0)
+						{
+							filetext += newline;
+						}
+					}
+				}
+			}
+		}
+	}
+	const std::array<std::byte, 3> utf8_bom = {mpt::byte_cast<std::byte>(uint8{0xef}), mpt::byte_cast<std::byte>(uint8{0xbb}), mpt::byte_cast<std::byte>(uint8{0xbf})};
+	std::vector<std::byte> filedata;
+	mpt::append(filedata, utf8_bom);
+	mpt::append(filedata, mpt::byte_cast<mpt::const_byte_span>(mpt::as_span(mpt::transcode<std::string>(mpt::common_encoding::utf8, filetext))));
+	file.write(filedata);
+}
+
+void CachedIniFileSettingsBackend::InvalidateCache()
+{
+	OPENMPT_PROFILE_FUNCTION(Profiler::Settings);
+	std::lock_guard l(file);
+	ReadFileIntoCache();
+}
+
+SettingValue CachedIniFileSettingsBackend::ReadSetting(const SettingPath &path, const SettingValue &def) const
+{
+	OPENMPT_PROFILE_FUNCTION(Profiler::Settings);
+	const auto sectionit = cache.find(path.GetRefSection());
+	if(sectionit != cache.end())
+	{
+		if(sectionit->second.has_value())
+		{
+			const std::map<mpt::ustring, std::optional<mpt::ustring>> &section = sectionit->second.value();
+			const auto it = section.find(path.GetRefKey());
+			if(it != section.end())
+			{
+				if(it->second.has_value())
+				{
+					return ParseValueFromIni(it->second.value(), def);
+				} else
+				{
+					return def;
+				}
+			} else
+			{
+				return def;
+			}
+		} else
+		{
+			return def;
+		}
+	} else
+	{
+		return def;
+	}
+}
+
+void CachedIniFileSettingsBackend::WriteAllSettings(const std::set<mpt::ustring> &removeSections, const std::map<SettingPath, std::optional<SettingValue>> &settings)
+{
+	OPENMPT_PROFILE_FUNCTION(Profiler::Settings);
+	std::lock_guard l(file);
+	ReadFileIntoCache();
+	MergeSettingsIntoCache(removeSections, settings);
+	WriteCacheIntoFile();
+}
+
+CachedIniFileSettingsBackend::CachedIniFileSettingsBackend(mpt::PathString filename_)
+	: IniFileBase(std::move(filename_))
+{
+	OPENMPT_PROFILE_FUNCTION(Profiler::Settings);
+	std::lock_guard l(file);
+	ReadFileIntoCache();
+}
+
+CachedIniFileSettingsBackend::~CachedIniFileSettingsBackend()
+{
+	return;
+}
+
+
 OPENMPT_NAMESPACE_END
Index: mptrack/SettingsIni.h
===================================================================
--- mptrack/SettingsIni.h	(revision 24610)
+++ mptrack/SettingsIni.h	(working copy)
@@ -15,11 +15,15 @@
 
 #include "Settings.h"
 
+#include "mpt/base/integer.hpp"
+#include "mpt/string/types.hpp"
 #include "mpt/io_file_atomic/atomic_file.hpp"
 
+#include <list>
 #include <map>
 #include <optional>
 #include <set>
+#include <utility>
 #include <vector>
 
 
@@ -26,9 +30,49 @@
 OPENMPT_NAMESPACE_BEGIN
 
 
+struct IniVersion
+{
+	uint8 major = 0;
+	uint8 minor = 0;
+	uint8 patch = 0;
+};
+
+
+enum class IniEncoding
+{
+	UTF32BE,
+	UTF32LE,
+	UTF16BE,
+	UTF16LE,
+	UTF8,
+	ANSI,
+};
+
+
+enum class IniLineEnding
+{
+	CRLF,
+	LF,
+	CR,
+};
+
+
+struct IniProbeResult
+{
+	IniVersion version;
+	IniEncoding encoding = IniEncoding::ANSI;
+	IniLineEnding lineending = IniLineEnding::CRLF;
+	bool case_sensitive = false;
+	bool has_bom = false;
+	mpt::IO::Offset data_offset = 0;
+};
+
+
 class IniFileHelpers
 {
 protected:
+	static IniProbeResult Probe(mpt::const_byte_span data);
+	static std::list<std::pair<SettingPath, SettingValue>> CreateIniHeader(IniVersion version);
 	static mpt::winstring GetSection(const SettingPath &path);
 	static mpt::winstring GetKey(const SettingPath &path);
 	static mpt::ustring FormatValueAsIni(const SettingValue &value);
@@ -36,19 +80,34 @@
 };
 
 
-class WindowsIniFileBase
+// Version 1: ANSI
+// Version 2: UTF-16LE with BOM
+// Version 3: UTF8 with BOM (not supported by Windows)
+// Version 4: UTF8 without BOM and explicit version
+
+
+class IniFileBase
 {
 protected:
 	const mpt::PathString filename;
 	mpt::IO::atomic_shared_file_ref file;
 protected:
+	IniFileBase(mpt::PathString filename_);
+	~IniFileBase() = default;
+public:
+	const mpt::PathString &Filename() const;
+	mpt::PathString GetFilename() const;
+};
+
+
+class WindowsIniFileBase
+	: public IniFileBase
+{
+protected:
 	WindowsIniFileBase(mpt::PathString filename_);
 	~WindowsIniFileBase() = default;
 public:
 	void ConvertToUnicode(const mpt::ustring &backupTag = mpt::ustring());
-public:
-	const mpt::PathString &Filename() const;
-	mpt::PathString GetFilename() const;
 };
 
 
@@ -105,6 +164,36 @@
 };
 
 
+class CachedIniFileSettingsBackend
+	: public ISettingsBackend<SettingsBatching::All>
+	, public IniFileBase
+	, protected IniFileHelpers
+{
+private:
+	struct comments
+	{
+		std::vector<mpt::ustring> before;
+		std::vector<mpt::ustring> after;
+	};
+private:
+	std::map<mpt::ustring, std::optional<std::map<mpt::ustring, std::optional<mpt::ustring>>>> cache;
+	std::map<std::pair<mpt::ustring, mpt::ustring>, comments> comments;
+private:
+
+private:
+	void ReadFileIntoCache();
+	void MergeSettingsIntoCache(const std::set<mpt::ustring> &removeSections, const std::map<SettingPath, std::optional<SettingValue>> &settings);
+	void WriteCacheIntoFile();
+public:
+	CachedIniFileSettingsBackend(mpt::PathString filename_);
+	~CachedIniFileSettingsBackend() override;
+public:
+	virtual void InvalidateCache() override;
+	virtual SettingValue ReadSetting(const SettingPath &path, const SettingValue &def) const override;
+	virtual void WriteAllSettings(const std::set<mpt::ustring> &removeSections, const std::map<SettingPath, std::optional<SettingValue>> &settings) override;
+};
+
+
 #ifndef IniFileSettingsBackend
 #define IniFileSettingsBackend ImmediateWindowsIniFileSettingsBackend
 #endif
Index: test/test.cpp
===================================================================
--- test/test.cpp	(revision 24610)
+++ test/test.cpp	(working copy)
@@ -3031,6 +3031,17 @@
 
 #ifdef MODPLUG_TRACKER
 
+	{
+		const mpt::PathString in = P_("D:\\stuff\\in.ini");
+		const mpt::PathString out = P_("D:\\stuff\\out.ini");
+		CopyFile(in.AsNative().c_str(), out.AsNative().c_str(), FALSE);
+		{
+			FileSettingsContainer<CachedIniFileSettingsBackend> ini{out};
+			ini.Write<mpt::ustring>(U_("Dummy"), U_("Value1"), U_("\r\n"));
+			ini.Write<mpt::ustring>(U_("Dummy"), U_("Value2"), U_(" \r\n "));
+		}
+	}
+
 	VERIFY_EQUAL(SettingPath(U_("a"),U_("b")) < SettingPath(U_("a"),U_("c")), true);
 	VERIFY_EQUAL(!(SettingPath(U_("c"),U_("b")) < SettingPath(U_("a"),U_("c"))), true);
 
@@ -3042,6 +3053,8 @@
 
 	TestIniSettingsBackendRead<BatchedWindowsIniFileSettingsBackend>(filename);
 
+	TestIniSettingsBackendRead<CachedIniFileSettingsBackend>(filename);
+
 	{
 		IniFileSettingsContainer conf{filename};
 		int32 foobar = conf.Read(U_("Test"), U_("bar"), 23);
@@ -3134,7 +3147,7 @@
 
 	{
 		{
-			IniFileSettingsContainer conf{filename};
+			FileSettingsContainer<ImmediateWindowsIniFileSettingsBackend> conf{filename};
 			conf.Write<mpt::ustring>(SettingPath(MPT_USTRING("Test"), MPT_USTRING("spacespacecharspacespace")), spacespacecharspacespace);
 			conf.Write<mpt::ustring>(SettingPath(MPT_USTRING("Test"), MPT_USTRING("threespaces")), threespaces);
 			conf.Write<mpt::ustring>(SettingPath(MPT_USTRING("Test"), MPT_USTRING("tab")), tab);
@@ -3143,7 +3156,7 @@
 			conf.Write<mpt::ustring>(SettingPath(MPT_USTRING("Test"), MPT_USTRING("cc0")), cc0);
 		}
 		{
-			IniFileSettingsContainer conf{filename};
+			FileSettingsContainer<ImmediateWindowsIniFileSettingsBackend> conf{filename};
 			//VERIFY_EQUAL(conf.Read<mpt::ustring>(SettingPath(MPT_USTRING("Test"), MPT_USTRING("spacespacecharspacespace"))), spacespacecharspacespace);
 			//VERIFY_EQUAL(conf.Read<mpt::ustring>(SettingPath(MPT_USTRING("Test"), MPT_USTRING("threespaces"))), threespaces);
 			//VERIFY_EQUAL(conf.Read<mpt::ustring>(SettingPath(MPT_USTRING("Test"), MPT_USTRING("tab"))), tab);
@@ -3160,6 +3173,28 @@
 		DeleteFile(mpt::support_long_path(filename.AsNative()).c_str());
 	}
 
+	{
+		{
+			FileSettingsContainer<CachedIniFileSettingsBackend> conf{filename};
+			conf.Write<mpt::ustring>(SettingPath(MPT_USTRING("Test"), MPT_USTRING("spacespacecharspacespace")), spacespacecharspacespace);
+			conf.Write<mpt::ustring>(SettingPath(MPT_USTRING("Test"), MPT_USTRING("threespaces")), threespaces);
+			conf.Write<mpt::ustring>(SettingPath(MPT_USTRING("Test"), MPT_USTRING("tab")), tab);
+			conf.Write<mpt::ustring>(SettingPath(MPT_USTRING("Test"), MPT_USTRING("tokens")), tokens);
+			conf.Write<mpt::ustring>(SettingPath(MPT_USTRING("Test"), MPT_USTRING("xcrlfy")), xcrlfy);
+			conf.Write<mpt::ustring>(SettingPath(MPT_USTRING("Test"), MPT_USTRING("cc0")), cc0);
+		}
+		{
+			FileSettingsContainer<CachedIniFileSettingsBackend> conf{filename};
+			VERIFY_EQUAL(conf.Read<mpt::ustring>(SettingPath(MPT_USTRING("Test"), MPT_USTRING("spacespacecharspacespace"))), spacespacecharspacespace);
+			VERIFY_EQUAL(conf.Read<mpt::ustring>(SettingPath(MPT_USTRING("Test"), MPT_USTRING("threespaces"))), threespaces);
+			VERIFY_EQUAL(conf.Read<mpt::ustring>(SettingPath(MPT_USTRING("Test"), MPT_USTRING("tab"))), tab);
+			VERIFY_EQUAL(conf.Read<mpt::ustring>(SettingPath(MPT_USTRING("Test"), MPT_USTRING("tokens"))), tokens);
+			VERIFY_EQUAL(conf.Read<mpt::ustring>(SettingPath(MPT_USTRING("Test"), MPT_USTRING("xcrlfy"))), xcrlfy);
+			VERIFY_EQUAL(conf.Read<mpt::ustring>(SettingPath(MPT_USTRING("Test"), MPT_USTRING("cc0"))), cc0);
+		}
+		DeleteFile(mpt::support_long_path(filename.AsNative()).c_str());
+	}
+
 #endif // MODPLUG_TRACKER
 
 }
manx

manx

2025-12-12 19:53

administrator   ~0006535

The patch does not replace the old INI backend, it just adds the new, but does not connect it yet.

  • Full bug-compatibility with WinAPI when reading. Verified via unit tests.
  • Sections and Keys are written alphabetically.
  • Comments retained and written as well, attached to the matching Section or Key.
  • Strings with whitespace padding are written in quotes, compatible with WinAPI.
  • The escape character is ^. This is an unusual choice but has the clear advantage of very low collision risk with existing INI files. Invalid escape sequences are silently read literally. No additional effort is made to read old INI files without the unescaping. We escape all syntactically significant characters: ^, ;, [, ], =, as well as all C0 and C1 control codes. 1, 2, 4 byte Unicode codepoints can be escaped via ^x00, ^u0000, ^U00000000 respectively.
  • The encoding is auto-detected via BOM, and line ending is irrelevant for parsing.
  • INI files are written with a small header in a special [!Type] INI section, for better forward compatibility.
  • INI files are written in UTF-8 encoding with BOM. Wine can parse this, WinAPI cannot. WinAPI will read these as ANSI encoding. We could change this to always writing UTF-16LE with BOM, which would further increase backwards compatibility.
  • The parser logs every syntax error, when logging is enabled.
  • Setting are initially loaded by parsing the whole file at once, after that only a std::map lookup happens for individual settings.
  • Settings are saved by:

    1. re-reading the settings from disk (to support the case where another instance has saved settings)
    2. merging the dirty settings into the just re-read settings.
    3. saving the merged settings

    These 3 steps happen atomically, guarded by a file lock.

manx

manx

2025-12-12 19:57

administrator   ~0006536

  • Settings are now case-sensitive. This also applies to the section-batched backend. This should not really matter unless someone edits the INI by hand and corrupts the casing.

Issue History

Date Modified Username Field Change
2022-01-30 17:48 Saga Musix New Issue
2022-01-30 17:51 Saga Musix Target Version => OpenMPT 1.31.01.00 / libopenmpt 0.7.0 (upgrade first)
2022-10-22 13:45 manx Assigned To => manx
2022-10-22 13:45 manx Status new => acknowledged
2023-04-10 08:24 manx Target Version OpenMPT 1.31.01.00 / libopenmpt 0.7.0 (upgrade first) => OpenMPT 1.32.01.00 / libopenmpt 0.8.0 (upgrade first)
2024-10-26 18:05 manx Target Version OpenMPT 1.32.01.00 / libopenmpt 0.8.0 (upgrade first) => OpenMPT 1.33 / libopenmpt 0.9 (goals)
2025-08-16 16:27 manx Note Added: 0006447
2025-08-30 13:00 Saga Musix Note Added: 0006454
2025-10-18 17:35 manx Note Added: 0006514
2025-10-18 17:35 manx File Added: speedup-ini-settings-v1.patch
2025-10-19 18:35 Saga Musix Note Added: 0006515
2025-10-25 18:52 manx Note Added: 0006517
2025-10-25 18:52 manx File Added: speedup-ini-settings-v3-wip.patch
2025-12-11 19:57 manx Note Added: 0006533
2025-12-11 19:57 manx File Added: speedup-ini-settings-v6-wip.patch
2025-12-11 20:12 manx Status acknowledged => assigned
2025-12-12 19:31 manx Note Added: 0006534
2025-12-12 19:31 manx File Added: speedup-ini-settings-v8-wip.patch
2025-12-12 19:53 manx Note Added: 0006535
2025-12-12 19:57 manx Note Added: 0006536