LittleFS is terribly slow at saving settings (partial file rewrite)

[TD-er]

As also mentioned before in #5932
I have now changed the code in ESPEasy to be able to switch between SPIFFS and LittleFS using a single define (or simply changing the PIO env label). See: letscontrolit/ESPEasy#3052
So I will now be testing LittleFS a lot on my test environments.

The read speed of Little FS is a lot (!!) faster than SPIFFS, but the write speed of our settings is awfully slow.
Saving settings:

• 67 msec on SPIFFS
• 2285 msec on LittleFS.

During write the node is completely blocked, so this is unacceptable slow.
The settings file we have is way too large to keep in memory.
I only update a part in the file (always start at multiple of 1k block in the file).

I would like to know what would be the best approach for fixing this as SPIFFS was unusable slow for larger file systems (> 1 MB), but this is really unusable slow for writes. (updates in a file)

[earlephilhower]

This is a known "design choice" of LittleFS: littlefs-project/littlefs#244

The simplest thing I can suggest is to split the file into objects. Make a subdir of the name of the file, and record individual objects in their own files in that subdir.

OTW, I suggest you open a new issue on upstream https://github.com/ARMmbed/littlefs/ to let them know that this design choice kind of sucks for a large class of applications...

[TD-er]

I've been thinking about this a bit more.
What I can do is split the parts of my settings file into separate pieces. But that makes backup or share of the file a lot more difficult.
So then I should make some special handler to combine them into a single file when it is requested as a single file.
But to me it seems like this should be part of the file system's task.
The file system should then have the option to set a flag to a file to mark it as "random write" or perhaps a more appropriate described flag.
This flag then under water stores the file in blocks of 4 or 8 k (for example) in a special directory.
When accessed it should then translate the calls for read/write into those separate file accesses.
Only problem then may be that you are using a lot more file handlers.

Is that some limit you're aware of?

[earlephilhower]

File handles are just extra memory. There's no hard-coded limit in littlefs or the 8266 File class. However, what you're discussing seems like basically making a new filesystem with all the corner cases/etc. that any other would need. Just in this case, instead of writing to flash, you're writing to a 4K file in LittleFS (which then writes to flash).

If it's just export/import, what about a basic streaming format like tar? Or simply concatenate the blobs together on export and then break them up on import?

[devyte]

There are no other known alternatives: it's either spiffs or littlefs. Spiffs upstream went stale, so we deprecated that for now. I suppose somebody could pick up spiffs maintenance, but it's been three years now.
If anybody knows an alternative fs that is reasonably fast in all cases, is safe against power cuts, is compatible with our license model, and has wear leveling, I'm all ears. Or eyes, as it were.
Closing, because there's nothing we can do about it at this time.

[TD-er]

What term do you plan for the deprecated state of SPIFFS and when is it no longer available in this library?

As mentioned, LittleFS is in our use case not a good replacement for SPIFFS, so I do need to dig into it to make our file handling fit LittleFS.

[devyte]

Per internal discussions, spiffs won't get pulled any time soon. The earliest, and this likely won't happen given the limitation reported here, would be v3.
You have time.

[TD-er]

Thanks.
I will of course try to make ESPEasy and LittleFS get along.
I'm not sure what side of it will have to change :)