cannot encode <Cyrillic-names> to local encoding 2252

[voroyam]

User in central reported this

https://central.owncloud.org/t/some-folders-with-cyrillic-names-are-ignored/16196
Client seems to have a problem with Cyrillic characters. Error message: "cannot encode За принтиране to local encoding 2252"

Client version 2.5

oC version 10.0.10

Platform win 7 pro

From his post:

For several weeks my desktop client ignores the main folders with cyrillic names and all they contain. This issue is only for one of the users. Other two have no problems.
Interestingly some folders, again with cyrillic names, were synced after I renamed the folder they were in.

Expected behaviour

Sync all folders and files within to the server and from the server

Actual behaviour

The client shows the folder as ignored with issue: "The filename cannot be encoded on your system."

Steps to reproduce

1. Rename folder ..\ownCloud\Представяне to "Presentation"
2. All files in it are uploaded
3. Rename again to "Представяне"
4. Client logs "Folder Presentation - deleted"
5. In not synced tab there is Issue: "The filename cannot be encoded on your system."

Server configuration

Shared hosting
Operating system: Linux

Web server: Apache 2.4.27

Database: mySQL

Server PHP version: 5.3.24
For ownCloud: 5.6

ownCloud version: latest - 10.0.10.4

Storage backend (external storage): -

Client configuration

Client version: 2.5.0

Operating system: Windows 7 profesional

OS language: english

Installation path of client: C:\Program Files (x86)\ownCloud

Logs

Client log output shows a lot of similar lines

10-12 11:00:09:037 [ info sync.csync.updater ]: cannot encode За принтиране to local encoding 2252

1. Client logfile: Output of owncloud --logwindow or owncloud --logfile log.txt
   (On Windows using cmd.exe, you might need to first cd into the ownCloud directory)
   (See also http://doc.owncloud.org/desktop/2.2/troubleshooting.html#client-logfile )

I hope this is enough to understand the problem.

[ogoffart]

This happens because the system locale codec is Windows-1252 which cannot encode cyrilic character. At least that's what Qt tells us.
Normally, most modern system are using UTF-8.
But perhaps this user configured his system to use Windows-1252 instead of UTF-8 ?

On Linux this is important since the file system is using that locale. On Windows however, the file system is always UCS-2, to my knowledge. And so we probably can encode everything

@ckamm: what do you think? Should we remove the "canEncode" test in _csync_detect_update on windows?

[ckamm]

@ogoffart It does indeed sound like we should not use the local encoding for this check on Windows. Is windows ucs-2 or utf-16? If it's ucs-2 keeping the canEncode() check would make sense to reject the 3 and 4 byte code points.

[ckamm]

According to wikipedia it migrated to utf-16 with win2k, so no check needed.

[ckamm]

Though it seems to be fully correct we might need to be wary of unpaired surrogates when reading existing paths (rust-lang/rust#12056). I'm not sure how the Qt filesystem abstraction handles these cases when converting to QString.

[ogoffart]

@ckamm: That's not the same thing, that would be taken care of by csync_vio_local_readdir. On Linux we indeed handle file that cannot be converted to utf-8, but we don't seem to do it on windows. But that's another topic.

[amc2002]

Same thing here with Chinese characters in 2.5.0 build 10560. When my boss went back to the previous version, sync issues were no longer an issue.

I tried a test on my own client (Windows 10), and I had the same issue:
10-23 11:29:12:178 [ warning sync.propagator ]: Could not complete propagation of "发送时.txt" by OCC::PropagateIgnoreJob(0x7f34e08) with status 6 and error: "The filename cannot be encoded on your file system."

Will the proposed fix also take care of this issue? Thanks.

[ogoffart]

@amc2002 It should, please try the 2.5.1 daily build.

[guruz]

@amc2002 https://download.owncloud.com/desktop/daily/ownCloud-2.5.1.10890.10790-daily20181024.en-US.msi