Corrupt files and folder when gocryptfs cipherdir is synchronised via Syncthing #549

savchenko · 2021-02-25T04:29:12Z

I have started observing this during the last 2 weeks or so, albeit on a smaller scale.

In previous cases corruption was mostly affecting git indexes, but here damage is more extensive:

$ gocryptfs -fsck gocrypt/
Password: 
Decrypting master key
OpenDir ".": invalid entry "Q42GWQmqmH5GvujP8DOrdg": bad message
fsck: corrupt entry in dir "": "Q42GWQmqmH5GvujP8DOrdg"
OpenDir "mhzofRHRP2shWMMtzv6ZEA": could not read gocryptfs.diriv: no such file or directory
fsck: error opening dir "xxxxxxxxx/xxxxxxxxxx/xx/xxxxxxxx/xxxx/xxxxxxxx": 5=input/output error
OpenDir "CmE5cc0HwXERKp3e6lO3gA": invalid entry "BS1jRx94o2SPZE3VVoKduw": bad message
fsck: corrupt entry in dir "xxxx/xxxx/xxx": "BS1jRx94o2SPZE3VVoKduw"
OpenDir "TNYugAMPIEi73eD6AlHZ3A": invalid entry "ErCyzALmrA_6iMQ2je18iQ": bad message
fsck: corrupt entry in dir "xxxx/xxxx/xxx/xxxxxx": "ErCyzALmrA_6iMQ2je18iQ"
OpenDir "YbtQ8679xCEYi324Okt7Bw": invalid entry "Q7lbJVgsU7EQJollS_EUng": bad message
fsck: corrupt entry in dir "xxxx/xxxx/xxx/xxxxxx/xxxxx": "Q7lbJVgsU7EQJollS_EUng"
OpenDir "8eOpARKnAQAbCBLVaVIBgg": invalid entry "r3UIaZYYMDQQjIOfM4WpJQ": bad message
OpenDir "8eOpARKnAQAbCBLVaVIBgg": invalid entry "XHaLuVAw3rTZ0A72ErwQDw": bad message
fsck: corrupt entry in dir "xxxx/xxxx/xxxxxx": "r3UIaZYYMDQQjIOfM4WpJQ"
fsck: corrupt entry in dir "xxxx/xxxx/xxxxxx": "XHaLuVAw3rTZ0A72ErwQDw"
OpenDir "EOiNnPbRmZLFlDZUy1hzRw": invalid entry "J9WwFnSWdFMcRtv88muvpg": bad message
fsck: corrupt entry in dir "xxxx/xxxx/xxxxxx/xxxxxxxxxxx": "J9WwFnSWdFMcRtv88muvpg"
OpenDir "xCcc7WlHYQ-CC31Q-T5yIw": invalid entry "LKwyeRtRF_0Bd-OWgPEAnA": bad message
fsck: corrupt entry in dir "xxxx/xxxx/xxxxxx/xxxxxxxxxxx/xxxx": "LKwyeRtRF_0Bd-OWgPEAnA"
OpenDir "mLOdyr1FCyOnfNa2332zIQ": invalid entry "nuPcy1D81IzB3sKpQiGSFQ": bad message
fsck: corrupt entry in dir "xxxx/xxxx/xxxxxxxx": "nuPcy1D81IzB3sKpQiGSFQ"
OpenDir "h5wPJK5dw21t-8bQLhfkuw": invalid entry "360a5xjQI5NYzUrCQzyb-A": bad message
fsck: corrupt entry in dir "xxxx/xxxx/xxxxxxxx/xxxxxxxx": "360a5xjQI5NYzUrCQzyb-A"
OpenDir "n4buocgPjyZ8eRSeAqCk7g": invalid entry "7Iifijp3w-TGxvt57TVMFw": bad message
fsck: corrupt entry in dir "xxxx/xxxx/xxxxxxxx/xxxxxxxxxxx": "7Iifijp3w-TGxvt57TVMFw"
OpenDir "cOo3UWTinz-Nrzmnojz2Og": invalid entry "Wo87J3NGSRa_D7KfTiF-0Q": bad message
fsck: corrupt entry in dir "xxxx/xxxx/xxxxxxxx/xxxxxxxxx": "Wo87J3NGSRa_D7KfTiF-0Q"
OpenDir "n15XaS9mDqLXjQXYUtkdUw": invalid entry "3cYjofmRbfOomOT_oZvRug": bad message
fsck: corrupt entry in dir "xxxx/xxxx/xxxxxxxxx": "3cYjofmRbfOomOT_oZvRug"
OpenDir "vPa5sYKAdS1HNODvxUnGcw": invalid entry "fNOALh19-Gk_Nfq3f9q5kQ": bad message
fsck: corrupt entry in dir "xxxx/xxxx/xxxxxxxxx/BSD": "fNOALh19-Gk_Nfq3f9q5kQ"
OpenDir "Re1tpEUGMO9Rh2aW8LI_yg": invalid entry "ww0XqKavBZqE3xoTXUPJLA": bad message
fsck: corrupt entry in dir "xxxx/xxxx/xxxxxxxxxxxx": "ww0XqKavBZqE3xoTXUPJLA"
OpenDir "xWrSDaS9XCmZxVin_3UkVA": invalid entry "4NtZBOl8UevnjabTYHT5XA": bad message
fsck: corrupt entry in dir "xxxx/xxxx/xxxxxxxxxxxx/xxxxxx": "4NtZBOl8UevnjabTYHT5XA"
OpenDir "dUr7Tfadq4vJXBetXhrAGw": invalid entry "QsBR3hImqbjav3TYaR7YxQ": bad message
fsck: corrupt entry in dir "xxxx/xxxx/xxxxxxxxxxxx": "QsBR3hImqbjav3TYaR7YxQ"
OpenDir "G0PrT6uQD760thIUu_PZfg": invalid entry "Ie_9LDDFJMkoOv2FhKgueg": bad message
fsck: corrupt entry in dir "xxxx/xxxx/xxxxx": "Ie_9LDDFJMkoOv2FhKgueg"
fsck summary: 19 corrupt files, 0 files skipped

Folder that shows 5=input/output error is displayed in clear-text as empty.

Is there anything I can do to help with investigating this issue? Or maybe there is some underlying incompatibility of GoCrypt share being synchronised bi-directionally?

Thanks.

The text was updated successfully, but these errors were encountered:

ghost · 2021-03-03T01:39:07Z

I have reached the same issue. I have 3 devices (server, PC and android) and I am syncing every device with all the others.

I do not see "corrupt entry errors" but I have OpenDir ".": invalid entry ".stfolder": illegal base64 data at input byte 0
`

ghost · 2021-03-05T22:13:04Z

Because of this I have moved to https://cryptomator.org/ for now. It seems to work well with syncthing so far. I would prefer gocryptfs because it has fully open-sourced android app, but until this issue is solved, it is not possible.

rfjakob · 2021-03-06T10:41:38Z

I see three different issues here:

OpenDir "mhzofRHRP2shWMMtzv6ZEA": could not read gocryptfs.diriv: no such file or directory

Looks like syncthing forgot to copy the gocryptfs.diriv file (what!?)

OpenDir "CmE5cc0HwXERKp3e6lO3gA": invalid entry "BS1jRx94o2SPZE3VVoKduw": bad message

The file names in the folder do not match the gocryptfs.diriv in that folder (huh!?)

OpenDir ".": invalid entry ".stfolder": illegal base64 data at input byte 0

Syncthing created a file .stfolder, and gocryptfs cannot decrypt it (obviously). I suggest you put the gocryptfs encrypted dir in a SUBFOLDER of the folder you sync with syncthing.

savchenko · 2021-03-06T11:04:18Z

Syncthing created a file .stfolder, and gocryptfs cannot decrypt it (obviously). I suggest you put the gocryptfs encrypted dir in a SUBFOLDER of the folder you sync with syncthing.

I will double-check the previous points, but can say now that gocryptfs-encrypted folder is already in a subfolder of the folder synced.

Please let me know if there is anything else that might help you to debug this.

rfjakob · 2021-03-06T11:30:13Z

@savchenko i only saw the .stlfolder in the messages from @peterflock

rfjakob · 2021-03-10T18:10:40Z

@savchenko Do you see the same corruption on all synchronised copies or does it look different?

rfjakob · 2021-03-14T15:34:41Z

Question to all syncthing users here: Do you use the -sharedstorage flag (you should!)?

I started syncthing three times like this

cd /var/tmp
mkdir a b c
syncthing -home=a &
syncthing -home=b &
syncthing -home=c &

and synced a folder between all three, so that I have three copies on disk, and mounted the three copies three times (without using `-sharedstorage). And after some trying I did manage to reprodruce git corruption:

$ git fsck
Checking object directories: 100% (256/256), done.
error: 60ac6e68fa5f74a0ae8cf39f72237e7b2b43cbf0: invalid sha1 pointer in cache-tree
broken link from    tree 979c573280be7d111e2874c0d7427df87171d20a
              to    tree 85a49a1f0b6b87b639763f280b5c7982cec06a81
...

rfjakob · 2021-03-14T18:59:50Z

Todos for go-fuse to make things more robust even without -sharedstorage:

Clear parents map if out.Nlink == 1? ---> https://github.com/hanwen/go-fuse/blob/0f728ba15b38579efefc3dc47821882ca18ffea7/fs/bridge.go#L210
Select last know-good parent instead of random? ---> https://github.com/hanwen/go-fuse/blob/0f728ba15b38579efefc3dc47821882ca18ffea7/fs/inode.go#L293
Implemented in https://review.gerrithub.io/c/hanwen/go-fuse/+/513646/2

savchenko · 2021-03-15T06:39:19Z

Do you see the same corruption on all synchronised copies or does it look different?

Different. Here is an output from another endpoint, same repository I've mentioned in the first post:

Decrypting master key
fsck: corrupt entry in dir "": "Q42GWQmqmH5GvujP8DOrdg"
fsck: error opening dir "xxxxx/xxxxxx/xxxxx/xxxxx/xxxxx/xxxxxxxx": 2=no such file or directory
fsck: corrupt entry in dir "xxxx/.git": "GGdbtiSuWnU4TaoWMpwPSA.sync-conflict-20210309-161800-K35IS4A"
fsck: corrupt entry in dir "xxxx/.git": "GGdbtiSuWnU4TaoWMpwPSA.sync-conflict-20210309-161819-K35IS4A"
fsck: corrupt entry in dir "xxxx/xxxx/xxxx": "BS1jRx94o2SPZE3VVoKduw"
fsck: corrupt entry in dir "xxxx/xxxx/xxxx/xxxx": "ErCyzALmrA_6iMQ2je18iQ"
fsck: corrupt entry in dir "xxxx/xxxx/xxxx/xxxx/xxxxx": "Q7lbJVgsU7EQJollS_EUng"
fsck: corrupt entry in dir "xxxx/xxxx/xxxxxx": "r3UIaZYYMDQQjIOfM4WpJQ"
fsck: corrupt entry in dir "xxxx/xxxx/xxxxxx": "XHaLuVAw3rTZ0A72ErwQDw"
fsck: corrupt entry in dir "xxxx/xxxx/xxxxxx/xxxxxx": "J9WwFnSWdFMcRtv88muvpg"
fsck: corrupt entry in dir "xxxx/xxxx/xxxxxx/xxxxxx/xxxx": "LKwyeRtRF_0Bd-OWgPEAnA"
fsck: corrupt entry in dir "xxxx/xxxx/xxxxxxx": "nuPcy1D81IzB3sKpQiGSFQ"
fsck: corrupt entry in dir "xxxx/xxxx/xxxxxxx/xxxxxxx": "360a5xjQI5NYzUrCQzyb-A"
fsck: corrupt entry in dir "xxxx/xxxx/xxxxxxx/xxxxxxx": "7Iifijp3w-TGxvt57TVMFw"
fsck: corrupt entry in dir "xxxx/xxxx/xxxxxxx/xxxxxxx": "Wo87J3NGSRa_D7KfTiF-0Q"
fsck: corrupt entry in dir "xxxx/xxxx/xxxxxx": "3cYjofmRbfOomOT_oZvRug"
fsck: corrupt entry in dir "xxxx/xxxx/xxxxxxx/xxxxxx": "fNOALh19-Gk_Nfq3f9q5kQ"
fsck: corrupt entry in dir "xxxx/xxxx/xxxxxxxxx": "ww0XqKavBZqE3xoTXUPJLA"
fsck: corrupt entry in dir "xxxx/xxxx/xxxxxxxxx/xxxxxx": "4NtZBOl8UevnjabTYHT5XA"
fsck: corrupt entry in dir "xxxx/xxxx/xxxxxxxxx": "QsBR3hImqbjav3TYaR7YxQ"
fsck: corrupt entry in dir "xxxx/xxxx/xxxxxxxxx": "Ie_9LDDFJMkoOv2FhKgueg"
fsck summary: 21 corrupt files

rfjakob · 2021-03-20T08:51:35Z

Thanks! The sync-conflict-20210309-161800-K35IS4A stuff is clearly different, but the rest looks like the same errors with slighly different fsck messages due to different gocryptfs versions?

tleydxdy · 2021-04-19T06:38:02Z

Similar? problem here. I'm also using syncthing and I realized I can't open the directory with my password anymore. it says cipher: message authentication failed I am certain that the password is correct, since I have a paper backup of it, and the master key on that same paper backup works. the directory is also shared with others and they can't use the password to decrypt it too.

Altho -fsck shows the directory as error free.

I did not have the sharedstorage flag, I didn't know it existed :( I don't think there was concurrent access tho, at least not concurrent writes to the same file

rfjakob · 2021-04-24T10:21:44Z

@tleydxdy you get this?

Decrypting master key
failed to unlock master key: cipher: message authentication failed
Password incorrect.

Decrypting the master key only depends on gocryptfs.conf being intact, and this file is never modified except when changing the password (gocryptfs -passwd).

Keyboard layout problem maybe?

Change was merged: hanwen/go-fuse@a90e1f4 Done using: $ go mod edit -dropreplace github.com/hanwen/go-fuse/v2 $ go get github.com/hanwen/go-fuse/v2@master go: downloading github.com/hanwen/go-fuse/v2 v2.1.1-0.20210423170155-a90e1f463c3f go get: upgraded github.com/hanwen/go-fuse/v2 v2.0.4-0.20210125162859-8e0bbdb16cb7 => v2.1.1-0.20210423170155-a90e1f463c3f Related: #549

tleydxdy · 2021-04-27T13:54:49Z

Keyboard layout problem maybe?
I don't think so, this is pretty strange I agree, but I haven't change devices or anything and this is on mutiple devices between two people. I guess until it happens again it'll only be a fluke (maybe I changed the password while sleeping)

rfjakob · 2021-05-03T17:40:46Z

Dear syncthing users, can you see if you can still trigger this problem with v2.0-beta3 ? This has the "track most recent parent" improvement (see changelog)

Download from https://github.com/rfjakob/gocryptfs/releases/tag/v2.0-beta3

rfjakob · 2021-08-17T13:37:25Z

gocryptfs v2.0 is out some time now containing the improvements. I tested some time ago with syncthing and was unable to cause problems. Even without -sharedstorage this should not happen anymore.

Unless somebody notices otherwise I'll call this fixed.

rfjakob changed the title ~~Corrupt files and folder when GoCrypt cipherdir is synchronised via Syncthing~~ Corrupt files and folder when gocryptfs cipherdir is synchronised via Syncthing Mar 14, 2021

rfjakob added the bug label May 3, 2021

rfjakob closed this as completed Aug 17, 2021

rfjakob mentioned this issue Aug 18, 2021

Minimum version of FUSE for 2.1 release #594

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Corrupt files and folder when gocryptfs cipherdir is synchronised via Syncthing #549

Corrupt files and folder when gocryptfs cipherdir is synchronised via Syncthing #549

savchenko commented Feb 25, 2021

ghost commented Mar 3, 2021

ghost commented Mar 5, 2021

rfjakob commented Mar 6, 2021

savchenko commented Mar 6, 2021

rfjakob commented Mar 6, 2021

rfjakob commented Mar 10, 2021

rfjakob commented Mar 14, 2021 •

edited

Loading

rfjakob commented Mar 14, 2021 •

edited

Loading

savchenko commented Mar 15, 2021

rfjakob commented Mar 20, 2021

tleydxdy commented Apr 19, 2021 •

edited

Loading

rfjakob commented Apr 24, 2021 •

edited

Loading

tleydxdy commented Apr 27, 2021

rfjakob commented May 3, 2021

rfjakob commented Aug 17, 2021

Corrupt files and folder when gocryptfs cipherdir is synchronised via Syncthing #549

Corrupt files and folder when gocryptfs cipherdir is synchronised via Syncthing #549

Comments

savchenko commented Feb 25, 2021

ghost commented Mar 3, 2021

ghost commented Mar 5, 2021

rfjakob commented Mar 6, 2021

savchenko commented Mar 6, 2021

rfjakob commented Mar 6, 2021

rfjakob commented Mar 10, 2021

rfjakob commented Mar 14, 2021 • edited Loading

rfjakob commented Mar 14, 2021 • edited Loading

savchenko commented Mar 15, 2021

rfjakob commented Mar 20, 2021

tleydxdy commented Apr 19, 2021 • edited Loading

rfjakob commented Apr 24, 2021 • edited Loading

tleydxdy commented Apr 27, 2021

rfjakob commented May 3, 2021

rfjakob commented Aug 17, 2021

rfjakob commented Mar 14, 2021 •

edited

Loading

rfjakob commented Mar 14, 2021 •

edited

Loading

tleydxdy commented Apr 19, 2021 •

edited

Loading

rfjakob commented Apr 24, 2021 •

edited

Loading