Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug]: filelocking and a lot of php-fpm-processes through preview generation #44190

Closed
5 of 8 tasks
ETES-Stuttgart opened this issue Mar 14, 2024 · 4 comments · Fixed by nextcloud/richdocuments#3831
Closed
5 of 8 tasks

[Bug]: filelocking and a lot of php-fpm-processes through preview generation #44190

ETES-Stuttgart opened this issue Mar 14, 2024 · 4 comments · Fixed by nextcloud/richdocuments#3831
Assignees
@mejo-
Labels
0. Needs triage Pending check for reproducibility or if it fits our roadmap 28-feedback bug feature: locking feature: previews and thumbnails

Comments

@ETES-Stuttgart
Copy link

ETES-Stuttgart commented Mar 14, 2024 •
edited
Loading

⚠️ This issue respects the following points: ⚠️

Bug description

We have two issues in our Nextcloud, which seem to have the same source:

  1. There are a lot of php-fpm-children spawning sometimes. We had over 300 processes in the past.
  2. When creating a new document (e.g. file.odt) from the WebGUI and opening it directly afterwards (with richdocuments/Collabora Online on a dedicated on-premise server), we are not able to save the file and get a message telling to check the permissions.

When I disable the preview generation with 'enable_previews' => false, , both issues are gone!

conceirning 1)
I cannot exactly tell when the processes are spawned, there is a coherence with the amount of users working with Nextcloud, anyway, we have setups around the same size with no issues conceirning the amount of php-fpm-children spawned.

conceirning 2)
I can see a php-fpm-process has opened the newly created file; and as long as it is open, I cannot save the file in Collabora Online.
It is unclear how long the process keeps the file open and it seems to differ from time to time.
Anyway, we often have this issue in the first ~ 10 minutes of the file-creation.

Steps to reproduce

  1. We did migrate the server from CentOS7 to RockyLinux 9; issue 2) persists, issue 1) is new
  2. We have a workaround with increasing the amount of php-fpm-children being allowed to spawn, so php-fpm is not crashing anymore and making our Nextcloud unusable..., anyway, this is not how the behaviour should be
  3. disabling the preview did solve both of the problems but keeps us with - obviously - no preview, which we want to have

Expected behavior

The amount php-fpm-processes should stay in a reasonable height; we normally are fine with something like 50 children max. For this instance we have increased it to > 400 to not have crashes while using Nextcloud.
The preview-generation of newly created (Office) files should not lock the file, therefore we can edit those files from the moment, they're created

Installation method

None

Nextcloud Server version

28

Operating system

RHEL/CentOS

PHP engine version

PHP 8.2

Web server

Apache (supported)

Database engine version

MariaDB

Is this bug present after an update or on a fresh install?

None

Are you using the Nextcloud Server Encryption module?

Encryption is Disabled

What user-backends are you using?

  • Default user-backend (database)
  • LDAP/ Active Directory
  • SSO - SAML
  • Other

Configuration report

"system": {
        "instanceid": "***REMOVED SENSITIVE VALUE***",
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "trusted_domains": {
            "0": "cloud.<domain>.<tld>",
            "1": "cloud.<internaldomain>.local",
            "2": "<IP>",
            "5": "localhost",
            "6": "127.0.0.1"
        },
        "overwrite.cli.url": "https:<domain>.<tld>",
        "datadirectory": "***REMOVED SENSITIVE VALUE***",
        "dbtype": "mysql",
        "version": "28.0.3.2",
        "dbname": "***REMOVED SENSITIVE VALUE***",
        "dbhost": "***REMOVED SENSITIVE VALUE***",
        "dbport": "",
        "dbtableprefix": "oc_",
        "mysql.utf8mb4": true,
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "installed": true,
        "ldapIgnoreNamingRules": false,
        "ldapProviderFactory": "\\OCA\\User_LDAP\\LDAPProviderFactory",
        "ldapUserCleanupInterval": 3330,
        "log_rotate_size": 8589934592,
        "loglevel": 0,
        "forcessl": true,
        "logtimezone": "Europe\/Berlin",
        "default_language": "de",
        "default_locale": "de_DE",
        "default_phone_region": "DE",
        "has_rebuilt_cache": true,
        "maintenance_window_start": "1",
        "memcache.local": "\\OC\\Memcache\\APCu",
        "memcache.locking": "\\OC\\Memcache\\Redis",
        "memcache.distributed": "\\OC\\Memcache\\Redis",
        "redis": {
            "host": "***REMOVED SENSITIVE VALUE***",
            "port": 6379,
            "timeout": 0
        },
        "mail_smtpmode": "sendmail",
        "mail_from_address": "***REMOVED SENSITIVE VALUE***",
        "mail_domain": "***REMOVED SENSITIVE VALUE***",
        "mail_smtphost": "***REMOVED SENSITIVE VALUE***",
        "mail_sendmailmode": "pipe",
        "mail_smtpport": "25",
        "updatechecker": false,
        "trashbin_retention_obligation": "auto",
        "enable_previews": false,
        "enabledPreviewProviders": [
            "OC\\Preview\\BMP",
            "OC\\Preview\\GIF",
            "OC\\Preview\\JPEG",
            "OC\\Preview\\WEBP",
            "OC\\Preview\\MarkDown",
            "OC\\Preview\\MP3",
            "OC\\Preview\\PNG",
            "OC\\Preview\\TXT",
            "OC\\Preview\\XBitmap"
        ],
        "preview_libreoffice_path": "\/usr\/bin\/libreoffice",
        "maintenance": false
    }
}

List of activated Apps

Enabled:
  - activity: 2.20.0
  - admin_audit: 1.18.0
  - cloud_federation_api: 1.11.0
  - comments: 1.18.0
  - contactsinteraction: 1.9.0
  - dav: 1.29.1
  - deck: 1.12.2
  - federatedfilesharing: 1.18.0
  - federation: 1.18.0
  - files: 2.0.0
  - files_antivirus: 5.4.2
  - files_external: 1.20.0
  - files_pdfviewer: 2.9.0
  - files_reminders: 1.1.0
  - files_sharing: 1.20.0
  - files_trashbin: 1.18.0
  - files_versions: 1.21.0
  - forms: 4.1.1
  - impersonate: 1.15.0
  - logreader: 2.13.0
  - lookup_server_connector: 1.16.0
  - nextcloud_announcements: 1.17.0
  - notes: 4.9.2
  - notifications: 2.16.0
  - oauth2: 1.16.3
  - password_policy: 1.18.0
  - photos: 2.4.0
  - polls: 6.1.6
  - previewgenerator: 5.4.0
  - privacy: 1.12.0
  - provisioning_api: 1.18.0
  - related_resources: 1.3.0
  - richdocuments: 8.3.2
  - serverinfo: 1.18.0
  - settings: 1.10.1
  - sharebymail: 1.18.0
  - survey_client: 1.16.0
  - text: 3.9.1
  - theming: 2.3.0
  - twofactor_backupcodes: 1.17.0
  - updatenotification: 1.18.0
  - user_ldap: 1.19.0
  - viewer: 2.2.0
  - workflowengine: 2.10.0
Disabled:
  - bruteforcesettings: 2.8.0 (installed 2.4.0)
  - circles: 28.0.0-dev (installed 0.21.4)
  - dashboard: 7.8.0 (installed 7.0.0)
  - encryption: 2.16.0
  - firstrunwizard: 2.17.0 (installed 1.1)
  - recommendations: 2.0.0 (installed 0.6.0)
  - support: 1.11.0 (installed 1.3.0)
  - suspicious_login: 6.0.0
  - systemtags: 1.18.0 (installed 1.8.0)
  - twofactor_totp: 10.0.0-beta.2
  - user_status: 1.8.1 (installed 1.0.1)
  - weather_status: 1.8.0 (installed 1.0.0)

Nextcloud Signing status

No errors have been found.

Nextcloud Logs

{"reqId":"ZfLPuP-n4QEvRz7WdrCByQAAAAc","level":3,"time":"2024-03-14T11:21:46+01:00","remoteAddr":"172.16.10.22","user":"<username>","app":"richdocuments","method":"POST","url":"/index.php/apps/richdocuments/wopi/files/5335619_oc4ad32a936a/contents?access_token=UEgq1qHLOf2VvZnKeOWzsTxO3xKKbKcS&access_token_ttl=0","message":"\"file.odt\" is locked, existing lock on file: 2 shared locks","userAgent":"COOLWSD HTTP Agent 23.05.9.4","version":"28.0.3.2","exception":{"Exception":"OCP\\Lock\\LockedException","Message":"\"file.odt\" is locked, existing lock on file: 2 shared locks","Code":0,"Trace":[{"file":"/var/www/html/nextcloud/lib/private/Files/View.php","line":647,"function":"changeLock","class":"OC\\Files\\View","type":"->"},{"file":"/var/www/html/nextcloud/lib/private/Files/Node/File.php","line":73,"function":"file_put_contents","class":"OC\\Files\\View","type":"->"},{"file":"/var/www/html/nextcloud/apps/richdocuments/lib/Controller/WopiController.php","line":514,"function":"putContent","class":"OC\\Files\\Node\\File","type":"->"},{"file":"/var/www/html/nextcloud/apps/richdocuments/lib/Controller/WopiController.php","line":806,"function":"OCA\\Richdocuments\\Controller\\{closure}","class":"OCA\\Richdocuments\\Controller\\WopiController","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/html/nextcloud/apps/richdocuments/lib/Controller/WopiController.php","line":783,"function":"retryOperation","class":"OCA\\Richdocuments\\Controller\\WopiController","type":"->"},{"file":"/var/www/html/nextcloud/lib/private/Files/Lock/LockManager.php","line":29,"function":"OCA\\Richdocuments\\Controller\\{closure}","class":"OCA\\Richdocuments\\Controller\\WopiController","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/html/nextcloud/apps/richdocuments/lib/Controller/WopiController.php","line":786,"function":"runInScope","class":"OC\\Files\\Lock\\LockManager","type":"->"},{"file":"/var/www/html/nextcloud/apps/richdocuments/lib/Controller/WopiController.php","line":513,"function":"wrappedFilesystemOperation","class":"OCA\\Richdocuments\\Controller\\WopiController","type":"->"},{"file":"/var/www/html/nextcloud/lib/private/AppFramework/Http/Dispatcher.php","line":230,"function":"putFile","class":"OCA\\Richdocuments\\Controller\\WopiController","type":"->"},{"file":"/var/www/html/nextcloud/lib/private/AppFramework/Http/Dispatcher.php","line":137,"function":"executeController","class":"OC\\AppFramework\\Http\\Dispatcher","type":"->"},{"file":"/var/www/html/nextcloud/lib/private/AppFramework/App.php","line":184,"function":"dispatch","class":"OC\\AppFramework\\Http\\Dispatcher","type":"->"},{"file":"/var/www/html/nextcloud/lib/private/Route/Router.php","line":315,"function":"main","class":"OC\\AppFramework\\App","type":"::"},{"file":"/var/www/html/nextcloud/lib/base.php","line":1069,"function":"match","class":"OC\\Route\\Router","type":"->"},{"file":"/var/www/html/nextcloud/index.php","line":39,"function":"handleRequest","class":"OC","type":"::"}],"File":"/var/www/html/nextcloud/lib/private/Files/View.php","Line":1984,"Previous":{"Exception":"OCP\\Lock\\LockedException","Message":"\"files/9d76f89cbccc8ec91c76acf71f20ee94\"(\"files/file.odt\") is locked, existing lock on file: 2 shared locks","Code":0,"Trace":[{"file":"/var/www/html/nextcloud/lib/private/Files/Storage/Wrapper/Wrapper.php","line":629,"function":"changeLock","class":"OC\\Files\\Storage\\Common","type":"->"},{"file":"/var/www/html/nextcloud/lib/private/Files/Storage/Wrapper/Wrapper.php","line":629,"function":"changeLock","class":"OC\\Files\\Storage\\Wrapper\\Wrapper","type":"->"},{"file":"/var/www/html/nextcloud/lib/private/Files/Storage/Wrapper/Wrapper.php","line":629,"function":"changeLock","class":"OC\\Files\\Storage\\Wrapper\\Wrapper","type":"->"},{"file":"/var/www/html/nextcloud/lib/private/Files/View.php","line":1975,"function":"changeLock","class":"OC\\Files\\Storage\\Wrapper\\Wrapper","type":"->"},{"file":"/var/www/html/nextcloud/lib/private/Files/View.php","line":647,"function":"changeLock","class":"OC\\Files\\View","type":"->"},{"file":"/var/www/html/nextcloud/lib/private/Files/Node/File.php","line":73,"function":"file_put_contents","class":"OC\\Files\\View","type":"->"},{"file":"/var/www/html/nextcloud/apps/richdocuments/lib/Controller/WopiController.php","line":514,"function":"putContent","class":"OC\\Files\\Node\\File","type":"->"},{"file":"/var/www/html/nextcloud/apps/richdocuments/lib/Controller/WopiController.php","line":806,"function":"OCA\\Richdocuments\\Controller\\{closure}","class":"OCA\\Richdocuments\\Controller\\WopiController","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/html/nextcloud/apps/richdocuments/lib/Controller/WopiController.php","line":783,"function":"retryOperation","class":"OCA\\Richdocuments\\Controller\\WopiController","type":"->"},{"file":"/var/www/html/nextcloud/lib/private/Files/Lock/LockManager.php","line":29,"function":"OCA\\Richdocuments\\Controller\\{closure}","class":"OCA\\Richdocuments\\Controller\\WopiController","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/html/nextcloud/apps/richdocuments/lib/Controller/WopiController.php","line":786,"function":"runInScope","class":"OC\\Files\\Lock\\LockManager","type":"->"},{"file":"/var/www/html/nextcloud/apps/richdocuments/lib/Controller/WopiController.php","line":513,"function":"wrappedFilesystemOperation","class":"OCA\\Richdocuments\\Controller\\WopiController","type":"->"},{"file":"/var/www/html/nextcloud/lib/private/AppFramework/Http/Dispatcher.php","line":230,"function":"putFile","class":"OCA\\Richdocuments\\Controller\\WopiController","type":"->"},{"file":"/var/www/html/nextcloud/lib/private/AppFramework/Http/Dispatcher.php","line":137,"function":"executeController","class":"OC\\AppFramework\\Http\\Dispatcher","type":"->"},{"file":"/var/www/html/nextcloud/lib/private/AppFramework/App.php","line":184,"function":"dispatch","class":"OC\\AppFramework\\Http\\Dispatcher","type":"->"},{"file":"/var/www/html/nextcloud/lib/private/Route/Router.php","line":315,"function":"main","class":"OC\\AppFramework\\App","type":"::"},{"file":"/var/www/html/nextcloud/lib/base.php","line":1069,"function":"match","class":"OC\\Route\\Router","type":"->"},{"file":"/var/www/html/nextcloud/index.php","line":39,"function":"handleRequest","class":"OC","type":"::"}],"File":"/var/www/html/nextcloud/lib/private/Files/Storage/Common.php","Line":833,"Previous":{"Exception":"OCP\\Lock\\LockedException","Message":"\"files/9d76f89cbccc8ec91c76acf71f20ee94\" is locked, existing lock on file: 2 shared locks","Code":0,"Trace":[{"file":"/var/www/html/nextcloud/lib/private/Files/Storage/Common.php","line":831,"function":"changeLock","class":"OC\\Lock\\MemcacheLockingProvider","type":"->"},{"file":"/var/www/html/nextcloud/lib/private/Files/Storage/Wrapper/Wrapper.php","line":629,"function":"changeLock","class":"OC\\Files\\Storage\\Common","type":"->"},{"file":"/var/www/html/nextcloud/lib/private/Files/Storage/Wrapper/Wrapper.php","line":629,"function":"changeLock","class":"OC\\Files\\Storage\\Wrapper\\Wrapper","type":"->"},{"file":"/var/www/html/nextcloud/lib/private/Files/Storage/Wrapper/Wrapper.php","line":629,"function":"changeLock","class":"OC\\Files\\Storage\\Wrapper\\Wrapper","type":"->"},{"file":"/var/www/html/nextcloud/lib/private/Files/View.php","line":1975,"function":"changeLock","class":"OC\\Files\\Storage\\Wrapper\\Wrapper","type":"->"},{"file":"/var/www/html/nextcloud/lib/private/Files/View.php","line":647,"function":"changeLock","class":"OC\\Files\\View","type":"->"},{"file":"/var/www/html/nextcloud/lib/private/Files/Node/File.php","line":73,"function":"file_put_contents","class":"OC\\Files\\View","type":"->"},{"file":"/var/www/html/nextcloud/apps/richdocuments/lib/Controller/WopiController.php","line":514,"function":"putContent","class":"OC\\Files\\Node\\File","type":"->"},{"file":"/var/www/html/nextcloud/apps/richdocuments/lib/Controller/WopiController.php","line":806,"function":"OCA\\Richdocuments\\Controller\\{closure}","class":"OCA\\Richdocuments\\Controller\\WopiController","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/html/nextcloud/apps/richdocuments/lib/Controller/WopiController.php","line":783,"function":"retryOperation","class":"OCA\\Richdocuments\\Controller\\WopiController","type":"->"},{"file":"/var/www/html/nextcloud/lib/private/Files/Lock/LockManager.php","line":29,"function":"OCA\\Richdocuments\\Controller\\{closure}","class":"OCA\\Richdocuments\\Controller\\WopiController","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/html/nextcloud/apps/richdocuments/lib/Controller/WopiController.php","line":786,"function":"runInScope","class":"OC\\Files\\Lock\\LockManager","type":"->"},{"file":"/var/www/html/nextcloud/apps/richdocuments/lib/Controller/WopiController.php","line":513,"function":"wrappedFilesystemOperation","class":"OCA\\Richdocuments\\Controller\\WopiController","type":"->"},{"file":"/var/www/html/nextcloud/lib/private/AppFramework/Http/Dispatcher.php","line":230,"function":"putFile","class":"OCA\\Richdocuments\\Controller\\WopiController","type":"->"},{"file":"/var/www/html/nextcloud/lib/private/AppFramework/Http/Dispatcher.php","line":137,"function":"executeController","class":"OC\\AppFramework\\Http\\Dispatcher","type":"->"},{"file":"/var/www/html/nextcloud/lib/private/AppFramework/App.php","line":184,"function":"dispatch","class":"OC\\AppFramework\\Http\\Dispatcher","type":"->"},{"file":"/var/www/html/nextcloud/lib/private/Route/Router.php","line":315,"function":"main","class":"OC\\AppFramework\\App","type":"::"},{"file":"/var/www/html/nextcloud/lib/base.php","line":1069,"function":"match","class":"OC\\Route\\Router","type":"->"},{"file":"/var/www/html/nextcloud/index.php","line":39,"function":"handleRequest","class":"OC","type":"::"}],"File":"/var/www/html/nextcloud/lib/private/Lock/MemcacheLockingProvider.php","Line":145}},"message":"\"file.odt\" is locked, existing lock on file: 2 shared locks","exception":{},"CustomMessage":"\"file.odt\" is locked, existing lock on file: 2 shared locks"}}

Additional info

[root@cloud ~]# php-fpm -v
PHP 8.2.16 (fpm-fcgi) (built: Feb 13 2024 15:22:59)
Copyright (c) The PHP Group
Zend Engine v4.2.16, Copyright (c) Zend Technologies
with Zend OPcache v8.2.16, Copyright (c), by Zend Technologies

[root@cloud ~]# sudo -u apache php /var/www/html/nextcloud/occ status

  • installed: true
  • version: 28.0.3.2
  • versionstring: 28.0.3
  • edition:
  • maintenance: false
  • needsDbUpgrade: false
  • productname: Nextcloud
  • extendedSupport: false
@ETES-Stuttgart ETES-Stuttgart added 0. Needs triage Pending check for reproducibility or if it fits our roadmap bug labels Mar 14, 2024
@ETES-Stuttgart ETES-Stuttgart changed the title [Bug]: [Bug]: filelocking and a lot of php-fpm-processes through preview generation Mar 15, 2024
@ETES-Stuttgart ETES-Stuttgart mentioned this issue Mar 15, 2024
Closed
@joshtrichards
Copy link
Member

Are you Office previews ultimately actually being successfully generated?

I see you have the previewgenerator app installed. How frequently is it running?

I assume you're not seeing any libreoffice/office.bin processes since you're using Nextcloud Office (which should handle it's own preview generation)?

@ETES-Stuttgart
Copy link
Author

@joshtrichards

Previews are generated when enabled - also for office documents.
The previewgenerator app has been installed for debugging purposes, we're not actually using it frequently.

Since we are using Nextcloud Office with a Collabora on-premise-server, yes, i do not see any libreoffice/office.bin processes.

@mejo-
Copy link
Member

mejo- commented Jul 9, 2024

@ETES-Stuttgart we figured that this happens at times when Nextcloud tries to access the Collabora instance via a private IP. Does setting 'allow_local_remote_servers' => true, solve the issue for you?

@ETES-Stuttgart
Copy link
Author

ETES-Stuttgart commented Jul 11, 2024 •
edited
Loading

@ETES-Stuttgart we figured that this happens at times when Nextcloud tries to access the Collabora instance via a private IP. Does setting 'allow_local_remote_servers' => true, solve the issue for you?

I added the setting to my config, as suggested and when testing, I could no longer reproduce the issue.
I am kind of optimistic, that this solves the problem but will have a look on that for some additional time with different users as well.
Anyway, thank you already!!

What I do not get is, when (and why) I need to set this?
Yes, our NC-server has the local IP of collabora in the /etc/hosts-file, since proxy-issues prevent the traffic going through the Internet, but it did work sometimes anyway. When is this setting set to 'false' preventing things? And what and why exactly?
Could you - or someone else - clarify this?

Thanks again & best regards :)

mejo- added a commit to nextcloud/richdocuments that referenced this issue Jul 24, 2024
@mejo-
fix(client): Use shared default options for HTTP client requests
9532339 
Makes sure we always set a sane timeout and `allow_local_address`.

Fixes: #3255
Fixes: #3435
Fixes: nextcloud/server#44190

Signed-off-by: Jonas <jonas@freesources.org>
@mejo- mejo- mentioned this issue Jul 24, 2024
Merged
3 tasks
@mejo- mejo- self-assigned this Jul 24, 2024
mejo- added a commit to nextcloud/richdocuments that referenced this issue Jul 24, 2024
@mejo-
fix(client): Use shared default options for HTTP client requests
a72bbf3 
Makes sure we always set a sane timeout and `allow_local_address`.

Fixes: #3255
Fixes: #3435
Fixes: nextcloud/server#44190

Signed-off-by: Jonas <jonas@freesources.org>
backportbot bot pushed a commit to nextcloud/richdocuments that referenced this issue Jul 25, 2024
@mejo- @backportbot
fix(client): Use shared default options for HTTP client requests
ff4f22b 
Makes sure we always set a sane timeout and `allow_local_address`.

Fixes: #3255
Fixes: #3435
Fixes: nextcloud/server#44190

Signed-off-by: Jonas <jonas@freesources.org>
mejo- added a commit to nextcloud/richdocuments that referenced this issue Sep 6, 2024
@mejo-
fix(client): Use shared default options for HTTP client requests
2740840 
Makes sure we always set a sane timeout and `allow_local_address`.

Fixes: #3255
Fixes: #3435
Fixes: nextcloud/server#44190

Signed-off-by: Jonas <jonas@freesources.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@mejo- mejo-

Labels
0. Needs triage Pending check for reproducibility or if it fits our roadmap 28-feedback bug feature: locking feature: previews and thumbnails
Projects
📝 Office team
Archived in project
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix(client): Use shared default options for HTTP client requests nextcloud/richdocuments
4 participants
@joshtrichards @mejo- @szaimen @ETES-Stuttgart