Support for FreeIPA v. 4.6 #157
Comments
|
There is Unfortunatelly, at this point even Fedora 26-based containers (built via so even newer versions might have even worse stability issues. |
|
Note that in Rawhide and F27 branched you should be OK when FreeIPA 4.6.0-2 packages reach them. You'd most likely need to rebuild your base docker images. Check following Bodhi update: https://bodhi.fedoraproject.org/updates/FEDORA-2017-a79e85e4d3, it is currently in pending state. |
|
Please note that FreeIPA 4.5 was never released into Fedora so it won't be available in containers. Fedora 27 container will have FreeIPA 4.6, though. |
The `ipa-server-install` program often fails during the "Configuring certificate server (pki-tomcatd)" stage in FreeIPA 4.5 and 4.6, at least for Fedora 25, 26 and rawhide. This seems to be a known problem [1]. The dogtag service is large and slow with many moving parts, and tends to be unresponsive right after start-up, especially for expensive operations that generate new certificates. The failures come from dbus clients timing out while attempting to run expensive operations through certmonger. This hack changes dbus client timeouts for some such operations to 120 seconds (from the default 25 seconds, IIRC). Of course it doesn't really belong here, but rather in the upstream FreeIPA project. [1]: freeipa#157 (comment)
The `ipa-server-install` program often fails during the "Configuring certificate server (pki-tomcatd)" stage in FreeIPA 4.5 and 4.6, at least in containers running Fedora 25, 26 and rawhide. This seems to be a known problem [1]. The dogtag service is large and slow with many moving parts, and tends to be unresponsive right after start-up, especially for expensive operations that generate new certificates. The failures come from dbus clients timing out while attempting to run these expensive operations through certmonger. This patch changes dbus client timeouts for some such operations to 120 seconds (from the default 25 seconds, IIRC). [1]: freeipa/freeipa-container#157 (comment)
The `ipa-server-install` program often fails during the "Configuring certificate server (pki-tomcatd)" stage in FreeIPA 4.5 and 4.6, at least for Fedora 25, 26 and rawhide. This seems to be a known problem [1]. The dogtag service is large and slow with many moving parts, and tends to be unresponsive right after start-up, especially for expensive operations that generate new certificates. The failures come from dbus clients timing out while attempting to run expensive operations through certmonger. This hack changes dbus client timeouts for some such operations to 120 seconds (from the default 25 seconds, IIRC). Of course it doesn't really belong here, but rather in the upstream FreeIPA project. [1]: freeipa#157 (comment)
The `ipa-server-install` program often fails during the "Configuring certificate server (pki-tomcatd)" stage in FreeIPA 4.5 and 4.6, at least for Fedora 25, 26 and rawhide. This seems to be a known problem [1]. The dogtag service is large and slow with many moving parts, and tends to be unresponsive right after start-up, especially for expensive operations that generate new certificates. The failures come from dbus clients timing out while attempting to run expensive operations through certmonger. This hack changes dbus client timeouts for some such operations to 120 seconds (from the default 25 seconds, IIRC). Of course it doesn't really belong here, but rather in the upstream FreeIPA project. [1]: freeipa#157 (comment)
|
I've also seen the same non-deterministic behavior that @adelton and (I assume) @stlaz are seeing. I just filed a PR against FreeIPA 4.5 upping the dbus timeouts in a couple of places that often for me. In the meantime, my There's a bunch of other stuff in that branch; if anything looks useful, I'll be happy to pick it out into a PR:
|
|
@zultron Thanks for your continuous contributions, I very much appreciate it. I'll put on my container hat tomorrow and will go through both your PRs (freeipa + freeipa-container), will try and test them, and will check whether we may adopt some of the changes in your git repo 👍 |
The `ipa-server-install` program often fails during the "Configuring certificate server (pki-tomcatd)" stage in FreeIPA 4.5 and 4.6, at least in containers running Fedora 25, 26 and rawhide. This seems to be a known problem [1]. The dogtag service is large and slow with many moving parts, and tends to be unresponsive right after start-up, especially for expensive operations that generate new certificates. The failures come from dbus clients timing out while attempting to run these expensive operations through certmonger. This patch changes dbus client timeouts for some such operations to 120 seconds (from the default 25 seconds, IIRC). [1]: freeipa/freeipa-container#157 (comment)
|
PR #158 submitted to fix |
|
During During From At this point, I'm not sure if it's another race condition or if it's my unfamiliarity with FreeIPA 4.5. The |
|
Any explanation why the D-Bus timeouts would be seen in the container setup and not on the host? |
|
@adelton, the timeouts addressed in freeipa/freeipa#1078 come from running on a memory-constrained system, not from running in a container per se. If there's any swapping going on as dogtag starts up, the default 25 second timeout halts the installation. |
|
IIRC, @stlaz found some issue with keyring which affected the dogtag startup in containers. So the timeout tweaks should not be needed. |
|
Indeed. It seems this might be a regression in systemd-233. Some related discussions: |
|
On 09/18/2017 02:23 AM, Jan Pazdziora wrote:
IIRC, @stlaz found some issue with keyring
which affected the dogtag startup in containers. So the timeout tweaks
should not be needed.
Yes he did. Initially I thought this was the same bug, since it
happened around the same place in the installer, but it turns out not to
be. My apologies for the confusion.
The timeouts I reported are here, and unfortunately won't be fixed with the dogtag issue @stlaz is working on:
freeipa/freeipa#1078
|
When running on memory-constrained systems, the `ipa-server-install` program often fails during the "Configuring certificate server (pki-tomcatd)" stage in FreeIPA 4.5 and 4.6. The memory-intensive dogtag service causes swapping on low-memory systems right after start-up, and especially new certificate operations requested via certmonger can exceed the dbus client default 25 second timeout. This patch changes dbus client timeouts for some such operations to 120 seconds (from the default 25 seconds, IIRC). See more discussion in FreeIPA PR freeipa#1078 [1] and FreeIPA container issue freeipa#157 [2]. [1]: freeipa#1078 [2]: freeipa/freeipa-container#157
When running on memory-constrained systems, the `ipa-server-install` program often fails during the "Configuring certificate server (pki-tomcatd)" stage in FreeIPA 4.5 and 4.6. The memory-intensive dogtag service causes swapping on low-memory systems right after start-up, and especially new certificate operations requested via certmonger can exceed the dbus client default 25 second timeout. This patch changes dbus client timeouts for some such operations to 120 seconds (from the default 25 seconds, IIRC). See more discussion in FreeIPA PR freeipa#1078 [1] and FreeIPA container issue freeipa#157 [2]. [1]: freeipa#1078 [2]: freeipa/freeipa-container#157
When running on memory-constrained systems, the `ipa-server-install` program often fails during the "Configuring certificate server (pki-tomcatd)" stage in FreeIPA 4.5 and 4.6. The memory-intensive dogtag service causes swapping on low-memory systems right after start-up, and especially new certificate operations requested via certmonger can exceed the dbus client default 25 second timeout. This patch changes dbus client timeouts for some such operations to 120 seconds (from the default 25 seconds, IIRC). See more discussion in FreeIPA PR freeipa#1078 [1] and FreeIPA container issue freeipa#157 [2]. Upstream ticket at [3]. [1]: freeipa#1078 [2]: freeipa/freeipa-container#157 [3]: https://pagure.io/freeipa/issue/7213
When running on memory-constrained systems, the `ipa-server-install` program often fails during the "Configuring certificate server (pki-tomcatd)" stage in FreeIPA 4.5 and 4.6. The memory-intensive dogtag service causes swapping on low-memory systems right after start-up, and especially new certificate operations requested via certmonger can exceed the dbus client default 25 second timeout. This patch changes dbus client timeouts for some such operations to 120 seconds (from the default 25 seconds, IIRC). See more discussion in FreeIPA PR freeipa#1078 [1] and FreeIPA container issue freeipa#157 [2]. Upstream ticket at [3]. [1]: freeipa#1078 [2]: freeipa/freeipa-container#157 [3]: https://pagure.io/freeipa/issue/7213
When running on memory-constrained systems, the `ipa-server-install` program often fails during the "Configuring certificate server (pki-tomcatd)" stage in FreeIPA 4.5 and 4.6. The memory-intensive dogtag service causes swapping on low-memory systems right after start-up, and especially new certificate operations requested via certmonger can exceed the dbus client default 25 second timeout. This patch changes dbus client timeouts for some such operations to 120 seconds (from the default 25 seconds, IIRC). See more discussion in FreeIPA PR freeipa#1078 [1] and FreeIPA container issue freeipa#157 [2]. Upstream ticket at [3]. [1]: freeipa#1078 [2]: freeipa/freeipa-container#157 [3]: https://pagure.io/freeipa/issue/7213
When running on memory-constrained systems, the `ipa-server-install` program often fails during the "Configuring certificate server (pki-tomcatd)" stage in FreeIPA 4.5 and 4.6. The memory-intensive dogtag service causes swapping on low-memory systems right after start-up, and especially new certificate operations requested via certmonger can exceed the dbus client default 25 second timeout. This patch changes dbus client timeouts for some such operations to 120 seconds (from the default 25 seconds, IIRC). See more discussion in FreeIPA PR freeipa#1078 [1] and FreeIPA container issue freeipa#157 [2]. Upstream ticket at [3]. [1]: freeipa#1078 [2]: freeipa/freeipa-container#157 [3]: https://pagure.io/freeipa/issue/7213
When running on memory-constrained systems, the `ipa-server-install` program often fails during the "Configuring certificate server (pki-tomcatd)" stage in FreeIPA 4.5 and 4.6. The memory-intensive dogtag service causes swapping on low-memory systems right after start-up, and especially new certificate operations requested via certmonger can exceed the dbus client default 25 second timeout. This patch changes dbus client timeouts for some such operations to 120 seconds (from the default 25 seconds, IIRC). See more discussion in FreeIPA PR freeipa#1078 [1] and FreeIPA container issue freeipa#157 [2]. Upstream ticket at [3]. [1]: freeipa#1078 [2]: freeipa/freeipa-container#157 [3]: https://pagure.io/freeipa/issue/7213
When running on memory-constrained systems, the `ipa-server-install` program often fails during the "Configuring certificate server (pki-tomcatd)" stage in FreeIPA 4.5 and 4.6. The memory-intensive dogtag service causes swapping on low-memory systems right after start-up, and especially new certificate operations requested via certmonger can exceed the dbus client default 25 second timeout. This patch changes dbus client timeouts for some such operations to 120 seconds (from the default 25 seconds, IIRC). See more discussion in FreeIPA PR freeipa#1078 [1] and FreeIPA container issue freeipa#157 [2]. Upstream ticket at [3]. [1]: freeipa#1078 [2]: freeipa/freeipa-container#157 [3]: https://pagure.io/freeipa/issue/7213
When running on memory-constrained systems, the `ipa-server-install` program often fails during the "Configuring certificate server (pki-tomcatd)" stage in FreeIPA 4.5 and 4.6. The memory-intensive dogtag service causes swapping on low-memory systems right after start-up, and especially new certificate operations requested via certmonger can exceed the dbus client default 25 second timeout. This patch changes dbus client timeouts for some such operations to 120 seconds (from the default 25 seconds, IIRC). See more discussion in FreeIPA PR freeipa#1078 [1] and FreeIPA container issue #157 [2]. Upstream ticket at [3]. [1]: freeipa#1078 [2]: freeipa/freeipa-container#157 [3]: https://pagure.io/freeipa/issue/7213 Reviewed-By: Stanislav Laznicka <slaznick@redhat.com> Reviewed-By: Rob Crittenden <rcritten@redhat.com>
When running on memory-constrained systems, the `ipa-server-install` program often fails during the "Configuring certificate server (pki-tomcatd)" stage in FreeIPA 4.5 and 4.6. The memory-intensive dogtag service causes swapping on low-memory systems right after start-up, and especially new certificate operations requested via certmonger can exceed the dbus client default 25 second timeout. This patch changes dbus client timeouts for some such operations to 120 seconds (from the default 25 seconds, IIRC). See more discussion in FreeIPA PR freeipa#1078 [1] and FreeIPA container issue freeipa#157 [2]. Upstream ticket at [3]. [1]: freeipa#1078 [2]: freeipa/freeipa-container#157 [3]: https://pagure.io/freeipa/issue/7213 Reviewed-By: Stanislav Laznicka <slaznick@redhat.com> Reviewed-By: Rob Crittenden <rcritten@redhat.com>
|
[After @stlaz's comment above, I renamed the issue to "... 4.6".] I'm returning to this project after some time, but it looks like familiar issues preventing FreeIPA running in the 4.6 container: Google shows this is a known issue at least amongst Ubuntu users, and it will be for Container Linux users as well. If the other participants on this issue think we've stalled out here and wish to close it, that's ok with me. |
|
Which image is this? |
|
Sorry, the fedora-27 image with FreeIPA 4.6. The Fedora-26 image still has FreeIPA 4.4.4, IIRC. |
|
I went ahead and filed issues on Pagure for FreeIPA and Dogtag PKI for the |
When running on memory-constrained systems, the `ipa-server-install` program often fails during the "Configuring certificate server (pki-tomcatd)" stage in FreeIPA 4.5 and 4.6. The memory-intensive dogtag service causes swapping on low-memory systems right after start-up, and especially new certificate operations requested via certmonger can exceed the dbus client default 25 second timeout. This patch changes dbus client timeouts for some such operations to 120 seconds (from the default 25 seconds, IIRC). See more discussion in FreeIPA PR freeipa#1078 [1] and FreeIPA container issue freeipa#157 [2]. Upstream ticket at [3]. [1]: freeipa#1078 [2]: freeipa/freeipa-container#157 [3]: https://pagure.io/freeipa/issue/7213
|
I ended up filing a PR for the I haven't investigated the next failure carefully yet, also during the "configuring certificate server instance" step. I'd appreciate a quick look. (The "Property internaldb.ldapconn.port missing value" error is a red herring.) These logs are from a custom That COPR builds Dogtag PKI with the above PR cherry-picked, and FreeIPA 4.6 with my dbus client timeout patch cherry-picked from master. pki-tomcat.ca.debug.log |
|
After a closer look, I'm not sure the "Property internaldb.ldapconn.port missing value" error is a red herring after all. I don't see any other errors, and the final one didn't have the "Swallow exception in pre-op mode" log message. Here's a full systemd journal. |
|
@zutron, Fedora 27 and Fedora 28 images (Dockerfiles*) seem to be stable in our tests, I've fixed rawhide recently. Do they work for you? You've done a great job bringing the issues you've hit to the respective upstreams so I wonder if there is anything needed on container side? |
|
Oops, sorry about that. |
|
@adelton, Although I haven't succeeded in running the 4.6-based images in a plain CoreOS Docker container yet, I have no reason to believe it's because of this project. I'm trying a new approach in my own FreeIPA project using Atomic instead of CoreOS, and running in Kubernetes instead of plain Docker. That should be a better-proven environment to run FreeIPA. Accordingly, I'm closing this issue. I really appreciate your and your team's support during this odyssey of mine through uncharted territory! |
|
Thanks for the info. |
|
Shouldn't this issue stay open anyway till it is resolved? IMO that way it's easier to track.. |
|
@LorbusChris, an open issue is an indication for someone in the community (for example for me) to investigate something or attempt to fix something in this project (containerization of FreeIPA). What is it exactly that needs to be addressed in this project? |
|
@LorbusChris Maybe I misunderstood something. Others have been reporting FreeIPA 4.6 does run in Docker, so I assumed this problem is something on my end at this point. For example, @adelton's comment says the F27 and F28 containers work in their testing, and the Fedora package database shows FreeIPA v. 4.6 packages in F27 and F28. @adelton Can you confirm that your tests running FreeIPA 4.6 in Docker do succeed? |
|
@zultron, yes, I have containers built from |
When running on memory-constrained systems, the `ipa-server-install` program often fails during the "Configuring certificate server (pki-tomcatd)" stage in FreeIPA 4.5 and 4.6. The memory-intensive dogtag service causes swapping on low-memory systems right after start-up, and especially new certificate operations requested via certmonger can exceed the dbus client default 25 second timeout. This patch changes dbus client timeouts for some such operations to 120 seconds (from the default 25 seconds, IIRC). See more discussion in FreeIPA PR freeipa#1078 [1] and FreeIPA container issue freeipa#157 [2]. Upstream ticket at [3]. [1]: freeipa#1078 [2]: freeipa/freeipa-container#157 [3]: https://pagure.io/freeipa/issue/7213 Reviewed-By: Stanislav Laznicka <slaznick@redhat.com> Reviewed-By: Rob Crittenden <rcritten@redhat.com>
It looks like the most recent FreeIPA version is in the
fedora-25tag, v. 4.4.4. When collaborating with the upstream project, I'm often asked to upgrade to the latest version 4.5 to reproduce a bug.Is there any plan to update the containers?
I started a branch with a new f26 Dockerfile pointing at the official
freeipa-4-5COPR. Of course the upgrade is non-trivial, and while I've solved a few initial problems, the install can still fail nondeterministically in a few places duringipa-server-install.The text was updated successfully, but these errors were encountered: