После обновления Tomcat, во FreeIPA пропала связь с сертификатами. Проблема временно решается добавлением аттрибута ‘secretRequired=“true”’ в AJP Connector.
2021-05-12
Симптомы
Сразу после установки новой FreeIPA на сервер RedOS 7.2, при посещении страницы “Сертификаты”, после долгого ожидания в виде вывески “В процессе”:
Далее появляется баннер с ошибкой:
IPA Error 4301: CertificateOperationError
Certificate operation cannot be completed: Unable to communicate with CMS (500)
Невозможно выполнение:
# ipa cert-show 1
ipa: ERROR: Certificate operation cannot be completed: Unable to communicate with CMS (500)
Диагноз
Строки из ‘journalctl -u pki-tomcatd@pki-tomcat’ указывают причину такого поведения:
SEVERE: Failed to start component [Connector[AJP/1.3-8009]]
Caused by: java.lang.IllegalArgumentException: The AJP Connector is configured with secretRequired="true" but the secret attribute is either null or "". This combination is not valid.
Полный листинг:
# journalctl -u pki-tomcatd@pki-tomcat
May 11 20:05:28 dev-ipa01.test.lan systemd[1]: Starting PKI Tomcat Server pki-tomcat...
May 11 20:05:30 dev-ipa01.test.lan pki-server[1894]: ----------------------------
May 11 20:05:30 dev-ipa01.test.lan pki-server[1894]: pki-tomcat instance migrated
May 11 20:05:30 dev-ipa01.test.lan pki-server[1894]: ----------------------------
May 11 20:05:32 dev-ipa01.test.lan pkidaemon[1923]: -----------------------
May 11 20:05:32 dev-ipa01.test.lan pkidaemon[1923]: Banner is not installed
May 11 20:05:32 dev-ipa01.test.lan pkidaemon[1923]: -----------------------
May 11 20:05:32 dev-ipa01.test.lan pkidaemon[1923]: ----------------------
May 11 20:05:32 dev-ipa01.test.lan pkidaemon[1923]: Enabled all subsystems
May 11 20:05:32 dev-ipa01.test.lan pkidaemon[1923]: ----------------------
May 11 20:05:32 dev-ipa01.test.lan systemd[1]: Started PKI Tomcat Server pki-tomcat.
May 11 20:05:32 dev-ipa01.test.lan server[2064]: Java virtual machine used: /usr/lib/jvm/jre-1.8.0-openjdk/bin/java
May 11 20:05:32 dev-ipa01.test.lan server[2064]: classpath used: /usr/share/tomcat/bin/bootstrap.jar:/usr/share/tomcat/bin
/tomcat-juli.jar:/usr/lib/java/commons-daemon.jar
May 11 20:05:32 dev-ipa01.test.lan server[2064]: main class used: org.apache.catalina.startup.Bootstrap
May 11 20:05:32 dev-ipa01.test.lan server[2064]: flags used: -DRESTEASY_LIB=/usr/share/java/resteasy
May 11 20:05:32 dev-ipa01.test.lan server[2064]: options used: -Dcatalina.base=/var/lib/pki/pki-tomcat -Dcatalina.home=/us
r/share/tomcat -Djava.endorsed.dirs= -Djava.io.tmpdir=/var/lib/pki/pki-tomcat/temp -Djava.util.logging.config.file=/var/lib/pki
/pki-tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.security.manag
er -Djava.security.policy==/var/lib/pki/pki-tomcat/conf/catalina.policy
May 11 20:05:32 dev-ipa01.test.lan server[2064]: arguments used: start
May 11 20:05:40 dev-ipa01.test.lan server[2064]: SEVERE: Failed to start component [Connector[AJP/1.3-8009]]
May 11 20:05:40 dev-ipa01.test.lan server[2064]: org.apache.catalina.LifecycleException: Protocol handler start failed
May 11 20:05:40 dev-ipa01.test.lan server[2064]: at org.apache.catalina.connector.Connector.startInternal(Connector.java:1
075)
May 11 20:05:40 dev-ipa01.test.lan server[2064]: at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
May 11 20:05:40 dev-ipa01.test.lan server[2064]: at org.apache.catalina.core.StandardService.startInternal(StandardService
.java:451)
May 11 20:05:40 dev-ipa01.test.lan server[2064]: at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
May 11 20:05:40 dev-ipa01.test.lan server[2064]: at org.apache.catalina.core.StandardServer.startInternal(StandardServer.j
ava:930)
May 11 20:05:40 dev-ipa01.test.lan server[2064]: at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
May 11 20:05:40 dev-ipa01.test.lan server[2064]: at org.apache.catalina.startup.Catalina.start(Catalina.java:772)
May 11 20:05:40 dev-ipa01.test.lan server[2064]: at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
May 11 20:05:40 dev-ipa01.test.lan server[2064]: at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.jMay 11 20:05:40 dev-ipa01.test.lan server[2064]: at org.apache.catalina.connector.Connector.startInternal(Connector.java:1
075)
May 11 20:05:40 dev-ipa01.test.lan server[2064]: at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
May 11 20:05:40 dev-ipa01.test.lan server[2064]: at org.apache.catalina.core.StandardService.startInternal(StandardService
.java:451)
May 11 20:05:40 dev-ipa01.test.lan server[2064]: at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
May 11 20:05:40 dev-ipa01.test.lan server[2064]: at org.apache.catalina.core.StandardServer.startInternal(StandardServer.j
ava:930)
May 11 20:05:40 dev-ipa01.test.lan server[2064]: at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
May 11 20:05:40 dev-ipa01.test.lan server[2064]: at org.apache.catalina.startup.Catalina.start(Catalina.java:772)
May 11 20:05:40 dev-ipa01.test.lan server[2064]: at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
May 11 20:05:40 dev-ipa01.test.lan server[2064]: at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.j
ava:62)
May 11 20:05:40 dev-ipa01.test.lan server[2064]: at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
May 11 20:05:40 dev-ipa01.test.lan server[2064]: at java.lang.reflect.Method.invoke(Method.java:498)
May 11 20:05:40 dev-ipa01.test.lan server[2064]: at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:342)
May 11 20:05:40 dev-ipa01.test.lan server[2064]: at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:473)
May 11 20:05:40 dev-ipa01.test.lan server[2064]: Caused by: java.lang.IllegalArgumentException: The AJP Connector is configured with secretRequired="true" but the secret attribute is either null or "". This combination is not valid.
May 11 20:05:40 dev-ipa01.test.lan server[2064]: at org.apache.coyote.ajp.AbstractAjpProtocol.start(AbstractAjpProtocol.java:270)
May 11 20:05:40 dev-ipa01.test.lan server[2064]: at org.apache.catalina.connector.Connector.startInternal(Connector.java:1072)
May 11 20:05:40 dev-ipa01.test.lan server[2064]: ... 12 more
Проверим версию установленных в систему Apache и Tomcat:
# rpm -qa httpd
httpd-2.4.29-9.el7.x86_64
# rpm -qa tomcat
tomcat-9.0.44-1.el7.noarch
В Tomcat’е, в предыдущей установленной в системе версии 9.0.13, присутствовала потенциальная уязвимость CVE-2020-1938. Для её решения ввели передачу secret’а при использовании протокола AJP, через который в FreeIPA общаются Apache и DogTag. В результате обновления Tomcat обновился до версии 9.0.44, где предлагается использовать secret, но Apache, и его настройки, остался прежним.
Пока не будет обновлён Apache до версии 2.5, то применяем обходной маневр. Так как установленный в системе Apache имеет версию 2.4, модуль mod_proxy_ajp которой не умеет передавать параметр ‘secret’, то необходимо в соответствующем месте Tomcat’а указать аттрибут, отключающий проверку секрета.
Лечение
Добавить в /etc/pki/pki-tomcat/server.xml в строку коннектора аттрибут ‘secretRequired=“false”’:
<!-- Define an AJP 1.3 Connector on port 8009 -->
<Connector port="8009" protocol="AJP/1.3" redirectPort="8443" address="localhost" secretRequired="false"/>
После чего перезапускаем сервис ‘pki-tomcatd@pki-tomcat’:
sudo systemctl restart pki-tomcatd@pki-tomcat
Результат
# ipa cert-show 1
Issuing CA: ipa
Certificate: MIIDgjCCAmqgAwIBAgIBATANBgkqhkiG9w0BAQsFADAzMREwDwYDVQQKDAhURVNULkxBTjEeMBwGA1UEAwwVQ2VydGlmaWNhdGUgQXV0aG9yaXR5MB4XDTIxMDUwNDE4MzYwNFoXDTQxMDUwNDE4MzYwNFowMzERMA8GA1UECgwIVEVTVC5MQU4xHjAcBgNVBAMMFUNlcnRpZmljYXRlIEF1dGhvcml0eTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBANwBn6gEBu73KL4OT/yuZNMNaL7esYVhQclx4d3lVVAFS7lXk/7Hh91MY39IL/N0teNvTnZn3I9E7VtkgATBvxXzzaLR+iMvDH1OWLIIcLuh4RfR8mVH5PRs1374aXrRRHlrQXElqm7kRt4yWQXb2ibagdnnQxdf2CvstA0HQCDg5DlNprVXu82RWomiacayyadnaZ/KKJmJAHhV4fY0I8tLn6uyVdOspfgA2ZQUaAXOuRtrbGuowegEbCurRO5egDl7GpSR6fOkxiFcnI/jEufatNe3ZQllF3ywT8rfT+oa+52BPCl9J25XrJYw0g71npocVkRi4gY86h0C2LEHeGMCAwEAAaOBoDCBnTAfBgNVHSMEGDAWgBSSy/ljEEs9wsV19d9s4SzwgaQnqjAPBgNVHRMBAf8EBTADAQH/MA4GA1UdDwEB/wQEAwIBxjAdBgNVHQ4EFgQUksv5YxBLPcLFdfXfbOEs8IGkJ6owOgYIKwYBBQUHAQEELjAsMCoGCCsGAQUFBzABhh5odHRwOi8vaXBhLWNhLnRlc3QubGFuL2NhL29jc3AwDQYJKoZIhvcNAQELBQADggEBAJ5t2ZOsPPl121dmCChndCD2aPAWjDZR5FYlEZZN8pk7ApoZ1yDDqPcX/NS+h3Lsj2SqDR1YjljkU6yLdaZy9RJ6NqfDnqt6EIvX5TM6KEuU8ga78tzP0/gJESeFr4oAh1wy/YwQrl2qhaI23SrRQEQXsxdi1s9fBBpAMlszcu5YYgvMimPAoKTdfAvm/nAfH6tcP6Hfvj5aNwQLPqc+S0xrJanAs97mktp84z1rtY5hjnrYKAYGQIE51h/v+f7grnmShiM55qu+A+qxYBqUozf0pNQ/Mo5zB8WongizCYyqjnhBHhdh/E9ca9kmkz7xEdgwxxTqe958WZ/9K+PgtWs=
Subject: CN=Certificate Authority,O=TEST.LAN
Issuer: CN=Certificate Authority,O=TEST.LAN
Not Before: Tue May 04 18:36:04 2021 UTC
Not After: Sat May 04 18:36:04 2041 UTC
Serial number: 1
Serial number (hex): 0x1
Revoked: False
# journalctl -u pki-tomcatd@pki-tomcat
May 11 20:40:20 dev-ipa01.test.lan systemd[1]: Starting PKI Tomcat Server pki-tomcat...
May 11 20:40:20 dev-ipa01.test.lan pki-server[3130]: ----------------------------
May 11 20:40:20 dev-ipa01.test.lan pki-server[3130]: pki-tomcat instance migrated
May 11 20:40:20 dev-ipa01.test.lan pki-server[3130]: ----------------------------
May 11 20:40:20 dev-ipa01.test.lan pkidaemon[3157]: -----------------------
May 11 20:40:20 dev-ipa01.test.lan pkidaemon[3157]: Banner is not installed
May 11 20:40:20 dev-ipa01.test.lan pkidaemon[3157]: -----------------------
May 11 20:40:20 dev-ipa01.test.lan pkidaemon[3157]: ----------------------
May 11 20:40:20 dev-ipa01.test.lan pkidaemon[3157]: Enabled all subsystems
May 11 20:40:20 dev-ipa01.test.lan pkidaemon[3157]: ----------------------
May 11 20:40:20 dev-ipa01.test.lan systemd[1]: Started PKI Tomcat Server pki-tomcat.
May 11 20:40:20 dev-ipa01.test.lan server[3284]: Java virtual machine used: /usr/lib/jvm/jre-1.8.0-openjdk/bin/java
May 11 20:40:20 dev-ipa01.test.lan server[3284]: classpath used: /usr/share/tomcat/bin/bootstrap.jar:/usr/share/tomcat/bin
May 11 20:40:20 dev-ipa01.test.lan server[3284]: main class used: org.apache.catalina.startup.Bootstrap
May 11 20:40:20 dev-ipa01.test.lan server[3284]: flags used: -DRESTEASY_LIB=/usr/share/java/resteasy
May 11 20:40:20 dev-ipa01.test.lan server[3284]: options used: -Dcatalina.base=/var/lib/pki/pki-tomcat -Dcatalina.home=/us
May 11 20:40:20 dev-ipa01.test.lan server[3284]: arguments used: start
Issue
This error can be seen in several different scenarios like what is already described in article: ca-error: Server failed request, will retry: 4301 (RPC failed at server. Certificate operation cannot be completed: Unable to communicate with CMS (Not Found))
The current article describes another scenario in which we can have similar error. It has been seen multiple times. The topology consists on a CRL master and a replica with no CA installed.
Then we do, in the replica, an operation involving a certificate operation like, for instance, a host-del that will show the following error:
ipa host-del myhost.example.com
ipa: ERROR: Certificate operation cannot be completed: Unable to communicate with CMS (Not Found)
Environment
Red Hat Identity Management 4.X
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.
Current Customers and Partners
Log in for full access
Log In
Contents
PKI#
This page contains PKI troubleshooting advice. For other issues,
refer to the index at Troubleshooting.
IPA won’t start, expired certificates#
Where available (>= v4.8.0?), the ipa-cert-fix command can be used
to recover from expired system certificate scenarios. See blog
post.
Starting with IPA 3.0.0 all FreeIPA certificates are
tracked by Certmonger and should be renewed
automatically. In case of problems, see
Certmonger#Manually_renew_a_certificate.
If your Certificate Authority certificate is expired, see CA
Certificate Renewal page .
For v2.0 see
IPA_2x_Certificate_Renewal.
DO NOT RUN ipa-cacert-manage renew in an effort to renew
expirted certificates. This is for renewing the CA certificate only,
which by default is good for 20 years. It is very unlikely you need to
do this.
PKI-tomcatd fails to start#
After an upgrade of IPA packages, pki-tomcatd fails to start. See
Troubleshooting pki-tomcatd fails to
start.
Authentication Errors#
If you see something like
4301 (RPC failed at server. Certificate operation cannot be completed: FAILURE
(Authentication Error)) or Invalid Credential the likely culprit
is the RA agent certificate that IPA uses to authenticate against
PKI.
Use getcert list -d /etc/httpd/alias -n ipaCert to show the current
status of the certificate tracked by Certmonger. It
should be MONITORING.
If it isn’t in MONITORING, or it is and things still aren’t working,
compare the serial number of the certificate with that on other IPA
masters.
If you have the [free]ipa-healthcheck package available in your
distribution run that as it will check for this condition.
For IPA < 4.7.0:
# certutil -L -d /etc/httpd/alias -n ipaCert | grep Serial
For IPA >= 4.7.0:
# openssl x509 -noout -serial -in /var/lib/ipa/ra-agent.pem
The serial number should match the value of the 2nd integer at:
``# ldapsearch -x -h localhost -p 389 -b uid=ipara,ou=People,o=ipaca description ``
(use port 7389 for 2.x servers)
If they are different this suggests that one has been renewed. Only the
most recent is allowed by dogtag. To repair this, go to the master with
the most recent certificate:
For IPA < 4.7.0:
``# certutil -L -d /etc/httpd/alias -n ipaCert -a > /tmp/ra.crt ``
This will export all the certificates. Edit this file and remove all but
the first certificate, You can double-check the result with:
# openssl x509 -text -in /tmp/ra.crt
For IPA >= 4.7.0 the certificate is in /var/lib/ipa/ra-agent.pem
It should have only the cert with the latest serial #.
After removing the unnecessary certificates from the file, for IPA <
4.7.0:
Now add it to your cert database:
# certutil -A -n ipaCert -d /etc/httpd/alias -t u,u,u -a -i /tmp/ra.crt
# service httpd restart
If the certificate is valid and the ou=People entry is ok then check the
PKI logs /var/log/pki or /var/log/pki-ca.
If you see an error like “Failed to connect LDAP server” then try
restarting the tomcat process, either pki-cad (for IPA 3.0) or
pki-tomcatd@pki-tomcat.service.
CRL gets very old#
If the main CRL file containing the list of invalidated certificates is
old and not updated, make sure you check that:
-
There is at least one PKI master server generating the CRL
— see CVE-2012-4546 for instructions. -
CRL generation on that server is not blocked by wrong ownership of
/var/lib/ipa/pki-ca/publish/directory or there are no SELinux
errors. Consult PKIsystemfor details. (related user
case)
External CA renewal with ipa-cacert-manage fails#
The second step of external CA renewal may fail for a number of reasons:
-
Subject name mismatch
The new CA certificate issued by the external CA uses a different
subject name than the old CA certificate. -
Subject name encoding mismatch
The new CA certificate issued by the external CA uses the same
subject name as the old CA certificate, but it is encoded
differently. Some CAs like to re-encode the subject name from
certificate signing requests in certificates they issue. This does
not work well with NSS, which considers subject names to be equal
only if they binary representation is exactly the same. To avoid
the problem, configure the external CA to either respect the CSR’s
encoding, or use UTF8String encoding (which is the FreeIPA
default). Microsoft Certificate Services / AD-CS uses
PrintableString as the default encoding in subject names. See
this GitHub
thread
for how to observe and configure the AD-CS treatment of Subject DN
encoding. -
Subject public key info mismatch
The new CA certificate issued by the external CA uses a different
public / private key pair than the old CA certificate. -
Not a valid CA certificate
The new CA certificate issued by the external CA is either missing
the Basic Constraints extension, or the Basic Constraints
extension indicates that the certificate is not a CA certificate.
Request for enhancement
A strange error is occurring when I try to access my FreeIPA.
Issue
The problem occurs when I try to access the FreeIPA portal.
«The message occurs saying IPA Error 4301: CertificateOperationError»
«Certificate operation cannot be completed: Unable to communicate with CMS (500)»
in Certificate Authority appear:
«cannot connect to ‘https://xyz.xxxxxhq.it:443/ca/rest/account/login’: [SSL: SSL_HANDSHAKE_FAILURE] ssl handshake failure (_ssl.c:1826)»
and if I try to connect with KINIT ADMIN command on the console appear this error:
«kinit: Cannot contact any KDC for realm ‘SUBITOHQ.IT’ while getting initial credentials»
Actual behavior
Serverweb and console with kinit admin doesn’t work. LDAPADMIN tool too.
Version/Release/Distribution
package freeipa-server is not installed
package freeipa-client is not installed
ipa-server-4.6.5-11.el7.centos.3.x86_64
ipa-client-4.6.5-11.el7.centos.3.x86_64
389-ds-base-1.3.9.1-12.el7_7.x86_64
pki-ca-10.5.16-5.el7_7.noarch
krb5-server-1.15.1-37.el7_7.2.x86_64
Additional info:
maybe it’s a problem with CA but how is the process to solve that issue? The fact is that this behavior it’s on a replica FreeIPA server with CA and DOMAIN. There is a resolution or a command to solve that?
Bug #1772450 reported by
gianluca
This bug affects 5 people
| Affects | Status | Importance | Assigned to | Milestone | |
|---|---|---|---|---|---|
|
freeipa (Ubuntu)
|
Fix Released |
High |
Unassigned
|
Bug Description
After having installed FreeIPA server on Ubuntu 18.04 and having sorted out all the other bugs, I still have problems with certificates.
In the web interface, every attempt to select the «Authentication -> Certificates» tab ends with the following error
IPA Error 4301: CertificateOper
Certificate operation cannot be completed: Unable to communicate with CMS (Start tag expected, ‘<‘ not found, line 1, column 1)
The problem also occur with command line utilities. For example, ‘ipa cert-show 1’ returns the error: ‘ipa: ERROR: Certificate operation cannot be completed: Unable to communicate with CMS (500)’