Problems with OpenSearch in Component Pack 8

FStahl · January 17, 2023, 9:52am

Hi everybody,

I can't get OpenSearch to work in Component Pack 8. I'm installing it manually (not via Ansible) to learn the in and outs of the new version. I installed CP 7 and previous versions this way. I'm using all the recommended versions of software for my Kubernetes Cluster on RedHat 7.9.

The problem is with TLS and the private key, that OpenSearch is trying to use. It gets created with the "bootstrap".

Here is the error in the Master Pod:

[2023-01-17T09:38:24,544][ERROR][o.o.b.OpenSearchUncaughtExceptionHandler] [opensearch-cluster-master-0] uncaught exception in thread [main]
org.opensearch.bootstrap.StartupException: java.lang.IllegalStateException: failed to load plugin class [org.opensearch.security.OpenSearchSecurityPlugin]
        at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:182) ~[opensearch-1.3.0.jar:1.3.0]
        at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:169) ~[opensearch-1.3.0.jar:1.3.0]
        at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:100) ~[opensearch-1.3.0.jar:1.3.0]
        at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) ~[opensearch-cli-1.3.0.jar:1.3.0]
        at org.opensearch.cli.Command.main(Command.java:101) ~[opensearch-cli-1.3.0.jar:1.3.0]
        at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:135) ~[opensearch-1.3.0.jar:1.3.0]
        at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:101) ~[opensearch-1.3.0.jar:1.3.0]
Caused by: java.lang.IllegalStateException: failed to load plugin class [org.opensearch.security.OpenSearchSecurityPlugin]
        at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:792) ~[opensearch-1.3.0.jar:1.3.0]
        at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:728) ~[opensearch-1.3.0.jar:1.3.0]
        at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:530) ~[opensearch-1.3.0.jar:1.3.0]
        at org.opensearch.plugins.PluginsService.<init>(PluginsService.java:193) ~[opensearch-1.3.0.jar:1.3.0]
        at org.opensearch.node.Node.<init>(Node.java:396) ~[opensearch-1.3.0.jar:1.3.0]
        at org.opensearch.node.Node.<init>(Node.java:319) ~[opensearch-1.3.0.jar:1.3.0]
        at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242) ~[opensearch-1.3.0.jar:1.3.0]
        at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[opensearch-1.3.0.jar:1.3.0]
        at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:412) ~[opensearch-1.3.0.jar:1.3.0]
        at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:178) ~[opensearch-1.3.0.jar:1.3.0]
        ... 6 more
Caused by: java.lang.reflect.InvocationTargetException
        at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
        at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
        at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
        at java.lang.reflect.Constructor.newInstance(Constructor.java:490) ~[?:?]
        at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:783) ~[opensearch-1.3.0.jar:1.3.0]
        at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:728) ~[opensearch-1.3.0.jar:1.3.0]
        at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:530) ~[opensearch-1.3.0.jar:1.3.0]
        at org.opensearch.plugins.PluginsService.<init>(PluginsService.java:193) ~[opensearch-1.3.0.jar:1.3.0]
        at org.opensearch.node.Node.<init>(Node.java:396) ~[opensearch-1.3.0.jar:1.3.0]
        at org.opensearch.node.Node.<init>(Node.java:319) ~[opensearch-1.3.0.jar:1.3.0]
        at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242) ~[opensearch-1.3.0.jar:1.3.0]
        at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[opensearch-1.3.0.jar:1.3.0]
        at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:412) ~[opensearch-1.3.0.jar:1.3.0]
        at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:178) ~[opensearch-1.3.0.jar:1.3.0]
        ... 6 more
Caused by: org.opensearch.OpenSearchSecurityException: Error while initializing transport SSL layer from PEM: java.lang.IllegalArgumentException: File does not contain valid private key: /usr/share/opensearch/config/certs/opensearch-transport.key
        at org.opensearch.security.ssl.DefaultSecurityKeyStore.initTransportSSLConfig(DefaultSecurityKeyStore.java:419) ~[?:?]
        at org.opensearch.security.ssl.DefaultSecurityKeyStore.initSSLConfig(DefaultSecurityKeyStore.java:258) ~[?:?]
uncaught exception in thread [main]
        at org.opensearch.security.ssl.DefaultSecurityKeyStore.<init>(DefaultSecurityKeyStore.java:179) ~[?:?]
        at org.opensearch.security.ssl.OpenSearchSecuritySSLPlugin.<init>(OpenSearchSecuritySSLPlugin.java:218) ~[?:?]
        at org.opensearch.security.OpenSearchSecurityPlugin.<init>(OpenSearchSecurityPlugin.java:252) ~[?:?]
        at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
        at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
        at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
        at java.lang.reflect.Constructor.newInstance(Constructor.java:490) ~[?:?]
        at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:783) ~[opensearch-1.3.0.jar:1.3.0]
        at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:728) ~[opensearch-1.3.0.jar:1.3.0]
        at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:530) ~[opensearch-1.3.0.jar:1.3.0]
        at org.opensearch.plugins.PluginsService.<init>(PluginsService.java:193) ~[opensearch-1.3.0.jar:1.3.0]
        at org.opensearch.node.Node.<init>(Node.java:396) ~[opensearch-1.3.0.jar:1.3.0]
        at org.opensearch.node.Node.<init>(Node.java:319) ~[opensearch-1.3.0.jar:1.3.0]
        at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242) ~[opensearch-1.3.0.jar:1.3.0]
        at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[opensearch-1.3.0.jar:1.3.0]
        at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:412) ~[opensearch-1.3.0.jar:1.3.0]
        at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:178) ~[opensearch-1.3.0.jar:1.3.0]
        ... 6 more
Caused by: java.lang.IllegalArgumentException: File does not contain valid private key: /usr/share/opensearch/config/certs/opensearch-transport.key
        at io.netty.handler.ssl.SslContextBuilder.keyManager(SslContextBuilder.java:386) ~[?:?]
        at io.netty.handler.ssl.SslContextBuilder.forServer(SslContextBuilder.java:120) ~[?:?]
        at org.opensearch.security.ssl.DefaultSecurityKeyStore.buildSSLServerContext(DefaultSecurityKeyStore.java:869) ~[?:?]
        at org.opensearch.security.ssl.DefaultSecurityKeyStore.initTransportSSLConfig(DefaultSecurityKeyStore.java:405) ~[?:?]
        at org.opensearch.security.ssl.DefaultSecurityKeyStore.initSSLConfig(DefaultSecurityKeyStore.java:258) ~[?:?]
        at org.opensearch.security.ssl.DefaultSecurityKeyStore.<init>(DefaultSecurityKeyStore.java:179) ~[?:?]
        at org.opensearch.security.ssl.OpenSearchSecuritySSLPlugin.<init>(OpenSearchSecuritySSLPlugin.java:218) ~[?:?]
        at org.opensearch.security.OpenSearchSecurityPlugin.<init>(OpenSearchSecurityPlugin.java:252) ~[?:?]
        at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
        at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
        at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
        at java.lang.reflect.Constructor.newInstance(Constructor.java:490) ~[?:?]
        at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:783) ~[opensearch-1.3.0.jar:1.3.0]
        at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:728) ~[opensearch-1.3.0.jar:1.3.0]
        at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:530) ~[opensearch-1.3.0.jar:1.3.0]
        at org.opensearch.plugins.PluginsService.<init>(PluginsService.java:193) ~[opensearch-1.3.0.jar:1.3.0]
        at org.opensearch.node.Node.<init>(Node.java:396) ~[opensearch-1.3.0.jar:1.3.0]
        at org.opensearch.node.Node.<init>(Node.java:319) ~[opensearch-1.3.0.jar:1.3.0]
        at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242) ~[opensearch-1.3.0.jar:1.3.0]
        at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[opensearch-1.3.0.jar:1.3.0]
        at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:412) ~[opensearch-1.3.0.jar:1.3.0]
        at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:178) ~[opensearch-1.3.0.jar:1.3.0]
        ... 6 more
Caused by: java.security.spec.InvalidKeySpecException: Cannot retrieve the PKCS8EncodedKeySpec
        at javax.crypto.EncryptedPrivateKeyInfo.getKeySpec(EncryptedPrivateKeyInfo.java:258) ~[?:?]
        at io.netty.handler.ssl.SslContext.generateKeySpec(SslContext.java:1090) ~[?:?]
        at io.netty.handler.ssl.SslContext.getPrivateKeyFromByteBuffer(SslContext.java:1144) ~[?:?]
        at io.netty.handler.ssl.SslContext.toPrivateKey(SslContext.java:1123) ~[?:?]
        at io.netty.handler.ssl.SslContextBuilder.keyManager(SslContextBuilder.java:384) ~[?:?]
        at io.netty.handler.ssl.SslContextBuilder.forServer(SslContextBuilder.java:120) ~[?:?]
        at org.opensearch.security.ssl.DefaultSecurityKeyStore.buildSSLServerContext(DefaultSecurityKeyStore.java:869) ~[?:?]
        at org.opensearch.security.ssl.DefaultSecurityKeyStore.initTransportSSLConfig(DefaultSecurityKeyStore.java:405) ~[?:?]
        at org.opensearch.security.ssl.DefaultSecurityKeyStore.initSSLConfig(DefaultSecurityKeyStore.java:258) ~[?:?]
        at org.opensearch.security.ssl.DefaultSecurityKeyStore.<init>(DefaultSecurityKeyStore.java:179) ~[?:?]
        at org.opensearch.security.ssl.OpenSearchSecuritySSLPlugin.<init>(OpenSearchSecuritySSLPlugin.java:218) ~[?:?]
        at org.opensearch.security.OpenSearchSecurityPlugin.<init>(OpenSearchSecurityPlugin.java:252) ~[?:?]
        at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
        at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
        at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
        at java.lang.reflect.Constructor.newInstance(Constructor.java:490) ~[?:?]
        at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:783) ~[opensearch-1.3.0.jar:1.3.0]
        at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:728) ~[opensearch-1.3.0.jar:1.3.0]
        at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:530) ~[opensearch-1.3.0.jar:1.3.0]
        at org.opensearch.plugins.PluginsService.<init>(PluginsService.java:193) ~[opensearch-1.3.0.jar:1.3.0]
        at org.opensearch.node.Node.<init>(Node.java:396) ~[opensearch-1.3.0.jar:1.3.0]
        at org.opensearch.node.Node.<init>(Node.java:319) ~[opensearch-1.3.0.jar:1.3.0]
        at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242) ~[opensearch-1.3.0.jar:1.3.0]
        at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[opensearch-1.3.0.jar:1.3.0]
        at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:412) ~[opensearch-1.3.0.jar:1.3.0]
        at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:178) ~[opensearch-1.3.0.jar:1.3.0]
        ... 6 more
Caused by: javax.crypto.BadPaddingException: Given final block not properly padded. Such issues can arise if a bad key is used during decryption.
        at com.sun.crypto.provider.CipherCore.unpad(CipherCore.java:975) ~[?:?]
        at com.sun.crypto.provider.CipherCore.fillOutputBuffer(CipherCore.java:1056) ~[?:?]
        at com.sun.crypto.provider.CipherCore.doFinal(CipherCore.java:853) ~[?:?]
        at com.sun.crypto.provider.PKCS12PBECipherCore.implDoFinal(PKCS12PBECipherCore.java:408) ~[?:?]
        at com.sun.crypto.provider.PKCS12PBECipherCore$PBEWithSHA1AndDESede.engineDoFinal(PKCS12PBECipherCore.java:440) ~[?:?]
        at javax.crypto.Cipher.doFinal(Cipher.java:2202) ~[?:?]
        at javax.crypto.EncryptedPrivateKeyInfo.getKeySpec(EncryptedPrivateKeyInfo.java:253) ~[?:?]
        at io.netty.handler.ssl.SslContext.generateKeySpec(SslContext.java:1090) ~[?:?]
        at io.netty.handler.ssl.SslContext.getPrivateKeyFromByteBuffer(SslContext.java:1144) ~[?:?]
        at io.netty.handler.ssl.SslContext.toPrivateKey(SslContext.java:1123) ~[?:?]
        at io.netty.handler.ssl.SslContextBuilder.keyManager(SslContextBuilder.java:384) ~[?:?]
        at io.netty.handler.ssl.SslContextBuilder.forServer(SslContextBuilder.java:120) ~[?:?]
        at org.opensearch.security.ssl.DefaultSecurityKeyStore.buildSSLServerContext(DefaultSecurityKeyStore.java:869) ~[?:?]
        at org.opensearch.security.ssl.DefaultSecurityKeyStore.initTransportSSLConfig(DefaultSecurityKeyStore.java:405) ~[?:?]
        at org.opensearch.security.ssl.DefaultSecurityKeyStore.initSSLConfig(DefaultSecurityKeyStore.java:258) ~[?:?]
        at org.opensearch.security.ssl.DefaultSecurityKeyStore.<init>(DefaultSecurityKeyStore.java:179) ~[?:?]
        at org.opensearch.security.ssl.OpenSearchSecuritySSLPlugin.<init>(OpenSearchSecuritySSLPlugin.java:218) ~[?:?]
        at org.opensearch.security.OpenSearchSecurityPlugin.<init>(OpenSearchSecurityPlugin.java:252) ~[?:?]
        at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
        at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
        at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
        at java.lang.reflect.Constructor.newInstance(Constructor.java:490) ~[?:?]
        at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:783) ~[opensearch-1.3.0.jar:1.3.0]
        at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:728) ~[opensearch-1.3.0.jar:1.3.0]
        at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:530) ~[opensearch-1.3.0.jar:1.3.0]
        at org.opensearch.plugins.PluginsService.<init>(PluginsService.java:193) ~[opensearch-1.3.0.jar:1.3.0]
        at org.opensearch.node.Node.<init>(Node.java:396) ~[opensearch-1.3.0.jar:1.3.0]
        at org.opensearch.node.Node.<init>(Node.java:319) ~[opensearch-1.3.0.jar:1.3.0]
        at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242) ~[opensearch-1.3.
        at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[opensearch-1.3.0.j
        at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:412) ~[opensearch-1.3.0.ja
        at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:178) ~[opensearch-1.3.0.
        ... 6 more
java.lang.IllegalStateException: failed to load plugin class [org.opensearch.security.OpenSe
Likely root cause: javax.crypto.BadPaddingException: Given final block not properly padded. 
        at java.base/com.sun.crypto.provider.CipherCore.unpad(CipherCore.java:975)
        at java.base/com.sun.crypto.provider.CipherCore.fillOutputBuffer(CipherCore.java:105
        at java.base/com.sun.crypto.provider.CipherCore.doFinal(CipherCore.java:853)
        at java.base/com.sun.crypto.provider.PKCS12PBECipherCore.implDoFinal(PKCS12PBECipher
        at java.base/com.sun.crypto.provider.PKCS12PBECipherCore$PBEWithSHA1AndDESede.engine
        at java.base/javax.crypto.Cipher.doFinal(Cipher.java:2202)
        at java.base/javax.crypto.EncryptedPrivateKeyInfo.getKeySpec(EncryptedPrivateKeyInfo
        at io.netty.handler.ssl.SslContext.generateKeySpec(SslContext.java:1090)
        at io.netty.handler.ssl.SslContext.getPrivateKeyFromByteBuffer(SslContext.java:1144)
        at io.netty.handler.ssl.SslContext.toPrivateKey(SslContext.java:1123)
        at io.netty.handler.ssl.SslContextBuilder.keyManager(SslContextBuilder.java:384)
        at io.netty.handler.ssl.SslContextBuilder.forServer(SslContextBuilder.java:120)
        at org.opensearch.security.ssl.DefaultSecurityKeyStore.buildSSLServerContext(Default
        at org.opensearch.security.ssl.DefaultSecurityKeyStore.initTransportSSLConfig(Defaul
        at org.opensearch.security.ssl.DefaultSecurityKeyStore.initSSLConfig(DefaultSecurity
        at org.opensearch.security.ssl.DefaultSecurityKeyStore.<init>(DefaultSecurityKeyStor
        at org.opensearch.security.ssl.OpenSearchSecuritySSLPlugin.<init>(OpenSearchSecurity
        at org.opensearch.security.OpenSearchSecurityPlugin.<init>(OpenSearchSecurityPlugin.
        at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native 
        at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeCo
        at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(Dele
        at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
        at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:783)
        at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:728)
        at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:530)
        at org.opensearch.plugins.PluginsService.<init>(PluginsService.java:193)
        at org.opensearch.node.Node.<init>(Node.java:396)
        at org.opensearch.node.Node.<init>(Node.java:319)
        at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242)
        at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242)
        <<<truncated>>>

The private key looks good to me:

"opensearch-transport.key": "LS0tLS1CRUdJTiBFTkNSWVBURUQgUFJJVkFURSBLRVktLS0tLQpNSUlFNmpBY0Jnb3Foa2lHOXcwQkRBRURNQTRFQ0Q2MFBIWDlnendUQWdJSUFBU0NCTWpUVEU2YktHd2tIV2NBCnFkNFg5THNDaUZJYVY4bEg4MXB5WGJxQ25QRnpRRDRmcndDWVVoZm9oNHZKTTBlMitBRXFOS1hJSTdzOHZESEsKME9FdXhLc09oVDgzM0Mxa2FoeS92L2dFS082ZzQzY3FQMG9FOUhwTGlyWWdYQkxDL096S1NwWmdmdVpsUUZSWAorQ3paL2E5eXJhNHBkZkdJTGdTTThBak5jSHpQa29yUzhjZSszU1NSSlhwMjRLZHNpV01naWwyL2VPWHI3L01qCkNZRlpmamVKeFB3cHhGbG5jS3o5akMvNGZzL1VNTnFEbEtieWowUjlDbDFXSkhzWk9RQ0FWUGVoTE1ITm9kZTkKV1QySXBYVTBpWVZZdXVIYTN6cFYvcVdQWDFIYWpTbTFXUWx5bFhCT0xNUWhzTTNZcERoVXZZM3dwK0pkNVE4QQpKRE85KzJ2b0U0TFV1RnRJdjdwby9UK1NkNGVsREwveVdNN3VaZjVWbU5XNlRobnZjd3k5NTQ5Q0VXbWFDdm51Ck93UWpFQzM1cXdFT0F3RjFHMnE5LzlSYmFBRDdRczNZcWtLRFZBZ091ZGFlS3FUdHBHb3pJRC8vZzUxWDFQd3QKaWdZRW1vbG9DN09lcHZQZmRkcEZ6TWxhR3p2MTNtMmhzWlpQSU10SmpLRkl0dWFzd2xhTVJGUWFEVUt5ZHNyMApTZHZoVHozZDRrZHFlbFR1aG4vRzdSOVJEV2VJMTBVRmRqMTJHYktZaHkvOWlXV2FVeG1RbzZZMVdDN3l4U2twClVTRWFxazRwR2lFeFp1elNRaUpnTUVaL0YrZDlYcXRYSDFCUS9zaHQrY2VneDN5bzJJMFZRYVlVMGN5emlCVXAKZ2RncW92aHl2Z3FtVnlocGUrNFR6ZHlvcnloN3NIMGZYWVNQK21zNW5POEprMWo2emdxSEUxaUdjSmYzNGNQWgppR3JhN243T2J0NThBOU5ZZWo4TWh5ZjFpM1hFWHRwNDF3QlMreW9XY0JpZGVzQWJwVnExVVF2ZzZ4Wk91b2JZCjdzaUZ2TEJER1BYVGM0RCtwcXVlN1hiUVNmRmw3M20rN2puTWF4bUsyZTdaM05BSFJjYWRiZWhmNnF4ODdmZ1cKUUNSM0NWM2s3YWN5NHQweXZCM3A3TFJQQW9ZcWN2NjdLQXhjdDh6Vnltcjh3TkFJZTloZDhYNHhqTnhvdXF4TAovZThzWVY5eFFhb3h4dXUzZW1uQmZuMkJ0N3VNVEExWmV2aUtaQ2Y0M25tVFczRGdwTjVVT1o1Z3JZVndtMEtsCmxwVnJZb1lMcVR1eFZPd3I1a001QitpcVVMdFVjdTdyM2FILzVadFF0VmFqbGNUMllaL3c0S1VDY3hDMGJHK2gKd1R4cmhSUzFuVXBOMGJRcHJBNEZTL1pDdkUwQUl6VzhVcjFFcW5UanVFV1BNd2xPd0g4dFpJVVNOMFh4aDdncwpNekNVM0xXTGxHcDQ3WkhESk5mOGFxWFdHekw3MHJnRyt2bUJJdVpYTER3STRPOUZUa1l4WllyM2tUMlJUOWpyCkdWR2hvbU1wMzlhbFZKSHZTM3FTMTQwZ1VkNG5kSTQwcC9rQWZJR2pvZHBDNUhYRzZrMHV0T0pQQm4xOGs5V20KWkF2b0FrcGFaNWlOaFY5MTNacEFmTFR3c2daT0tXbHBUeFQwajVYQWJrR3h1dUxycFNFOVZjcFkwR2hsQnp1RApaaW9RODFiQnducEt0VGhlWG40QnRqMDlGN0FpTU5KendFQ2dOTm1ncHJNZVZ0NW0ybHFBdWE4QjhUTXhTdkV5ClhKNTVEdnNyM3MxaWdINWpDckVYM2xaUmY5U0JpOWx2SE05TWRUSjZoOGNwWW1yYkxBZS9FN1VCTEVuQ0ovY1UKMXVFNEVZOVlnYVNwUkxIdnV1bTlTemlsbXZLY3dSKzhjNzE2MUJwSUdjZ1VIbmMzekdQam92YkgrQkZCRXhuMwpwZW5SSFdXbEg3V2tGTysyUm9TYkRVWTdsM1FYMEN6WjFaWXBjY3I3QUEyVVNPY0ZWV2VGSjlvM24xNnhBaGdjCjJrTTRFbVF6Z3VqRjB3dnFHNTBXNVByQWVhRmNFK1VWTjFNRzlIRDlFcGdzVWdOcHZCZDRWcDFoTnh2MjlWcC8KbzMyNzZISm9oREZXa1AvcGxWaz0KLS0tLS1FTkQgRU5DUllQVEVEIFBSSVZBVEUgS0VZLS0tLS0K"

Did anybody else have a similiar problem or a solution for this?

I tried a few things already, but always ended up with the same error.

Thanks

Florian

anon-2231 · January 17, 2023, 10:15am

Hi Florian,

Well I have the same problem. I don't have a real solution, but at least a workaround.

After deploying a helm chart, I edited the corresponding configmap (e.g. opensearch-cluster-master-config) and replaced all occurrences of the standard "pemkey_password: password" with my custom password that was set during bootstrap.

For the my testing deployment, this solved the problem for now. However, according to support, it should get set via pemkeyPass variable during deployment, but that doesn't work for me.

Would be happy, if there is a better solution.

Regards,
Daniel

stoeps · January 17, 2023, 10:43am

Hi,

just downloaded the OpenSearch helm chart from Harbor and checked values.yaml in the tar file.

There is no variable defined for the password of pemkey.

The opensearch.yml is hardcoded in values.yaml with pemkey_password: password.

The whole helm chart does not contain a variable with pemkey_pass:

grep pemkey * -ir

ci/ci-values.yaml: pemkey_filepath: esnode-key.pem

ci/ci-values.yaml: pemkey_filepath: esnode-key.pem

ci/ci-rbac-enabled-values.yaml: pemkey_filepath: esnode-key.pem

ci/ci-rbac-enabled-values.yaml: pemkey_filepath: esnode-key.pem

ci/ci-ingress-class-name-values.yaml: pemkey_filepath: esnode-key.pem

ci/ci-ingress-class-name-values.yaml: pemkey_filepath: esnode-key.pem

values.yaml: \ pemkey_filepath: certs/opensearch-transport.key\n pemtrustedcas_filepath:\

values.yaml: \ certs/chain-ca.pem\n enforce_hostname_verification: false\n pemkey_password:\

values.yaml: \ true\n pemcert_filepath: certs/opensearch-http.crt.pem\n pemkey_filepath:\

values.yaml: \ pemkey_password: password\n clientauth_mode: REQUIRE\n allow_unsafe_democertificates:\

So if you set a password in bootstrap != password it will fail to read the pemkey.

Hugs

Stoeps

FStahl · January 17, 2023, 10:54am

Thank you for the workaround Daniel!

It works now. I had a suspicion it was related to the password (I too did set "pemPasskey"), but didn't really properly investigate.

Thanks to Stoeps for investigating further. That explains the problem. And the "password" in the ConfigMap.