Openvpn ошибка tls error

I am configuring OpenVPN 2.3.6-1 on my Arch Linux server in order to encrypt SMB traffic over the public Internet. When I test the setup on one of my Linux virtual machine clients, I get the error: TLS Error: TLS handshake failed.

I quickly read (OpenVPN on OpenVZ TLS Error: TLS handshake failed (google suggested solutions not helping)) and tried to switch from the default UDP to TCP, but that only caused the client to repeatedly report that the connection timed out. I also tried disabling the cipher and TLS authentication, but that caused the server to fail with Assertion failed at crypto_openssl.c:523. In both instances, the required changes were made to both the client and server configurations.

I have been following the instructions at (https://wiki.archlinux.org/index.php/OpenVPN) to set up OpenVPN and the instructions at (https://wiki.archlinux.org/index.php/Create_a_Public_Key_Infrastructure_Using_the_easy-rsa_Scripts) to create the keys and certificates. The only deviations I have made from these instructions have been specifying my own computers’ names and their corresponding key/certificate file names.

See also my original question about securing SMB traffic over the Internet: (Simple encryption for Samba shares)

Can anybody explain how I can solve this issue?

Details:

Server: Arch Linux (up to date) connected directly to gateway via ethernet cable. No iptables.

Client: Arch Linux (up to date) virtual machine on VirtualBox 4.3.28r100309 Windows 8.1 host, bridged network adapter. No iptables. Windows Firewall disabled.

Gateway: Port forwarding for port 1194 enabled, no firewall restrictions.

Here are the configuration files on the server and client, respectively. I created these according to the instructions on the Arch Wiki.

/etc/openvpn/server.conf (Non-comment lines only):

port 1194
proto udp
dev tun
ca /etc/openvpn/ca.crt
cert /etc/openvpn/server-name.crt
key /etc/openvpn/server-name.key
dh /etc/openvpn/dh2048.pem
server 10.8.0.0 255.255.255.0
ifconfig-pool-persist ipp.txt
keepalive 10 120
tls-auth /etc/openvpn/ta.key 0
comp-lzo
user nobody
group nobody
persist-key
persist-tun
status openvpn-status.log
verb 3

/etc/openvpn/client.conf (Non-comment lines only):

client
dev tun
proto udp
remote [my public IP here] 1194
resolv-retry infinite
nobind
user nobody
group nobody
persist-key
persist-tun
ca /etc/openvpn/ca.crt
cert /etc/openvpn/client-name.crt
key /etc/openvpn/client-name.key
remote-cert-tls server
tls-auth /etc/openvpn/ta.key 1
comp-lzo
verb 3

Here are the outputs of running openvpn on the machines with the above configurations. I started the server first, then the client.

The output of openvpn /etc/openvpn/server.conf on the server:

Thu Jul 30 17:02:53 2015 OpenVPN 2.3.6 x86_64-unknown-linux-gnu [SSL (OpenSSL)] [LZO] [EPOLL] [MH] [IPv6] built on Dec  2 2014
Thu Jul 30 17:02:53 2015 library versions: OpenSSL 1.0.2d 9 Jul 2015, LZO 2.09
Thu Jul 30 17:02:53 2015 NOTE: your local LAN uses the extremely common subnet address 192.168.0.x or 192.168.1.x.  Be aware that this might create routing conflicts if you connect to the VPN server from public locations such as internet cafes that use the same subnet.
Thu Jul 30 17:02:53 2015 Diffie-Hellman initialized with 2048 bit key
Thu Jul 30 17:02:53 2015 Control Channel Authentication: using '/etc/openvpn/ta.key' as a OpenVPN static key file
Thu Jul 30 17:02:53 2015 Outgoing Control Channel Authentication: Using 160 bit message hash 'SHA1' for HMAC authentication
Thu Jul 30 17:02:53 2015 Incoming Control Channel Authentication: Using 160 bit message hash 'SHA1' for HMAC authentication
Thu Jul 30 17:02:53 2015 Socket Buffers: R=[212992->131072] S=[212992->131072]
Thu Jul 30 17:02:53 2015 ROUTE_GATEWAY 192.168.0.1/255.255.255.0 IFACE=enp5s0 HWADDR=##:##:##:##:##:##
Thu Jul 30 17:02:53 2015 TUN/TAP device tun0 opened
Thu Jul 30 17:02:53 2015 TUN/TAP TX queue length set to 100
Thu Jul 30 17:02:53 2015 do_ifconfig, tt->ipv6=0, tt->did_ifconfig_ipv6_setup=0
Thu Jul 30 17:02:53 2015 /usr/bin/ip link set dev tun0 up mtu 1500
Thu Jul 30 17:02:53 2015 /usr/bin/ip addr add dev tun0 local 10.8.0.1 peer 10.8.0.2
Thu Jul 30 17:02:53 2015 /usr/bin/ip route add 10.8.0.0/24 via 10.8.0.2
Thu Jul 30 17:02:53 2015 GID set to nobody
Thu Jul 30 17:02:53 2015 UID set to nobody
Thu Jul 30 17:02:53 2015 UDPv4 link local (bound): [undef]
Thu Jul 30 17:02:53 2015 UDPv4 link remote: [undef]
Thu Jul 30 17:02:53 2015 MULTI: multi_init called, r=256 v=256
Thu Jul 30 17:02:53 2015 IFCONFIG POOL: base=10.8.0.4 size=62, ipv6=0
Thu Jul 30 17:02:53 2015 IFCONFIG POOL LIST
Thu Jul 30 17:02:53 2015 Initialization Sequence Completed

The output of openvpn /etc/openvpn/client.conf on the client:

Thu Jul 30 21:03:02 2015 OpenVPN 2.3.6 x86_64-unknown-linux-gnu [SSL (OpenSSL)] [LZO] [EPOLL] [MH] [IPv6] built on Dec  2 2014
Thu Jul 30 21:03:02 2015 library versions: OpenSSL 1.0.2d 9 Jul 2015, LZO 2.09
Thu Jul 30 21:03:02 2015 WARNING: file '/etc/openvpn/client-name.key' is group or others accessible
Thu Jul 30 21:03:02 2015 WARNING: file '/etc/openvpn/ta.key' is group or others accessible
Thu Jul 30 21:03:02 2015 Control Channel Authentication: using '/etc/openvpn/ta.key' as a OpenVPN static key file
Thu Jul 30 21:03:02 2015 Outgoing Control Channel Authentication: Using 160 bit message hash 'SHA1' for HMAC authentication
Thu Jul 30 21:03:02 2015 Incoming Control Channel Authentication: Using 160 bit message hash 'SHA1' for HMAC authentication
Thu Jul 30 21:03:02 2015 Socket Buffers: R=[212992->131072] S=[212992->131072]
Thu Jul 30 21:03:02 2015 NOTE: UID/GID downgrade will be delayed because of --client, --pull, or --up-delay
Thu Jul 30 21:03:02 2015 UDPv4 link local: [undef]
Thu Jul 30 21:03:02 2015 UDPv4 link remote: [AF_INET][my public IP here]:1194
Thu Jul 30 21:04:02 2015 TLS Error: TLS key negotiation failed to occur within 60 seconds (check your network connectivity)
Thu Jul 30 21:04:02 2015 TLS Error: TLS handshake failed
Thu Jul 30 21:04:02 2015 SIGUSR1[soft,tls-error] received, process restarting
Thu Jul 30 21:04:02 2015 Restart pause, 2 second(s)

@Nixellion hi, have You found the cause of this issue?
I stopped iptables, checked /etc/hosts.deny and i am not aware about other blocking software (did «ps aux»).

I tried to connect VPN from my home PC and phone which is using different internet/IP, but can not connect, seeing same status messages as mentioned. What else to try to discover why vpn server is not proceeding?
@Nyr
Hi, i have new log entries from the server log, these are interesting and shows the progress of me connecting to the server, please can you check it?

Wed Apr 25 12:00:24 2018 us=309178 myip:44173 Local Options String (VER=V4): ‘V4,dev-type tun,link-mtu 1558,tun-mtu 1500,proto UDPv4,comp-lzo,keydir 0,cipher AES-256-CBC,auth SHA1,keysize 256,tls-auth,key-method 2,tls-server’
Wed Apr 25 12:00:24 2018 us=309195 myip:44173 Expected Remote Options String (VER=V4): ‘V4,dev-type tun,link-mtu 1558,tun-mtu 1500,proto UDPv4,comp-lzo,keydir 1,cipher AES-256-CBC,auth SHA1,keysize 256,tls-auth,key-method 2,tls-client’
RWed Apr 25 12:00:24 2018 us=309288 myip:44173 TLS: Initial packet from [AF_INET]myip:44173, sid=8289a011 23bf96ff
WRRWWRRWRWed Apr 25 12:00:24 2018 us=513010 myip:44173 VERIFY ERROR: depth=0, error=CRL has expired: CN=client
Wed Apr 25 12:00:24 2018 us=513151 myip:44173 OpenSSL: error:140890B2:SSL routines:SSL3_GET_CLIENT_CERTIFICATE:no certificate returned
Wed Apr 25 12:00:24 2018 us=513172 myip:44173 TLS_ERROR: BIO read tls_read_plaintext error
Wed Apr 25 12:00:24 2018 us=513186 myip:44173 TLS Error: TLS object -> incoming plaintext read error
Wed Apr 25 12:00:24 2018 us=513198 myip:44173 TLS Error: TLS handshake failed
Wed Apr 25 12:00:24 2018 us=513312 myip:44173 SIGUSR1[soft,tls-error] received, client-instance restarting

It do not seem to me like there is NAT misbehaving/firewall problem like you mentioned, because above log entries shows i was able to connect, but VPN server have some problem with TLS or OpenSSL?

Wondering how to resolve TLS key negotiation failed error in OpenVPN? We can help you.

As part of our Server Management Services, we assist our customers with several OpenVPN queries.

Today, let us see how our Support techs resolve this error.

How to resolve TLS key negotiation failed error in OpenVPN?

First and foremost, to diagnose problems with an OpenVPN server or client, it is helpful to look at the log files.

Locating the server log files

The log files are located in specific areas on your computer systems.

Log files are the place to check whenever you’re having any problems making a connection with an OpenVPN client program to the OpenVPN Access Server.

On the OpenVPN Access Server there is the server side log:

/var/log/openvpnas.log /var/log/openvpnas.node.log (in case of a failover setup)

In the event that you are having problems with starting the Access Server or certain portions of it, for example the web services, then it may be useful to stop the Access Server service.

Then, move the log file aside, then start the Access Server service, and stop it again immediately.

This creates a new clean log file that contains the startup and shutdown sequence of the Access Server and no other extraneous information.

This makes analysis of the log file much easier.

To do so use these commands in order:

service openvpnas stop
mv /var/log/openvpnas.log /var/log/openvpnas.log.old
service openvpnas start
service openvpnas stop

You can then grab the /var/log/openvpnas.log file for analysis and start the Access Server again:

service openvpnas start

Locating the client log files

Log file location for the OpenVPN Connect Client for Windows:

C:\Program Files (x86)\OpenVPN Technologies\OpenVPN Client\etc\log\openvpn_(unique_name).log

The OpenVPN Connect Client for Mac:

/Library/Application Support/OpenVPN/log/openvpn_(unique_name).log

To get to the /Library folder, open Finder and in the menu at the top choose Go followed by Go to folder and then enter the path /Library to get into that directory.

You can then go to the correct folder and look up the log file.

Please also note that the OpenVPN Connect Client for Macintosh will have permissions set on the log file so that you cannot normally open it.

To bypass this, right click the log file and choose the Get info option in the menu.

Then at the bottom, under Sharing & Permissions, you will be able to use the yellow padlock icon to unlock the settings and to give everyone read access.

Then, you will be able to open the log file with a right click and selecting Open with and then choosing something like Text editor to view the contents of the log file.

TLS key negotiation failed error

Typical error will look as shown below:

TLS Error: TLS key negotiation failed to occur within 60 seconds (check your network connectivity)

This particular error can have multiple different causes as it is a fairly generic error message.

A possible explanation is that the client program is old and supports only TLS 1.0, but the server is expecting TLS level 1.1 or higher.

To see if this is the case log on to the server and check the server side log file.

The chances are high that your client program is an older version, like version 2.2 or older, and that it doesn’t know how to handle a modern TLS minimum level requirement, when you see messages that look like this on the server side:

OpenSSL: error:140760FC:SSL routines:SSL23_GET_CLIENT_HELLO:unknown protocol’
TLS_ERROR: BIO read tls_read_plaintext error’
TLS Error: TLS object -> incoming plaintext read error’
TLS Error: TLS handshake failed’
SIGUSR1[soft,tls-error] received, client-instance restarting’

The solution to this particular problem is to upgrade the client software to the latest version.

Another possible explanation is that the settings regarding TLS minimum requirement level have been altered but the OpenVPN client is using an older copy of the connection profile which has incorrect instructions.

The settings on the client and the server must match for the connection to be successful.

In this situation installing a new copy of the configuration profile will solve the issue.

A complete uninstall, redownload, and reinstall of the OpenVPN Connect Client should take care of that for you.

And yet another possible explanation is that there is a blockage in place in a firewall or at the Internet service provider that is blocking or interfering with the TLS handshake in some way.

[Stuck in between? We’d be glad to assist you]

Conclusion

In short, today we saw steps followed by our Support Techs to resolve TLS key negotiation failed error in OpenVPN.