ssh stuck at SSH2_MSG_KEX_DH_GEX_REQUEST when trying to access CHR v7+ on GCE

I’ve been trying out the cloud hosted router on Google Cloud, using v6.48.3, with no issue. Wanting to check the wireguard support of ROS7 out, I upgraded to v7.4.1. After the upgrade, ssh would not log in anymore. It would get stuck at SSH2_MSG_KEX_DH_GEX_REQUEST and then timeout after a while with “auth timeout”

The client ssh is the SSH from Ubuntu 20.04, OpenSSH_8.2p1 Ubuntu-4ubuntu0.5, OpenSSL 1.1.1f 31 Mar 2020

Since I’ve seen some ssh-related changes in 7.3 and 7.4, I tried the latest minor versions from 7.1 and 7.2, but no change. Also tried to install the CHR whole disk image, also no change. Reinstalling 6.48.3 works again.

Any idea what could be happening ?

Output of ssh -v (minus some IPs and stuff):

OpenSSH_8.2p1 Ubuntu-4ubuntu0.5, OpenSSL 1.1.1f 31 Mar 2020
debug1: Reading configuration data /home/doudou/.ssh/config
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files
debug1: /etc/ssh/ssh_config line 21: Applying options for *
debug1: Connecting to XXXXXX.
debug1: Connection established.
debug1: identity file /home/doudou/.ssh/id_rsa type 0
debug1: identity file /home/doudou/.ssh/id_rsa-cert type -1
debug1: identity file /home/doudou/.ssh/id_dsa type -1
debug1: identity file /home/doudou/.ssh/id_dsa-cert type -1
debug1: identity file /home/doudou/.ssh/id_ecdsa type -1
debug1: identity file /home/doudou/.ssh/id_ecdsa-cert type -1
debug1: identity file /home/doudou/.ssh/id_ecdsa_sk type -1
debug1: identity file /home/doudou/.ssh/id_ecdsa_sk-cert type -1
debug1: identity file /home/doudou/.ssh/id_ed25519 type -1
debug1: identity file /home/doudou/.ssh/id_ed25519-cert type -1
debug1: identity file /home/doudou/.ssh/id_ed25519_sk type -1
debug1: identity file /home/doudou/.ssh/id_ed25519_sk-cert type -1
debug1: identity file /home/doudou/.ssh/id_xmss type -1
debug1: identity file /home/doudou/.ssh/id_xmss-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_8.2p1 Ubuntu-4ubuntu0.5
debug1: Remote protocol version 2.0, remote software version ROSSSH
debug1: no match: ROSSSH
debug1: Authenticating to XXXXX
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: diffie-hellman-group-exchange-sha256
debug1: kex: host key algorithm: rsa-sha2-256
debug1: kex: server->client cipher: aes192-ctr MAC: hmac-sha2-256 compression: none
debug1: kex: client->server cipher: aes192-ctr MAC: hmac-sha2-256 compression: none
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(2048<8192<8192) sent
^C

Noone ? No idea … ?

http://forum.mikrotik.com/t/cannot-ssh-to-mikrotik-rb750-with-dsa-key/125060/1

/ip ssh regenerate-host-key

Right. So, this was an install from scratch of a brand new firmware … regenerating the host key was unlikely to solve anything (and indeed nothing got solved by it).

What I had to do to make it work:

  • explicitly set hmac-sha2-256 as the MAC on the openssh side
  • disable SetEnv stanzas (I had some that were setting git-related environment)

Without the latter, I would not get a terminal. With the former, it would simply not pass the key exchange phase.

Sorry to dig up an old thread, but ran into a similar issue with 7.11. It definitely appears to be related to the number/type of key exchange algorithms offered. In virtually all cases, when more than 5 key exchange algorithms are offered by the SSH client, the router stops responding and the SSH session times out after 30 seconds. Depending on which algorithms are offered, the number may be as low as 3.

This particular command line seems to work:

$ ssh -o KexAlgorithms=curve25519-sha256,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1  mikrotik

whereas the following will fail:

$ ssh -o KexAlgorithms=diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha1,curve25519-sha256 mikrotik
Received disconnect from UNKNOWN port 65535:11: auth timeout
Disconnected from UNKNOWN port 65535

What I see from the MikroTik side is this:

18:35:20 ssh,debug new connection: ::ffff:192.0.2.11:55904 (4)
18:35:20 ssh,debug transport state: 1 --> 2
18:35:20 ssh,debug,packet sending string
18:35:20 ssh,debug,packet SSH-2.0-ROSSSH\r
18:35:20 ssh,debug,packet
18:35:20 ssh,debug client version: SSH-2.0-OpenSSH_8.7
18:35:20 ssh,debug transport state: 2 --> 3
18:35:20 ssh,debug,packet packet create: kex init
18:35:20 ssh,debug,packet ----- sending ----- #0
18:35:20 ssh,debug,packet => size:592 [0x250]
18:35:20 ssh,debug,packet 0000024c 0a14c159 5f14fc20 fda6b6bb
18:35:20 ssh,debug,packet 0a3cc7fa 02d70000 009b6375 72766532
18:35:20 ssh,debug,packet 35353139 2d736861 3235362c 64696666
18:35:20 ssh,debug,packet 69652d68 656c6c6d 616e2d67 726f7570
18:35:20 ssh,debug,packet 2d657863 68616e67 652d7368 61323536
18:35:20 ssh,debug,packet 2c646966 6669652d 68656c6c 6d616e2d
18:35:20 ssh,debug,packet 67726f75 702d6578 6368616e 67652d73
18:35:20 ssh,debug,packet 6861312c 64696666 69652d68 656c6c6d
18:35:20 ssh,debug,packet 616e2d67 726f7570 31342d73 6861312c
18:35:20 ssh,debug,packet 64696666 69652d68 656c6c6d 616e2d67
18:35:20 ssh,debug,packet 726f7570 312d7368 61312c65 78742d69
18:35:20 ssh,debug,packet 6e666f2d 73000000 14727361 2d736861
18:35:20 ssh,debug,packet 322d3235 362c7373 682d7273 61000000
18:35:20 ssh,debug,packet 78616573 3132382d 6374722c 61657331
18:35:20 ssh,debug,packet 39322d63 74722c61 65733235 362d6374
18:35:20 ssh,debug,packet 722c6165 73313238 2d67636d 406f7065
18:35:20 ssh,debug,packet
18:35:20 ssh,debug,packet 6e737368 2e636f6d 2c616573 3235362d
18:35:20 ssh,debug,packet 67636d40 6f70656e 7373682e 636f6d2c
18:35:20 ssh,debug,packet 61657331 32382d63 62632c61 65733139
18:35:20 ssh,debug,packet 322d6362 632c6165 73323536 2d636263 
18:35:20 ssh,debug,packet 2c336465 732d6362 63000000 78616573
18:35:20 ssh,debug,packet 3132382d 6374722c 61657331 39322d63
18:35:20 ssh,debug,packet 74722c61 65733235 362d6374 722c6165
18:35:20 ssh,debug,packet 73313238 2d67636d 406f7065 6e737368
18:35:20 ssh,debug,packet 2e636f6d 2c616573 3235362d 67636d40
18:35:20 ssh,debug,packet 6f70656e 7373682e 636f6d2c 61657331
18:35:20 ssh,debug,packet 32382d63 62632c61 65733139 322d6362
18:35:20 ssh,debug,packet 632c6165 73323536 2d636263 2c336465
18:35:20 ssh,debug,packet 732d6362 63000000 2e686d61 632d7368
18:35:20 ssh,debug,packet 61322d32 35362c68 6d61632d 73686132
18:35:20 ssh,debug,packet 2d353132 2c686d61 632d7368 61312c68
18:35:20 ssh,debug,packet 6d61632d 6d643500 00002e68 6d61632d
18:35:20 ssh,debug,packet 
18:35:20 ssh,debug,packet 73686132 2d323536 2c686d61 632d7368
18:35:20 ssh,debug,packet 61322d35 31322c68 6d61632d 73686131
18:35:20 ssh,debug,packet 2c686d61 632d6d64 35000000 046e6f6e
18:35:20 ssh,debug,packet 65000000 046e6f6e 65000000 00000000
18:35:20 ssh,debug,packet 00000000 00005a5b f4aa5330 8e078b93
18:35:20 ssh,debug,packet --------------------
...
18:35:50 ssh,info auth timeout
18:35:50 ssh,debug,packet packet create: disconnect
18:35:50 ssh,debug,packet ----- sending ----- #1
18:35:50 ssh,debug,packet => size:40 [0x28]
18:35:50 ssh,debug,packet 00000024 0a010000 000b0000 000c6175
18:35:50 ssh,debug,packet 74682074 696d656f 75740000 00000442
18:35:50 ssh,debug,packet a934bb77 6d5386d8
18:35:50 ssh,debug,packet --------------------
18:35:50 ssh,debug transport state: 3 --> 0
18:35:50 ssh,debug closing connection: <auth timeout> ::ffff:192.0.2.11:55904 (4)

and from the OpenSSH client:

debug1: re-parsing configuration
debug1: Reading configuration data /etc/ssh/ssh_config
debug3: /etc/ssh/ssh_config line 55: Including file /etc/ssh/ssh_config.d/04-ipa.conf depth 0
debug1: Reading configuration data /etc/ssh/ssh_config.d/04-ipa.conf
debug2: checking match for 'exec true' host 172.26.0.47 originally 172.26.0.47
debug1: Executing command: 'true'
debug3: command returned status 0
debug3: /etc/ssh/ssh_config.d/04-ipa.conf line 9: matched 'exec "true"'
debug2: match found
debug3: /etc/ssh/ssh_config line 55: Including file /etc/ssh/ssh_config.d/50-redhat.conf depth 0
debug1: Reading configuration data /etc/ssh/ssh_config.d/50-redhat.conf
debug2: checking match for 'final all' host 172.26.0.47 originally 172.26.0.47
debug3: /etc/ssh/ssh_config.d/50-redhat.conf line 3: matched 'final'
debug2: match found
debug3: /etc/ssh/ssh_config.d/50-redhat.conf line 5: Including file /etc/crypto-policies/back-ends/openssh.config depth 1
debug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config
debug3: gss kex names ok: [gss-curve25519-sha256-,gss-nistp256-sha256-,gss-group14-sha256-,gss-group16-sha512-,gss-gex-sha1-,gss-group14-sha1-]
debug3: kex names ok: [curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1]
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/home/user/.ssh/known_hosts'
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/home/user/.ssh/known_hosts2'
debug1: Executing proxy command: exec /usr/bin/sss_ssh_knownhostsproxy -p 22 172.26.0.47
debug1: identity file /home/user/.ssh/id_rsa type -1
debug1: identity file /home/user/.ssh/id_rsa-cert type -1
debug1: identity file /home/user/.ssh/id_dsa type -1
debug1: identity file /home/user/.ssh/id_dsa-cert type -1
debug1: identity file /home/user/.ssh/id_ecdsa type -1
debug1: identity file /home/user/.ssh/id_ecdsa-cert type -1
debug1: identity file /home/user/.ssh/id_ecdsa_sk type -1
debug1: identity file /home/user/.ssh/id_ecdsa_sk-cert type -1
debug1: identity file /home/user/.ssh/id_ed25519 type -1
debug1: identity file /home/user/.ssh/id_ed25519-cert type -1
debug1: identity file /home/user/.ssh/id_ed25519_sk type -1
debug1: identity file /home/user/.ssh/id_ed25519_sk-cert type -1
debug1: identity file /home/user/.ssh/id_xmss type -1
debug1: identity file /home/user/.ssh/id_xmss-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_8.7
debug1: Remote protocol version 2.0, remote software version ROSSSH
debug1: compat_banner: no match: ROSSSH
debug2: fd 7 setting O_NONBLOCK
debug2: fd 5 setting O_NONBLOCK
debug1: Authenticating to 172.26.0.47:22 as 'cfg-mon'
debug1: load_hostkeys: fopen /home/user/.ssh/known_hosts2: No such file or directory
debug3: order_hostkeyalgs: no algorithms matched; accept original
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug3: receive packet: type 20
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,ext-info-c,kex-strict-c-v00@openssh.com   
debug2: host key algorithms: ssh-ed25519-cert-v01@openssh.com,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,sk-ssh-ed25519-cert-v01@openssh.com,sk-ecdsa-sha2-nistp256-cert-v01@openssh.com,rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh-ed25519,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ssh-ed25519@openssh.com,sk-ecdsa-sha2-nistp256@openssh.com,rsa-sha2-512,rsa-sha2-256,ssh-rsa
debug2: ciphers ctos: aes256-gcm@openssh.com,chacha20-poly1305@openssh.com,aes256-ctr,aes256-cbc,aes128-gcm@openssh.com,aes128-ctr,aes128-cbc
debug2: ciphers stoc: aes256-gcm@openssh.com,chacha20-poly1305@openssh.com,aes256-ctr,aes256-cbc,aes128-gcm@openssh.com,aes128-ctr,aes128-cbc
debug2: MACs ctos: hmac-sha2-256-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha2-256,hmac-sha1,umac-128@openssh.com,hmac-sha2-512
debug2: MACs stoc: hmac-sha2-256-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha2-256,hmac-sha1,umac-128@openssh.com,hmac-sha2-512
debug2: compression ctos: none,zlib@openssh.com,zlib
debug2: compression stoc: none,zlib@openssh.com,zlib
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1,ext-info-s
debug2: host key algorithms: rsa-sha2-256,ssh-rsa
debug2: ciphers ctos: aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-cbc,aes192-cbc,aes256-cbc,3des-cbc
debug2: ciphers stoc: aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-cbc,aes192-cbc,aes256-cbc,3des-cbc
debug2: MACs ctos: hmac-sha2-256,hmac-sha2-512,hmac-sha1,hmac-md5
debug2: MACs stoc: hmac-sha2-256,hmac-sha2-512,hmac-sha1,hmac-md5
debug2: compression ctos: none
debug2: compression stoc: none
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: rsa-sha2-256
debug1: kex: server->client cipher: aes256-gcm@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: aes256-gcm@openssh.com MAC: <implicit> compression: none
debug1: kex: curve25519-sha256 need=32 dh_need=32
debug1: kex: curve25519-sha256 need=32 dh_need=32
debug3: send packet: type 30
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug3: receive packet: type 1
Received disconnect from UNKNOWN port 65535:11: auth timeout
Disconnected from UNKNOWN port 65535