Какая кодировка используется для ключей при использовании `ssh-keygen -t rsa`?

Ключи ssl(частный и открытый )обычно хранятся в так называемом формате PEM .

Privacy-Enhanced Mail (PEM) is a de facto file format for storing and sending cryptographic keys, certificates, and other data, based on a set of 1993 IETF standards defining "privacy-enhanced mail." While the original standards were never broadly adopted, and were supplanted by PGP and S/MIME, the textual encoding they defined became very popular. The PEM format was eventually formalized by the IETF in RFC 7468.

Этот формат на самом деле является заголовком, затем base64 закодированными двоичными данными и нижним колонтитулом.

Base64 is a group of similar binary-to-text encoding schemes that represent binary data in an ASCII string format by translating it into a radix-64 representation. The term Base64 originates from a specific MIME content transfer encoding. Each Base64 digit represents exactly 6 bits of data. Three 8-bit bytes (i.e., a total of 24 bits) can therefore be represented by four 6-bit Base64 digits.

Ключи sshсм. ниже от Дэйва _Томпсона _085 комментариев:

Note ssh-keygen uses (several) PEM formats but never the one(s) in 7468. In the past for RSA it defaulted to OpenSSL's two 'traditional' (aka 'legacy') formats, either unencrypted whcih is 7468-like except containing PKCS1, or password-encrypted which is1421-like with Proc-type and DEK-Info and base64 of encrypted PKCS1, but not 7468-like. Since 7.8 it defaults to OpenSSH's own 'new format' (previously invoked by option -o) which is 7468-like but the contents are entirely different (XDR-style not ASN.1). There are numerous Qs about these already on several Stacks.

OpenSSH public key formats are never PEM (although commercial 'SSH2' sort-of are), just base64 of SSH wire format. And I was recently reminded this Q/A covers the private key formats quite thoroughly