Aaaaand we’re back! This time we’re talking about setting up and using SSH keys to run Ansible playbooks. Once again, I’m using BitBucket as my repo and specifically using BitBucket on-prem runners to run the playbooks via a container. This time around I’m feeling some Made in Heights, but it seems that’s all but disappeared from the internet so we’ll go with Kelsey Bulkin herself to scratch a similar itch.
First, The SSH Keys
Before you can connect to a machine using SSH keys, you need a few things. You need a single user account on each machine that you want to connect with and you need some SSH keys generated for that user. When you generate SSH keys you will create a public and private key, the public key can be distributed to all the machines you want to connect to and the private key will be used to actually connect to those machines. Needless to say, you want to keep the private key safe. Best practice is to keep this on the same machine it was generated on or to transfer it to a safer storage method like a secret management setup. You really don’t want to be shipping the private key around as it contains the public key data as well and is ripe for abuse should unsavory folk get ahold of it.
Now, I say all of that but of course I’m going to commit just about every SSH key sin you can in this example. To be fair, this is an example of how to generate keys and use them to run Ansible playbooks, this is not an example of how you should set this up in an enterprise setting!
With all the caveats out of the way, I have included two playbooks in my repo, Generate_Keys & Push_Keys. First we’ll look at how we’re generating the keys.
---
- hosts: localhost
become: yes
tasks:
- name: Create user
no_log: true
user:
name: '#NewUser'
password: "{{'#NewPass' | password_hash('sha512')}}"
update_Password: on_create
- name: Create user dir
ansible.builtin.file:
path: ~/ansible
state: directory
mode: '0755'
- name: Create SSH Key-Pair #I haven't tested this nested become but the keygen needs to be run as the new user.
become: yes
become_user: '#NewUser'
local_action: command ssh-keygen -t rsa -b 4096 -N "" -q -f ~/ansible/id_rsaRemember that we’re running all of this on a runner container, so we first want to create a local instance of the user that we want the ssh keys generated for. You can see that’s what the first play is doing, the second is creating a unique directory in which to land our keys on the runner. Finally, we’re creating the keypair by running a local_action AS the new user. I haven’t tested this kind of nested become directly after creating a new user, so it’s possible this key creation would need to be broken into a separate playbook where you specify become_user: NewUser
I do want to note that the key creation here is being done insecurely, I’m not even setting a password for the private key file, which is definitely something we should do using the -P flag. Again, don’t do this in production 🙂
---
- hosts: Management
become: yes
tasks:
- name: Copy Private Key
ansible.builtin.copy:
src: ~/ansible/id_rsa
dest: /home/placeyouwantkeys
owner: '#NewUser'
group: sudo
mode: '0644'
- name: Copy Public Key
ansible.builtin.copy:
src: ~/ansible/id_rsa.pub
dest: /home/placeyouwantkeys
owner: '#NewUser'
group: sudo
mode: '0644'Speaking of things you shouldn’t do in production, here’s a playbook that takes our newly generated keys and copies them off to whatever server(s) we’ve set under the management group in our inventory. With the above two playbooks you should be able to create the keys we need and ship them off to a server for later retrieval. So lets get into how we use SSH keys to run playbooks!
SSH Keys in Ansible – The Inventory
If you take a look at the inventory file I’ve got in my example repo, you’ll see there’s a test group with a single server setup. That same server is setup higher up in the Admin_Group1 group as well. This is because some of our playbooks will target this host using an existing management account specified as inventory vars. However, we also need to have the inventory setup to use SSH keys with this host for a later demonstration, thus the second “test” group with user and key vars specified for SSH key use.
---
All:
hosts:
children:
Admin_Group1:
hosts:
target1.fqdn:
target2.fqdn:
Management:
hosts:
manbox1.fqdn:
Test:
hosts:
target1.fqdn:
ansible_ssh_user: '#NewUser'
ansible_ssh_private_key_file: /home/ansible/.ssh/id_rsa
vars:
ansible_user: '#AdminUser'
ansible_password: '#AdminPass'Now that we understand the inventory side, let’s look at the pipeline we’re going to run and break down each of the playbooks being run. First we set our ansible_host_key_checking option to false as usual, to help fight off issues with running playbooks against “unknown” hosts. Then we perform our variable substitution using SED, and finally we get to the good stuff.
We first pull the SSH keys we plan to use for our new admin account, then we run the playbook that uses our existing admin creds to create this new admin user across our inventory hosts, and finally we run a playbook that tests using our new admin user with SSH keys.
image: alpine:latest
pipelines:
branches:
master:
- step:
name: Deploy New User
deployment: Prod
runs-on:
- "self-hosted"
- "ansible"
image: quay.io/ansible/ansible-runner:latest
script:
- export ANSIBLE_HOST_KEY_CHECKING=False
- sed -i 's/#AdminUser/'"${AdminUser}"'/g' ./hosts.yml
- sed -i 's/#AdminPass/'"${AdminPass}"'/g' ./hosts.yml
- sed -i 's/#NewUser/'"${NewUser}"'/g' ./hosts.yml
- sed -i 's/#NewUser/'"${NewUser}"'/g' ./create_user.yml
- sed -i 's/#NewPass/'"${NewPass}"'/g' ./create_user.yml
- sed -i 's/#NewUser/'"${NewUser}"'/g' ./key_test.yml
- ansible-playbook ./pull_keys.yml -i ./hosts.yml
- ansible-playbook ./create_user.yml -i ./hosts.yml
- ansible-playbook ./key_test.yml -i ./hosts.yml
options:
docker: trueOur playbook to pull our SSH keys is pretty dang simple. Since we previously generated on or copied our keys over to a management host, we’re just going to target that same host to pull the keys over from. We need to secure the private key after it’s copied over, otherwise Ansible won’t actually let us use it, so we’ve got a play delegated to the localhost to lock that file down once it’s pulled over.
---
- hosts: Management
become: yes
tasks:
- name: Pull Private Key
ansible.builtin.fetch:
src: /home/user/pathtokeys/ir_rsa
dest: /home/ansible/.ssh/id_rsa
flat: yes
- name: Secure Private Key
ansible.builin.file:
path: /home/ansible/.ssh/id_rsa
mode: '0600'
delegate_to: localhost
- name: Pull Public Key
ansible.builtin.fetch:
src: /home/user/pathtokeys/ir_rsa.pub
dest: /home/ansible/.ssh/id_rsa.pub
flat: yesCreating the Admin User Across Inventory
So we’ve got our SSH keys on the runner, and we’ve got an inventory file setup to use the private key to connect to one of our hosts. Now we need to create the associated user account on whichever hosts we plan to connect to using our SSH keys. This is done with the create_user playbook. There are a few things going on here so let’s break this down.
The first thing we’re doing here is using the user command to create a new user account. Of note here, I am setting this account up with a null password by passing in ! as the NewPass variable. You can also do this manually using ‘sudo passwd -d useraccounthere’ in case you aren’t setting a null password on creation. Notice I’ve got the play set to always update the user password in case you need to set an actual password to test logging in with. Simply change your NewPass repo var and re-run.
Next up, we’ve got some rudimentary logic to decide if we should add the user to the sudo or wheel group based on which Linux distribution is being targeted. These are based on facts gathered by Ansible when you run these playbooks, so if you set gather_facts to disabled, these won’t work.
The “Set Passwordless” play is where we use the lineinfile function to setup the sudoers file to allow our new user account to use passwordless sudo. Otherwise we’d be able to log in with our SSH key but we wouldn’t be able to escalate to sudoer privilege without providing a password. It is possible to provide a password in Ansible to handle these situations but in my experience it’s kludgey and I’ve had problems getting it to work right. Also of note in this play is that we’re backing the file up in case things go wrong, we’re checking if the line we’d like to add exists already so as not to create unnecessary noise, and we’re using visudo to validate the sudoers file on top of that.
Finally, we’ve got to add our public key to the authorized_keys file on each host. You’ll notice I’m using the lineinfile function again here to ensure we don’t upset any existing configurations. This will also create the file if it doesn’t exist and evaluate any existing file to ensure it’s not stuffing the same key in more than once. Additionally, we’re creating a backup of the file if it exists, in case anything untoward should happen and we need to restore the previous version.
---
- hosts: Admin_Group1
become: yes
tasks:
- name: Create user
no_log: true
user:
name: '#NewUser'
password: "{{'#NewPass' | password_hash('sha512')}}"
password_lock: yes
update_password: always
- name: Add to sudo group
shell: usermod -aG 'sudo' '#NewUser'
when: ansible_facts['distribution'] == "CentOS"
- name: Add to wheel group
shell: usermod -aG 'wheel' '#NewUser'
when: ansible_facts['distribution'] == "Ubuntu"
- name: Set Passwordless
lineinfile:
path: /etc/sudoers
backup: yes
state: present
regexp: '#NewUser'
line: '#NewUser ALL=(ALL) NOPASSWD: ALL'
insertafter:
validate: 'visudo -cf %s'
- name: Add Key to Authorized_Keys
blockinfile:
block: "{{lookup('file', '/home/ansible/.ssh/id_rsa.pub') }}"
path: /home/#NewUser/.ssh/authorized_keys
backup: yes
state: present
insertafter:Giving it a Go
Phew. That’s the playbook doing most of the heavy lifting here. Once all of that is complete we can finally use our private SSH key to run an Ansible playbook. I’ve got a little key_test playbook setup specifically to do just that. All this playbook does is target our test host group (the one where we define what SSH key to use!) and creates a directory. If this succeeds it validates that we can login with our SSH key and use passwordless sudo with the new admin account we’ve setup.
---
- hosts: Test
user: '#NewUser'
become: yes
tasks:
- name: Create dir
ansible.builtin.file:
path: /home/#NewUser/testdir
owner: '#NewUser'
state: directoryGotchas & Notes
The main thing to note here, again, is that this isn’t really a solution you should be using in production. Yes, you can make the argument that things are relatively secure as you’re using SSH connections all around when keys are being transferred, but I still wouldn’t suggest doing this. At the very least, try to get the private key setup in a secret management service and then push/pull from that when you need to use it. I’d also suggest setting a password on the private key using the -P option on creation, provided you can get Ansible to play nice with consuming the password when needed.
Otherwise the standard advice to keep an eye on your variables applies here. There’s likely to be some debugging needed as none of this has been run/tested anywhere.
As usual, if you have any questions feel free to drop them below and I’ll try to answer or point you in the right direction!