The goal is the use of Gitlab's CI/CD pipeline to fire off the building of a test environment within Azure, where knock_rs' iptables rule changes can be assessed.
Create a gitlab-runner on azure which will be used to manage the test environment. This test environment will contain two machines, a client and a server. Once both machines are up and running the client will send a port knock sequence to the server and the assumed response of the server will be recorded making sure the appropriate iptables rules were created.
cloud_env
├── ansible_playbook
│ ├── ansible.cfg
│ ├── inventory
│ ├── main.yml
│ ├── setup
│ │ ├── config_common.yml
│ │ ├── config_knock_client.yml
│ │ ├── config_knock_server.yml
│ │ ├── instance_configuration.yml
│ │ └── instance_creation.yml
│ └── test
│ ├── get_results_client.yml
│ ├── get_results_server.yml
│ └── run_system_test.yml
├── client_login_test.py
├── flag.txt
└── vault.pass.py
gitlab's docs has everything needed to create and start a runner. We can setup multiple runners one for testing and building a debug build, and a second to fire off the anisble playbooks for setup and teardown of the test environment.
I wrote a post that contains an Ansible playbook to setup and start a runner here
stages:
- test_unit
- test_system
- test_cleanup
- build
test_build:
image: "rust:latest"
stage: test_unit
tags:
- docker
script:
- rustc --version && cargo --version # Print version info for debugging
- cargo build
- cargo test
artifacts:
paths:
- target/debug/
- config/
- tests/cloud_env/client_login_test.py
- tests/cloud_env/flag.txt
test_system:
only:
- schedules
stage: test_system
tags:
- shell
- ansible
script:
- STATE=present TEST=true ANSIBLE_CONFIG=./tests/cloud_env/ansible_playbook/ansible.cfg ansible-playbook -i ./tests/cloud_env/ansible_playbook/inventory ./tests/cloud_env/ansible_playbook/main.yml --vault-password-file ./tests/cloud_env/vault.pass.py
- cp tests/cloud_env/ansible_playbook/test/test_results_server.log .
- cp tests/cloud_env/ansible_playbook/test/test_results_server_ipt.log .
- cp tests/cloud_env/ansible_playbook/test/test_results_client.log .
artifacts:
untracked: false
expire_in: 7 days
paths:
- test_results_server.log
- test_results_server_ipt.log
- test_results_client.log
test_system_cleanup:
only:
- schedules
stage: test_cleanup
tags:
- shell
- ansible
script:
- STATE=absent ansible-playbook ./tests/cloud_env/ansible_playbook/setup/instance_creation.yml --vault-password-file ./tests/cloud_env/vault.pass.py
build:
image: "rust:latest"
only:
- schedules
stage: build
tags:
- docker
script:
- cargo build --release
- cp target/release/knock_rs ./knockd_rs
- cp target/release/client ./knock_rs
artifacts:
untracked: false
expire_in: 30 days
paths:
- knockd_rs
- knock_rs
These Microsoft docs link to info about creating an Azure app leaving you with:
Then following the Ansible Azure docs guide we can setup a simple Ansible playbook for creating a vm on Azure:
The above guide is missing some changes needed due to Ansible's move to full qualified collection names (fqcn) with Ansible 2.10.
- name: Install azure modules from galaxy
command: |
ansible-galaxy collection install community.azure && \
ansible-galaxy collection install azure.azcollection
- name: Install azure requirements
command: |
cd /home/gitlab-runner/.ansible/collections/ansible_collections/azure/azcollection/; \
python3 -m pip install -r requirements-azure.txt
This snippet shows the need to install requirements located within anisbles collections folder. Make sure these requirements are installed to the python that Ansible uses.
A top level main file is used to run these creation and configuration playbooks as well as start the test playbooks:
---
- name: Run instance creation
import_playbook: setup/instance_creation.yml
- name: Run instance configuration
import_playbook: setup/instance_configuration.yml
when: lookup('env', 'STATE') == 'present'
- name: Run tests
import_playbook: test/run_system_test.yml
when: lookup('env', 'TEST') == 'true'
Environment variables are used to switch between setup, teardown, and to determine if tests should be run.
---
- hosts: 127.0.0.1
connection: local
environment:
AZURE_CLIENT_ID: "{{ azure_client_id }}"
AZURE_SECRET: "{{ azure_secret }}"
AZURE_SUBSCRIPTION_ID: "{{ azure_subscription_id }}"
AZURE_TENANT: "{{ azure_tenant }}"
vars:
state: "{{ lookup('env', 'STATE') }}"
action: "{{ 'Create' if state == 'present' else 'Destroy' }}"
azure_client_id: !vault |
...
azure_secret: !vault |
...
azure_subscription_id: !vault |
...
azure_tenant: !vault |
...
tasks:
- name: "{{ action }} knock_rs server"
azure.azcollection.azure_rm_virtualmachine:
state: "{{ state }}"
resource_group: gitlab_ci_cd
name: knock-server
vm_size: Standard_B1s
admin_username: knock
admin_password: "Myreally5ecurepassword"
ssh_public_keys:
- path: /home/knock/.ssh/authorized_keys
key_data: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsEhd8AxSy6cIlqyVT84jd9zpiqSTae5+CBE4Sp5HJay93gWjBMD9HZa2UzZ67bTgfgQ3s3W+O1eWUUJjpDT9OnpBIVm5zqJ3SPJClx6RU6X4f7B3Ek66pgeFnqoXyMzCwN/1iDAO+6CXtF78hzxuy0CgL0fl81ic2VlLxTUUM3GKOM0XxPwK5SEUjCKWUzxXrHive3FtGcxxDkqTDtDq3Ht67XChJRAzULMbutcrel6AkdNyfEyH3fjsSDb35xu20D0CgQgDnUcnDXz1YWYonqD1AP1nxeFZZAZh48DQnU57DblOPdu/XHz/+/nZTPaZSfrhS2LawR3C8FP7isVg5 gitlab-runner@gitlab-runner
public_ip_allocation_method: Disabled
image:
offer: UbuntuServer
publisher: Canonical
sku: '18.04-LTS'
version: latest
I've omitted the values for the secrets stored within Anisble's vault feature. You can check it out here. It's a great way to store variables securely within playbooks and allows for great portability when deploying, allowing you to check self contained playbooks into version control and not have to deal with moving keys and secrets to wherever you may be launching your deploy. As is the case here, this playbook will be launched from a server on Azure and the only necessary secure transfer will be the password which locks the vault.
So we've got a server to run the knock_rs daemon lets also start up a server for a knock_rs client.
...
- name: "{{ action }} knock_rs client"
azure.azcollection.azure_rm_virtualmachine:
state: "{{ state }}"
resource_group: gitlab_ci_cd
name: knock-client
vm_size: Standard_B1s
admin_username: knock
ssh_password_enabled: false
ssh_public_keys:
- path: /home/knock/.ssh/authorized_keys
key_data: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsEhd8AxSy6cIlqyVT84jd9zpiqSTae5+CBE4Sp5HJay93gWjBMD9HZa2UzZ67bTgfgQ3s3W+O1eWUUJjpDT9OnpBIVm5zqJ3SPJClx6RU6X4f7B3Ek66pgeFnqoXyMzCwN/1iDAO+6CXtF78hzxuy0CgL0fl81ic2VlLxTUUM3GKOM0XxPwK5SEUjCKWUzxXrHive3FtGcxxDkqTDtDq3Ht67XChJRAzULMbutcrel6AkdNyfEyH3fjsSDb35xu20D0CgQgDnUcnDXz1YWYonqD1AP1nxeFZZAZh48DQnU57DblOPdu/XHz/+/nZTPaZSfrhS2LawR3C8FP7isVg5 gitlab-runner@gitlab-runner
public_ip_allocation_method: Disabled
image:
offer: UbuntuServer
publisher: Canonical
sku: '18.04-LTS'
version: latest
Once these instances are created we can run some common tasks for them.
---
- hosts: server
tasks:
- name: Server setup
include_tasks: config_knock_server.yml
- hosts: client
tasks:
- name: Client setup
include_tasks: config_knock_client.yml
---
- name: Download binaries and config files
get_url:
url: https://gitlab.com/andrewkreuzer/knock_rs/-/jobs/artifacts/master/download?job=test_build
dest: /home/knock/knock_rs.zip
- name: Upgrade apt packages
become: yes
apt:
upgrade: yes
- name: Install unzip
become: yes
apt:
name: unzip
state: present
update-cache: yes
- name: Install pip
become: yes
apt:
name: python3-pip
state: present
update-cache: yes
# This seems to fix segfault on pip version from ubuntu repo
- name: Instantly upgrade pip
shell: python3 -m pip install -U pip
- name: Create knock_rs directory
file:
path: /home/knock/knock_rs/
state: directory
mode: '0755'
- name: Unzip knock_rs.zip
unarchive:
remote_src: yes
src: /home/knock/knock_rs.zip
dest: /home/knock/knock_rs/
creates: /home/knock/knock_rs/target/
Here we grab some prebuilt binaries and config files, built during a separate stage of the CI/CD pipeline. This is one of the nice things about Gitlab's CI/CD, being able to store artifacts from builds or tests and use them in deployment or further on in the testing pipeline is quite handy.
For the specific tasks for each of the servers I've separated the setup into client and sever files.
---
- include: config_common.yml
- name: Install paramiko
pip:
name: paramiko
state: present
---
- include: config_common.yml
- name: Install psutils python module
pip:
name: psutil
state: present
- name: Ensure gitlab-runner can always connect
iptables:
chain: INPUT
source: gitlab-runner
jump: ACCEPT
become: yes
- name: Move flag into home directory
copy:
remote_src: yes
src: /home/knock/knock_rs/tests/cloud_env/flag.txt
dest: /home/knock/flag.txt
- name: Check for running knock_rs daemon
community.general.pids:
name: knock_rs
register: knock_rs_pids
- name: Run knock_rs
shell: cd /home/knock/knock_rs; nohup /home/knock/knock_rs/target/debug/knock_rs $(ip -4 -o route show default | awk '{print $5}') </dev/null >/dev/null 2>&1 &
become: yes
when: knock_rs_pids.pids|length == 0
- name: Log packets that reach KNOCK_RS iptables CHAIN
become: yes
iptables:
chain: KNOCK_RS
action: append
state: present
limit: 2/second
limit_burst: '20'
log_prefix: "KNOCK_RS:INFO "
log_level: info
Each of these files includes the config_common.yml file then setups up the various needs for each server.
In the case of the client all that's needed is to grab paramiko a library for Python to connect through ssh, the client uses this in a script to ssh into the server and grab a flag.txt.
The server grabs psutils to monitor if knock_rs is already running; adds some iptables rules to ensure the gitlab-runner box can always connect; moves a flag.txt file into the home directory (the client will grab this later); then starts knock_rs in the background and logs packets which reach the KNOCK_RS chain.
In the run_system_test.yml file we simply call a "get results" playbook for each server.
---
- import_playbook: get_results_client.yml
- import_playbook: get_results_server.yml
In the get_results_server.yml file we grab the logs that reached the KNOCK_RS chain and copy them back to the runner. And in the get_results_client.yml file we run a python script which attempts connections to the knock_rs daemon and copies the results back to the runner.
---
- hosts: server
tasks:
- name: Get iptables logs
shell: journalctl -k | grep "KNOCK_RS:INFO" > /tmp/iptables.log
become: yes
- name: Copy test results back to runner
fetch:
flat: yes
src: /tmp/iptables.log
dest: test_results_server.log
---
- hosts: client
tasks:
- name: Run client knock
command: python3 ~/knock_rs/tests/cloud_env/client_login_test.py
- name: Get test results
fetch:
flat: yes
src: test_results.log
dest: test_results_client.log
And finally we have our test file which is launched by the client. This attempts ssh connections to the server until it is successful or there has been 5 attempts. Inbetween each attempt it calls knock_rs's client bin to initiate a port knock against the server. The attempts are logged to the test_results.log file which is copied into the artifacts for this pipeline stage.
#!/usr/bin/python3
import paramiko
import time
import json
import os
def connect_SSH():
time.sleep(10)
client = paramiko.SSHClient()
try:
client.set_missing_host_key_policy(paramiko.client.AutoAddPolicy)
client.connect('knock-server', username='knock', password='Myreally5ecurepassword', timeout=5)
print("connected successfully")
# stdin, stdout, stderr
_, stdout, _ = client.exec_command('cat flag.txt')
for line in stdout:
flag = line.strip()
if flag == "ya done did it":
return True
except paramiko.ssh_exception.SSHException as e:
print(f"SSHException {e.args[0]}")
return False
except Exception as e:
print(f"SSH Connection failed: {e}")
return False
def run_knocker(closeSession):
close_sequence = "8083 8084 8085"
open_sequence = "7070 7071 7072"
if closeSession:
os.system(f"~/knock_rs/target/debug/client -r knock-server -p udp -s { close_sequence }")
return
os.system(f"~/knock_rs/target/debug/client -r knock-server -p udp -s { open_sequence }")
def write_to_file(results):
with open("test_results.log", 'w') as f:
j = json.dumps(results)
f.write(j)
def check_SSH():
results = {}
count = 0
while not connect_SSH():
results[count] = "failed attempt"
run_knocker(False)
if count >= 5:
results['final'] = "Ending test due to 5 failed attempts"
write_to_file(results)
return
count += 1
results['final'] = "Successful connection"
write_to_file(results)
run_knocker(True)
if __name__ == '__main__':
check_SSH()
Was this necessary, probably not. But it was an interesting project to test out. Definitely learned some more about Ansible and Azure so all time wasn't lost. I wonder though if mocking iptables for the daemon and an ssh connection for the client, would have been a better solution. Those seem like they wouldn't be particularity fun either though.