This is a bonus step for our Stack of NUCs lab. We are going to create and run an Ansible playbook to set up Hashtopolis. Cyber security professionals use Hashtopolis to create a cluster of systems all running Hashcat, a versatile password hash cracking tool.

It is important to discuss GPUs at this point.

• hashcat is no longer CPU-only; it uses GPUs and CPUs via OpenCL
• if your NUCs have a supported GPU, great it; otherwise you will be using OpenCL and CPU
  • in my lab this worked out to only 61039 kH/s for md5 and 330 H/s for mode 1880 (Unix)
• if you have an NVIDIA GPU install hashcat-nvidia for better performance
• because of this complexity, we are installing the hashcat package and its requirements instead of the traditional hashtopolis way of simply copying the binary and running it

There are two pieces to set up:

• server - central server distributes the keyspace of a task, aggregates jobs, and collects results in MySQL database
  • communicates over HTTPS with agent machines
  • passes over files, binaries and task commands
• agents - act on the commands, execute the hash cracking application, and report “founds” to the server

Purpose:

• Demonstrate a running a cluster of hash cracking nodes

References:

• https://github.com/hashtopolis
• https://jakewnuk.com/posts/hashtopolis-infrastructure/
• https://github.com/peterezzo/hashtopolis-docker
• https://resources.infosecinstitute.com/topic/hashcat-tutorial-beginners/
• https://infosecscout.com/install-hashcat-on-ubuntu/

1. Log in to the Ansible control node (NUC 2)
2. Create directory /home/ansible/my-project/hashtopolis and change to it
   • mkdir hashtopolis
   • cd hashtopolis
3. Create the inventory, putting one of the worker nodes in the [server] section and the rest in the [agents] section
   • /home/ansible/my-project/hashtopolis/inventory

   • inventory

     [all:vars]
     ansible_python_interpreter=/usr/bin/python3
     ansible_user='ansible'
     ansible_become=true
     ansible_become_method=sudo
     [server]
      
     [agents]

4. Create the ansible.cfg file
   • /home/ansible/my-project/hashtopolis/ansible.cfg

   • ansible.cfg

     [defaults]
     inventory = inventory

5. Create the apache.conf.j2 file

   • apache.conf.j2

     <VirtualHost *:{{ http_port }}>
        ServerAdmin webmaster@localhost
        ServerName {{ http_host }}
        ServerAlias www.{{ http_host }}
        DocumentRoot /var/www/{{ http_host }}
        ErrorLog ${APACHE_LOG_DIR}/error.log
        CustomLog ${APACHE_LOG_DIR}/access.log combined
      
        <Directory /var/www/{{ http_host }}>
              Options -Indexes
        </Directory>
      
        <IfModule mod_dir.c>
            DirectoryIndex index.php index.html index.cgi index.pl  index.xhtml index.htm
        </IfModule>
      
     </VirtualHost>

6. Create the info.php.j2 file

   • info.php.j2

     <?php
     phpinfo();

7. Create the .my.cnf.j2 file

   • .my.cnf.j2

     [client]
     port		= 3306
     socket		= /var/run/mysqld/mysqld.sock
     user=root
     password={{ mysql_root_password }}

The server runs on a LAMP stack. This playbook installs the LAMP stack and git clones the Hashtopolis server application.

TODO: add php.ini tweaking to the playbook, as necessary

1. Create the playbook /home/ansible/my-project/hashtopolis/hashtopolis-server.yml

   • hashtopolis-server.yml

     ---
     - hosts: localhost
       tasks:
         - name: Download wordlist common passwords
           get_url:
             url: https://raw.githubusercontent.com/danielmiessler/SecLists/master/Passwords/Common-Credentials/10-million-password-list-top-100000.txt
             dest: /home/ansible/my-project/hashtopolis/10-million-password-list-top-100000.txt
         - name: Download wordlist rockyou.txt
           get_url:
             url: http://downloads.skullsecurity.org/passwords/rockyou.txt.bz2
             dest: /home/ansible/my-project/hashtopolis/rockyou.txt.bz2
         - name: Download oneruletorulethemall
           get_url:
             url: https://raw.githubusercontent.com/NotSoSecure/password_cracking_rules/master/OneRuleToRuleThemAll.rule
             dest: /home/ansible/my-project/hashtopolis/OneRuleToRuleThemAll.rule
     - hosts: server
       become: true
       vars:
         mysql_root_password: "my_sql_root_password"
         app_user: "ansible"
         http_host: "hashtopolis"
         http_conf: "hashtopolis.conf"
         http_port: "80"
         disable_default: true
         hashtopolis_password: "my_hashtopolis_password"
       tasks:
         - name: Install prerequisites
           apt:
             pkg:
               - aptitude
               - git
               - phpmyadmin
      
         # Apache Configuration
         - name: Install LAMP Packages
           apt:
             pkg:
               - apache2
               - mysql-server
               - python3-pymysql
               - php
               - php-pear
               - php-mysql
               - libapache2-mod-php
         - name: Create document root
           file:
             path: "/var/www/{{ http_host }}"
             state: directory
             owner: "{{ app_user }}"
             mode: '0755'
         - name: Set up Apache virtualhost
           template:
             src: "apache.conf.j2"
             dest: "/etc/apache2/sites-available/{{ http_conf }}"
           notify: Reload Apache
         - name: Enable new site
           shell: /usr/sbin/a2ensite {{ http_conf }}
           notify: Reload Apache
         - name: Disable default Apache site
           shell: /usr/sbin/a2dissite 000-default.conf
           when: disable_default
           notify: Reload Apache
      
         # MySQL Configuration
         - name: start and enable mysql service
           service:
             name: mysql
             state: started
             enabled: true
         - name: manage MySQL root password
           become: true
           template:
             src: /home/ansible/my-project/hashtopolis/.my.cnf.j2
             dest: /root/.my.cnf
         - name: Sets the hashtopolis password
           mysql_user:
             name: hashtopolis
             password: "{{ hashtopolis_password }}"
             priv: "*.*:ALL"
             login_user: root
             login_password: "{{ mysql_root_password }}"
             state: present
         - name: Removes all anonymous user accounts
           mysql_user:
             name: ''
             host_all: true
             state: absent
             login_user: root
             login_password: "{{ mysql_root_password }}"
         - name: Removes the MySQL test database
           mysql_db:
             name: test
             state: absent
             login_user: root
             login_password: "{{ mysql_root_password }}"
         - name: Create new databases
           mysql_db:
             name:
               - hashtopolis
             state: present
             login_user: root
             login_password: "{{ mysql_root_password }}"
      
         # UFW Configuration
         - name: "UFW - Allow HTTP on port {{ http_port }}"
           ufw:
             rule: allow
             port: "{{ http_port }}"
             proto: tcp
      
         # PHP Info Page
         - name: Sets Up PHP Info Page
           template:
             src: "info.php.j2"
             dest: "/var/www/{{ http_host }}/info.php"
     
         - name: Clone Hashtopolis github repository
           git:
             repo: https://github.com/s3inlc/hashtopolis.git
             dest: /home/ansible/repos/
             clone: true
             update: true
         - name: Copy hastopolis/src/* to /var/www/hashtopolis
           copy:
             src: /home/ansible/repos/src/
             dest: "/var/www/{{ http_host }}/"
             remote_src: true
             owner: www-data
             group: www-data
     
         - name: Copy wordlist 100k
           copy:
             src: /home/ansible/my-project/hashtopolis/10-million-password-list-top-100000.txt
             dest: /var/www/hashtopolis/import/10-million-password-list-top-100000.txt
             owner: www-data
             group: www-data
         - name: Copy wordlist rockyou.txt
           copy:
             src: /home/ansible/my-project/hashtopolis/rockyou.txt.bz2
             dest: /var/www/hashtopolis/import/rockyou.txt.bz2
             owner: www-data
             group: www-data
         - name: Decompress rockyou.txt
           command: /usr/bin/bunzip2 /var/www/hashtopolis/import/rockyou.txt.bz2
         - name: Copy oneruletorulethemall
           copy:
             src: /home/ansible/my-project/hashtopolis/OneRuleToRuleThemAll.rule
             dest: /var/www/hashtopolis/import/OneRuleToRuleThemAll.rule
             owner: www-data
             group: www-data
       handlers:
         - name: Reload Apache
           service:
             name: apache2
             state: reloaded
     
         - name: Restart Apache
           service:
             name: apache2
             state: restarted

2. Run the playbook
   • ansible-playbook hashtopolis-server.yml

1. Configure the server using the Web UI
   • open web browser and point to the server's IP address
   • complete the installation gui to configure the server
     • server hostname: localhost
     • server port: 3306
     • mysql user: hashtopolis
     • mysql password: my_hashtopolis_password
     • database name: hashtopolis
   • create a login account when prompted
2. Allow voucher reuse
   • Click Config > Server
   • Click Server
   • Check “Vouchers can be used multiple times and will not be deleted automatically.”
   • Click Save Changes
3. Import word lists
   • Click Files
   • Click Wordlists
   • Under Import files select 10-million-password-list-top-100000.txt and rockyou.txt
   • Click Import
4. Import rule
   • Click Files
   • Click Rules
   • Under Import files select OneRuleToRuleThemAll.rule
   • Click Import
5. After configuration is complete, remove the install directory.

   • remove-hashtopolis-installer.yml

     ---
     - hosts: server
       become: true
       vars:
         http_host: "hashtopolis"
       tasks:
         - name: Remove install directory
           file:
             path: "/var/www/{{ http_host }}/install"
             state: "absent"

   • ansible-playbook remove-hashtopolis-installer.yml

1. Log in and create enough vouchers for all your worker nodes
   • Click Agents > New
   • Under Vouchers, and next to the New voucher button, click Create
   • Repeat to generate vouchers for all your workers
   • Save these voucher codes to /home/ansible/my-project/hashtopolis/vouchers.txt

Intel CPUs require this runtime: “OpenCL Runtime for Intel Core and Intel Xeon Processors” (16.1.1 or later)

• https://github.com/intel/compute-runtime/releases
• hmmmm sudo apt install intel-opencl-icd
• http://registrationcenter-download.intel.com/akdlm/irc_nas/vcp/15532/l_opencl_p_18.1.0.015.tgz

Testing: sudo crackers/1/hashcat.bin -a6 -m0 hashlists/1 ?d?d?d?d?d?d?d?d

1. Create a j2 template for the agent configuration file

   • config.json.j2

     {
       "files-path": "/home/ansible/files",
       "crackers-path": "/home/ansible/crackers",
       "hashlists-path": "/home/ansible/hashlists",
       "zaps-path": "/home/ansible",
       "preprocessors-path": "/home/ansible/preprocessors",
       "url": "http://{{ server_ip }}/api/server.php",
       "voucher": "{{ vouchers[play_hosts.index(inventory_hostname)]}}",
       "token": "",
       "uuid": ""
     }

   • See more options for this config file at https://github.com/hashtopolis/agent-python
2. Create unit file for the new hashtopolis-agent service

   • hashtopolis-agent.service

     [Unit]
     Description=Hashtopolis Agent
     After=network.target
      
     [Service]
     Type=simple
     ExecStart=/usr/bin/python3 /home/ansible/hashtopolis.zip
     Restart=on-failure
     StandardOutput=syslog
     StandardError=syslog
     SyslogIdentifier=hashtopolis-agent
     WorkingDirectory=/home/ansible
      
     [Install]
     WantedBy=multi-user.target

3. Create playbook to install the agent

   • hashtopolis-agent.yml

     ---
     - hosts: server
     - hosts: agents
       become: true
       vars:
         server_ip: "{{groups['server'].0}}"
         vouchers: "{{ lookup('file', '/home/ansible/my-project/hashtopolis/vouchers.txt').splitlines() }}"
       tasks:
         - name: Install prerequisites
           apt:
             pkg:
               - git
               - zip
               - curl
               - hashcat
               - python3
               - python3-psutil
               - python3-requests
               - pciutils
         - name: Pull agent
           get_url:
             url: "http://{{ server_ip }}/agents.php?download=1"
             dest: /home/ansible/
         - name: Create config file
           template:
             src: "/home/ansible/my-project/hashtopolis/config.json.j2"
             dest: "/home/ansible/config.json"
         - name: Create systemd unit file
           copy:
             src: /home/ansible/my-project/hashtopolis/hashtopolis-agent.service
             dest: /etc/systemd/system
             owner: root
             mode: 644
     
         - name: Just force systemd to reread configs (2.4 and above)
           systemd:
             daemon_reload: true
     
         - name: Start hashtopolis-agent service
           systemd:
             name: hashtopolis-agent
             enabled: true
             state: started

4. ansible-playbook hashtopolis-agent.yml
5. If some agents are not coming on-line, check the config.json for a missing voucher. Put in a voucher code and sudo systemctl restart hashtopolis-agent.service

1. Check the services using a playbook

   • check-agent-service.yml

     ---
     - hosts: agents
       tasks:
         - name: Get Service Status
           ansible.builtin.systemd:
             name: "hashtopolis-agent"
           register: hta_service_status
         - debug:
             var: hta_service_status.status.ActiveState

2. Log in to the Hashtopolis dashboard and view the agents
3. Edit each agent “Trust” setting by checking the box for “Trust agent with secret data”

1. Create a list of passwords you want to crack
   • Use a variety of passwords
     • poor passwords
     • kids passwords (https://www.dinopass.com/)
     • short leet passwords
     • short truly random passwords

       • sudo apt install pwgen -ypwgen
         pwgen 5 1
         pwgen 7 1

2. Put the passwords in a file passwords.txt

   • passwords.txt

     Butterfly123!
     returnofthejedi
     J@sonHouse
     sillywombat11
     mi$tyHelp55
     January2022
     P@$$w0rd
     Ewug4
     ieMuth6
     covidsucks

3. Create a list of md5 hashes of these passwords (we are cracking with very old NUCs after all) in the file hashes.txt

   • hash-passwords.sh

     #!/bin/bash
     file="passwords.txt"
     output="hashes.txt"
     while read -r line
     do
         /bin/echo -n "$line" | md5sum | cut -d' ' -f 1 >> "$output"
     done < $file

   • hashes.txt

     7c67bd5694775b082d3d858a1882afb1
     7dd02e107e35921b778bd4d61be734b1
     966ffda037af61805f2d797d4cafce12
     f81e55522520a7646c6d5a1c643cda9c
     2f43b4850a2ecd83471d7e938d54a636
     27d745dd658451e50f969e132e86de9f
     c53e479b03b3220d3d56da88c4cace20
     fd350c0534cff7b3e4bc6b99b11c1286
     cfbf38310f0bb6972addb10494db51db
     26dc3dac9e39bfa97fdf180899b9e81f

4. SORT the file
   • sort -o hashes.txt hashes.txt

   • hashes.txt

     26dc3dac9e39bfa97fdf180899b9e81f
     27d745dd658451e50f969e132e86de9f
     2f43b4850a2ecd83471d7e938d54a636
     7c67bd5694775b082d3d858a1882afb1
     7dd02e107e35921b778bd4d61be734b1
     966ffda037af61805f2d797d4cafce12
     c53e479b03b3220d3d56da88c4cace20
     cfbf38310f0bb6972addb10494db51db
     f81e55522520a7646c6d5a1c643cda9c
     fd350c0534cff7b3e4bc6b99b11c1286

5. Upload the hashes.txt file to Hashtopolis
   • Lists > New hashlist
     • Name: hashes.txt
     • Hashtype: 0 (md5)
     • Hashlist format: Text File
     • Hash source: Upload
     • File to upload: Click Choose File, then select the file
     • Click Create hashlist

1. Tasks > New Task
   • Name: demo
   • Hashlist: hashes.txt
   • Command:
     • Click Rules then check (under T) OneRuleToRuleThemAll.rule
     • Click Wordlists then check (under T) rockyou.txt
     • Command should be: “#HL# -r OneRuleToRuleThemAll.rule rockyou.txt”
   • Priority: leave 10 (greater than 0)
   • Maximum number of agents: leave 0
   • Task notes: demo
   • Color: A00000
   • Click Create Task
   • Under Assigned agents
     • For each node click Assign
     • WARNING if the task assignment fails, modify the agent(s) to be “trusted” with secret data
2. Wait for your job to complete
   • Click Lists > Cracks to view cracked passwords
     • First to be cracked:
       • P@$$w0rd
       • Butterfly123!
       • January2022
       • covidsucks
       • sillywombat11
       • Ewug4
     • Consider what the difference would with without use the rule or with the smaller word list
3. Tasks > New Task
   • Name: demo-brute
   • Hashlist: hashes.txt
   • Command:
     • -a3 #HL#
   • Priority: 10
   • Maximum number of agents: leave 0
   • Task notes: demo
   • Color: 00A000
   • Click Create Task

Classic Hashcat

root@kali:~/Desktop# hashcat -m 0 -a 0 -o cracked.txt target_hashes.txt /usr/share/wordlists/rockyou.txt
</bash>
 
  * -m 0 designates the type of hash we are cracking (MD5)
  * -a 0 designates a dictionary attack ("straight" mode)
    * 1 combination
    * 3 brute-force
    * 6 hybrid wordlist + mask
    * 7 hybrid mask + worklist
    * 9 association
  * -o cracked.txt is the output file for the cracked passwords
  * target_hashes.txt is our input file of hashes
  * /usr/share/wordlists/rockyou.txt is the absolute path to the wordlist file for this dictionary attack
 
rules
<code bash>
hashcat -m 0 <$hash/file> rockyou.txt -r rules\OneRuleToRuleThemAll.rule

optimized rule that works great for md5 and other fast hashes

https://github.com/stealthsploit/Optimised-hashcat-Rule

1. Create the file /home/ansible/my-project/fah/remove-hashtopolis.yml

   • remove-hashtopolis.yml

     ---
     - hosts: server
       become: true
       become_user: root
       tasks:
         - name: Stop and disable services
           service:
             name: "{{ item }}"
             state: stopped
             enabled: false
           with_items:
             - apache2
             - mysql
         - name: Remove packages
           apt:
             name:
               - apache2
               - mysql
               - python3-pymysql
               - php
               - php-pear
               - php-mysql
               - libapache2-mod-php
             state: absent
             autoclean: true
             purge: true
         - name: Remove directory
           file:
             state: absent
             path: /var/www/hashtopolis
     - hosts: agents
       become: true
       become_user: root
       tasks:
         - name: Stop and disable services
           service:
             name: "{{ item }}"
             state: stopped
             enabled: false
           with_items:
             - hashtopolis-agent
         - name: Remove packages
           apt:
             name:
               - git
               - zip
               - curl
               - hashcat
               - python3-psutil
               - python3-requests
               - pciutils
             state: absent
             autoclean: true
             purge: true

2. Run the playbook: ansible-playbook remove-hashtopolis.yml

NOTE After running playbook to remove Hashtoplis, I would that upon reinstalling the server, the Hashtopolis server PHP stopped working. The following are commands to fix that issue.

sudo apt install php-fpm
sudo a2enmod proxy_fcgi setenvif
sudo a2enconf php8.1-fpm
sudo systemctl restart apache2

Try cracking other hashes,

1. Create a user account name tryhackme on NUC 1 with a simple password, like password
2. Dump the hash for the user tryhackme
   • sudo grep tryhackme/etc/shadow | cut -f 2 -d“:”

attempt 1 hashlist:

• name Unix
• $6$R7h5ysTPvsdNZNO6$54ndCR5fYQEf.W5aTnk9lRXBBNq/SeASUB3yVf43vO/DgJtD/JUn6h3n9TayPBCXF4fL/bWmF/OFay1RKfuGi1
• hashtype 1800 - sha512crypt, SHA512(UNIX)
• No check for salted hashes, separator
• No check for salt is in hex (only when salted hashes)

attempt 1 task:

• name unix
• hashlist Unix
• just worklist rockyou.txt
• priority 5

Note that I'm still working out the specifics on why benchmarks are seeming to get stuck.

1. On the Windows 10 PC
   1. Add an account with the a simple password, such as password
      • net user add person /active:yes /add
      • net localgroup administrators /add person
        • making this user an administrator makes it show up easier to find in the password hash dump
      • net user person *
        • set the password to something easy like password
   2. Extract the SAM and SYSTEM registry hives
      • Open a shell as Administrator
        • reg save hklm\sam c:\sam
        • reg save hklm\system c:\system
        • the last parameter is the location to copy the file two
2. On Linux
   1. Copy the SAM and SYSTEM registry hives to the Linux system
   2. Install samdump2
      • sudo apt install samdump2 -y
   3. Dump the system keys and hashes
      • If the files are named “sam” and “system”
        • samdump2 system sam
   4. For your test user (i.e. person) there are two hashes
      • one for LM authentication, (deprecated and only populated with a value to ensure the syntax remains constant)
      • the other is the NTLM string
      • For example, for person:
        • LM aad3b435b51404eeaad3b435b51404ee
        • NTLM 31d6cfe0d16ae931b73c59d7e0c089c0
3. In Hashtopolis
   1. Create new hashlists
      • LM
        • Name: LM
        • Hashtype: 3000 - LM
        • Paste in text, the LM hash you dumped
        • Create
        • Trying Runtime Benchmark
      • NTLM
        • Name: NTLM
        • Hashtype: 1000 - NTLM
        • Paste in text, the LM hash you dumped
        • Create
   2. Create new tasks
      • LM
        • Name: LM
        • Hashlist: LM
        • Enable rule OneRuleToRuleThemAll.rule
        • Enable worklist rockyou.txt
        • Priority: 10
        • Attack command: #HL# rockyou.txt -r OneRuleToRuleThemAll.rule
      • NTLM
        • Name: LM
        • Hashlist: LM
        • Enable rule OneRuleToRuleThemAll.rule
        • Enable worklist rockyou.txt
        • Priority: 9
        • Attack command: #HL# rockyou.txt -r OneRuleToRuleThemAll.rule
        • If the plaintext comes back blank….??