I took an extended leave from IT for most of 2009 until returning later in December and therefore haven’t had to deal with the Citrix CVE-2019-19781 vulnerability but I just so happened to come across an unpatched Citrix ADC the other day so I thought I’d go through the process as outlined by my friend Thomas Poppelgaard (https://www.poppelgaard.com/cve-2019-19781-what-you-should-know-and-how-to-fix-your-citrix-adc-access-gateway) in detail to capture and show some screenshots of what an exploited Citrix ADC would reveal during the tests as well as the remediation I performed at the end. I won’t be reiterating all the details about the tests so please refer to his blog post for a more in depth description of why and what the test are for.
Confirming Vulnerability and Compromise
CISA and Citrix both offer tools to check the Citrix ADC for the vulnerability:
CISA UTILITY TO CHECK IF CUSTOMER IS VULNERABLE
https://github.com/cisagov/check-cve-2019-19781
CVE-2019-19781 – VERIFICATION TOOL FROM CITRIX
https://support.citrix.com/article/CTX269180
… but a community tool created by zentura_cp is available on Azure to perform the check quickly. That tool can be found here: https://cve-2019-19781.azurewebsites.net
Scanning the Citrix ADC in this environment confirmed that it is vulnerable to CVE-2019-19781:
The wildcard certificate check did not reveal a match but I have recommended the client to revoke and reissue the certificate.
Checking the Template Files
As indicated by Thomas Pooelgaard’s article, the exploits all write files to the following three directories:
/netscaler/portal/templates/
/var/tmp/netscaler/portal/templates
/var/vpn/bookmark/
There is no need to switch to the shell prompt from the CLI as you can execute an ls on these directories as such:
shell ls /netscaler/portal/templates/*.xml
shell ls /var/tmp/netscaler/portal/templates
shell ls /var/vpn/bookmark/*.xml
Initiating a ls command for those directories return the following:
shell ls /netscaler/portal/templates/*.xml
> shell ls /netscaler/portal/templates/*.xml
/netscaler/portal/templates/02a81b35.xml
/netscaler/portal/templates/04c86902.xml
/netscaler/portal/templates/06fb0cee.xml
/netscaler/portal/templates/0bc583f8.xml
/netscaler/portal/templates/17EOk.xml
/netscaler/portal/templates/1Z5g92Aky.xml
/netscaler/portal/templates/1d9363f1.xml
/netscaler/portal/templates/2YMBK.xml
/netscaler/portal/templates/3QJW6.xml
/netscaler/portal/templates/3YDd6gheCtB8mrJL2zfyO4WKl7sG0MQa.xml
/netscaler/portal/templates/473ce5e1.xml
/netscaler/portal/templates/48ZwobXq0yjVtU3GeC2a95kBfAHNu1Qz.xml
/netscaler/portal/templates/4GgZh.xml
/netscaler/portal/templates/53963742.xml
/netscaler/portal/templates/5BdZA.xml
/netscaler/portal/templates/5qIJsMV1x9aUGzKirLuNtRYX032QjnvS.xml
/netscaler/portal/templates/5t4ls.xml
/netscaler/portal/templates/64c2192d.xml
/netscaler/portal/templates/6fef6135.xml
/netscaler/portal/templates/77bd07f9.xml
/netscaler/portal/templates/7BA1C.xml
/netscaler/portal/templates/7ESdi.xml
/netscaler/portal/templates/8HtqLVavu.xml
/netscaler/portal/templates/8W7Cgvyck.xml
/netscaler/portal/templates/8cc4d5fb.xml
/netscaler/portal/templates/9HS1qhTa5cyMXYjeIdbV06LQum3nlxB2.xml
/netscaler/portal/templates/BbMZ6.xml
/netscaler/portal/templates/CHQsB.xml
/netscaler/portal/templates/E9TG5.xml
/netscaler/portal/templates/FLQBEv6FQfjjxFfjn2dSRjGHf8WjgpN6.xml
/netscaler/portal/templates/FUcrY.xml
/netscaler/portal/templates/FnMd4HqxlN0Tar9eX3sYD1L8OtPbA5gG.xml
/netscaler/portal/templates/GWwH1.xml
/netscaler/portal/templates/GbRqm.xml
/netscaler/portal/templates/GgJlo.xml
/netscaler/portal/templates/GiDNJIaYtW7EubqBCLF3rHKodsQ16wRc.xml
/netscaler/portal/templates/H8yeh.xml
/netscaler/portal/templates/H9Qwv12zTE3bUcVZMK7Wga5hBfJGxiu0.xml
/netscaler/portal/templates/HitNA.xml
/netscaler/portal/templates/JTQ5sTyzWXWdQRLHrhWukGTgvAHX8Be9.xml
/netscaler/portal/templates/MAkw4mge4.xml
/netscaler/portal/templates/NpWPYQq2DCndZfFVix74PB6owv0QrOUp.xml
/netscaler/portal/templates/O0aAJQSycUFHDsk32gflC98iTK6bRpm1.xml
/netscaler/portal/templates/O9Bon2fAzapuXry7Q0JM41LTKS8l6DmR.xml
/netscaler/portal/templates/OhjZf.xml
/netscaler/portal/templates/R8p9l.xml
/netscaler/portal/templates/RWM4aJ9GhcvSTZLxAreg0qOo5fsCuz8y.xml
/netscaler/portal/templates/TlkhDg4uyEWRHX1fxV2t9AJ3OM0iKmYz.xml
/netscaler/portal/templates/UHKl9QzvkTpJi34wqWORhGN0bDYsCPX1.xml
/netscaler/portal/templates/X3h40kvVpMxjYIKbqlwsa5eLtuNTf281.xml
/netscaler/portal/templates/YOrPCVxZizJjLauTmRyoWUf5Xg1F4G6E.xml
/netscaler/portal/templates/YP68c.xml
/netscaler/portal/templates/YjpRX.xml
/netscaler/portal/templates/a436n.xml
/netscaler/portal/templates/ad140234.xml
/netscaler/portal/templates/aenp7.xml
/netscaler/portal/templates/bf327bcc.xml
/netscaler/portal/templates/c1ktNfwQ76mDi9y2bVEY4Ze8qJajSCvs.xml
/netscaler/portal/templates/cUAwMikvxEePnrOV0bs6NRzT28lt71jq.xml
/netscaler/portal/templates/cVSMUbaDF4TeAL8C9tpmGf15PNR6sd2W.xml
/netscaler/portal/templates/cs0lxv5ppnrxebo4.xml
/netscaler/portal/templates/czqhO.xml
/netscaler/portal/templates/d12affe3.xml
/netscaler/portal/templates/dddd[%template.new({‘BLOCK’=’print`id`’})%].xml
/netscaler/portal/templates/dqe82VQf8.xml
/netscaler/portal/templates/eMlUc.xml
/netscaler/portal/templates/f81961c6.xml
/netscaler/portal/templates/fhLCc.xml
/netscaler/portal/templates/hK3Sn.xml
/netscaler/portal/templates/hjXNi.xml
/netscaler/portal/templates/jLBscdX0Q.xml
/netscaler/portal/templates/k42bayNsu1hxf7pZ98UwWPviVYDOAmIM.xml
/netscaler/portal/templates/niphc4KmTRS9E0XkbAo5eBQWMtsYF3zG.xml
/netscaler/portal/templates/oQZFe.xml
/netscaler/portal/templates/orfuxqzupb.xml
/netscaler/portal/templates/pswIV.xml
/netscaler/portal/templates/qTvCLFw9HnfAzFycKGJSdQ9XxOnCityF.xml
/netscaler/portal/templates/qVIgn5Mljz7as2Q6PrkDHBbmoRuhx13v.xml
/netscaler/portal/templates/qrfgO.xml
/netscaler/portal/templates/qui4o.xml
/netscaler/portal/templates/ryMG7.xml
/netscaler/portal/templates/sDZIbl7ZNsL1HVBCzowAq3oVLY2aiexj.xml
/netscaler/portal/templates/somuniquestr.xml
/netscaler/portal/templates/tcHzZ6um0MrlEVXT8dyAwBOx3kYs2go7.xml
/netscaler/portal/templates/v905U.xml
/netscaler/portal/templates/wTJIb.xml
/netscaler/portal/templates/xfWjDEZ0K4lSaRgHvecQ2987GihIYTUX.xml
/netscaler/portal/templates/yR39d.xml
/netscaler/portal/templates/yVStWwCFy9BDXBxjIGvCk3h67Gx4Zm8E.xml
/netscaler/portal/templates/yndBMhT2L.xml
Done
The results from above indicate that there are plenty of xml files that have filenames suggesting the Citrix ADC has been compromised.
Proceeding to execute the following:
shell ls /var/tmp/netscaler/portal/templates
… reveals that the /var/tmp/netscaler/portal/templates directory appears to be fine.
Executing the last command:
shell ls /var/vpn/bookmark/*.xml
> shell ls /var/vpn/bookmark/*.xml
/var/vpn/bookmark/1Fx3kynwZ.xml
/var/vpn/bookmark/OqU8uS1bW.xml
/var/vpn/bookmark/bNRrtwSjF.xml
/var/vpn/bookmark/cx4yylzvg.xml
/var/vpn/bookmark/fKchklbXP.xml
/var/vpn/bookmark/fSdeA2hEn.xml
/var/vpn/bookmark/lztbVXgWEKqmoQLWPjbM.xml
/var/vpn/bookmark/nsroot.xml
/var/vpn/bookmark/pwnpzi1337.xml
/var/vpn/bookmark/testtest.xml
/var/vpn/bookmark/z8P0sSAGK.xml
Done
>
… reveals that the /var/vpn/bookmark/ directory also displays the same type of files and suggests the ADC has been compromised.
Reviewing Apache Log Files
Executing the first 3 commands to review the Apache logs does not reveal any results:
shell cat /var/log/httpaccess.log | grep vpns | grep xml
shell cat /var/log/httpaccess.log | grep “/../”
shell gzcat /var/log/httpaccess.log.*.gz | grep vpns | grep xml
However, execute the following command does:
shell gzcat /var/log/httpaccess.log.*.gz | grep “/../”
The suspicious xml files are notably present in the results above.
Executing the commands to determine whether there are any scheduled tasks to maintain access after patches does not reveal any jobs that a compromised Citrix ADC would have:
shell cat /etc/crontab
shell crontab -l -u nobody
Checking for Backdoor Scripts
Executing the following two commands to look for backdoor scripts:
shell ps -aux | grep python
shell ps -aux | grep perl
… reveal the following:
> shell ps -aux | grep python
nobody 82094 0.0 0.1 60980 1852 ?? Ss 28Jan20 0:37.76 /var/python/bin/python2 -c -c (python2.7)
root 36830 0.0 0.1 9096 912 0 RL+ 1:51PM 0:00.00 grep python
> shell ps -aux | grep perl
nobody 53632 100.0 0.0 5804 272 ?? R Sun05AM 3176:31.20 /tmp/.perl
nobody 77911 0.0 0.0 5804 412 ?? I 28Jan20 0:07.52 /tmp/.perl
root 36832 0.0 0.1 9096 1024 0 S+ 1:51PM 0:00.00 grep perl
>
The presence of nobody scripts is a red flag that there are backdoor scripts currently on the NetScaler and we can see that there are 3 of them listed.
Checking for Cyrpto Mining
Executing the following command to list the current running processes:
shell top -n 10
… reveal the following:
> shell top -n 10
last pid: 36937; load averages: 2.00, 2.00, 2.00 up 123+23:58:22 13:53:29
77 processes: 3 running, 71 sleeping, 3 zombie
Mem: 27M Active, 3504K Inact, 1487M Wired, 11M Cache, 169M Buf, 2528K Free
Swap: 4198M Total, 299M Used, 3899M Free, 7% Inuse
PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND
1174 root 1 44 r0 1210M 1210M CPU1 1 2976.0 100.00% NSPPE-00
53632 nobody 1 119 0 5804K 272K RUN 0 53.0H 100.00% .perl
1176 root 1 44 0 83392K 5668K kqread 0 30.7H 0.00% nsnetsvc
1297 root 1 44 0 37128K 3232K kqread 0 387:11 0.00% nsrised
1434 root 1 44 0 54752K 444K nanslp 0 316:34 0.00% nsprofmon
994 root 1 44 0 76720K 1356K select 0 112:35 0.00% vmtoolsd
1199 root 1 44 0 99M 11940K kqread 0 111:55 0.00% nsaggregatord
1331 root 2 76 0 70184K 13168K ucond 0 41:09 0.00% nscopo
25 root 1 44 0 29784K 29872K kqread 0 37:05 0.00% pitboss
1314 root 1 44 0 63928K 9820K nanslp 0 30:48 0.00% nscollect
Done
>
The presence of the nobody account running a perl script that is consuming 100.00% of the WCPU suggests malicious activity currently running on the Citrix ADC.
Reviewing the Bash Logs
Executing the following 2 commands to review the bash logs:
shell cat /var/log/bash.log | grep nobody
shell gzcat /var/log/bash.*.gz | grep nobody
… does not reveal the nobody account present:
The following 2 commands display the full output of both logs allowing you to manually comb through the entries but the results from all the tests above are sufficient enough to deem the appliance as being compromised:
shell cat /var/log/httperror.log
shell gzcat /var/log/httperror.log.*.gz
Remediation
This particular environment did not have a backup pre-December 2019 for me to restore so the only viable option that was available is to redeploy the VPX appliance and restore the configuration. The latest backup I could find dates back to 2 years ago but given that there has not been any change to the Citrix ADC, I decided to download the same build, redeploy, restore backup configuration, upgrade appliance and swap out the existing one.
Prior to completing the swap it is also important to perform the following:
- Review all of the servers that the Citrix ADC interacts with to determine whether they have been breached. These services could include domain controllers, Citrix StoreFronts, Citrix Delivery Controllers, Licensing servers and others.
- Update all of the passwords both local to the ADC and credentials used to interact with other services such as LDAP lookups against Domain Controllers.
- Force password changes for all users who have logged onto the ADC to access resources. As most Citrix ADCs allow Active Directory integration, it is important to force password changes for AD accounts.
- Revoke and renew all SSL certificates on the Citrix ADC. Use the list of files on the ADC’s /nsconfig/ssl/ directory as the hackers who have breached the appliance would have been able to retrieve the files in there.
- Ensure that the version of the Citrix ADC is not vulnerable to the CVE-2019-19781.
- Once the above have been completed, proceed to place the Citrix ADC back on the internet and run the vulnerability scan against it.
Restoring Configuration from Backup
As indicated above, the backup I had to work with was a few versions back from the NS12.1 51.19.nc that the appliance was currently running:
Citrix’s official Backup and Restore documentation (https://docs.citrix.com/en-us/netscaler/12/system/basic-operations/backup-restore-netscaler-appliance.html) indicates we can use the same or newer build but just to avoid any unintended issues, I downloaded the same build and redeployed the appliance as the running version would need to be upgraded anyways. The first step for the restore is to get the backup file uploaded and since a restore can only be performed via the GUI, I will use it to upload the configuration backup.
Begin by navigating to System > Backup and Restore then click on the Backup button:
Select the Add button:
Click on the down arrow beside Choose File and select Local as the source then select the backup TGZ file:
**Note that it is important to choose the Local option because the Appliance option would allow you to upload but it will not be displayed as a backup in the options to restore.
With the backup uploaded, proceed to restore it from the TGZ package:
Note that there is no need to have backup the appliance since it’s a new deployment so select the Skip Backup checkbox:
The restore should be fairly quick and will display a Restore Successful at the bottom right hand corner of the window:
The restored configuration will not be active until you restart the appliance and it is important to not select Save configuration if you restart it from the GUI or else the empty configuration will overwrite what you have just restored.
Also note that the IP address configuration will match the original appliance so if you have left the other one online for forensics but prevented it from accessing the internet then you should either change its IP address to avoid conflict or move it to another network.
If you’re staging the new appliance and have not given it network connectivity then the following are useful commands to verify the restored configuration:
- show ip – list the NSIP, SNIP, SIP, MIP, VIP IP addresses
- show lb vserver – displays the Load Balancing Virtual Servers details
- show lb vserver -summary -fullvalues – displays the list of Load Balancing Virtual Servers without truncating names (use | grep to add specific values to list such as UP or DOWN)
- show ssl certKey – to list all certificates on the appliance
- show ssl certLink – to list the links between Root CA and Subordinate CA certificates
It’s important to note that the previously installed Client certificates will be present but you should deem these as being compromised so proceed to revoke, reissue and reinstall those certificates.
