In previous parts of this series we discussed nmap parsing in Part I http://isc.sans.edu/diary.html?storyid=9091, and nessus parsing in Part II http://isc.sans.edu/diary.html?storyid=9328. The nessus parse script has been updated as a number of people have recommended changes or improvements. The latest version of the .nessus v2 report parsing script is here: parsenessusv2mysql.pl
The majority of the examples I give use a command line interface and are typically run on Linux. They can be ported to Windows or other platforms quite easily.
For Part III of vulnerability assessment testing automation and reporting we will discuss:
using nmap results to configure nessus scans;
using the nessus XMLRPC interface to kick off nessus scans;
converting .nessus v1 files to .v2;
and splitting up large .nessus v2 files.
Using nmap results with nessus
A good reference for using nmap with nessus is the documentation at the following:
http://www.nessus.org/documentation/index.php?doc=nmap-usage
The first step is to download the nasl files to use nmap with nessus:
wget http://www.nessus.org/documentation/nmapxml.nasl
wget http://www.nessus.org/documentation/nmap.nasl
Copy them to the nessus plugins directory:
sudo cp nmapxml.nasl /opt/nessus/lib/nessus/plugins/
sudo cp nmap.nasl /opt/nessus/lib/nessus/plugins/
Stop the nessus daemon (on Ubuntu):
sudo /etc/init.d/nessusd stop
Recompile the nessus plugins:
sudo /opt/nessus/sbin/nessusd -R
Start nessus back up (on Ubuntu):
sudo /etc/init.d/nessusd start
One way to tell is the above was successful is to log into https://localhost:8834 (nessus XMLRPC interface using a web browser, requires Flash) and create a new policy, if Nmap (NASL wrapper) and Nmap (XML file importer) show up under the preferences it worked. The script to make use of the nmap XML files as part of the nessus configuration via the XMLRPC interface is shown in the next section.
Using the nessus XMLRPC interface to kick off nessus scans
If in the past you kicked off your nessus scans on the command line it probably looked something like this:
/opt/nessus/bin/nessus -V -T nessus -q localhost 1241 username password $host.ip $host.nessus
Which was great for scripting and running cron jobs. The side effect of using this method is that the output formats typically used were .nbe or . nessus v1 files. .nbe files were easily parsed however they contained very little of the wealth of metadata available in the .nessus formats. The .nessus formats being XML are even easier to parse, and contain much more information than previous nessus reports. Tenable has indicated that .nbe file and the old nessus command line tool are being phased out. In order to achieve the same output format as the GUI clients (v2) and still be able to run scans and interact with the nessus server via a command line interface the XMLRPC interface was introduced. The Net::Nessus::XMLRPC perl module is a great way to script nessus scans. I also use it to convert the many .v1 files I have to the .v2 format, as in the script in the next section. The perl module can be found here: http://search.cpan.org/~kost/Net-Nessus-XMLRPC-0.30/ and often installed like this: sudo cpan Net::Nessus::XMLRPC
The script is located here: http://handlers.dshield.org/adebeaupre/runscannessusxmlrpc.pl
Usage: runscannessusxmlrpc.pl [options]
Options:
-u, --user Nessus username (string)- defaults to nessus
-p, --password Nessus password (string) - defaults to password
-s, --scanname Name to give the nessus scan (string) - defaults to XMLRPC scan
-n, --policyname Name of the nessus policy template to use - defaults to first in list
-l, --list Text file containing a list of targets (string) -required
-x, --nmapxml File containing nmap xml results (string) - optional
-c, --copy Flag to copy the policy (flag) - optional
-q, --quiet Only print errors (flag)
-h, --help Brief help message (flag)
Example: ./runscannessusxmlrpc.pl -u nessus -p password -s scan-name -l targetlist
Converting .nessus v1 files
If you run (or used to) your nessus scans using the old cli interface the reports generated follow the .nessus v1 file format. These can be parsed using a variety of scripts and tools, however the .nessus v2 file format contains much more data and is easier to process. One simple method to convert them is simply to upload them via the nessus XMLRPC interface. This can be achieved using the GUI, or an easier solution is to create a text file that contains the names of the .v1 files to upload and run a script.
The script is located here:http://handlers.dshield.org/adebeaupre/uploadreports2nessusxmlrpc.pl
Usage: uploadreports2nessusxmlrpc.pl [options]
Options:
-u, --user Nessus username (string)- defaults to nessus
-p, --password Nessus password (string) - defaults to password
-q, --quiet Only print errors (flag)
-f, --file List of Nessus version 1 results files (string)
-h, --help Brief help message (flag)
Example: ./uploadreports2nessusxmlrpc.pl -u nessus -p password -f list-of-nessus-v1-files.txt
When the upload and scans are complete the reports are here: /opt/nessus/var/nessus/users/nessususername/reports/
Splitting up large .nessus v2 files
One of the issues with parsing XML files is their format, another is choosing an XML parser, and a third is their size. In the case of nessus reports they can get quite large if you either scan a large number of hosts or have a large number of findings per hosts. This can happen quite easily with a credentialed scan for example. If you parse a large report with the parsenessusv2mysql.pl script you can get out of memory errors. The script uses the XML::DOM module, there are other XML parsers that do not load the entire XML document into a tree structure. There are a few ways to avoid this issue, one of which is to split the nessus report into smaller chunks. The script below uses xml_split which is part of the XML::TWIG perl module. It is a simple bash script which takes a single parameter of the .nessus file to split.The number of hosts per report is configured to 10 in the script, but can easily be changed (divisor). If the number of hosts in the original report is equal to or less than the divisor the new report is identical to the old.
The script is located here: http://handlers.dshield.org/adebeaupre/splitnessusv2.sh
Example: ./splitnessusv2.sh nessusv2-test.nessus
I am more than open to suggestions, or better ways of doing things. Part IV will be parsing other tool outputs for database import that don't have a fancy XML format. Part V will be the scripts and techniques to wrap all of the other parts together following a reasonable methodology. Let us know if you use this script, something like it, or some other technique to manage security test data. Contact us or use the comment fields below.
Cheers,
Adrien de Beaupr
EWA-Canada.com
(c) SANS Internet Storm Center. http://isc.sans.org Creative Commons Attribution-Noncommercial 3.0 United States License.