jaybosamiya/GSoC 2014 Mini Projects.md

## GSoC 2014 Mini Projects.md

      
    Raw
  

              GSoC 2014 Mini Projects.md
            
          
    GSoC 2014 Mini Projects

Reducing Zenmap Topology Noise


Zenmap produces some really noisy Topology views if there are multiple (parallel) anonymous hops.
For example: a->b->X->c->d, and a->b->X->c->e (where X means anonymous) would cause 2 anonymous nodes to appear.
Anders Sundman had sent a patch a long time ago for this but it handled only a very small subset of all the possible cases.
Wrote a patch that handles almost all the cases. Not sure of what to do in two cases so have mentioned it in the mail to list and have asked for feedback.
Address comments: Waiting for comments
Commit: Waiting for review

Analyzing MemoryError Catching Patch (with statistics)

Look here for the analysis.
Look here for the post on nmap-dev list.
--exclude-ports option for Nmap


Identify feature: Nmap has had multiple requests for an option to exclude ports. This would allow for easy writing of scans similar to nmap -p 1-1000 --exclude-ports 23,443 instead of writing the cumbersome nmap -p 1-22,24-442,444-1000 which is currently required to do the same
Requirements:

Look at the RFC post on Nmap Dev List
Should also handle excluding ports from host discovery


Establish test cases to determine success:

Different combination of -p, --top-ports, and --exclude-ports are used. They are selected in such a way that all combinations of the following are tested.

No ports remain after exclusion
Some ports that are included are excluded
The excluded ports and included ports are disjoint


Made a test script to test proper behaviour.


Code: Sent to list
Test: Passes all test cases.
Address comments: Host discovery exclusion was not working when --top-ports was also used. Fixed this (and added it to the test set). Waiting for further review.
Commit: After reviews

Catching the MemoryError bug in Zenmap


Identify bug: We get many reports on the mailing list about Zenmap crashing with a MemoryError. We want to atleast catch this bug and provide the user a more useful error message so that we get less of the out-of-memory "crash reports".
Steps to reproduce the bug:

Running Zenmap on a system with lesser memory (or in a VM) is an option; however this is cumbersome. Thus the softlimit tool is used.
Some scans cause a MemoryError and some don't. Upon further analysis, it seems that scans that generate large output cause the MemoryError.
These points lead to the following steps to cause the bug to appear reliably

Run softlimit -a 1000000000 python zenmap
Use nmap 127.0.0.1/24 --packet-trace -p- -r as the Nmap Command in the GUI.
Run the scan and wait for a while
BOOM!


Analysis: The error seems to occur due to the large Nmap output. Further analysis shows that the problem seems to occur whenever the output window is refreshed. Correcting this would be a huge task involving file based paging (kept this for the next Mini Project). On the other hand, just showing an error message and quiting Zenmap would be much more easier. A more preferred outcome would be to continue running the scan without showing the output (and instead showing a message that says this).
Coding a fix: The MemoryError is not localized to just the output window and multiple try-except wraps are needed. Each wrap causes one bug to disappear but allows for the next to be caught. It is a very time consuming procedure, especially with the Nmap Command taking extremely long to run. I seem to have caught all the places now and am showing a useful error message but further testing is required.
Test: Running multiple Nmap commands seems to work fine. Tried using nmap --packet-trace -r --top-ports 1000 127.0.0.1/24 instead now and it seems to be a better command for testing (since it runs faster). Tried to click on all options and places in Zenmap to make sure that things work as they should. As far as I can see, everything works perfectly (except for the Nmap Output window, obviously). The only things that seem to have a problem are Save (Ctrl+S), Save all scans to directory, and Print... feature. I cannot test Opening large scans right now, but I believe that this may also have some problem (since it'll try parsing the large file). Some analysis showed that the opening large file thing is slow due to a problem with slow concatenation of strings. Will look into this as a different Mini Project.
Comments: Waiting
Addressing comments: Only after receiving comments
Commit: Only after addressing comments

Done

Making terminal sane after Canceling a scan in Zenmap


Zenmap sends a SIGKILL to the Nmap process that is running whenever a user presses "Cancel Scan"
Sending a SIGTERM allows for cleaner shutdown
Sometimes SIGTERM might not work
At that time, need to send SIGKILL
Best idea - Send SIGTERM. Keep polling every 0.5 seconds. If process is still not TERMinated at the end of 5 seconds then KILL it.
Thread on list
Committed as revision 33106.

Adding the --comment --data <hex string> and --data-string <string> options to Nmap


Identify new feature: The feature is to add a comment to the sent packet data during scans (for example "pH33r my l3eT s|<iLLz!  I'll 0wN UR b0x!").
Requirements: --comment should accept a string and attach it to the data similar to the way --data-length attaches random data to the sent packets. After reading Kris's reply: Error should be thrown if --data-length and/or --comment are used multiple times or together.
Test cases to determine success: Options and errors should work fine. What needs to be tested is whether the comment is actually sent. How to test this is to be seen.
Code: Done and put on mailing list. Modified the code and added in the --data-string <string> and --data <hex string>. Needed to take some code from nping/utils.* and put it into utils.* with modifications so that hex string can be successfully converted.
Test: Options and errors work fine. Whether the comment is sent or not is to be tested.
Comments: Kris had coded this in a while ago but nmap was in feature freeze at the time. Waiting for Fyodor to comment on this. Got a comment from Dan which says that naming it --data-string <string> may be better. Also, can include a --data <hex string> to take a hex payload.
Addressing comments: Done
Commit: Committed as revision 33014
Possibilities of follow-up: Testing for max size (since IPv6 shouldn't have the 1400 limit warning). This'll need support for jumbograms to be added.
Follow-up: Need to update docs/refguide.xml to document the new feature.

Speeding up Zenmap's Scan Opening routine


Any large scan in Zenmap takes an unnecessarily large time to open
The problem lies in Python's string concatenation thing being slow. This makes the whole "Open File" algorithm quadratic in time complexity while it should be linear for speed.
Fixing this would require refactoring some of the nmap_output and _nmap_output code since this is in a tangle
Using StringIO.StringIO() seems the best bet
Patch: Submitted
Tests: Ran many scans. Saved them. Reopened them. Works much faster. Scans open in seconds instead of hours now.
Review: Passes all checks
Commit: Committed as revision 32983
Follow-up Commit: cStringIO is faster than StringIO. Sent in a patch to use cStringIO whenever possible and fallback on StringIO otherwise. Committed as revision 32984.

Updating libpcap to 1.5.3


Wrote up a patch to modify libpcap/NMAP_MODIFICATIONS so that the modifications work with libpcap 1.5.3
Wrote a script (bash script) that checks out nmap's latest svn, downloads libpcap 1.5.3, runs the patching process and generates a diff.patch file that can then be used to update the svn.
Testing: All tests worked perfectly (on Ubuntu 14.04 x64).
Reviews: (previously got a "works fine" from Gisle, in March) Turns out, patch 0003-*.patch and 0004-*.patch are not required at all. They have been fixed upstream.
Started off a discussion as to whether we should keep "ports" of 3rd party libraries in nmap (i.e. do we need to keep tagging them along esp. since they are so easily available). Will lead to either removal of these libraries or with the update of libpcap.
Commit: Committed as revision 32966
Follow up commit: 2 files are generated when we run ./configure and these need not be there in the SVN repo. So I set svn:ignore property for them in revision 32967.

Weird behavior with arp ping


Nmap received a report which said that Nmap behaved differently from other IP stacks (Linux, win). It sent ff:ff:ff:ff:ff:ff instead of 00:00:00:00:00:00 for the arp target MAC address.
The behaviour can be traced to scan_engine.cc which has a function sendArpScanProbe which uses some libdnet functions, macros and constants.
At one place, there is a call which uses the constant ETH_ADDR_BROADCAST which is ff:ff:ff:ff:ff:ff (or "\xff\xff\xff\xff\xff\xff" as it is in the code)
Research: Using Wireshark to scan the network that I have access to, all arp packets (arp requests) have 00:00:00:00:00:00 (except for packets from Nmap). Changing the call to have any value seems to have no effect on the working. Conforming to the OSes seems to be the best course of action.
Fix: Changing the call to have a "\x00\x00\x00\x00\x00\x00" instead does the trick.
Testing: All tests work as normal. Wireshark shows that the new arp packets are indistinguishable from the ones sent by to OS.
Reviews: Turns out that RFC 826 says that the field doesn't matter; and Wireshark Wiki on Gratuitous ARP says that Solaris uses ff:ff:ff:ff:ff:ff. Thus Nmap's implementation is not "wrong" but that the patch will make it conform to the way currently popular OSes do things.
Commit: Committed in revision 32920

KRAD Ascii Art (Randomizing the ASCII art shown from the ./configure script)


Nmap shows the dragon ASCII art at the end of the ./configure script
The todo list had an item which said that a random ASCII art must be shown from docs/leet-nmap-ascii-art*.txt
Coded this in initially using sort -R, but Dan pointed out that sort -R is a GNU extension so will break on non-GNU systems
Modified it to use the POSIX-specified awk
Review: Dan said that the new patch was fine. He said that the ASCII art, however, was part of the "branding" of Nmap, so the decision of the images that I included in the patch should be upto Fyodor. He said that just the randomization part could be committed in. This was back in March.
Commit: Committed the randomization part in revision 32919
Follow up commit: Turns out, if anyone ran autoconf then the randomization would be gone. Had to modify configure.ac so that this randomization is not removed on runnning autoconf. Committed this as revision 32965.

Fixing runtime interaction error when using sudo


Steps to reproduce bug: Run any nmap command using sudo or through a script and try to use keyboard interaction (eg: press ?). Nothing happens.
Some extra requirements: nmap should quit with terminal in sane state when Ctrl+C etc. is sent. Core dump ability should be preserved.
Fix: In nmap_tty.cc, change getpid() to getpgrp() at two places to allow for keyboard interaction even when called through sudo or through a script. This however throws in a problem of terminal not remaining in a sane state when Ctrl+C is pressed. Add a signal handler for SIGINT, SIGTERM and SIGQUIT and make it call tty_done(), reset default handler and raise() same signal.
Test: Run nmap with and without sudo. Try different key presses when it is running. Try Ctrl+C, Ctrl+\. Try ulimit -c unlimited; sudo nmap 127.0.0.1 -p- and then pressing Ctrl-\. Core dump generation is preserved and terminal remains in sane state in all these. No tests failed.
Review: Review done by Patrick (basically, the patch's main idea was from Patrick's code in Chirp).
Commit: Committed in revision 32873.

Giving a more descriptive error on --ssl in Ncat when OpenSSL is not compiled in


Steps to reproduce "bug": Compile Ncat without OpenSSL and run ncat with any of the --ssl options (i.e. --ssl, --ssl-cert, --ssl-key, --ssl-verify, --ssl-trustfile)
Some extra requirements: Should ncat show a message saying "OpenSSL not compiled in" and continue? or should it quit?
Code in a fix: Decided that quitting will be better due to security issues that may arise by continuing.
Test: Compiled both with and without OpenSSL and ran make check and compared the results. Ran the ssl options with and without other options and checked whether it works fine.
Review: "Looks good"
Commit: Committed in revision 32881.