Working from the command line is the easiest way to explore a new dataset of text files. However, some tasks require more higher level tools. I wrote a bunch of python scripts for exploring data to complement the standard command-line utilities. The scripts are vary from general (e.g., binning log lines by a common column value or values) to domain specific (e.g., computing the stack distance between values in the file).
Many times I find the need to split a task across multiple physical machines. For example, bandwidth bound tasks like using curl to repeatedly fetch web content for many websites is faster using the pipes of multiple servers or processing bound tasks likes clustering samples is quicker using many CPUs. So, I wrote a python wrapper around ssh that split input across a set of machines and collects output.
Ever find yourself needing to know what content delivery networks (CDNs) a website uses? Me too. This tool collects all of the DNS hostnames of the objects embedded in the root HTML object of websites and compares the hostnames against a dictionary of known hostnames used by common CDNs. The dictionary can always be improved. Please write me with suggestions!
Many people use their ISP's resolvers for DNS resolution. However, there are many public DNS resolvers available on the Internet which may offer better performance. Use this tool to see your options and compare performance to get the fastest resolutions possible.
The Domain Name System (DNS) is responsible for converting domain names, e.g., www.google.com, into Internet Protocol addresses that are used to route traffic on the Internet. To convert a domain name, your computer issues a DNS resolution request which is performed on your behalf by a set of servers known as DNS resolvers.
This tool is designed to help you learn about how your DNS resolutions are being handled. It works by sending specifically designed DNS resolution requests that cause the DNS infrastructure to interact with a server under our control. In addition to discovering the DNS resolvers that you are using, we also test the resolvers for vulnerabilities to several well known attacks. Go here to test your resolver now.