httppp logo

HTTP Passive Probe (httppp)

 

About

Httppp is a network capture analyzer which extract HTTP request response times from a previously recorded network capture file.

It has been written because no free software which perform this analysis existed. Even Wireshark does not provides those figures.

It is intended for measuring response time of HTTP inter-application calls (so-called web-services), not to measure HTTP browsing user experience quality, for which there are far better tools on the market (such as Fiddler, Firebug and a bunch of expensive but far richer softwares hidden behind buzzwords like "user experience management").

Actually, httppp does not perform any statistical analysis, but produces a CSV file which can be easily imported into a spreadsheet software to be analyzed through pivot table feature and to compute percentiles of the TTLB or TTFB distribution (see snapshots below)

Features

Httppp key features are to:

  • Read network capture file in libpcap format, which is produced by most network sniffers such as tcpdump or Wireshark.
  • Analyze TCP packets to find HTTP among them and match requests and replies to compute response time (TTFB/time to first byte and TTLB/time to last byte) along with usual information about an HTTP hit (host, path, return code, source IP address, etc.).
  • Provide a graphical user interface (GUI) to interactively browse hits and TCP packets with interactive sort to easily find longest service calls (or shortest, or 5xx return codes, etc.).
  • Produce hits table in CSV format, to make it easy to import into a spreadsheet software.
    • The CSV format is designed to be easily transformed into a pivot table to group response times by host, path, source IP, return code, custom fields, etc. depending on needs. Disclaimer: I personnaly use a very well known non-free spreadsheet software developed at Redmond, WA and am happy with it. If you use another one, you're maybe a better free software advocate than I am, but don't complain with its pivot table feature not being as powerfull.
    • The CSV format is also usable to draw a distribution diagram by letting the spreadsheet software compute percentiles. This is often the first diagram you'll wan't since it displays information such as "80 % of requests are served in less than 600 ms".
  • Support customizable fields to extract any information from the request or reply. This is done with regular expressions and make it possible to catch any header (such as Referer, Content-size or SOAPAction) or any fragment of a POST payload (again, the SOAP action may be relevant) or the reply payload. See snapshot below. Some interesting regexp are:
    • Referer: Referer:\s*(\S+)
    • SOAP action in header: SOAPAction:\s*"(\S+)"
    • SOAP action in POST: <\w*:?Action[^>]*>([^<]+)</
    • URI path without query string: GET\s+(/[^?#]*)\S*\s+HTTP/1\.\d
    • invoked method name in Spring HTTPInvoker serialized Java request: ([A-Za-z0-9._]*)..\[[^[]*$
    • class name in Spring HTTPInvoker serialized Java request: (com.mycompany.[A-Za-z0-9._]*[A-Za-z0-9_])
    For exact syntax reference, please refer to Qt's QRegExp class documentation, since it's the regexp engine used by httppp.

Screenshots

Main GUI displaying (from left to right): TCP conversations list, TCP packets tree, analyzed HTTP hits table, packet content:

Main GUI with everything closed but HTTP hits table, exhibiting regexp custom extraction of referer header in custom field 1:

CSV HTTP hits file produced by httppp, displayed inside a famous (whereas not free) spreadsheet software:

Same information analyzed in a pivot table (only 30 seconds of manual work after previous snapshot):

Percentiles of TTLB distribution (yet another 30 seconds of manual work later):

Known limitations and bugs

The most obvious limitation is due to the fact that, under heavy workload, most network capture tools don't capture every packets but only most of them. Therefore most capture files will lack packets, which makes the analysis harder (it's harder to reorder TCP packets when some packets are lacking) and produces incorrect data (when the lost packet is the one containing the HTTP request, even guessing that there was a request packet cannot tell at which microseconds it occured). Even though httppp is resilient to packet lost in many cases, they will create artifacts, such as:

  • Some hits will be lost. If one of them is the longest or shortest, the computing of maximum or minimum TTLB in your spreadsheet will be false.
  • If a request packet is lost, its response will be considered as a part of previous hit response if there is a previous hit in the same TCP connection, hence the previous hit will be reported as longer than it is, possibly far longer (especialy if the TCP connection has been inactive for a while).

HTTP features not (yet) supported:

  • HTTPS/SSL traffic isn't analyzed.
  • Proxied HTTP traffic isn't correctly analyzed (i.e. the host is assumed to be set in request headers and the request URL is assumed to begin with a slash).
  • HTTP pipelining would be misinterpreted and produce inconsistent figures. Please note that even though it can be an optimization for web browsing in some circonstances, HTTP pipelining is most of the time inefficient for web-services and should not be activated.
  • IP fragmentation is not supported either (fragmented IP packets are ignored). Please note that in regular operation conditions any modern OS won't use IP fragmentation for TCP communication and will instead enable path MTU discovery algorithm therefore you should never have to analyze HTTP traffic inside fragmented IP packets.
  • IPv6 is not supported.

Other limitations

  • The GUI crashes with large capture files, reading more than 100 MB is not a problem, but several GB will crash the GUI. I apologize for that, but it won't be fixed until it is easy for me to provide httppp in 64 bits, i.e. until 64 bits Qt is easy to handle on MinGW on Windows (until now 64 bits Qt on Windows only work out of the box for MSVC compiler). This is likely to change in a few months.

There are no known bugs (provided you read a capture files without any packet lost, which is nearly impossible), but feel free to report any bug (see contact address at the bottom of the page).

Download source code or binary packages

Source code is hosted at bitbucket:

Some precompiled binary packages of recent versions are also available here:

Roadmap and News

There is no official roadmap. However next versions are likely to fix bugs that may be reported and to circumvent limitations described above.

If you want to be warned when a new version is released, feel free to follow the project on Freecode.

Technological details (a.k.a. geek's corner)

Httppp is a Qt/C++ free software making intensive use of some Qt idioms such as signals and slots and implicit sharing. It uses libpcap to read network capture files.

If you want to contribute to the software (or fork it), feel free to use bitbucket links above and clone, do what you want with git, and then to submit merge/pull requests.

The capture file size limitation is mainly due to httppp GUI being compiled as a 32 bits executable.

Everything in the source code should be portable to other OSes, especially MacOS X which is well supported with Qt. I didn't compile it yet by myself on MacOS X because it would imply finding, downloading, installing and testing libpcap and a network sniffer.

 
Contact:

Copyright © 2012 all rights reserved