How can I capture all HTTP traffic on a specific port using code?

December 02, 2005 • 14 comments

On this page:

Seems like there’s a huge variety of people stopping by here with all sorts of skillsets and so I’m going to take advantage of that and pick YOUR brain today …

I need to build an application that needs to capture Web requests and Web traffic from a client. If you’ve ever used Microsoft Web Application Stress Tool or ACT in Visual Studio you get the idea. Or heck a better example yet - Fiddler or some similar type of HTTP proxy. Basically I’d like to bring up a browser and capture ALL request traffic the browser is running and capture the HTTP headers to replay them later.

You might think this is pretty easy – after all you can use IE and BeforeNavigate to capture URLs, POST data and headers. Unfortunately BeforeNavigate captures only top level URLS and only those you actually navigated to in some way, not things like images stylesheets etc. There are other events that look like they might work like DownloadFile etc. but none of them seem to capture the supporting files for a page itself. So I don’t think this can be done using the IE COM interfaces.

My next thought was to use some sort of packet filtering and capture the Web traffic that way. There are actually a number of reasonably good free packet capture tools out there for .NET and I played around with that for a while. Problem is most do not work against localhost – they actually need to run a full network connection. Accck... Also, even if I managed to get this to work this seems like a pretty hairy way to go.

The last thing I’ve been thinking of is to build some sort of forwarding proxy. Listen on a proxy port capture the data then forward. But even that looks to be a pretty major undertaking given the way HTTP chunks data – it’s not going to be trivial to put the HTTP responses together properly…

Peter Bromberg suggested taking a look at Fiddler with Reflector – which I did a while back actually, but the proxy stuff in Fiddler is pretty spread out.

I guess I’m wondering what approaches that I missed here or whether somebody knows of some sort of pre-packaged mechanism that can accomplish this task without me having to get my hands into low level network plumbing…

The Voices of Reason

Sietse Wijnker
December 02, 2005

# re: How can I capture all HTTP traffic on a specific port?

Hi Rick,
I guess you want to log all file downloads within IE. It must be possible using an add-on, because thats how some content-blockers work. They are plugged into IE and will exmine the url of each file and will allow/deny downloading the file based on the domain/filename/extension/whatever. Maybe you can research in that direction also?
Some titles you can look for:
-IEWatch
-IE HTTP Analyzer
-IE Spy (VB Source code):http://www.freevbcode.com/ShowCode.asp?ID=4301

Matt
December 02, 2005

# re: How can I capture all HTTP traffic on a specific port?

Ethereal will let you do this, and you can view the packets as a TCP/IP stream which provides, which keeps you out of the networking guts. If your browser can be Firefox there's the Tamper Data and Live HTTP Headers extensions. I think one of those extensions has an IE version as well.

Rick Strahl
December 02, 2005

# re: How can I capture all HTTP traffic on a specific port?

Well I know there are tools that can do this - I need something I can build into my application to do this capturing for me so I can distribute it with my app.

Josh Twist
December 02, 2005

# re: How can I capture all HTTP traffic on a specific port?

Hi Rick,

This will capture the traffic from your client
http://www.pocketsoap.com/pcaptrace/ (it's a really neat tool) - but it won't be part of the application.

Can you reiterate exactly what you want to do? I think you want to capture all traffic leaving a 'client' and targeting your application including the headers? But you want it to live on the server?

Bertrand
December 02, 2005

# re: How can I capture all HTTP traffic on a specific port?

Ask Nikhil how he does it in his own tool.

Rick Strahl
December 02, 2005

# re: How can I capture all HTTP traffic on a specific port?

Hi Josh,

Yes pcap is something that I've been meaning to look into but last I checked a long while back it looks at hte network adapter so it didn't work against localhost.

What I need to do is capture all HTTP traffic leaving the local machine. Basically I'm building a customized stress tool, so I need to capture where the user goes to fire the requests including 'related' content.

Matt
December 03, 2005

# re: How can I capture all HTTP traffic on a specific port using code?

Would the IE plugins be applied if you used the IE control? Similiarly, if you were using the Gecko engine maybe one of the Firefox extensions could be convinced it wanted to work?

Rick Strahl's WebLog
December 04, 2005

# Add a Web Server to your .NET 2.0 app with a few lines of code

.NET 2.0 includes a very useful and more importantly easily integrated HttpListener class that makes it very easy to add a Web Server to your own applications. Can you say quick communication between two apps?

JosephCooney
December 06, 2005

# re: How can I capture all HTTP traffic on a specific port using code?

I used this http proxy written in C# to write my own proxy that "dopped packets" (to simulate some kinds of network failures) http://www.mentalis.org/soft/projects/proxy/ - it hasn't been worked on in a while but IIRR it was pretty well done and it might get you started. I can't help but think you'll end up knowing the System.Net namespace REALLY well after you write this....

gozh2002
January 10, 2006

# re: How can I capture all HTTP traffic on a specific port using code?

it seems in C++ you can hook up a LSP layer on top of existing tcp/ip layer, and I think that mostly how the current spam adware does to monitoring your network flow...

my question is how possible to do that in .net?

Emre Eren
December 01, 2007

# re: How can I capture all HTTP traffic on a specific port using code?

This article is really old but I need the exact functionality with ie like yours. I wonder did you find a solution or at least an idea for this?

Altug
August 04, 2008

# re: How can I capture all HTTP traffic on a specific port using code?

Hi Rick,

I was looking for a solution similar to yours. I need to capture the http traffic that is generated from a flash file in a browser and save the requests to a file. I thought I could do this with javascript but I doubt that. Any ideas?

Thanks
Al

Tony
November 26, 2008

# re: How can I capture all HTTP traffic on a specific port using code?

Hi Rick,

Heard you can capture all the requests using wininet. But have not been able to as of yet.
Have you been able to capture requests without using a proxy?
Iam looking into wininet cause it lets you clear cookies, noticed this in the new http analyser.
Anyone been able to do this ?

Thanks,
Paul

megha
June 30, 2010

# re: How can I capture all HTTP traffic on a specific port using code?

hi rick

I need a help I have to write a c code to read a url and to get all the html web page data in a file or string,plz help me how can i write this kind of application in c.
thanking you in anticipation

megha

Rick Strahl's Weblog