If you don't find an answer, please click here to post your question.

Helpful Tools for Capturing Network Traffic

0 Kudos
Started ‎04-11-2017 by
Modified ‎04-11-2017 by

Helpful Tools for Capturing Network Traffic

by Ooyalan on ‎04-11-2017 07:34 AM (616 Views)
There are some options for getting a 'debugging proxy' set up on your desktop so you can capture the network traffic of a browser or native app. A debugging proxy is otherwise known as a man-in-the-middle proxy and can be used to log any communications that a browser or native mobile app makes when that software is using it as the proxy.
 
• Charles - https://www.charlesproxy.com/download/ - Paid/Free-trial - available for Mac (and now PC, though at the moment not so good as the Mac where it originated)
• Fiddler - http://www.telerik.com/fiddler - Free - available for PC (and now Mac - but it was originally developed on PC)
• MITMProxy - https://mitmproxy.org/ - Free - for Linux and other terminal-oriented systems.
 
It's usually straightforward to set up and capture HTTP traffic - Charles and Fiddler both have good documentation on running them and configuring the browser or device to use that machine as the proxy for internet requests.
 
On browsers it varies:
• Safari (Mac) - you need to set the proxy for *all* apps running on the Mac, not just Safari - https://support.apple.com/kb/PH21420?locale=en_GB - note that Charles can do this for you automagically
• Firefox (Mac/Windows/Linux) - you can set this in the Firefox Preferences, you can also use an add-on like FoxyProxy to make this a bit easier to enable and disable. Note that Charles on Mac with Firefox will make it easier; Fiddler has some integration on Windows to make that easier also.
• IE/Edge (Windows) - Fiddler has integration with the Windows system and browsers can be set to proxy or the entire system can proxy. https://www.google.co.uk/search?client=firefox-b-ab&q=setting+proxy+on+IE+edge
• Chrome - on Windows you can access proxy in the advanced settings under tools, on Mac it's fixed to system wide so you'll need to follow instructions for Safari above.
 
On Mobile App Development 'IDEs' like Xcode or Android Studio - you can set the simulators to use Proxies as you would on their hardware counterparts. But please remember debugging in simulators for video/adtech is not always the best - things like DRM will never work - so definitely confirm there is a bug when it runs on real hardware where possible.
 
For HTTPS traffic all of the debugging proxies will need permission to decrypt the passing responses that are encoded using SSL. When, for example, a web browser speaks to a secure website - a default configured proxy without being able to handle SSL, will simply log 'gobble-de-goock' as it will be unable to decrypt it.
 
Therefore you need to have the browser or app/device accept the proxy's own SSL certificate. Once you have done this you will be able to intercept the SSL data for that website using the debugging proxy. I've listed some useful things from each proxy software which might help you...
 
Charles
========== 
General intro to SSL proxying with Charles:
 
 
For instructions on accepting Charles SSL Certificates see here...
 
 
On iPhone here's some additional detail...

https://www.charlesproxy.com/documentation/faqs/ssl-connections-from-within-iphone-applications/

 
Note that, although you will have enabled SSL Certificates to be trusted you will also need to tell Charles which hostnames you want debugged - it will only debug a whitelist of hosts listed in the preferences.
 
 
Fiddler:
==========
 
 

 
 
Note that once you have the proxy running for iOS / Android device, or other device 'remote' to your Fiddler installation, you should be able to use the default browser on that device to visit
 
 
MITMProxy
============
 
Since this is really for developers and linux-wranglers - I'm not going to go into detail here, the documentation is available on the website, and SSL Certificates are available like with Fiddler and Charles - a special hostname accessed via the browser of a device proxied via MITMProxy will let you proxy SSL data. Please note that, my experience with MITMProxy is limited and it is difficult to get a nice 'HTTPArchive' (aka 'HAR') log file out of it at this point - Fiddler and Charles do this no problem.
 
 
Final comment: remote debugging over the public internet
==============================================
 
Whilst Charles and Fiddler were originally designed for basic debugging on a local network, so you might be debugging an iOS iPhone app and you have the iPhone connected on wifi to the same network that the machine running Charles/Fiddler is on, they can be used to remote debug a device on an unknown network. MITMProxy is also designed for this sort of thing.
For any of them - you just need a computer instance (virtual maybe) running in an environment on an internet connection that allows that computer to be addressable from the public internet. For example - I have had success spinning up an Amazon AWS Workspace with Windows, remotely connecting to the desktop, making sure the workspace has a public IP address, installing Fiddler and allowing it to receive proxy requests from the public internet - then on my Android device connected on a local Mobile Operator (e.g. Three in the UK) I configured the proxy to connect via the Fiddler instance. It then meant I could capture the traffic even when the device was connected on cellular/mobile data connection. This is something that you could do for debugging an end user's device with your app if you really needed to do that and the end user was suitably skilled at configuring their device with a proxy.