cs146a Project 2: Blocking Web Proxy

Introduction

You will create a simple web proxy that forwards requests from a client (your browser) to remote web servers, and then forwards the response back to the client. Your web proxy will be single-threaded and so can only service one request at a time.

Both Server and Client

From the perspective of a user of your proxy (i.e., you while you are testing), the web browser is the client and the web proxy is the server. From the perspective of your web proxy, however, the proxy is the client and the remote web server is the server. In this way, your web proxy will behave as both client and server.

Setting up a Proxy

Web browsers can be configured to use a web proxy; but, the configuration is a bit different for different web browsers. You may find testing with links to be convenient. You can fetch webpage source with links by entering the following command:

$ http_proxy=http://localhost:8888/ links -source http://www.brandeis.edu

This assumes of course that your proxy has been started on port 8888 (this will be explained below). Also, if your shell isn't Bash (but it probably is, type echo $SHELL to find out), you may need to specify the proxy environment variable differently.

Because you must ensure that images work too, you will want to test in a graphical browser as well. To use a proxy in Firefox, do the following:

1. Visit the page about:config
2. Scroll down until you see the Preference Names beginning with "network" (the Filter box can help)
3. Make the following changes to your settings:
   • network.http.proxy.version = 1.0
   • network.http.proxy.pipelining = false
4. Go to the Firefox preferences, and under the Advanced tab click the connections settings button and change the proxy to manual config. Set the server to localhost and the port to whatever you start your proxy on (e.g., 8888).

Be aware that since your proxy will not implement all of HTTP/1.0 some sites and activities will not work correctly, but GET-ing HTML, Javascript, and image data should all work.

Network Connections

Like your previous project (project 1), your proxy will use the socket abstraction for building network applications. Unlike your previous project, you will be responsible for both client- and server-side socket programming.

Resources

• Using TCP Through Sockets by David Mazières, Frank Dabek, and Eric Petererson

Project Requirements

Proxy Functionality

Your proxy will work with a useful subset of the HTTP/1.0 spec (see RFC 1945). Specifically, your proxy must be able to handle the following:

Please read these requirements carefully!

• Short- and Full-Requests must be handled.
  • A short request is an HTTP/0.9 request, which takes the following format:

    GET /path/to/resource
    

    Note that this command only needs to be terminated by a single carriage-return (\r\n).
  • A full request is an HTTP/1.0 request that may include an arbitrary number of headers:

    GET /path/to/resource HTTP/1.0
    Host: www.example.com
    Accept: text/html text/plain
    Accept-Encoding: gzip, compress
    Accept-Language: en
    

    Note that this command must be terminated by two carriage returns (\r\n\r\n). These are just a few examples of HTTP headers. You should pass all headers through the proxy verbatim as part of the request you make to the remote web server.
• You must terminate any request that is longer than 65535 bytes (this might happen if the client sends too many headers).
• Your proxy must accept both hostnames and numeric addresses in the URL.
• You should limit the hostname to no more than 1024 bytes.
• Your proxy must transfer both text and binary files (e.g., images).
• Your proxy must be able to browse typical web pages like the Brandeis homepage without leaking memory or failing to load some of the page content.
• If your proxy encounters an error (like a connection being reset by a client or server), it should print an error on stderr and close any open file descriptors, but it should not stop running. Your proxy must be able to encounter errors and then continue running. (You may terminate for errors like out of memory, although an out of memory error might indicate a bug like excessive malloc()ing).
• You may not allocate more than 1MB of buffer space.
• You must be able to handle responses of any size (remember, requests have a limited size, but responses do not). This means that your program must not buffer the entire response before sending it to the client.

Non-Requirements

Your proxy does not need to do the following:

• Handle POST or HEAD (or anything from HTTP/1.1)
• URLs of any protocol other than http (e.g., you do not need to handle https)
• Parse or reorder headers (you may treat all lines beyond the "GET ..." line as opaque data; however, it is required that you send all headers verbatim to the remote server provided that the request is under the length limit).
• Your code does not need to be multithreaded or use asynchronous I/O. This means that only one connection at a time can be proxied.

Parsing the Request

When a browser sends an HTTP/1.0 request to a proxy server, it sends the same format request that it would send directly to a web server, except that it includes the domain name in the resource locator.

Here is an example of an HTTP/1.0 request sent directly to a web server:

GET / HTTP/1.0
Host: www.brandeis.edu
Accept: text/html, text/plain
Accept-Encoding: gzip, compress
Accept-Language: en
User-Agent: Lynx/2.8.4rel.1 libwww-FM/2.14

And here is an example of how the browser requests the same resource through a proxy server:

GET http://www.brandeis.edu/ HTTP/1.0
Host: www.brandeis.edu
Accept: text/html, text/plain
Accept-Encoding: gzip, compress
Accept-Language: en
User-Agent: Lynx/2.8.4rel.1 libwww-FM/2.14

As you can see, the only difference is in the first line, the full URL including the protocol and hostname are given.

It is up to you to parse the first line of any HTTP/1.0 request sent to your proxy server and

1. Find out the host name (and port if given, remember that in a URL the port is specified in the form "http://www.example.com:8080"; i.e., separated from the domain name by a colon); and,
2. Rewrite the request to only include the path part, not the protocol, domain name, or port (essentially, you need to follow the rewriting example shown above).

The results of this parse will be a rewritten request that can be sent to the remote web server, the hostname (or IP address) of the remote web server, and the port to use to connect to the remote web server. These results are sufficient to forward the request and proxy the response back to the client.

Invocation

Your client program will be named proxy, and will be invoked on the command line like this:

$ ./proxy port

• port : port on which to listen for proxy requests; this is the port you will use when configuring your browser to use a web proxy.

For example:

$ ./proxy 8888

Where it Must Run

We recommend that you develop on a Linux machine. You can log into any of the Brandeis CS department public machines to work, or you can work in the berry patch. You are also free to develop on your own computer, but be aware that in order to receive credit your assignment must compile and run on one of the RHEL machines in the berry patch.

Tips and Suggestions

• You may find that a port number is already used on a machine you are testing on. The error when this happens will look something like this:

  Cannot bind to TCP port 8888 (Address already in use)
  

  If you occurs you can just chose another port.
• You may find this line of code helpful to keep "Broken Pipe" errors from killing your program (issue it from main before you start accepting connections):

  signal(SIGPIPE, SIG_IGN);
  

  This is discussed on page 18 of the TCP Through Sockets handout.
• If you are feeling overwhelmed, try to code up a simple echo server first, and test it with telnet (or with your sc program from project 1!); then add a parser and test that you can correctly rewrite the request and extract the host and port; finally, add the client portion and forwarding of requests and responses.

Extra Credit

You may earn 10% extra credit on this assignment by augmenting your code to use fork() to service multiple requests at once. In order to earn extra credit you must do the following:

1. Submit both versions of your code: the single-process version and the multi-process version that uses fork
2. Demonstrate that your multi-process version maintains the correctness properties of your single-process version
3. Submit a short experiment demonstrating the improved speed of your proxy when using multiple processes

Your next assignment will give you an opportunity to program with concurrency, so do not feel that you will be missing out on concepts by not completing the extra credit. The extra credit is intended for students who really enjoy programming and are looking for a greater challenge.

Collaboration

• You may discuss the assignment with other students in the course
• You may use the Internet for general help with socket programming
• You may not look at any solutions by other students from the current or previous years, and you may not look for solutions to similar problems on the Internet
• You may not turn in any code written by another student, nor may you share your code with any other student

How to Hand In

Follow the instructions in the FAQ.