So, recently I've been wondering why people are scrambling to do everything on top of HTTP (SOAP? Atom?). Why not make a separate protocol for each, using different ports, etc?. I know that would be a pain to convince hosting providers to open ports for all of your pet applications, but isn't this the way the internet is supposed to work? Aren't we just adding another level of needless indirection?
Here's my tenative answer.
First, but least importantly, there's probably stuff that the HTTP protocol defines that may be useful in a variety of settings and which you don't need to redefine.
Second, the web usually defines a namespace over part of the server machine's directory structure. That is, by accessing http://example.com/cgi-bin/program3.cgi, I am pointing to a certain program in the cgi-bin directory of the server. This allows the client to specify which program on the server they want executed. TCP/IP alone only defines a way to point to a target program by means of a port #.
Third, and more importantly, using HTTP simplifies configuration for the server administrator; in order to run a new non-HTTP application, your hosting service must run a program that you provide as a daemon. In order to run a CGI application, your hosting service just has to drop a file into the cgi-bin directory.
Forth, also very importantly, the web server acts as a dispatcher to these incoming program requests. This allows CGI scripts to avoid the operating-system specific details of being a TCP/IP server. If you wrote some application independently of HTTP, and made it available via the internet, it would have to be a server. If you wanted to port it to Windows, you'd have to learn about how to write Windows server.
By writing applications as CGI scripts, the web server does the interaction with the OS's TCP/IP interfaces for you.
In other words, the HTTP+CGI protocol is a remote procedure call protocol (akin to a very lightweight CORBA), and the webserver is middleware. These components abstract away some of the details of client-server interaction. Compared to making a new TCP/IP standard, developing a standard on top of HTTP makes it easier:
SOAP and friends try to build in some of that "lost" complexity and infrastructure on top of CGI. However, HTTP is oriented towards connectionless, asymmetric processing. Hence, it cannot be ideal for all applications (for example, one would probably not want to run a real-time, graphical MMORPG over HTTP).