[Rabbit-dev] build a powerful server utilizing multiple proxy servers

[Robert Olofsson]

Hi!

On Thu, 28 Jul 2011 23:24:26 +0800
蔡超 <toppiprc at gmail.com> wrote:
>     I want to build a powerful proxy server using rabbit. The main idea is
> to make a pool of proxy addresses, and maintain a mapping from a target host
> to a proper proxy server which can reach the target fastest. At the same
> time, there should be a powerful cache system with 10 GB level volume. I
> have write a ProxyChain which can choose a Resolver for a given url. I can't
> find any detail document for rabbit architecture, there are still two
> problems:

The code is the documentation of the architecture. There is no real roadmap
or similar, things get added when someone either supplies a patch or requests
a feature that someone cares to develop.

If you have specific questions I can probably answer them.

> 1.  How to know the time a proxy cost to fetch an url? where is the code
> using the given proxy address?

Rabbit does not currently try to guess how much time a request will take.
Internet is unreliable so how can you really say.
Rabbit does not do anything special today, it checks the request header
to see if it may use its own cache and if so checks if the resource
is currently in the cache. If the requested resource is in the cache
(or needs a revalidation) rabbit contacts the real server (possibly 
through a chained proxy) and asks for it. 
If the response from the real server allows caching then rabbit will
cache the resource.

> 2.  Can Rabbit  cache 10GB files?

I do not really know. I have personally never had that large a cache for
rabbit, but in theory it ought to be ok. You could try to write a test
case and see what happen, or you can try it for real.
My rabbit cache is currently about 20 000 files, but that is only 63 MB
resource data. Most resource on the net are very small. Do you plan to 
have any large files in the cache?

I would guess that we may want to do some modifications to the cache
if we want it to support 10GB.

I also guess it depends on what filesystem and hardware you are going
to put it on.

/robo