Installing and Configuring Squid
Pages: 1, 2
Dos and Don'ts
- DO put a name server on the machine with Squid. It's an extra level of caching, and minimizes choke points. Don't overload a single DNS for a cluster of boxes, especially if that server has other things to do.
- DO use the national root DNS closest to you as a forwarder. If someone else has looked up the same address recently, it should be cached. Do not use it as your only forwarder. Use at least one other machine on your network, and one on your provider's network.
- DO increase the size of your
ipcache. Bigger is better. Stale addresses are less important than many entries and long TTLs. Cache addresses for at least 24 hours, and negative cache for at least 5 minutes.
- DON'T cache big objects. Next to CPU and RAM, disk I/O is your biggest bottleneck. Try not to cache anything over about a megabyte.
- DO split your cache over several physical drives. Four 5-Gbyte drives are better than one 20-Gbyte drive -- you save time using multiple spindles. IDE will serve you as well as SCSI, unless you're using Ultra Wide SCSI.
- DON'T put two cache drives on the same IDE controller, but you can put four high-speed or six ordinary-speed SCSI drives on a single SCSI controller. Don't mix Ultra-Wide SCSI drives with non-Ultra-Wide SCSI drives on the same chain.
- DO keep your logs on a non-cache drive, and preferably on a different chain or controller.
- DO test your hardware before trusting it. Some servers have incompatible or semi-compatible combinations of hardware: It's tragic to see a USD$30,000 server with Ultra Wide SCSI that has poorer disk I/O throughput than a P100 with a 5400-rpm IDE drive. You may have to buy it before you can test it, but at least you can warn the boss before it goes into service.
Configuration for economy
Saving dollars is a large part of what proxy caches are all about. It's easy to waste your proxy -- and real money in bandwidth -- if you don't understand what's going on. It's also easy to save money with one if you know the issues. Most web servers and web content are operated or produced by people who don't really understand the HTTP protocol. As cache administrators, their errors are going to land on your shoulders.
Disk space and memory
A cache can always use more disk space, but as the size of your disk-cache grows, you will need more memory to index it. There's a straightforward rule for memory.
Divide the size of your disk cache by 13 Kbytes, and multiply that by 130 bytes. Add the size of
cache_mem, and add about 2.5 Mbytes more for executable files, libraries, and other sundry overhead. For example: We have a 10-Gbyte drive, and a
cache_mem of 8 Mbytes.
10 Gbytes/13 Kbytes = 769,230
769,230 x 130 bytes = 99,999,900 bytes (or 97,656 Kbytes)
97,656 Kbytes + 2.5 Mbytes + 8 Mbytes = 10,849,656 Kbytes or about 108 Mbytes
The example server needs 108 Mbytes available to Squid to support 10 Gbytes of
Provide as much disk space as you can provide RAM to support it. Squid performs very badly when it starts to swap. Remember to set aside memory for anything else on the machine (DNS,
cron, operating system, etc.).
Refresh patterns determine the lifetime of the object. Within an object lifetime, Squid will serve the object without requesting an IMS ("if modified since") request. Once the lifetime is exceeded, Squid will keep the object but will send an IMS request to the origin server. If the object has been modified since it was first cached, Squid requests the new copy. If not, it keeps the old copy. Either way, the object is marked as fresh again.
Here's our (default) basic refresh pattern:
refresh_pattern . 0 20% 4320
The dot (
.) is the the regular expression pattern, and matches anything. It uses POSIX regular expressions. (See
man 7 regex).
The zero (
0) is the minimum freshness time. If it's anything other than zero, it will override any expiration headers given with the object. If the content provider actually provided an expiration header, we should usually honor it.
The last term (
4320) is the maximum freshness time. The object becomes stale after this many minutes in the cache.
The 20 percent is used for our default case, for when there's no information from the content provider about the lifetime of the object. Squid takes x percent (20 percent in this example) of the difference between the last-modified time of the object and the current time, and uses that as the object lifetime. If the object lifetime is less than the minimum set by the
refresh_pattern, it is increased to at least that. If it's greater than the supplied maximum, it's reduced to that.
Some kinds of files can be maintained much longer than others. Zip,
.exe files rarely change content without also changing name. Using regular expressions, we can create a set of refresh patterns like this:
refresh_pattern -i exe$ 0 50% 999999 refresh_pattern -i zip$ 0 50% 999999 refresh_pattern -i tar\.gz$ 0 50% 999999 refresh_pattern -i tgz$ 0 50% 999999
Refresh pattern options
Note that these options violate the HTTP standard. Do not use them lightly.
override-expire pretends there is no expiration header on the object and calculates purely based on last-modified times. This permits you to cache sites that abuse the use of expiration headers, but also inhibits updates of frequently changed content (such as news sites).
ignore-reload prevents the object being refreshed when the user presses the refresh button on their browser. This does not perform well when the object has no content length -- you may wind up with a broken object that the users cannot reload.
reload-into-ims transforms reloads into validations. Beware: Web servers may permit an object to be updated without the last-modified time being altered. The server may then insist that the object is still valid when it actually is not.
More Dos and Don'ts
- DO increase
maximum_object_size. 40 Mbytes is not too large. 800 Mbytes might cache the large downloads.
- DO make sure you have at least one or two local name servers for Squid to query. Don't let it query any other servers directly. This keeps Squid's offsite DNS requests to a minimum.
- DO increase the
- DO have a parent proxy query if you can. It's cheaper to talk through a hierarchy than directly to multiple sites.
- DON'T use ICP if you have a single parent you always use.
- DO use
calamaristo analyze your logs periodically and look for changes you can make to your
refresh_patterns. There is no single "good" set. Users change their browsing habits and create new files, and new technology is always being developed. Adapt.
Caveats and gotchas
In many clients, "reload" forces the cache to reload from the origin server. This can cause testing problems.
Don't test Squid with pages that have "do not cache" headers. Squid will not cache them.
access.logwhen testing to verify that you're pulling pages from Squid.
Access control lists have an implicit last line which reverses the rule of the last explicit line.
Squid can improve browsing speed and reduce HTTP bandwidth. The
squid.conf file gives great flexibility, but can be initially daunting. These settings let you get started -- but are just a start. Experiment!
- Squid, A User's Guide
- Squid configuration manual
- RFC 2616: Hypertext Transfer Protocol -- HTTP/1.1
Jennifer Vesperman is the author of Essential CVS. She writes for the O'Reilly Network, the Linux Documentation Project, and occasionally Linux.Com.
Return to the Linux DevCenter.
2007-11-01 04:26:06 IRAQ [View]
Squid access pages direct from origin no caching is done
2007-03-31 13:04:27 moodylicious [View]
- Trackback from http://furrymuck.q2u.net/archives/001082.html
try try try ...
2003-12-10 05:19:14 [View]
what about of transparent proxy?
2003-09-02 21:04:28 anonymous2 [View]
2003-08-25 21:09:26 anonymous2 [View]
2002-12-04 19:52:20 anonymous2 [View]
How to merge different lease line connection to a single bandwidth on linux
2003-01-28 15:58:41 Jennifer Vesperman | [View]
2001-08-23 15:41:37 sharumpe [View]
Minimum-effort emergency bypass options?
2001-11-18 18:15:52 Jennifer Vesperman | [View]
Followup: Minimum-effort emergency bypass options?
2001-08-31 09:57:22 sharumpe [View]
Minimum-effort emergency bypass options?
2001-08-23 17:50:25 turpie [View]