Vortexmind: free your mind Tu ne cede malis sed contra audentior ito

Apache .htaccess tweaking tutorial

In this tutorial we are going to improve our website by tweaking out the .htaccess file. Why I wrote this article? Because on the net I have found many articles about this little beast, but every one of them dealt with a specific issue and not look at the overall usage of these files, or they are just too big when you need to do a thing in little time. So I'm trying to collect all the useful bits of data in a monolithic but slim tutorial, which will be updated as I collect more information. But first, let's see what .htaccess file is...

Here we have the definitions from Wikipedia:

.htaccess (Hypertext Access) is the default name of Apache's directory-level configuration file. It provides the ability to customize configuration directives defined in the main configuration file. The configuration directives need to be in .htaccess context and the user needs appropriate permissions.

Let's now deal with most common issues!

Tweaks Index

(Last updated 28th Feb 2006)

1) Folders Access Control

You may want to totally disable access in one folder (for example, you have a directory with programming libraries that are included in your main files: in this case only the main files will access these trought the filesystem, but no one from the web should be able to open it). Well, just create an .htaccess file in that folder and put this in it


#deny all access
deny from all


If you'd like to allow access from one specific IP


#deny all access
deny from all
allow from 10.0.0.1


or from a specific IP range (which you enforce with a bit mask)


allow from 192.168.0.0/24


you can also block a specific file from access


<Files private.html>
Order allow,deny
Deny from all
</Files>


./ Back to Index

2) Folder Listing

If you want to make your folders browsable, then you should add this line in .htaccess file

Options +Indexes +MultiViews +FollowSymlinks

And this one if you have the appropriate module installed on your webserver

<ifmodule mod_autoindex.c>
IndexOptions FancyIndexing
</ifmodule>


You may want to prevent folder listing


IndexIgnore *

./ Back to Index

3) Enable Compression

You can enable PHP's built in data compression to save bandwidth


<ifModule mod_php4.c>
php_value zlib.output_compression 16386
</ifModule>

./ Back to Index

4) Hide your files

To disable access to a particular file you can use a regular expression and the Files directive to deny access to any file beginning with .ht
You can modify it to deny a specific file (like configuration files, robots.txt, log files and whatever you want)

<Files ~ "^\.ht">
Order allow,deny
Deny from all
Satisfy All
</Files>

./ Back to Index

5) Customized HTTP 404 error page

If you'd like to redirect your visitors every time they catch into an HTTP 404 error, use this code:


ErrorDocument 404 /errors/notfound.html


This redirects the user to /errors/notfound.html whenever a 404 error happen. You can of course redefine also other http errors codes (403, 500... and so on). Read below what I've found here!

Tip: Internet Explorer has a lightly-documented “feature” that stops it from serving any custom 404 error page that is less than 512 bytes long. Your visitors will instead be sent to IE’s own 404 page (screenshot), which is generic and suggests they use an MSN search to “look for information on the Internet.” That’s one way to lose visitors! Make sure your custom 404 error page is over this limit — about 10 full lines of text and HTML should be enough.

./ Back to Index

6) Blocking bad referers - No hotlinking

If you want to block some parts of your site from any bad referer:


<IfModule mod_rewrite.c>
RewriteEngine on
RewriteCond %{HTTP_REFERER} example\.com [NC,OR]
RewriteCond %{HTTP_REFERER} otherexample\.com
RewriteRule .* - [F]
</ifModule>


Using rewrite engine, you will deny access to all your site from any visitor incoming from badguy.com or othernastywebsite.com
To prevent bandwidth stealing, you can also block access to particular files (images, zip, avi and so on)


<IfModule mod_rewrite.c>
RewriteEngine on
RewriteCond %{HTTP_REFERER} !^$
RewriteCond %{HTTP_REFERER} !^http://([-a-z0-9]+\.)?example\.com[NC]
RewriteRule .*\.(zip|mp3|avi|wmv|mpg|mpeg)$ http://www.example.com/images/nohotlink.gif [R,NC,L]
</ifModule>

This says: "If the visitor is not coming from mywebsite.net, then redirect all requests for (zip,mp3,avi,wmv,mpg,mpeg) files to a nice image that says "NO HOTLINKING HERE". Got it? You can redirect to a page, or whatever you want, or you can modify the file extension list to include/exclude other files. CAUTION: when you decide to block image hotlinking, remember that you can potentially block ALL traffic outside your domain scope! For example, if you have a feedburner feed you have to modify the rule to let him get the images ... or you feed will look quite nasty!
./ Back to Index

7) Blocking Bad Bots | Fetchers

In some cases you want to block some nasty spiders or site downloaders. Then we have to use mod_rewrite again. Usually bad bots ignore robots.txt directive so you may want to enforce them to a 403 error whenever they try to spider or fetch your website


<IfModule mod_rewrite.c>
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} ^BlackWidow [OR]
RewriteCond %{HTTP_USER_AGENT} ^Bot\ mailto:[email protected] [OR]
RewriteCond %{HTTP_USER_AGENT} ^ChinaClaw [OR]
RewriteCond %{HTTP_USER_AGENT} ^Custo [OR]
RewriteCond %{HTTP_USER_AGENT} ^DISCo [OR]
RewriteCond %{HTTP_USER_AGENT} ^Download\ Demon [OR]
RewriteCond %{HTTP_USER_AGENT} ^eCatch [OR]
RewriteCond %{HTTP_USER_AGENT} ^EirGrabber [OR]
RewriteCond %{HTTP_USER_AGENT} ^EmailSiphon [OR]
RewriteCond %{HTTP_USER_AGENT} ^EmailWolf [OR]
RewriteCond %{HTTP_USER_AGENT} ^Express\ WebPictures [OR]
RewriteCond %{HTTP_USER_AGENT} ^ExtractorPro [OR]
RewriteCond %{HTTP_USER_AGENT} ^EyeNetIE [OR]
RewriteCond %{HTTP_USER_AGENT} ^FlashGet [OR]
RewriteCond %{HTTP_USER_AGENT} ^GetRight [OR]
RewriteCond %{HTTP_USER_AGENT} ^GetWeb! [OR]
RewriteCond %{HTTP_USER_AGENT} ^Go!Zilla [OR]
RewriteCond %{HTTP_USER_AGENT} ^Go-Ahead-Got-It [OR]
RewriteCond %{HTTP_USER_AGENT} ^GrabNet [OR]
RewriteCond %{HTTP_USER_AGENT} ^Grafula [OR]
RewriteCond %{HTTP_USER_AGENT} ^HMView [OR]
RewriteCond %{HTTP_USER_AGENT} HTTrack [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Image\ Stripper [OR]
RewriteCond %{HTTP_USER_AGENT} ^Image\ Sucker [OR]
RewriteCond %{HTTP_USER_AGENT} Indy\ Library [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^InterGET [OR]
RewriteCond %{HTTP_USER_AGENT} ^Internet\ Ninja [OR]
RewriteCond %{HTTP_USER_AGENT} ^JetCar [OR]
RewriteCond %{HTTP_USER_AGENT} ^JOC\ Web\ Spider [OR]
RewriteCond %{HTTP_USER_AGENT} ^larbin [OR]
RewriteCond %{HTTP_USER_AGENT} ^LeechFTP [OR]
RewriteCond %{HTTP_USER_AGENT} ^Mass\ Downloader [OR]
RewriteCond %{HTTP_USER_AGENT} ^MIDown\ tool [OR]
RewriteCond %{HTTP_USER_AGENT} ^Mister\ PiX [OR]
RewriteCond %{HTTP_USER_AGENT} ^Navroad [OR]
RewriteCond %{HTTP_USER_AGENT} ^NearSite [OR]
RewriteCond %{HTTP_USER_AGENT} ^NetAnts [OR]
RewriteCond %{HTTP_USER_AGENT} ^NetSpider [OR]
RewriteCond %{HTTP_USER_AGENT} ^Net\ Vampire [OR]
RewriteCond %{HTTP_USER_AGENT} ^NetZIP [OR]
RewriteCond %{HTTP_USER_AGENT} ^Octopus [OR]
RewriteCond %{HTTP_USER_AGENT} ^Offline\ Explorer [OR]
RewriteCond %{HTTP_USER_AGENT} ^Offline\ Navigator [OR]
RewriteCond %{HTTP_USER_AGENT} ^PageGrabber [OR]
RewriteCond %{HTTP_USER_AGENT} ^Papa\ Foto [OR]
RewriteCond %{HTTP_USER_AGENT} ^pavuk [OR]
RewriteCond %{HTTP_USER_AGENT} ^pcBrowser [OR]
RewriteCond %{HTTP_USER_AGENT} ^RealDownload [OR]
RewriteCond %{HTTP_USER_AGENT} ^ReGet [OR]
RewriteCond %{HTTP_USER_AGENT} ^SiteSnagger [OR]
RewriteCond %{HTTP_USER_AGENT} ^SmartDownload [OR]
RewriteCond %{HTTP_USER_AGENT} ^SuperBot [OR]
RewriteCond %{HTTP_USER_AGENT} ^SuperHTTP [OR]
RewriteCond %{HTTP_USER_AGENT} ^Surfbot [OR]
RewriteCond %{HTTP_USER_AGENT} ^tAkeOut [OR]
RewriteCond %{HTTP_USER_AGENT} ^Teleport\ Pro [OR]
RewriteCond %{HTTP_USER_AGENT} ^VoidEYE [OR]
RewriteCond %{HTTP_USER_AGENT} ^Web\ Image\ Collector [OR]
RewriteCond %{HTTP_USER_AGENT} ^Web\ Sucker [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebAuto [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebCopier [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebFetch [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebGo\ IS [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebLeacher [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebReaper [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebSauger [OR]
RewriteCond %{HTTP_USER_AGENT} ^Website\ eXtractor [OR]
RewriteCond %{HTTP_USER_AGENT} ^Website\ Quester [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebStripper [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebWhacker [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebZIP [OR]
RewriteCond %{HTTP_USER_AGENT} ^Wget [OR]
RewriteCond %{HTTP_USER_AGENT} ^Widow [OR]
RewriteCond %{HTTP_USER_AGENT} ^WWWOFFLE [OR]
RewriteCond %{HTTP_USER_AGENT} ^Xaldon\ WebSpider [OR]
RewriteCond %{HTTP_USER_AGENT} ^Zeus
RewriteRule .* - [F]
</ifModule>

(List taken from here)
./ Back to Index

8) Do not show 'www'

To do this, you can usea simple rewrite rule


<IfModule mod_rewrite.c>
Options +FollowSymlinks
RewriteEngine on
RewriteCond %{http_host} ^www\.example\.com[nc]
RewriteRule ^(.*)$ http://example.com/$1 [r=301,nc]
</IfModule>


Why removing www? You can read it here.

./ Back to Index

9) Hide scripting language extension

You can improve your security by changing script extensions so your visitors don't know what scripting language you are using:


# Make PHP code look like unknown types
AddType application/x-httpd-php .133t


This way the .133t files will be parsed as PHP files. You must rename your files with the new extension.
./ Back to Index

10) Various Tips & Tricks

• Keep .htaccess small: the file is processed by the webserver at EACH request (performance issues)
• Keep your .htaccess organized. Use comments (# lines) and keep it logically consistent. Is very difficult to understand a untidy .htaccess file once it grows in size
• When using URL rewriting rules, add the flag [L] to the rules that redirects the users to a last page (like no hotlinking rules and so on). You will tell the server to not process any more the rules (performance issues)
• Beware of inheritance: root level .htaccess files are applied also in folders, and any htaccess rule in the folder can override the root rules

./ Back to Index

11) Password Protection with htpasswd

This is useful if you want to add password protection to some pages/folders

• Create a .htpasswd file in the folder you want to protect
• The file will contain login data in the form username:password. Username is plain text. Password should be encrypted or it won't work! Use this tool to get your string to add
• If you create the file on your local pc, be sure to upload it on the webserver in ASCII mode
• Now you can modify your .htaccess file. The authentication will apply to the folder where you place it and its subfolders:
  
  AuthUserFile /home/pathto/.htpasswd
  AuthType Basic
  AuthName "My Secret Folder"

  <LIMIT GET POST>
require valid-user
</LIMIT>

  You can protect a single file by placing this into a <Files> directive.

• Be sure to protect your .htaccess file from viewing using the 1) tip

./ Back to Index

12) Enabling SSI

Use this instructions to enable SSI parsing


AddType text/html .html
AddType text/html .shtml
AddHandler server-parsed .html
AddHandler server-parsed .shtml


./ Back to Index

13) Changing default page

You can use these instructions to change default page (order is important!)


DirectoryIndex home.html index.htm index.html index.php


./ Back to Index

14) Avoid 500 Error

By passing the charset you avoid the 500 error display

AddDefaultCharset utf-8


./ Back to Index

15) CheckSpelling directive

This directive can be useful to auto-correct simple spelling errors in the URL

<IfModule mod_speling.c>
CheckSpelling On
</IfModule>


./ Back to Index

16) Add MD5 Digest

If you aren't worried about performance issues, you can add a MD5 hash calculation to attach a MIC (Message Integrity Check) on each request. This is useful to check the integrity of the message.

ContentDigest On


./ Back to Index

A) Sources

• http://corz.org/serv/tricks/htaccess.php
• http://underscorebleach.net/jotsheet/2004/06/htaccess-prevent-hotlinking
• http://brainstormsandraves.com/archives/2005/10/09/htaccess/
• http://www.usphp.com/security.hiding.html

./ Back to Index

B) Tools

• .htaccess password generator

./ Back to Index