Squid Proxy CacheServer Installation and Configuration

 Squid is a proxy caching server. If you are Linux sysadmin, you can use squid to control internet access at your work environment.
This beginners guide will give a jump-start on how to setup squid on Linux to restrict internet access in an network.

Install Squid
You should install the following three squid related packages on your system.
squid
squid-common
squid-langpack
On Debian and Ubuntu, use aptitude to install squid as shown below. On CentOS, use yum to install the squid package.
$ yum install squid
Check Configuration and Startup scripts
Apart from installing the squid related packages, it also creates the /etc/squid/squid.conf and /etc/init.d/squid startup script.
By default Squid runs on 3128 port. You can verify this from the squid.conf file. You can also set the visible_hostname parameter in your squid.conf, which will be used in error_log. If you don’t define, squid gets the hostname value using gethostname() function.
# vim /etc/squid/squid.conf
visible_hostname ubuntuserver
httpd_port 3128
Note: The http port number (3128) specified in the squid.conf should be entered in the proxy setting section in the client browser. If squid is built with SSL, you can use https_port option inside squid.conf to define https squid.
Start Squid and View Logs
Start the Squid proxy caching server as shown below.
# service squid start
squid start/running, process 11743
Squid maintains three log files (access.log, cache.log and store.log) under /var/log/squid directory.
From the /var/log/squid/access.log, you can view who accessed which website at what time. Following is the format of the squid access.log record.
time elapsed remotehost code/status bytes method URL rfc931     peerstatus/peerhost
To disable logging in squid, update the squid.conf with the following information.
# to disable access.log
cache_access_log /dev/null

# to disable store.log
cache_store_log none

# to disable cache.log
cache_log /dev/null
Squid Usage 1: Restrict Access to Specific Websites
This is how you can restrict folks from browsing certain website when they are connected to your network using your proxy server.
Create a file called restricted_sites and list all sites that you would want to restrict the access.
# vim /etc/squid/restricted_sites
www.youtube.com
mail.yahoo.com
www.hotmail.com
www.gmail.com
Modify the squid.conf to add the following.
# vim /etc/squid/squid.conf
acl RestrictedSites  dstdomain "/etc/squid/restricted_sites"
http_access deny RestrictedSites

Squid Usage 2: Allow Access to Websites Only During Specific Time
Some organization might want to allow employees to surf or download from the internet only during specific timeperiods.
The squid.conf configuration shown below will allow internet access for employees only between 9:00AM and 18:00 during weekdays.
# vim /etc/squid/squid.conf
acl official_hours time M T W H F 09:00-18:00
http_access deny all
http_access allow official_hours
Squid Usage 3 : Restrict Access to Particular Network
Instead of restricting specific sites, you can also provide access only to certain network and block everything else. The example below, allows access only to the 192.168.10.* internal network.
# vim /etc/squid/squid.conf
acl branch_offices src 192.168.10.0/24
http_access deny all
http_access allow branch_offices

Squid Usage 4 : Use Regular Expression to Match URLs
You can also use regular expression to allow or deny websites.
First create a blocked_sites files with a list of keywords.
# cat /etc/squid/blocked_sites
soccer
movie
www.example.com
Modify the squid.conf to block any sites that has any of these keywords in their url.
# vim /etc/squid/squid.conf
acl blocked_sites url_regex -i "/etc/squid/blocked_sites"
http_access deny blocked_sites
http_access allow all
In the above example, -i option is used for ignoring case for matching. So, while accessing the websites, squid will try to match the url with any of the pattern mentioned in the above blocked_sites file and denies the access when it matches.
SARG – Squid Analysis Report Generator
Download and install SARG to generate squid usage reports.
Use the sarg-reports command to generate reports as shown below.
# to generate the report for today
sarg-report today

# on daily basis
sarg-report daily

# on weekly basis
sarg-report weekly

# on monthly basis
sarg-report monthly

Add the sarg-report to the crontab.
The reports generated by sarg are stored under /var/www/squid-reports. These are html reports can you can view from a browser.

$ ls /var/www/squid-reports
Daily  index.hyml

$ ls /var/www/squid-reports/Daily
2010Sept1-2010Sept2  images  index.html

Vi & Vim Commands

Command mode ESC

                dd       delete
                u        undelete
                y        yank (copy to buffer)
                p/P      p before cursor/P after cursor

                Ctl-g    show current line number
                shft-G   end of file
              n shft-G   move to line n

               /stuff/   search
                  n   repeat in same direction
                  N   repeat in opposite direction
                  /return  repeat seach forward
                  ?return  repeat seach backward

               "dyy  Yank current line to buffer d
               "a7yy Yank next 7 lines to buffer a
                    or
               :1,7ya a  Yank [ya] lines 1,7 to buffer a
               :1,7ya b  Yank [ya] lines 1,7 to buffer b

               :5 pu b   Put [pu] buffer b after line 5

               "dP   Put the content of buffer d before cursor
               "ap   Put the contents of buffer a after cursor

               :1,4 w! file2  Write lines 1,4 to file2
               :1,3

               :set nu     Display line numbers
               :set nonum  Turns off display

               :set ic     Ignore Case

               :e  Edit a file in a new buffer

               :g//p   Print matching regular expression

            vim
               :split
               :split
               :sp
               :split new

                   ctl-w   To move between windows
                   ctl-w+
                   ctl-w-  To change size
                   ctl+wv  Split windows vertically
                   ctl-wq  Close window

               :only       To view only 1 window

            vim dictionary - put the following command in ~/.vimrc

                   set dictionary+=/usr/share/dict/words
                   set thesaurus+=/usr/share/dict/words
             
               Now, after you type a word  and to
               go back in the listing

                   butter

           Scripting - you can script vi commands using ex. For example
               suppose you want to replace all occurrences of "one" with "two", then
               exit the file if there are changes. You would put the following in a file call script

               Contents of script

                   %s/one/two/g|x

               If you want to run this on all files with the patten "example*"

                   for i in $(ls example*); do ex - $i

Diffrence between Hardlink & Softlink

Hard link: Hard link refers to "The specific location of physical data".
Hard Link is a mirror copy of the original file.
Hard links share the same inode.
Any changes made to the original or Hard linked file will reflect the other.
Even if you delete any one of the files, nothing will happen to the other hard links.
But soft link which points to deleted hard link become a dangling soft link.
You can't link a directory even within the same file system.
Hard links can't cross file systems.
Soft link( also called symbolic link): Soft link refers to "A symbolic path indicating the abstract location of another file".
Soft Link is a symbolic link to the original file.(more like windows shortcuts)
Soft Links will have a different Inode value.
Any changes made to the soft link will reflect the original file and its hard links.
A soft link points to the original file. If you delete the original file, the soft link fails. It would become dangling symbolic link.
If you delete the soft link, nothing will happen.
You can link a directory using soft link on same file system and also on other file system.
Soft links can cross file systems

IPTables on Redaht / Cent O/s

Since kernel version 2.4, there is a built in system for package filtering known as Netfilter. To use Netfilter, during kernel compiling CONFIG_NETFILTER must be included. Also ip_forward must be enabled:
echo 1 > /proc/sys/net/ipv4/ip_forward
Package filtering works on Internet layer of TCP/IP protocol. Filtering rules can be defined based on a transport layer header(port number) and connection layer (source IP address). During filtering, package content is not being checked.
Netfilter filtering chains work in kernel mode. In user mode works special tool called – iptables, which requires root user privilegies and it's used to configure:
- filter chains,
- NAT tables,
- mangle tables.
Netfilter uses three filters, INPUT, OUTPUT, FORWARD, realised in form of chains. Each chain contains a set of rules that filters packages. If some package sattisfies a rule, an action gets to be applied, like accepting or rejecting package.
Iptables commands:
-A (Add rule to the end of chain).
-D (Delete rule from chain).
-R (Replace rule in chain).
-I (Add numeric rule in chain).
-L (List rules).
-F (Delete all rules from chain).
Deleting the chains:
# iptables -F INPUT
# iptables -F OUTPUT
# iptables -F FORWARD
Here are some basic examples of iptables usage.
1. Blocking IP with iptables:
# iptables -A INPUT -s xxx.xxx.xxx.xxx -j DROP
# iptables -A OUTPUT -d xxx.xxx.xxx.xxx -j DROP
* instead xxx.xxx.xxx.xxx www.abc.com can be added.
2. Opening ports:
First thing you need to do is check if ports are already opened. It's done using nmap, free program, and it's distributed in most distros.
# nmap -sT xxx.xxx.xxx.xxx
PORT STATE SERVICE
22/tcp open ssh
80/tcp open http
443/tcp open https
Nmap run completed -- 1 IP address scanned in 0.941 seconds.
As we see, on this list port 25 is not opened. Let's open port 25 for SMTP traffic.
# iptables -A INPUT -p tcp --dport 25 -j ACCEPT
3. Enabling other hosts to ping:
# iptables -A INPUT -p icmp -s xxx.xxx.xxx.xxx/xx –icmp-type echo-request -j ACCEPT
# iptables -A INPUT -p icmp -d xxx.xxx.xxx.xxx/xx –icmp-type echo-reply -j ACCEPT
4. Restricting access by time of the day:
# iptables -A INPUT -p tcp -s 0/0 --sport 513:65535 -d xxx.xxx.xxx.xxx --dport 22 -m state
--state NEW,ESTABLISHED -m time --timestart 09:00 --timestop 18:00
--days Mon,Tue,Wed,Thu,Fri -j ACCEPT 

5. Keeping logs about rejected packages:

# iptables -A OUTPUT -j LOG
# iptables -A OUTPUT -j DROP
# iptables -A INPUT -j LOG
# iptables -A INPUT -j DROP
# iptables -A FORWARD -j LOG
# iptables -A FORWARD -j DROP

Top Command Line Utility

When you need to see the running processes on your Linux in real time, you have top as your tool for that.
top also displays other info besides the running processes, like free memory both physical and swap
Usage
top [options]
Options
-d ss.tt
Delay -- Specifies the seconds and tenths of seconds of delay between the updates of the info showed on the screen, being the default 3 seconds
-i
Starts top with the last remembered 'i' state reversed. When this toggle is Off, tasks that are idled or zombied will not be displayed.
-n n
Specifies the maximum number of iterations, or frames, top should produce before ending.
-p n
Monitor only processes with specified process IDs. This option can be given up to 20 times, or you can provide a comma delimited list with up to 20 pids. Co-mingling both approaches is permitted. This is a command-line option only. And should you wish to return to normal operation, it is not necessary to quit and and restart top -- just issue the '=' interactive command.
-s
- Secure - Runs top in secure mode, restricting the commands you can use while top is running even for root
-S (Sum)
Starts top with the last remembered 'S' state reversed. When 'Cumulative mode' is On, each process is listed with the cpu time that it and its dead children have used. See the 'S' interactive command for additional information regarding this mode.
Description of the fields
a: PID -- Process Id
The task's unique process ID, which periodically wraps, though never restarting at zero.
b: PPID -- Parent Process Pid
The process ID of a task's parent.
c: RUSER -- Real User Name
The real user name of the task's owner.
d: UID -- User Id
The effective user ID of the task's owner.
e: USER -- User Name
The effective user name of the task's owner.
f: GROUP -- Group Name
The effective group name of the task's owner.
g: TTY -- Controlling Tty
The name of the controlling terminal. This is usually the device (serial port, pty, etc.) from which the process was started, and which it uses for input or output. However, a task need not be associated with a terminal, in which case you'll see '?' displayed.
h: PR -- Priority
The priority of the task.
i: NI -- Nice value
The nice value of the task. A negative nice value means higher priority, whereas a positive nice value means lower priority. Zero in this field simply means priority will not be adjusted in determining a task's dispatchability./dd>
j: P -- Last used CPU (SMP)
A number representing the last used processor. In a true SMP environment this will likely change frequently since the kernel intentionally uses weak affinity. Also, the very act of running top may break this weak affinity and cause more processes to change CPUs more often (because of the extra demand for cpu time).
k: %CPU -- CPU usage
The task's share of the elapsed CPU time since the last screen update, expressed as a percentage of total CPU time. In a true SMP environment, if 'Irix mode' is Off, top will operate in 'Solaris mode' where a task's cpu usage will be divided by the total number of CPUs. You toggle 'Irix/Solaris' modes with the 'I' interactive command.
l: TIME -- CPU Time
Total CPU time the task has used since it started. When 'Cumulative mode' is On, each process is listed with the cpu time that it and its dead children has used. You toggle 'Cumulative mode' with 'S', which is a command-line option and an interactive command. See the 'S' interactive command for additional information regarding this mode.
m: TIME+ -- CPU Time, hundredths
The same as 'TIME', but reflecting more granularity through hundredths of a second.
n: %MEM -- Memory usage (RES)
A task's currently used share of available physical memory.
o: VIRT -- Virtual Image (kb)
The total amount of virtual memory used by the task. It includes all code, data and shared libraries plus pages that have been swapped out.
VIRT = SWAP + RES.
p: SWAP -- Swapped size (kb)
The swapped out portion of a task's total virtual memory image.
q: RES -- Resident size (kb)
The non-swapped physical memory a task has used.
RES = CODE + DATA.
r: CODE -- Code size (kb)
The amount of physical memory devoted to executable code, also known as the 'text resident set' size or TRS.
s: DATA -- Data+Stack size (kb)
The amount of physical memory devoted to other than executable code, also known as the 'data resident set' size or DRS.
t: SHR -- Shared Mem size (kb)
The amount of shared memory used by a task. It simply reflects memory that could be potentially shared with other processes.
u: nFLT -- Page Fault count
The number of major page faults that have occurred for a task. A page fault occurs when a process attempts to read from or write to a virtual page that is not currently present in its address space. A major page fault is when disk access is involved in making that page available.
v: nDRT -- Dirty Pages count
The number of pages that have been modified since they were last written to disk. Dirty pages must be written to disk before the corresponding physical memory location can be used for some other virtual page.
w: S -- Process Status
The status of the task which can be one of:
'D' = uninterruptible sleep
'R' = running
'S' = sleeping
'T' = traced or stopped
'Z' = zombie
Tasks shown as running should be more properly thought of as 'ready to run' -- their task_struct is simply represented on the Linux run-queue. Even without a true SMP machine, you may see numerous tasks in this state depending on top's delay interval and nice value.
x: Command -- Command line or Program name
Display the command line used to start a task or the name of the associated program. You toggle between command line and name with 'c', which is both a command-line option and an interactive command.
When you've chosen to display command lines, processes without a command line (like kernel threads) will be shown with only the program name in parentheses, as in this example:
( mdrecoveryd )
Either form of display is subject to potential truncation if it's too long to fit in this field's current width. That width depends upon other fields selected, their order and the current screen width.
Note: The 'Command' field/column is unique, in that it is not fixed-width. When displayed, this column will be allocated all remaining screen width (up to the maximum 512 characters) to provide for the potential growth of program names into command lines.
y: WCHAN -- Sleeping in Function
Depending on the availability of the kernel link map ('System.map'), this field will show the name or the address of the kernel function in which the task is currently sleeping. Running tasks will display a dash ('-') in this column.
Note: By displaying this field, top's own working set will be
increased by over 700Kb. Your only means of reducing that overhead
will be to stop and restart top.
z: Flags -- Task Flags
This column represents the task's current scheduling flags which are expressed in hexadecimal notation and with zeros suppressed. These flags are officially documented in . Less formal documentation can also be found on the 'Fields select' and 'Order fields' screens.
Interactive commands
While top is running you may issue some options that will interact immediately with top these options are:
h
Help, displays a summary of command that will modify the behavior of top
k
Kills a process, you will be able to kill only your own processes, unless you are running top as root
n
Once this command is entered top will ask you how many lines you want on your screen, if you enter 0 top will display as much as it can
q
Exits top
r
Change the priority of a process, as well as with k you will only be able to act on your own processes unless you are root
W
Writes the current configuration to your personal configuration file, which is $HOME/.toprc

Linux Directory Structure

/bin - This directory contains most of your non-privileged system commands such as ls, mkdir, rm, etc.

/boot - Contains the systems boot image, bootloader, and the kernel

/dev - Symbolic links to system devices such as optical and removable drives

/etc - Contains all system configuration files and most configurations for installed packages

/home - Contains a directory for each user and contains profile information

/lib - Contains dynamic libraries and modules for the Linux system and installed packages

/media - Contains mount points for optical drives and removable media

/mnt - Used as a location for mounted drives and shares

/opt - Contains user installed packages and custom software not handled by the system or package manager

/proc - An interface between the kernel and the system, useful for diagnostics and system information

/root - The root superuser's home directory

/sbin - Contains privileged commands that are usually run as superuser (root/sudo)

/sys - An interface between the kernel and the system, used for modifying system settings

/tmp - A location for temporary files such as sessions on a web server

/usr - Contains most installed packages that are not part of the system, user installed programs

/usr/bin - Contains commands related to user installed packages in /usr

/usr/sbin - Contains privileged commands related to user installed packages in /usr

/var - Contains files that change often or accessed frequently

/var/log - Contains all system logs and most logs generated by installed packages

Linux LVM

1. create LVM partition:

fdisk /dev/[disk]

change partition type from LINUX to LVM

2. initialization LVM partition:

pvcreate /dev/[disk_partiton1] /dev/[disk_partition2]

check commands:

pvs
pvdisplay

3. create LVM group:

vgcreate [group_name] /dev/[disk_partition1] /dev/[disk_partition2]

check commands:

vgs
vgdisplay

4. create logical volumes which is attached to group:

lvcreate -L[size]M -n [vol_name] [group_name]

check commands:

lvs
lvdisplay

5. format volumes:

mkfs.ext3 [LV_name_from_lvdisplay]

---

EXTEND
change size volume:

lvextend -L +[size]M [LV_name_from_lvdisplay]

change size partition:

resize2fs [LV_name_from_lvdisplay]

REDUCE

fsck -f [LV_name_from_lvdisplay]

change size partition:

resize2fs [LV_name_from_lvdisplay] [size]G

change size volume:
lvreduce -L -[size]G [LV_name_from_lvdisplay]