AWK Onliners

Simple AWK programs enclosed in single quotes can be typed and executed right at the Unix prompt. For example, the program

awk 'BEGIN { FS = ":" } { print $1 | "sort" }' /etc/passwd
This program prints a sorted list of the login names of all users.

If an input file or output file are not specified, awk will expect input from stdin or output to stdout.

AWK is very flexible about matching patterns. Patterns can be

* regular expressions enclosed by slashes, e.g.: /regular expression/
* relational expressions, e.g.: $3!=$4
* pattern-matching expressions, e.g.: $1 !~ /string/
* or any combination of these, e.g.:
(substr($0,5,2)=="xx" && $3 ~ /nasty/ ) || /^The/ || /$mean/ || $4>2

(This last example selects lines where the two characters starting in fifth column are xx and the third field matches nasty, plus lines beginning with The, plus lines ending with mean, plus lines in which the fourth field is greater than two.)

AWK procedures are enclosed in {curly brackets}. Procedures can

(1) assign variables or arrays. For example:

BEGIN {FS = ","}
(resets the field-separator character to comma before reading input)

/string/ { count["string"]++ }
(creates array count indexing the occurrences of string)

split (substr($0,4,12),N,",")
(splits the substring at commas into arrays N[1], N[2], ...

newvar = $4*sqrt($5)

AWK operators by order of (decreasing) precedence:

Field reference: $
Increment or decrement: ++ --
Exponentiate: ^
Multiply, divide, modulus: * / %
Add, subtract: + -
Concatenation: (blank space)
Relational: < <= > >= != ==
Match regular expression: ~ !~
Logical: && ||
C-style assignment: = += -= *= /= %= ^=

AWK arithmetic functions: exp, int, log and sqrt

AWK string functions:

index(string,substring)
returns position of first occurrence of substring in string, or 0

length[(argument)]
returns length of argument, if specified, or $0.

split(string,array[,f])
splits string by separator character f (or blanks) into array[1], array[2], ...

substr(string,s[,l])
extracts from string starting at position number s with length l (or the rest of string).

(2) print output. Output can be unformatted (print) or formatted (printf):

{ print $2 $1 ":" $4/$3 }

prints the first two fields in reverse order, a colon and the integer ratio of the next fields.)

{ printf "Clump %d: \t%4.1f acres \t%s. \n", $2, $4*640, $6 }

prints formatted output: %d specifies a decimal number format for the clump ID; %n.mf specifies a floating-point number format for the acreage, converted from square miles; %s specifies a string format. \t is the tab character; \n is the newline character. The input line

28 4 12 0.072 vegcov forest spearfish

would be printed as the tab-aligned output line

Clump 4: 46.1 acres forest

(3) perform flow-control (you won't need these for this class):

Do-loops:


for ( [initial expression];
[test expression];
[increment counter expression] )
{ commands }

example: for (i = 1; i <= 20; i++) does 20 iterations

If-Then-Else:


if (condition)
{ commands1 }
[ else
{ commands2 } ]

does commands1 if condition is true; commands2 (or nothing) if false; condition is any expression with relational or pattern-match operators.

Other flow-control commands:

break exits from a for loop.
continue begins next iteration of a for loop.
exit terminates remaining procedures; terminates input; executes END procedure, if any.

The following command runs a simple awk program that searches the input file /etc/passwd for the character string `foo' (a grouping of characters is usually called a string; the term string is based on similar usage in English, such as “a string of pearls,” or “a string of cars in a train”):

awk '/foo/ { print $0 }' /etc/passwd

When lines containing `foo' are found, they are printed because `print $0' means print the current line. (Just `print' by itself means the same thing, so we could have written that instead.)

You will notice that slashes (`/') surround the string `foo' in the awk program. The slashes indicate that `foo' is the pattern to search for. This type of pattern is called a regular expression, which is covered in more detail later (see Regexp). The pattern is allowed to match parts of words. There are single quotes around the awk program so that the shell won't interpret any of it as special shell characters.

Here is what this program prints:

$ awk '/foo/ { print $0 }' BBS-list
-| fooey 555-1234 2400/1200/300 B
-| foot 555-6699 1200/300 B
-| macfoo 555-6480 1200/300 A
-| sabafoo 555-2127 1200/300 C

In an awk rule, either the pattern or the action can be omitted, but not both. If the pattern is omitted, then the action is performed for every input line. If the action is omitted, the default action is to print all lines that match the pattern.

Thus, we could leave out the action (the print statement and the curly braces) in the previous example and the result would be the same: all lines matching the pattern `foo' are printed. By comparison, omitting the print statement but retaining the curly braces makes an empty action that does nothing (i.e., no lines are printed).

Many practical awk programs are just a line or two. Following is a collection of useful, short programs to get you started. Some of these programs contain constructs that haven't been covered yet. (The description of the program will give you a good idea of what is going on, but please read the rest of the Web page to become an awk expert!) Most of the examples use a data file named data. This is just a placeholder; if you use these programs yourself, substitute your own file names for data. For future reference, note that there is often more than one way to do things in awk. At some point, you may want to look back at these examples and see if you can come up with different ways to do the same things shown here:

* Print the length of the longest input line:

awk '{ if (length($0) > max) max = length($0) }
END { print max }' data

* Print every line that is longer than 80 characters:

awk 'length($0) > 80' data

The sole rule has a relational expression as its pattern and it has no action—so the default action, printing the record, is used.

* Print the length of the longest line in data:

expand data | awk '{ if (x < length()) x = length() }
END { print "maximum line length is " x }'

The input is processed by the expand utility to change tabs into spaces, so the widths compared are actually the right-margin columns.
* Print every line that has at least one field:

awk 'NF > 0' data


This is an easy way to delete blank lines from a file (or rather, to create a new file similar to the old file but from which the blank lines have been removed).
* Print seven random numbers from 0 to 100, inclusive:

awk 'BEGIN { for (i = 1; i <= 7; i++)
print int(101 * rand()) }'


* Print the total number of bytes used by files:

ls -l files | awk '{ x += $5 }
END { print "total bytes: " x }'


* Print the total number of kilobytes used by files:

ls -l files | awk '{ x += $5 }
END { print "total K-bytes: " (x + 1023)/1024 }'

* Print a sorted list of the login names of all users:

awk -F: '{ print $1 }' /etc/passwd | sort

* Count the lines in a file:

awk 'END { print NR }' data

* Print the even-numbered lines in the data file:

awk 'NR % 2 == 0' data

If you use the expression `NR % 2 == 1' instead, the program would print the odd-numbered lines.

Other AWK Onliners

awk '{ if (NF > max) max = NF }
END { print max }'
This program prints the maximum number of fields on any input line.

awk 'length($0) > 80'
This program prints every line longer than 80 characters. The sole rule has a relational expression as its pattern, and has no action (so the default action, printing the record, is used).

awk 'NF > 0'
This program prints every line that has at least one field. This is an easy way to delete blank lines from a file (or rather, to create a new file similar to the old file but from which the blank lines have been deleted).

awk '{ if (NF > 0) print }'
This program also prints every line that has at least one field. Here we allow the rule to match every line, then decide in the action whether to print.

awk 'BEGIN { for (i = 1; i <= 7; i++)
print int(101 * rand()) }'
This program prints 7 random numbers from 0 to 100, inclusive.

ls -l files | awk '{ x += $4 } ; END { print "total bytes: " x }'
This program prints the total number of bytes used by files.

expand file | awk '{ if (x < length()) x = length() }
END { print "maximum line length is " x }'
This program prints the maximum line length of file. The input is piped through the expand program to change tabs into spaces, so the widths compared are actually the right-margin columns.

awk 'BEGIN { FS = ":" }
{ print $1 | "sort" }' /etc/passwd
This program prints a sorted list of the login names of all users.

awk '{ nlines++ }
END { print nlines }'
This programs counts lines in a file.

awk 'END { print NR }'
This program also counts lines in a file, but lets awk do the work.

awk '{ print NR, $0 }'
This program adds line numbers to all its input files, similar to `cat -n'.

HANDY ONE-LINERS FOR AWK

USAGE:
awk '/pattern/ {print "$1"}' # standard Unix shells

FILE SPACING:

# double space a file
awk '1;{print ""}'
awk 'BEGIN{ORS="\n\n"};1'

# double space a file which already has blank lines in it. Output file
# should contain no more than one blank line between lines of text.
# NOTE: On Unix systems, DOS lines which have only CRLF (\r\n) are
# often treated as non-blank, and thus 'NF' alone will return TRUE.
awk 'NF{print $0 "\n"}'

# triple space a file
awk '1;{print "\n"}'

NUMBERING AND CALCULATIONS:

# precede each line by its line number FOR THAT FILE (left alignment).
# Using a tab (\t) instead of space will preserve margins.
awk '{print FNR "\t" $0}' files*

# precede each line by its line number FOR ALL FILES TOGETHER, with tab.
awk '{print NR "\t" $0}' files*

# number each line of a file (number on left, right-aligned)
# Double the percent signs if typing from the DOS command prompt.
awk '{printf("%5d : %s\n", NR,$0)}'

# number each line of file, but only print numbers if line is not blank
# Remember caveats about Unix treatment of \r (mentioned above)
awk 'NF{$0=++a " :" $0};{print}'
awk '{print (NF? ++a " :" :"") $0}'

# count lines (emulates "wc -l")
awk 'END{print NR}'

# print the sums of the fields of every line
awk '{s=0; for (i=1; i<=NF; i++) s=s+$i; print s}'

# add all fields in all lines and print the sum
awk '{for (i=1; i<=NF; i++) s=s+$i}; END{print s}'

# print every line after replacing each field with its absolute value
awk '{for (i=1; i<=NF; i++) if ($i < 0) $i = -$i; print }'
awk '{for (i=1; i<=NF; i++) $i = ($i < 0) ? -$i : $i; print }'

# print the total number of fields ("words") in all lines
awk '{ total = total + NF }; END {print total}' file

# print the total number of lines that contain "Beth"
awk '/Beth/{n++}; END {print n+0}' file

# print the largest first field and the line that contains it
# Intended for finding the longest string in field #1
awk '$1 > max {max=$1; maxline=$0}; END{ print max, maxline}'

# print the number of fields in each line, followed by the line
awk '{ print NF ":" $0 } '

# print the last field of each line
awk '{ print $NF }'

# print the last field of the last line
awk '{ field = $NF }; END{ print field }'

# print every line with more than 4 fields
awk 'NF > 4'

# print every line where the value of the last field is > 4
awk '$NF > 4'


TEXT CONVERSION AND SUBSTITUTION:

# IN UNIX ENVIRONMENT: convert DOS newlines (CR/LF) to Unix format
awk '{sub(/\r$/,"");print}' # assumes EACH line ends with Ctrl-M

# IN UNIX ENVIRONMENT: convert Unix newlines (LF) to DOS format
awk '{sub(/$/,"\r");print}

# IN DOS ENVIRONMENT: convert Unix newlines (LF) to DOS format
awk 1

# IN DOS ENVIRONMENT: convert DOS newlines (CR/LF) to Unix format
# Cannot be done with DOS versions of awk, other than gawk:
gawk -v BINMODE="w" '1' infile >outfile

# Use "tr" instead.
tr -d \r outfile # GNU tr version 1.22 or higher

# delete leading whitespace (spaces, tabs) from front of each line
# aligns all text flush left
awk '{sub(/^[ \t]+/, ""); print}'

# delete trailing whitespace (spaces, tabs) from end of each line
awk '{sub(/[ \t]+$/, "");print}'

# delete BOTH leading and trailing whitespace from each line
awk '{gsub(/^[ \t]+|[ \t]+$/,"");print}'
awk '{$1=$1;print}' # also removes extra space between fields

# insert 5 blank spaces at beginning of each line (make page offset)
awk '{sub(/^/, " ");print}'

# align all text flush right on a 79-column width
awk '{printf "%79s\n", $0}' file*

# center all text on a 79-character width
awk '{l=length();s=int((79-l)/2); printf "%"(s+l)"s\n",$0}' file*

# substitute (find and replace) "foo" with "bar" on each line
awk '{sub(/foo/,"bar");print}' # replaces only 1st instance
gawk '{$0=gensub(/foo/,"bar",4);print}' # replaces only 4th instance
awk '{gsub(/foo/,"bar");print}' # replaces ALL instances in a line

# substitute "foo" with "bar" ONLY for lines which contain "baz"
awk '/baz/{gsub(/foo/, "bar")};{print}'

# substitute "foo" with "bar" EXCEPT for lines which contain "baz"
awk '!/baz/{gsub(/foo/, "bar")};{print}'

# change "scarlet" or "ruby" or "puce" to "red"
awk '{gsub(/scarlet|ruby|puce/, "red"); print}'

# reverse order of lines (emulates "tac")
awk '{a[i++]=$0} END {for (j=i-1; j>=0;) print a[j--] }' file*

# if a line ends with a backslash, append the next line to it
# (fails if there are multiple lines ending with backslash...)
awk '/\\$/ {sub(/\\$/,""); getline t; print $0 t; next}; 1' file*

# print and sort the login names of all users
awk -F ":" '{ print $1 | "sort" }' /etc/passwd

# print the first 2 fields, in opposite order, of every line
awk '{print $2, $1}' file

# switch the first 2 fields of every line
awk '{temp = $1; $1 = $2; $2 = temp}' file

# print every line, deleting the second field of that line
awk '{ $2 = ""; print }'

# print in reverse order the fields of every line
awk '{for (i=NF; i>0; i--) printf("%s ",i);printf ("\n")}' file

# remove duplicate, consecutive lines (emulates "uniq")
awk 'a !~ $0; {a=$0}'

# remove duplicate, nonconsecutive lines
awk '! a[$0]++' # most concise script
awk '!($0 in a) {a[$0];print}' # most efficient script

# concatenate every 5 lines of input, using a comma separator
# between fields
awk 'ORS=%NR%5?",":"\n"' file



SELECTIVE PRINTING OF CERTAIN LINES:

# print first 10 lines of file (emulates behavior of "head")
awk 'NR < 11'

# print first line of file (emulates "head -1")
awk 'NR>1{exit};1'

# print the last 2 lines of a file (emulates "tail -2")
awk '{y=x "\n" $0; x=$0};END{print y}'

# print the last line of a file (emulates "tail -1")
awk 'END{print}'

# print only lines which match regular expression (emulates "grep")
awk '/regex/'

# print only lines which do NOT match regex (emulates "grep -v")
awk '!/regex/'

# print the line immediately before a regex, but not the line
# containing the regex
awk '/regex/{print x};{x=$0}'
awk '/regex/{print (x=="" ? "match on line 1" : x)};{x=$0}'

# print the line immediately after a regex, but not the line
# containing the regex
awk '/regex/{getline;print}'

# grep for AAA and BBB and CCC (in any order)
awk '/AAA/; /BBB/; /CCC/'

# grep for AAA and BBB and CCC (in that order)
awk '/AAA.*BBB.*CCC/'

# print only lines of 65 characters or longer
awk 'length > 64'

# print only lines of less than 65 characters
awk 'length < 64'

# print section of file from regular expression to end of file
awk '/regex/,0'
awk '/regex/,EOF'

# print section of file based on line numbers (lines 8-12, inclusive)
awk 'NR==8,NR==12'

# print line number 52
awk 'NR==52'
awk 'NR==52 {print;exit}' # more efficient on large files

# print section of file between two regular expressions (inclusive)
awk '/Iowa/,/Montana/' # case sensitive


SELECTIVE DELETION OF CERTAIN LINES:

# delete ALL blank lines from a file (same as "grep '.' ")
awk NF
awk '/./'

GETTIN LAST FIELD OF THE STRING

# seperated with / get the last FIELD (useful when when you don't have the fixed position of the field)
awk -F/ '{print $(NF)}'
# Note the -F option this only tobe used when using $(NF) to get the field {FS="/"} will always give an error. Also, the -F option is to used onle if the seperator is other than space or tab. If the seperator is space or tab use following
awk '{print $NF)}'

Perl One Liner

List of one line perl programs to do many common command-line tasks.

# run contents of "my_file" as a program
perl my_file

# run debugger "stand-alone"
perl -d -e 42

# run program, but with warnings
perl -w my_file

# run program under debugger
perl -d my_file

# just check syntax, with warnings
perl -wc my_file

# useful at end of "find foo -print"
perl -nle unlink

# simplest one-liner program
perl -e 'print "hello world!\n"'

# add first and penultimate columns
perl -lane 'print $F[0] + $F[-2]'

# just lines 15 to 17
perl -ne 'print if 15 .. 17' *.pod

# in-place edit of *.c files changing all foo to bar
perl -p -i.bak -e 's/\bfoo\b/bar/g' *.c

# command-line that prints the first 50 lines (cheaply)
perl -pe 'exit if $. > 50' f1 f2 f3 ...

# delete first 10 lines
perl -i.old -ne 'print unless 1 .. 10' foo.txt

# change all the isolated oldvar occurrences to newvar
perl -i.old -pe 's{\boldvar\b}{newvar}g' *.[chy]

# command-line that reverses the whole file by lines
perl -e 'print reverse <>' file1 file2 file3 ....

# find palindromes
perl -lne 'print if $_ eq reverse' /usr/dict/words

# command-line that reverse all the bytes in a file
perl -0777e 'print scalar reverse <>' f1 f2 f3 ...

# command-line that reverses the whole file by paragraphs
perl -00 -e 'print reverse <>' file1 file2 file3 ....

# increment all numbers found in these files
perl i.tiny -pe 's/(\d+)/ 1 + $1 /ge' file1 file2 ....

# command-line that shows each line with its characters backwards
perl -nle 'print scalar reverse $_' file1 file2 file3 ....

# delete all but lines beween START and END
perl -i.old -ne 'print unless /^START$/ .. /^END$/' foo.txt

# binary edit (careful!)
perl -i.bak -pe 's/Mozilla/Slopoke/g' /usr/local/bin/netscape

# look for dup words
perl -0777 -ne 'print "$.: doubled $_\n" while /\b(\w+)\b\s+\b\1\b/gi'

# command-line that prints the last 50 lines (expensively)
perl -e 'lines = <>; print @@lines[ $#lines .. $#lines-50' f1 f2 f3 ...

# command-line that prints the line Title Case
echo "line needed in title case"| perl -p -e 's/(\w+)/\u\L$1/g;'

Output is
Line Needed In Title Case

Shell One Liners

$ for i in *.input; do mv $i ${i/name\.old/name\.new}; done # renames file name.old to name.new - To test things first, insert 'echo' between 'do mv' (above).

$ for i in *.input; do ./application $i; done # runs application in loops on many input files

$ for i in *.input; do fastacmd -d /data/../database_name -i $i > $i.out; done # runs fastacmd in loops on many *.input files and creates *.out files

$ for i in *.pep; do target99 -db /usr/../database_name -seed $i -out $i; done # runs SAM's target99 on many input files

$ for j in 0 1 2 3 4 5 6 7 8 9; do grep -iH *$j.seq; done # searches in > 10,000 files for pattern and prints occurences together with file names.

$ for i in *.pep; do echo -e "$i\n\n17\n33\n\n\n" | ./tmpred $i > $i.out; done # example of how to run an interactive application (tmpred) that asks for file name input/output

$ for i in *.fasta1; do blast2 -p blastp -i $i -j ${i/_*fasta1/_*fasta2} >> my_out_file; done # runs BLAST2 for all *.fasa1/*.fasta2 file pairs in the order specified by file names and writes results into one file. This example uses two variables in a for loop. The content of the second variable gets specified in each loop by a replace function.

$ for i in *.fasta; do for j in *.fasta; do blast2 -p blastp -F F -i $i -j $j >> my_out_file; done; done; # runs BLAST2 in all-against-all mode and writes results into one file; '-F F' turns low-complexity filter off

Often useful, often forgotten, unix commands.

Unix One-Liners

Listed here are a bunch of unix commands.

–> change file date stamp touch –t 199906042020 filename

–> move partitions ufsdump 0f - /dev/rdsk/c0t0s0s0 | (cd /home; ufsrestore xv -)

–> lay down file system with 1% minfree and inode density newfs –m1 –i81920 /dev/rdsk/c0t0d0s0

–> check file system fsck /dev/rdsk/c0t0d0s0

Q: starting sybase login as sybase, run: ./install/RUN_SYBASE

Q: logging in as sybase sa isql -U sa

–> dump a partition and pipe to gzip. Watch > 2GB limit ufsdump 0f - /home | gzip - >/tmp/home.dump.gz

–> rewind offline a tape mt –f /dev/rmt/0 rewoffl

–> only allow 300MB for user /tmp access swap - /tmp tmpfs – yes SIZE=300M

–> verbose interactive restore ufsrestore –ivf /dev/rmt/1

–> remove a printer from a class lpadmin –p level5-line1 –r level5-line

–> truss a command truss –-f -–o /tmp/log.txt

–> [DB] feed a script into sybase isql –Urfe_xfer -Uuser -Ppassword -isqlscript >>blah.txt

–> make a printer class lpadmin –p level5-line1 –c level5-line

–> remove level2-line2 printer from printer class level2-line lpadmin -p level2-line2 -r level2-line

–> add level2-line3 to printer class lpadmin -c level2-line -p level2-line3

–> [DB] how to change your password in isql sp_password password, password-new

–> move a directory tar cf - ./games | (cd /tmp; tar xvBpf - )

–> [DB] run a sybase script, and dump to file $ISQL -i$SCRIPTFILE -U$USER -D$DATABASE -P$PASS_ENC >> $SCRIPTLOGFILE

–> move a directory to another server tar cf - ./games | rsh brucey cd /tmp; tar xvBpf -

–> check for SUID SGID files ncheck -F ufs -s /dev/dsk/c3t0d0s

– remove core files find / -name core –exec rm –f {} ; -o –fstype nfs –prune

–> rebuild man pages catman –w –M man-page-directory or /usr/lib/makewhatis

–> vi command to show special characters : set list

–> adding an account useradd -u 120 –g dls -d /apps/dls –s /bin/ksh -c “comment” -m dls

–> create a mysql database mysqladmin -uroot -ppassword create ebs

–> starting mysql database /etc/rc.d/init.d/mysql.server start /usr/local/bin/safe_mysqld

–> Invoke CPAN module install perl –MCPAN –eshell

–> dump to zip ufsdump 0f - /filesystem | /opt/local/gzip - > /tmp/dump.gz

–> shutdown mysql databse /usr/local/bin/mysqladmin shutdown -ppassword /etc/rc.d/init.d/mysql.server stop

–> test the loading of a module PERLDLDEBUG=255 perl -e ‘use CGI;’

–> shows open files fuser –cu /

–> Writing a Daemon: 1. edit /etc/services add service and port. 2. edit /etc/inetd.conf add in: edwardd stream tcp nowait root /bin/sh /bin/sh /home/sextone/bin/SERVER.mine 3. kill –HUP inetd.conf

–> how to mount a file system mount /dev/dsk/c3t0d0s4 /apps/data/easysoft/DEVT

–> look at sar log sar –f /var/adm/sa/sa24

–> write file checksums and size cksum filename

–> show storage array info ssaadm display /dev/rdsk/c1t5d2s0 –> show all disks on device d luxadm display d

–> examine for a specific OS finerprint nmap –sS -p 80 -O -v = examine OS

–> show print jobs /usr/ucb/lpq –Plevel6

–> Scan for known ports. log it. do OS scan.
nmap –sS -F -o foo.log -v -O www.foo.com//24 =

–> show status of printer /usr/ucb/lpc status

–> make a swap file: dd if=/dev/zero of=swapfile bs=1024 count=65535 mkswap ./swapfile chmod 600 ./swapfile swapon ./swapfile

–> show open files for process lsof –p PID

–> show open files for all TCP connections lsof –iTCP

–> show open files for internet address lsof -iTCP@10.20.2.9

–> as above lsof -i @10.20.2.122

–> examine tcp ports lsof -iTCP@sarah:1-50000

–> show open files for user. lsof –u username

–> show processes that has the file in use. lsof /apps/cms/ECMS-Server

–> show open files and retry every 5 seconds lsof –p process-id –r 5

–> mount a floppy mount -t vfat /dev/fd0 /mnt/floppy

–> check here for debugging processes and errno.h for errors /usr/include /usr/include/sys /usr/include/sys/errno.h

–> scp a whole directory, preserve mods sudo scp -prv devel webadmin@203.19.123.140:/home/httpd/cgi-bin

–> take processor 2 and 3 offline. psradm -f 2 3 –> show processor stats verbose. psrinfo –v

–> how to skip grant tables in mysql (over ride security) /usr/local/libexec/mysqld -Sg

–> how to feed in an SQL program mysql rm all files in directories find . -type f -exec rm {} ;

–> dump packets to a capture file sudo snoop –o /tmp/tcp.txt cp

–> backup one liner tar cvf - /home/ebs | gzip - > ebs.tar.gz

–> Look at selected packets in capture file sudo snoop -i /tmp/tcp.txt

–> unzip and pipe to tar gzip -dc watch packets from two servers. snoop sarah brucey

–> enable ip masquerading /sbin/ipchains -P forward DENY /sbin/ipchains -A forward -s 10.100.100.100/8 -j MASQ

-> view su log file cat /var/adm/sulog

–> establish a default router or gateway. echo “sagacity.com” > /etc/defaultrouter echo “10.100.100.100 sagacity.com sagacity” >> /etc/hosts change /etc/nsswitch.conf so that hosts has files, dns edit resolv.conf put in search . nameserver 203.7.132.98

–> turn off automounter on /export/home. vi /etc/auto_master, comment out /export/home

–> configuration file for sudoers /opt/local/etc/sudoers

–> building ssh-1.2.27 on x86Solaris2.6 needed a few things: /usr/openwin/bin in path /usr/xpg4/bin in path declare AR=”/usr/xpg4/bin/ar” declare NM_PATH=”/usr/xpg4/bin/nm”

–> snoop network packets and get size and time stamp entries. snoop -S -ta empa1

–> access perl CPAN perl -MCPAN -e shell install DBI

–> search for no password entries and lock all accounts. for i in passwd –sa | grep NP | awk ‘{print $1’ do echo “locking $i�? passwd –l $i done

–> delete from a tar tar –delete -f fsbackupSunday.tar home/ebs/tmp

–> Example on backing up files to tape. Must specify non rewinding, else you will over-write the files. for file in ls do echo “sending $file to tape…” echo date tar cvpf /dev/rmt/0n $file done

–> making/adding a partition. 1. use fdisk to make a parition. 2. mkfs -t ext2 -c /dev/hda11 3. mount -t ext2 /dev/hda11 /opt2 4. update /etc/fstab

–> rebuild the windex file catman –w –M /usr/share/man

–> execute tar on remote host sarah and send tarball to standard output, which becomes standard input for tar xvf – and the file gets dumped locally, in this case on crawl. you have to cd to dir before tar or else you will include path in tar ssh maggie “cd $DIRNAME; tar cvf - $BASENAME” | (cd $TPATH; tar xvf - )

–> dump a remote filesystem and send it to local tape drive. ssh -–x $fw /usr/sbin/ufsdump 0cf - $fs | dd obs=63k of=$TAPE

–> encrypt filename 1 and output to 1.crypt file crypt 1.crypt ; rm 1

–> decrypt filename 1.crypt and stdout to screen crypt send a file to tape tar cvpf /dev/rmt/0 filename

–> quicker way to search and replace in vi : %s/existing/new/g

–> shows where and which shared library files an application uses. ldd binary

–> shell script stuff:
repeat a command 100 times

x=100 while [ $x -gt 0 ] do command x=$(($x-1)) done

–> Something very important to remember about partitions It is important to note that Cylinder 0 contains the disklabel, which contains the partition table. A normal filesystem can be placed starting at Cylinder 0, since it will not touch the disklabel.
If you lay down a raw device, for a database, over Cylinder 0, then you will completely lose all your partitions. You will then have to restore the disklabel, and backup from tape if you happen to do this.

–> move a partition find . |cpio -pdm /apps

–> cron structure min hour day-of-month month weekday command

–> PatchDiag Tool. Get patches from: http://sunsolve.sun.com/private-cgi/patchpage.pl patchdiag.xref is available at: http://sunsolve.sun.com/sunsolve/patchdiag/ /opt/local/bin/patchdiag -x /opt/local/lib/patchdiag.xref > patchdiag.uname -n

–> command showing system parameters /usr/sbin/sysdef

–> Get Ambient Temperature of Server /usr/platform/SUNW,Ultra-4/sbin/prtdiag -v

–> good ps formatting showing percent cpu first. ps -edf -o pcpu,pid,user,arg

–> full details on ps /usr/bin/ps –A -o user,pid,pcpu,pmem,vsz,rss,tty,s,stime,time,args

–> chown the hidden files as well. find . -print -exec chown -R sextone:staff {} ;

–> The nsradmin command is a command-line based administrative program for the NetWorker system. Normally nsradmin monitors and modifies NetWorker resources over the network. /usr/sbin/nsr/nsradmin

–> Spray a server -c number of packets -d delay in microseconds -l pakcet size in bytes /usr/sbin/spray -c 1 –d 20 -l 4096 maggie

–> Turn on bold. bold=tput smso offbold=tput rmso echo “${bold}You must be the “root” user to run this script.${offbold}”

–> good way to send a dir to tape tar cf /dev/rmt/0n directory

–> example of bringing up an interface ifconfig hme0:1 inet 10.2.25.52 up

–> show all connections netstat –f inet

–> rpcinfo makes an RPC call to an RPC server and reports what it finds. rpcinfo -b 390109 2 | sort -u

–> rewind a tape fast show loaded modules /usr/sbin/modinfo

–> find world readable files and dirs find / -type d –perm -2 –print find . -type f –perm -2 -print

–> adding in a boot alias, eg: boot sarahroot1 –s nvalias sarahroot1 /sbus@1f,0/sunw,fas@e,8800000/sd@9,0:a

–> clever way to archive tar cvf - find . –print >/tmp/dumpfile.tar tar xvf - tee to a file echo “Start Date/Time: date” | tee -a $LOG_FILE

–> read a snoop file snoop -i anz-telnet.snoop

–> write a snoop log (this will count the number of connections, which is pretty neat). snoop –osnoop.log sarah

–> set default run level. 5 for gui. /etc/inittab

–> show all exported filesystems showmount -e crawl

–> shows all configurable variables for tcp interface. sudo ndd -get /dev/tcp - ? eg: sudo ndd -get /dev/tcp tcpconnreqmaxq 128 ndd /dev/arp ? ndd /dev/ip ? ndd /dev/tcp ? ndd /dev/udp ? ndd /dev/icmp ?

–> set sticky bit on group files, only the owner can change the mode.
–> the +l is mandatory file and record locking while a program –> is accessing that file. chmod g+s,+l file

–> print duplex landscape 4 qudrant printing mpage –t –l –4

–> install a patch installpatch .

–> check to see if a patch has been installed showrev –p |grep package name

–> unzip, untar in a /tmp directory zcat 104708-16.tar.gz | ( cd /tmp; sudo tar xvf - )

–> check out revision level on ssa controller /usr/sbin/ssaadm display controller

–> unzip and untar a file without having to create an intermediate tar file sudo gzip -dc /tmp/270599/post-EOD.tar.gz |tar xvf -

–> selectively extract from a tar archive tar xvf /tmp/iona.tar ./iona/.sh_history

–> send a bunch of files to tape tar cf /tmp/rules.tar ruleb* objects.C *.W

–> examine section 5 of man man -s 5 signal

–> shows signals and definitions of structures, eg sigaction /usr/include/sys/signal.h

–> location of the limits file on solaris /usr/include/limits.h

–> send an attachment via email from command prompt uuencode file.tar.gz file.tar.gz | mailx –s “backup�? root@crawl

–> zero a file cat /dev/null > isam.log

–> good way to restore from cdrom a binary file zcat running su as a user then ssh su - dls-PROD -c “/opt/local/bin/ssh drp-stagger “cd /tmp; /bin/ls” “

–> verify a newfs format sudo newfs –Nv /dev/md/dsk/d96

–> making lost_found. must be 8192 bytes in size. mkdir ./lost+found;chown root ./lost+found; chgrp root ./lost+found ;chmod 700 ./lost+found’; cd ./lost+found nofiles=0 ; while [ “$nofiles” -le 650 ] ; do ; /usr/ucb/touch $nofiles ; nofiles=expr $nofiles + 1 ; done

–> execute lynx lynx -cfg /usr/lib/lynx.cfg

–> sed search example sed ‘/Sep 25/!d; /castill/!d’ /var/log/syslo

–>should only be used at the EEPROM boot –r –> should be used at single user mode reboot — -r –> should be used in multiuser mode touch /reconfigure

–> performing a remote dump

find MFASYS |cpio -oc |gzip -c |ssh brucey -l chaup dd obs=18k of=/dev/rmt/0n

* to extract - cd /ssa/emphasys/sybase/dump dd ibs=18k if=|gunzip -c |cpio –idc

–> boot block located here. /usr/platform/uname –i/lib/fs/ufs

–> getting a server on the network add hosts entry for IP address clear configs: ifconfig pe0 unplumb ifconfig pe0 10.20.2.27 netmask 255.0.0.0 up route add default 10.20.0.1 1 verify the routing table: netstat –rn add resolv.conf entry: domain rabobank.com.au nameserver 192.192.192.252 edit /etc/nsswitch.conf change hosts to files, dns

lesson here is to unplumb interface, and let ifconfig setup the routing.
if you specify an ip address and a netmask it will manage the routing and the broadcasting.

–> find all, files associated with PID 22240 /usr/proc/bin/pfiles 22240 find file based on inode find –i number “ncheck –i number

–> good redirection example ./a.out trash

–> synchronize files from one server to another. This is useful for synchronizing database dump files, binary files, etc. This is definitely a powerful tool.

rsync -avz -e ssh –rsync-path=”/usr/local/bin/rsync” pwd myhost.com:/home/ebs/public_html

–> Example Awk Script
run with awk -f/tmp/1.awk /etc/group

BEGIN { FS = “:” } { print $1 | “sort” } { nlines++ } END { print nlines }

–> awk example. awk ‘/#/ {print “Got a comment”}’ /etc/hosts

–> delete every 2nd field in file awk ‘{$2= “”; print}’ datafile > datafile.new

–> awk average/standard deviation program

x1 += $1 x2 += $1*$1

END { x1 = x1/NR x2 = x2/NR sigma = sqrt(x2 - x1*x1) if (NR > 1) stderr = sigma/sqrt(NR - 1) print “Number of points = ” NR print “Mean = ” x1 print “Standard Deviation = ” sigma print “Standard Error = ” stderr

–> Setting Prompt PS1=”hostname($LOGNAME)–>”

mount syntax
mount 10.0.20.41:/mnt/cdrom /mnt/cdrom

ldapsearch syntax
ldapsearch -h mainldap -b 'o=vialta.com,c=us' cn="*"

ldapmodify syntax
ldapmodify -h masterldap -D 'cn=Directory Manager,o=vialta.com,c=US' -r -f /tmp/user.ldif -w
"password123"

ldapadd syntax
ldapadd -f user.ldif -D 'cn=Directory Manager,o=vialta.com,c=US' -w "password"

ldapdelete syntax
ldapdelete -f martin.ldif -D "cn=Directory Manager,o=vialta.com,c=US" -w "password"

sendmail debugging
sendmail -bt -d0.1,21.12
truss -fae -o /tmp/truss.log -p
Count System Calls
truss -c

Debugging processes in Linux: strace
strace -a80 -f
strace -a80 -f -p

manually setting date/time
date 07091427.00 Syntax: (mmddhhmm.ss)


make a large file
(linux) dd if=/dev/zero of=bigfie bs=1024 count=65536
(sun) mkfile bigfile 65m

count number of open files
lsof | awk '{ print $1 }' | uniq -c | sort -n | awk '{print $1}' | awk ' BEGIN { a=0; } {a+=$1;} END {print a; }'


apache bench marking
/usr/sbin/ab -n 100 -c 100 http://register.vialta.com:80/registe/index.ecgi

using dtterm with ssh, $1 is the hostname argument, eg: ./go va5-prod-101
dtterm -geometry 80x50 -n $1 -title $1 -bg $BG -fg $FG -cr $CR -sb -aw -e /usr/local/bin/ssh.binary
$LOGNAME@$1 &

global search and replace in vi
:%s/oldstring/newstring/g

Using php to md5 encrypt
echo "" | php -q

delete all ldap entries
$ ldapsearch -h mainldap -b 'o=vialta.com,c=us' cn="*" >/tmp/all.ldif
$ ldapdelete -c -f /tmp/all.ldif -D "directory Manager,o=vialta.com,c=us" -w password
run again the ldapdelete, this will remove non-leaf nodes. it is a hack but it works.


testing radius logins
Usage: radtest login passwd server:port nas_port_id secretkey

radtest ebs password localhost localhost testing123

Writing to HPOV
opcmsg sev=normal app=Apache MsgGroup=Web_Apps node=va5-prod-101 msg_text=Your message
goes here.
sev=normal|warning|minor|major|critical

LDAP and STDIN
ldapsearch -h mainldap -b 'o=vialta.com,c=US' cn=ed_904 | ldapdelete -c -D "cn=Directory
Manager,o=vialta.com,c=US" -w secret_password
ldapsearch -h mainldap -b 'o=vialta.com,c=US' cn=ed_904 | ldapdelete -c -D "cn=Directory
Manager,o=vialta.com,c=US" -w secret_password
Use -c to continue if errors are detected.

Oracle: deleting a username
sqlplus register ; SQL> exec maint.del_user('USERID');

Oracle: Decryption
select reg_crypt.pwd_decrypt(screen_pwd) from screen_name where screen_name='username';
select reg_crypt.pwd_decrypt(screen_pwd) from screen_name where screen_name='username';
select reg_crypt.pwd_decrypt(security_question_answer) from screen_name where
screen_name='username';
select vbl_crypt.cc_decrypt(credit_card_num) from vbl_user_creditcard;
select reg_crypt.pwd_decrypt(CENTER_PASSWORD) from CENTER where
CENTER_PUBLIC_SCREENNAME='username';

LDAP Information
ldapsearch -s base -b cn=monitor 'objectclass=#'

MPortal: fix mysql links for web guide management
use portal; select * from web_title where title_id=16 and title_code=0; update web_title set
title_code=11 where title_id=16 and title_code=0;

keyword search all files and print file names and date/size
find . -type f -exec grep -l search_word {} ; | xargs ls -al

remove encrypted key from cert
( cd /etc/httpd/conf/ssl.key && openssl rsa -in home.vialta.com.key -out home.vialta.com.key)

broadcast ping
for host in `ping -b 10.0.101.255 -c 2 | awk '{print $4}' | grep "^10" |sed s/.$//g`; do echo -n "$host ";
nslookup $host 2>/dev/null | grep Name; echo ;done


Find all files a process tries to open
truss -t !all -t open

ldapsearch operators

AND operator:
ldapsearch -h mainldap -b 'ou=1,o=vialta.com, c=US' "(&(suffixflag=0)(loginname=eval*))"

OR operator:
ldapsearch -h mainldap -b 'ou=1,o=vialta.com, c=US' "(|(loginname=thanurak)(loginname=ebs))"

Adding a NewLine character with sed. Use a backslash
ldapsearch -h mainldap -b 'ou=1,o=vialta.com, c=US' cn=ebs | sed
s/suffixflag=0/userid=000000000000/g

Adding a New field to existing LDAP database (this will add new field: foo=bar
ldapsearch -h masterldap -b o=vialta.com,c=us cn=loopy | awk '{print} /suffixflag=0/{print
"foo=bar"}' | ldapmodify -r -h masterldap -D "cn=Directory Manager, o=vialta.com, c=US" -w xxx


Continuing a process
If /proc/$PID/status ever shows a State: T (Stopped), then start it with this signal:
kill -SIGCONT pid

Oracle shutdown
export ORACLE_SID ; sqlplus internal ; select * from v$database; shutdown immediate;

Interesting Oracle views
v$sga;
v$session


using uuencode for file transfer
$ uuencode filename filename | mail esexton@sun1-noc
eg: uuencode tnsnames.ora tnsnames.ora | mail esexton@sun1-noc
tar cvf - * | uuencode backup.tar | mail esexton@sun1-noc
(linux) $ tar zcvf - * | uuencode backup.tar.gz | mail esexton@sun1-noc
(sun) $ tar cvf - * | gzip - | uuencode backup.tar.gz | mailx esexton@sun1-noc

LDAP protocol 2 specific commands

LDAP protocol 2 query:
ldapsearch -LLL -P2 -x -h mainldap.vialta-inc.com -b 'o=vialta.com, c=us' cn=guest
ldapdelete -x "cn=j2, ou=1, o=vialta.com, c=US" -w
ldapsearch -x -LLL -b 'o=vialta.com, c=US' cn=j2
ldapadd -x -f newadd1.ldif -D "cn=Directory Manager,o=vialta.com,c=us" -w ""

Tailing tcpdump
tcpdump -l udp > dat & tail -f dat


mysql date select
SELECT * FROM `connection` where login_time > "2002-12-01 00:00:01"


RPM extract commands
Get a content listing:
rpm2cpio web-programs.rpm | cpio -it

Extract Specific File:
rpm2cpio web-programs.rpm | cpio -idm

Extract Entire contents:
rpm2cpio web-programs.rpm | cpio -ivd

Sendmail
Testing aliases and routing.
echo "3,0 nreynolds@orchestream.com" | sendmail -bt -d60.1
echo "3,0 nreynolds@orchestream.com" | sendmail -bt -d60.1 -d21.12

use sed to translate a space to a newline
sed 's/ /

/g' filename

Bind version
nslookup -q=txt -class=CHAOS version.bind. 0
dig @host version.bind chaos txt
named -v

Query MX record
nslookup -q=mx server-name

freebsd
pkg_add package.tgz
MySQL

Too many indexes on a table will cause delete operations to be slow.

Linux Route route add -net 10.0.19.0 netmask 255.255.255.0 gw 10.0.101.4 /etc/sysconfig/static-routes: eth0 net 10.0.19.0 netmask 255.255.255.0 gw 10.0.101.4 route del -net 10.0.19.0 gw 10.0.101.4 netmask 255.255.255.0

Sed Onliners

FILE SPACING:

# double space a file
sed G

# double space a file which already has blank lines in it. Output file
# should contain no more than one blank line between lines of text.
sed '/^$/d;G'

# triple space a file
sed 'G;G'

# undo double-spacing (assumes even-numbered lines are always blank)
sed 'n;d'

# insert a blank line above every line which matches "regex"
sed '/regex/{x;p;x;}'

# insert a blank line below every line which matches "regex"
sed '/regex/G'

# insert a blank line above and below every line which matches "regex"
sed '/regex/{x;p;x;G;}'

NUMBERING:

# number each line of a file (simple left alignment). Using a tab (see
# note on '\t' at end of file) instead of space will preserve margins.
sed = filename | sed 'N;s/\n/\t/'

# number each line of a file (number on left, right-aligned)
sed = filename | sed 'N; s/^/ /; s/ *\(.\{6,\}\)\n/\1 /'

# number each line of file, but only print numbers if line is not blank
sed '/./=' filename | sed '/./N; s/\n/ /'

# count lines (emulates "wc -l")
sed -n '$='

TEXT CONVERSION AND SUBSTITUTION:

# IN UNIX ENVIRONMENT: convert DOS newlines (CR/LF) to Unix format
sed 's/.$//' # assumes that all lines end with CR/LF
sed 's/^M$//' # in bash/tcsh, press Ctrl-V then Ctrl-M
sed 's/\x0D$//' # gsed 3.02.80, but top script is easier

# IN UNIX ENVIRONMENT: convert Unix newlines (LF) to DOS format
sed "s/$/`echo -e \\\r`/" # command line under ksh
sed 's/$'"/`echo \\\r`/" # command line under bash
sed "s/$/`echo \\\r`/" # command line under zsh
sed 's/$/\r/' # gsed 3.02.80

# IN DOS ENVIRONMENT: convert Unix newlines (LF) to DOS format
sed "s/$//" # method 1
sed -n p # method 2

# IN DOS ENVIRONMENT: convert DOS newlines (CR/LF) to Unix format
# Can only be done with UnxUtils sed, version 4.0.7 or higher.
# Cannot be done with other DOS versions of sed. Use "tr" instead.
sed "s/\r//" infile >outfile # UnxUtils sed v4.0.7 or higher
tr -d \r outfile # GNU tr version 1.22 or higher

# delete leading whitespace (spaces, tabs) from front of each line
# aligns all text flush left
sed 's/^[ \t]*//' # see note on '\t' at end of file

# delete trailing whitespace (spaces, tabs) from end of each line
sed 's/[ \t]*$//' # see note on '\t' at end of file

# delete BOTH leading and trailing whitespace from each line
sed 's/^[ \t]*//;s/[ \t]*$//'

# insert 5 blank spaces at beginning of each line (make page offset)
sed 's/^/ /'

# align all text flush right on a 79-column width
sed -e :a -e 's/^.\{1,78\}$/ &/;ta' # set at 78 plus 1 space

# center all text in the middle of 79-column width. In method 1,
# spaces at the beginning of the line are significant, and trailing
# spaces are appended at the end of the line. In method 2, spaces at
# the beginning of the line are discarded in centering the line, and
# no trailing spaces appear at the end of lines.
sed -e :a -e 's/^.\{1,77\}$/ & /;ta' # method 1
sed -e :a -e 's/^.\{1,77\}$/ &/;ta' -e 's/\( *\)\1/\1/' # method 2

# substitute (find and replace) "foo" with "bar" on each line
sed 's/foo/bar/' # replaces only 1st instance in a line
sed 's/foo/bar/4' # replaces only 4th instance in a line
sed 's/foo/bar/g' # replaces ALL instances in a line
sed 's/\(.*\)foo\(.*foo\)/\1bar\2/' # replace the next-to-last case
sed 's/\(.*\)foo/\1bar/' # replace only the last case

# substitute "foo" with "bar" ONLY for lines which contain "baz"
sed '/baz/s/foo/bar/g'

# substitute "foo" with "bar" EXCEPT for lines which contain "baz"
sed '/baz/!s/foo/bar/g'

# change "scarlet" or "ruby" or "puce" to "red"
sed 's/scarlet/red/g;s/ruby/red/g;s/puce/red/g' # most seds
gsed 's/scarlet\|ruby\|puce/red/g' # GNU sed only

# reverse order of lines (emulates "tac")
# bug/feature in HHsed v1.5 causes blank lines to be deleted
sed '1!G;h;$!d' # method 1
sed -n '1!G;h;$p' # method 2

# reverse each character on the line (emulates "rev")
sed '/\n/!G;s/\(.\)\(.*\n\)/&\2\1/;//D;s/.//'

# join pairs of lines side-by-side (like "paste")
sed '$!N;s/\n/ /'

# if a line ends with a backslash, append the next line to it
sed -e :a -e '/\\$/N; s/\\\n//; ta'

# if a line begins with an equal sign, append it to the previous line
# and replace the "=" with a single space
sed -e :a -e '$!N;s/\n=/ /;ta' -e 'P;D'

# add commas to numeric strings, changing "1234567" to "1,234,567"
gsed ':a;s/\B[0-9]\{3\}\>/,&/;ta' # GNU sed
sed -e :a -e 's/\(.*[0-9]\)\([0-9]\{3\}\)/\1,\2/;ta' # other seds

# add commas to numbers with decimal points and minus signs (GNU sed)
gsed ':a;s/\(^\|[^0-9.]\)\([0-9]\+\)\([0-9]\{3\}\)/\1\2,\3/g;ta'

# add a blank line every 5 lines (after lines 5, 10, 15, 20, etc.)
gsed '0~5G' # GNU sed only
sed 'n;n;n;n;G;' # other seds

SELECTIVE PRINTING OF CERTAIN LINES:

# print first 10 lines of file (emulates behavior of "head")
sed 10q

# print first line of file (emulates "head -1")
sed q

# print the last 10 lines of a file (emulates "tail")
sed -e :a -e '$q;N;11,$D;ba'

# print the last 2 lines of a file (emulates "tail -2")
sed '$!N;$!D'

# print the last line of a file (emulates "tail -1")
sed '$!d' # method 1
sed -n '$p' # method 2

# print only lines which match regular expression (emulates "grep")
sed -n '/regexp/p' # method 1
sed '/regexp/!d' # method 2

# print only lines which do NOT match regexp (emulates "grep -v")
sed -n '/regexp/!p' # method 1, corresponds to above
sed '/regexp/d' # method 2, simpler syntax

# print the line immediately before a regexp, but not the line
# containing the regexp
sed -n '/regexp/{g;1!p;};h'

# print the line immediately after a regexp, but not the line
# containing the regexp
sed -n '/regexp/{n;p;}'

# print 1 line of context before and after regexp, with line number
# indicating where the regexp occurred (similar to "grep -A1 -B1")
sed -n -e '/regexp/{=;x;1!p;g;$!N;p;D;}' -e h

# grep for AAA and BBB and CCC (in any order)
sed '/AAA/!d; /BBB/!d; /CCC/!d'

# grep for AAA and BBB and CCC (in that order)
sed '/AAA.*BBB.*CCC/!d'

# grep for AAA or BBB or CCC (emulates "egrep")
sed -e '/AAA/b' -e '/BBB/b' -e '/CCC/b' -e d # most seds
gsed '/AAA\|BBB\|CCC/!d' # GNU sed only

# print paragraph if it contains AAA (blank lines separate paragraphs)
# HHsed v1.5 must insert a 'G;' after 'x;' in the next 3 scripts below
sed -e '/./{H;$!d;}' -e 'x;/AAA/!d;'

# print paragraph if it contains AAA and BBB and CCC (in any order)
sed -e '/./{H;$!d;}' -e 'x;/AAA/!d;/BBB/!d;/CCC/!d'

# print paragraph if it contains AAA or BBB or CCC
sed -e '/./{H;$!d;}' -e 'x;/AAA/b' -e '/BBB/b' -e '/CCC/b' -e d
gsed '/./{H;$!d;};x;/AAA\|BBB\|CCC/b;d' # GNU sed only

# print only lines of 65 characters or longer
sed -n '/^.\{65\}/p'

# print only lines of less than 65 characters
sed -n '/^.\{65\}/!p' # method 1, corresponds to above
sed '/^.\{65\}/d' # method 2, simpler syntax

# print section of file from regular expression to end of file
sed -n '/regexp/,$p'

# print section of file based on line numbers (lines 8-12, inclusive)
sed -n '8,12p' # method 1
sed '8,12!d' # method 2

# print line number 52
sed -n '52p' # method 1
sed '52!d' # method 2
sed '52q;d' # method 3, efficient on large files

# beginning at line 3, print every 7th line
gsed -n '3~7p' # GNU sed only
sed -n '3,${p;n;n;n;n;n;n;}' # other seds

# print section of file between two regular expressions (inclusive)
sed -n '/Iowa/,/Montana/p' # case sensitive

SELECTIVE DELETION OF CERTAIN LINES:

# print all of file EXCEPT section between 2 regular expressions
sed '/Iowa/,/Montana/d'

# delete duplicate, consecutive lines from a file (emulates "uniq").
# First line in a set of duplicate lines is kept, rest are deleted.
sed '$!N; /^\(.*\)\n\1$/!P; D'

# delete duplicate, nonconsecutive lines from a file. Beware not to
# overflow the buffer size of the hold space, or else use GNU sed.
sed -n 'G; s/\n/&&/; /^\([ -~]*\n\).*\n\1/d; s/\n//; h; P'

# delete all lines except duplicate lines (emulates "uniq -d").
sed '$!N; s/^\(.*\)\n\1$/\1/; t; D'

# delete the first 10 lines of a file
sed '1,10d'

# delete the last line of a file
sed '$d'

# delete the last 2 lines of a file
sed 'N;$!P;$!D;$d'

# delete the last 10 lines of a file
sed -e :a -e '$d;N;2,10ba' -e 'P;D' # method 1
sed -n -e :a -e '1,10!{P;N;D;};N;ba' # method 2

# delete every 8th line
gsed '0~8d' # GNU sed only
sed 'n;n;n;n;n;n;n;d;' # other seds

# delete ALL blank lines from a file (same as "grep '.' ")
sed '/^$/d' # method 1
sed '/./!d' # method 2

# delete all CONSECUTIVE blank lines from file except the first; also
# deletes all blank lines from top and end of file (emulates "cat -s")
sed '/./,/^$/!d' # method 1, allows 0 blanks at top, 1 at EOF
sed '/^$/N;/\n$/D' # method 2, allows 1 blank at top, 0 at EOF

# delete all CONSECUTIVE blank lines from file except the first 2:
sed '/^$/N;/\n$/N;//D'

# delete all leading blank lines at top of file
sed '/./,$!d'

# delete all trailing blank lines at end of file
sed -e :a -e '/^\n*$/{$d;N;ba' -e '}' # works on all seds
sed -e :a -e '/^\n*$/N;/\n$/ba' # ditto, except for gsed 3.02*

# delete the last line of each paragraph
sed -n '/^$/{p;h;};/./{x;/./p;}'

SPECIAL APPLICATIONS:

# remove nroff overstrikes (char, backspace) from man pages. The 'echo'
# command may need an -e switch if you use Unix System V or bash shell.
sed "s/.`echo \\\b`//g" # double quotes required for Unix environment
sed 's/.^H//g' # in bash/tcsh, press Ctrl-V and then Ctrl-H
sed 's/.\x08//g' # hex expression for sed v1.5

# get Usenet/e-mail message header
sed '/^$/q' # deletes everything after first blank line

# get Usenet/e-mail message body
sed '1,/^$/d' # deletes everything up to first blank line

# get Subject header, but remove initial "Subject: " portion
sed '/^Subject: */!d; s///;q'

# get return address header
sed '/^Reply-To:/q; /^From:/h; /./d;g;q'

# parse out the address proper. Pulls out the e-mail address by itself
# from the 1-line return address header (see preceding script)
sed 's/ *(.*)//; s/>.*//; s/.*[:<] *//'

# add a leading angle bracket and space to each line (quote a message)
sed 's/^/> /'

# delete leading angle bracket & space from each line (unquote a message)
sed 's/^> //'

# remove most HTML tags (accommodates multiple-line tags)
sed -e :a -e 's/<[^>]*>//g;/
# extract multi-part uuencoded binaries, removing extraneous header
# info, so that only the uuencoded portion remains. Files passed to
# sed must be passed in the proper order. Version 1 can be entered
# from the command line; version 2 can be made into an executable
# Unix shell script. (Modified from a script by Rahul Dhesi.)
sed '/^end/,/^begin/d' file1 file2 ... fileX | uudecode # vers. 1
sed '/^end/,/^begin/d' "$@" | uudecode # vers. 2

# zip up each .TXT file individually, deleting the source file and
# setting the name of each .ZIP file to the basename of the .TXT file
# (under DOS: the "dir /b" switch returns bare filenames in all caps).
echo @echo off >zipup.bat
dir /b *.txt | sed "s/^\(.*\)\.TXT/pkzip -mo \1 \1.TXT/" >>zipup.bat

TYPICAL USE: Sed takes one or more editing commands and applies all of
them, in sequence, to each line of input. After all the commands have
been applied to the first input line, that line is output and a second
input line is taken for processing, and the cycle repeats. The
preceding examples assume that input comes from the standard input
device (i.e, the console, normally this will be piped input). One or
more filenames can be appended to the command line if the input does
not come from stdin. Output is sent to stdout (the screen). Thus:

cat filename | sed '10q' # uses piped input
sed '10q' filename # same effect, avoids a useless "cat"
sed '10q' filename > newfile # redirects output to disk

For additional syntax instructions, including the way to apply editing
commands from a disk file instead of the command line, consult "sed &
awk, 2nd Edition," by Dale Dougherty and Arnold Robbins (O'Reilly,
1997; http://www.ora.com), "UNIX Text Processing," by Dale Dougherty
and Tim O'Reilly (Hayden Books, 1987) or the tutorials by Mike Arst
distributed in U-SEDIT2.ZIP (many sites). To fully exploit the power
of sed, one must understand "regular expressions." For this, see
"Mastering Regular Expressions" by Jeffrey Friedl (O'Reilly, 1997).
The manual ("man") pages on Unix systems may be helpful (try "man
sed", "man regexp", or the subsection on regular expressions in "man
ed"), but man pages are notoriously difficult. They are not written to
teach sed use or regexps to first-time users, but as a reference text
for those already acquainted with these tools.

QUOTING SYNTAX: The preceding examples use single quotes ('...')
instead of double quotes ("...") to enclose editing commands, since
sed is typically used on a Unix platform. Single quotes prevent the
Unix shell from intrepreting the dollar sign ($) and backquotes
(`...`), which are expanded by the shell if they are enclosed in
double quotes. Users of the "csh" shell and derivatives will also need
to quote the exclamation mark (!) with the backslash (i.e., \!) to
properly run the examples listed above, even within single quotes.
Versions of sed written for DOS invariably require double quotes
("...") instead of single quotes to enclose editing commands.

USE OF '\t' IN SED SCRIPTS: For clarity in documentation, we have used
the expression '\t' to indicate a tab character (0x09) in the scripts.
However, most versions of sed do not recognize the '\t' abbreviation,
so when typing these scripts from the command line, you should press
the TAB key instead. '\t' is supported as a regular expression
metacharacter in awk, perl, and HHsed, sedmod, and GNU sed v3.02.80.

VERSIONS OF SED: Versions of sed do differ, and some slight syntax
variation is to be expected. In particular, most do not support the
use of labels (:name) or branch instructions (b,t) within editing
commands, except at the end of those commands. We have used the syntax
which will be portable to most users of sed, even though the popular
GNU versions of sed allow a more succinct syntax. When the reader sees
a fairly long command such as this:

sed -e '/AAA/b' -e '/BBB/b' -e '/CCC/b' -e d

it is heartening to know that GNU sed will let you reduce it to:

sed '/AAA/b;/BBB/b;/CCC/b;d' # or even
sed '/AAA\|BBB\|CCC/b;d'

In addition, remember that while many versions of sed accept a command
like "/one/ s/RE1/RE2/", some do NOT allow "/one/! s/RE1/RE2/", which
contains space before the 's'. Omit the space when typing the command.

OPTIMIZING FOR SPEED: If execution speed needs to be increased (due to
large input files or slow processors or hard disks), substitution will
be executed more quickly if the "find" expression is specified before
giving the "s/.../.../" instruction. Thus:

sed 's/foo/bar/g' filename # standard replace command
sed '/foo/ s/foo/bar/g' filename # executes more quickly
sed '/foo/ s//bar/g' filename # shorthand sed syntax

On line selection or deletion in which you only need to output lines
from the first part of the file, a "quit" command (q) in the script
will drastically reduce processing time for large files. Thus:

sed -n '45,50p' filename # print line nos. 45-50 of a file
sed -n '51q;45,50p' filename # same, but executes much faster