Common Linux tools usage

Jephe Wu - http://linuxtechres.blogspot.com/


  • bash  - if
http://tldp.org/LDP/Bash-Beginners-Guide/html/sect_07_01.html
  • vi
30G - go to 30th  line
30| - go to 30th column
change a file to remove the ending new line
vi file , :%s#$\n# #g
  • sed (delete one line from the file itself)
for i in B*;do sed -i /A443/d $i;done 
JEPHE=A1234; for i in B*;do sed -i "/$JEPHE/d" $i;done
  • Tar
Backup files and exclude a list of files from a file:
tar --exclude-from=/path/to/excludedfilelist -cvpzf  file.tar.gz *

note: inside excludedfilelist, give file or directory name line by line, don't put / for directories behind.

delete a file from a tar archive:
tar --delete --file=file.tar tobedeletedfile

transfer sparse file on the network:


tar cvzSpf - *|ssh jephe@remoteserver '(cd /path/to; tar xzSpf -)'


  • rsync
use rsync to transfer specified files under some directories.
rsync -av -r  --include-from=include.txt  /cygdrive/e/ root@10.0.0.1:/data/backup/

$ cat include.txt
+ a
+ a/b/
+ b
+ b/c/
+ a/b/*.dat
+ b/c/*.exe
- *

/usr/bin/rsync --timeout=600 -v --progress --include=*.gpg --exclude=* -a -e ssh --delete /data/db 10.0.0.1:/data/db/ > /tmp/dbbackup

Note: refer to http://samba.anu.edu.au/ftp/rsync/rsync.html


transfer-root directory referes to the source directory on the source server.

Here are some examples of exclude/include matching:

o --exclude "*.o" would exclude all filenames matching *.o
o --exclude "/foo" would exclude a file called foo in the transfer-root directory
o --exclude "foo/" would exclude any directory called foo
o --exclude "/foo/*/bar" would exclude any file called bar two levels below a directory called foo in the transfer-root directory
o --exclude "/foo/**/bar" would exclude any file called bar two or more levels below a directory called foo in the transfer-root directory
o --include "*/" --include "*.c" --exclude "*" would include all directories and C source files
o --include "foo/" --include "foo/bar.c" --exclude "*" would include only foo/bar.c (the foo/ directory must be
explicitly included or it would be excluded by the "*")

o --exclude "*/foo/" would exclude any directory called foo which is one level below the transfer root directory
o --exclude "**/foo/" would exclude any directory called foo which is one or more levels below the transfer-root directory
  •  curl & wget
download a iso file  -  curl -CO complete_download_url and wget -c complete_download_url
check web response header  - curl -I www.domain.com

  • nc
nc -v -z -s 10.0.0.1 10.0.10.1 25
Connection to 10.0.10.1 25 port [tcp/smtp] succeeded!
for udp, use:
nc -vuz destination_ip_addr 53  (udp might always report it's successful when using -z option)

  • socat  (linux or cygwin)
$ socat -d -d tcp4-listen:25,bind=192.168.100.20,fork,reuseaddr tcp4:smtp.gmail.com:25,bind=192.168.100.20 &

  • awk
ls -l *.3.gz.gpg  | awk '{total += $5} END { print total}'
2298937968

db2 list application | awk '($3==495)'  => print the third column which equals 495

  • netstat
netstat -tunelp  --list listening tcp and udp port numbers
netstat -natup
  • ssh port forwarding 
Local port forwarding
$ ssh jephe@server1 -L 1234:10.0.0.1:22 [-g]  (-g means allows remote hosts to connect to local forwarded ports)
$ ssh jephe@localhost -p 1234

Remote port forwarding
$ ssh jephe@office_server -R 1234:10.0.0.2:22  (on office server, the user can connect to localhost at port 1234 to access the local ssh server at port 22)

or


$ ssh jephe@office_server -R 2222:server_on_internet:22 (when user ssh to port 2222 to office server, it actually goes to internet server ssh port)

If you want to anyone from your office network to access office server at port 2222 which will be forwarded to server_on_internet ssh server, uncomment GatewayPorts line as
GatewayPorts yes


or
$ ssh jephe@office_server -R 80:your_home_web_server:80 (your home web server is not necessarily same as your home ssh client pc, can be another server)


Reference: http://souptonuts.sourceforge.net/sshtips.htm

You can use Windows cygwin ssh server plus ssh remote port forwarding to achieve something. Assume you have lease line connected to remote office, you can only ssh into remote office, now you need to install Linux in one of machine, you can use this method to install from local office http server.

Use above local and remote port forwarding + cygwin + openssh + putty + proxytunnel at http://proxytunnel.sourceforge.net/ , you can do a lot of things you might think it's impossible before.

If your company only allows to use proxy to access Internet and you control one of ssh server on Internet, then you got the power to access office network from home.
  • rpm
rpm -Uvh --root=/tmp/ --nodeps /mnt/iso/CentOS/glibc-2.5-34.x86_64.rpm 
or
  1. Make a temporary directory to extract the rpm in and copy the rpm into the directory: 
    
     mkdir tempdir
     cp bash.rpm tempdir
     

  2. Execute rpm2cpio in the temporary directory: 
    
     cd tempdir
     rpm2cpio bash.rpm | cpio -idmv
     
  3. show architecture
 rpm -qf /bin/ls --qf '%{NAME}-%{VERSION}-%{RELEASE}.%{ARCH}\n'

  • stty and setterm/xset
stty erase ^H  (ctrl V H at the same time) 

setterm -blank nn will tell the console driver to blank the screen after nn minutes of inactivity. (With nn = 0, screensaving is turned off. In some old kernels this first took effect after the next keyboard interrupt.)

The s option of xset(1) will set the X screensaving parameters: xset s off turns off the screensaver, xset s 10 blanks the screen after 10 minutes.
  • ssh 
ssh user@host -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null
ssh-keygen -F 10.0.3.1  - find this host from .ssh/known_hosts file
ssh-keygen -R 10.0.3.1  - remove this host from .ssh/known_hosts file
 
$ sshfs jephe@server:/path/to /mnt/sshfs/ 

Working in a ssh shell that used forwarding:
Supported escape sequences:

~. – terminate connection
~B – send a BREAK to the remote system
~C – open a command line
~R – Request rekey (SSH protocol 2 only)
~^Z – suspend ssh
~# – list forwarded connections
~& – background ssh (when waiting for connections to terminate)
~? – this message
~~ – send the escape character by typing it twice
(Note that escapes are only recognized immediately after newline.)
~. and ~# are particularly useful.

  • ftp (windows)
literal pasv (to change to passive mode under Windows ftp CLI) 

  • sftp
$ sftp -b /tmp/1 -o port=2222 jephe@1.2.3.4

$ more 1
put /etc/hosts

$ more 1
rm hosts
  • grep/cat
grep -v "^#" /etc/httpd/conf/httpd.conf | cat -s | less
  • tcpdump 
    Collecting a TCP dump from the server using a command like the following:
    tcpdump -s0 -w /tmp/tcpdump.pcap -i any host <client ip> and port 80

    and generating traffic (HTTP or LDAP) to the server captures evidence of the server not responding to the TCP SYN packets.  The output file can be analyzed with a command like
    tcpdump -r /<path>/<to>/tcpdump.pcap
For monitoring openbsd firewall PF rules, you can use 'tcpdump -n -e -ttt -i pflog0' to see which pf rules is matching the traffic, pass or block. use 'pfctl -sr' to get output of in-memory rules, the first block or pass rule will be rule 0, followed by rule 1, rule 2 and so on.
  • Tshark
     tshark -i eth0 -f 'host 10.0.2.1' -w /tmp/upstream.cap -S
  • screen
 screen -S main (manually)

/home/jephe/.bash_profile contact 'screen -D -R main'. this way, if I login server, it will disconnect then reconnect my screen session 'main' automatically every time. So we keep one session only.