Collected from CBRC wiki:
Original source: http://dragon.cbrc.kaust.edu.sa/wiki/index.php/Linux_Tips_and_Tricks

Linux Tips and Tricks

Jump to: navigation, search

Contents

• 1 Useful commands
  • 1.1 General commands
  • 1.2 Directories processing
  • 1.3 Files processing
  • 1.4 Find/Replace
  • 1.5 Processes
• 2 File System /Disk Management
  • 2.1 Disk space report
  • 2.2 Resizing Shared Memory (/dev/shm)
  • 2.3 Resizing partitions CentOS
  • 2.4 Disk crash recovery
  • 2.5 Mounting WebDAV Share
    • 2.5.1 Installing and configuring WebDAV for non-sudo on Ubuntu

Useful commands

General commands

• find Linux version

$ cat /etc/issue

Directories processing

$ # recursively delete backup files
$ find ./ -name '*~' | xargs rm 
 
$ # Process all files in a directory
$ ls dir_path/*.xml | xargs -n 1 MyProcessingProgram 
 
$ # gzip/guznip directory:
$ tar czvf myfile.tar.gz mydir/*
$ tar xzvf filename.tar.gz (or: tar xzvf filename.tar.gz -C foldername/)
$ gunzip -c foo.tar.gz | tar xvf

Files processing

$ # batch convert dos to unix type text files (e.g. .pl)
$ find . -name "*.pl" -type f -exec dos2unix {} \; 
 
$ # remove blank lines from a text file
$ sed '/^$/d' in.txt > out.txt
 
$ # sort file from shortest to longest line
$ cat myfile.txt | awk '{ print length, $0 }' | sort -n | cut -d" " -f2- > out.txt
$ # or from longest to the shortest
$ cat myfile.txt | awk '{ print length, $0 }' | sort -nr | cut -d" " -f2- > out.txt
 
$ # Count word occurence in a file
$ grep -w 'searchword' filename.txt | tr ' ' '\n' | wc -l

Find/Replace

$ # find words listed in one file (e.g. dictionary) in another (e.g. free text)
$ grep -i -f dictionary.txt text.txt   # ignore case (-i) and print matching lines
$ grep -io -f dictionary.txt text.txt  # ignore case (-i) and print matching pattern
$ while read line; do grep -io -P "\Q$line\E" text.txt; done < dictionary.txt # if there are special chars
 
$ # replace char1 with char2 in a file, e.g comma with tab
$ tr ',' '\t' < in.csv > out.tsv

Processes

$ # print process name, pid and running time for the process "xyz"
$ ps -eo fname,pid,etime | awk '{if ($1=="xyz") print $1,$2,$3}' | more
$ % or show pid for process XYZ that is running more than an hour:
$ ps --no-headers -C XYZ -o pid,etime | awk  '/[0-23]:[0-9][0-9]:[0-9][0-9]/ {print $1}'

File System /Disk Management

Disk space report

$ ncdu

To install ncdu on ubuntu:

$ sudo apt-get install ncdu

To install ncdu on Scientific Linux you need to install Extra Packages for Enterprise Linux (or EPEL) repository. The easiest way to do it is from the Synaptic Package Manager: just search for: Extra Packages for Enterprise Linux. After the EPEL is installed, you can install ncdu:

$ sudo yum install ncdu

Resizing Shared Memory (/dev/shm)

You can use /dev/shm to improve the performance of application software or overall Linux system performance. Use 'df -h' to see partition allocation. To permanently increase size of shared memory, append or modify /dev/shm entry as follows to set size to 8G

$ sudo nano /etc/fstab
# modify the following line by adding new size
none      /dev/shm        tmpfs   defaults,size=8G        0 0
# save and close the file. For the changes to take effect immediately remount /dev/shm:
$ sudo mount -o remount /dev/shm
# verify the same:
$ df -h

Resizing partitions CentOS

1. display partition information

$ df -kh
Filesystem                      Size  Used  Avail  Use% Mounted on
/dev/mapper/vg_centos-lv_root    50G   3.6G   44G   8%    /
/tmpfs                          1.9G   228K  1.9G   1%   /dev/shm
/dev/sda1                       485M    68M  393M  15%   /boot
/dev/mapper/vg_centos-lv_home   2.0T   200M  1.9T   1%   /home

1. Drop into the single user terminal to avoid read and writing to the partitions that will be resized

$ sudo telinit 1

if filesystem is used, try booting in a single user-mode (press any key to interrupt boot->letter 'e'->'kernel' option->letter 'e'->type single->letter 'b' to boot)

1. Decrease /dev/mapper/vg_web-lv_home

$ umount /dev/mapper/vg_centos-lv_home
$ e2fsck -f /dev/mapper/vg_centos-lv_home
$ resize2fs /dev/mapper/vg_centos-lv_home 200G
$ lvresize -L 200G /dev/mapper/vg_centos-lv_home
$ mount /home
$ mount -o remount /home
$ df -kh # check the size

1. Increase /dev/mapper/vg_centos-lv_root

$ lvresize -l +100%FREE /dev/mapper/vg_centos-lv_root
$ # or, alternatively: lvresize -L 1800G /dev/mapper/vg_centos-lv_root
$ resize2fs /dev/mapper/vg_centos-lv_root
$ df -kh # check the size
$ shutdown -r now

Disk crash recovery

My Linux (kubuntu) suddenly crashed. The screen blinked several times and the keyboard stopped responding. A new reboot failed, it either hung or went to a recovery mode (that did not recover anything). The (almost) only solution in such situations is to run a disk-checking program (e.g. e2fsck) that can fix bad sectors; in this case the suspicious was the Linux partition table, or the Linux boot sector. Therefore, this is what I did:

1. Get the latest version of Ubuntu on a CD.

1. Boot from it (on MacBook hold "C" during the reboot; that's how MacBook knows thta you want to boot from a CD). Go to a recovery mode.

1. Start partition management program parted. I needed to run this program to find out what is the device name of my Linux partition - and parted told me that it was /dev/sda5.

1. Then I started e2fsck (with the found device /dev/sda5 as a parameter) to check and fix the disk. I answered yes (meaning: yes, fix it please) to all prompts (actually I started it with the -y option).

Mounting WebDAV Share

$ mkdir ~/<my mount point>
$ sudo mount -t davfs http://10.75.106.38:5005 ~/<my mount point>

Installing and configuring WebDAV for non-sudo on Ubuntu

1. install davfs: sudo apt-get install davfs2
2. Reconfigure davfs2 to enable non-sudo mount: sudo dpkg-reconfigure davfs2
3. Edit /etc/davfs2/davfs2.conf to enable automatic credentials by uncommenting the line: secrets ~/.davfs2/secrets
4. Edit ~/.davfs2/secrets file to add credentials to remote WebDav diectory. Add a line to the end of file in following style: http://
5. Set the permission: chmod 600 ~/.davfs2/secrets
6. Add a line to /ect/fstab about the remote WebDav directory: http:// davfs user,noauto,file_mode=600,dir_mode=700 0 1
7. Add your user to the davfs2 group: sudo vi /etc/group as follows: davfs2:x:134: (134 can be some other gorup number - don't change it)
8. You can use following commands without being a root user to mount/umount

$ mount <mount point>
$ umount <mount point>

You can also use nautilus to mount/umount the directory.

Change User password in linux

sudo passwd username

- It will then prompt the new password setup for username.