Monday, April 10, 2017

Map with R



How it really works in R ggplot2 


http://eriqande.github.io/rep-res-web/lectures/making-maps-with-R.html

You need to create data in following format:

                long      lat        group order region subregion
#> 1 -101.4078 29.74224     1     1   main      
#> 2 -101.3906 29.74224     1     2   main      
#> 3 -101.3620 29.65056     1     3   main      
#> 4 -101.3505 29.63911     1     4   main      
#> 5 -101.3219 29.63338     1     5   main      
#> 6 -101.3047 29.64484     1     6   main      


Some useful packages


  • ggmap
  • googleVis







Sunday, April 9, 2017

Teradata Tutorial




Import CSV file as a table in Teradata


1. First create the table you want with the structure of the file.

Create table dp_cad_analtyics.A
(
Testcol1 varchar(20)
, Testcol2 varchar(20)
);

2. Then modify the below query accordingly so that the number of question marks are equal to the number of columns

Insert into dp_cad_analytics.A
Values
(?,?);


3. Select the option of "import data" from the "File" menu and execute query. It will ask for the file.




Wednesday, March 22, 2017

SAS tutorial


SAS Tutorial



SAS E-Guide


1. Library Creation

Create a program and write following command:

libname tanvir '/sasdata2/SAS-USERS/taaalam/';





2. Import File into Library

File > Import Data >  Select Local File >  Output SAS data set (  use SAS server and Library name that you created in step 1 )




SAS E-Miner


1. Map Libraries in SAS Miner (Let E-miner knows the library you created in E-Guide)

File > New > Library


2. Bring Files under "Data Source"

Right click "Data Source" > "Create Data Soruce"





Thursday, January 14, 2016

OpenMP Tutorial


Some tutorial:


http://people.math.umass.edu/~johnston/PHI_WG_2014/OpenMPSlides_tamu_sc.pdf

http://www3.nd.edu/~zxu2/acms60212-40212-S12/Lec-11-03.pdf


Scheduling Tutorial

For beginners: https://people.sc.fsu.edu/~jburkardt/c_src/schedule_openmp/schedule_openmp.html

https://www.buffalo.edu/content/www/ccr/support/training-resources/tutorials/advanced-topics--e-g--mpi--gpgpu--openmp--etc--/2011-09---practical-issues-in-openmp--hpc-1-/_jcr_content/par/download/file.res/omp-II-handout-2x2.pdf

Fibonacci Number generation


https://gist.github.com/CarlEkerot/2601195

Monday, October 19, 2015

SVN tutorial




SVN cheat sheet:

http://www.cheatography.com/davechild/cheat-sheets/subversion/

For more useful svn commands, check the following tutorial:
http://openoffice.apache.org/svn-basics.html



Server : Create a  SVN repository


svn co https://svn.apache.org/repos/asf/openoffice/trunk  mySVNrepository

Local machine: Make a repository and connect with server


Just on your local machine, run the following command:

Then, all current testing scripts will be added to your local machine. 

Add and commit your changes to server:


In order to share with us your work, inside your own folder (e.g. ChallengeScripts/Wail ), create any folder, file or scripts and do:
1- svn add folder1
2- svn commit folder1 -m "Comments of your changes"


Monday, October 12, 2015

Linux Tips and Tricks


Collected from CBRC wiki:
Original source: http://dragon.cbrc.kaust.edu.sa/wiki/index.php/Linux_Tips_and_Tricks


Linux Tips and Tricks

Jump to: navigation, search

Contents

Useful commands

General commands

  • find Linux version
$ cat /etc/issue

Directories processing

$ # recursively delete backup files
$ find ./ -name '*~' | xargs rm 
 
$ # Process all files in a directory
$ ls dir_path/*.xml | xargs -n 1 MyProcessingProgram 
 
$ # gzip/guznip directory:
$ tar czvf myfile.tar.gz mydir/*
$ tar xzvf filename.tar.gz (or: tar xzvf filename.tar.gz -C foldername/)
$ gunzip -c foo.tar.gz | tar xvf

Files processing

$ # batch convert dos to unix type text files (e.g. .pl)
$ find . -name "*.pl" -type f -exec dos2unix {} \; 
 
$ # remove blank lines from a text file
$ sed '/^$/d' in.txt > out.txt
 
$ # sort file from shortest to longest line
$ cat myfile.txt | awk '{ print length, $0 }' | sort -n | cut -d" " -f2- > out.txt
$ # or from longest to the shortest
$ cat myfile.txt | awk '{ print length, $0 }' | sort -nr | cut -d" " -f2- > out.txt
 
$ # Count word occurence in a file
$ grep -w 'searchword' filename.txt | tr ' ' '\n' | wc -l

Find/Replace

$ # find words listed in one file (e.g. dictionary) in another (e.g. free text)
$ grep -i -f dictionary.txt text.txt   # ignore case (-i) and print matching lines
$ grep -io -f dictionary.txt text.txt  # ignore case (-i) and print matching pattern
$ while read line; do grep -io -P "\Q$line\E" text.txt; done < dictionary.txt # if there are special chars
 
$ # replace char1 with char2 in a file, e.g comma with tab
$ tr ',' '\t' < in.csv > out.tsv

Processes

$ # print process name, pid and running time for the process "xyz"
$ ps -eo fname,pid,etime | awk '{if ($1=="xyz") print $1,$2,$3}' | more
$ % or show pid for process XYZ that is running more than an hour:
$ ps --no-headers -C XYZ -o pid,etime | awk  '/[0-23]:[0-9][0-9]:[0-9][0-9]/ {print $1}'

File System /Disk Management

Disk space report

$ ncdu
To install ncdu on ubuntu:
$ sudo apt-get install ncdu
To install ncdu on Scientific Linux you need to install Extra Packages for Enterprise Linux (or EPEL) repository. The easiest way to do it is from the Synaptic Package Manager: just search for: Extra Packages for Enterprise Linux. After the EPEL is installed, you can install ncdu:
$ sudo yum install ncdu

Resizing Shared Memory (/dev/shm)

You can use /dev/shm to improve the performance of application software or overall Linux system performance. Use 'df -h' to see partition allocation. To permanently increase size of shared memory, append or modify /dev/shm entry as follows to set size to 8G
$ sudo nano /etc/fstab
# modify the following line by adding new size
none      /dev/shm        tmpfs   defaults,size=8G        0 0
# save and close the file. For the changes to take effect immediately remount /dev/shm:
$ sudo mount -o remount /dev/shm
# verify the same:
$ df -h


Resizing partitions CentOS

  1. display partition information
$ df -kh
Filesystem                      Size  Used  Avail  Use% Mounted on
/dev/mapper/vg_centos-lv_root    50G   3.6G   44G   8%    /
/tmpfs                          1.9G   228K  1.9G   1%   /dev/shm
/dev/sda1                       485M    68M  393M  15%   /boot
/dev/mapper/vg_centos-lv_home   2.0T   200M  1.9T   1%   /home
  1. Drop into the single user terminal to avoid read and writing to the partitions that will be resized
$ sudo telinit 1
if filesystem is used, try booting in a single user-mode (press any key to interrupt boot->letter 'e'->'kernel' option->letter 'e'->type single->letter 'b' to boot)
  1. Decrease /dev/mapper/vg_web-lv_home
$ umount /dev/mapper/vg_centos-lv_home
$ e2fsck -f /dev/mapper/vg_centos-lv_home
$ resize2fs /dev/mapper/vg_centos-lv_home 200G
$ lvresize -L 200G /dev/mapper/vg_centos-lv_home
$ mount /home
$ mount -o remount /home
$ df -kh # check the size
  1. Increase /dev/mapper/vg_centos-lv_root
$ lvresize -l +100%FREE /dev/mapper/vg_centos-lv_root
$ # or, alternatively: lvresize -L 1800G /dev/mapper/vg_centos-lv_root
$ resize2fs /dev/mapper/vg_centos-lv_root
$ df -kh # check the size
$ shutdown -r now

Disk crash recovery

My Linux (kubuntu) suddenly crashed. The screen blinked several times and the keyboard stopped responding. A new reboot failed, it either hung or went to a recovery mode (that did not recover anything). The (almost) only solution in such situations is to run a disk-checking program (e.g. e2fsck) that can fix bad sectors; in this case the suspicious was the Linux partition table, or the Linux boot sector. Therefore, this is what I did:
  1. Get the latest version of Ubuntu on a CD.
  1. Boot from it (on MacBook hold "C" during the reboot; that's how MacBook knows thta you want to boot from a CD). Go to a recovery mode.
  1. Start partition management program parted. I needed to run this program to find out what is the device name of my Linux partition - and parted told me that it was /dev/sda5.
  1. Then I started e2fsck (with the found device /dev/sda5 as a parameter) to check and fix the disk. I answered yes (meaning: yes, fix it please) to all prompts (actually I started it with the -y option).

Mounting WebDAV Share

$ mkdir ~/<my mount point>
$ sudo mount -t davfs http://10.75.106.38:5005 ~/<my mount point>

Installing and configuring WebDAV for non-sudo on Ubuntu

  1. install davfs: sudo apt-get install davfs2
  2. Reconfigure davfs2 to enable non-sudo mount: sudo dpkg-reconfigure davfs2
  3. Edit /etc/davfs2/davfs2.conf to enable automatic credentials by uncommenting the line: secrets ~/.davfs2/secrets
  4. Edit ~/.davfs2/secrets file to add credentials to remote WebDav diectory. Add a line to the end of file in following style: http://
  5. Set the permission: chmod 600 ~/.davfs2/secrets
  6. Add a line to /ect/fstab about the remote WebDav directory: http:// davfs user,noauto,file_mode=600,dir_mode=700 0 1
  7. Add your user to the davfs2 group: sudo vi /etc/group as follows: davfs2:x:134: (134 can be some other gorup number - don't change it)
  8. You can use following commands without being a root user to mount/umount
$ mount <mount point>
$ umount <mount point>
You can also use nautilus to mount/umount the directory.

Thursday, August 13, 2015

GIT tutorial



Tutorial: 

http://gitready.com/
http://blog.osteele.com/posts/2008/05/my-git-workflow/

Basic structure







Image from: From : http://blog.osteele.com/posts/2008/05/my-git-workflow/

Following tips are collected from CBRC git (Thanks to John and Craig)
Original LInk: http://dragon.cbrc.kaust.edu.sa/wiki/index.php/Cbrcgit

Cbrcgit

Jump to: navigation, search

Contents

What is git

http://en.wikipedia.org/wiki/Git_%28software%29
Git is a free distributed revision control, or software source code management, project with an emphasis on being fast. Git was initially designed and developed by Linus Torvalds for Linux kernel development.
Every Git working directory is a full-fledged repository with complete history and full revision tracking capabilities, not dependent on network access or a central server.
Git has a very powerfull command and option set. Some defaults need to be set in new projects to streamline the workflow.

CBRC git server

The central git server for CBRC runs at git.cbrc.kaust.edu, or if you want to use a friendlier GitLab version, have a look at Gitlab wiki.
Also, notice that in communications with the cbrcgit server, everyone should be using the 'git' username. This minimizes the chances of having conflicts with different users creating files in the server.
To access the server, you need to send the CBRC IT support your public ssh key (~/.ssh/id_rsa.pub). If your work computer does not have one, you can generate it:
 $  ssh-keygen -t rsa # with an empty pass phrase
You can find more info about this on FAQ pages.

Quick start

  • On your development computer (provided that git has been installed):
$ cd <my_project_directory>
$ git init
$ git config user.name 'My Name'
$ git config user.email 'My Email Address'
$ git remote add origin ssh://git@git.cbrc.kaust.edu.sa/datapool/git/datagit/My_Project_Name.git/
  • Login to the CBRC git repository and create your project.
$ ssh git@git.cbrc.kaust.edu.sa
$ mkdir /datapool/git/datagit/My_Project_Name.git
$ cd /datapool/git/datagit/My_Project_Name.git
$ git --bare init
$ vim description
$ exit
  • Back on your development machine, add files to the local and remote repository: 
A. Make your local version integrate all from server
$ git pull ssh://git@git.cbrc.kaust.edu.sa/datapool/git/datagit/LDmotif.git
 
B . Now upload your changes to server 

$ git add -A
$ git commit -am 'note about the commit'
$ git push origin master
Please do not forget to edit description file. 3 lines are enough, but you can add more:
   project name
   web location or short description
   contact

Using git

Start by telling git your contact details:
 $ git config --global user.name "Your Name"
 $ git config --global user.email "your.name@kaust.edu.sa"
Since git is a distributed version controlling system, you can use it for any directory. Simply run:
 $ git init
After that you can add subdirectories and files and modify existing ones. Note that git commit command takes note of only those files that have been 'add'ed.
 $ git add .
 $ git commit -a -m 'note about the commit'
Git contains powerful mechanisms to branch, merge versions and study older versions.
Important: If you want to rename a file or directory that is under git, do not use the normal 'mv' command. Git can keep track of the change, if you do:
 $ git mv filename newfilename
Similarly, remove a file from disk and git at the same time:
 $ git rm filename
Committing to a remote git: The latest git does not allow you to commit before you select the push method. The following command sets it globally for all your git repositories:
 $ git config --global push.default matching

See: man pages git, gittutorial


Creating a new remote repository

Once your public ssh key is installed on the git machine (see the procedure in the previous chapter), you can add a new project, e.g. 'myproj', into the git server.
 $ ssh git@git.cbrc.kaust.edu.sa
 $ mkdir /datapool/git/datagit/myproj.git
 $ cd /datapool/git/datagit/myproj.git
 $ git --bare init
 Initialized empty Git repository in /datapool/git/datagit/myproj.git/
 $ exit
Note that this remote directory name needs to end '.git' for the connection to work.

Creating a new local git repository

You can create a local git repository and start using it to keep track of changes a few simple commands:
 $ mkdir myproj
 $ cd myproj/
 $ git --bare init
 # add files and content, e.g.:
 $ echo "text" > readme
 # tell git not to keep track of backup files, e.g. 
 $ echo '*~' > .gitignore
 # tell git to track all files and commit them for the first time
 $ git add .
 $ git commit -a -m 'first commit'
The local git repository sits on your machine and reflects all changes in your files after you commit them. Remember, therefore: the commit command does not update anything outside of your machine (as it does with few others version control systems). If you need to share your project and its files with others, you need to propagate your changes to the CBRC central git server. Read on.

These following commands are needed only when you want to link the local to the CBRC central server:
 $ git remote add origin ssh://git@git.cbrc.kaust.edu.sa/datapool/git/datagit/myproj.git
 $ git push origin master
 

Very important command to let git know using master branch

 
# tell git that by default you are working on the head of the master
 # branch in the remote repository
 $ git config branch.master.remote origin
 $ git config branch.master.merge refs/heads/master

Other CBRC members (who have their public ssh keys installed - see above) can now download their copy of the project with:
 $ git clone ssh://git@git.cbrc.kaust.edu.sa/datapool/git/datagit/myproj.git
From there on, before working on the project, you should first check if someone has update the central repository:
 $ git pull
 # After setting the defaults above, this is same as:
 $ git pull ssh://git@git.cbrc.kaust.edu.sa/datapool/git/datagit/myproj.git master
While working with files, you can check the status any time with:
 $ git status
 $ git log (filename)
 $ git diff filename
 $ git blame filename
and commit any changes in logical sets.
After working with a set of tasks, you should push the changes to the central repository for others to see:
 $ git push origin master

Project repository

The current git repository location is: git@git.cbrc.kaust.edu.sa:/datapool/git/datagit/
CBRC Git repository browser: http://dragon.cbrc.kaust.edu.sa/cbrcwg/

See also

Best Tutorial


http://classic.scottr.org/presentations/git-in-5-minutes/

1. CREATE : ## Make a REMOTE repository in server as git repository

 

  •  On your git  server machine

$ ssh git@git.cbrc.kaust.edu.sa
$ mkdir /datapool/git/datagit/My_Project_Name.git
$ cd /datapool/git/datagit/My_Project_Name.git
$ git --bare init
Initialized empty Git repository in /datapool/git/datagit/myproj.git/
$ vim description $ exit

2. CREATE :  ## Make a LOCAL repository and upload to git repository

  •  On your development computer (provided that git has been installed):
$ cd <my_local_project_directory>
$ git --bare init
$ git config user.name 'My Name'
$ git config user.email 'My Email Address'
$ git remote add origin ssh://git@git.cbrc.kaust.edu.sa/datapool/git/datagit/My_Project_Name.git/


  • add files to the local and remote repository:
$ git add -A
# tell git to track all files and commit them for the first time
$ git add .
$ git commit -am 'note about the commit' $ git push origin master



3. DOWNLOAD : ALREADY EXISTING git repository from server FIRST TIME



 

4. WORK : ALREADY EXISTING git repository


  • From there on, before working on the project, you should first check if someone has update the central repository:

 $ git pull
 # After setting the defaults above, this is same as:
 $ git pull ssh://git@git.cbrc.kaust.edu.sa/datapool/git/datagit/myproj.git master


  • While working with files, you can check the status any time with:

 $ git status
 $ git log (filename)
 $ git diff filename
 $ git blame filename

and commit any changes in logical sets.



  • After working with a set of tasks, you should push the changes to the central repository for others to see:

 $ git push origin master


If git is not sure which branch to use, use master

# tell git that by default you are working on the head of the master
 # branch in the remote repository
 $ git config branch.master.remote origin
 $ git config branch.master.merge refs/heads/master

 

5. Some useful commands

## Make local index from remote git index when you create git local index First time


git clone


## Show the remote git repository

git remote -v

## Download from git repository


git pull


## check difference between local file and local git index
git diff   file.html

## Commit locally
git commit -m "uploading a single file"  help.html
git commit -m "uploading all changes"  .

## Commit in server
git push