Linux Administration

From Hawk Wiki
Jump to: navigation, search


How to Install SVN on Ubuntu


AWS Redhat RHEL Install LAMP

Install VPN PPTPD on RHEL 7

AWS Redhat RHEL Install Ruby and Rails

Ubuntu Install GUI

clean up linux disk space

sudo journalctl --vacuum-time=2d
sudo apt-get autoremove --purge

tar and gzip

Archive a set of files:

tar -cvf tarfile.tar /var/log/syslog /var/log/messages

Archive and compress (gzip) a set of files:

tar -cvzf file.tar.gz /var/log/syslog /var/log/messages

Archive and compress (bzip2) a set of files:

tar -cvjf file.tar.bz2 /var/log/syslog /var/log/messages

Extract a tar file:

tar -xvf file.tar
tar -xvzf file.tar.gz
tar -xvjf file.tar.bz2

Display the content of a tar file:

tar -tvf file.tar
tar -tvzf file.tar.gz
tar -tvjf file.tar.bz2

Replace a file in an existing tar file:

tar -rvf tarfile.tar filetoreplace

Update a file in an existing tar file:

tar -uvf tarfile.tar newfile

Copy all files in one directory to another directory on local host:

cd /etc; tar cf – . | (cd /etc.bak; tar xvpf -)

#parallel zip
# compress
pigz -2 ./abc.vdi
# extract / decompress
pigz -d ./abc.vdi.gz

zip with split and password


zip --encrypt -s 4g -r files


// recursive zip all folder
// with password
// split 4gb
zip --password password -s 4g -r input &
// with store only and 1gb split
zip -Z store -r -s 1g input/

no hangup even if logged out of ssh

nohup bash &> script.log &

DB backup

  1. mysqldump -u root -p [root_password] [tabdaase_name] > dumpfilename.sql
  2. mysqldump -u root -p password sugarcrm > sugarcrm.sql


  1. mysqldump -u root -ptmppassword --databases bugs sugarcrm > bugs_sugarcrm.sql

mysql -u root -p[root_password] [database_name] < dumpfilename.sql

Cron job backup mysql databse every week

# Back up db. Run once a week at midnight on Sunday morning
0 0 * * 0 mysqldump -u root -pPASSWORD DB_NAME | gzip > /tmp/wiki_database_`date +'%Y-%m-%d'`.sql.gz

Search specific text among files

grep -H "textToFind" /var/www/*
grep -H "textToFind" /var/www/sites/*/html/.htaccess

Search file

find /home/david -name 'index*'

SVN cheat sheet
Set up svn in Ubuntu

Apache Rewrite

CodeIgniter Rewrite Rules

<IfModule mod_rewrite.c>
	#Disable file listing
	Options -Indexes
    RewriteEngine On
    RewriteBase /
	#force www
	RewriteCond %{HTTP_HOST} ^hawkguide\.com [NC] 
	RewriteRule ^(.*)$1 [R=301,QSA,L] 
    #Removes access to the system folder by users.
    #Additionally this will allow you to create a System.php controller,
    #previously this would not have been possible.
    #'system' can be replaced if you have renamed your system folder.
    RewriteCond %{REQUEST_URI} ^system.*
    RewriteRule ^(.*)$ /index.php?/$1 [L]
    #When your application folder isn't in the system folder
    #This snippet prevents user access to the application folder
    #Submitted by: Fabdrol
    #Rename 'application' to your applications folder name.
    RewriteCond %{REQUEST_URI} ^application.*
    RewriteRule ^(.*)$ /index.php?/$1 [L]

    #Checks to see if the user is attempting to access a valid file,
    #such as an image or css document, if this isn't true it sends the
    #request to index.php
    RewriteCond %{REQUEST_FILENAME} !-f
    RewriteCond %{REQUEST_FILENAME} !-d
    RewriteRule ^(.*)$ index.php?/$1 [L]

Create User

adduser newuser

Add a existing user to a group usermod -a -G www-data jerry

ImageMagick merge images

montage f*.jpg -tile 1x22 -geometry 1576x930+0+0 tile_.jpg

listen ports

 netstat -a | egrep 'Proto|LISTEN'

Disable prompt when ssh to unknown host

echo -e "StrictHostKeyChecking no\n" >> ~/.ssh/config

Apache server log permission denied

In CentOS
Permission denied: [client xx:xx] AH00132: file permissions deny server access: /var/www/ Run

restorecon -R /var/www/foldername

squid 3.3 on centos

Configure SSL Bump on squid 3.3

List disk usage by directory

du -h <dir> | grep '[0-9\.]\+G'

Ubuntu 16.4 create swap

sudo fallocate -l 1G /swapfile
ls -lh /swapfile
sudo chmod 600 /swapfile
ls -lh /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile
sudo swapon --show

to auto mount

echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab

How to auto run scripts at system start in Cent OS 7

/etc/rc.local or /etc/rc.d/rc.local are no longer executed by default due to systemd-changes. to still use those, you need to make /etc/rc.d/rc.local executable:

chmod +x /etc/rc.d/rc.local
sudo vim /etc/rc.d/rc.local
# then add yout commands to run

Mac fix mongodb max file limit issue / Adjust ulimit

echo kern.maxfiles=65536 | sudo tee -a /etc/sysctl.conf
echo kern.maxfilesperproc=65536 | sudo tee -a /etc/sysctl.conf
sudo sysctl -w kern.maxfiles=65536
sudo sysctl -w kern.maxfilesperproc=65536

Then set ulimit in bash_profile

echo "ulimit -n 65536 65536" >> ~/.bash_profile

Ubuntu set timezone

sudo dpkg-reconfigure tzdata

Mac show git branch name in terminal

Add following code into ~/.bash_profile

parse_git_branch() {
  git branch 2> /dev/null | sed -e '/^[^*]/d' -e 's/* \(.*\)/ (\1)/'

export PS1="\u@\h \W\[\033[32m\]\$(parse_git_branch)\[\033[00m\] $ "

Prepare extra ~/bin folder for shortcut commands

edit ~/.bash_profile add line

export PATH="~/bin/:$PATH"

Auto generate git commit message with one file name

touch ~/bin/autocommit && chmod +x ~/bin/autocommit && vim ~/bin/autocommit 


BRANCH_NAME=$(git symbolic-ref --short HEAD)
COMMIT_MSG=$(LANG=C git -c color.status=false status \
| sed -n -E -e '1,/Changes to be committed:/ d'\
            -e '1,1 d' \
            -e '/git reset/ d' \
            -e '/^Untracked files:/,$ d' \
            -e 's/^\s*//' \
            -e 's/      modified:   //' \
            -e '/./p')
git commit -m "[${BRANCH_NAME}] ${FIRST_FILE}"

usage: just type autocommit and it will generate a commit message containing the current branch name and first file changed

Auto commit and push

git config --global push.default current
echo "autocommit && git push" > ~/bin/autopush
chmod +x ~/bin/autopush

usage: just type autopush

Mount hard drive

#find uuid
sudo blkid
#mount hard drive
sudo mount /dev/sda1 /mnt/ssd1

# auto mount
sudo vim /etc/fstab
UUID=0c18cc09-0028-4cb1-97ef-6222fe477432   /mnt/ssd   ext3    defaults     0        2

Better method

UUID=$(sudo blkid /dev/${DRIVE} -sUUID -ovalue)
echo $UUID
echo "UUID=${UUID} /mnt/${FOLDER} xfs defaults 0 1" | sudo tee -a /etc/fstab
sudo mount --uuid $UUID /mnt/${FOLDER}

fix corrupt hard drive file system

sudo gdisk /dev/sdb

sudo e2fsck /dev/sdb1

using gparted for over 2tb

sudo apt-get install -y gparted xfsprogs


sudo parted /dev/${DRIVE}

mklabel gpt
mkpart primary 0G 100%

sudo mkfs.xfs /dev/${DRIVE}1
# mount hard drive
UUID=$(sudo blkid /dev/${DRIVE}1 -sUUID -ovalue)
echo $UUID
echo "UUID=${UUID} /mnt/${FOLDER} xfs defaults 0 1" | sudo tee -a /etc/fstab
sudo mount --uuid $UUID /mnt/${FOLDER}

Test disk write speed

sudo chown -R $USER:$USER ./
dd if=/dev/zero of=./test1.img bs=1G count=1 oflag=dsync && rm ./test1.img

Splunk installation

download splunk then follow

Change port

sudo vim $SPLUNK_HOME/etc/system/local/web.conf


httpport = 8866

Configure to start on boot time

sudo $SPLUNK_HOME/bin/splunk enable boot-start
sudo service splunk start

List file sizes by directory (depth = 1)

du -ah --max-depth=1 | sort -n

Allow pm2 run via ssh and cron

n=$(which node); \
n=${n%/bin/node}; \
chmod -R 755 $n/bin/*; \
sudo cp -r $n/{bin,lib,share} /usr/local

method 2:

sudo rm -f /usr/local/bin/pm2 && sudo ln -s $(which pm2) /usr/local/bin/pm2

control cpu fan speed


sudo apt-get install fancontrol lm-sensors
sudo sensors-detect
# answer yes to all questions

check temp


edit /etc/default/grub to add acpi_enforce_resources=lax to the GRUB_CMDLINE_LINUX_DEFAULT

then run
sudo update-grub
sudo service kmod start
sudo pwmconfig

view current config


finally start fan control

sudo service fancontrol start

auto start

sudo update-rc.d fancontrol enable

config explaination

MINTEMP: The temperature (°C) at which to SHUT OFF the CPU fan. Efficient CPUs often will not need a fan while idling. Be sure to set this to a temperature that you know is safe. Setting this to 0 is not recommended and may ruin your hardware!

MAXTEMP: The temperature (°C) at which to spin the fan at its MAXIMUM speed. This should be probably be set to perhaps 10 or 20 degrees (°C) below your CPU's critical/shutdown temperature. Setting it closer to MINTEMP will result in higher fan speeds overall.

MINSTOP: The PWM value at which your fan stops spinning. Each fan is a little different. Power tweakers can echo different values (between 0 and 255) to /sys/class/hwmon/hwmon0/device/pwm1 and then watch the CPU fan. When the CPU fan stops, use this value.

MINSTART: The PWM value at which your fan starts to spin again. This is often a higher value than MINSTOP as more voltage is required to overcome inertia.

raw disk backup

if: input file

of: output file (image)

dd if=/dev/mmcblk0 of=/media/usb500/pi_system_torrent.image

search for file name or extension

search for file name or extension find . -iname "*.jpg"

delete files by matches

find . -iname "*.jpg" -delete

rsync file from remote to local

nohup rsync -avh -e "ssh -x -T -o Compression=no"* ./ >~/rsync.log &

The above command turn off compression so it will be faster.

rsync locally

sudo rsync -aqxP /mnt/ssd1/docker/ /nvmex4/docker


grep --color -n -C 1 "search_text" lib/npm.js


grep --color -n -e "search_text[abc|abe]" lib/npm.js

search all files recursive

grep -HR --color "loan" ./

backup whole system

sudo su
dd if=/dev/sdc | bzip2 > /mnt/ssdbackup/ubuntuBackUp201803.bz2
bzcat /mnt/ssdbackup/ubuntuBackUp201803.bz2 | dd of=/dev/sdc

Grow linux partitions on live system

Start socks proxy server

see use docker-compose

Install fusion io drive2 driver in ubuntu 16.04

fusionio driver

Port Forwarding Simple TCP Traffic

use rinetd

Cron start service if stopped


if pgrep "$SERVICE" >/dev/null
    echo "$SERVICE is running"
    echo "$SERVICE stopped"
    /root/frp/frpc -c /root/frp/frpc.ini &

rinetd simple tcp traffic forwarding

This is useful for forwarding traffic to server behind vpn

Auto restart process if exit

echo $RIGID
STATUS=`ps -A | grep gethbsc`
# change channel to slack webhook
if [ ! -z "$STATUS" -a "$STATUS" != " " ] ; then
  echo $STATUS
  echo "Restart service"
  cd ~/bsc && bash ~/bsc/
  curl -X POST -H 'Content-type: application/json' --data "{\"text\":\"server=${RIGID} node down\"}" $CHANNEL

RDMA NDS system via infiniband

this only works in centOS8

on server

change nfs disk array server ib ip to

change client server ib ip to

vim /etc/exports
# edit and add

sudo exportfs -a
sudo modprobe xprtrdma
sudo modprobe ib_ipoib
echo "rdma 20049" | sudo tee -a /proc/fs/nfsd/portlist

on client

sudo modprobe xprtrdma
sudo modprobe ib_ipoib
sudo mount -o rdma,port=20049 /mnt/nfs

LVM disk merged drive

follow instructions at

sudo apt-get -y install lvm2
sudo lvmdiskscan
sudo pvcreate /dev/${DRIVE}
sudo vgcreate nvmeg /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 /dev/nvme5n1 /dev/nvme0n1 /dev/nvme1n1
sudo vgextend nvme4 /dev/${DRIVE}
# create logical volume to take 100% space
sudo lvcreate -l 100%FREE -n main nvmeg
sudo fdisk -l
# Disk /dev/mapper/nvmeg-main: 9.1 TiB, 10001997365248 bytes, 19535151104 sectors
sudo lvmdiskscan
sudo fdisk -l
# extend volume and grow
sudo lvextend -l +100%FREE /dev/mapper/nvme4-nvme4t
sudo xfs_growfs -d /mnt/nvme1

setup frp

Setup FRP

setup apache reverse proxy

Setup apache2 reverse proxy



Github Action Runner

Github Action Runner

Squid https proxy

Squid proxy docker