Networking, Python, BigData and Linux

What happens during Linux boot process ? Read here: https://www.thegeekstuff.com/2011/02/linux-boot-process What happens during the Login by  Non-privileged user post reboot? Init process (pid=1, uid =0 ) spwans the Login process  (pid-x,uid=0)   Fork() system call   Exec system call Login Process  Fork() System Call setuid for the User (pid x, uid=0) Exec System call  Shell Uid >0, PID x Execute ~/.bashrc ; exports all the enviornment variables etc/

Let's consider the linux machine is having two interfaces eth0 and eth1. eth1 is connected to the internet and eth0 is connected to the private network. How can i make internet accessible via the clients/servers residing in the private networks? in this article i'll give the steps to achieve the same in Ubuntu. Step1: Enable the ipv4 traffic forwarding  echo 1 > /proc/sys/net/ipv4/ip_forward To enable permanently, add following line in /etc/sysctl.conf net.ipv4.ip_forward = 1 Step2:  Add following rules to IP Tables sudo iptables -A FORWARD -o eth1 -i eth0 -m conntrack --ctstate -NEW -j ACCEPT sudo iptables -t nat -F POSTROUTING sudo iptables -t nat -A POSTROUTING -s [ Private network eth0 belong to ex:10.2.0.0/16 ]  -o eth1 -j MASQUERADE  Step3: Save IP Tables  sudo iptables-save | sudo tee /etc/iptables.sav sudo iptables-restore < /etc/iptables.sav Step4: Update the DNS server as 8.8.8.8 in private network nodes Add 'nameserver 8...

tcpdump -G 300 -W 10 -w 'BR_tcpdump_%Y%m%d-%H%M%S.pcap' -i eth0:0 'port 2055'  -Z admin ; tar -cvzf tcpdump.tar.gz BR_tcpdump_* cd /tmp/;tcpdump -G 300 -W 12 -w 'BR_tcpdump_%Y%m%d-%H%M%S.pcap' -i eth0:0 'port 2055' -Z admin -z gzip & disown

1. Check the Used memory with free command: [admin@BIZ-UPGRADE-NN1 ~]# free -g              total       used       free     shared    buffers     cached Mem:            44         17         26          0          0          8 -/+ buffers/cache:          8         35 Swap:           20          0         19 2.  Check the Top 20 Processes consuming memory:  ps -eo pmem,pcpu,vsize,pid,cmd | sort -k 1 -nr | head -20  => Sort by Resident  memory & ps -eo pmem,pcpu,vsize,pid,cmd | sort -k 3 -nr | head -20  => Sort by Virtual memory

How to check the utilisation of each cores: mpstat -P ALL 1 & lscpu or cat /proc/cpuinfo  ps -aeF  gives the details of which core ID is being used for a process. top -H -p => gives all the details of threads of all the processes. lsof -t => Gives the PID of a file name. netstat -a => all ; listening + established; -n => suppress host/port name resolution; -t => only tcp ; -p => program name; -r routing; -i interface   Find files smaller than 2K: find -type f -mindepth 2 -size -1c  Find files which are not older than 2 days: find -type f -mtime -2  stat zzz (Gives all the statistics of a file); stat %i ; %F; %G; %U; %b ; atime => when it was last accessed , mtime => when the file was last modified, ctime when the file was  last changed (changed means file attributes were changed)  Unix File system: A directory has name ...

Well, working with IPV6 addresses is little bit tricky unlike IPV4 addresses. IPV6 addresses are 128 bit-length with 8 octets each of 16 bits. In this post,  i'll explain my working experience with IPV6 addresses. While working with IPV6, i found that sipcalc command is very useful on deciding which ipv6 address to use and what is the network corresponding to the ipv6 address. Hence, lets install sipcalc in Linux using bellow steps: Step1: Download the sipcalc rpm wget ftp://ftp.univie.ac.at/systems/linux/fedora/epel/6/x86_64/sipcalc-1.1.6-4.el6.x86_64.rpm Step2: Install the rpm rpm -ivh sipcalc-1.1.6-4.el6.x86_64.rpm Lets suppose i need at most 2 IPV6 addresses in a subnet. I have 128-2 => 126 network bits & 2 host bits. That means 2^2 hosts per subnet of length /126. Example: 2001:db8:1:1:0:0:0:0/126 is a subnet having 4 possible ipv6 addresses as: 2001:db8:1:1:0:0:0:0 => network address 2001:db8:1:1:0:0:0:1 & 2001:db8:1:1:0:0:0:2 => Usable host...

Suppose you want to mount a remote directory and use it as if directory is present on the same server, here are the steps: A. Execute these steps on the remote server: /etc/init.d/nfs start under /etc/exports add this line :  [Directory to allow access on this server] [Server IP to allow access]     (rw,sync) run exportfs -r B. Execute these steps on the Local server: mount -o nolock :  [Remote server IP] :[Remote directory]    [Mount Directory]

NTP/SNTP is a protocol which is used for syncing the time between multiple nodes in the network. Here are the steps to Configure NTP client in Ubuntu: update /etc/ntp.conf with your server. server ntpdate /etc/init.d/ntpd restart Update the firewalls rule to accept NTP traffic: iptables -A OUTPUT -p udp --dport 123 -j ACCEPT iptables -A INPUT -p udp --sport 123 -j ACCEPT Check if the server is picking up right NTP server:  sudo ntpq -np    remote           refid      st t when poll reach   delay   offset  jitter ============================================================================== *81.76.22.20   8.78.8.80      2 u  116  128  377    0.540   -9.006   5.55 Star indicates the server is picking up right server.

1. How to print the line which matches /regex/?    awk '/regex/{print $0}' 2. How to print the line immediately above the line containing /regex/? awk '{x=y"\n"$0;y=$0};x~/regex/{gsub(/\n.*/,"",x);print x}' 3. How to print the line immediately after the line containing /regex/? awk ' {x=NR+1};NR==x{print $0}' 4. How to get the uniq lines from a file & the count the line is repeated? awk '{arr[$0]++;};END { for (i in arr) { print i "\t" arr[i];} }' 5. Precede each lines with its line number. awk ‘{print NR “\t” $0}’ 6. Precede line numbers only on the non-blank lines. awk 'NF{++c; print  c "t" $0};NF==0{print $0}' 7. Count the Lines. awk ‘END{print NR}’ 8. Printing words of a line in reverse order awk '{for (i=NF;i>0;i--){ printf  "%s%s",$i,FS  } print "" }' 9. Printing lines of a file in Reverse order. awk ‘{ arr[NR]=$0...

Following are the some advanced UNIX commands which i found very useful. DNS (Domain Name Service) host string Perform forward or reverse lookup on string dig @nameserver string Lookup string's host info from nameserver System Hardware Information grep -i memtotal /proc/meminfo Find total amount of RAM in the system dmidecode -q Display DMI/SMBIOS information cat /proc/cpuinfo Display CPU information egrep '(vmx|svm)' /proc/cpuinfo See if a processor supports hardware virtualization lspci -tv Display PCI information lsusb -tv Display USB information hdparm -i /dev/sda Display disk information for sda Quick HTML Editing Site Wide find . -type f -print -exec sed -i -e 's|X|Y|g' {} \; Replace all X's with Y's in all files this directory and below. perl -pi -w -e 's/X/Y/g;' *.txt Replace all X's with Y's in all files ending with *.txt find -name '*' | xargs grep 'string' Print filename and all lines containing ...

Well, i was asked a question(which was last one)  write a shell script which renames all *.txt files in a directory to *.sh ? Immediately, mv command came into my mind as it does in many peoples' mind.Then i wrote like: find . -name "*.txt" -exec mv *.txt *.sh {}\;  however this was wrong. He told me to explain how the above would work and i realized this was not correct. then i tried something else like: find . -name "*.txt" -exec ls {}\; | perl -e '`rename *.txt *.sh`' I wasn't pretty much sure that this would work.He told thats fine if i could  try without perl. I asked for some hints and he reminded me of xargs & sh.Then i tried like: find . -name "*.txt" -exec ls {}\;| xargs -n1 sh -c 'mv $1 `basename $1 .txt`.sh' - and then i got selected in the unix rounds... :-) :-)  xargs is really a great unix command like sed & awk.It is quite often used for parallel processing.Never miss this if you  like t...