shell script: array

# space separated
alphabets=("a" "b" "c" "d")

#To access any element by index
${alphabets[1]}

# to loop
for f in "${alphabets[@]}"

python: tips and tricks

1.

m = {'a': 1, 'b': 2}

m[ 'c' ] will throw error. Instead use m.get( 'c', 'default' )

2.

'foo'.index( 'bar' ) throws Exception

'foo'.find( 'bar' ) returns -1

shell script: file information

file=/foo/bar/myfile.txt

File modify date and time

file_modify_date=$(stat -c %y $file | cut -d" " -f1)

file_modify_time="$(stat -c %y $file | cut -d" " -f2 | cut -d":" -f1,2 | tr -d ":")

File directory, name and extension

filename="${file##*/}"  # get filename

extn="${filename##*.}"

filename="${filename%.*}" # removing extension

if [[ $filename == $extn ]]; then
    extn=""
else
    extn=.$extn
fi

dirname="${file%/*}" # get dirname

basename is a good command too

References:

man stat
man basename

http://www.thegeekstuff.com/2010/07/bash-string-manipulation/

nginx: try_files, proxy_pass and rewrite

Situation: serves files locally if not found get it from a proxy server

location ^~ /site {

  try_files $uri $uri.html @proxy;

}

location @proxy {
  resolver 8.8.8.8;   proxy_pass http://mysite.com/; 
}

The prefix "@" specifies a named location. Such locations are not used during normal processing of requests, they are intended only to process internally redirected requests (see error_page, try_files).

To avoid any non-named location to be not processed by external requests try internal.

While passing request nginx replaces URI part which corresponds to location with one indicated in proxy_pass directive. So

location /mymatch {
  proxy_pass http://10.11.12.13;
}

e.g. For the request http://host/mymatch/foo/bar the request to proxy would be http://10.11.12.13/foo/bar stripping the /mymatch from the requested URI.

In the above case there is no replacement because we are named location.

Since domain name (mysite.com) is used and not IP it is required to set the resolved. I have set Goolge's resolver IP.

Difference between try_files and rewrite

try_files sets the internal URI pointer (does not change the URI) and only the last parameter causes an internal redirect.

Since try_files sets the internal pointer a leading slash might be required.

try_files $uri /index.html;

So it will look for <root>/index.html. Otherwise <root>index.html

rewrite directive changes URI in accordance with the regular expression and the replacement string. Directives are carried out in order of appearance in the configuration file.

Flags make it possible to end the execution of rewrite directives.

References:

http://wiki.nginx.org/HttpCoreModule#try_files

http://wiki.nginx.org/HttpCoreModule#location

http://wiki.nginx.org/HttpCoreModule#internal

http://wiki.nginx.org/HttpRewriteModule#rewrite

http://wiki.nginx.org/HttpProxyModule#proxy_pass

python: import a custom file

suppose you wrote a python file with common functions, say util.py and you want use it in another of your python files

If they in the same directory the below should work

import util

If they in different directories the below will not work. To make it work add another entry

sys.path.append('<path/to/the/folder/of/util.py>')
import util

This fixes it. This is the simplest way to me but there must be more.I will explore it further and update.

python: import function

This is interesting to me. We can import only a particular function of a module inside python

Say there is a function boto.ec2.regions()

So I would write

>>> import boto.ec2
or
>>> from boto import ec2

>>> boto.ec2.regions()
or
>>> ec2.regions()

But we can also write

>>> from boto.ec2 import regions

>>> regions()

I had this scenario. Inside my python file I had import my util file. In the util file I had the above import.

from boto.ec2 import regions

So in my python file I could write

import util

util.regions()

Awesome right??

linux: logging top processes by cpu or memory

I wanted to log the processes consuming cpu and memory

First thought was to use top in the batch mode (-b).
-c to show the process name.
-n 1 to capture for 1, to run for 1 frame
   
$ top -b -c -n 1 > top_$(date +"%Y-%m-%d_%H%M").log

But I was not getting the complete process/command name.

Later I  used the below technique and it was quite helpful

#!/bin/bash

log_file=top_$(date +"%Y-%m-%d_%H%M").log
echo 'user    %cpu    %mem    pid    elapsed time    command' > $log_file
echo ' =========== CPU ===========' > $log_file
ps -eo user,pcpu,pmem,pid,etime,command | sort -rn -k2 | head -11 > $log_file
echo ' ========== MEMORY ==========' >> $log_file
ps -eo user,pcpu,pmem,pid,etime,command | sort -rn -k3 | head -11 >> $log_file

Observation:

1. 'ps aux' is a wonderful command

2. ps command itself has --sort <fieldname>, but there is no reverse sorting