shell script: array

# space separated
alphabets=("a" "b" "c" "d")

#To access any element by index
${alphabets[1]}

# to loop
for f in "${alphabets[@]}"

python: tips and tricks

1.

m = {'a': 1, 'b': 2}

m[ 'c' ] will throw error. Instead use m.get( 'c', 'default' )

2.

'foo'.index( 'bar' ) throws Exception

'foo'.find( 'bar' ) returns -1

shell script: file information

file=/foo/bar/myfile.txt

File modify date and time

file_modify_date=$(stat -c %y $file | cut -d" " -f1)

file_modify_time="$(stat -c %y $file | cut -d" " -f2 | cut -d":" -f1,2 | tr -d ":")

File directory, name and extension

filename="${file##*/}"  # get filename

extn="${filename##*.}"

filename="${filename%.*}" # removing extension

if [[ $filename == $extn ]]; then
    extn=""
else
    extn=.$extn
fi

dirname="${file%/*}" # get dirname

basename is a good command too

References:

man stat
man basename

http://www.thegeekstuff.com/2010/07/bash-string-manipulation/

nginx: try_files, proxy_pass and rewrite

Situation: serves files locally if not found get it from a proxy server

location ^~ /site {

  try_files $uri $uri.html @proxy;

}

location @proxy {
  resolver 8.8.8.8;   proxy_pass http://mysite.com/; 
}

The prefix "@" specifies a named location. Such locations are not used during normal processing of requests, they are intended only to process internally redirected requests (see error_page, try_files).

To avoid any non-named location to be not processed by external requests try internal.

While passing request nginx replaces URI part which corresponds to location with one indicated in proxy_pass directive. So

location /mymatch {
  proxy_pass http://10.11.12.13;
}

e.g. For the request http://host/mymatch/foo/bar the request to proxy would be http://10.11.12.13/foo/bar stripping the /mymatch from the requested URI.

In the above case there is no replacement because we are named location.

Since domain name (mysite.com) is used and not IP it is required to set the resolved. I have set Goolge's resolver IP.

Difference between try_files and rewrite

try_files sets the internal URI pointer (does not change the URI) and only the last parameter causes an internal redirect.

Since try_files sets the internal pointer a leading slash might be required.

try_files $uri /index.html;

So it will look for <root>/index.html. Otherwise <root>index.html

rewrite directive changes URI in accordance with the regular expression and the replacement string. Directives are carried out in order of appearance in the configuration file.

Flags make it possible to end the execution of rewrite directives.

References:

http://wiki.nginx.org/HttpCoreModule#try_files

http://wiki.nginx.org/HttpCoreModule#location

http://wiki.nginx.org/HttpCoreModule#internal

http://wiki.nginx.org/HttpRewriteModule#rewrite

http://wiki.nginx.org/HttpProxyModule#proxy_pass

python: import a custom file

suppose you wrote a python file with common functions, say util.py and you want use it in another of your python files

If they in the same directory the below should work

import util

If they in different directories the below will not work. To make it work add another entry

sys.path.append('<path/to/the/folder/of/util.py>')
import util

This fixes it. This is the simplest way to me but there must be more.I will explore it further and update.

python: import function

This is interesting to me. We can import only a particular function of a module inside python

Say there is a function boto.ec2.regions()

So I would write

>>> import boto.ec2
or
>>> from boto import ec2

>>> boto.ec2.regions()
or
>>> ec2.regions()

But we can also write

>>> from boto.ec2 import regions

>>> regions()

I had this scenario. Inside my python file I had import my util file. In the util file I had the above import.

from boto.ec2 import regions

So in my python file I could write

import util

util.regions()

Awesome right??

linux: logging top processes by cpu or memory

I wanted to log the processes consuming cpu and memory

First thought was to use top in the batch mode (-b).
-c to show the process name.
-n 1 to capture for 1, to run for 1 frame
   
$ top -b -c -n 1 > top_$(date +"%Y-%m-%d_%H%M").log

But I was not getting the complete process/command name.

Later I  used the below technique and it was quite helpful

#!/bin/bash

log_file=top_$(date +"%Y-%m-%d_%H%M").log
echo 'user    %cpu    %mem    pid    elapsed time    command' > $log_file
echo ' =========== CPU ===========' > $log_file
ps -eo user,pcpu,pmem,pid,etime,command | sort -rn -k2 | head -11 > $log_file
echo ' ========== MEMORY ==========' >> $log_file
ps -eo user,pcpu,pmem,pid,etime,command | sort -rn -k3 | head -11 >> $log_file

Observation:

1. 'ps aux' is a wonderful command

2. ps command itself has --sort <fieldname>, but there is no reverse sorting

bash script: arithmetic comparison

Arithmetic in BASH is integer math only. You can't do floating point math in Bash; if you need that capability, see Bash FAQ #22.

Remember few tricks

1. use [[ .. ]] for strings and files

2. use (( .. )) for numbers

To compare arithmetic numbers use bc function

$(echo "1.4 < 2.5" | bc)

> and < is for ASCII comparison and so 100 > 75 is false

-gt, -lt is only integer comparison.

This works for me

    if (( $(echo "$mem_util > 75" | bc) == 1 ))
    then
        ...
    fi

I am wondering why there was no floating point support??

References:

http://mywiki.wooledge.org/ArithmeticExpression

http://mywiki.wooledge.org/BashFAQ/031

linux: complete command

I have my .ssh/config with a lot of connection names. What would be better than to have auto complete with ssh <TAB> <TAB>

Added this line to my bash profile and I was good to go.

complete -W "$(awk '/^\s*Host\s*/ { sub(/^\s*Host /, ""); print; }' ~/.ssh/config)" ssh

man complete

There is a lot that human can do, for everything else there is [shell] script :)

rsync - folder to folder

rsync is wonder utility to sync files.

I had a situation and I was sure the developer must have thought of it. And I was right :) He had solved it with a trailing /

rsync [option] /home/foo/bar dest:/home/foo

will sync bar directory of source into destinations /home/foo.

Note: this requires access to /home/foo

Suppose you have /bar under root / of the destination and you want to sync the same source. Now this will get tricky as it is rare to have open access to /. The solution is

rsync [option] /home/foo/bar/ dest:/bar

The trailing / in the source syncs everything under bar and fits the solution to the problem.

linux: RANDOM is so simple

I was impressed with $RANDOM in linux. Simply use $RANDOM to get a random number.

Shell script thingy

Did you know: Hypen, -, is not allowed in script variable names!

a="A"
b="B"

echo "$a_$b"

will print B??!!

it will treat $a_ as the first variable name.

To fix it

echo "${a}_${b}"

Running redis as background process

I am big fan of redis and there are many reasons for it! The first being its simplicity to set up and go live.

Copy the redis.conf from the source and make necessary changes. There are very less configuration keys and they are supported with clean in-line documentation.

To start:
redis-server <location to the redis.conf>

To Stop:
killall -w redis-server

Note:

You might have observed in the conf file. Let me put it here. To run redis as daemon you should configure these two keys

daemonize yes

pidfile /home/<user>/pids/redis.pid

For everything else redis.io

nginx $request_uri vs $uri

$request_uri is what comes in the request from the client and $uri is what is processed as part of the request fulfillment.

www.blogger.com/blogger

$request_uri = blogger

Suppose if we have rewrite rule as

rewrite ^ $request_uri.html

$uri = blogger.html

HTH

Redis - vertical scaling (migrating to large machine)

Redis is an open source, advanced key-value store. It is often referred to as a data structure server since keys can contain strings, hashes, lists, sets and sorted sets. (http://redis.io)

I am talking w.r.t v 2.4.14

The latest configuration suggests all the data should be in memory and we should not use Virtual Memory. So if your product is growing one day you would need to vertically scale redis (i.e. move to a large memory machine).

There are tricks for Memory Optimization. Also one can prefer sharding (keeping data divided in different redis servers based on key group). Handling expiry time wisely is another important consideration.

We followed the following approach to migrate with 0 down time!

We had two small machines (master - slave).

Cerated two large machines and set up master - slave.

With latest version of redis we can have a chain of master slaves

Master - slave1 - slave2 ..

Slave2 listens to slave1.

Put the new master as slave to old slave and let it sync. Verified the total number of keys. (Note the sync starts with some delay)

After that was done switched the new machine as the master redis in application configuration files and we were good to go. Remove the new master being slave of the old slave. Thanks to redis for all this support!

So writing to a slave is allowed :)

In case it your redis does not support slave to a slave then we can do something like this

run 'SLAVEOF NO ONE' in the old slave and on the new master run 'SLAVEOF <old master> <port>'
Once the sync is complete switch to new master and stop it being slave of the old master. Add a new slave to the new master.

HTH

Please take care of Lucene :)

My company's Order Search, which is based on Lucene, is like a miracle to me. I was wondering what does Lucene stand for? After the dose of Sentinel, Lucene was little off the hook :)

http://wiki.apache.org/lucene-java/LuceneFAQ

Where does the name Lucene come from?

Lucene is Doug Cutting's wife's middle name, and her maternal grandmother's first name.

--

Doug Cutting is the developer of Lucene. Just wondering how different even great minds think to come up with a name or is it that Java people are more Loving ;)

Redis sentinel

sen·ti·nel/ˈsentn-əl/

Noun:	A soldier or guard whose job is to stand and keep watch.
Verb:	Station a soldier or guard by (a place) to keep watch.
Synonyms:	noun.  sentry - guard - watch - watchman - guardsman - keeper verb.  guard - watch

http://redis.io/topics/sentinel-spec

running an jnlp - unix like system

you need javaws (JAVA WebStart) to run jnlp file. Installing Java Jre will install javaws

$ which javaws

$ [javaws path] [jnlp file path]

Javascript: Sorting complex object (e.g. with S, M, L, XL sizes for clothing)

        var sorted_size_chart = ["XS", "S", "M", "L", "XL", "XXL", "STD"];
                
        // sort the array
        data.sort(function(a, b) {
            var a_size_index = a["size"];
            if (a["size"] && $.inArray(a["size"].toUpperCase(), sorted_size_chart) > -1) {
                a_size_index = $.inArray(a["size"].toUpperCase(), sorted_size_chart);
            }

            var b_size_index = b["size"];
            if (b["size"] && $.inArray(b["size"].toUpperCase(), sorted_size_chart) > -1) {
                b_size_index = $.inArray(b["size"].toUpperCase(), sorted_size_chart);
            }
            return a_size_index - b_size_index;
        });