Wednesday, October 31, 2012

nginx: try_files, proxy_pass and rewrite

Situation: serves files locally if not found get it from a proxy server

location ^~ /site {

  try_files $uri $uri.html @proxy;


location @proxy {
  resolver;   proxy_pass; 

The prefix "@" specifies a named location. Such locations are not used during normal processing of requests, they are intended only to process internally redirected requests (see error_page, try_files).

To avoid any non-named location to be not processed by external requests try internal.

While passing request nginx replaces URI part which corresponds to location with one indicated in proxy_pass directive. So

location /mymatch {

e.g. For the request http://host/mymatch/foo/bar the request to proxy would be stripping the /mymatch from the requested URI.

In the above case there is no replacement because we are named location.

Since domain name ( is used and not IP it is required to set the resolved. I have set Goolge's resolver IP.

Difference between try_files and rewrite

try_files sets the internal URI pointer (does not change the URI) and only the last parameter causes an internal redirect.

Since try_files sets the internal pointer a leading slash might be required.

try_files $uri /index.html;

So it will look for <root>/index.html. Otherwise <root>index.html

rewrite directive changes URI in accordance with the regular expression and the replacement string. Directives are carried out in order of appearance in the configuration file.

Flags make it possible to end the execution of rewrite directives.


Tuesday, October 30, 2012

python: import a custom file

suppose you wrote a python file with common functions, say and you want use it in another of your python files

If they in the same directory the below should work

import util

If they in different directories the below will not work. To make it work add another entry

import util

This fixes it. This is the simplest way to me but there must be more.I will explore it further and update.

python: import function

This is interesting to me. We can import only a particular function of a module inside python

Say there is a function boto.ec2.regions()

So I would write

>>> import boto.ec2
>>> from boto import ec2

>>> boto.ec2.regions()
>>> ec2.regions()

But we can also write

>>> from boto.ec2 import regions

>>> regions()

I had this scenario. Inside my python file I had import my util file. In the util file I had the above import.

from boto.ec2 import regions

So in my python file I could write

import util


Awesome right??

Saturday, October 27, 2012

linux: logging top processes by cpu or memory

I wanted to log the processes consuming cpu and memory

First thought was to use top in the batch mode (-b).
-c to show the process name.
-n 1 to capture for 1, to run for 1 frame
$ top -b -c -n 1 > top_$(date +"%Y-%m-%d_%H%M").log

But I was not getting the complete process/command name.

Later I  used the below technique and it was quite helpful


log_file=top_$(date +"%Y-%m-%d_%H%M").log
echo 'user    %cpu    %mem    pid    elapsed time    command' > $log_file
echo ' =========== CPU ===========' > $log_file
ps -eo user,pcpu,pmem,pid,etime,command | sort -rn -k2 | head -11 > $log_file
echo ' ========== MEMORY ==========' >> $log_file
ps -eo user,pcpu,pmem,pid,etime,command | sort -rn -k3 | head -11 >> $log_file


1. 'ps aux' is a wonderful command

2. ps command itself has --sort <fieldname>, but there is no reverse sorting

bash script: arithmetic comparison

Arithmetic in BASH is integer math only. You can't do floating point math in Bash; if you need that capability, see Bash FAQ #22.

Remember few tricks

1. use [[ .. ]] for strings and files

2. use (( .. )) for numbers

To compare arithmetic numbers use bc function

$(echo "1.4 < 2.5" | bc)

> and < is for ASCII comparison and so 100 > 75 is false

-gt, -lt is only integer comparison.

This works for me

    if (( $(echo "$mem_util > 75" | bc) == 1 ))

I am wondering why there was no floating point support??


Thursday, October 25, 2012

linux: complete command

I have my .ssh/config with a lot of connection names. What would be better than to have auto complete with ssh <TAB> <TAB>

Added this line to my bash profile and I was good to go.

complete -W "$(awk '/^\s*Host\s*/ { sub(/^\s*Host /, ""); print; }' ~/.ssh/config)" ssh
man complete

There is a lot that human can do, for everything else there is [shell] script :)

Thursday, October 18, 2012

rsync - folder to folder

rsync is wonder utility to sync files.

I had a situation and I was sure the developer must have thought of it. And I was right :) He had solved it with a trailing /

rsync [option] /home/foo/bar dest:/home/foo

will sync bar directory of source into destinations /home/foo.

Note: this requires access to /home/foo

Suppose you have /bar under root / of the destination and you want to sync the same source. Now this will get tricky as it is rare to have open access to /. The solution is

rsync [option] /home/foo/bar/ dest:/bar

The trailing / in the source syncs everything under bar and fits the solution to the problem.

linux: RANDOM is so simple

I was impressed with $RANDOM in linux. Simply use $RANDOM to get a random number.

Shell script thingy

Did you know: Hypen, -, is not allowed in script variable names!


echo "$a_$b"

will print B??!!

it will treat $a_ as the first variable name.

To fix it

echo "${a}_${b}"