Saturday, May 14, 2016

What do the // in a URI stand for?

The url spec basically has a scheme id followed by a colon, followed by a scheme specific part (RFC3986)

As you can see, the mailto scheme basically has just the email address, so something like is a complete URL.

For the http scheme, the slashes have a historical reason. In the old days (pre-1990s) many networks used double slash to indicate that the first token is a hostname.

Therefore /someplace/file.txt would mean file.txt under the /someplace directory on the current file system but //someplace/file.txt would mean /file.txt at server "someplace"

Therefore the full translation of would be:

http:Use the http protocol
//example.comConnect to
/item.htmlAsk the server for item.html

(I've omitted port and fragment for simplicity)

The // is technically unnecessary to distinguish from local server because http requires that you need to specify a host to connect to.

When the file uri was originally implemented, it had the following forms and meanings

file:./file.txtLoad file from current directory (Browser's working directory)
file:/directory/file.txtLoad file from root of current host
file://mount/file.txtAttempts to load file.txt from mount point. If mount point is invalid, attempts to load file.txt from root directory
file:///directory/file.txtLoads file.txt from root directory. This is the same as above with mount token missing.

In modern browsers, everything upto the first slash is pretty much ignored. The mountpoint still works but often the browser ignores it and treats file://mount/file.txt as file:///file.txt

Friday, May 25, 2012

Simple way to use getopts

Most getopts examples show complex case switches in a while loop and complicated logic to set defaults.

In general, getopts is very simple to use:

# example options -t and -a with args and -x with no args
while getopts xt:a: arg
    # remove punctuations to prevent shell injection
    V=$(echo $OPTARG | perl -lpe 's/[^a-zA-Z0-9_. ]/_/gs')
    eval "$arg='${V:-true}'"
    if [ -n "$OPTARG" ]; then C=$[C+2] ; else  C=$[C+1] ; fi
shift $C
echo $@  # args after params

# in case option was not passed set defaults

#err check/usage
if [ -z "$a" ] 
    echo Usage: $0 [-x] [-t text] -a address; exit 1; 

# use it!
echo T is $t and A is $a and X is $x

Wednesday, April 25, 2012

Installing FreeNAS using virt-install

A simple way to install FreeNAS on a VM is to use virt-install. I had to add a sata controller since I have more than 4 drives:

virt-install -nFreeNAS -r 6144 --vcpus=2 --autostart --os-variant=freebsd7 --accelerate --network bridge=br0,model=rtl8139 --disk path=/mnt/img/FreeNAS.img --disk path=/dev/sda2,bus=sata --disk path=/dev/sdb2,bus=sata --disk path=/dev/sdb1,bus=sata -c /mnt/img/FreeNAS8.2.0B3.iso --vnc --controller sata

Note: I ended up not using the FreeNAS on a VM because attaching the host partitions to the VM ended up giving terrible performance on disk IO. This was true for both UFS and ZFS on fully or paravirtualized drivers.

Abandoning this approach and having the host manage its own disks greatly improved performance and am very happy with it. For every compartmentalized task such as media servers I'm still using a VM with the filesystem from the host mounted over NFS.

Saturday, April 21, 2012

Running FreeNAS on a virtual machine

I prefer to have my physical servers bare, containing only the barest changes required to run virtual machines. Thus, when I need to migrate to a different hardware, its as simple as moving the VMs.

Usually, I only have the base server packages running with the packages needed to act as a virtual machine host. On linux systems, this translates to libvirt and related utilities. In addition, I also usually install hardware monitoring packages to check for overheating, etc.

The only config change I make are usually to make the network interface bridged:

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
auto br0
iface br0 inet dhcp
        bridge_ports eth0
        bridge_fd 9
        bridge_hello 2
        bridge_maxage 12
        bridge_stp off

So when I need to install FreeNAS, I wanted it to directly interact with the underlying disks yet the ability to simply ship my VM (and disks) to another server if desired. So I decided to install FreeNAS on a VM using qemu.

Here is how I finally got it working:

Your root filesystem where FreeNAS is installed should be ide. FreeNAS refuses to install on other types. You can get it to work on scsi by adding a boot option, but using ide is a lot simpler.

Once you install the system, you need to enable virtio for paravirtualized devices.

mount -uw /wget virtio-kmod-0.228301.tbz > /tmp/out

/tmp/out contains instructions to enable virtio drivers. Essentially, change /boot/loader.conf to load virtio drivers and /etc/rc.conf and /etc/fstab to point to the virtualized device names.

You can then go to the admin console and setup the volumes. My ZIL partitions were only 2GB and it failed to create a zpool with zil. So I had to create a volume without a ZIL, ssh into the box and add it manually using:

zpool add log zfs_pool_name mirror vtbd1 vtbd2 

(Use actual names of the devices from the FreeNAS volume manager in place of vtbd1,vtbd2)

You can now use the admin GUI to perform the rest of the setup.

Sunday, February 26, 2012

Flaw in yahoo mail allows spam messages to stay at the top of Inbox

Its sunday afternoon and imagine my surprise when I see an email sent from Monday. First, neutrinos time travel and now email!

Turns out yahoo mail blindly trusts the Date header in the message and uses it to sort and present the email. So as long as its Monday in any part of the world, (i.e. international date line) an email can be sent with that date in the header and is accepted as valid. This means that a spam message can be sent to the other side of the international date line with 11:59 PM in the date header and the email will stay on top of the inbox for upto 24 hours (if you were just on the other side of the dateline)

Note that the actual email could be sent from anywhere. This particular email was sent from Massachusetts.

Note the Received-SPF header. This is a great way to pass spam controls since yahoo (and many other mail providers) trust that such a header is automatic guarantee of legitimate email. This of course, is nonsense. A spammer can register a domain and add IP addresses of the nodes he controls as a designated sender and until the domain is banned, he can bypass spam easily.

Return-Path: <>
Received-SPF: pass (domain of designates as permitted sender)

Message-Id: <>
Reply-To: <>
From: =?ISO-8859-1?Q? <>
Subject: =?ISO-8859-1?Q?Photos of Black Singles in DALLAS=A0 =A0=A0=A0=A0=A0  =A0?=
Date: Mon, 27 Feb 2012 11:31:05 GMT
List-Unsubscribe: <

Sunday, October 16, 2011

Prime algorithm in assembler

One of the greatest strengths of assembly language is the extreme compactness of code. Digging though my old files I found this gem that prints all prime numbers under 2^32. The COM file size is 165 bytes. That's right! You could memorize the bytes if you wanted to and hand assemble it.

The algorithm is not sieve (which would create 2^31 locations and strike off multiples of primes). Rather this is a space optimized algorithm that repeatedly divides each number n from 2 to sqrt(n) to determine if a number is prime. This algorithm uses no additional storage than the 165 bytes to hold the code.

.model tiny
        org     100h
main    proc
        xor     eax,eax
        mov     al,3
        call    isprime
        jnc     @@m1
        call    printdword
        call    printcrlf
        inc     eax
        jnz     @@m2

        mov     ax,4c00h
        int     21h
main    endp

isprime proc
        mov     esi,eax         ; save number
        xor     ecx,ecx
        mov     ebx,ecx
        mov     edx,ecx         ; edx = 0
        call    sqrt
        mov     cx,ax           ; save sqrt
        mov     bl,2            ; ebx = 2 (divisor)
        mov     eax,esi         ; restore number
        cmp     ecx,ebx         ; is sqrt < divisor?
        jl      @@i4            ; yes, no need to divide any more
        div     ebx
        or      edx,edx
        jz      @@i1
        xor     edx,edx
        inc     ebx
        jnz     @@i2
        jmp     @@i3
isprime endp

sqrt    proc
        push    ebx
        push    ecx

        xor     ecx,ecx
        mov     ebx,ecx
        inc     ebx
        sub     eax,ebx
        jl      @@s1
        inc     cx
        inc     ebx
        jmp     @@s2
        mov     ax,cx

        pop     ecx
        pop     ebx
sqrt    endp

printdword      proc
        xor     ebx,ebx
        mov     cx,bx
        mov     bl,10
        xor     edx,edx
        div     ebx
        push    dx
        inc     cx
        or      eax,eax
        jnz     @@p1
        pop     ax
        add     al,'0'
        mov     dl,al
        mov     ah,02
        int     21h
        loop    @@p2

printdword      endp

printcrlf       proc
        mov     dl,13
        mov     ah,02
        int     21h
        mov     dl,10
        mov     ah,2
        int     21h
printcrlf       endp
        end     start