system booting

 system booting (PC)

Any system administrator worth their weight in silicon, knows that you need to understand how the booting process works. From the moment you press power until you are presented with the login prompt, a number of things will have occurred:

• Hardware Boot (POST)
• OS Loader and Master Boot Record
• Kernel Initialization
• Run Levels & Init Scripts

Note: Most of the information below has been obtained from my cpsc550 (system admin class), researching online, and man pages. For a quick intro on the booting process I would advise a 'man boot'.

Hardware Boot (POST)

When power is turned on or a reset occurs. The cpu has a hardwired address that it begins to load instructions out of nvram (none volitile ram) at address 0xFFF0. This program is referred to in the PC world as the bios ( basic input/output system ) and usually exists in memory space 0xF000 : 0xFFFF. Since there is not much room left on the runway, the first instruction usually consists of a jump back into lower bios memory. This nvram is also called CMOS memory which stands for "Complementary Metal Oxide Semiconductor". The bios program now conducts a POST ( Power on Self Test ) where it makes sure that the memory and other peripherals are there and functioning correctly. If there are errors at this point, your system lets you know this with a series of beep codes. If you check your motherboard manufacturer's website there are details to what these codes mean. A minimal set of instructions now exists in the bios to begin probing a set of system devices for bootable code. This bootable code should begin loading the OS.

OS Loader and Master Boot Record

Once the system has located a bootable device, in most cases your hard disk, it transfers control over to the first sector of that device a.k.a the Master Boot Record (MBR). In a PC this loader has some limitations due to the bios; the most notable is the fact that we need the boot program and partition table to exist in the first 512 bytes. The format of the MBR is as follows:

• 446 byte program code starts at 0x0000
• 64 byte partition table starts at 0x01be ( decimal 446)
• 2 byte signature of ( 0x55aa ) starts at 0x01fe ( decimal 510 )

This all sounds good in theory, but as a programmer I want to know a little more about the mysterious MBR. I remember when I was a youngster that boot sector virus's were prevalent. It makes sense to put one there, seeing as how the OS is not loaded and the system has no protection at this point. I believe that these have all but faded away since the internet has come along, providing a better medium of transport. In order to get a better look at the contents of the boot sector we can run the 'dd' command as root. If we specify 'if='dev/hda' and 'of=mbr_contents' and put a 'count' on it we should get what we are looking for. The 'count' part of the last instruction is important; without it you will attempt to read the whole disk. On a modern system you will generate a very large file in a few seconds. Once we have the these bytes we could make the contents into something that is a little more readable. For this I used the command 'hexdump mbr_contents > mbr_hexdump'. Here are the results from my debian machine.

0000000 ebfa 0120 01b5 494c 4f4c 0516 6b4c 413f
0000010 0000 0000 0fe8 404f b836 b836 8080 5560
0000020 13c0 b800 07c0 d08e 00bc fb08 5352 5606
0000030 8efc 31d8 60ed 00b8 b312 cd36 6110 0db0
0000040 68e8 b001 e80a 0163 4cb0 5ee8 6001 071e
0000050 fa80 75fe 8802 bbf2 0200 768a 891d 80d0
0000060 80e4 e030 0a78 103c 0673 46f6 401c 2c75
0000070 f288 8b66 187e 0966 74ff 5221 08b4 80b2
0000080 13cd 5572 9892 ba91 007f 6642 c031 e840
0000090 0071 3b66 b8bf 7401 e203 5aef 8a53 1e76
00000a0 1fbe e800 004b 99b4 8166 fc7f 494c 4f4c
00000b0 2775 685e 0880 3107 e8db 0035 fb75 06be
00000c0 8900 b9f7 000a a6f3 0d75 02b0 75ae 0608
00000d0 b055 e849 00d5 b4cb b09a e820 00cd bae8
00000e0 fe00 004e 0874 e8bc 6107 e960 ff60 ebf4
00000f0 66fd 66ad c009 0a74 0366 1046 04e8 8000
0000100 02c7 60c3 5555 5066 5306 016a 106a e689
0000110 f653 60c6 5874 c6f6 7420 bb14 55aa 41b4
0000120 13cd 0b72 fb81 aa55 0575 c1f6 7501 524a
0000130 b406 cd08 0713 5872 c051 06e9 e986 cf89
0000140 c159 08ea 4092 e183 f73f 93e1 448b 8b08
0000150 0a54 da39 3873 f3f7 f839 3277 e4c0 8606
0000160 92e0 f1f6 e208 d189 5a41 c688 06eb 5066
0000170 5859 e688 01b8 eb02 b402 5b42 05bd 6000
0000180 13cd 0f73 744d 3109 cdc0 6113 f1eb 40b4
0000190 46e9 88ff 1f64 648d 6110 c1c3 04c0 03e8
00001a0 c100 04c0 0f24 0427 14f0 6040 07bb b400
00001b0 cd0e 6110 00c3 6344 b836 b836 0000 fe00
00001c0 ffff fe05 ffff d222 0704 ca12 0023 0180
00001d0 0001 fe83 ffff 003f 0000 d1e3 0704 0000
00001e0 0000 0000 0000 0000 0000 0000 0000 0000
00001f0 0000 0000 0000 0000 0000 0000 0000 aa55
0000200 0000 0000 0000 0000 0000 0000 0000 0000


There are the aa55 bytes right before the 512 byte boundary ... Sweet! That means we found what we were looking for... but unless your hex op-code reading is better than mine... we still have a problem. My first thought was to take our original file and run it threw gdb (gnu debugger), but this turned out not to work at all. It just complains about it not being an executable file. Next I gave 'ndisasm' the NAS Disassembler a try. This one ate the whole file and spat back the nasm I was looking for. So let's see what the lilo MBR boot loader does:

Click here to have a look at the MBR assembly code.

Kernel Initialization

Once the kernel code has been located and control has been passed over to it, the kernel first initializes the devices through the device drivers on the system. Once this is complete the kernel creates its first process which is the scheduler or the swapper. This process is a kernel process called kswapd. The kernel then mounts the root file system and starts its first user process PID 1 called 'init' ( /sbin/init ). Init is often referred to as the mother of all processes. This can be verified by the reader by typing the command 'pstree'. When init loads it looks in '/etc/initab' for instructions on what to do. The inittab specifies the default run level for the system and calls another script passing that run level as an argument to the script '/etc/init.d/rc 6'. At this point you really should go check out the 'rc' script. It is a small script and the syntax is easy to read. To check what run level your system is in, run the 'runlevel' command. The last thing the inittab does on my system is start up a bunch of virtual consoles that I can access with 'ctr+alt+F1 through F6'.

Run Levels & Init Scripts

The scripts that actually start and stop the services on my system are located in ( /etc/rc[0-6].d ). Once you have looked at the '/etc/init.d/rc' script, you know that it gets past the run level and attempts to call Star and Kill scripts in the appropriate run level directory. My system default runlevel is 6 ( X-graphical mode ), so if I go into '/etc/rc6.d/' and do an 'ls -al' we will see that the files in this directory are nothing more then symbolic links back into ( /etc/init.d ). This is were all the real scripts reside. The run level directories are just there to provide a convenient mechanism to talk to these scripts. The conveniance comes from the way the links are named. They all have an 'S' or 'K' in front of them and then a number. If you go back and look at the ( /etc/init.d/rc ) script you can see that the 'S' starts a service and the 'K' will kill a service. The numbers allow you to specify the order in which the system will execute the scripts. This will allow a service that relies on another service to be started after the service that it requires.

So if we would like to add a new service to be started at runlevel 6 we would do the following things:

1. Create an init script using the example provided here.
2. Put your new script in ( /etc/init.d/ )
.
3. Create a symbolic link from ( /etc/rc6.d/ ) called S[0-99]scriptname that points to your new script
.

That is all it takes. Finally, if you need a configuration file that your service is going to need when it starts, the standard place to put one is in the ( /etc/sysconfig ) directory. I hope that you have learned and had as much fun as I did playing around with this stuff..

link editing

 link editing (ld)

ink-editor, ld(1) concatenates one or more input files (relocatable objects, shared objects or archive libraries) to produce one output file ( relocatable object, exe, or shared object ). Most commonly evoked as a part of the compilation ( cc, gcc ).

• Link Editing (ld) Intro
• Executable Linker Format (ELF)
• ld Execution
• Your Compiler
• Symbol Processing and Resolution

Link Editing (ld)

Takes input files from cc, as, or ld and produces one output file of the following formats: relocatable objects, static exe, dynamic exe or shared object. All input files to ld are in the Executable Linker Format (ELF). It is therefore crucial that we understand ELF file format in order to understand link editing. First we shall examine the types of ELF files one can have and there purpose.

• Relocatable Objects - concatenation of relocatable object input files into one output that can be used again in link-editing. These files contain data telling the linker how to link them to other relocatable objects, shared objects, and executable's.
• Static exe - all symbol references get bound to the exe, and thus represent a ready to run process. Both forms of executable files contain the data necessary for the operating system to produce an executable image.
• Dynamic exe - concatenation of relocatable objects that requires intervention by the runtime linker to produce the runnable process. The symbols in the symtab might need binding at runtime. The dynamic executable may also be dependent on shared objects(so). Dynamic executable's are the default output of a compilation.
• Shared Objects - concatenation of relocatable objects that provides services to dynamic executable's bound at runtime by the runtime linker ld.so.1. Shared objects might also be dependent on other shared objects. Think of Shared objects as dynamic executable's that have not been assigned any virtual address space.

The graphic below demonstrates how to create the various file format discussed above.

Executable Linker Format (ELF)

The ELF file format was created by Unix System Laboratories as a better alternative to a.out and COFF binary formats. Some capabilities of the ELF format include: dynamic linking, dynamic loading, imposing runtime control on a program, and an improved method for creating shared libraries. ELF files contain five section types that may or may not be included in the file. The five types include:

1. The ELF header.
2. The Program header table.
3. The Section header table.
4. ELF sections. (linker view)
5. ELF segments. (executable view)

Each of the ELF file formats described above can be looked at in 2 ways (called views). The first view is the linker view and the second is the executable view. The views are summarized in the figure below:

The linker view of ELF files is partitioned into sections while the executable view is partitioned into segments. Sections represent the smallest indivisible unit that can be processed in the ELF file. A segment is a collection of sections and is the smallest unit that can be mapped (mmap) to memory by (exec) or (ld.so.1). These two views allows us to look at information that is specific to linking such as the symbol table and relocation information separate from information specific to creating the process image, like text and data segments. The bulk of the data is therefore stored in sections and segments with the rest of the file (headers) devoted to the organization and access of those sections/segments. The following is a brief description of each of the five file parts.

ELF Header.

This is the only fixed portion of the ELF file, always occurring at the start. It provides information such as: ELF version, target architecture, location of program header table, location of section header table, location of strings table(storing the names of sections), along with the size of each table, and lastly the location of the first instruction that is going to be executed.

#define EI_NIDENT 16 typedef struct {    unsigned char e_ident[EI_NIDENT];    uint16_t e_type;    uint16_t e_machine;    uint32_t e_version;    ElfN_Addr e_entry;    ElfN_Off e_phoff;    ElfN_Off e_shoff;    uint32_t e_flags;    uint16_t e_ehsize;    uint16_t e_phentsize;    uint16_t e_phnum;    uint16_t e_shentsize;    uint16_t e_shnum;    uint16_t e_shstrndx; } ElfN_Ehdr;

Program Header Table

The program header table is only useful to executables and shared objects. This provides organizational information on the array of segments in the file. Each entry in the program header table contains the type, file offset, physical address, virtual address, file size, memory image size, and alignment for a segment in the program. Each segment is copied into memory if its pt_type=PT_LOAD. ?? Question how do we know the physical address ??

typedef struct {    uint32_t p_type;    Elf32_Off p_offset;    Elf32_Addr p_vaddr;    Elf32_Addr p_paddr;    uint32_t p_filesz;    uint32_t p_memsz;    uint32_t p_flags;    uint32_t p_align; } Elf32_Phdr;

Section Header Table

Provides organization information on the array of sections in the ELF file. These entries provide the name, type, memory image starting address (if loadable), file offset, the section's size in bytes, alignment, and how the information in the section should be interpreted.

typedef struct {    uint32_t sh_name;    uint32_t sh_type;    uint32_t sh_flags;    Elf32_Addr sh_addr;    Elf32_Off sh_offset;    uint32_t sh_size;    uint32_t sh_link;    uint32_t sh_info;    uint32_t sh_addralign;    uint32_t sh_entsize; } Elf32_Shdr;

ELF Sections

Sections can hold executable code, data, dynamic linking information, debugging data, symbol tables, relocation information, comments, string tables, and notes. Some sections provide information on liking, others are loaded into the process image, while others provide information on building an executable.

ELF Segments

Segments are a groupings of like sections ( text segment, data segment). A process image is created by loading segments into virtual memory segments described by the program header.

Tools readelf

readelf is a tool for viewing elf files. Click here to view and example elfdump. Make sure to view the sections in the example file and return to the example when needed. I found that it gave me a better understanding of the material having an example elf file handy.

Sections of Interest to us

So the basic idea from here is that the link editor concatenates program .text, .data, and .bss sections into the new output file. The rest of the relocation and symbol information is modified or generated to the output file.

ld Execution

So the basic idea from here is that the link editor concatenates program .text, .data, and .bss sections into the new output file. The rest of the relocation and symbol information is modified or generated to the output file.

Here is the program flow for the linker:

• Verify options passed to it.
• Concatenate like sections (type, attribute, name) from input relocatable objects to form sections within the output file.
• Read symbol tables from relocatable object's and shared object's and apply the info to output file by updating other input sections. In addition an output relocation section might be generated.
• Generate program headers that describe all the segments created.
• generate dynamic linking info section providing shared object's dependencies and symbol bindings to the runtime linker.

You can change how these sections get mapped by creating a mapping file and using the -M option with (ld). More on this later.

Your Compiler

In practice you rarely invoke ld yourself and it is generally good practice not to. This is because the linker will not attach init and termination code to your program. But we will run some tests on our example program to better understand this (example test.c - the simplest c program).


int main( )
{
return 0;
}


Then we can ask nicely for gcc to compile our test program but not to link it. Once we are done this we can try to manually link the file

gcc -c test.c

ld test.o
ld: warning: cannot find entry symbol _start; defaulting to 0000000008048094


Click here to view the "readelf -a" of the resulting file

The normal way is to have the compiler dirver invoke the linker as follows

gcc test.o

Click here to view the readelf of the resulting file. The deference is rather substantial. To the tune of a lot of extra crap gets included into my simple little program. There is actually more stuff added then there is stuff in my program. At this point it could be said that gcc is the author of my program and not me. So what is all this extra crap that is being added? Lets find out.

One of the only times that it is acceptable to invoke the linker on your own is when you are creating another relocatable object. This is done with the -r option for ld.

ld -r test.o

The moral of the story is that during compilation there is a bunch of extra stuff that gets included in your file. Upon realizing this a good question is what is it? On a Solaris box we can use the -# option to have the compiler display these mysterious files that are included into our code. In linux and gcc you can get the same output with a call to gcc --verbose..

cc -# -o prog test.c

Here is the results on Solaris.

/opt/SUNWspro/bin/../WS6U1/bin/acomp -i test.c -y-fbe -y/opt/SUNWspro/bin/../WS6U1/bin/fbe -y-xarch=generic -y-o -ytest.o -y-s -y-verbose -y-xmemalign=4s -Qy -D__SunOS_5_8 -D__SUNPRO_C=0x520 -D__SVR4 -D__unix -D__sun -D__sparc -D__BUILTIN_VA_ARG_INCR -D__SUN_PREFETCH -Xa -D__PRAGMA_REDEFINE_EXTNAME -Dunix -Dsun -Dsparc -D__RESTRICT -I/opt/SUNWspro/WS6U1/include/cc "-g/opt/SUNWspro/bin/../WS6U1/bin/cc -c "
### Note: LD_LIBRARY_PATH = <null>
### Note: LD_RUN_PATH = <null>
/usr/ccs/bin/ld /opt/SUNWspro/WS6U1/lib/crti.o
/opt/SUNWspro/WS6U1/lib/crt1.o
/opt/SUNWspro/WS6U1/lib/values-xa.o -o prog test.o -Y "P,/opt/SUNWspro/WS6U1/lib:/usr/ccs/lib:/usr/lib" -Qy -lc /opt/SUNWspro/WS6U1/lib/crtn.o gcc --verbose test.c

Here is the results under debian linux

Reading specs from /usr/lib/gcc-lib/i486-linux/3.3.4/specs
Configured with: ../src/configure -v --enable-languages=c,c++,java,f77,pascal,objc,ada,treelang --prefix=/usr --mandir=/usr/share/man --infodir=/usr/share/info --with-gxx-include-dir=/usr/include/c++/3.3 --enable-shared --with-system-zlib --enable-nls --without-included-gettext --enable-__cxa_atexit --enable-clocale=gnu --enable-debug --enable-java-gc=boehm --enable-java-awt=xlib --enable-objc-gc i486-linux
Thread model: posix
gcc version 3.3.4 (Debian)
/usr/lib/gcc-lib/i486-linux/3.3.4/cc1 -quiet -v -D__GNUC__=3 -D__GNUC_MINOR__=3 -D__GNUC_PATCHLEVEL__=4 test.c -quiet -dumpbase test.c -auxbase test -version -o /tmp/ccSbXIgh.s
GNU C version 3.3.4 (Debian) (i486-linux)
compiled by GNU C version 3.3.4 (Debian)
GGC heuristics: --param ggc-min-expand=98 --param ggc-min-heapsize=129048
ignoring nonexistent directory "/usr/i486-linux/include"
#include "..." search starts here:
#include <...> search starts here:
/usr/local/include
/usr/lib/gcc-lib/i486-linux/3.3.4/include
/usr/include
End of search list.
as -V -Qy -o /tmp/ccWmNHhp.o /tmp/ccSbXIgh.s
GNU assembler version 2.15 (i386-linux) using BFD version 2.15
/usr/lib/gcc-lib/i486-linux/3.3.4/collect2 --eh-frame-hdr -m elf_i386 -dynamic-linker /lib/ld-linux.so.2 /usr/lib/gcc-lib/i486-linux/3.3.4/../../../crt1.o /usr/lib/gcc-lib/i486-linux/3.3.4/../../../crti.o /usr/lib/gcc-lib/i486-linux/3.3.4/crtbegin.o -L/usr/lib/gcc-lib/i486-linux/3.3.4 -L/usr/lib/gcc-lib/i486-linux/3.3.4/../../.. /tmp/ccWmNHhp.o -lgcc -lgcc_eh -lc -lgcc -lgcc_eh /usr/lib/gcc-lib/i486-linux/3.3.4/crtend.o /usr/lib/gcc-lib/i486-linux/3.3.4/../../../crtn.o


Initialization and Termination Sections

Dynamic Objects provide code for runtime initialization and termination. This code may be in the form of function pointers or one entire block. Each of these sections is built from like section types given by input relocatable objects. Sections:

• .preinit_array
• .init_array
• .fini_array

When creating dynamic objects the link editor identifies these arrays with .dynamic tags DT_PREINIT_ARRAY, DT_PREINIT_ARRAYSZ, AND DT_INIT_ARRAY, DT_INIT_ARRAYSZ, AND DT_FINI_ARRAY, DT_INI_ARRAYSZ.

The sections .init and .fini provide the runtime initialization and termination code for your dynamic executable. Compiler drivers usually supply these sections as files that are tacked onto the beginning and end of the input file list. These sections are provide the requred code in the form of two reserved functions named _init and _fini. When creating a dynamic object the link editor provides symbols with .dynamic tags DT_INIT and DT_FINI. One thing that is very kewl is that you can add functions to the ini_array and the fini_array.

refer back to our ELF file to locate these symbols.

Symbol Processing and Resolution

During input file processing the link editor passes any local symbols straight through to the output file, while global symbols are accumulated internally. The internal symbol table is searched for each new global symbol entry to determine if two are the same and some form of resolution needs to occur.

Basic types of symbol resulution

• Undefined - global
• Tentative - occupy storage at runtime
• Defined - occupy storage in file

Play Framework Comet / Chunking Support

So my first choice when it comes to full duplex communication with a web server is websockets. Play has amazing support (non-blocking/async) support for websockets and there are plenty of example inside activator templates. But what do we do for those browsers that don't support websockets? In particular Android default browser 4.3 (Jellybeen) and below.

Note: 4.4+ Kitkat will now have chrome as the native browser [websockets, webrtc, webgl, and much more]

In cases like this we need to fallback to some form of "comet" or "long-polling" technique. However in the spirit of play we want to be non-blocking and asynchronous with our approach. With a little bit of digging on google we can find some info on how this was done in play 2.0. This stackoverflow article talks a bit about how comet works in play 2.0

public static void newMessages() {
    List messages = Message.find("date > ?", request.date).fetch();
    if (messages.isEmpty()) {
        suspend("1s");
    }
    renderJSON(messages);
}

The key bit here is suspend("1s") which is what holds the HTTP request open, checking for new data once per second.

However suspend is not going to work for us in play 2.1+ so we need to find another solution. A number of things changed from 2.0 to 2.1 and in particular the "SimpleResult" is now the return type for all actions. They have done away with the other result types and folded them in under simple result. There are some good notes here on migrating to play 2.2. One of the things we can see is the use of a Chunked result

Working with chunked results

Previously the stream method on Status was used to produce chunked results. This has been deprecated, replaced with a chunked method, that makes it clear that the result is going to be chunked. For example:

def cometAction = Action {
  Ok.chunked(Enumerator("a", "b", "c") &> Comet(callback = "parent.cometMessage"))
}

Advanced uses that created or used ChunkedResult directly should be replaced with code that manually sets/checks the TransferEncoding: chunked header, and uses the new Results.chunk and Results.dechunk enumeratees.

So this looks promising. However we still need to make a choice on what we are going to do as a generator for the Enumerator. My first idea was to use to use Enumerator.fromCallback1. This looked like it should work, and even compiles fine, but no chuncked response. After doing some digging I found that my return type looked a little strange and that this may be a bug in "fromCallback1". I found someone else having the same issue in this post here.

Next I played around with a Promise.timeout and a Enumerator.generateM. There is an example of this style of comet usage on the comet-clock. The lines to note here are the following:

Enumerator.generateM {
   Promise.timeout(Some(dateFormat.format(new Date)), 100 milliseconds)
}

...

def liveClock = Action {
   Ok.chunked(clock &> Comet(callback = "parent.clockChanged"))
}

After some more time trying to get this new approach to work, I stumbled into a discussion around Concurrent.broadcast. Which as it turns out... is exactly how I am doing my producer / consumer for my websocket... doh! Should have looked here right at the start :)

Concurrent.broadcast

This article on stackoverflow talks about the Concurrent.broadcast. After a little bit of time playing with this I got a working example. Here are the steps involved.

First you need to get you producer and consumer tuple.

   val (cometOut, cometChanel) = Concurrent.broadcast[JsValue]

Now you can use the cometChannel to push Json data into the cometOut Enumerator.

   cometChanel.push(Json.obj("data" -> "test"))

And finally you can plug your enumerator into the play Comet helper for a chuncked data response.

   Ok.chunked(cometOut &> Comet(callback = "parent.cometMessage"))

Chuncked response and what can go wrong

First off the comet helper in play does some "padding" to get around an issue with chunking. This can be seen with this gist.

 def apply[E](callback: String, initialChunk: Html = Html(Array.fill[Char](5 * 1024)(' ').mkString + ""))(implicit encoder: CometMessage[E]) = new Enumeratee[E, Html] {

Note the Array.fill[Char](5 * 1024)(' ').mkString. So if you are not using the Comet helper in play, you will need to do something similar to have your chunked response work. You can find more information on this here.

Nginx and HTTP 1.1

If you are using nginx to proxy your request like I am, then there are further problems to deal with. First Nginx defaults to HTTP 1.0 which does not have chucking support. You need to explicitly tell your nginx config to use HTTP 1.1.

   location / {
     proxy_http_version 1.1;
     proxy_pass http://localhost:9009/;
     proxy_set_header Host $http_host;
  }

There is another problem

If you view your endpoint now using curl, everything will work fine... but wait. If you try your comet solution in chrome or firefox you will get the infinite loading spinner and you will never see any chunked data until the http request is closed.

If you point your browser directly at play however you will notice that everything is working fine. This again points to a problem within nginx. After using curl to examine the headers for both request, I noticed that nginx response headers contain the line Connection: keep-alive. Browsers seem to wait for the connection to close for a chunked transfer before returning the data.

So we are left trying to get rid of the Connection: keep-alive from the nginx response headers. Unfortunately this is not a configurable option inside nginx and you are left patching it from source. I have posted a comment here.

Update: The solution is to turn GZIP off for nginx
The white space that play pushes into the buffer (described about) obviously compresses very well with gzip... totally defeating the purpose of the buffer to begin with. The solution is to turn off gzip. NOTE: that I don't like this solution and will be looking for alternatives to play http 1.1 chunking.

Update: There is more info surrounding this issue in the above post

Nginx Websocket configuration

https://github.com/dryan/decss-sync/issues/1

Also ran into problems with nginx closing the socket after 1 min. This is related to proxy_read_timeout.. more about this here http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_read_timeout.

Neo4J user service plugin for secure social (play framework/scala)

SecureSocial-Neo4JUserService

Neo4J user service plugin for secure social (play framework/scala)

Project on GitHub: SecureSocial-Neo4JUserService

Background

Building reactive web applications with play framework and scala often starts with a social login system. One great way to get started here is to use Secure Social. They provide a simple way to get up and running with a large number of providers. Once you are running you will need to store your account information and more the likely the relationships between your social users "friends". This is where Neo4J really shines.

Requirements

• Play Framework play
• Secure Social plugin for play Secure Social

Setup

Once you have secure social up and running all you need to do is add this scala file to your play project. I created a app/servicies directory to place the file. Next you simply need to add a line to your play.plugins

play.plugins

9998:service.Neo4JUserService

Neo4J Structure

Users will now be added to your neo4J with the following structure

(u:User)-[:HAS_ACCOUNT]->(p:Provider)

You can also use the utility methods outlined below to make users friends

(u1:User)-[:FRIEND]->(u2:User)

Helper methods

Here are number of usefull methods for helping you to work with secure social and Neo4J

object  Neo4JUserService{
  def socialUserFromMap( umap: Map[String, String]): Identity = {
    ...
  }

  def makeUsersFriends( uuid1: String, uuid2: String ) = future{
    ...
  }

  def socialUserFromToken(provider: String, token: String) = future{
    ...
  }

  def uuidFromProviderInfo( provider: String, id: String, name: String = "" ) = future {
    ...
  }


  private def createUserAndProvider(name: String, id: String, fullname: String) = {
    ...
  }
}

Things to consider.

You will still need a way for users to "link" multiple providers on your back-end. Currently if a user signs in using another provider, they will get another user and provider record in Neo4J. You could try to combat this at the login level by looking for emails that are same as other providers (you would want to verify the email before linking for security reasons)

Another way would be a settings section on your site when a user is loged in, that would allow them to "link" their other accounts. In this manor you would want to create to continue to build a structure like the following

        /[:HAS_ACCOUNT]->(p0:Provider)
(u:User)-[:HAS_ACCOUNT]->(p1:Provider)
        \[:HAS_ACCOUNT]->(p2:Provider)

Base3d a simple c++ library

Base3d lib C++

GitHub https://github.com/coreyauger/cpp_base3d_lib

Overview

This is a small and simple base 3d library written in C++. You would be crazy to write your own 3d library in todays world… However there are still a number of applications for using a compact 3d library to learn from or implement application specific problems.

This code was written a while back but used in a number of 3d applications during my time in University. I have decided to share it in the hopes that someone could find it as useful as I once did.

Example projects

Collision Detection Project

This project I designed a large "ball" course that I let serve as a demonstration of the physics I wrote. This was a super fun project and ended up working very well.

Water Simulation

This was a really fun project to work on. The main idea for the movement of the water came from one of my favorite sites on the net: The good-looking textured light-sourced bouncy fun smart and stretchy page

Radiosity

Dividing each face of our scene into light-map texel regions, we are going to perform a gathering type of radiosity solution. First by using GL to render the scene from the perspective of a texel into a hemi-cubed texture map, we solve the visibility problem. More info here

FLV Socket Writer Direct Show Filter

flv socket filter

Project on GitHub: https://github.com/coreyauger/flv_socket_filter

Windows Direct Show filter. Stream FLV and header information to any connecting application. Takes care of late joiners by resending h264/aac headers.

Overview

FLV files consist of a single header, and metadata, chunk followed by N interleaved audio and video chunks.
Most streaming filter implement RTMP and provide a single client with the ability to connect and start injesting a stream.
This filter provides a much more flexible and free form way of Interprocess Communication with you data.

• Supports M number of simultaneous connection steams to FLV
• Supports "late joiners" headers(Meta,Head) + AAC/H264 packets are retransmitted to new connections
• Actionscript ByteStream clients will receive data on chunk boundaries for easy parsing.
• Simple to right stream out to file on disk at the same time as other operations.

Requirements

The only requirement are the direct show base classes. Sockets are winsock implementation.

Usage

Currently the filter has the listening port hardcoded. You could add this as an option to the filter option page if you so desire. Once this filter is up and running you can connect to it via the port number and start to injesting your FLV.

Example

Here is an example client written in C# that will connect and write the FLV to disk.

This example uses the FlvStream2FileWriter that is also an open source project I have on GIT here: (https://github.com/coreyauger/flv-streamer-2-file)[https://github.com/coreyauger/flv-streamer-2-file]

This takes care of inseting the new timing information into the FLV so that your file on disk will allow seek operations.

internal void SaveStreamToFile(string path)
        {
            const int count = 4096;
            byte[] buffer = new byte[count];

            // Declare the callback.  Need to do that so that
            // the closure captures it.

            AsyncCallback callback = null;

            System.IO.File.Delete(path);
            FlvStream2FileWriter stream2File = new FlvStream2FileWriter(path);

            // Assign the callback.
            callback = ar =>
            {
                try
                {
                    // Call EndRead.
                    int bytesRead = _clientStream.GetStream().EndRead(ar);

                    // Process the bytes here.
                    if (bytesRead != buffer.Length)
                    {
                        stream2File.Write(buffer.Take(bytesRead).ToArray());
                    }
                    else
                    {
                        stream2File.Write(buffer);
                    }

                    // Determine if you want to read again.  If not, return.                
                    // Read again.  This callback will be called again.
                    if (!_clientStream.Connected)
                    {
                        throw new Exception("Just catch and exec finallize");
                    }
                    _clientStream.GetStream().BeginRead(buffer, 0, count, callback, null);
                }
                catch
                {   // most likly discontinuity ...just fall through                   
                    stream2File.FinallizeFile();
                    return;
                }               
            };
            _clientStream = new TcpClient("127.0.0.1", 22822);  // (CA) TODO: make port configurable on the filter and here
            // Trigger the initial read.
            _clientStream.GetStream().BeginRead(buffer, 0, count, callback, null);
        }

Blue Tooth Low Energy Scanner for Android written in scala

android-scala-ble

Blue Tooth Low Energy Scanner for Android written in scala

Overview

This project serves to wrap up some common functionality regarding scanning for a BLE sensor device.
Right now the code is simple and to the point. Install a filter (device name, mac address) and then start scanning for devices.

Features

I will try to keep adding to this list as I go. For now here is a short list of features:

• Scan for a BLE sensor devices
  • Filter device list
• Callback function to report device and signal strength

Example Usage

class MainActivity extends Activity with BleDeviceScanner{
  // Notice the "with BleDeviceScenner"
  // ...

  @Override
    protected override def onCreate(savedInstanceState: Bundle) = {
      // ... normal android init
      initBleScanner(this)

      val bscan = findViewById( R.id.bscan ).asInstanceOf[Button]
        bscan.setOnClickListener(new View.OnClickListener {
          override def onClick(v: View) = {         
             val filter = {
                d: BluetoothDevice =>
                    d.getName() != null     // You could filter by device name or address here..                                    
              }
              startScanWithFilter(filter){
                di: BleDeviceInfo =>  // This ia a callback with the located device 
                  Log.d(TAG,"Found device[%s] with signal stregth: %s".format(di.getBluetoothDevice.getAddress, di.getRssi) )
              }             
          }
        })

    }

}

Example Projects

Was used in a hackathon to try to do accurate indoor positioning. We ended up having to make a ton of modifications since the TI Sensor Tag can not be used to accuratly interpolate position.

Here is a screen shot TODO://