Published by Ostap Cherkashin on 2011 06 22.
New functionality implemented in v3 makes it possible to run bandicoot programs on multiple computers in a TCP/IP network. This way you can improve data availability and also increase the transaction throughput (though the later depends on the frequency of writes, if there are any).
Let's have a look at how bandicoot handles a function invocation. It all starts with a client initiating an HTTP request (e.g. GET /List HTTP/1.1) then bandicoot executes the specified function (e.g. List) and returns the result back to the client. Here is a sequence diagram which illustrates this process in more details:
exec tx vol
---- -- ---
| | |
HTTP GET/POST | | |
- - - - - - - - >| | |
| start a transaction | |
|- - - - - - - - - - - >| |
| | |
| read (optional) | |
|- - - - - - - - - - - - - - - - - - - - >|
| | |
| calculation | |
|- - - - - - + | |
| | | |
|< - - - - - + | |
| | |
| write (optional) | |
|- - - - - - - - - - - - - - - - - - - - >|
| | |
| commit / revert | |
|- - - - - - - - - - - >| |
< - - - - - - - -| | |
| | |
Internally, bandicoot is split into three components:
The key idea here is that it is possible to start one or more executor to offload the CPU-intensive work, and one or more volume to offload the IO-intensive work. Transaction manager is not distributed (yet) so only one instance could be run per cluster (it is indeed a single point of failure for now). On the other hand, tx is quite modest in terms of resources so scaling it to multiple computers might not be of a great benefit, it is more about system availability. Volumes provide storage for global variables but when there are multiple volumes running they synchronize periodically between each other, so the data is eventually distributed. From data-consistency perspective, executors always go to the volume which has the correct version of a given variable (with a strong preference for localhost volumes).
Obviously, the startup parameters had to change in order to support different modes of operation. The quickest way to start is by reviewing the usage instructions:
$ bandicoot usage: bandicoot <command> <args> standalone commands: start -p <port> -d <data.dir> -c <source.file> -s <state.file> distributed commands: tx -p <port> -c <source.file> -s <state.file> vol -p <port> -d <data.dir> -t <tx.host:port> exec -p <port> -t <tx.host:port> [bandicoot v3, http://bandilab.org, built on Sun Jun 12 11:30:55 UTC 2011]
There are two types of commands: standalone and distributed. In standalone mode, when bandicoot starts up, it bootstraps all three components and then interconnects them together. In distributed all three components can be run individually. It all starts with the tx which is a centralised entity in the whole cluster. Then executor and volume take transaction manager connection details as arguments.
In general, it is up to a user to decide how many executors and volumes to start and where to run all of them. If there are 3 or more computers available you can setup a cluster using the following topology, e.g.:
host1 host2 host3
+------+ +----+ +------+
| exec |------| tx |------| exec |
+------+ +----+ +------+
| |
+------+ | | +------+
| vol |-------+ +-------| vol |
+------+ +------+
This can be done in the following way:
host2$ bandicoot tx -p 12300 -c program.b -s state
host1$ bandicoot exec -p 12345 -t host2:12300 host1$ bandicoot vol -p 12301 -t host2:12300 -d data/
host3$ bandicoot exec -p 12345 -t host2:12300 host3$ bandicoot vol -p 12301 -t host2:12300 -d data/
This way you get two executors running on host1:12345 and on host3:12345. All requests executed via host1 are always consistent with the ones executed via host3. Volumes periodically sync the data, so once the required version is available on host1 executor will always go to localhost thus avoiding the network round-trips.
Hopefully this post provides enough information to get started with the new functionality. If you have more questions please leave your comments below.