LeraoSoft - Knowledge Distributed

It has been a long time since we have updated and we figure it’s time for a status update.The first question i’d like to answer is what have we been doing with our time.  The answer is of course being code monkeys.   In our last post we said that it worked but there was still a lot of work to do.  Of that work we now have message authentication and moved all of the configurations into one file.  Also lots of cleaning has been done with more to do.  The final improvement is in usability.  Users and Storage Nodes will be able to automatically register.  This means that installation will be extremely easy just drop some files and change the configuration a little and wuala. Speaking of Wua.la I’d like to take some time to talk about our “Competitors”.  Of them there really are only two one is RobuSTore and the other is Wuala.  First lets talk about RobuSTore they are the closest thing to our project and are from UC SD.  They use LT Codes like our project does and have some of the same goals.  What is different is that they are concerned mostly with speed and want some redundancy on top of that, and are also concerned with large files on a pre configured network.  They do not use multicast or peer to peer systems (or so it appears) and they know where all the bits of data are.  This is different from our goals.  We hope to have a fast system yes but we want a dynamic system on unreliable dynamically configured networks with unreliable disks and computers in a word Byzantine.  The reason for this is that we want to replace SANS and NAS with peoples laptops and PC’s not with cheaper servers.  We also want the network to be flexible and work in a peer to peer way so that it can recover from individual network failures.Wuala on the other hand is very different they use ReedSolomon encoding which while an optimal erasure code is much more computationally complex and would slow down storage.  Also ReedSolomon does not scale with the number of nodes in a dynamic way it needs pre defined variables to encode and decode thus is not Rate-less like LT Codes are.  It also works in a peer to peer system though.  However it is designed to be a “web service” instead of something a company could have for their personal use.  It is also designed to be very distributed and has no central meta data storage which would not work for companies who need to know who saved what and get access to it.  Also by the end of the quarter we will have a paper we will try to get published.

Great news we have gotten the system to work.  It is still a little slow taking about 30 seconds to save a file and anywhere from 0.5 seconds to 45 seconds to retrieve a file, depending on size.  Currently the system supports listing storing and removing files as well as creating directories.

 It seems to perform better with more nodes on the network and is more reliable then (in a quick connect and ensure message sense not in a data integrity sense). We do however have lots of work still.  We need to be-able to split up files into multiple parts so we can store lager files more effectively, ensure all of the certificate based authentication is working correctly and  encrypt the file before sending it.  The final encryption will probably be done last so that there is the least amount hinging upon it but it will be easy to implement since the encoding code can take any stream we need only to create a encrypted file stream, one such stream is actually already created and part of the pastry utilities.  

Of key importance to our project is testing.To get closer to our goal of testing this on somewhat realistic systems we need to make it much easier to deploy and configure to do this we have made it use a variety of configuration files and we are working on creating an installer program to do most of the work for us in this regard aswell.  However many of our tests will be done with unit tests some basic ones have already been created and will be expanded upon. 

The latest design document can be found HERE or a copy can be found there->Winter Design Document

Last night we began integrating the file encoding and decoding functions into the network and client code.  This caused us to commit and pass revision 100 and we currently have written 2763 lines of code.  Within the next week we hope to test the code and perform the first file save on the network.After we have done that we will be working on ensuring configurability and security. 

So the programming has begun in full swing.  There is already a good start to the network clients for the Meta Data Controller, MimirClient(the “user” client) and the Storage Client.  Also a SubVersion repository has been set up and can be viewed on our Trac page at http://compsciguy.homeftp.org/trac/mimir.  Also i want to take this time to thank Chris Deeterly for his suggestion to implement a task handler that works like a queue to process tasks. The deadline is coming up soon (feb 11th) and we are working at a fever pitch to get this first HUGE milestone done in time.