Welcome to the Films on the Fly tech blog.
This is for any tech related information. Especially scripts, projects, general news and any thoughts or resources regarding the general state of web development.
Welcome
Featured
Reply
Code Balance – Fast and throw away or slow and solid?
I just wrote this on the Fishburners Skype chat in response to someone trying to Outsource their iPhone dev then someone saying that when they (goCatch) did it they had to throw away all the code and start again which took 10 intense weeks.
——
Hi Tom, I’m currently grappling with the difference between doing fast, throw away later code and good quality code which can be a foundation for future functionality.
I suspect there’s a couple of things to consider when trying to make the (faustian?) bargin :
1. How likely is the code going to need to be re-written anyway once you’ve done some A/B testing or validated some assumptions?
e.g If you are creating a task app or calendar app and there’s a number of ways of doing the task and you are trying to see which way has the most traction then going MVP or even wireframe can give you a lot of feedback.
2. What is the risk the function or style changes will be used? If it isn’t core to your product or brand then the likelyhood it will be used is probably much lower than if it is what you are basing your whole company on.
3. Is it a important component or one that can be re-used in various parts of the app?
An example might be password hashing for a web platform, getting it right is important so when your database is stolen and spread around the Internet it will be far far harder for hackers to log into users other services.
Other functionality might be a notifications service or an none (number once) component for verifying an action should be taken, e.g that the persons account is valid, that they really did want to drop the whole database.
If you are coding Instagram or Flickr then having decent photo gallery code which can scale is probably really important.
When it comes to programming for the long term I’ve started coding with the end aim in mind but only coding the bits that are needed e.g adding some shell classes which will later do things like filtering or be used to help scale, but currently just pass the data through to what is doing the main work.
In my mind it’s concepually similar to building a bigger warehouse and throwing up some scaffolding which indicates where walls will be. You do the little bit of extra work in designing and building the bigger warehouse, but it is far better than quickly filling up a small house and having to build a warehouse around it whilst what you are storing is spilling out all over the place.
Anyway, the main question I wanted to know was : Did you learn anything useful by doing the re-write? Did you do it in an architectually different way because of the knoledge gained?
—–
[3:25:23 AM] James Zaki: Tom, was that 10 weeks x 2 devs, doing both front and back end?
[3:27:40 AM] James Zaki: And 10 “insane” weeks is >50hrs per week each?
[7:12:03 AM] Tom Horn: @James. Just the client side, so x 1 dev … and yep > 50 hrs …
[7:15:47 AM] James Zaki: Ah ok, I thought the server side was getting an upgrade/re-write at the same time
[7:16:25 AM] Tom Horn: Server has got lots of love, but not a re-write from scratch
[7:16:49 AM] Tom Horn: @Michael I wasn’t involved in the 1.x codebase to the very end, so architectually they were basically seperate products. I think I have accidentally placed myself on the wrong side of this argument. I actually think getting an MVP built cheaply is a great way to kickstart a product, as long as you go in with the expectation that you will have to pay back that technical debt at some point.
[7:17:08 AM] Tom Horn: err … (to the very end) -> (till the very end)
[7:17:39 AM] Maliblu: interesting
[7:17:39 AM] James Zaki: Non-tech people need to appreciate “technical debt”, I love that expression
[7:18:16 AM] Tom Horn: hehe … yep .. it is awesome 
[7:18:54 AM] James Zaki: its like karma in a way 
[7:19:19 AM] Tom Horn: A little bit … and it is a lot like debt. The longer you leave it unpaid, the hard it is to pay back.
[7:19:31 AM] Tom Horn: Hence the ten weeks of insanity …
[7:19:32 AM] Tom Horn: ;_)
[7:19:55 AM] James Zaki: then you just declare bankrupt and start from scratch
[7:20:11 AM] Tom Horn: Yep. Which I guess is what we did.
[7:20:26 AM] Tom Horn: feels good to be on the other side of that
[7:35:34 AM] Jason MacLulich: Just to put this to bed a little: the problem with goCatch was that we had no POC-code exit strategy — and that’s something that can trip you up if you’re not careful. The problem is if that you don’t have a strong technical presence locally you’re beholden to your outsources for everything and if you can’t be sure of code quality you accumulate, yes this technical debt, which can kill you if you can’t get rid of it. Everything you do takes more and more time to do and harder to achieve and the likelyhood of BUGS in the field increases. We’ve had to delay feature development for some months to get rid of this technical debt which in the startup field can be deadly if you’re being closely hounded by competitors and features are what you need to differentitate you.
[7:36:23 AM] Jason MacLulich: @James: Yeah no complete rewrite for the backend, I could get away with not having to do that and just replace chunks of it where we needed. That’s why I’m introducing a generic messaging layer with 0MQ and adding extra functioanlity (rewriting where needed) in Scala at the moment.
[7:36:55 AM] Jason MacLulich: I see in about 2 months a complete rewrite through an organic process rather then a stop the press and rewrite everything from scratch method which we had to with the client.
[7:37:47 AM] Tom Horn: Yep. Nice thing about the server is that a incremental re-write is possible. That just wasn’t an option on the client side.
How to waste a $35K grant
This is a post about money… Or more specifically the grants and investments in the Australian Startup scene. This is based on my own experience and I may be completely wrong. I’m interested in a dialogue, so please let me know if my assumptions or experiences differ from yours.
![[Picture] Burning Money](http://w00t.filmsonthefly.com/wp-content/uploads/2011/08/Burning-Money.jpg "Burning Money")
Sydney Startup Weekend – Classmate
OK, so I participated in the Sydney Startup Weekend (29th to 31st of July, 2011) and was the web developer for the winning team : www.getclassmate.com
Classmate is creating a web platform to help let teachers share their smart whiteboard lesson plans with other teachers.
It was an awesome time and working with the other people in the team was particularly awesome. Something I haven’t had a chance to do like that before.
I setup a Dropbox folder for the web development and had Dropbox running on the server with a hacked together cp script which we’d manually run to copy the newly changed Dropbox files onto the web server folder. It worked amazingly well. There was myself and Michael I coding in Netbeans at the same time, using the Yii PHP framework.
Silvio was working on the MySQL database and also reverse engineering the .notebook smart whiteboard file format. Turns out it’s a zip file and that inside it is a preview.png file which is perfect for us to grab and use as the preview image of the project (it just shows the first page of the lesson).
We worked on two main models : Lessons and Teachers.
Michael Imstepf did most of the front end work. Setting up a jQuery carousel on the front page, writing some of the CSS, etc.. I mainly worked on the back end, getting the right information to be displayed, to automate some of the object saving. E.g To automatically add the createdAt timestamp or update the updatedAt timestamp when edited.
I also created a widget to display the updated/created at time in a nice human readable format. For example it would show “Updated 23 seconds ago“, or “Updated 19 minutes ago” and so on, for hours, days and weeks (I’ll have to add months and years when I next look at it). Creating the widget in Yii took a little longer than maybe it should have as I had to learn all the different ways to extend Yii and the most appropriate way. I simply wanted to create a static class function that I could call, but from a generic class that anything can access, but the two standard models are extending CActiveRecord or CFormModel which isn’t what I wanted. Widgets seemed designed for the task and made sense at the time. Although I’ll re-evaluate that at a later date.
The most powerful part of the whole experience was that we had such a great range of people all working together so well. I really enjoyed Mary’s UX/UI expert advice. Especially when creating wireframes and helping with layout suggestions.
The ability for people like Vik and Silvio to quickly add content to the Dropbox was a boon. They could easily just copy a picture of someone into the right folder and create a new teacher or lesson to fill out our sample very quickly.
Unfortunately during the weekend we only reached the proof-of-concept stage as the actual upload and download functionality didn’t work properly. We faked it for the pitch, not that it mattered as the pitch didn’t have time to show off the actual platform, which made me a little sad having only slept 3 hours cause I was working my ass off on it, but Treffyn, Jesse, Ehon and the others did an AMAZING job and created the most amazing pitch. Also, big ups to Mark for suggesting unbounce and creating an awesome landing page. David was also full of ideas.
Overall the team rocked and we are still kicking. We are actively emailing, Skyping and Google Docs’ing. We also got into the beta of Asana the collaborative task list manager thanks to Ehon’s fantastically vast network of contacts. He seems to know nearly everyone in the web business/startup space from Sydney to Brisbane!
Anyway, that’s enough ranting from me.
Cheers!
[Update] Starcast have put up a video of the Startup Weekend.
Interesting posts 23rd July 2011
Here’s a bunch of interesting posts, articles and videos I’ve come across over the past week. It’s quite a mix-mash
[Tech articles of interest]
http://engineering.tumblr.com/post/7658008285/efficiently-copying-files-to-multiple-destinations — How to copy files between multiple computers (in a streaming fashion) which is especially useful when starting up a number of new server instances.
http://blog.backblaze.com/2009/09/01/petabytes-on-a-budget-how-to-build-cheap-cloud-storage/ — An oldie, but some people might not have seen it. Build your own storage racks full of HDDs, probably only useful for really high storage applications, and would require a decent amount of CapEx compared to S3, but lower OpEx.
http://www.smashingmagazine.com/2011/07/18/seven-guidelines-for-designing-high-performance-mobile-user-experiences/ — I enjoyed this article as it highlights the fact that design and UI decisions have an effect on performance and reliability.
The 7 guidelines for designing high performance mobile user experiences are :
- Define UI Brand Signatures
- Focus the Portfolio of Products
- Identify the Core User Stories
- Optimize UI Flows and Elements
- Define UI Scaling Rules
- Use a Performance Dashboard
- Champion Dedicated UI Engineering Skills
[Business Articles]
http://blog.thomvest.com/youhavetwoyears/ — According to previous patterns we have 2 years until the latest version of the dot com bubble bursts.http://cameronkeng.com/hn-how-do-i-pull-a-google/ — Google, Microsoft, Facebook and the like only have an effective tax rate of 2.4%. This explains the current loop holes and how.
http://www.avc.com/a_vc/2011/07/financing-options-preferred-stock.htm – Talking about preferred stock (preference shares). Which is used a lot in startup circles.
http://timbull.com/adventures-with-venture-capital-outside-the-v — This is a story I’ve heard a couple of times. Getting 95% through a VC funding deal then it goes flat. Sounds nasty 
http://www.uxbooth.com/blog/psychological-manipulation-in-ecommerce-design/ — How to use tricks like lowering the friction and increasing the pressure to make people more interested in buying your products.
Anyone else on the Startup Digest? It’s where I got some of the links from.
http://startupdigest.com/ Some cool article links emailed to you weekly.
Photobox
This is an idea for a new company which I will be pitching at the Sydney Startup Weekend.
Situation : Pro and Prosumer Photographers create vast quantities of photos. e.g after an intense photoshoot or wedding they can end up with 64GB+ of data.
Problem : Backing up 1TB or more of photos is a daunting task. Most photographers don’t do a good job at this and usually have to do very manual processes.
Solution : Photobox. An online backup system with a combination of local storage that synchronises small changes using the Internet, but uses a couriered Hard drive for allowing uploading of large quantities of data.
Background
Backing up data is harder than most people think. If you don’t have at least 3 copies of the data in at least 2 geographically separate places then you are going to lose your photos, it is just a matter of when. But it is harder than just that, as the TAO of backup explains :
- Coverage — Backup everything you can.
- Frequency — Backup as often as you are making changes.
- Separation — Geographically redundant storage.
- History — Versioning/snapshots of data changes over time. This helps recover from corruption, virus infiltration and PEBKAC.
- Testing — If you haven’t tested your backups then Murphy’s law says they will likely fail when you need them the most.
- Security — If I want to hack you, I’ll go for your unsecured backups.
- Integrity — If your original data has been corrupted without you knowing then your copies of it are useless.
Most photographers (at least the one’s I have sampled) use a basic HDD or raid array (especially good is something like a DROBO). The Photobox software would work similar to Dropbox, in that files on the drive would be synchronised with an online backup service. However, if a large number of new files were added, or edited (especially PSDs), they will be couriered a portable hard drive. Connect the drive up, it encrypts and stores the images for transport then put the drive into the return addressed packaging and drop it off at the nearest post box.
Business model
This is based off the awesome Business Model Generation canvas : http://www.businessmodelgeneration.com/downloads/business_model_canvas_poster.pdf
Value Proposition : An easier way for photographers to backup large quantities of data using a combination of Internet and couriered hard drives.
Customer Segments : Professional and prosumer photographers. Especially those taking lots of photos in Raw mode and don’t have a robust, automated, geographically redundant storage system.
Customer Relationships : Primarily the relationship is around ensuring the automated software application is working properly and backing up small changes or suggesting they send in the portable HDD for faster, safer, backup. As we are maintaining a copy of their photos and data it should be a long term relationship with good customer service
Channels : Possible options include Photography websites, Photog magazines (e.g DigitalPhoto Pro), local photography shops (In Adelaide : Total Photo, Twin City Camera House, ProAV solutions, Diamonds, TEDs, Photographic Wholesalers In the US : Adorama, B&H).
Revenue Streams : This will have to be tested and worked out with first customers. A monthly tiered pricing structure, a pay for what you use system (cost per HDD courier and cost of online storage), or a bundled pack with something like a Drobo.
Cost Structure : The cost of storage for the backups, the courier costs, portable HDD costs and application development costs. One option for helping the courier costs could be by partnering with a delivery company. Even Domino’s Pizza. Order a pizza and they’ll take your drive.
Key Partners : Amazon Web Services (especially for the S3 storage and AWS Import/Export). A courier company (such as FedEx, Toll IPEC, AAE) and hopefully an existing backup app developer (MemoryBox, BackBlaze, Jungle Disk, Dropbox?).
Key Activities : Occasionally couriering a Hard Drive to customers, with drives received from them having their data uploaded to the cloud for backup. Also running a desktop application to help upload smaller changes and detect larger data changes which get encrypted and copied onto the external hard drive which gets sent.
Key Resources : Being about data storage the biggest requirement is hard drives, lots and lots of them.
Spare Portable Hard drives will likely be kept at the Photographers premises, connected up to their computer until they need to send it off.
Growth Model : Once the service has taken off it can be scaled by going to cheaper storage, although that requires some CapEx.
Lean Startup
Start small. No coding, just a landing page. Then get some photogs on board and get them to send you an hard drive. If they do then start uploading their data to the cloud (can be done via Amazon Web Services batch import). It’s trivial to write or rebrand a web app which lets them view their S3 bucket (each photog would have their own). Can start working on a backup app, possibly partnering with an existing backup provider (Backblaze, Memory Box, etc..), or with a Drobo app.
- Encrypt couriered HDD.
- Add online album/gallery option.
- Print from album (people viewing a wedding album for example can order prints which get sent to a local print ).
- Add online processing options (e.g face detection and tagging).
- Start looking into using BackBlaze style storage pods (which would require running your own data centre equipment, but could dramatically reduce storage costs).
![[Diagram] BackBlaze : Cost of storage](http://blog.backblaze.com/wp-content/uploads/2009/08/cost-of-a-petabyte-chart.jpg "BackBlaze - Cost of storage")
Evolution
After creating a working business which is successfully dealing with photographers the service can be expanded to videographers, researchers and other data intensive applications.
Collections and organising photos
So far of the 4 photographers I’ve talked to (including myself), Adobe Lightroom seems to be the most convenient way of organising and dealing with large quantities photos.
The ability to utilise a photographers lightroom collection and create an online album from stared (or rated/tagged) photos would be very useful.
Customer Validation and Understanding
I’ve talked to some potential customers so far. The first being Benjamin Liew, a photographer in Adelaide. Another being an event photographer in Sydney. Ben currently has a collection of about 3 or 4 portable/external HDDs, some of which he tries to keep at his office with others at his home, plus whatever he is working on is on the CF cards and on his Macbook.
The photographer in Sydney meets up with a friend every week or two and they exchange hard drives, so each has a copy of the others. But what happens when his friend goes away for a while? Or what happens if the office burns down and his friend happens to get broken into at about the same time? Unlikely, yes. But Murphy’s law has a habit of striking when you are down.
The problem with teaching Mastery
The problem with teaching Mastery of a craft is that those who have it can’t explain it.
Imagine this, you are a budding animator. You are trying to learn how to create amazing visual animations (2d, 3d, puppetry.. take your pick). You join a forum and find there are a range of articles and information of interest. Occasionally you come across some real nuggets of information. But after a year you realise that those people with real mastery of the craft aren’t on the forums and rarely do they even post something more than a couple of tips or an interview. The people you are spending most of your time talking to and learning from are maybe only a year or two more advanced than yourself.
It’s been widely established that to be particularly amazing at a craft you need to have been passionately doing it for 10 years.
To have a proper mastery of your craft, be it cooking, programming, sales or engineering, you need to have been actively learning and getting better at it for about a decade.
The problem seems to be in trying to explain and teach other people the techniques things that you do. When you have achieved such a mastery of the craft you go into what is known as a state of Flow. So much of your brain power is being dedicated to the task at hand that it is like you are having an out of body experience.
My suggestion and I don’t know how well it would work, but is probably worth a try, is to assign a documenting trainee to people who have achieved such mastery. Those pushing the edge of their field should have someone who is maybe only 4 or 5 years into their learning of the craft to not just be a trainee, but to document what they are learning and hopefully attempt to understand what the master is doing and why. Then to share those insights online.
Documenting can obviously be done in a wide range of ways. From simple notes to photos and a full audio/video recording.
My guess is that the skills that such people have honed aren’t something that can be shared with just a couple of hints and tips. A master cook’s secrets aren’t hidden in his recipe, a master programmers abilities extend beyond just the final code. It is the underlying process of getting to the final product which is what a new trainee should be interested in learning and sharing with others.
If someone is interested in trying to document a master of their craft, I’d be interested in how well it works!
Do it smart and do it with passion and purpose
My new mantra is ”If I am going to do it, I’m going to do it smart and with passion and purpose“.
For many years I’ve meditated. I would quite often do it when stressed or tired in order to motivate me.
Whilst meditating it is best to do something, or think about something and keep repeating that. Some people make “Ooohm” like noises, others picture a ball circling around. I would think to myself “I can do it“.
There are some issues with thinking that and thus I have decided my new mantra is “If I am going to do it, I’m going to do it smart and with passion and purpose“.
This invariably gets shortened to “Do it smart and with passion and purpose“.
Update – Lean Startup, Interface Options and a need to change.
So. I was reading the 37 Signals online book ‘Getting Real‘, which is very much about Lean Startup when it finally struck me how I can use the lean startup approach to get the web platform working TOMORROW (which was actually 2 days ago).
We have a Flash application, a modified version of the RaVis explorer, which would allow us to create the imagery for the Nodemap relationship, however it isn’t working 100% and I couldn’t get the required Javascript mouseover, and click events with the extra information of which ID the video thumbnail the person was hovering over was.
I fixed this by screenshotting the Nodemap then using an Imagemapping application to create the require Img map.
For those that don’t know an Image Map is a piece of HTML which has the co-ordinates of the various shapes on the image and allows you to click on the different areas and go to different links. By adding the videoID information to each area, in this case as a data-videoid attribute, (although could have grabbed it from the href), then using some jQuery magic I finally got exactly what I wanted. A Nodemap which I could play with.
I’m now in the process of creating 4 different Nodemap views (by manually creating different layouts then screenshotting them and Image Mapping them), plus 4 different interfaces.
Interface Options :
Infobox - When you hover over a film thumbnail a section of the webpage which I call the Infobox shows the information about that film. If you click the infobox it’ll load up Flowplayer and you can watch the video. Currently if you are watching a video but then hover over another thumbnail then the Infobox changes and the video disappears. This is the only version I’ve implemented so far.
Light Box – Displays the video over top of the page.
Video Page – Hover over a thumbnail and it’ll display information about the video. Click it and it’ll take you directly to the video page.
Playlist – Allows you to drag and drop the videos into your playlist.
Platform Staging Site - Yes, I know the Nodemap goes over the site width, will fix that.
Right now the site needs a good overhaul regarding the styling and look and feel. I’ve been putting out a call for a UI designer, asking everyone I got business cards from at the X Media Lab, however haven’t had any luck, although have had a lot of people forwarding the request onto their friends and have at least had a decent response rate since putting the WHY in front of the explanation (as per the previous post).
I started to get a bit depressed and pretty much wasted all of yesterday just watching some online videos and not doing any programming. I felt myself starting to loose enthusiasm and energy. Starting to feel stressed and lethargic, yet am so close to getting it ready for a limited beta trial. I realised that I hadn’t been getting enough exercise and after doing something akin to the beep test for 5mins (running between two points), I already started to feel a lot better. Now I feel like I just need someone to bounce ideas off. A Technical co-founder.
Actually, my girlfriend and I talked and we realised that something needs to change or else I’ll run out of money before the end of August and I’ll have burnt out before then anyway. Meaning I’ll likely have to go back to film production work and will miss the current window of opportunity.
To make things happen I really need to change the environment I’m in, get some help and also get some more finances
Environment – Currently I’m either sitting in bed programming on the laptop, or occasionally going to the office in the city, although usually then end up having to do other work such as networking with people or basic admin stuff so don’t get much programming done.
What I really need is what I came to Sydney for. I need to be around other web developers. I need to talk to them, hear their problems, bounce ideas off them. I came here wanting to have a spare seat at the Pollenizer office, but they were busy moving. I then wanted to get a place at Fishburners, but they wouldn’t let me stay cause I was only going to be there for 3 months and they wanted me to commit to 6 months. I think things would be very different (and likely far more advanced) if they could’ve made an exception. They wanted someone for longer so that they could have people who contribute to the atmosphere. I could have whipped up a storm in there if they needed. Ohh well. Instead I’m staying at the office RIGHT NEXT to Fishburners, it is a decently sized open office with a couple of sales staff, but I usually only go there from 6pm until midnight as there is no one there and I can concentrate slightly more.
All of this means that the feeling of being alone is creeping up on me and taking hold of me. I especially miss my girlfriend and can’t wait to see her at the start of next month as she visits Sydney
Help - As mentioned above, I need someone to bounce ideas off, I also need a graphic, web or UI designer to help with the site. I have a very good understanding of design, but I’m channelling my creativity into programming not design. Another PHP developer would also be really awesome, although it would likely take a while to get them up to speed on everything. In the near future I’ll also need some help redoing the database design to make it more flexible so we can try out a whole lot more stuff.
Finances – My savings have nearly run out and I’m starting on my Credit Card debt. I accidentally went over my mobile phone cap so instead of paying $69 for the month it’s now over $450. That, plus some other bills means I no longer have the $500 earmarked for a designer. I’m also questioning if I’ll be able to stay in Sydney much longer. Whilst I’d love $150K to actually get this platform off the ground, I will at least need a few grand to be able to show its potential. Even $5K would make an awesome difference.
Unfortunately the next Sydney Angels meeting isn’t until ~30th of August and the next Adelaide meeting is in July but I doubt I’ll be able to make it back in time even if I somehow managed to get accepted to be seen. I was hoping to put up an online video of me explaining the Business Model, however the hard drive with that on it got wiped as I travelled from Adelaide to Sydney so I’ll have to re-film it when my girlfriend comes to visit.
If anyone knows of any Angel investors who are interested in Web 2.0 startups and want to help change the face of Entertainment and Education (and be a thought leader in economics) then let me know
I’m not going to give up. I might be temporarily subdued. But the real problem is that if things don’t start changing soon then we are going to miss the window of opportunity and the next wave likely won’t have the same impact as catching this one will.
Some Interesting TED talks – Flow and WHY?
Flow is the state of being whereby you are working with such concentration that you almost don’t feel like you are controlling yourself, the work is just flowing through you. However this only happens when you are a master in your field. When you have been honing your skill for over 10 years. I’ve still got a couple more years before I get to this point when programming in PHP, although have very nearly reached it when bash scripting.
Simon Sinek: How great leaders inspire action
Whist Simon talks about money too much in this talk for my liking the core point has very interesting ramifications for any marketing department. The core concept is that you need to talk about WHY you are doing something before talking about how you are doing it and what you are doing. As an aside I think it is even more important when teaching Maths.
I particularly like the line about a leader is someone who holds a position of power or authority, but those who lead inspire us. In this case I’ve realised that I’ve not been able to inspire people nearly as much as I’d hoped because I was going about it the wrong way.
Previously at a conference when people asked what I do I’d explain the Multipath storytelling concept then explain that I was writing a web platform for that. But always in the back of my mind I was thinking of the why as my motivator.
Now I should change it and explain that I am trying to foster new paradigm changes in entertainment, education and economics. Entertainment through the Multipath concept, education through the collaborative communities and economics through the for-benefit principles.
Changing Torrent – Bigger than Dropbox?
As part of my work with the Zeitgeist Movement and in thinking about collaborative communities, one of the tools we currently use to share files is Dropbox.
It’s a great way of transferring files between computers. It keeps old versions, lets you undelete and automatically synchronises the files between the computers. You can also share the folders with other Dropbox users, which is what is especially useful in a collaborative community environment.
The problem is that the entire structure of Dropbox means that your files get saved to the cloud (Amazon S3) and require servers to synchronise the changes. The system effectively uses a combination of rSync and SVN. Rsync only transfers the small changes you’ve made to a file and SVN allows you to keep versioning information.
However there are still storage, bandwidth and processing costs involved which make the system reasonably hard to scale at near free prices.
I had a thought. What if you created a Dropbox style system, but based on Bittorent?
Currently if you upload file(s) using Bittorent you need to create a .torrent manifest file, which can’t really allow changes to the group of files without invalidating them. But imagine an ever changing .torrent file. Files could be updated and the updates transferred between peers… actually I would think of it like a Bittorent folder. You put new files in the folder and instead of being directly uploaded to the cloud, they are peered to your friends. Instead of being limited to a default 2GB or so free account, you would be limited by your Internet connection’s quota and your hard drive space.
My guess is that a combination of rSync for minor file changes and bittorent for new files or large file changes would be the most appropriate. Versioning control would be the hardest as running SVN on files of such size would be a pain. Git would likely be a better alternative.
This will be especially useful for video and audio data or anything which is usually far too big to transfer normally.
Some use cases :
Video Editors : Sunny Wu and I often film a project, but have issues editing it. Even miniDV (SD or HDV) footage is 12GB per hour. When you are trying to edit 3 hours of footage down to a 5min video that’s not something you can put on the Dropbox. We currently use a draft version of the video, exported as ~iPod sized. It looks shite but we get enough of an idea that we can critique it. The down side is that even though we both have a copy of the raw video, I don’t have his copy of the Vegas Video project. Whilst the project file could be stored in the Dropbox, we both keep the video files in different places on our computer. Now thankfully Vegas is good at re-mapping where the files are stored, however having to do this every time I wanted to make a change would make us go crazy. However if we had a Bittorent/rSync style synchronisation, then we would both be able to edit the video very easily. Your usually only changing the project file and occasionally adding in some extra audio and images, maybe 50-500MB worth.
In this case, the ability to create multiple torrent shared folders anywhere on your computer is important.
Audio Engineers or narrators : Lets say I’ve got a video file that someone’s exported for me. From 50MB to 1GB in size. I now need to record a high quality voice over (i.e uncompressed raw). This could be a few GB in size. The Audio Engineer has to first add some filters and sound effects before the video editor can use them. These three people would be able to easily access the shared folder and work together.
Collaborative Communities : With the Films on the Fly platform there will be various sites, communities, groups and organisations. These will want to collaborate in different ways. On top of that each film, or even film in planning (project), will be able to upload various resources associated with the film. These could be the 3D Animation files, the script, headshots of the actors, even the raw video footage. By letting people download these they can use the content to extend and support the rich stories and information being created and shared. In this case the Films on the Fly servers would likely act as a seeder. Ensuring no project resources end up dead.
Libraries : You want to share your (creative commons licensed) music library with your friends. You want them to contribute to it as well. Or maybe you run an actual library and want to synchronise new ebooks between the 500 other libraries in the network.
Quick Test : An easy way to test if this is something worth pursuing is to use Dropbox or similar tech and let people put .torrent files into a folder. Others can then run a bittorent client on their machine which monitors the folder and loads new torrents from it.
I already use such a system for general torrents. But to have the level of control I’m advocating would require creating new .torrent files and running your own torrent server.
Existing Dropbox Alternatives
These are just some of the many alternatives to Dropbox.
Sparkles Share – http://sparkleshare.org : Free and Open Source, but Linux and Mac only. Can use Github or you can run your own server.
Memory Box – http://www.memorybox.net.au : Peer to Peer Distributed hard drive backups. A similar use case to Backblaze, but the data is encrypted and stored on other peoples computers.
Box.net – http://box.net/ : Similar to Dropbox but aimed more at the Enterprise market so is more extensible.
Note : I know Dropbox has paid accounts. I have one myself. I still love Dropbox and even if such a peer to peer synchronisation system was setup, I’d likely still use Dropbox for a number of things.