CFBackup, a simple backup utility for Cloud Files

jmstacey

I've just released CFBackup, a small ruby program that transfers files or directories from the local machine to Mosso Cloud Files. It is meant to serve as a useful tool for automated backups. If you're interested, you can download it and find out more on the project page: http://jonsview.com/projects/cfbackup.

One of the key features is the "--local_net" argument, which will upload files over the unmetered local network (i.e. you don't pay for transfer, only storage). In order for this to work you must meet the following two conditions:
1. Your slice is in DFW1, or you have a Cloud Server.
2. You have a private IP with the necessary static route setup (see http://forum.slicehost.com/post.php?CategoryID=6)

I will warn you that this project is in its infancy, so let me know about any problems that you run into. If you know a thing or two about ruby I'll take patches any day :-)

bartee

GREAT idea. I will look at it. I have been concerned with a method of backing up my mysql database on a daily basis... This should work and give me an intro into Mosso...

Thanks

jmstacey

I've pushed the repository up to GitHub (Repository | Wiki) and released 0.0.4.

This version contains an experimental feature to pipe data straight to a Cloud Files container, skipping the intermediate temp directory. For example, you could pipe a MySQL dump straight to Cloud Files with a command like this:

mysqldump -u root -p --all-databases | gzip -9 | ruby cfbackup.rb --pipe_data --container backups:mysql_backup.sql.gz

Something like this might be useful when there is not enough disk space to contain the temporary backups before upload.

jspringman

I tried your gem out, but the --local-net argument doesn't work. I can see a small amount of traffic to and from 10.191.196.33, but no files are pushed. When I leave out that argument, it works perfectly. I think Mosso has changed something that has caused the private IP to cease working.

jmstacey

jspringman, my nightly slice backups over local_net are working. I also just tried a single file push to confirm and everything is working on my end. Do you get a response when pinging the storage network (snet-storage.clouddrive.com)?

Edit: Well if you're seeing traffic... CFBackup has very limited error handling, so no errors will show up unless something throws an unhandled exception, which authentication failure will last I checked. Do you get the full "Establishing connection... done." string when simply running "cfbackup -v"?

jspringman

jmstacey, it's encouraging that you have no problems. I can ping snet-storage.clouddrive.com; I can even telnet to its port 443. I do get confirmation of the connection when I run cfbackup, but after that, it just stalls. If I remove the --local-net argument, it works. Let me know if you have any ideas.

jmstacey

Have you tried pulling or deleting files to see if those are also dysfunctional? If you're not averse to getting your hands a little dirty I would try establishing a connection directly with cURL to rule out any ruby library issues. The Cloud Files developer documentation contains some samples around page 38. If you do successfully get an auth token returned, start stepping through the other examples with cURL. Remember to swap out the URL given to you with the auth token with the snet-storage one.

jspringman

Everything works fine so long as I don't use snet-storage.clouddrive.com:

@$ curl -D - -H "X-Auth-Key: xxxxxxxxxxxxxxxxxxxxxxx" -H "X-Auth-User: xxxxxxxxxx" \
> https://api.mosso.com/auth
HTTP/1.1 204 No Content
Date: Mon, 22 Jun 2009 23:06:36 GMT
Server: Apache/2.2.3 (Mosso Engineering)
X-Storage-Token: xxxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
X-Storage-Url: https://storage4.clouddrive.com/v1/MossoCloudFS_e4fd555b-757a-436e-973b-0a80521afff7
X-CDN-Management-Url: https://cdn.clouddrive.com/v1/MossoCloudFS_e4fd555b-757a-436e-973b-0a80521afff7
X-Auth-Token: xxxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
Content-Length: 0
Connection: close
Content-Type: application/octet-stream@

@$ curl -X "GET" -D - -H "X-Auth-Token: xxxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" \
> https://storage4.clouddrive.com/v1/MossoCloudFS_e4fd555b-757a-436e-973b-0a80521afff7/test/test.txt
HTTP/1.1 200 OK
Date: Mon, 22 Jun 2009 23:07:09 GMT
Server: Apache
Last-Modified: Mon, 22 Jun 2009 23:05:21 GMT
Etag: 3f3309312cc3264a1389b2d263dece8d
Content-Length: 20
Content-Type: text/plain@

@This is a text file.@

@$ curl -X "GET" -D - -H "X-Auth-Token: xxxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" \
> https://snet-storage.clouddrive.com/v1/MossoCloudFS_e4fd555b-757a-436e-973b-0a80521afff7/test/test.txt
HTTP/1.1 401 Authorization Required
Date: Mon, 22 Jun 2009 23:08:00 GMT
Server: Apache
Transfer-Encoding: chunked
Content-Type: text/html@

@401 Unauthorized401 Unauthorized

jmstacey

The only response difference that I can see is X-Storage-Url which is https://storage.clouddrive.com/v1... on my end. I don't think I've ever gotten any other subdomain. I was told when I first wrote cfbackup (~3 months ago) that they were eventually going to return the internal/external URL as part of the authentication, but I don't know if that has actually happened yet.

I would definitely start barking up some trees. While support is only marginally better than Amazon's (virtually nonexistent) I can at least say that S3 worked reliably. If we could get free transfer with S3 I'd switch back.

jspringman

It seems to me that I need the private IP of storage4.clouddrive.com or another subdomain that resolves to that private IP (like snet-storage4.clouddrive.com, which doesn't work). I wonder how long it will take to get that.

jspringman

The Cloud Files people got this straightened out for me today; snet-storage4.clouddrive.com is resolving, and I'm able to transfer files over the local network. The support was not exactly fanatical, but it came through eventually.

jmstacey

Did they say what the exact problem was and if it's possible to resolve without a ticket?

Categories