bup - backup system based on the git packfile format...

Share interesting information or links related to portable apps here.

Moderators: usdcs, Andrew Lee, webfork

Post Reply
Message
Author
User avatar
lintalist
Posts: 173
Joined: Sat Apr 19, 2014 12:52 am
Contact:

bup - backup system based on the git packfile format...

#1 Post by lintalist » Fri Aug 19, 2016 12:28 pm

The Rclone - CLI cloud backup via rsync post by shnbwmn reminded me of bup. I didn't see a post about it yet so here it is...

bup https://github.com/bup/bup

It currently only works on Linux, FreeBSD, NetBSD, OS X >= 10.4, Solaris, or Windows + Cygwin
Very efficient backup system based on the git packfile format, providing fast incremental saves and global deduplication (among and within files, including virtual machine images).

bup has a few advantages over other backup software:
  • It uses a rolling checksum algorithm (similar to rsync) to split large files into chunks. The most useful result of this is you can backup huge virtual machine (VM) disk images, databases, and XML files incrementally, even though they're typically all in one huge file, and not use tons of disk space for multiple versions.
  • It uses the packfile format from git (the open source version control system), so you can access the stored data even if you don't like bup's user interface.
  • Unlike git, it writes packfiles directly (instead of having a separate garbage collection / repacking stage) so it's fast even with gratuitously huge amounts of data. bup's improved index formats also allow you to track far more filenames than git (millions) and keep track of far more objects (hundreds or thousands of gigabytes).
  • Data is "automagically" shared between incremental backups without having to know which backup is based on which other one - even if the backups are made from two different computers that don't even know about each other. You just tell bup to back stuff up, and it saves only the minimum amount of data needed.
  • You can back up directly to a remote bup server, without needing tons of temporary disk space on the computer being backed up. And if your backup is interrupted halfway through, the next run will pick up where you left off. And it's easy to set up a bup server: just install bup on any machine where you have ssh access.
  • Bup can use "par2" redundancy to recover corrupted backups even if your disk has undetected bad sectors.
  • Even when a backup is incremental, you don't have to worry about restoring the full backup, then each of the incrementals in turn; an incremental backup acts as if it's a full backup, it just takes less disk space.
  • You can mount your bup repository as a FUSE filesystem and access the content that way, and even export it over Samba.
  • It's written in python (with some C parts to make it faster) so it's easy for you to extend and maintain.
This is a very early version

User avatar
webfork
Posts: 7516
Joined: Wed Apr 11, 2007 8:06 pm
Location: US, Texas
Contact:

Re: bup - backup system based on the git packfile format...

#2 Post by webfork » Wed Aug 31, 2016 3:36 pm

Data is "automagically" shared between incremental backups without having to know which backup is based on which other one - even if the backups are made from two different computers that don't even know about each other. You just tell bup to back stuff up, and it saves only the minimum amount of data needed.
If true, that's incredible. I can see two system backups having quite a bit in common. Many file systems (including dropbox) do something called de-duplication to zap extra files, but for incremental backups? That's a neat trick.

Also the ability to mount the repository and view it over Samba is ... well, that's amazing.

The only thing that keeps me from going towards this is the "very early version" note. Still, very stable systems don't get that way in isolation: someone's got to break them. Hopefully someone is willing to dig into that.
Supporting Net Neutrality - BattleForTheNet | Why this matters | More from EFF.org

User avatar
lintalist
Posts: 173
Joined: Sat Apr 19, 2014 12:52 am
Contact:

Re: bup - backup system based on the git packfile format...

#3 Post by lintalist » Thu Sep 01, 2016 8:59 am

I'm probably going to try this in a few months just to see how it works, there is (four year old) talk on BUP on https://www.youtube.com/watch?v=N5qj94B3WkE

You can browse BUP repositories via your web browser which is nice ("bup web starts a web server that can browse bup repositories" https://github.com/bup/bup/blob/master/ ... bup-web.md )

The "very early version" is probably used as a catch all disclaimer - as it has been around for a while it should work.

I'm curious to see how it would handle a very large email archive (mbox format).

User avatar
lintalist
Posts: 173
Joined: Sat Apr 19, 2014 12:52 am
Contact:

Re: bup - backup system based on the git packfile format...

#4 Post by lintalist » Fri Jan 19, 2018 12:18 pm

Similar it seems https://github.com/borgbackup/borg

Easy installation on multiple platforms
We offer single-file binaries that do not require installing anything - you can just run them on these platforms:
  • Linux
  • Mac OS X
  • FreeBSD
  • OpenBSD and NetBSD (no xattrs/ACLs support or binaries yet)
  • Cygwin (experimental, no binaries yet)
  • Linux Subsystem of Windows 10 (experimental)

Post Reply