From baa6f68706406c19af17d68acaa82c4bc40b1a8f Mon Sep 17 00:00:00 2001 From: Marco Cetica Date: Wed, 3 Apr 2024 08:55:06 +0200 Subject: [PATCH] Added integrity check option(--checksum) --- README.md | 88 +++++++++---- backup.sh | 115 +++++++++++----- backup.sh.1 | 369 ++++++++++++++++++++++++++++++++++++---------------- man.md | 126 +++++++++++------- tests.sh | 8 +- 5 files changed, 490 insertions(+), 216 deletions(-) diff --git a/README.md b/README.md index 906d358..b8a6112 100644 --- a/README.md +++ b/README.md @@ -1,14 +1,9 @@ # backup.sh ![](https://github.com/ceticamarco/backup.sh/actions/workflows/backup.sh.yml/badge.svg) `backup.sh` is a POSIX compliant, modular and lightweight backup utility to save and encrypt your files. This tool is intended to be used on small scale UNIX environments such as VPS, personal servers and -workstations. `backup.sh` uses [rsync](https://linux.die.net/man/1/rsync), [tar](https://linux.die.net/man/1/tar) -and [gpg](https://linux.die.net/man/1/gpg) to copy, compress and encrypt the backup. - -`backup.sh` works under the following operating systems: -- GNU/Linux; -- OpenBSD -- FreeBSD; -- Apple MacOS. +workstations. `backup.sh` uses [rsync](https://linux.die.net/man/1/rsync), [tar](https://linux.die.net/man/1/tar), +[gpg](https://linux.die.net/man/1/gpg) and [sha256sum](https://linux.die.net/man/1/sha256sum) +to copy, compress, encrypt the backup and verify the backup. ## Installation `backup.sh` consists in a single source file, to install it you can copy the script wherever you want. @@ -30,18 +25,23 @@ To show the available options, you can run `backup.sh --help`, which will print ```text backup.sh - POSIX compliant, modular and lightweight backup utility. -Syntax: ./backup.sh [-b|-e|-h] +Syntax: ./backup.sh [-b|-c|-e|-h] options: --b|--backup SOURCES DEST PASS Backup folders from SOURCES file. --e|--extract ARCHIVE PASS Extract ARCHIVE using PASS. --h|--help Show this helper. +-b|--backup SOURCES DEST PASS Backup folders from SOURCES file. +-c|--checksum Generate/check SHA256 of a backup. +-e|--extract ARCHIVE PASS Extract ARCHIVE using PASS. +-h|--help Show this helper. + +General help with the software: https://github.com/ceticamarco/backup.sh +Report bugs to: Marco Cetica() ``` -As you can see, `backup.sh` supports two options: **backup creation** and **backup extraction**, the former requires -root permissions, while the latter does not. Let us see them in details. +As you can see, `backup.sh` supports three options: **backup creation**, **backup extraction** and **checksum** to verify the +integrity of a backup. The first option requires +root permissions, while the second one does not. The checksum option must be used in combination of one of the previous options. ### Backup creation -To specify the directories to backup, `backup.sh` uses an associative array +To specify the directories to back up, `backup.sh` uses an associative array defined in a text file(called _sources file_) with the following syntax: ```text @@ -68,7 +68,7 @@ backup-ssh- ``` You can add as many entries as you want, just be sure to use the proper syntax. In particular, -the _sources file_, **should not** includes: +the _sources file_, **should not** include: - Spaces between the label and the equal sign; - Empty lines; - Comments. @@ -88,19 +88,26 @@ In the previous example, this would be: $> sudo ./backup.sh --backup sources.bk /home/john badpw1234 ``` +You can also tell `backup.sh` to generate a SHA256 file containing the hash of each file using the `-c` option. +In the previous example, this would be: +```sh +$> sudo ./backup.sh --checksum --backup sources.bk /home/john badpw1234 +``` + The backup utility will begin to copy the files defined in the _sources file_: ```text Copying nginx(1/2) Copying ssh(2/2) Compressing backup... Encrypting backup... -File name: /home/marco/backup--.tar.gz.enc +File name: /home/john/backup--.tar.gz.enc +Checksum file: /home/john/backup--.sha256 File size: 7336400696(6.9G) -File hash: 0e75ca393117f389d9e8edfea7106d98 Elapsed time: 259 seconds. ``` -After that, you will find the final backup archive in `/home/john/backup--.tar.gz.enc`. +After that, you will find the backup archive and the checksum file in +`/home/john/backup--.tar.gz.enc` and `/home/john/backup--.sha256`, respectively. You can also use `backup.sh` from a crontab rule: ```sh @@ -114,7 +121,8 @@ key is stored in a local file(with fixed permissions) to avoid password leaking adopt this practice while using the `--extract` option to avoid password leaking in shell history. ### Backup extraction -`backup.sh` can also extract the encrypted backup archive using the following syntax: +`backup.sh` can also be used to extract the encrypted backup as well to verify the integrity +of the backup data. To do so, use the following commands: ```sh $> ./backup.sh --extract @@ -128,16 +136,31 @@ For instance: $> ./backup.sh --extract backup--.tar.gz.enc badpw1234 ``` -This will create a new folder called `backup.sh.tmp` in your local directory. Be sure to rename any directory -with that name to avoid collisions. From the previous example, you should have the following directories: +This will create a new folder called `backup.sh.tmp` in your local directory with the following content: ```text backup-nginx- backup-ssh- ``` +**note:**: be sure to rename any directory with that name to avoid collisions. + + +Instead, if you also want to verify the integrity of the backup data, use the following commands: +```sh +$> ./backup.sh --checksum --extract +``` + +For instance: + +```sh +$> ./backup.sh --checksum --extract backup--.tar.gz.enc badpw1234 $PWD/backup--.sha256 +``` + +**note:** be sure to provide the ABSOLUTE PATH of the checksum file. ## How does backup.sh work? -**backup.sh** uses _rsync_ to copy the files, _tar_ to compress the backup and _gpg_ to encrypt it. +**backup.sh** uses _rsync_ to copy the files, _tar_ to compress the backup, _gpg_ to encrypt it and +_sha256sum_ to verify it. By default, rsync is being used with the following parameters: ``` @@ -153,10 +176,15 @@ That is: - q: quiet mode: reduces the amount of information rsync produces; - delete: delete mode: forces rsync to delete any extraneous files at the destination dir. +If specified(`--checksum` option), `backup.sh` can also generate the checksum of each file of the backup. +To do so, it uses `sha256sum(1)` to compute the hash of every single file using the SHA256 hashing algorithm. +The checksum file contains nothing but the checksums of the files, no other information about the files stored +on the backup archive is exposed on the unencrypted checksum file. This may be an issue if you want plausible +deniability(see privacy section for more information). + After that the backup folder is being encrypted using gpg. By default, it is used with the following parameters: - ``` $> gpg -a \ --symmetric \ @@ -177,6 +205,18 @@ This command encrypts the backup using the AES-256 symmetric encryption algorith - `--output`: Specify output file; - `$INPUT`: Specify input file. +## Plausible Deniability +While `backup.sh` provide some pretty strong security against bruteforce attack(assuming a strong passphrase is being used) +it should by no means considered a viable tool against a cryptanalysis investigation. Many of the copying, compressing and +encrypting operations made by `backup.sh` during the backup process can be used to invalidate plausible deniability. +In particular, you should pay attention to the following details: + +1. The `--checksum` option generates an **UNENCRYPTED** checksum file containing the _digests_ of **EVERY** +file in your backup archive. If your files are known to your adversary(e.g., a banned book), they may use a rainbow table attack to +determine whether you own a given file, voiding your plausible deniability; +2. Since `backup.sh` is essentially a set of shell commands, an eavesdropper could monitor the whole backup process to extract +the name of the files or the encryption password. + ## Unit tests `backup.sh` provides some unit tests inside the `tests.sh` script. This script generates some dummy files inside the following directories: diff --git a/backup.sh b/backup.sh index 594d22c..ddd8137 100755 --- a/backup.sh +++ b/backup.sh @@ -13,19 +13,23 @@ # logs=/var/log/ # # After that you can launch the script with(sample usage): -# sudo ./backup.sh --backup sources.bk /home/john badpw1234 +# sudo ./backup.sh --checksum --backup sources.bk /home/john badpw1234 # # This will create an encrypted tar archive(password: 'badpw1234') -# in '/home/john/backup--.tar.gz.enc' containing +# in '/home/john/backup--.tar.gz.enc' containing # the following three directories: # backup-nginx- # backup-ssh- # backup-logs- # -# You can then decrypt it using: -# ./backup.sh --extract backup--.tar.gz.enc badpw1234 +# as well as a SHA256 file('/home/john/backup--.sha256') +# containing the file hashes of the backup. # -# You can read the full guide on https://github.com/ice-bit/backup.sh +# You can then decrypt it using: +# ./backup.sh --checksum --extract backup--.tar.gz.enc badpw1234 $PWD/backup--.sha256 +# which will also check the integrity of the backup(optional feature). +# +# You can read the full guide on https://github.com/ceticamarco/backup.sh # or on the manual page. # Copyright (c) 2018,2023,2024 Marco Cetica # @@ -47,29 +51,23 @@ checkdeps() { fi } -checksum() { - BACKUP_SH_FILENAME="$1" - BACKUP_SH_OS="$(uname | tr '[:lower:]' '[:upper:]')" - - if [ "$BACKUP_SH_OS" = "LINUX" ]; then - RES="$(md5sum "$BACKUP_SH_FILENAME" | awk '{print $1}')" - else - RES="$(md5 -q "$BACKUP_SH_FILENAME")" - fi - - echo "$RES" -} - +# $1: sources.bk file +# $2: output path +# $3: password +# $4: compute sha256(0,1) make_backup() { BACKUP_SH_SOURCES_PATH="$1" BACKUP_SH_OUTPATH="$2" BACKUP_SH_PASS="$3" + BACKUP_SH_SHA256="$4" + BACKUP_SH_COMMAND="rsync -aPhrq --delete" BACKUP_SH_DATE="$(date +'%Y%m%d')" BACKUP_SH_FOLDER="backup.sh.tmp" BACKUP_SH_OUTPUT="$BACKUP_SH_OUTPATH/$BACKUP_SH_FOLDER" BACKUP_SH_START_TIME="$(date +%s)" BACKUP_SH_FILENAME="$BACKUP_SH_OUTPATH/backup-$(uname -n)-$BACKUP_SH_DATE.tar.gz.enc" + BACKUP_SH_CHECKSUM_FILE="$BACKUP_SH_OUTPATH/backup-$(uname -n)-$BACKUP_SH_DATE.sha256" # Check for root permissions if [ "$(id -u)" -ne 0 ]; then @@ -96,6 +94,13 @@ make_backup() { mkdir -p "$BACKUP_SH_SUBDIR" printf "Copying %s(%s/%s)\n" "$label" "$BACKUP_SH_PROGRESS" "$BACKUP_SH_TOTAL" + + # Compute SHA256 of all files of the current directory + if [ "$BACKUP_SH_SHA256" -eq 1 ]; then + find "$path" -type f -exec sha256sum {} + | sort -k 2 | awk '{print $1}' >> "$BACKUP_SH_CHECKSUM_FILE" + fi + + # Copy files $BACKUP_SH_COMMAND "$path" "$BACKUP_SH_SUBDIR" BACKUP_SH_PROGRESS=$((BACKUP_SH_PROGRESS+1)) done < "$BACKUP_SH_SOURCES_PATH" @@ -124,20 +129,24 @@ make_backup() { BACKUP_SH_END_TIME="$(date +%s)" BACKUP_SH_FILE_SIZE="$(find "$BACKUP_SH_FILENAME" -exec ls -l {} \; | awk '{print $5}')" BACKUP_SH_FILE_SIZE_H="$(find "$BACKUP_SH_FILENAME" -exec ls -lh {} \; | awk '{print $5}')" - BACKUP_SH_HASH="$(checksum "$BACKUP_SH_FILENAME")" echo "File name: $BACKUP_SH_FILENAME" + [ "$BACKUP_SH_SHA256" -eq 1 ] && { echo "Checksum file: $BACKUP_SH_CHECKSUM_FILE"; } echo "File size: $BACKUP_SH_FILE_SIZE($BACKUP_SH_FILE_SIZE_H)" - echo "File hash: $BACKUP_SH_HASH" printf "Elapsed time: %s seconds.\n" "$((BACKUP_SH_END_TIME - BACKUP_SH_START_TIME))" } +# $1: archive file +# $2: archive password +# $3: sha256 file(optional) extract_backup() { BACKUP_SH_ARCHIVE_PATH="$1" BACKUP_SH_ARCHIVE_PW="$2" + BACKUP_SH_SHA256_FILE="$3" # Decrypt the archive gpg -a \ + --quiet \ --decrypt \ --no-symkey-cache \ --pinentry-mode=loopback \ @@ -148,7 +157,20 @@ extract_backup() { # Extract archive tar -xzf backup.sh.tar.gz 1> /dev/null 2>&1 - # Remove temporary files + # If specified, use SHA256 file to compute checksum of files + if [ -n "$BACKUP_SH_SHA256_FILE" ]; then + for file in $(find "backup.sh.tmp" -type f | sort -k 2); do + # Compute sha256 for current file + sha256="$(sha256sum "$file" | awk '{print $1}')" + # Check if checksum file contains hash + if ! grep -wq "$sha256" "$BACKUP_SH_SHA256_FILE"; then + printf "[FATAL] - integrity error for '%s'.\n" "$file" + rm -rf backup.sh.tar.gz backup.sh.tmp + exit 1 + fi + done + fi + rm -rf backup.sh.tar.gz } @@ -158,13 +180,14 @@ helper() { cat <) EOF } @@ -179,6 +202,7 @@ main() { exit 1 fi + CHECKSUM_FLAG=0 # Parse CLI arguments while [ $# -gt 0 ]; do case $1 in @@ -192,19 +216,46 @@ main() { echo "For more informatio, try --help" exit 1 fi - make_backup "$BACKUP_SH_SOURCES_PATH" "$BACKUP_SH_OUTPATH" "$BACKUP_SH_PASSWORD" + + if [ "$CHECKSUM_FLAG" -eq 1 ]; then + make_backup "$BACKUP_SH_SOURCES_PATH" "$BACKUP_SH_OUTPATH" "$BACKUP_SH_PASSWORD" 1 + else + make_backup "$BACKUP_SH_SOURCES_PATH" "$BACKUP_SH_OUTPATH" "$BACKUP_SH_PASSWORD" 0 + fi + exit 0 ;; + -c|--checksum) + [ $# -eq 1 ] && { echo "Use this option with '--backup' or '--extract'"; exit 1; } + CHECKSUM_FLAG=1 + shift 1 + ;; -e|--extract) BACKUP_SH_ARCHIVE_PATH="$2" BACKUP_SH_ARCHIVE_PW="$3" + BACKUP_SH_SHA256_FILE="$4" - if [ -z "$BACKUP_SH_ARCHIVE_PATH" ] || [ -z "$BACKUP_SH_ARCHIVE_PW" ]; then - echo "Please, specify an encrypted archive and a password." - echo "For more informatio, try --help" - exit 1 + if [ "$CHECKSUM_FLAG" -eq 1 ]; then + if [ -z "$BACKUP_SH_ARCHIVE_PATH" ] || [ -z "$BACKUP_SH_ARCHIVE_PW" ] || [ -z "$BACKUP_SH_SHA256_FILE" ]; then + echo "Please, specify an encrypted archive, a password and a SHA256 file." + echo "For more informatio, try --help" + exit 1 + fi + else + if [ -z "$BACKUP_SH_ARCHIVE_PATH" ] || [ -z "$BACKUP_SH_ARCHIVE_PW" ]; then + echo "Please, specify an encrypted archive and a password." + echo "For more informatio, try --help" + exit 1 + fi fi - extract_backup "$BACKUP_SH_ARCHIVE_PATH" "$BACKUP_SH_ARCHIVE_PW" + + if [ "$CHECKSUM_FLAG" -eq 1 ]; then + [ -e "$BACKUP_SH_SHA256_FILE" ] || { echo "Checksum file does not exist"; exit 1; } + extract_backup "$BACKUP_SH_ARCHIVE_PATH" "$BACKUP_SH_ARCHIVE_PW" "$BACKUP_SH_SHA256_FILE" + else + extract_backup "$BACKUP_SH_ARCHIVE_PATH" "$BACKUP_SH_ARCHIVE_PW" + fi + exit 0 ;; -h|--help) diff --git a/backup.sh.1 b/backup.sh.1 index 3e37d46..9d73b58 100644 --- a/backup.sh.1 +++ b/backup.sh.1 @@ -1,265 +1,406 @@ -.\" Automatically generated by Pandoc 3.1.8 +.\" Automatically generated by Pandoc 2.17.1.1 .\" -.TH "backup.sh" "1" "February 27, 2024" "Marco Cetica" "General Commands Manual" +.\" Define V font for inline verbatim, using C font in formats +.\" that render this, and otherwise B font. +.ie "\f[CB]x\f[]"x" \{\ +. ftr V B +. ftr VI BI +. ftr VB B +. ftr VBI BI +.\} +.el \{\ +. ftr V CR +. ftr VI CI +. ftr VB CB +. ftr VBI CBI +.\} +.TH "backup.sh" "1" "April 3, 2024" "Marco Cetica" "General Commands Manual" +.hy .SH NAME +.PP \f[B]backup.sh\f[R] - POSIX compliant, modular and lightweight backup utility to save and encrypt your files. .SH SYNOPSIS .IP -.EX -Syntax: backup.sh [-b|-e|-h] +.nf +\f[C] +Syntax: ./backup.sh [-b|-c|-e|-h] options: --b|--backup SOURCES DEST PASS Backup folders from SOURCES file. --e|--extract ARCHIVE PASS Extract ARCHIVE using PASS. --h|--help Show this helper. -.EE +-b|--backup SOURCES DEST PASS Backup folders from SOURCES file. +-c|--checksum Generate/check SHA256 of a backup. +-e|--extract ARCHIVE PASS Extract ARCHIVE using PASS. +-h|--help Show this helper. +\f[R] +.fi .SH DESCRIPTION +.PP \f[B]backup.sh\f[R] is a POSIX compliant, modular and lightweight backup utility to save and encrypt your files. This tool is intended to be used on small scale UNIX environment such as VPS, small servers and workstations. -\f[B]backup.sh\f[R] uses \f[I]rsync\f[R], \f[I]tar\f[R] and -\f[I]gpg\f[R] to copy, compress and encrypt the backup. +\f[B]backup.sh\f[R] uses \f[I]rsync\f[R], \f[I]tar\f[R], +\f[I]sha256sum\f[R] and \f[I]gpg\f[R] to copy, compress, verify and +encrypt the backup. .SH OPTIONS -\f[B]backup.sh\f[R] supports two options: \f[I]backup creation\f[R] and -\f[I]backup extraction\f[R]. -The former requires root permissions, while the latter does not. -Let us see them in details: +.PP +\f[B]backup.sh\f[R] supports three options: \f[B]backup creation\f[R], +\f[B]backup extraction\f[R] and \f[B]checksum\f[R] to verify the +integrity of a backup. +The first option requires root permissions, while the second one does +not. +The checksum option must be used in combination of one of the previous +options. .SS Backup creation -To specify the directories to backup, \f[B]backup.sh\f[R] uses an -associative array defined in a text file(called sources file) with the -following syntax: -.IP -.EX -