Casual_blog/content/hidden/preview/test3.md

+++
title = 'test'
hidden = true
image = "https://i.extremetech.com/imagery/content-types/017c7K9UIE7N2VnHK8XqLds/images-5.jpg"
+++


<meta property="og:image" content="https://imgs.xkcd.com/comics/backup_batteries.png" />


In short: 3-2-1 backup strategy + Disaster recovery plan.


## Backup strategy

You should have:
 - 3 copies of data
 - on 2 different types of storages
 - including 1 off-site copy

AND you must test disaster recovery plan

### Why so many copies?

What if you accidently delete important files that you frequently edit? That's the reason to have snapshots.

What if your main drive with data will die? That's the reason to have backup nearby.

You think that you smart and have RAID for all those cases? Did you know that in drive arrays, one drive's failure significantly increases the short-term risk of a second drive failing.  That's the reason to have off-site backup.

What if your main storage Server will die with all drives in it due to power spike (flood,etc...)? So, do off-site backups.

![](https://imgs.xkcd.com/comics/backup_batteries.png)

### 3 copies of data

You should have:
 - Original data
 - 1 backup at place (another drive)
 - 1 backup in another place (encrypted in cloud, HDD stored in another remote location (friend's house))

Backups which should be made regularly (daily or more frequently for critical data, +depends how "hot" data (how fast it changes)).  
My take on it - have a trusted source of data - RAID/Ceph and use snapshots to have copy of data to save some money on backup drives.

### 2 types of storages

You need to have different 2 types of storage to metigate if some error may affect all devices of 1 type. 

Storage types examples:
 - Internal HDD/SSD (we will focus on them)
 - External HDD (them)
 - USB drive/SSD
 - Tape library
 - Cloud storage (and them)

### 1 off-site copy

It's pretty simple:
 - encrypted cloud backup
 - encrypted HDD with backup in another town in friend's house (secured by bubble wrap)
 - or at least encrypted HDD in another house (also secured by bubble wrap)

The more distant this off-site backup the better.


## Disaster recovery plan

People fall into 3 categories: 
 - those who don't do backups yet 
 - those who already do them
 - and those who do them and tested them

You should be in 3rd catergory.

__So what is disaster recovery plan?__

You must be prepared in case if your main data and in-site backup dies. You must beforehand imitate:
 - accidental data removal (to test in-site snapshots)
 - drive failure and its change (to test RAID/Ceph solution)
 - main storage failure (to test restore from in-site backup)
 - entire site unavailability (to test off-site backup)

Ideally you should write for yourself step-by-step guide what to do in any of those situations


{{< spoiler Examples >}}

 &nbsp;

## Examples


### Enterprise-ish (Expensive at start, hard setup, easy to maintain)

Ceph cluster:
 - requires 3 servers (at least) (OS - Proxmox)   
 Ideally server motherboard, ECC RAM, Intel Xeon E5 v4 CPU Family or better / AMD Epyc analog
 - any number of drives (but at least 3 drives)
 Ideally enterpise-class (or with "RAID support"). The more IOPS - the better
 - [automatic snapshots](https://github.com/draga79/cephfs-snp)
 - 10Gb network (if you expect total 9-ish (or more) HDD drives or some SSDs)
 - Setup Samba/WebDAV/Nextcloud server which will share this storage to your network
 - and ideally SSD cache (at least 2 SSDs with PLP) (1tb each more than enough for 10TB of raw storage) 
 
Off-site backup:

Cloud storage + [dublicati](https://github.com/duplicati/duplicati)  
OR  
Proxmoxx Backup Server at another city (e.g. at friend's house) with RAID1/5/6   
 (thou you should set it up so if malware/hacker would get to root user it won't overwrite backups) 

#### Pros
 - Ideal if you already have homeserver and want to expand
 - Low chances of loosing data because you essentially have 3 copies (by default, 2 min) of  data + hourly/daily/weekly/montly snapshots  
 So if you get 2 dead drives in a same time - you still won't loose your data  
 Essentially it covers 2 copies of data 
 - If drive fails - you simple take it out, put new drive in and say that you want add this drive to pool via WebGUI
 - With SSD cache you can throw in any trashy HDD drives until they start to fail
 - You can add any number of drives
 - And if you need/want to be able to freely shutdown one of a servers and still be able to access data - you need to distribute drives so their raw storage would be even on each server.
 Or just add in few more server and distribute drives between them so you would still be able to access this storage 
 - If you get your house+servers destroed - you wouldn't loose your data  
 - You can access your storage from any device in your network as if it is on it device

#### Cons
 - Expect 30% usable space from raw storage (you can use Erasure Coding (RAID5 analog) but it will be even slower)
 - Bad/Slow (in terms of IOPS and delay times) drives without PLP SSD cache can have amazingly bad total speed
 - Power usage might be a burden if you don't have any 
 - More performance comes with more drives because speed = available IOPS and avarage access time for 2-3 drives that have that data. So more drives, more IOPS we have (excluding SSD cache case)
 - Ceph can be complicated to understand and maintain in case of failures


### Home-server (Medium cost, medium difficulty, hard to maintain)

CIFS/WebDAV/Nextcloud Share:
 - get any PC, install linux on it, setup Samba/WebDAV/Nextcloud share
 - X number of drives in RAIDZ (4+ even drives) (ideally RAIDZ2)
 - ZFS automatic Snapshots

Off-site backup:  
Cloud storage + [dublicati](https://github.com/duplicati/duplicati)   
OR  
Regular (montly/weekly) manual encrypted backup to external HDD which is given to friend.

#### Pros
 - It's relativly cheap
 - You get storage space from X-1 (or X-2) of drives 
 - You can access your storage from any device in your network as if it is on it device
 - You can loose 1 (RAIDZ2 - 2) drive

#### Cons
 - If drive fails - storage should be inaccessable for some time after you put new drive instead of failed drive. 
 - If 2/3 drives fails in short perioud of time - you loose data
 - Hard to upgrade storage by using bigger disks, then more disks
 - Drives should have same size


### Home PC (low cost, low difficulty, easy to maintain)

We will just put 2 (or more) drives in RAID1 in your PC.
Ideally - buy different drives with same-ish specs so they die in different time. And use file system with snapshot support

Off-site backup:  
Cloud storage + [dublicati](https://github.com/duplicati/duplicati)   
OR  
Regular (montly/weekly) manual encrypted backup to external HDD which is given to friend.


#### Pros
 - It's cheap
 - Setup easy to understand


#### Cons
 - 50% space from raw storage
 - Potentially no snapshots if file system don't support it
 - All of the drives should die to loose data


### Laptop (High cost, easy setup, easy to maintain)

This time we will do opposite:
 - laptop with cloud storage synchronized in laptop and cloud (so files stored on laptop and cloud) 
 - ideally file system snapshot support 

Off-site backup:  
Regular (montly/weekly) manual encrypted backup to external HDD which is given to friend.

#### Pros
 - It's cheap at first, but costly in the long run
 - It's easy to setup and cloud providers give support (not the best but neverthless)
 - It's much easier to maintain since you don't have to deal with hardware

#### Cons
 - It's the most privacy unfriendly setup because you will have unencrypted data in cloud - or way that will sync only encrypted data to cloud
 - Cloud subcription are costly in the long run
 - To have backup - you should be connected to internet
 - You may be affected by troubles by cloud provider


### Laptop+PC (Low cost, easy setup, may be hard to maintain)

We will use available hardware and its space, laptop+PC+off-site (friend's) PC for encrypted backups.
The trick is - we will use [syncthing](https://github.com/syncthing/syncthing) - amazing tools, allows P2P sync storage. 

#### Pros
 - P2P, no other servers involved!
 - We can specify where data will be stored encrypted and where freely accessable
 - as easy to setup as cloud provider
 
#### Cons
 - The issue may be if file edited in 2 places before sync = version conflict 
 - Another problem - is storage space, it's easy to setup but it maybe hard to maintain if data drives have different free storage space.
 

{{< /spoiler >}}

{{< source >}}
https://raidz-calculator.com/raidz-types-reference.aspx
https://www.techtarget.com/searchdatabackup/definition/3-2-1-Backup-Strategy
https://en.wikipedia.org/wiki/Hard_disk_drive
{{< /source >}}
test 2024-08-10 20:37:08 +00:00			`+++`
			`title = 'test'`
			`hidden = true`
image preview 2024-08-11 01:40:47 +00:00			`image = "https://i.extremetech.com/imagery/content-types/017c7K9UIE7N2VnHK8XqLds/images-5.jpg"`
test 2024-08-10 20:37:08 +00:00			`+++`

testing image preview 2024-08-10 22:41:18 +00:00
			`<meta property="og:image" content="https://imgs.xkcd.com/comics/backup_batteries.png" />`


test 2024-08-10 20:37:08 +00:00			`In short: 3-2-1 backup strategy + Disaster recovery plan.`



			`## Backup strategy`

			`You should have:`
			`- 3 copies of data`
			`- on 2 different types of storages`
			`- including 1 off-site copy`

			`AND you must test disaster recovery plan`

			`### Why so many copies?`

			`What if you accidently delete important files that you frequently edit? That's the reason to have snapshots.`

			`What if your main drive with data will die? That's the reason to have backup nearby.`

			`You think that you smart and have RAID for all those cases? Did you know that in drive arrays, one drive's failure significantly increases the short-term risk of a second drive failing. That's the reason to have off-site backup.`

			`What if your main storage Server will die with all drives in it due to power spike (flood,etc...)? So, do off-site backups.`

			`![](https://imgs.xkcd.com/comics/backup_batteries.png)`

			`### 3 copies of data`

			`You should have:`
			`- Original data`
			`- 1 backup at place (another drive)`
			`- 1 backup in another place (encrypted in cloud, HDD stored in another remote location (friend's house))`

			`Backups which should be made regularly (daily or more frequently for critical data, +depends how "hot" data (how fast it changes)).`
			`My take on it - have a trusted source of data - RAID/Ceph and use snapshots to have copy of data to save some money on backup drives.`

			`### 2 types of storages`

			`You need to have different 2 types of storage to metigate if some error may affect all devices of 1 type.`

			`Storage types examples:`
			`- Internal HDD/SSD (we will focus on them)`
			`- External HDD (them)`
			`- USB drive/SSD`
			`- Tape library`
			`- Cloud storage (and them)`

			`### 1 off-site copy`

			`It's pretty simple:`
			`- encrypted cloud backup`
			`- encrypted HDD with backup in another town in friend's house (secured by bubble wrap)`
			`- or at least encrypted HDD in another house (also secured by bubble wrap)`

			`The more distant this off-site backup the better.`


			`## Disaster recovery plan`

			`People fall into 3 categories:`
			`- those who don't do backups yet`
			`- those who already do them`
			`- and those who do them and tested them`

			`You should be in 3rd catergory.`

			`__So what is disaster recovery plan?__`

			`You must be prepared in case if your main data and in-site backup dies. You must beforehand imitate:`
			`- accidental data removal (to test in-site snapshots)`
			`- drive failure and its change (to test RAID/Ceph solution)`
			`- main storage failure (to test restore from in-site backup)`
			`- entire site unavailability (to test off-site backup)`

			`Ideally you should write for yourself step-by-step guide what to do in any of those situations`



			`{{< spoiler Examples >}}`

			` `

			`## Examples`



			`### Enterprise-ish (Expensive at start, hard setup, easy to maintain)`

			`Ceph cluster:`
			`- requires 3 servers (at least) (OS - Proxmox)`
			`Ideally server motherboard, ECC RAM, Intel Xeon E5 v4 CPU Family or better / AMD Epyc analog`
			`- any number of drives (but at least 3 drives)`
			`Ideally enterpise-class (or with "RAID support"). The more IOPS - the better`
			`- [automatic snapshots](https://github.com/draga79/cephfs-snp)`
			`- 10Gb network (if you expect total 9-ish (or more) HDD drives or some SSDs)`
			`- Setup Samba/WebDAV/Nextcloud server which will share this storage to your network`
			`- and ideally SSD cache (at least 2 SSDs with PLP) (1tb each more than enough for 10TB of raw storage)`

			`Off-site backup:`

			`Cloud storage + [dublicati](https://github.com/duplicati/duplicati)`
			`OR`
			`Proxmoxx Backup Server at another city (e.g. at friend's house) with RAID1/5/6`
			`(thou you should set it up so if malware/hacker would get to root user it won't overwrite backups)`

			`#### Pros`
			`- Ideal if you already have homeserver and want to expand`
			`- Low chances of loosing data because you essentially have 3 copies (by default, 2 min) of data + hourly/daily/weekly/montly snapshots`
			`So if you get 2 dead drives in a same time - you still won't loose your data`
			`Essentially it covers 2 copies of data`
			`- If drive fails - you simple take it out, put new drive in and say that you want add this drive to pool via WebGUI`
			`- With SSD cache you can throw in any trashy HDD drives until they start to fail`
			`- You can add any number of drives`
			`- And if you need/want to be able to freely shutdown one of a servers and still be able to access data - you need to distribute drives so their raw storage would be even on each server.`
			`Or just add in few more server and distribute drives between them so you would still be able to access this storage`
			`- If you get your house+servers destroed - you wouldn't loose your data`
			`- You can access your storage from any device in your network as if it is on it device`

			`#### Cons`
			`- Expect 30% usable space from raw storage (you can use Erasure Coding (RAID5 analog) but it will be even slower)`
			`- Bad/Slow (in terms of IOPS and delay times) drives without PLP SSD cache can have amazingly bad total speed`
			`- Power usage might be a burden if you don't have any`
			`- More performance comes with more drives because speed = available IOPS and avarage access time for 2-3 drives that have that data. So more drives, more IOPS we have (excluding SSD cache case)`
			`- Ceph can be complicated to understand and maintain in case of failures`


			`### Home-server (Medium cost, medium difficulty, hard to maintain)`

			`CIFS/WebDAV/Nextcloud Share:`
			`- get any PC, install linux on it, setup Samba/WebDAV/Nextcloud share`
			`- X number of drives in RAIDZ (4+ even drives) (ideally RAIDZ2)`
			`- ZFS automatic Snapshots`

			`Off-site backup:`
			`Cloud storage + [dublicati](https://github.com/duplicati/duplicati)`
			`OR`
			`Regular (montly/weekly) manual encrypted backup to external HDD which is given to friend.`

			`#### Pros`
			`- It's relativly cheap`
			`- You get storage space from X-1 (or X-2) of drives`
			`- You can access your storage from any device in your network as if it is on it device`
			`- You can loose 1 (RAIDZ2 - 2) drive`

			`#### Cons`
			`- If drive fails - storage should be inaccessable for some time after you put new drive instead of failed drive.`
			`- If 2/3 drives fails in short perioud of time - you loose data`
			`- Hard to upgrade storage by using bigger disks, then more disks`
			`- Drives should have same size`



			`### Home PC (low cost, low difficulty, easy to maintain)`

			`We will just put 2 (or more) drives in RAID1 in your PC.`
			`Ideally - buy different drives with same-ish specs so they die in different time. And use file system with snapshot support`

			`Off-site backup:`
			`Cloud storage + [dublicati](https://github.com/duplicati/duplicati)`
			`OR`
			`Regular (montly/weekly) manual encrypted backup to external HDD which is given to friend.`


			`#### Pros`
			`- It's cheap`
			`- Setup easy to understand`


			`#### Cons`
			`- 50% space from raw storage`
			`- Potentially no snapshots if file system don't support it`
			`- All of the drives should die to loose data`



			`### Laptop (High cost, easy setup, easy to maintain)`

			`This time we will do opposite:`
			`- laptop with cloud storage synchronized in laptop and cloud (so files stored on laptop and cloud)`
			`- ideally file system snapshot support`

			`Off-site backup:`
			`Regular (montly/weekly) manual encrypted backup to external HDD which is given to friend.`

			`#### Pros`
			`- It's cheap at first, but costly in the long run`
			`- It's easy to setup and cloud providers give support (not the best but neverthless)`
			`- It's much easier to maintain since you don't have to deal with hardware`

			`#### Cons`
			`- It's the most privacy unfriendly setup because you will have unencrypted data in cloud - or way that will sync only encrypted data to cloud`
			`- Cloud subcription are costly in the long run`
			`- To have backup - you should be connected to internet`
			`- You may be affected by troubles by cloud provider`



			`### Laptop+PC (Low cost, easy setup, may be hard to maintain)`

			`We will use available hardware and its space, laptop+PC+off-site (friend's) PC for encrypted backups.`
			`The trick is - we will use [syncthing](https://github.com/syncthing/syncthing) - amazing tools, allows P2P sync storage.`

			`#### Pros`
			`- P2P, no other servers involved!`
			`- We can specify where data will be stored encrypted and where freely accessable`
			`- as easy to setup as cloud provider`

			`#### Cons`
			`- The issue may be if file edited in 2 places before sync = version conflict`
			`- Another problem - is storage space, it's easy to setup but it maybe hard to maintain if data drives have different free storage space.`



			`{{< /spoiler >}}`

			`{{< source >}}`
			`https://raidz-calculator.com/raidz-types-reference.aspx`
			`https://www.techtarget.com/searchdatabackup/definition/3-2-1-Backup-Strategy`
			`https://en.wikipedia.org/wiki/Hard_disk_drive`
			`{{< /source >}}`