The objectif of this Tutorial is to present the implementation of a synchronization script based on the indispensable and (well?) known "rsync", between a local computer and a Hubic Cloud space, made available by the OVH Company. The object is obviously not to present the client provided by OVH, rather simple to set up, and powerful (when it works ;-)), but to propose a third party or backup solution, which also allows direct access to the storage space as a "local" system, ranging from 25 GB (offered free by OHV) to 10 TB, for 5 €/month. I also made sure to increase the synchronization performance as much as possible compared to a "simple" rsync command.
This script has to be analyzed and used as needed, without any guarantee: I myself have deleted a few gigas of my personal data by developing it. Luckily I had backups, so do the same.
Mounting the Hubic file system.
The access to the storage space is done through the hubicfuse tool and will not be detailed here, as others have already done it very well. I just bring some customizations that can be useful.
I followed this page, simple and well explained.
Here is my config file ".hubicfuse" after having followed P. Diancourt's Tutorial :
client_id=api_hubic_XXXXXXXXXXXXXXXXXXXXX
client_secret=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
refresh_token=XXXXXXXXXXXXXXXXX
cache_timeout = 600
temp_dir = /home/my_user/tmp
get_extended_metadata = true
curl_verbose = false
curl_progress_state = true
cache_statfs_timeout = 15
debug_level = 0
enable_chmod = false
enable_chown = false
Personal Adjustments :
I modify the default temporary directory ( /tmp ) that I put in my user space for space reasons. Because if like me you have a 5 GB /tmp partition, this one can be quickly saturated (which happened to me) if you synchronize several large files.
The mount options I use are different. I have adjusted them after "DuckDugué" on different blogs and it seems to work pretty well, so here's what I add to the /,etc./fstab file:
hubicfuse /home/my_user/hubic rw,noauto_cache,sync_read,allow_other,no_splice_write,no_splice_move
You don't have to mount this system permanently, the script provided below will do it if necessary.
As Pierre Diancourt points out, this file system has a few limitations in terms of performance and functionality, but it will be fine with an rsync-based script configured accordingly.
The HubicTool synchronization script
This script, pompously called "hubictool" brings 6 rather basic arguments:
Synchronization (argument -sync) allows first to synchronize a local directory of your choice to the "default" remote directory of hubic. In a second step it will do the reverse operation, in case you upload files to hubic via web client or smartphone. The sync options chosen don't delete anything, you just add or modify in one way or another.
Sending (-send argument): updates the cloud only
Receive (-receive argument): updates the local directory only
The backup (argument -backup) allows you to backup your local directory to the hubic backup directory. Its directory name depends on the one you gave to your synchronization under hubic. Lacking imagination, I called it "Synchro". My backup directory is therefore called "HubiC-DeskBackup_Synchro", at the same level as "default". Otherwise create a directory of your choice. You will then encounter your first disappointment due to the limitations mentioned above. To get around this, create a directory named as you wish on your desktop for example, then move it to your Hubic space. Because if you create it directly, you won't be able to rename it!
The restore (-restore argument) will synchronize the backup space to the local directory.
This script is strongly inspired by the one we can find here in which the principle is to browse directories with a "find" and to launch a rsync thread per directory. The number of threads is limited so as not to bring the machine to its knees.
I took the principle of rsync thread parallelization, but by parallelizing at the level of files and not directories. This adapts better to directories with a strong disparity in the size of the files. For example a multitude of directories with small files (office DOCs, scanning of administrative papers, etc.), and one directory with films. By parallelizing at the directory level, small files are quickly synchronized, and there is often only one rsync process left for the directory with large files.
I also took this script to work by directory level (it synchronizes first the root, then the level 1, 2 directories ... <= $maxdepth
The script can be retrieved here.
Here are some explanations about the code.
The first block allows to initialize the most common variables: directories, exclusions and mount point for hubicfuse. Note that the exclusions refer to "what starts with". The different PATTERNS are separated by | (pipes). Slashes are escaped by backslashes. For example the /_NO_ exclusion allows to quickly exclude any directory starting with "_NO_". See the egrep command (REGEXP) for notation.
Next is the main synchronization function, with its variables, some of which can be adjusted:
maxdepth: it is important to adjust this variable according to the depth of the directories to be synchronized. Setting a number too small will exclude some files from synchronization, a number too large will significantly increase the synchronization time. A command like 'find . -mindepth x -maxdepth x -type d -print' (with x strictly identical) will allow you to determine which value to choose.
rsyncopts: Long hours have led me to make this choice, which will not necessarily suit everyone. However in the 'Hubic' context the -inplace argument is indispensable, because rsync writes by default to a temporary file in the destination directory, then renames the file. But the renaming fails and the replacement is not done.
t: conservation of the timestamp. Essential for hubic to know if a file has been modified and needs to be resynchronized.
LK: allow to treat symbolic links as "normal" directories. This allows to align in its synchro directory directories scattered elsewhere on the machine. Adding a symbolic link to a "photos" directory is enough to make it "in sync"! You can also mount remote systems (NAS, SMB shares ...) in your sync directory ... And then suddenly the 10TB offer makes sense, doesn't it? (no no, I swear that OVH doesn't pay me anything for this Tuto).
W: I read somewhere that it was better with inplace, so I added it
maxthreads: sets the number of simultaneous rsync threads. A rsync command triggering 3 forks will synchronize about ten files simultaneously with the current value of 30. Also to be set according to the available bandwidth.
sleeptime: waiting time between 2 checks of the number of threads. Reducing it can save time on a large number of small files.
The hubicmount function: as its name indicates it mounts the hubic file system, it is not unmounted when the script ends normally.
The "main" is simple since it is based on a "box" listing the 3 arguments of this : -sync, -backup, -restore.
Finally, pressing Ctrl-C during the execution of the script allows to exit by killing ALL the rsync processes. This is only possible (for the moment) by also killing the hubicfuse process.
That is why you are asked to force unmount the hubicfuse file system (sudo umount -l hubicfuse) when exiting, otherwise it will still appear as mounted to the system.
Want to know what is going through your ip address?
Cybersecurity is not only a problem of financial means. It is also a question of defensive tools, technical perimeter, unfiltered access and above all, running after time.
It is therefore necessary to be able to analyze these unfiltered accesses in real time, to detect those that are fraudulent.
CrowdSec is a 100% open source tool that combines 2 ideas very well known by server administrators.
Crowdsec is able to automatically ban an IP connecting to your server, a bit like a Fail2ban.
And it is able to offer a database of these IPs to the CrowdSec user community, a bit like AbuseIPDB does, so that information can circulate quickly and everyone can be protected in turn.
This allows you to detect all types of attacks and respond proportionally to them.
ISocial Plugin