Posts Tagged bash
A friend of mine accidentally formatted her 2TB backup drive and brought it to me to see what I could do. PhotoRec was the only tool that would give more than a hint of recoverable data from my Linux-based laptop. After letting it run for several hours, I had a mess: PhotoRec recovered over 230,000 files, and spread them out over 430 directories!
Finding and sorting them by hand was obviously out of the question as it would take way too long. I took a hint from another recovery program I had [unsuccessfully] tried, and thought of sorting them into directories based on their file extensions. I.e. .jpg into ‘jpg’, .doc into ‘doc’, etc.
However, since I was working with the large number of files and directories, I had to take this a bit further. This will sort files into directory names like ‘jpg.1’, ‘jpg.2’, etc. These are numbered to prevent putting too many files in a single directory and killing system performance :)
I came up with the following bash script for use on Linux systems. A word of caution is due: This script does very little error checking, though the commands are quite sensible and I ran it as-is with fine results. I’m not responsible for unanticipated results.
UPDATE: This has moved to github, here.
Questions, comments, and feedback are welcome. Please leave them in the comments section below. Thank you!
Every now and then I’ll run into an issue with a website’s uploader. They ask me to upload a picture, but then when I click their upload button, none of my pictures appear in the dialog. After troubleshooting for a few, it turns out that they’re limiting file masks to [*.jpg, *.jpeg, *.bmp, *.png] etc. But — because I copied my pictures over from a windows installation, they have all-capital file extensions. Linux uses a case-sensitive file system, so it regards these as different. Renaming a file to a lowercase extension [*.jpg] caused it to show up in the dialog which is what I wanted — but manually renaming thousands of pictures in dozens of directories was out of the question.
I could have written a bash script to do the renaming in a few minutes but I found something better — convmv. This simple utility makes filename conversions / renaming a breeze. By default, it runs in ‘test’ mode so that you can see what will happen before it does the job.
For my case, I needed to rename all the files to lowercase, so I used:
convmv --lower *
That showed me a verbose listing of everything it would do (test mode). However, I wanted to do the entire Pictures folder and everything under it. The new command from my Pictures folder became:
convmv --lower -r *
To get it to actually do the job, I had to specify
--notest as well.
convmv --lower -r --notest *
It did it’s work within seconds and everything was lowercase. In my opinion, much easier and better than a bash script.
Convmv has plenty of other options, so next time you need to do filename conversion, check it out.
Questions, comments, feedback? Please share in the comments below. Thank you.
The one bad thing I’ve come to notice about the JungleDisk Server Edition is, over time, it tends to hog a lot of memory, even when it’s not running backups. The author at geektank.net noticed this too, and recommended it may not be a good fit for low-memory VPS configurations.
But if JungleDisk is a good fit for your needs, and the memory usage is the only issue, here’s something to try. It’s either a clever solution or an ugly workaround. Call it what you will.
What we’re going to do is create a cron job that will restart jungledisk when it is done running the backup, which will free up any potentially wasted memory.
So, we’ll start by creating a postbackup.sh script to run after your backup job. For advice on how to create and schedule this script, see my previous article, Backing up your server using JungleDisk Server Edition – part 2.
Create your postbackup.sh file with the following line:
Now, create the following jd-check.sh script and make it executable. It should be setuid root.
#!/bin/bash if [ -e /etc/jungledisk/.restartjd ] then rm /etc/jungledisk/.restartjd && /etc/init.d/junglediskserver restart fi
That’s about as simple as it gets, right there.
The new script should be run on a cron job that will cause it to run often enough to restart jungledisk after a backup. A suggestion would be to have it run about a half-hour to an hour after your backups are scheduled to start.
There are some security implications to where you store your temp file, what your name it, and what permissions you give it, so use your head. If you carefully read part 2, you can get a good handle on how to be mindful of the security issues.
It’s also possible to simply restart junglediskserver on a cron job, but there’s the potential you could restart it when it’s in the middle of a backup. This would cause the backup to either postpone the backup, or resume immediately, and leave stale memory allocations again, which defeats the point. What I’m aiming for here is to have it restart as quickly as possible once the update completes.
Do you have any thoughts on this approach? Know of a way that might work better? Feel free to share your thoughts in the comments below! Thank you.
In part 1, I told you how to set up JungleDisk backup for your Linux server. In this part 2, I’ll tell you how to automatically have it dump and backup your MySQL databases (correctly)!
There are security implications if this permissions are not set correctly on the files, as you’re storing your MySQL password in the script, and you’re going to have complete database dumps sitting on your hard drive. However, I will attempt to explain as clearly as possible my solution, and I’m not responsible if it doesn’t work for you.
So, start out by deciding where you want your database-dumping script to run from. A few good example spots are /root (root users home directory), and /etc/jungledisk (with the configuration files). I’ve called my backup script prebackup.sh. You’ll understand the name further below, but I’m going to use that name the rest of the way through.
So create your prebackup.sh script as root, and set it to mode 700.
touch prebackup.sh && chmod 0700 prebackup.sh
This makes sure that root is the only user who can read, write, or execute it.
Now, using your favorite text editor, you can use the following sample script:
#!/bin/bash date=`date -I` dir=/var/backups file=sqlbackup-$date.sql touch $dir/$file && chmod 600 $dir/$file && mysqldump --all-databases -p'__MySQLPassword__' >> $dir/$file find $dir -ctime +7 -delete
Danny pointed out that creating gzipped MySQL dumps, as I had in my original script, is discouraged as it defeats the data de-dup feature of jungledisk. The above script has been changed from the original to make uncompressed dumps. Thanks, Danny!
The last line, with the find statement, is responsible for the cleanup of the directory. It will delete any file which is older than 7 days. If you want more or less time, simply change
+7 to your desired number of days (keeping the
Warning: There’s very little (read: none) sanity-checking in this script. Replace
__MySQLPassword__ with your root MySQL password.
Jay pointed out that there will likely be issues with handling special characters in the SQL password. If you have any suggestions, please feel free to post them in the comments below.
After saving, you should have a 183-byte (give or take) file with 0700 mode:
root@ve:/etc/jungledisk# ls -l prebackup.sh
-rwx------ 1 root root 183 Mar 24 17:08 prebackup.sh
You should make sure that the directory in $dir is owned by user and group root and mode 0700. This will make sure that noone else has access to your dumped databases. Now that you have your script, it’s time to automate it. You could schedule a cron job to run the script, but it’s easier to have JungleDisk run it for you at the start of every backup job.
Start your management-side program, login, and go to your server’s backup configuration. Under Backup Options, check Enable Pre and Post Backup Scripts. Now, click Configure, and in the Pre-Backup Script, enter the full path and filename of your newly created script, i.e. /etc/jungledisk/prebackup.sh.
Now, on your assigned schedule, the server engine will run your database dumping script, and start your backup job. Of course, make sure whatever directory you’re dumping your databases to is included in your backup set.
The last thing to note is this script is amazingly light; it doesn’t delete any old databases and it doesn’t do much else. You’re free to modify it, and I would greatly appreciate any feedback on your modifications.
Comments are welcome, as always.
If you’re a jungledisk user on linux, and you put junglediskdesktop in your Startup Applications you may receive unusual errors when you log in.
Such errors are:
- The jungledisk tray icon does not appear
- The jungledisk window floats and cannot be closed
- The jungledisk app gives unusual errors
The problem appears to be that there’s a race condition where junglediskdesktop starts before Gnome is ready to handle it as a tray app.
- Create a text file with gedit (or your editor of choice)
- In the file, enter these two lines:
#!/bin/bash sleep 3 && /usr/local/bin/junglediskdesktop
You’ll need to make your new script executable, so at a terminal do:
chmod +x filename
Now, in startup applications, use your new script instead of junglediskdesktop.
What this script does:
It ‘sleeps’ for 3 seconds before starting the junglediskdesktop application.
Doing that allows Gnome to be ready to handle junglediskdesktop correctly.
It’s my opinion that this is an issue with junglediskdesktop itself (not waiting for Gnome to be ready) rather than an issue with the Gnome itself.
CrashPlan on Linux depends on the inotify kernel module to know when files update in real-time.
Inotify was merged into the 2.6.13 Linux kernel, so if you’re running a kernel equal to or newer than this, it’s already installed. If not, you’ll have to install it yourself. If inotify is installed, you may need to increase the number of watches that can be created.
The inotify module is govered by a property called max_user_watches. If you attempt to exceed the max number you’ll get the following error in the engine_error.log (but the process lives on).
inotify_add_watch: No space left on device
Any file not covered by a watch does not have real-time backup protection.
The default on my Ubuntu 11.04 box is 524288, which seems plenty sufficient for me. I have not experienced any issues, but if you find that you are, you may want to increase the watch value.
Updating the Watch Value
You can temporarily update the value with:
echo 1048576 > /proc/sys/fs/inotify/max_user_watches
You can update the value permanently by putting the following value in /etc/sysctl.conf and restarting:
For more information, see CrashPlan’s Forums.