Archive for category Uncategorized

Restoring and downloading S3 Glacier objects using s3cmd

I currently have a portion of my backups on S3, with a life-cycle policy that includes moving the objects to Glacier after a period of time. This makes the storage much cheaper ($0.01/GB/Mo from $0.03/GB/Mo – Source), but has the downside that objects require a 4-hour restore period  before they can become available for download. I have had need for some objects quickly, and so the 4-hour restore time isn’t worth the savings. Unfortunately, once an object has had this life-cycle applied to it, it can only be temporarily restored. In order to make it a standard object again, you have to download it, delete the Glacier object, and then re-upload it. Unfortunately, doing it all wasn’t quite as straightforward as I thought it might be. But, (I think) I figured out a way to get it done rather painlessly.

I’m going to be using s3cmd and a few cron jobs to automate this.

First, get s3cmd version 1.5. This version supports initiating restores on the Glacier objects. You can recursively initiate a restore on every object in the bucket, but when it hits a non-Glacier object it will stop. You can also use s3cmd to initiate a download of all the objects in the bucket, but when it hits a Glacier object, the download will stop. And you will end up with a zero-byte file. (Hey s3cmd developers, would you mind fixing this behavior, or at least writing in something to force progression on a failure, so we can walk through the entire bucket in one go?)

The solution had to involve initiating restores, waiting at least 4 hours for the restore, then going back for the restored data and deleting it from the buckets, then deleting any zero-byte files, and then doing it all over again later.

Ain’t nobody got time for that. Except cron. Cron has plenty of time for that.

First of all, make sure you have s3cmd installed and configured (with s3cmd --configure). Then you can configure the following script to run every 4 hours. I’m not going to go into much detail on this. If you’re familiar with s3cmd and Amazon S3/Glacier, you can probably figure out how it works. I wrote it as a short-term fix, but it’s worth sharing.

#!/bin/bash

# This script should be fired every 4 hours from a cron job until all
# data from the desired bucket is restored.
# Requires s3cmd 1.5 or newer

# Temp file
TEMPFILE=~/.s3cmd.restore.tmp

# Bucket to restore data from. Use trailing slash.
BUCKET="s3://bucketname/"

# Folder to restore data to. Use trailing slash.
FOLDER="/destination_folder/"

# Because of the way s3cmd handles errors, we have to run in a certain method
# 1: download/delete files from bucket,
# 2: run restore on the remaining objects
# 3: Do housekeeping on the downloaded data

if [ ! -f $TEMPFILE ]
then
touch $TEMPFILE
echo === Starting Download Phase
s3cmd -r --delete-after-fetch --rexclude "/$" sync $BUCKET $FOLDER
echo === Starting Restore Phase
s3cmd -r -D 30 restore $BUCKET
echo === Starting cleanup
# s3cmd doesn't delete empty folders, and can create empty files. Clean this up.
find $FOLDER -empty -delete
# but it might accidentially delete the target directory if the download didn't
# happen, so we have to fix that now
mkdir $FOLDER
rm $TEMPFILE
fi

Note that restore, download, and delete operations can incur extra costs. Be aware of that before proceeding.

So that’s it. I *should* have my entire S3 bucket downloaded completely within the next few days, and then I can migrate to what I hope is a more simplified archiving plan.

, , ,

Leave a comment

External optical drive woes

My old laptop, a Dell Latitude E6510, has a bad DVD drive. It won’t read most discs, and almost everything I try to burn, fails. I purchased an external USB drive to use with it, and that works well. It was cheaper than replacing the drive itself. Replacing the drive is straightforward, I just don’t want to spend the money right now to get a new drive.

About a week ago I wanted to install some software from a CD (Who does that anymore, right?) on my Asus tablet, which lacks an optical drive, but has a USB port. I plugged the external USB drive into the port, and put the disc in it. Shortly after, the drive kept disappearing from ‘My Computer’, I kept hearing the Windows disconnect and reconnect sounds. It wouldn’t stay ‘connected.’

My son likes to get into things that he shouldn’t, so I nearly chewed him out for it straight away, but after taking a bit to calm down, I decided to troubleshoot. I thought it was a bad cable at first, so I tried a few different cables. No change. Tried wiggling the USB port to see if the tablet port was faulty, no change. I plugged the drive back into the Dell, and it worked without a problem. This got me thinking.

After some of that thinking, I finally figured out what the issue was, and I’m posting this to see if anyone else can figure it out too, because you’ll learn something from my ordeal. Something that may surprise you.

This is a photo of the bottom of the USB drive, showing the label and the USB plug. Most of the information you need to figure this out is in this photo. You may need to go and do a web search if you’re missing a piece of information.

Post your comments, questions, or guesses in the comments below. The first person who figures it out will get recognition, and I’ll clarify the answer if it needs any. 

Hint: Again, almost everything you need to know to figure this out is in the photo of the USB drive. Almost. The rest is a quick web search away.

Hint: The Dell laptop has USB 2.0 ports. The tablet has USB 3.0. This is not the reason, although it’s going in the right direction. 

Guess/Hint: Someone guessed that the drive was made on April Fool’s Day. A clever guess sir, but not correct. You are looking in the right area.

Guess: Someone guessed that I might have been using the microUSB “On The Go” (OTG) port. A good guess, as this tablet has one of those ports as well, but no. I was using the USB 3 port that’s on the tablet’s keyboard dock. I did just try using the OTG port instead of the USB 3.0 port and got the same result. I am not surprised. 

Here are photos. Click any for a larger image:

Answer: My friend Gary got the idea that it might have been a power issue, but cwyenberg went so far as to point that 1.6A is a lot for a USB draw. 

Explanation: From Wikipedia:

USB power standards
Specification Current Voltage Power
USB 1.x and 2.0 500 mA[a] 5 V 2.5 W
USB 3.x 900 mA[b] 5 V 4.5 W

USB devices can only ask for so much power, (up to 500mA for a USB 2.0 device), and 1.6A is far beyond that. This drive can’t be expected to work under all circumstances according to the USB specification. The reason that it worked on the Dell laptop is probably that Dell is allowing the device to draw more power for the sake of compatibility. This is why you see external hard drives with Y-cables — the device can pull from 2 USB ports to get more power and still adhere to the USB standard, allowing the device to be expected to work under all circumstances. Interestingly enough, the drive does work with a powered USB hub rated for 2.5A. So, if you’re having a similar issue as me, try getting a powered USB hub. 

There’s a handy program I found called USBTreeView that can show you the requested power for each device connected to your USB bus. In this case, USBTreeView shows the following lines for this drive:

iManufacturer : 0x01
Language 0x0409 : "Hitachi-LG Data Storage Inc"
iProduct : 0x02
Language 0x0409 : "Portable Super Multi Drive"
...
MaxPower : 0xFA (500 mA)

Thanks for reading (and for your responses)!

Leave a comment

Fixing Windows 8.1 blank start menu

Some older programs not updated to work correctly with Windows 8/8.1 will produce some very undesirable results when uninstalling them, such as clearing out your start and app screens.

Echolink is one program that affects Windows 8 in this way when using its bundled uninstaller. You can read about how to extract the exe to run it as a standalone program in this post.

You can see below a sample of an affected system, and that all the tiles have been removed from the start screen and all apps screen:

EL_UnInst_Start_1EL_UnInst_Start_2

In this situation, the search from the start screen is available, but will not produce any results. The only way to navigate is to right-click on the start button and make selections from that menu, or to use the charms bar.

Various sites offer various fixes for this issue, and I haven’t found one that works without either refreshing the PC or having to use a restore point. If you use a restore point to recover, the program will likely be reinstalled. If you refresh your PC, you will keep your data but you may lose some settings or customizations.

For either of these two methods, follow the link below:

How to refresh, reset, or restore your PC [microsoft.com]

I wish I could find a consistent fix without having to use the above, but they methods do work.

If you have a solution that works, please feel free to share it in the comments below. Thank you!

, ,

Leave a comment

ACL fix for Synology DiskStations

A reader got in touch with me regarding my previous post, Quick sh script cronjob to fix user homes permissions on Synology. That script was initially intended to fix user homes file ownership, but this reader shared a script that uses the synoacltool to fix the Access Control List on directories.

A few thoughts regarding this script:

First, it was mentioned that these issues may be fixed in the latest DSM release. If you’re still experiencing file ownership and permissions issues, please feel free to use the solution linked to above or posted below.

Second, the script linked to above and the script below take different approaches on the problem. You may find a solution in one, or you may elect to use both.

Third, it was mentioned that this was a “one and done” solution. Due to the changing nature of filesystem content, I don’t believe that to be the case. You may want to save this as a sh script and run it as a scheduled task, or you may want it to run on every boot up. If you decide you want to run it on every boot, edit (or create) the file /etc/rc.local, and paste the below. I can’t say for certain whether this script is preserved on an upgrade, though this page strongly suggests that it would be preserved.

I don’t have a Synology unit right now to test this on, so I can’t offer any insight other than what I’ve shared above.

Here’s the script:

#!/bin/sh
synouser --enum all > user.list
sed -i 's/\\/\\\\/g' user.list
cat user.list | while read line
do
echo -n "$line: "
USERDIR=`synouser --get "$line" | grep "User Dir"`
if [ $? != 0 ]; then
echo "user: [$line] not found"
continue
fi
HOMEPATH=`echo "$USERDIR" | cut -d'[' -f2 | cut -d']' -f1`
synoacltool -get-archive "$HOMEPATH" | grep is_support_ACL > /dev/null 2>&1
if [ $? != 0 ]; then
echo "[$HOMEPATH] not support ACL or not exist"
continue
fi
synoacltool -get "$HOMEPATH" | grep -F "user:$line:allow:rwxpdDaARWcCo:fd--" > /dev/null 2>&1
if [ $? = 0 ]; then
echo "[$HOMEPATH] exist user's Full Control ACL"
continue
fi
synoacltool -add "$HOMEPATH" "user:$line:allow:rwxpdDaARWcCo:fd--"
done
rm user.list

Any feedback is welcome and appreciated. Thank you!

Leave a comment

Fixing wireless issues on Asus T100

I’ve seen issues with the Asus T100 where the Wi-Fi will frequently not reconnect after coming out of standby, hibernate, or a power off. Toggling airplane mode or rebooting the device will fix it, but not always the first time.

I found a fix for this, after reviewing the fix for frequently disconnecting Bluetooth devices, and the issues appear related.

Again, go into device manager and right-click the wireless network adapter, and click Properties. Go to the Advanced tab and change Minimum Power Consumption to Disabled.

Broadcom_Minimum_Power

After doing this, no more Wi-Fi issues!

, ,

Leave a comment

Fixing frequent Bluetooth disconnection on Asus T100

I found this fix specifically when looking for a way to fix disconnection issues with a Bluetooth mouse, but it fixes a multitude of issues as well.

Asus T100 frequently disconnects Bluetooth devices in an overly-aggressive effort to save power. Fix this by disabling Bluetooth power management.

First, make sure you have all the up-to-date drivers from Windows Update and from ASUS Live Update.

Next, open Device Manager and  right-click Bluetooth Radio, then click Properties, Power Management tab, and uncheck “Allow the computer to turn off this device to save power” as shows in the screenshot below.

ASUS_T100_Bluetooth_Properties

That’s all. After doing that, no more BT device disconnects!

Leave a comment

How to repair files identified as corrupted by sfc /scannow

A frequently used tool to identify Windows issues is the System File Checker, or sfc. It’s usually run as such:

sfc /scannow

This will scan through your system files and tell you if any files are corrupt. But — is there an easy way to repair those corrupt files, other than doing a full re-install? Turns out, there is: Connecting to the Internet, and using dism, as shown below:

dism /online /cleanup-image /restorehealth

See below for a copy-paste log that I did on the system I’m currently using to write this post. SFC identified corrupt files, I ran dism, and then re-ran sfc.

C:\WINDOWS\system32>sfc /scannow

Beginning system scan. This process will take some time.

Beginning verification phase of system scan. 
Verification 100% complete.

Windows Resource Protection found corrupt files but was unable to fix some of them. Details are included in the CBS.Log windir\Logs\CBS\CBS.log. For example C:\Windows\Logs\CBS\CBS.log. Note that logging is currently not supported in offline servicing scenarios.

C:\WINDOWS\system32>dism /online /cleanup-image /restorehealth

Deployment Image Servicing and Management tool 
Version: 6.3.9600.17031

Image Version: 6.3.9600.17031

[==========================100.0%==========================] 
The restore operation completed successfully. The component store corruption was repaired. 
The operation completed successfully.

C:\WINDOWS\system32>sfc /scannow

Beginning system scan.  This process will take some time.

Beginning verification phase of system scan.
Verification 100% complete.

Windows Resource Protection did not find any integrity violations.

Credits to this forum for having the original post of the fix.

Leave a comment

Event ID 129 on Windows Server 2012 on HP MicroServer N40L

Storage Spaces driver.

Symptoms of this issue:

  • Hard drive activity light solid on
  • Occasional disk hangs
  • Event ID 129 in Windows logs storahci “Reset to device, \Device\RaidPort0, was issued.”

Fix:

  • Go to Power Options > Advanced > PCI Express > Link State Power Management and set it to Off.
  • Reboot the server

Source: Event ID 129 – storachi – Reset to device, \Device\RaidPort0, was issued. (blogs.technet.com)

Fully resolved the issue for me.

 

, , , ,

Leave a comment

Installation of IE9 via Windows Update fails with 0x80092004

There are two ways to fix this, and I found I had to do both steps to make it work.

1) Open Internet Explorer. Go to Tools > Internet Options > Advanced > Reset and reset all settings. Try the update again.

Source: Installation of IE 9 fails with error code 80092004 (answers.microsoft.com)

2) Open an administrator-level command prompt, and run the following:

FORFILES /P %WINDIR%\servicing\Packages /M Microsoft-Windows-InternetExplorer-*9.*.mum /c "cmd /c echo Uninstalling package @fname && start /w pkgmgr /up:@fname /norestart"

Source: How to install ie9 (Internet Explorer 9) on Windows 7 despite 80092004 error (stackoverflow.com)

 

, ,

Leave a comment

How to format a disk GPT in FreeNAS

gpart create -s gpt adaX
gpart add -t freebsd-ufs adaX
newfs /dev/adaXp1

Also reference this post for more info:

https://mikebeach.org/2013/02/28/adding-and-removing-zfs-zpool-zil-disk-live-by-gptid/

,

Leave a comment