Extremely slow USB to USB transfer

[rizzyc]

150 kb/s transfer rate.
I have a 1.5 TB (exFat 4) hard drive hooked up to my Pi, I wanted to transfer a file from that hard hard drive to a 16 gig (FAT32) USB stick. Both devices were plugged into the Pi directly (no hub was used).

I tried transferring a 8 gig file from my HD to my USB stick and the transfer rates were abysmally slow. Around 150 kb/s. I thought this would be the fastest way to transfer the files. Is this some limitation on the FAT32 partion? My 1.5 TB is formatted as the default linux partition.

On a side note, i was able to SFTP files from my Pi to my Macbook over the network at almost 1 MB/second, but I thought the USB to USB should have been much faster.

[mk01]

this we need to put on wiki as well.

mostly this is caused by default XBian setup to mount external media with "sync" mount option. this provides additional layer of filesystem protection as it avoids corruptions if the USB is removed without proper UNMOUNT before, but other side of story is speed (and much faster wear out for USB sticks / SD cards).

150kbps sound like this exactly.

I don't which version we are talking about, with Beta2 you can change this in xbian-config under xbmc, if Beta1X check /etc/usbmount/usbmount.conf for mount options and change "sync" to "async" where you find it.

then reboot. drives will be re-mounted with new params and transfers should be better. You should be able to write to USB via network let's say at sustained 6-7MB (if RPI is not otherwise busy) - at beginning you can peak at 10 until buffers fill up. USB - USB never tested but don't expect much higher from this as RPI's USB implementation is not the perfect one and is quite CPU consuming.

in any case it should be more acceptable than 150kbps

[Jingles]

I am having similar problems. I am running beta 2 and I have disable the auto mount feature in favour of fstab with the options 'defaults' (i.e. inc. async).

I am trying to copy a file at the minute, which is 1.3GB. I'm using rsync to copy from one from a 1TB drive to a flash drive. The copy starts off at ~5MB/s but very quickly drops to <150kB/s! Every know and then it does go back up to around 400MB/s but then drops again.

I have noticed that the timer you get when using the -P flag with rsync doesn't seem to change. The % copied, speed and time left seem to jump rather than continuously tick.

This only seems to have happened since putting beta 2 on. Despite the troubles (there were many!) I was having with beta 1 file transfer like this remained around 4-5MB/s.

Put simply this is driving me crazy! Help!

[Jingles]

I think I have the cause of the problem but don't know how to fix it.

I have two 1TB external USB HDDs attached via a USB hub. I have some tests using dd to see the read and write speeds of my drives. Results show write speed of both drives is fine (~20-28MB/s depending on block size and amount of data) but the read speed for one drive is really slow which will obviously be causing the slow data transfer between the two disks. However when I attach the drive to a different PC it reads at 35 MB/s?! I have summarised the results below and also attached detailed copies of the dd results, and hdpram test, and fdisk outputs.

To me is seems there is a problem with the drive working on my Raspberry Pi but I have no idea what to do now to fix it. It is really an issue because the drive concerned has all my tv shows on and consequently I can't watch anything because xbmc can't get the video quick enough to play it!

I really hope someone can help

Drive 'vexed' read speeds:
19.1 MB/s (bs = 1048576, count = 1024)
122 MB/s (bs = 8k, count = 10k)

Drive 'vashet' read speeds:
446kB/s (bs = 1048576, count = 1024)
319 kB/s (bs = 8k, count = 10k)

When I attached the problem drive to another PC (desktop PC running opensuse 12.x) I got the following read/write speeds (bs = 1048576, count = 1024):
read = 35.8 Mb/s
write = 29.7 MB/s

[IriDium]

Just for a test - can you try hddparm -tT on each drive again - but with only one drive connected. (is it the USB port?)
Also - can you swap the drives around, so that they are the other way around. (Is is sdb1 that is causing the issue?)

[Jingles]

I will try that when I get home on Sunday. Thanks

[ganchito55]

(28th Nov, 2013 09:52 AM)Jingles Wrote:  I am having similar problems. I am running beta 2 and I have disable the auto mount feature in favour of fstab with the options 'defaults' (i.e. inc. async).

I am trying to copy a file at the minute, which is 1.3GB. I'm using rsync to copy from one from a 1TB drive to a flash drive. The copy starts off at ~5MB/s but very quickly drops to <150kB/s! Every know and then it does go back up to around 400MB/s but then drops again.

I have noticed that the timer you get when using the -P flag with rsync doesn't seem to change. The % copied, speed and time left seem to jump rather than continuously tick.

This only seems to have happened since putting beta 2 on. Despite the troubles (there were many!) I was having with beta 1 file transfer like this remained around 4-5MB/s.

Put simply this is driving me crazy! Help!

I just do it and the speed has changed from 150kb/s to ~6.5MB/s (via Samba). I use a ext4 partition.
Try to do it.

Also I think that the sync option should be disable.

[mk01]

@ganchito55

sync mount option can be enabled/disabled directly from xbian-config in XBMC, but the XBian default is "sync" by default for external auto mounted media.

@Jingles

can you please post output of "mount" command with the drives attached. also can you retest with the destination / source drives attached via powered hub?

can you also test copying when the stick is formatted as ext4 OR btrfs?

[Jingles]

Hi IriDium, mk01,

Thank you so much for your replies.

The drives are both 1TB portable hard rives formatted with ext4. They are always connected to a powered USB hub as they are bus powered drives, and the Pi doesn't like it if I plug them into it directly. (@mk01 I can't test them when formatted with btrfs as they both have 400GB+ of movies and TV on them atm!)

I didn't get chance to look at this until yesterday. I ran the hdparm -tT on the drives connected in a variety of ways and got the same results each time – both drives scored 100+ MB/s on the cached read, and 'vexed' scored ~20 MB/s on the buffered, while 'vashet' only scored something around 1-300 kB/s. I don't think the drives were mounted during these tests. The connections i tested:
*vexed only on /dev/sda and /dev/sdb
*vashet only on /dev/sda and /dev/sdb
*vexed on /dev/sda and vashet on /dev/sdb
*vashet on /dev/sda and vexed on /dev/sdb

@IriDium I think that rules out it being /dev/sda that was causing the problem. And whether i have one or both drive connected, and the USB port. The powered hub is plugged into one port and the drives have been switched around on the hub itself many times.

Monday night I also rebooted my Pi as I was tidying the power cables for my media setup and ran some tests afterwards. 'vashet' now reads fine and it is 'vexed' that is reading slow! They are currently attached with vashet on /dev/sda, vexed on /dev/sdd (despite what the mount output says!). I have attached the mount, fdisk and hdparm commands for the drives as they are currently attached. I mentioned earlier that I have disable the auto-mount feature in xbian and use fstab to mount the drives now. The fstab entries are:
LABEL={label} /media/{name} ext4 defaults 0 0

I have also noticed that whenever running any of the commands above the Pi always take a while to get the data for the disk that is reading slow. For example if I run fdisk it spews out everything very quickly, but then takes maybe a minute or two to get the information for the 'slow-disk'. I have seen the outputs come out in the wrong order to (say the 'slow-disk' is on /dev/sda my fdsik output can read: dev/mmcblk0 info, then dev/sdb info, pause then dev/sda info. It is similarly slow to put the headers for the hdparm command on screen when running that on the 'slow-disk'

I have tried to be a clear and comprehensive as possible in my descriptions, but am still relatively new to Linux so if there is anything I have missed or need to clarify then please say. I look forward to hearing your suggestions!

Jingles

[mk01]

@Jingles

you are fine and clear - we just ask stupid questions - to be sure we really have something there and not ugly hidden misconfiguration. this really now looks as some kind of hw/kernel/rpi combo issue. you told us if you connect the hub as it is to pc (with all the disks as they are normally attached) the pc has no issues with them, right?

can you turn on debug (put "debug" into cmdline.txt), reboot, open console, sudo -i and start

Code:

while sleep 0.2; do dmesg -c -T -x; done

then from second console start even the fdisk - I wan't to see the output from kernel when there is this long time to access the device.

post to pastebin. maybe would be better to redirect the dmesg to file " > /tmp/dmesg.log" as with debug logging it could be more than one screen size of output.

thanks

[Jingles]

Well that is good to know

When I tried it on the desktop PC I only tried it with the one hard drive I was having difficulties with, and with out the hub. Mainly because the PC I used was at my office not at home, and at that time I was just checking it wasn't a problem with that particular drive.

I have never used debug mode so when you say put "debug" into cmdline.txt, where is that file? And do you literally mean once I am in the console after rebooting enter:

Code:

sudo -i
while sleep 0.2; do dmesg -c -T -x > /tmp/dmesg.log; done

Did I get that right how to send the output to a log file?

Then when that has finished:

Code:

sudo fdisk -l

Do you want the fdisk output in a log as well?

I'll try this when I get home tonight.

[mk01]

if there are no error on fdisks output then no.

---

put the redirect at the end of the line after "done", or keep where it is but put >> instead. otherwise the log will be truncated (recreated fresh) on each round of dmesg.

cmdline.txt is at /boot and hold "kernel command line parameters". it is one line, doesn't matter order of all options. just be sure put space before and after.

mk

[Jingles]

(5th Dec, 2013 06:28 AM)mk01 Wrote:  cmdline.txt is at /boot and hold "kernel command line parameters". it is one line, doesn't matter order of all options. just be sure put space before and after.

What do you mean by " hold 'kernel command line parameters'"?

[mk01]

@Jingles

"/boot/cmdline.txt file holds parameters sent to kernel for booting". as from grub or other boot loader (if you know any other linux a bit).

[Jingles]

I just read your message wrong makes total sense and now feel slightly daft!

Anyway...

I added debug in cmdline.txt, rebooted and ran the commands. I have attached the log file. I have also attached the fdisk, and a hdparm with the -I option just in case.

I don't think the dmesg has the info you want though. When I ran the command I got a new line in terminal with a blinking cursor ( _ ), but it didn't come back to the normal prompt when it's ready for the next command (i.e. root@sierra #~), even when i left it overnight. I had to kill it with ^ C.

When I looked at the log it only has info for /dev/sda and it was still /dev/sdb that was reading slowly (see hdparm.log). NB the last five entries (resetting USB device) appeared after I ran the hdparm command.