The Gentle News

Dems Darth Vader Farmers.

The Gentle News header image 2

File Transfer Woes

October 31st, 2005 · 7 Comments · Tech

I’m shuffling around all of the PCs at home. The two main goals are to (1) give B a nicer machine on which to play her silly human simulator games and (2) to have a more robust and snappy file/game server in the living room.

The bitch is that I have about 500 gigabytes of data on one machine that needs to be on another and, due to logistics, I can’t just move the harddrives into the same box and copy them. So, I’m doing all of this over my 100Mbit network. I’m seeing some dramatically varied transfer rates depending on the protocol I’m using. If you can shed any light onto this, please share with the rest of the class:

1) If I use ssh/sftp via Nautilus, I can only get an effective throughput of about 10Mbit/s.
2) If I use ssh/sftp via command-line sftp, I can only get an effective throughput of about 20Mbit/s.
3) If I use SMB via Nautilus, I can only get about 35Mbit/s.
4) If I use ftp, whether via Nautilus or various command-line ftp clients, I get a nice 90-95Mbit/s.

This isn’t a huge deal, except that I would prefer to use ssh/sftp, and Ubuntu & Nautilus make this very easy to do, interface-wise. It’s no easier than ftp via Nautilus, but it’s just the principle that gets me.

Anyway, I guess I’m going to use Nautilus & FTP to finish all this up, but first, I’m going to install some new toys. I finally broke down and bought a WRT54G and a couple of 802.11g cards for B’s machine and my arcade machine (John Williams, if you’re nasty).

In case I hadn’t mentioned it before, I am now a more or less total Ubuntu convert. If you’re a desktop Linux user, and you don’t have any good reason not to do so, I’d suggest you take a look at Ubuntu the next time you reinstall. The ease of installation, straight-forward UI, and debianish methods of cooking up your own additions make this distribution ballsy enough for you to rice it up as much as you’d like, but idiot-proof enough for your mother to use on a daily basis.

Not that your mother’s an idiot. I’ve met her, and she’s a damn nice gal. A little too fond of the firewater on occasion, but who can blame her?

pixelstats trackingpixel

Tags:

7 Comments so far ↓

  • Justinm

    Not being anywhere near an expert on the subject, I would assume that the basic difference between sftp and plain ol’ ftp is the overhead that is going into the encryption/decryption. Normally this overhead doesn’t bug you, but, at the data quantity you’re talking about, that overhead adds up quickly. As far as why the interface versions are slower than their command-line counterparts, I’ll leave that to more knowledgeable folk.

    And, as far as my mother is concerned, she did usually have a shot or two before walking out the door of The Plantation, but, who can discredit her for that?

  • Farris

    It can’t be overhead, because the numbers I’m giving are actual throughput on the interface, as measured by iptraf. I sorta misspoke when I said “effective throughput”.

    Oh, and… um… Yeah. Your mama.

  • Jim

    I think by overhead… JustinM means CPU overhead.

    With FTP, for instance, the CPU has very little to do. The data is not converted in anyway. It’s simply crammed into packets and shipped across the network. With SSH it has to be encrypted on one end and then decrypted on the other, and, unfortunately, that’s a lot of math. So, your CPU will be VERY involved with an SSH transaction. Now, if you use SFTP over SSH, you have to deal with an SSH subsystem and any of the sillyness involved there…. and it just gets dirtier and dirtier.

    As far as why nautilus SSH would be slower than command line SSH more than likely has to do with packet size and any extra CPU being sucked out by whatever fancy dialog box the GUI is trying to show you. Nautilus isn’t know for speed.

    What I’m trying to say is, your network and your disks are probably NOT your bottleneck. CPU is playing a large role here.

    At work about a year ago they pushed out a requirement to use encryption for all data travelling over the intranet, even if it isn’t destined to leave our network. We saw a HUGE slow down from almost EVERY application… even though they are all written in different languages, on different platforms, with different purposes.

  • Josh-Daniel Davis

    Most SFTP clients still run synchronously. They wait for ACK before sending another packet. This is a huge perf loss. SCP doesn’t do that, but there’s still alot more protocol overhead.

    I recommend TLS over FTP. It’s still not as fast as ftp, but it’s as close as you can get with encryption.

    vsftpd supports TLS. I use filezilla (from sourceforge) for winders client.

  • Josh-Daniel Davis

    That doesn’t account for the SMB slowdown.

    Network tuning (send and receive buffers) would help SMB.

    SFTP is poorly implemented in many places.

    TLS/FTP is pretty good, all things considered.

  • Farris

    The next time I need to move data securely, I’ll probably go with FTP/TLS. Thanks, dude.

    It’s GOTTA be the acks that are slowing this thing down. No amount of CPU overhead on a P4 2GHz (I forget the bus speed) can account for an almost 90% performance loss.

  • Frenchy

    Hi Farris, did you find a solution to why the transfers are slow when using SFTP through Nautilus? I have the same problem and I, like you, am also an Ubuntu convert. I wonder if it’s something in Ubuntu.

    I’ve read through the responses and thought that you got a pretty raw deal. Firstly, the ACK is done at the TCP/IP stack level, not by client software. Happy to be corrected on this.

    Also, rule out CPU overhead because the command line version is fast and therefore SSH (i.e. the CPU time required for compression) isn’t a problem. Also, to backup my claim, when I transferred a 600 MB file, the idle time on both client and server remained above 90%. CPU time is not the problem.

    Just as a comparison, my 600 MB file took 10 minutes to transfer which , by my calculations, is 1 MB/sec which is 8 Mbits/sec, very slow.

    The last piece to the puzzle is that all network communications to the box slow down when I’m transferring the file through Nautilus using SSH/SFTP. I think that I’m going to run a packet sniffer on the network to see what’s being sent, more to the point, what’s taking up the bandwidth.

Leave a Comment