File copy problem.

[MuntyScruntfundle]

Hi there.

I'm pretty sure this more of a unix problem than related directly to the pi, but I thought I'd ask here just in case.

I've just been copying a bunch of files from a pi (raspbian) to a share on an iMac. When I run "ls | wc -l" on the source and the local directory share there's a deficit of over 4,000 files on the destination. If I go to the mac and run the same command I get the correct number.

So, where is the data going missing? Why is the count different on the pi/share and the destination machine?

Thanks.

[B.Goode]

It may not be the problem in this case, but note that you are not counting files, you are counting the number of lines in the output.

Are the formats of the output of the ls command identical on both systems?

[Heater]

You should use the -l switch to ls :

Code: Select all

$ ls -l | wc -l

Then the directory listing is one file per line plus an extra line giving total bytes:

Code: Select all

$ ls -l
total 3164744
-rw-r--r-- 1 rsm rsm       2598 Aug 13 02:47 app.js
-rw-r--r-- 1 rsm rsm       1613 Aug 13 02:47 box.js
-rw-r--r-- 1 rsm rsm       2167 Aug 13 02:49 chart.js
-rw-r--r-- 1 rsm rsm       1902 Aug  4 18:33 index.html
-rw-r--r-- 1 rsm rsm       1262 Aug 13 02:53 loadjs.js

Or better still, what about this:

Code: Select all

$ ls -1
app.js
box.js
chart.js
index.html
loadjs.js

That is the number "1" not a letter "l"

[n67]

Incidentally, you don't need the -1 option if the output is going to a pipe. It happens "as if by magic". Observe:

Code: Select all

$ ls | wc
     31      31     285
 $ ls | cat
(31 files, in 31 rows)
 $ ls
(31 files, arranged in 7 rows)
$ unbuffer ls | wc
      7      31     361
$ 

unbuffer is part of the "Expect" package. It tricks the program into thinking it is running on a pty, even though the output is redirected to a pipe.

[Heater]

By George you are right. Is that true of ls on the Mac and elsewhere?

[jahboater]

Just another ...

find . -type f | wc -l

[jahboater]

By George you are right. Is that true of ls on the Mac and elsewhere?

Its done that on all UNIX like systems for many years.

[Heater]

Excellent.

The only non-Linux Unix-like system I have ever used is a Sun 360 workstation back in 1990 something.

[n67]

By George you are right. Is that true of ls on the Mac and elsewhere?

Its done that on all UNIX like systems for many years.

Yes, I think it is pretty generic ls - has been that way since more or less forever.

In fact, on OSX, it is actually documented in "man ls":

-1 (The numeric digit ``one''.) Force output to be one entry per line. This is the
default when output is not to a terminal.

[jahboater]

The only non-Linux Unix-like system I have ever used is a Sun 360 workstation back in 1990 something.

I think I may have used that, was it SunOS then ?

I started with Amdahl UTS (Unix Timesharing System) running as a guest OS under the IBM VM/SP hypervisor on a large mainframe, in the early 80's. I used many many different UNIX's after that, and always dreamed of having my own UNIX system at home. Now we have perfectly good Linux machines running on a tiny $5 board!!

[jahboater]

People used to boast when presenting new software at a conference, that they had produced it "without modifying the kernel and without adding any options to ls"

[Heater]

I guess it was SunOS. I don't really recall.

I was just given this machine to use and a requirement to write crypto keys to magnetic cards connected to a card writer on it's serial port. It was an up hill struggle given that I had never seen Unix before and there was no internet to search for advice. I was a programmer of little 8 bit embedded systems at the time.

Loved every minute of it.

Had a bit of a shock when the tape cartridge drive broke. It cost thousands of pounds to get a field service engineer out from Sun to replace it!

Never mind the Pi, I'm totally stoked that I can run a Unix like system on anything as small as the Onion 2 https://www.kickstarter.com/projects/on ... d-by-linux

[B.Goode]

All fascinating history...

But to drag this back to the present day and the OP's question. My hunch was that there was some difference in behaviour between Raspbian and MacOS in respect of the command pipe that the OP was using. I don't have access to a MacOS system to verify this. Can anyone comment from actual experience?

(But I agree it would be safer to use commands for which the behaviour is understood and predictable.)

[Heater]

OK, coming back to the OP's question.

This may be nothing to do with whatever command one is using to count files.

It is stated that 4000 files might be missing.

Grief, how many files were being copied?!

On that scale I would be packaging them all up in a tar archive and compressing it. Then copying that and unpacking at the other end.

I'd also be tempted to verify the copy with an md5sum or such.

[cpc464]

Hi OP, as I understand it you have shared out a disk on the Mac and it is mounted on the Pi. You are copying files over the share, from the Pi to the Mac. Afterwards, you count the copied files from the Pi and they are 4000 short, but counting them on the Mac gets the right number.

If all the files are counted correctly on the Mac (destination) then the copy was good. I am not sure why the count is different on the Pi, unless you can post further information.

Your "ls | wc -l" command is correct and should work on both systems in the same way.

[n67]

Note to other posters "ls -1" is not the same as "ls", but it will output the same number of lines (files).

It is when the output is to a pipe (or, more specifically, when it is not to a terminal).

When output to a terminal, "ls -1" and "ls" output a different number of lines (except in certain edge cases, of course).

Anyway, back to topic: Yes, it is weird (and almost certainly not the case; I.e., there is something lost in the translation) for the source to have fewer files than the destination. If something had gone wrong with the copy, it would be the other way around. Unless, of course, there were files already present in the destination. Maybe that's what's really going on - the OP already had 4000 files in the destination directory; he copied over all, say, 2000, from the source, and now there are 6000 in the destination.

Or, I wonder if this is some variation of the "files hidden in a mounted directory" problem... (Just a wild guess here)

[cpc464]

You are right n67. I have just edited my last post to remove the erroneous last sentence.

As I understand it, the OP has fewer files on the target than the source, ie. some files have been missed. Unless he comes back with more information, it is impossible to say what has happened. The target disk might have filled up, or there might be a permissions issue, or he might have accidentally copied stuff into a sub direcrory, or something else.

[n67]

The original post was unclear, to say the least. I was going more on what some of the responders had written about the problem (I.e., their interpretations of it) than on the literal text of the OP.