How to exclude a directory from rsync

[Mick]

I am not sure how to exclude a directory on an ntfs partition from
being accessed during rsync. The attributes do not seem to be right
and it comes up with this error:
===============================
'rsync -a -l -v --exclude "/mnt/User_WinXP/System Volume Information"
-e "ssh -c blowfish -l root" /mnt/User_WinXP/
10.10.10.25:/home/httpd/backup'

sending incremental file list
rsync: opendir "/mnt/User_WinXP/System Volume Information" failed:
Permission denied (13)

sent 975792 bytes received 3760 bytes 23048.28 bytes/sec
total size is 11957666624 speedup is 12207.28
rsync error: some files/attrs were not transferred (see previous
errors) (code 23) at main.c(1042) [sender=3.0.7]
===============================

Is my syntax wrong?

This what mount shows for the partition (I use ntfs-3g):

/dev/sda7 on /mnt/User_WinXP type fuseblk
(rw,nosuid,nodev,noatime,allow_other,blksize=4096)

[Stefan G. Weichinger]

Post by Mick
I am not sure how to exclude a directory on an ntfs partition from
being accessed during rsync. The attributes do not seem to be right
===============================
'rsync -a -l -v --exclude "/mnt/User_WinXP/System Volume Information"

try something like "--exclude "./System Volume Information"

(relative paths)

or even "--exclude "./System\ Volume\ Information"

(escaping the spaces)

S

[Mick]

Post by Stefan G. Weichinger

Post by Mick
I am not sure how to exclude a directory on an ntfs partition from
being accessed during rsync. The attributes do not seem to be right
===============================
'rsync -a -l -v --exclude "/mnt/User_WinXP/System Volume Information"

try something like "--exclude "./System Volume Information"
(relative paths)
or even "--exclude "./System\ Volume\ Information"
(escaping the spaces)

Thanks Stefan, I'm afraid I'm still getting the same problem:

rsync: opendir "/mnt/User_WinXP/System Volume Information" failed:
Permission denied (13)

Why is rsync trying to open this directory, when I thought I've asked it to
exclude it?

[Stefan G. Weichinger]

Post by Mick
Permission denied (13)
Why is rsync trying to open this directory, when I thought I've asked it to
exclude it?

Maybe you did it wrong?

;-)

You don't show us what you did ...

[Mick]

Post by Stefan G. Weichinger

Post by Mick
Permission denied (13)
Why is rsync trying to open this directory, when I thought I've asked it to
exclude it?

Maybe you did it wrong?
;-)
You don't show us what you did ...

I ran the same as before but changed the path to the one you suggested:

===============================
'rsync -a -l -v --exclude "./System Volume Information"
-e "ssh -c blowfish -l root" /mnt/User_WinXP/
10.10.10.25:/home/httpd/backup'

sending incremental file list
rsync: opendir "/mnt/User_WinXP/System Volume Information" failed:
Permission denied (13)

[snip ...]

rsync error: some files/attrs were not transferred (see previous
errors) (code 23) at main.c(1042) [sender=3.0.7]
===============================

The System Volume is shown as 0700 (mounted with default permissions)
and is owned by root when viewed from Gentoo.

[Alan McKinnon]

Apparently, though unproven, at 15:28 on Tuesday 16 November 2010, Mick did

Post by Mick

Post by Stefan G. Weichinger

Post by Mick
Permission denied (13)
Why is rsync trying to open this directory, when I thought I've asked it
to exclude it?

Maybe you did it wrong?
;-)
You don't show us what you did ...

===============================
'rsync -a -l -v --exclude "./System Volume Information"
-e "ssh -c blowfish -l root" /mnt/User_WinXP/
10.10.10.25:/home/httpd/backup'
sending incremental file list
Permission denied (13)
[snip ...]
rsync error: some files/attrs were not transferred (see previous
errors) (code 23) at main.c(1042) [sender=3.0.7]
===============================
The System Volume is shown as 0700 (mounted with default permissions)
and is owned by root when viewed from Gentoo.

...

Don't think of --exclude as being a file path match, think of it as more a
regex (usually just a literal one). It specifies a pattern that if found if
the full pathname, results in the file not being synced.

The string "./System Volume Information" of course appears nowhere in the
rsync output list of files.

I think you want:

--exclude "/System Volume Information"

Then rsync will not attempt to open the directory at all

[Mick]

Post by Alan McKinnon
Apparently, though unproven, at 15:28 on Tuesday 16 November 2010, Mick did

Post by Mick

Post by Stefan G. Weichinger
You don't show us what you did ...

===============================
'rsync -a -l -v --exclude "./System Volume Information"
-e "ssh -c blowfish -l root" /mnt/User_WinXP/
10.10.10.25:/home/httpd/backup'
sending incremental file list
Permission denied (13)
[snip ...]
rsync error: some files/attrs were not transferred (see previous
errors) (code 23) at main.c(1042) [sender=3.0.7]
===============================
The System Volume is shown as 0700 (mounted with default permissions)
and is owned by root when viewed from Gentoo.

Don't think of --exclude as being a file path match, think of it as more a
regex (usually just a literal one). It specifies a pattern that if found if
the full pathname, results in the file not being synced.
The string "./System Volume Information" of course appears nowhere in the
rsync output list of files.
--exclude "/System Volume Information"
Then rsync will not attempt to open the directory at all

...

Excellent, it worked! :-)

Thank you all for your help.

[Alan McKinnon]

Apparently, though unproven, at 23:12 on Tuesday 16 November 2010, Mick did

Post by Mick

Post by Alan McKinnon
Don't think of --exclude as being a file path match, think of it as more
a regex (usually just a literal one). It specifies a pattern that if
found if the full pathname, results in the file not being synced.
The string "./System Volume Information" of course appears nowhere in the
rsync output list of files.
--exclude "/System Volume Information"
Then rsync will not attempt to open the directory at all

Excellent, it worked! :-)

Glad to hear it.

I could help because part of my job is running a rather big public ftp mirror
that management graciously pay for. And I went down this rsync road a long
time ago myself.

You have no idea how many brain cells died in agony to figure out this
specific piece of rsync behaviour :-)

[Stefan G. Weichinger]

Post by Alan McKinnon
Apparently, though unproven, at 23:12 on Tuesday 16 November 2010, Mick did

Post by Mick
Excellent, it worked! :-)

Glad to hear it.
I could help because part of my job is running a rather big public ftp mirror
that management graciously pay for. And I went down this rsync road a long
time ago myself.
You have no idea how many brain cells died in agony to figure out this
specific piece of rsync behaviour :-)

;-)

I would like to know if my suggestion also works ;-)

Yeah, include/exclude-patterns are rather hard to figure out sometimes
... nearly like regexes -> write once, read never ....

S

[Mick]

Post by Stefan G. Weichinger

Post by Alan McKinnon
Apparently, though unproven, at 23:12 on Tuesday 16 November 2010, Mick did

Post by Mick
Excellent, it worked! :-)

Glad to hear it.
I could help because part of my job is running a rather big public ftp
mirror that management graciously pay for. And I went down this rsync
road a long time ago myself.
You have no idea how many brain cells died in agony to figure out this
specific piece of rsync behaviour :-)

;-)
I would like to know if my suggestion also works ;-)
Yeah, include/exclude-patterns are rather hard to figure out sometimes
... nearly like regexes -> write once, read never ....

...

Ha, ha! True!

Stefan, I tried escaping the spaces (even tried \\ double and \\\ triple
escapes in case it makes a difference because of using ssh) but still did not
work. In my head I couldn't see how the full path would not work, but the
relative path would, but I tried it out all the same.

I still don't understand why Alan's recommendation works ;-)

[Renat Golubchyk]

Post by Mick

Post by Stefan G. Weichinger

Post by Alan McKinnon
Apparently, though unproven, at 23:12 on Tuesday 16 November 2010, Mick did

Post by Mick
Excellent, it worked! :-)

Glad to hear it.
I could help because part of my job is running a rather big
public ftp mirror that management graciously pay for. And I went
down this rsync road a long time ago myself.
You have no idea how many brain cells died in agony to figure out
this specific piece of rsync behaviour :-)

;-)
I would like to know if my suggestion also works ;-)
Yeah, include/exclude-patterns are rather hard to figure out
sometimes ... nearly like regexes -> write once, read never ....

Ha, ha! True!
Stefan, I tried escaping the spaces (even tried \\ double and \\\
triple escapes in case it makes a difference because of using ssh)
but still did not work. In my head I couldn't see how the full path
would not work, but the relative path would, but I tried it out all
the same.
I still don't understand why Alan's recommendation works ;-)

...

I'm probably late with my reply, but I'll post it so it will be in the
archives for future reference.

The man page is actually pretty clear on this issue. Quote:

if the pattern starts with a / then it is anchored to a particular
spot in the hierarchy of files, otherwise it is matched against the
end of the pathname. This is similar to a leading ^ in regular
expressions. Thus "/foo" would match a name of "foo" at either the
"root of the transfer" (for a global rule) or in the merge-file’s
directory (for a per-directory rule). An unqualified "foo" would
match a name of "foo" anywhere in the tree because the algorithm is
applied recursively from the top down; it behaves as if each path
component gets a turn at being the end of the filename. Even the
unanchored "sub/foo" would match at any point in the hierarchy
where a "foo" was found within a directory named "sub".

"Root of the transfer" is the directory you want to sync. Thus, if you
run e.g. "rsync /var/log/ /mnt/backups/ --exclude=/portage/" then root
of the transfer is /var/log, and therefore the directory
/var/log/portage will be excluded. If on the other hand you write
--exclude=portage/ then a directory named portage anywhere in the tree
under /var/log will be excluded. Without the trailing slash, i.e. just
--exclude=portage any file (regular file, directory, link, whatever)
named portage anywhere in the tree gets excluded. And finally
--exclude=/portage would exclude a file only at the top of the tree that
is going to be synchronsed.

Hope it helps.

Cheers,
Renat

[Mick]

Post by Renat Golubchyk

Post by Mick

Post by Stefan G. Weichinger

Post by Alan McKinnon
Apparently, though unproven, at 23:12 on Tuesday 16 November 2010, Mick did

Post by Mick
Excellent, it worked! :-)

Glad to hear it.
I could help because part of my job is running a rather big
public ftp mirror that management graciously pay for. And I went
down this rsync road a long time ago myself.
You have no idea how many brain cells died in agony to figure out
this specific piece of rsync behaviour :-)

;-)
I would like to know if my suggestion also works ;-)
Yeah, include/exclude-patterns are rather hard to figure out
sometimes ... nearly like regexes -> write once, read never ....

Ha, ha! True!
Stefan, I tried escaping the spaces (even tried \\ double and \\\
triple escapes in case it makes a difference because of using ssh)
but still did not work. In my head I couldn't see how the full path
would not work, but the relative path would, but I tried it out all
the same.
I still don't understand why Alan's recommendation works ;-)

I'm probably late with my reply, but I'll post it so it will be in the
archives for future reference.
if the pattern starts with a / then it is anchored to a particular
spot in the hierarchy of files, otherwise it is matched against the
end of the pathname. This is similar to a leading ^ in regular
expressions. Thus "/foo" would match a name of "foo" at either the
"root of the transfer" (for a global rule) or in the merge-file’s
directory (for a per-directory rule). An unqualified "foo" would
match a name of "foo" anywhere in the tree because the algorithm is
applied recursively from the top down; it behaves as if each path
component gets a turn at being the end of the filename. Even the
unanchored "sub/foo" would match at any point in the hierarchy
where a "foo" was found within a directory named "sub".
"Root of the transfer" is the directory you want to sync. Thus, if you
run e.g. "rsync /var/log/ /mnt/backups/ --exclude=/portage/" then root
of the transfer is /var/log, and therefore the directory
/var/log/portage will be excluded. If on the other hand you write
--exclude=portage/ then a directory named portage anywhere in the tree
under /var/log will be excluded. Without the trailing slash, i.e. just
--exclude=portage any file (regular file, directory, link, whatever)
named portage anywhere in the tree gets excluded. And finally
--exclude=/portage would exclude a file only at the top of the tree that
is going to be synchronsed.
Hope it helps.

...

Yes it does! Thank you, it's clear to me now why it behaved so.

[Renat Golubchyk]

On Fri, 26 Nov 2010 09:02:49 +0100 Helmut Jarausch

Post by Renat Golubchyk
if the pattern starts with a / then it is anchored to a particular
spot in the hierarchy of files, otherwise it is matched against the
end of the pathname. This is similar to a leading ^ in regular
expressions. Thus "/foo" would match a name of "foo" at either
the "root of the transfer" (for a global rule) or in the
merge-file’s directory (for a per-directory rule). An
unqualified "foo" would
match a name of "foo" anywhere in the tree because the
algorithm is
applied recursively from the top down; it behaves as if each path
component gets a turn at being the end of the filename. Even the
unanchored "sub/foo" would match at any point in the
hierarchy where a "foo" was found within a directory named "sub".
"Root of the transfer" is the directory you want to sync. Thus, if you
run e.g. "rsync /var/log/ /mnt/backups/ --exclude=/portage/" then root
of the transfer is /var/log, and therefore the directory
/var/log/portage will be excluded. If on the other hand you write
--exclude=portage/ then a directory named portage anywhere in the tree
under /var/log will be excluded. Without the trailing slash, i.e. just
--exclude=portage any file (regular file, directory, link, whatever)
named portage anywhere in the tree gets excluded. And finally
--exclude=/portage would exclude a file only at the top of the tree that
is going to be synchronsed.

Let me add some caveat which has trapped me recently.
I had (in your terms)
rsync /var/log/ /mnt/backups/ --delete --exclude=/portage/
and /var/log/portage was just a symlink to some other directory
while /mnt/backups/portage was a real directory.
In that case rsync deletes /mnt/backups/portage !
It looks as if the "directory property" is check in the source tree
only. Quite an unpleasant surprise.

...

The behavior is logical and consistent with the command you provided.
If you tell rsync to synchronise two directories and delete everything
in the destination that is not in the source then rsync does just that.
And a symbolic link is not a directory. One might think that exclude
rule applies to both source and destination, but it doesn't. That's a
common pitfall. But the documentation is clear on that too. It says,

"This option allows you to add rules to selectively exclude certain
files from the list of files to be transferred."

Since destination directory is not going to be transferred it is clear
that this option doesn't apply there.

In order to avoid the problem of files being accidentally deleted I
just never run rsync without running it with --dry-run first if I
specify a --delete option. Thus I have an opportunity to review the
file list before anything unpleasant is done.


Cheers,
Renat

[Stefan G. Weichinger]

Post by Mick
===============================
'rsync -a -l -v --exclude "./System Volume Information"
-e "ssh -c blowfish -l root" /mnt/User_WinXP/
10.10.10.25:/home/httpd/backup'

Try it with escaped spaces as I mentioned also:

--exclude "./System\ Volume\ Information"

or even

--exclude "./System*"

if there is nothing else starting with "System" ;-)

S

[Robin Atwood]

On Tuesday 16 November 2010, Mick wrote:

Try:

'rsync -a -l -v --exclude "*/System Volume Information"
-e "ssh -c blowfish -l root" /mnt/User_WinXP/
10.10.10.25:/home/httpd/backup'

From "man rsync"
"Note that if you don’t specify --backup-dir, (1) the --omit-dir-times option
will be implied, and (2) if --delete is also in effect (without --delete-
excluded), rsync will add a "protect" filter-rule for the backup suffix to the
end of all your existing excludes (e.g. -f "P *~"). This will prevent
previously backed-up files from being deleted. Note that if you are supplying
your own filter rules, you may need to manually insert your own
exclude/protect rule somewhere higher up in the list so that it has a high
enough priority to be effective (e.g., if your rules specify a trailing
inclusion/exclusion of ’*’, the auto-added rule would never be reached)."

HTH
-Robin