Discussion:
DFS - can I copy data between locations physically to reduce replication?
(too old to reply)
boe
2009-05-05 03:22:42 UTC
Permalink
Hello,

I have an office overseas. I have about 200 GB of data over there that I
want on our server over here in the US. Using DFS and starting from
scratch it could take a very very very long time to replicate and clog up my
bandwidth at both offices. Could I send a hard drive of all the data from
the overseas office to the US and then create the DFS after the data is
copied on to our US server? Will it still want to replicate the data since
the creation/modified dates will be different?

Thanks
Isaac Oben [MCITP:EA, MCSE]
2009-05-05 04:27:01 UTC
Permalink
Hello Boe,

I haven't tried it yet, but I think it should work if you keep same folder
names, permissions etc, then w2k8 dfsr will just replicate the changes
accross.

Isaac
Post by boe
Hello,
I have an office overseas. I have about 200 GB of data over there that I
want on our server over here in the US. Using DFS and starting from
scratch it could take a very very very long time to replicate and clog up my
bandwidth at both offices. Could I send a hard drive of all the data from
the overseas office to the US and then create the DFS after the data is
copied on to our US server? Will it still want to replicate the data since
the creation/modified dates will be different?
Thanks
Isaac Oben [MCITP:EA, MCSE]
2009-05-05 04:30:59 UTC
Permalink
To followup from previous post, you can test by setting up a test lab before
you embark on transporting data..Or you can replicate data at night when
client traffic is less.

Isaac
Post by boe
Hello,
I have an office overseas. I have about 200 GB of data over there that I
want on our server over here in the US. Using DFS and starting from
scratch it could take a very very very long time to replicate and clog up my
bandwidth at both offices. Could I send a hard drive of all the data from
the overseas office to the US and then create the DFS after the data is
copied on to our US server? Will it still want to replicate the data since
the creation/modified dates will be different?
Thanks
Dave Warren
2009-05-06 07:06:33 UTC
Permalink
Post by boe
I have an office overseas. I have about 200 GB of data over there that I
want on our server over here in the US. Using DFS and starting from
scratch it could take a very very very long time to replicate and clog up my
bandwidth at both offices. Could I send a hard drive of all the data from
the overseas office to the US and then create the DFS after the data is
copied on to our US server? Will it still want to replicate the data since
the creation/modified dates will be different?
Research "prestaging"

The short answer is that yes, you can. If you're using 2008 on both
sides, I'd suggest just copying the data and letting DFS-R worry about
permissions and attributes, DFS-R is *very* efficient at these sorts of
things, I did this with over a million files and around 3TB of data,
only a couple GB went across the wire during the sync-up process.

If you're using 2003 R2 on either or both sides, research how to
prestage if you're using robocopy then something like the following will
do the trick:

/e /sec /w:3 /r:2 /purge /xd DfsrPrivate

It's not perfect, you'll still see a reasonable chunk of data flow back
and forth during the sync-up process, but it's better then all 200GB.

IIRC the other big thing that you need to do in 2003 before using the
above is to turn off the "inherit from parent" option on both sides and
manually set the root's permissions, otherwise robocopy's writes aren't
quite right.

Regardless of how close your initial prestaging is, DFS-R does it's best
to only transfer the data required, as long as the file content matches,
DFS-R will update the meta data on it's own.

The only other thought that comes to mind is to make your staging
directories as large as possible, on 2008 I'd go as large as 64GB if you
have the free space, this increases efficiency significantly during the
initial replication, once both sides agree that they're in sync then
drop this back to a sane value.
boe
2009-05-06 14:04:00 UTC
Permalink
Thanks!
Post by Dave Warren
Post by boe
I have an office overseas. I have about 200 GB of data over there that I
want on our server over here in the US. Using DFS and starting from
scratch it could take a very very very long time to replicate and clog up my
bandwidth at both offices. Could I send a hard drive of all the data from
the overseas office to the US and then create the DFS after the data is
copied on to our US server? Will it still want to replicate the data since
the creation/modified dates will be different?
Research "prestaging"
The short answer is that yes, you can. If you're using 2008 on both
sides, I'd suggest just copying the data and letting DFS-R worry about
permissions and attributes, DFS-R is *very* efficient at these sorts of
things, I did this with over a million files and around 3TB of data,
only a couple GB went across the wire during the sync-up process.
If you're using 2003 R2 on either or both sides, research how to
prestage if you're using robocopy then something like the following will
/e /sec /w:3 /r:2 /purge /xd DfsrPrivate
It's not perfect, you'll still see a reasonable chunk of data flow back
and forth during the sync-up process, but it's better then all 200GB.
IIRC the other big thing that you need to do in 2003 before using the
above is to turn off the "inherit from parent" option on both sides and
manually set the root's permissions, otherwise robocopy's writes aren't
quite right.
Regardless of how close your initial prestaging is, DFS-R does it's best
to only transfer the data required, as long as the file content matches,
DFS-R will update the meta data on it's own.
The only other thought that comes to mind is to make your staging
directories as large as possible, on 2008 I'd go as large as 64GB if you
have the free space, this increases efficiency significantly during the
initial replication, once both sides agree that they're in sync then
drop this back to a sane value.
Kyle BLake
2009-05-07 19:15:03 UTC
Permalink
Another point is you can limit the bandwidth used for replication. So you
could just set it up and limit is and let it take its time.
Post by boe
Thanks!
Post by Dave Warren
Post by boe
I have an office overseas. I have about 200 GB of data over there that I
want on our server over here in the US. Using DFS and starting from
scratch it could take a very very very long time to replicate and clog up my
bandwidth at both offices. Could I send a hard drive of all the data from
the overseas office to the US and then create the DFS after the data is
copied on to our US server? Will it still want to replicate the data since
the creation/modified dates will be different?
Research "prestaging"
The short answer is that yes, you can. If you're using 2008 on both
sides, I'd suggest just copying the data and letting DFS-R worry about
permissions and attributes, DFS-R is *very* efficient at these sorts of
things, I did this with over a million files and around 3TB of data,
only a couple GB went across the wire during the sync-up process.
If you're using 2003 R2 on either or both sides, research how to
prestage if you're using robocopy then something like the following will
/e /sec /w:3 /r:2 /purge /xd DfsrPrivate
It's not perfect, you'll still see a reasonable chunk of data flow back
and forth during the sync-up process, but it's better then all 200GB.
IIRC the other big thing that you need to do in 2003 before using the
above is to turn off the "inherit from parent" option on both sides and
manually set the root's permissions, otherwise robocopy's writes aren't
quite right.
Regardless of how close your initial prestaging is, DFS-R does it's best
to only transfer the data required, as long as the file content matches,
DFS-R will update the meta data on it's own.
The only other thought that comes to mind is to make your staging
directories as large as possible, on 2008 I'd go as large as 64GB if you
have the free space, this increases efficiency significantly during the
initial replication, once both sides agree that they're in sync then
drop this back to a sane value.
boe
2009-05-14 19:51:18 UTC
Permalink
Thanks - I did not know that. Do you have any links on how that is done?
Post by Kyle BLake
Another point is you can limit the bandwidth used for replication. So you
could just set it up and limit is and let it take its time.
Post by boe
Thanks!
Post by Dave Warren
Post by boe
I have an office overseas. I have about 200 GB of data over there
that
I
want on our server over here in the US. Using DFS and starting from
scratch it could take a very very very long time to replicate and clog
up
my
bandwidth at both offices. Could I send a hard drive of all the data from
the overseas office to the US and then create the DFS after the data is
copied on to our US server? Will it still want to replicate the data since
the creation/modified dates will be different?
Research "prestaging"
The short answer is that yes, you can. If you're using 2008 on both
sides, I'd suggest just copying the data and letting DFS-R worry about
permissions and attributes, DFS-R is *very* efficient at these sorts of
things, I did this with over a million files and around 3TB of data,
only a couple GB went across the wire during the sync-up process.
If you're using 2003 R2 on either or both sides, research how to
prestage if you're using robocopy then something like the following will
/e /sec /w:3 /r:2 /purge /xd DfsrPrivate
It's not perfect, you'll still see a reasonable chunk of data flow back
and forth during the sync-up process, but it's better then all 200GB.
IIRC the other big thing that you need to do in 2003 before using the
above is to turn off the "inherit from parent" option on both sides and
manually set the root's permissions, otherwise robocopy's writes aren't
quite right.
Regardless of how close your initial prestaging is, DFS-R does it's best
to only transfer the data required, as long as the file content matches,
DFS-R will update the meta data on it's own.
The only other thought that comes to mind is to make your staging
directories as large as possible, on 2008 I'd go as large as 64GB if you
have the free space, this increases efficiency significantly during the
initial replication, once both sides agree that they're in sync then
drop this back to a sane value.
Continue reading on narkive:
Loading...