Good day, Cloud Gurus,
and welcome back for this lesson
where we're going to take a look
at optimizing your file server capacity.
Let's take a look at what we're going to cover.
First up, we'll enter the question
what is Data Deduplication?
Then we'll take a look at how Data Deduplication works.
We'll take a look at some of the important considerations,
and we'll wrap up this lesson
with a bit of a demo where we will implement
and manage Data Deduplication.
Let's get started.
So what is Data Deduplication?
Well, Data Deduplication
reduces storage capacity requirements
by scanning and storing duplicate data once
on a volume and allows you to optionally compress the data
to save even more space.
And Data Deduplication is great
for basically any Windows file server.
There are some limitations, though,
and we'll take a look at those later in the lesson,
but you can save disk space with general file shares,
software and operating system shares,
virtualization images, and archives.
And you can save anywhere between 30
and 95% of the capacity
when you're using Data Dedup.
And Data Deduplication can be used
with a range of other Windows file services
to combine resiliency,
replication or management capabilities.
Some of these include File Server Resource Manager,
which we covered earlier in this section,
Storage Replica, and Distributed File System.
Now let's take a look at how Data Deduplication works.
The Data Deduplication process starts
with a scan of files
to determine which files can be optimized
based on the optimization policy.
And the files are broken into variable chunks
and unique and duplicated chunks are identified.
And chunks are just portions of data
that make up a file,
and are likely to be repeated between similar files.
The chunks are then moved to a chunk store,
retaining unique chunks and removing duplicate chunks.
And then the original file stream is updated
to point to the chunk store
for its unique and duplicated content.
And if you've never heard the term file stream before,
it's basically what Windows uses to read
and write files from the file system.
What's important to know about this
is that reads are read from the deduplicated chunk store
but writes are made unoptimized.
This improves performance by offloading deduplication
to a later time.
So it won't slow things down
when you're waiting for data to be written to the disk.
And there are a few background jobs that make this possible.
The first is the file optimization job
that runs the process shown here.
And it runs every hour by default.
And there are 2 other jobs that run weekly by default.
The first is garbage collection,
which removes chunks that are no longer being referenced.
And the other is an integrity scrubbing job
that corrects corruption in the chunk store.
Now let's take a look at some of the considerations
for data dedup
because there are some fairly decent gotchas
to be aware of.
Firstly, you should be aware that data dedup
can be used with ReFS-formatted volumes
but you need to be running at least Windows Server 2019.
You also should be aware that Data Deduplication works great
with Distributed File System
and can even be used with Storage Replica.
But with Storage Replica,
you need to enable the dedup feature
on both the primary and secondary server.
This is because Storage Replica replicates block of data
but DFS replicates files.
And how the blocks are stored does not impact DFS.
There is one caveat here to be aware of though.
Storage space is direct
with a stretched Storage Replica cluster
is not recommended by Microsoft.
There are also a few products
and server roles that you shouldn't use with dedup.
These include Windows Server Update Services
and databases like SQL and Exchange Server.
Windows Search also doesn't work properly
with deduped volumes.
So just be aware that it impact your search results.
You also should be careful using
the very popular ROBOCOPY command-line tool
with deduplicated volumes
because it can corrupt the chunk store.
And ROBOCOPY is commonly used to copy or migrate data
on Windows Server.
So this is a pretty decent gotcha to be aware of.
All right, now let's take a look
at a demo of Data Deduplication.
In this demo, we will enable the deduplication feature
on our File Server CORP-BNE-FS01
and configure it for our volume.
And then we'll copy some data to the volume
and manage deduplication using the built-in optimization
and garbage collection tasks.
And then just for fun, we'll fail our file server
and attempt to recover the deduplicated data volume.
Let's take a look.
All right, so here I am logged on to CORP-BNE-FS01.
And we can see we have a data directory here
and it's got some operating system disk images
and some software.
And this is a great candidate for Data Deduplication.
Let's go ahead and get started
by installing the data dedup feature,
and we'll do that using PowerShell.
So I'll just go ahead and launch PowerShell
and then we'll use install-WindowsFeature
and we'll install the FS-Data-Deduplication feature.
We'll go ahead and run that.
This will just take a moment to install,
so we'll come back when that's done.
All right, the installation of that feature is now complete.
Now let's use Server Manager
to enable data dedup on the M drive volume.
So we'll navigate to Server Manager.
We'll jump into File and Storage Services.
We'll jump into Volumes.
And we'll right-click M drive
and select Configure Data Deduplication.
We'll go ahead and enable it.
And we'll use a general-purpose file server,
which is ideal for our purposes.
And because we're doing this in a lab environment,
we're doing to dedup files older than zero days
instead of the default 3 days.
And we're okay with the remainder of the defaults,
so we'll go ahead and click OK.
Now let's take a look
at managing deduplication tasks using PowerShell.
So I'll minimize Server Manager.
Here we are back in PowerShell.
Let's start our Data Deduplication job on the M drive volume
using the Start-DedupJob PowerShell cmdlet.
We'll specify the Volume as M.
We'll specify the type to be the Optimization task.
We'll specify to use up to 75% of the server's memory
and we'll specify that we want to wait
for the task to complete.
Now, Microsoft recommends you use a maximum
of 50% of the memory to run this task
but because we want to do it fast,
and we're in a lab, we'll use 75%,
using the Start-DedupJob PowerShell cmdlet,
this will just take a moment to run.
So we'll come back when this is done.
All right, that DedupJob is now finished.
Now let's take a look at the status
of the deduplication and we'll do that
using the get-DedupStatus PowerShell cmdlet
and we'll specify the volume as M.
We'll go ahead and run that.
And we can see we've saved about 2.1 gigabytes
on just those small files that we had on that server.
Now we're going to use
the Get-DedupMetadata PowerShell cmdlet
to have a look at the chunk store size.
So we'll run Get-DedupMetadata
and specify the volume as M.
And we can see that our ChunkStoreSize
is currently just under 9 gigabytes.
Now let's go ahead and delete some
of the large files from M drive
using the Remove-Item PowerShell cmdlet,
and we'll go ahead and remove all of the isos
by specifying the wildcard
and we'll specify that we want to force delete them.
Now, what do you think will happen
to the ChunkStoreSize
when we run the Get-DedupMetadata PowerShell cmdlet again?
Do you think it will have changed?
Let's take a look.
Now, as we can see, even though we've removed a couple
of large files from the M drive,
the TotalChunkStoreSize has not changed.
Now let's run the Start-Dedup PowerShell cmdlet again.
This time we'll specify
that we want to run a garbage collection job
to clean up the chunk store.
So I'll run Start-DedupJob, specify the volume as M.
We'll specify the type of task as GarabageCollection,
set the memory to 75% so we don't have to wait as long,
and specify that we want to wait for completion.
We'll go ahead and run that.
And this will just take a moment to complete.
So we'll come back when this is done.
All right, the GarbageCollection task has now finished.
Now let's run the Get-DedupMetadata PowerShell cmdlet again
and we'll have a look and see
if the chunk size has decreased.
So we'll just press up to get to that command.
We'll run it again
and we can see that the TotalChunkStoreSize
has decreased down to about 80 megabytes
so that you can see that it would be a good idea
to run the GarbageCollection job
when you've deleted a large amount of files
from a server that is using data dedup.
All right, now let's simulate the failure
of this server, and then we'll attach the disk
to FS02 and then we'll try and access the data.
So I'll come back once I've shut down this server
and moved the disk over to File Server 2
and then we'll try and access the data.
Be back in just a moment.
All right, so here we are on CORP-BNE-FS02,
and I've attached that data disk to this server.
Let's go ahead and have a look at the data.
So I'll launch Windows Explorer.
I can see we have access to the data, as we would expect.
Let's go ahead and run the installer.
So double click that.
Now, this is unexpected.
Everything looked good right up until we went
to run the file.
What do you think we could do to fix this?
Well, if you guessed install the data dedup feature,
you would be right.
So let's go ahead and install that using PowerShell.
I'll just click OK on this error message
and then we'll swap to PowerShell.
We'll install the FS-Data-Deduplication feature.
This will just take a moment to install.
So I'll come back when this is done.
All right, the Data-Deduplication feature is now installed.
Now let's go ahead and try and run that installer again.
So I'll switch back to Windows Explorer.
We'll go ahead and try and run the installer,
and we can see we're all good.
All right, that concludes this demo.
Now let's wrap up with a bit of a summary
of what we covered in this lesson.
So what have we covered?
Data Dedup can improve storage utilization
on your Windows File Server volumes
by anywhere between 30 and 90%,
depending on the type of data stored.
And deduplication is performed
by an optimization task that runs every hour by default.
And the garbage collection task
to reclaim space runs every week by default
but you can run it manually
if you delete a large number of files
from a deduped volume.
And Data Deduplication volumes can only be read from servers
that have the Data Deduplication feature enabled.
It's also important to remember
that you should not enable dedup on servers
that run incompatible products, like Exchange Server
or SQL database servers.
All right, thanks for being with me in this lesson.
I hope you found it useful
and I look forward to seeing you in the next lesson.