Using "dd" to mirror an entire drive
|
| Author | Content |
|---|---|
| Bob_Robertson Mar 08, 2010 9:59 AM EDT |
Ok, first I'm assuming that the drive is smaller than the one on my laptop, that the USB disk interface is working, etc. I know I can take a CD/DVD and write a .ISO image of it, like this... $ dd if=/dev/hdc of=/home/bob/DiskImage.iso ...and that the image can be written back out to CD/DVD using K3b and etc, as well as mounting it as a CD/DVD image in VirtualBox, because I've done that. It works fine. Now the question. Can I do the following and create a "ghost", a complete and BOOTABLE IDE disk, with (for example) some other OS than Linux? $ dd if=/dev/sdb of =/home/bob/LameOS_Disk.img then, after putting a disk of equal or larger size on the adapter, $ dd if=/home/bob/LameOS_Disk.img of=/dev/sdb I did pull down G4L, but the interface does not inspire confidence. Sorry, G4L developers. |
| jdixon Mar 08, 2010 11:06 AM EDT |
Bob: It's my understanding that your method should work to give you a complete backup, yes. However, as to it still being bootable when restored to another drive, that may depend on things like the drive geometry matching, et.al. As for G4L, take a look at PartImage or Clonezilla. I've used PartImage on occasion, and it seems to work fairly well. |
| Bob_Robertson Mar 08, 2010 11:21 AM EDT |
Thanks, JD. The bootable issue makes sense. I wonder how the "real" tools get around that? I did find "How to do everything with dd", the author makes repeated use of "conv=notrunc", and I can get the words "tells not to truncate" but that doesn't tell me what would happen if it DID "truncate" the output. I'll look up PartImage and Clonezilla right away. Thanks again. ...Ok, PartImage isn't going to work, NTFS support is "experimental". Clonezilla looks good, and saves to the local disk (which G4L didn't seem able to do). Ooookay. |
| techiem2 Mar 08, 2010 12:30 PM EDT |
To efficiently clone full disks without the pain of the dd process (it's looong), what we do is: 1. Backup the MBR: dd if=/dev/hda of=backup-hda.mbr count=1 bs=512 2. Backup the partition table: sfdisk -d /dev/hda > backup-hda.sf 3. Image the partitions with your tool of choice. partimage save /dev/hda1 my-boot-partition partimage save /dev/hda2 my-root-partition etc. We generally use partimage because it works well and is easy to use and automate. NTFS support has been called experimental in partimage for years. I've had it fail to make an image of ntfs maybe once, years ago (I think that machine needed a defrag or chkdisk really bad or something). So yes, it is technically Experimental, but it works reliably (if it didn't we wouldn't be using it on several hundred Windows machines that get reimaged every 3-6 months). The downside is that it's a special image format, so you can't loopback mount the image. Of course, most image formats are. Otherwise you use something like dd or ntfsclone to make a raw image file and pump it through gz to save space. But then you need to extract it anyway to loopback mount (unless someone knows a way to loopback mount an image insize a gz...I'd be interested it knowing). We tried Clonezilla, but it is a bit limited as to the location options to save to, and the CD had serious issues on some of our machines(like taking 10+ minutes to boot). Clonezilla basically does what I said above, just automated. It uses different imaging tools based on the partition type (partimage/ntfsclone/etc) for the actual partitions. To restore a full disk: 1. Restore the mbr: dd if=backup-hda.mbr of=/dev/hda 2. Restore the partition table: sfdisk /dev/hda < backup-hda.sf 3. Restore the partitions: partimage restore /dev/hda1 my-boot-partition.000 partimage restore /dev/hda2 my-root-partition.000 etc. partimage has lots of options, such as --batch to allow for better automation (--batch when used with restore just does it rather than making you go through the verify screens - there's a similar option for the save command, but I don't have it in head right now). I |
| techiem2 Mar 08, 2010 12:39 PM EDT |
Another note: partimage saves the image to a local file. So you can put your image on anything you can mount. We generally sshfs mount the server and use that for storing images. |
| Bob_Robertson Mar 08, 2010 1:52 PM EDT |
I must say that the compression idea sounds great. dd if=/dev/sda | zip OtherOS.image.zip - unzip -p OtherOS.image.zip | dd of=/dev/sdb conv=notrunc When investigating compression, I found out that bzip2 doesn't like stdin. More investigation needed. Found out what "notrunc" does. If the device/file being written to is bigger than the data being written, it does not truncate the resulting device/file to that size but leave the rest alone. Or so it seems. Considering that the used space on the disks I expect to be working on is a few gig at most, I don't want to get stuck with 80GB image files that are mostly zeros. Anyway, I grabbed Clonezilla and will see what shakes. Yesterday, I had a 160G IDE drive completely flake out on me as I was trying to use it externally on a USB disk adapter. I may have static fried it, mea maxima culpa. So I'm a big gun-shy right now about the whole thing. |
| gus3 Mar 08, 2010 2:21 PM EDT |
"bzip2 -c" reads from stdin, writes to stdout. That's the necessary standard for tar helpers. Add "-z" to compress, or "-d" to decompress (or just use "bzcat"). You might also reduce disk I/O kernel calls with "dd bs=1048576". At least, it can't hurt, and might help. |
| techiem2 Mar 08, 2010 2:30 PM EDT |
gzip supports stdin.
That's the recommended for using ntfsclone with compression.
partimage uses gzip by default as well. You can do bzip2 with either as well. It's generally not recommended because of the extra processing time. Though that may not be an issue with today's systems (I haven't done any comparisons). Clonezilla basically uses whichever tools (partimage/ntfsclone/dd/etc) it feels are best for the job and combines it with the mbr and partition table backups. |
| Bob_Robertson Mar 08, 2010 2:44 PM EDT |
Gus, I tried -c, and it errored. $ cat 773dd5.pdf | bzip2 -c test.bz2 bzip2: Can't open input file test.bz2: No such file or directory. Ok, Tech, I'll look at gzip. All this because a client said, "You need to learn Ghost." |
| gus3 Mar 08, 2010 3:39 PM EDT |
$ cat 773dd5.pdf | bzip2 -c > test.bz2 Should work. You need to use redirection. When you specify a filename directly to bzip2, it takes that as the file it should read. EDIT: Yes, it works that way. (note to self: try before posting) |
| techiem2 Mar 08, 2010 4:33 PM EDT |
Quoting: All this because a client said, "You need to learn Ghost." lol. We still have 1 lab that we have to reimage with ghost (we finally replaced the other one) because the machines don't have CDs and won't boot USB, so we have to use ghostcast with boot floppies for that lab. Everything else is partimage for now. Although we are hoping to find something that will be a little more efficient for full-lab imaging (having to boot every machine, mount the server, and start the image loading is a tad annoying at times - especially when you have several labs to do). We are thinking fogproject looks interesting. |
| Bob_Robertson Mar 08, 2010 7:12 PM EDT |
Thanks, Gus, I knew it _had_ to work, just not obvious. First thing I'm going to try is the boot image for Clonezilla, I figure it will either do what I want obviously, or not at all. Ugh. |
| hkwint Mar 08, 2010 7:35 PM EDT |
Bob: Be sure to zero out the unused bits (free space) before zipping! 'Unused' space may appear as random data, and the thing 'random' and 'state with the most entropy' share is that you cannot compress the data! Also, when space is an issue, you might want to look at xz-tools as it reaches slightly higher compression than bzip2 and may be the next standard for compression in Linux (for some distro's, it already is). $dd if=/dev/urandom of=~/test.raw count=50000 $ls -halF test.raw -rw-r----- 1 kwint wheel 25M Mar 9 00:24 test.raw $sudo mkfs.ext4 -F test.raw Now there's an ext4 filesystem on it. Let's try to 'zip' it: $xz -z -9 test.raw $ls -halF test.raw.xz -rw-r----- 1 kwint wheel 23M Mar 9 00:28 test.raw.xz You see, almost not zipped, due to 'high entropy' of the 'free space'. $unxz test.raw.xz $sudo mount -o loop test.raw /home/kwint/mnt/ $cd /home/kwint/mnt (Now we write to a new file on the filesystem we want to zip, the file is named 'empty', contains zeros, and we write to that file until dd raises an exception because the unused space is full with zeros:) $ sudo dd if=/dev/zero of=empty dd: writing to `empty': No space left on device 45265+0 records in 45264+0 records out 23175168 bytes (23 MB) copied, 0.199184 s, 116 MB/s $cd .. $sudo unmount /home/kwint/mnt $ xz -9 test.raw $ls -halF test.raw.xz -rw-r----- 1 kwint wheel 342K Mar 9 00:28 test.raw.xz See the difference? |
| Bob_Robertson Mar 08, 2010 8:22 PM EDT |
Very interesting idea, just writing to a file so that it naturally fills up unused space. I like it. Elegant, but unobvious. |
| gus3 Mar 08, 2010 9:36 PM EDT |
If you then use "shred -z" on that file, you'll get a security wipe on the unused space. |
| azerthoth Mar 08, 2010 11:28 PM EDT |
Careful Gus, on a journaled filesystem the success rate of shred drops off. There are warnings about the behavior in the man page. |
| gus3 Mar 08, 2010 11:59 PM EDT |
That may be true on smaller files, but once the shred's write exceeds the size of the journal, it's guaranteed that the log entries are forced out to the main filesystem. That'll be true for any filesystem with a journal; or, if it isn't true, it's a severe bug in the FS code. |
| msetzerii Apr 02, 2010 10:17 PM EDT |
Being the current maintainer of G4L, I would appreciate some additional information on:
"I did pull down G4L, but the interface does not inspire confidence. Sorry, G4L developers." Which version did you download? Version 0.32 is the latest released version, but I have already done 36 alpha versions with 0.33, and just compiled the 4 updated kernels from 4/1/2010 that kernel.org updated? The dialog menu is a simple way to provide the information for using dd to do most of the imaging, and it supports gzip, lzop, and bzip compression. It also supports local hard disks, usbs, ftp, cifs, sshfs, and recently nfs. The NTFSCLONE options make backuping up windows partitions faster than regular dd. It had udpcast for multicast imaging. It also has various options for clearing space on the drive before doing a dd image. NTFSCLONE images only backup used data. It also has options for backing up the MBR and EBR. If you can provide information on what the issue is, I could attempt to resolve it. I use the program all the time for my lab systems. Note. It can be loaded from the cd image, or installed directly if the grub or grub4dos menus. |
| Bob_Robertson Apr 03, 2010 4:11 PM EDT |
msetzerii, thank you. Part of my difficulties, in fact I will now say the majority, was in not having tried "Ghost" previously, so I had no idea what it was trying to accomplish in the first place. So as soon as networking was brought up, I started getting lost. This is entirely my fault, this must be how a manager feels when a technical issue comes up and people throw techie-buzz-words out and the manager simply has no context for what they mean. I knew I was coming into this cold, what I didn't realize was just HOW cold. Now that I have seen "Ghost" function and used it, even if only to a limited degree, the various options presented in G4L make much more sense. It's much more comprehensible for someone who already knows what they're doing. So let me get into a position to use it for real, before I go making suggestions of how to fix it. :^) |
You cannot post until you login.