October 18, 2015

Fun with e2fsck and debugfs

Hello there, 
We are going to corrupt the system and then repair, restore sanity. Things we are going to do:


  • fun1 → Corrupt the superblock.
  • fun2 → Multiply owned block.
  • fun3 → Corrupted inode.
  • fun4 → Orphaned inode.

Lets begin with perfectly working file system, which has say 10 files. 
mkfs.ext3 /dev/xvdb 


 Superblock backups stored on blocks:                                            
        32768, 98304, 163840, 229376, 294912, 819200, 884736   

Go ahead and mount it then create 10 files. 
mount /dev/xvdb /mnt 

for i in {1..10}; do echo "this is file$i" > /mnt/file$i.txt ; done 
Verify everything looks fine on mount point. Let the fun begin!

Fun1: Attack Super block

Our first task is to corrupt the super block. Run following command which will zero-out super block! 
dd if=/dev/zero of=/dev/xvdb count=1 bs=1024 seek=1 
above command,writes 0's on superblock location at offset 1024! Now lets try to mount it : 
umount /mnt && mount /dev/xvdb /mnt 

 mount: you must specify the filesystem type   

failed right? that's what we wanted :) Now how to fix this ? For cases like this ext3 has backup superblock (have look at above mkfs.ext3 output more closely!) All you need to do is tell mount command to use that copy of superblock. 
mount command expects block address in 1KB. so we need to convert our address like 98304 * 4 = 393216 we will try again with alternate superblock option "-o sb=" now: 
mount -o sb=393216 /dev/xvdb /mnt 
Worked! Just go ahead and explore everything is fine on mountpoint! 

Fun 2: Multiply owned blocks - No, Its MY block!

a Lets check out our existing file contents. 
cat /mnt/file1.txt /mnt/file2.txt 

this is file1                                                                   
this is file2

We are going to corrupt file system in such way that file2.txt will also point to file1.txt content!!!! 
To modify file system structure, we will a command called debugfs - an ext2/ext3/ext4 file system debugger 

First fetch the data block address of file1.txt (i.e where 'this is file1' is stored). 
debugfs -s 98304 -b 4096 /dev/xvdb -w 
debugfs: stat file1.txt 

 Inode: 12   Type: regular    Mode:  0644   Flags: 0x0                           
Generation: 1325309953    Version: 0x00000000                                   
User:     0   Group:     0   Size: 14                                           
File ACL: 0    Directory ACL: 0                                                 
Links: 1   Blockcount: 8                                                        
Fragment:  Address: 0    Number: 0    Size: 0                                   
ctime: 0x561ff500 -- Thu Oct 15 18:48:32 2015                                   
atime: 0x561ff7f1 -- Thu Oct 15 19:01:05 2015                                   
mtime: 0x561ff500 -- Thu Oct 15 18:48:32 2015  

Okay, debugfs stat output tells us the file1.txt data block address is 2048 

Now lets point file2.txt to this block with another debugfs command named mi which is powerful command. It gives up ability to modify inode contents! run 

debugfs: mi file2.txt 
and keep pressing the enter key

                       Mode    [0100644]                                     
                       User ID    [0]                                           
                      Group ID    [0]                                           
                          Size    [14]                                          
                 Creation time    [1444934912]                                  
             Modification time    [1444934912]                                  
                   Access time    [1444935703]                                  
                 Deletion time    [0]                                           
                    Link count    [1]                                           
              Block count high    [0]  

until you reach Direct Block #0 
Here type file1.txt block address here (2048)

             Direct Block #0    [2049] 2048                                   

Now just press enter until debugfs 

debugfs: quit 
and quit. To summarize, we figure-out the block address (2048) of file1.txt with debugfs-stat command, then replaced file2.txt block address 2049 with 2048. 
By following unmounting and remounting - we ensure our above changes written to disk.
umount /mnt 

mount -o sb=393216 /dev/xvdb /mnt -t ext3 
Now go head print file1.txt file2.txt 
cat /mnt/file1.txt /mnt/file2.txt 

this is file1                                                                   
this is file1

I hope you can spot the difference between our previous file1.txt and file2.txt output ans this one, right? Now lets run fsck to repair this. Since we can't run file system checker (fsck) on mounted partitions 
umount /mnt 
e2fsck /dev/xvdb 

 e2fsck 1.41.12 (17-May-2010)                                                    
e2fsck: Superblock invalid, trying backup blocks...                             
/dev/xvdb was not cleanly unmounted, check forced.                              
Pass 1: Checking inodes, blocks, and sizes                                      
Running additional passes to resolve blocks claimed by more than one inode...   
Pass 1B: Rescanning for multiply-claimed blocks                                 
Multiply-claimed block(s) in inode 12: 2048                                     
Multiply-claimed block(s) in inode 13: 2048                                     
Pass 1C: Scanning directories for inodes with multiply-claimed blocks           
Pass 1D: Reconciling multiply-claimed blocks                                    
(There are 2 inodes containing multiply-claimed blocks.)                        
File /file1.txt (inode #12, mod time Thu Oct 15 18:48:32 2015)                  
  has 1 multiply-claimed block(s), shared with 1 file(s):                       
        /file2.txt (inode #13, mod time Thu Oct 15 18:48:32 2015)               
Clone multiply-claimed blocks? yes 

Hey! see it figures our file1.txt and file2.txt has shared block 2038 which is wrong. It prompts how to deal with this case, we indeed cloned it.
Now it go on to update other book keeping entries: Just say yes to them. 

lock bitmap differences:  -2049                                                
Fix? yes                                                                     
Free blocks count wrong for group #0 (31863, counted=31853).                    
Fix? yes                                                                     
Free blocks count wrong (1512305, counted=1512295).                             
Fix? yes                                                                     
Free inodes count wrong for group #0 (8181, counted=8171).                      
Fix? yes                                                                     
Free inodes count wrong (393205, counted=393195).                               
Fix? yes           

Fun 3: Corrupt permission/type - can you fix it?

Lets use debugfs again, open the drive with debugfs -this time we are going corrupt file 'mode' with modify inode option. 
debugfs: mi file3.txt 

Mode    [0100644] 0
//keep pressing enter key

and then quit debugfs. 
debugfs: quit 
unmount and mount again if needed, and do: 
ls -l /mnt/file3.txt 

 ?---------. 1 root root 14 Oct 15 18:48 /mnt/file3.txt  

hihi :) can you see that, we corrupted file permission!!! 
I want you to fix this 'mode' with debugfs mi. If you are wondering what values to put, just look above..no,not the sky :p just scroll above for original mode values

Fun 4: Sad, Orphaned inodes

You might have seen this error message sometimes, lets recreate it. we will create 2 files on a new directory 'dir1' 
mkdir /mnt/dir1 

 echo "this is file-a" > /mnt/dir1/a
echo "this is file-b" > /mnt/dir1/b

with the help of debugfs,we will corrupt dir1 entries! 
debugfs clri dir1 

what this will do is unlink the chain between directory and its file. Now go back and list ls -l /mnt/dir1 won't show this file. You get error like

ls: cannot access /mnt/dir1: Input/output error

Remember file 'a' and 'b' are not deleted. Its laying on our file system without any directory pointing to it. 
If we run e2fcck now, it will find these orphaned inodes and put it under lost+found . So ahead and run it, you get message like 

 Pass 4: Checking reference counts                                               
Inode 2 ref count is 4, should be 3.  Fix? yes                               
 Unattached inode 32770          
Connect to /lost+found? yes                                                  
Inode 32770 ref count is 2, should be 1.  Fix? yes                           
Unattached inode 32771                                                          
Connect to /lost+found? yes  
Inode 32771 ref count is 2, should be 1.  Fix? yes  

e2fsck has done its job, lets go ahead and explore the end result. 
ls -l /mnt/lost+found/ 

 total 8                                                                         
-rw-r--r-- 1 root root 15 Oct 15 20:28 #32770                                   
-rw-r--r-- 1 root root 15 Oct 15 20:28 #32771         

Verify the content of files: 
cat \#32770 

 this is file-a

cat \#32771 

 this is file-b

Can you guess the how these filename are created? 


hmm..No? Have a close look at our e2fsck output. Yes, these the inode numbers of 'a' and 'b'. 

That's it, I hope you learned new corruption tricks, please feel free try these corruption on your production system^ 

^  Disclaimer : I'm not responsible, if you get fired!  


Click Here!