[RFC] Start using guestfish for U-Boot fs tests

[Sean Anderson]

On 7/2/21 3:48 PM, Tom Rini wrote:
> On Fri, Jul 02, 2021 at 03:22:14PM -0400, Sean Anderson wrote:
>> 
>> 
>> On 7/2/21 3:01 PM, Tom Rini wrote:
>> > Hey all,
>> >
>> > I started taking a look at moving to guestfish to see if this resolves
>> > the latest problem I've run in to:
>> > https://source.denx.de/u-boot/u-boot/-/jobs/284763#L307
>> > which I think is due to guestmount not being done in time for the test.
>> > So I started converting things to use guestfish directly:
>> >
>> > diff --git a/test/py/tests/test_fs/conftest.py b/test/py/tests/test_fs/conftest.py
>> > index 7325486cdb1a..e8899cfdd118 100644
>> > --- a/test/py/tests/test_fs/conftest.py
>> > +++ b/test/py/tests/test_fs/conftest.py
>> > @@ -265,10 +265,10 @@ def fs_obj_basic(request, u_boot_config):
>> >       fs_ubtype = fstype_to_ubname(fs_type)
>> >       check_ubconfig(u_boot_config, fs_ubtype)
>> >
>> > -    mount_dir = u_boot_config.persistent_data_dir + '/mnt'
>> > +    data_dir = u_boot_config.persistent_data_dir + '/data'
>> >
>> > -    small_file = mount_dir + '/' + SMALL_FILE
>> > -    big_file = mount_dir + '/' + BIG_FILE
>> > +    small_file = data_dir + '/' + SMALL_FILE
>> > +    big_file = data_dir + '/' + BIG_FILE
>> >
>> >       try:
>> >
>> > @@ -279,26 +279,14 @@ def fs_obj_basic(request, u_boot_config):
>> >           return
>> >
>> >       try:
>> > -        check_call('mkdir -p %s' % mount_dir, shell=True)
>> > +        check_call('mkdir -p %s' % data_dir, shell=True)
>> >       except CalledProcessError as err:
>> >           pytest.skip('Preparing mount folder failed for filesystem: ' + fs_type + '. {}'.format(err))
>> >           call('rm -f %s' % fs_img, shell=True)
>> >           return
>> >
>> >       try:
>> > -        # Mount the image so we can populate it.
>> > -        mount_fs(fs_type, fs_img, mount_dir)
>> > -    except CalledProcessError as err:
>> > -        pytest.skip('Mounting to folder failed for filesystem: ' + fs_type + '. {}'.format(err))
>> > -        call('rmdir %s' % mount_dir, shell=True)
>> > -        call('rm -f %s' % fs_img, shell=True)
>> > -        return
>> > -
>> > -    try:
>> > -        # Create a subdirectory.
>> > -        check_call('mkdir %s/SUBDIR' % mount_dir, shell=True)
>> > -
>> > -        # Create big file in this image.
>> > +        # Create big file to copy in to the image.
>> >           # Note that we work only on the start 1MB, couple MBs in the 2GB range
>> >           # and the last 1 MB of the huge 2.5GB file.
>> >           # So, just put random values only in those areas.
>> > @@ -309,10 +297,14 @@ def fs_obj_basic(request, u_boot_config):
>> >           check_call('dd if=/dev/urandom of=%s bs=1M count=1 seek=2499'
>> >               % big_file, shell=True)
>> >
>> > -        # Create a small file in this image.
>> > +        # Create a small file to copy in to the image.
>> >           check_call('dd if=/dev/urandom of=%s bs=1M count=1'
>> >   	    % small_file, shell=True)
>> >
>> > +        # Copy the files in to the image and add a subdirectory.
>> > +        # Create a subdirectory.
>> > +        check_call('guestfish add %s : run : mount /dev/sda / : mkdir /SUBDIR : copy-in %s %s /'
>> > +            % (fs_img, big_file, small_file), shell=True)
>> >           # Delete the small file copies which possibly are written as part of a
>> >           # previous test.
>> >           # check_call('rm -f "%s.w"' % MB1, shell=True)
>> > @@ -357,13 +349,11 @@ def fs_obj_basic(request, u_boot_config):
>> >
>> >       except CalledProcessError as err:
>> >           pytest.skip('Setup failed for filesystem: ' + fs_type + '. {}'.format(err))
>> > -        umount_fs(mount_dir)
>> >           return
>> >       else:
>> > -        umount_fs(mount_dir)
>> >           yield [fs_ubtype, fs_img, md5val]
>> >       finally:
>> > -        call('rmdir %s' % mount_dir, shell=True)
>> > +        call('rmdir %s' % data_dir, shell=True)
>> >           call('rm -f %s' % fs_img, shell=True)
>> >
>> >   #
>> >
>> > The problem here is that a test run went from taking about 5 minutes to
>> > taking about 17 minutes.  I can reduce this to closer to 15 minutes with
>> > LIBGUESTFS_BACKEND=direct and using libguestfs-make-fixed-appliance to
>> > make an appliance we reuse.  But that's still too long to be usable.
>> > I'm hoping someone has some ideas here on how to improve things.
>> 
>> Have we tried using fuse? I've had good results with fuse2fs (although
>> it does not support the journal), and there is a fusefat package as
>> well. Example usage:
>> 
>> 	mkdir -p root
>> 	truncate -s 1G root.img
>> 	mkfs.ext4 -q $@
>> 	fuse2fs -o fakeroot root.img root
>> 	fakeroot tar -C root -xf rootfs.tar
>> 	fusermount -u root
>> 
>> If you don't need correct permissions, you can skip the fakeroot stuff.
> 
> I think fuse directly (guestfs stuff uses fuse under the hood) is
> probably out as we wouldn't be able to convert squashfs for example.

Why not squashfuse?

--Sean