[RFC] Start using guestfish for U-Boot fs tests
Alper Nebi Yasak
alpernebiyasak at gmail.com
Fri Jul 2 22:14:00 CEST 2021
On 02/07/2021 23:03, Alper Nebi Yasak wrote:
> On 02/07/2021 22:01, Tom Rini wrote:
>> Hey all,
>>
>> I started taking a look at moving to guestfish to see if this resolves
>> the latest problem I've run in to:
>> https://source.denx.de/u-boot/u-boot/-/jobs/284763#L307
>> which I think is due to guestmount not being done in time for the test.
>
> That failing test's setup uses virt-make-fs, different from what you're
> changing below. I locally only see that failure for the clang build, and
> it still fails after adding time.sleep(300) after its virt-make-fs
> calls. I don't think it's an issue in the test setup.
>
>> So I started converting things to use guestfish directly:
>> diff --git a/test/py/tests/test_fs/conftest.py b/test/py/tests/test_fs/conftest.py
>> index 7325486cdb1a..e8899cfdd118 100644
>> --- a/test/py/tests/test_fs/conftest.py
>> +++ b/test/py/tests/test_fs/conftest.py
>> @@ -265,10 +265,10 @@ def fs_obj_basic(request, u_boot_config):
>> fs_ubtype = fstype_to_ubname(fs_type)
>> check_ubconfig(u_boot_config, fs_ubtype)
>>
>> - mount_dir = u_boot_config.persistent_data_dir + '/mnt'
>> + data_dir = u_boot_config.persistent_data_dir + '/data'
>>
>> - small_file = mount_dir + '/' + SMALL_FILE
>> - big_file = mount_dir + '/' + BIG_FILE
>> + small_file = data_dir + '/' + SMALL_FILE
>> + big_file = data_dir + '/' + BIG_FILE
>>
>> try:
>>
>> @@ -279,26 +279,14 @@ def fs_obj_basic(request, u_boot_config):
>> return
>>
>> try:
>> - check_call('mkdir -p %s' % mount_dir, shell=True)
>> + check_call('mkdir -p %s' % data_dir, shell=True)
>> except CalledProcessError as err:
>> pytest.skip('Preparing mount folder failed for filesystem: ' + fs_type + '. {}'.format(err))
>> call('rm -f %s' % fs_img, shell=True)
>> return
>>
>> try:
>> - # Mount the image so we can populate it.
>> - mount_fs(fs_type, fs_img, mount_dir)
>> - except CalledProcessError as err:
>> - pytest.skip('Mounting to folder failed for filesystem: ' + fs_type + '. {}'.format(err))
>> - call('rmdir %s' % mount_dir, shell=True)
>> - call('rm -f %s' % fs_img, shell=True)
>> - return
>> -
>> - try:
>> - # Create a subdirectory.
>> - check_call('mkdir %s/SUBDIR' % mount_dir, shell=True)
>> -
>> - # Create big file in this image.
>> + # Create big file to copy in to the image.
>> # Note that we work only on the start 1MB, couple MBs in the 2GB range
>> # and the last 1 MB of the huge 2.5GB file.
>> # So, just put random values only in those areas.
>> @@ -309,10 +297,14 @@ def fs_obj_basic(request, u_boot_config):
>> check_call('dd if=/dev/urandom of=%s bs=1M count=1 seek=2499'
>> % big_file, shell=True)
>>
>> - # Create a small file in this image.
>> + # Create a small file to copy in to the image.
>> check_call('dd if=/dev/urandom of=%s bs=1M count=1'
>> % small_file, shell=True)
>>
>> + # Copy the files in to the image and add a subdirectory.
>> + # Create a subdirectory.
>> + check_call('guestfish add %s : run : mount /dev/sda / : mkdir /SUBDIR : copy-in %s %s /'
>> + % (fs_img, big_file, small_file), shell=True)
>
> It could be faster to do things within guestfish as much as possible,
> instead of preparing the files outside and copying them in.
>
> Also it looks like python bindings are available as python3-guestfs on
> Debian and Ubuntu, just not on pypi.org.
>
>> # Delete the small file copies which possibly are written as part of a
>> # previous test.
>> # check_call('rm -f "%s.w"' % MB1, shell=True)
>> @@ -357,13 +349,11 @@ def fs_obj_basic(request, u_boot_config):
>>
>> except CalledProcessError as err:
>> pytest.skip('Setup failed for filesystem: ' + fs_type + '. {}'.format(err))
>> - umount_fs(mount_dir)
>> return
>> else:
>> - umount_fs(mount_dir)
>> yield [fs_ubtype, fs_img, md5val]
>> finally:
>> - call('rmdir %s' % mount_dir, shell=True)
>> + call('rmdir %s' % data_dir, shell=True)
>> call('rm -f %s' % fs_img, shell=True)
>>
>> #
>>
>> The problem here is that a test run went from taking about 5 minutes to
>> taking about 17 minutes. I can reduce this to closer to 15 minutes with
>> LIBGUESTFS_BACKEND=direct and using libguestfs-make-fixed-appliance to
>> make an appliance we reuse. But that's still too long to be usable.
>> I'm hoping someone has some ideas here on how to improve things.
>
> If libguestfs is falling back to slow emulation because /dev/kvm isn't
> available, maybe it's appropriate to check for that and skip the fs tests...
Also the User-Mode Linux backend might be worth exploring.
More information about the U-Boot
mailing list