cannot lock /dev/ttyusb_port1: Resource temporarily unavailable

[Rahix]

Hi Simon,

On Sun, 2023-10-08 at 14:16 -0600, Simon Glass wrote:
> Hi,
> 
> Since an update to Ubuntu 2022.04 on my lab machine I cannot get
> boards to connect:
> 
> Revision 7d1e5aacdb44bdc2139197a60d8c97c9062d5064, board rpi3
> 
> Checking revision 7d1e5aacdb44bdc2139197a60d8c97c9062d5064
> /vid/software/devel/ubtest
> tbot starting ...
> ├─Parameters:
> │     rev        = '7d1e5aacdb44bdc2139197a60d8c97c9062d5064'
> │     clean      = False
> ├─Calling uboot_build_and_flash ...
> │   ├─Calling uboot_build ...
> │   │   ├─Calling uboot_checkout ...
> │   │   │   ├─Builder: rpi3
> │   │   │   └─Done. (6.016s)
> │   │   ├─Configuring build ...
> │   │   ├─Calling uboot_make ...
> │   │   │   └─Done. (10.188s)
> │   │   └─Done. (18.444s)
> │   ├─Calling uboot_flash ...
> │   │   ├─Calling copy ...
> │   │   │   └─Done. (0.004s)
> │   │   └─Done. (0.646s)
> │   └─Done. (19.390s)
> ├─────────────────────────────────────────
> └─SUCCESS (19.486s)
> tbot starting ...
> ├─Calling interactive_board ...
> │   ├─POWERON (rpi3)
> │   ├─Entering interactive shell (CTRL+D to exit) ...
> 
> FATAL: cannot lock /dev/ttyusb_port1: Resource temporarily unavailable

This is a telltale sign that there is still a session with the console
open somewhere. This can either happen

  - on the remote side (e.g. a picocom process lingering after the ssh
    session closed)

  - or locally (an ssh process not getting SIGTERMed correctly).

The latter shouldn't be possible with the paramiko connector unless you
start tbot multiple times within a single python interpreter and somehow
manage to leak channels.

> (after it hangs forever here, I press ctrl-D)
> 
> logout
> │   ├─POWEROFF (rpi3)
> │   ├─Exiting poweroff
> │   └─Fail. (77.542s)
> ├─Warning: Found dangling <class 'rpi3.Rpi3'> instance in this context
> ├─Warning: Teardown went wrong!  A <class 'rpi3.Rpi3'> instance is still alive.
> │ Please report this to https://github.com/rahix/tbot/issues!

This here is looking very interesting to me.  Even when everything is
broken, you should not be able to produce this error... Might be
related.

In any case, I think it would be easiest to diagnose this with a bit
more info about your setup.  Can you share

  - how are you invoking tbot?
  - what terminal emulator are you using to open the console sessions?
  - if possible, the board & lab config which produces this error

Best regards,

Harald
-- 

> ├─Exception:
> │   Traceback (most recent call last):
> │     File "/home/sglass/.local/lib/python3.10/site-packages/tbot/main.py",
> line 345, in main
> │       func(**params)
> │     File "/home/sglass/.local/lib/python3.10/site-packages/tbot/decorators.py",
> line 62, in wrapped
> │       return tc(*args, **kwargs)
> │     File "/home/sglass/.local/lib/python3.10/site-packages/tbot/tc/callable.py",
> line 43, in interactive_board
> │       with tbot.ctx.request(tbot.role.Board) as b:
> │     File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__
> │       self.gen.throw(typ, value, traceback)
> │     File "/home/sglass/.local/lib/python3.10/site-packages/tbot/context.py",
> line 359, in request
> │       with instance.request(exclusive, self._keep_alive) as m:
> │     File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__
> │       self.gen.throw(typ, value, traceback)
> │     File "/home/sglass/.local/lib/python3.10/site-packages/tbot/context.py",
> line 94, in request
> │       self.teardown()
> │     File "/home/sglass/.local/lib/python3.10/site-packages/tbot/context.py",
> line 65, in teardown
> │       self._cx.close()
> │     File "/usr/lib/python3.10/contextlib.py", line 584, in close
> │       self.__exit__(None, None, None)
> │     File "/usr/lib/python3.10/contextlib.py", line 576, in __exit__
> │       raise exc_details[1]
> │     File "/usr/lib/python3.10/contextlib.py", line 561, in __exit__
> │       if cb(*exc_details):
> │     File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__
> │       next(self.gen)
> │     File "/home/sglass/.local/lib/python3.10/site-packages/tbot/machine/connector/common.py",
> line 126, in from_context
> │       with contextlib.ExitStack() as cx:
> │     File "/usr/lib/python3.10/contextlib.py", line 576, in __exit__
> │       raise exc_details[1]
> │     File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__
> │       self.gen.throw(typ, value, traceback)
> │     File "/home/sglass/.local/lib/python3.10/site-packages/tbot/context.py",
> line 379, in request
> │       raise e from None
> │     File "/home/sglass/.local/lib/python3.10/site-packages/tbot/context.py",
> line 366, in request
> │       yield m
> │     File "/usr/lib/python3.10/contextlib.py", line 561, in __exit__
> │       if cb(*exc_details):
> │     File "/home/sglass/.local/lib/python3.10/site-packages/tbot/machine/machine.py",
> line 186, in __exit__
> │       self._cx.__exit__(*args)
> │     File "/usr/lib/python3.10/contextlib.py", line 576, in __exit__
> │       raise exc_details[1]
> │     File "/usr/lib/python3.10/contextlib.py", line 561, in __exit__
> │       if cb(*exc_details):
> │     File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__
> │       next(self.gen)
> │     File "/home/sglass/.local/lib/python3.10/site-packages/tbot/machine/connector/common.py",
> line 133, in _connect
> │       with self.host.clone() as cloned, self.connect(cloned) as ch:
> │     File "/home/sglass/.local/lib/python3.10/site-packages/tbot/machine/channel/channel.py",
> line 562, in __exit__
> │       self.close()
> │     File "/home/sglass/.local/lib/python3.10/site-packages/tbot/machine/channel/channel.py",
> line 540, in close
> │       self._c.close()
> │     File "/home/sglass/.local/lib/python3.10/site-packages/tbot/machine/channel/paramiko.py",
> line 58, in close
> │       raise channel.ChannelClosedException()
> │   tbot.machine.channel.channel.ChannelClosedException
> ├─────────────────────────────────────────
> └─FAILURE (77.579s)
> 
> 
> If I then connect again, it works, mostly. Has something changed in
> paramiko? Any clues?
> 
> Regards,
> Simon