cannot lock /dev/ttyusb_port1: Resource temporarily unavailable

[Simon Glass]

Hi Harald,

On Sun, 8 Oct 2023 at 17:07, Rahix <rahix at rahix.de> wrote:
>
> Hi Simon,
>
> On Sun, 2023-10-08 at 14:16 -0600, Simon Glass wrote:
> > Hi,
> >
> > Since an update to Ubuntu 2022.04 on my lab machine I cannot get
> > boards to connect:
> >
> > Revision 7d1e5aacdb44bdc2139197a60d8c97c9062d5064, board rpi3
> >
> > Checking revision 7d1e5aacdb44bdc2139197a60d8c97c9062d5064
> > /vid/software/devel/ubtest
> > tbot starting ...
> > ├─Parameters:
> > │     rev        = '7d1e5aacdb44bdc2139197a60d8c97c9062d5064'
> > │     clean      = False
> > ├─Calling uboot_build_and_flash ...
> > │   ├─Calling uboot_build ...
> > │   │   ├─Calling uboot_checkout ...
> > │   │   │   ├─Builder: rpi3
> > │   │   │   └─Done. (6.016s)
> > │   │   ├─Configuring build ...
> > │   │   ├─Calling uboot_make ...
> > │   │   │   └─Done. (10.188s)
> > │   │   └─Done. (18.444s)
> > │   ├─Calling uboot_flash ...
> > │   │   ├─Calling copy ...
> > │   │   │   └─Done. (0.004s)
> > │   │   └─Done. (0.646s)
> > │   └─Done. (19.390s)
> > ├─────────────────────────────────────────
> > └─SUCCESS (19.486s)
> > tbot starting ...
> > ├─Calling interactive_board ...
> > │   ├─POWERON (rpi3)
> > │   ├─Entering interactive shell (CTRL+D to exit) ...
> >
> > FATAL: cannot lock /dev/ttyusb_port1: Resource temporarily unavailable
>
> This is a telltale sign that there is still a session with the console
> open somewhere. This can either happen
>
>   - on the remote side (e.g. a picocom process lingering after the ssh
>     session closed)
>
>   - or locally (an ssh process not getting SIGTERMed correctly).
>
> The latter shouldn't be possible with the paramiko connector unless you
> start tbot multiple times within a single python interpreter and somehow
> manage to leak channels.

Thanks for the info.

I don't see either of those...e.g. I can see the picocom going away on
the remote machine (with the lab).

>
> > (after it hangs forever here, I press ctrl-D)
> >
> > logout
> > │   ├─POWEROFF (rpi3)
> > │   ├─Exiting poweroff
> > │   └─Fail. (77.542s)
> > ├─Warning: Found dangling <class 'rpi3.Rpi3'> instance in this context
> > ├─Warning: Teardown went wrong!  A <class 'rpi3.Rpi3'> instance is still alive.
> > │ Please report this to https://github.com/rahix/tbot/issues!
>
> This here is looking very interesting to me.  Even when everything is
> broken, you should not be able to produce this error... Might be
> related.
>
> In any case, I think it would be easiest to diagnose this with a bit
> more info about your setup.  Can you share
>
>   - how are you invoking tbot?

tbot -l ${lab} -b ${board}.py -T tbot/contrib -p rev=\"${rev}\" \
-p clean=${clean} $patch $V uboot_build_and_flash
tbot -l ${lab} -b ${board}.py interactive_board

my tree is here:
https://github.com/sjg20/tbot/pull/new/lab8

>   - what terminal emulator are you using to open the console sessions?

In this case I am just running tbot and relying on it to attach the console

>   - if possible, the board & lab config which produces this error

I see it on all boards - this one happens to be a rpi4, but I can
repeat this on any of them.

The lab machine is remote (in the next room).

Regards,
Simon

>
> Best regards,
>
> Harald
> --
>
> > ├─Exception:
> > │   Traceback (most recent call last):
> > │     File "/home/sglass/.local/lib/python3.10/site-packages/tbot/main.py",
> > line 345, in main
> > │       func(**params)
> > │     File "/home/sglass/.local/lib/python3.10/site-packages/tbot/decorators.py",
> > line 62, in wrapped
> > │       return tc(*args, **kwargs)
> > │     File "/home/sglass/.local/lib/python3.10/site-packages/tbot/tc/callable.py",
> > line 43, in interactive_board
> > │       with tbot.ctx.request(tbot.role.Board) as b:
> > │     File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__
> > │       self.gen.throw(typ, value, traceback)
> > │     File "/home/sglass/.local/lib/python3.10/site-packages/tbot/context.py",
> > line 359, in request
> > │       with instance.request(exclusive, self._keep_alive) as m:
> > │     File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__
> > │       self.gen.throw(typ, value, traceback)
> > │     File "/home/sglass/.local/lib/python3.10/site-packages/tbot/context.py",
> > line 94, in request
> > │       self.teardown()
> > │     File "/home/sglass/.local/lib/python3.10/site-packages/tbot/context.py",
> > line 65, in teardown
> > │       self._cx.close()
> > │     File "/usr/lib/python3.10/contextlib.py", line 584, in close
> > │       self.__exit__(None, None, None)
> > │     File "/usr/lib/python3.10/contextlib.py", line 576, in __exit__
> > │       raise exc_details[1]
> > │     File "/usr/lib/python3.10/contextlib.py", line 561, in __exit__
> > │       if cb(*exc_details):
> > │     File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__
> > │       next(self.gen)
> > │     File "/home/sglass/.local/lib/python3.10/site-packages/tbot/machine/connector/common.py",
> > line 126, in from_context
> > │       with contextlib.ExitStack() as cx:
> > │     File "/usr/lib/python3.10/contextlib.py", line 576, in __exit__
> > │       raise exc_details[1]
> > │     File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__
> > │       self.gen.throw(typ, value, traceback)
> > │     File "/home/sglass/.local/lib/python3.10/site-packages/tbot/context.py",
> > line 379, in request
> > │       raise e from None
> > │     File "/home/sglass/.local/lib/python3.10/site-packages/tbot/context.py",
> > line 366, in request
> > │       yield m
> > │     File "/usr/lib/python3.10/contextlib.py", line 561, in __exit__
> > │       if cb(*exc_details):
> > │     File "/home/sglass/.local/lib/python3.10/site-packages/tbot/machine/machine.py",
> > line 186, in __exit__
> > │       self._cx.__exit__(*args)
> > │     File "/usr/lib/python3.10/contextlib.py", line 576, in __exit__
> > │       raise exc_details[1]
> > │     File "/usr/lib/python3.10/contextlib.py", line 561, in __exit__
> > │       if cb(*exc_details):
> > │     File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__
> > │       next(self.gen)
> > │     File "/home/sglass/.local/lib/python3.10/site-packages/tbot/machine/connector/common.py",
> > line 133, in _connect
> > │       with self.host.clone() as cloned, self.connect(cloned) as ch:
> > │     File "/home/sglass/.local/lib/python3.10/site-packages/tbot/machine/channel/channel.py",
> > line 562, in __exit__
> > │       self.close()
> > │     File "/home/sglass/.local/lib/python3.10/site-packages/tbot/machine/channel/channel.py",
> > line 540, in close
> > │       self._c.close()
> > │     File "/home/sglass/.local/lib/python3.10/site-packages/tbot/machine/channel/paramiko.py",
> > line 58, in close
> > │       raise channel.ChannelClosedException()
> > │   tbot.machine.channel.channel.ChannelClosedException
> > ├─────────────────────────────────────────
> > └─FAILURE (77.579s)
> >
> >
> > If I then connect again, it works, mostly. Has something changed in
> > paramiko? Any clues?
> >
> > Regards,
> > Simon