From sjg at chromium.org  Sun Oct  8 22:16:53 2023
From: sjg at chromium.org (Simon Glass)
Date: Sun, 8 Oct 2023 14:16:53 -0600
Subject: cannot lock /dev/ttyusb_port1: Resource temporarily unavailable
Message-ID: <CAPnjgZ0L8U-hxa+bgkB4kT3L1TNmDz8QYB9QhKWcwkQcnfdjEA@mail.gmail.com>

Hi,

Since an update to Ubuntu 2022.04 on my lab machine I cannot get
boards to connect:

Revision 7d1e5aacdb44bdc2139197a60d8c97c9062d5064, board rpi3

Checking revision 7d1e5aacdb44bdc2139197a60d8c97c9062d5064
/vid/software/devel/ubtest
tbot starting ...
??Parameters:
?     rev        = '7d1e5aacdb44bdc2139197a60d8c97c9062d5064'
?     clean      = False
??Calling uboot_build_and_flash ...
?   ??Calling uboot_build ...
?   ?   ??Calling uboot_checkout ...
?   ?   ?   ??Builder: rpi3
?   ?   ?   ??Done. (6.016s)
?   ?   ??Configuring build ...
?   ?   ??Calling uboot_make ...
?   ?   ?   ??Done. (10.188s)
?   ?   ??Done. (18.444s)
?   ??Calling uboot_flash ...
?   ?   ??Calling copy ...
?   ?   ?   ??Done. (0.004s)
?   ?   ??Done. (0.646s)
?   ??Done. (19.390s)
??????????????????????????????????????????
??SUCCESS (19.486s)
tbot starting ...
??Calling interactive_board ...
?   ??POWERON (rpi3)
?   ??Entering interactive shell (CTRL+D to exit) ...

FATAL: cannot lock /dev/ttyusb_port1: Resource temporarily unavailable

(after it hangs forever here, I press ctrl-D)

logout
?   ??POWEROFF (rpi3)
?   ??Exiting poweroff
?   ??Fail. (77.542s)
??Warning: Found dangling <class 'rpi3.Rpi3'> instance in this context
??Warning: Teardown went wrong!  A <class 'rpi3.Rpi3'> instance is still alive.
? Please report this to https://github.com/rahix/tbot/issues!
??Exception:
?   Traceback (most recent call last):
?     File "/home/sglass/.local/lib/python3.10/site-packages/tbot/main.py",
line 345, in main
?       func(**params)
?     File "/home/sglass/.local/lib/python3.10/site-packages/tbot/decorators.py",
line 62, in wrapped
?       return tc(*args, **kwargs)
?     File "/home/sglass/.local/lib/python3.10/site-packages/tbot/tc/callable.py",
line 43, in interactive_board
?       with tbot.ctx.request(tbot.role.Board) as b:
?     File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__
?       self.gen.throw(typ, value, traceback)
?     File "/home/sglass/.local/lib/python3.10/site-packages/tbot/context.py",
line 359, in request
?       with instance.request(exclusive, self._keep_alive) as m:
?     File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__
?       self.gen.throw(typ, value, traceback)
?     File "/home/sglass/.local/lib/python3.10/site-packages/tbot/context.py",
line 94, in request
?       self.teardown()
?     File "/home/sglass/.local/lib/python3.10/site-packages/tbot/context.py",
line 65, in teardown
?       self._cx.close()
?     File "/usr/lib/python3.10/contextlib.py", line 584, in close
?       self.__exit__(None, None, None)
?     File "/usr/lib/python3.10/contextlib.py", line 576, in __exit__
?       raise exc_details[1]
?     File "/usr/lib/python3.10/contextlib.py", line 561, in __exit__
?       if cb(*exc_details):
?     File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__
?       next(self.gen)
?     File "/home/sglass/.local/lib/python3.10/site-packages/tbot/machine/connector/common.py",
line 126, in from_context
?       with contextlib.ExitStack() as cx:
?     File "/usr/lib/python3.10/contextlib.py", line 576, in __exit__
?       raise exc_details[1]
?     File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__
?       self.gen.throw(typ, value, traceback)
?     File "/home/sglass/.local/lib/python3.10/site-packages/tbot/context.py",
line 379, in request
?       raise e from None
?     File "/home/sglass/.local/lib/python3.10/site-packages/tbot/context.py",
line 366, in request
?       yield m
?     File "/usr/lib/python3.10/contextlib.py", line 561, in __exit__
?       if cb(*exc_details):
?     File "/home/sglass/.local/lib/python3.10/site-packages/tbot/machine/machine.py",
line 186, in __exit__
?       self._cx.__exit__(*args)
?     File "/usr/lib/python3.10/contextlib.py", line 576, in __exit__
?       raise exc_details[1]
?     File "/usr/lib/python3.10/contextlib.py", line 561, in __exit__
?       if cb(*exc_details):
?     File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__
?       next(self.gen)
?     File "/home/sglass/.local/lib/python3.10/site-packages/tbot/machine/connector/common.py",
line 133, in _connect
?       with self.host.clone() as cloned, self.connect(cloned) as ch:
?     File "/home/sglass/.local/lib/python3.10/site-packages/tbot/machine/channel/channel.py",
line 562, in __exit__
?       self.close()
?     File "/home/sglass/.local/lib/python3.10/site-packages/tbot/machine/channel/channel.py",
line 540, in close
?       self._c.close()
?     File "/home/sglass/.local/lib/python3.10/site-packages/tbot/machine/channel/paramiko.py",
line 58, in close
?       raise channel.ChannelClosedException()
?   tbot.machine.channel.channel.ChannelClosedException
??????????????????????????????????????????
??FAILURE (77.579s)


If I then connect again, it works, mostly. Has something changed in
paramiko? Any clues?

Regards,
Simon

From rahix at rahix.de  Mon Oct  9 01:07:14 2023
From: rahix at rahix.de (Rahix)
Date: Mon, 09 Oct 2023 01:07:14 +0200
Subject: cannot lock /dev/ttyusb_port1: Resource temporarily unavailable
In-Reply-To: <CAPnjgZ0L8U-hxa+bgkB4kT3L1TNmDz8QYB9QhKWcwkQcnfdjEA@mail.gmail.com>
References: <CAPnjgZ0L8U-hxa+bgkB4kT3L1TNmDz8QYB9QhKWcwkQcnfdjEA@mail.gmail.com>
Message-ID: <e58b7448b75dded41c7e5234dde989c586f9b739.camel@rahix.de>

Hi Simon,

On Sun, 2023-10-08 at 14:16 -0600, Simon Glass wrote:
> Hi,
> 
> Since an update to Ubuntu 2022.04 on my lab machine I cannot get
> boards to connect:
> 
> Revision 7d1e5aacdb44bdc2139197a60d8c97c9062d5064, board rpi3
> 
> Checking revision 7d1e5aacdb44bdc2139197a60d8c97c9062d5064
> /vid/software/devel/ubtest
> tbot starting ...
> ??Parameters:
> ????? rev??????? = '7d1e5aacdb44bdc2139197a60d8c97c9062d5064'
> ????? clean????? = False
> ??Calling uboot_build_and_flash ...
> ??? ??Calling uboot_build ...
> ??? ??? ??Calling uboot_checkout ...
> ??? ??? ??? ??Builder: rpi3
> ??? ??? ??? ??Done. (6.016s)
> ??? ??? ??Configuring build ...
> ??? ??? ??Calling uboot_make ...
> ??? ??? ??? ??Done. (10.188s)
> ??? ??? ??Done. (18.444s)
> ??? ??Calling uboot_flash ...
> ??? ??? ??Calling copy ...
> ??? ??? ??? ??Done. (0.004s)
> ??? ??? ??Done. (0.646s)
> ??? ??Done. (19.390s)
> ??????????????????????????????????????????
> ??SUCCESS (19.486s)
> tbot starting ...
> ??Calling interactive_board ...
> ??? ??POWERON (rpi3)
> ??? ??Entering interactive shell (CTRL+D to exit) ...
> 
> FATAL: cannot lock /dev/ttyusb_port1: Resource temporarily unavailable

This is a telltale sign that there is still a session with the console
open somewhere. This can either happen

  - on the remote side (e.g. a picocom process lingering after the ssh
    session closed)

  - or locally (an ssh process not getting SIGTERMed correctly).

The latter shouldn't be possible with the paramiko connector unless you
start tbot multiple times within a single python interpreter and somehow
manage to leak channels.

> (after it hangs forever here, I press ctrl-D)
> 
> logout
> ??? ??POWEROFF (rpi3)
> ??? ??Exiting poweroff
> ??? ??Fail. (77.542s)
> ??Warning: Found dangling <class 'rpi3.Rpi3'> instance in this context
> ??Warning: Teardown went wrong!? A <class 'rpi3.Rpi3'> instance is still alive.
> ? Please report this to https://github.com/rahix/tbot/issues!

This here is looking very interesting to me.  Even when everything is
broken, you should not be able to produce this error... Might be
related.

In any case, I think it would be easiest to diagnose this with a bit
more info about your setup.  Can you share

  - how are you invoking tbot?
  - what terminal emulator are you using to open the console sessions?
  - if possible, the board & lab config which produces this error

Best regards,

Harald
-- 

> ??Exception:
> ??? Traceback (most recent call last):
> ????? File "/home/sglass/.local/lib/python3.10/site-packages/tbot/main.py",
> line 345, in main
> ??????? func(**params)
> ????? File "/home/sglass/.local/lib/python3.10/site-packages/tbot/decorators.py",
> line 62, in wrapped
> ??????? return tc(*args, **kwargs)
> ????? File "/home/sglass/.local/lib/python3.10/site-packages/tbot/tc/callable.py",
> line 43, in interactive_board
> ??????? with tbot.ctx.request(tbot.role.Board) as b:
> ????? File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__
> ??????? self.gen.throw(typ, value, traceback)
> ????? File "/home/sglass/.local/lib/python3.10/site-packages/tbot/context.py",
> line 359, in request
> ??????? with instance.request(exclusive, self._keep_alive) as m:
> ????? File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__
> ??????? self.gen.throw(typ, value, traceback)
> ????? File "/home/sglass/.local/lib/python3.10/site-packages/tbot/context.py",
> line 94, in request
> ??????? self.teardown()
> ????? File "/home/sglass/.local/lib/python3.10/site-packages/tbot/context.py",
> line 65, in teardown
> ??????? self._cx.close()
> ????? File "/usr/lib/python3.10/contextlib.py", line 584, in close
> ??????? self.__exit__(None, None, None)
> ????? File "/usr/lib/python3.10/contextlib.py", line 576, in __exit__
> ??????? raise exc_details[1]
> ????? File "/usr/lib/python3.10/contextlib.py", line 561, in __exit__
> ??????? if cb(*exc_details):
> ????? File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__
> ??????? next(self.gen)
> ????? File "/home/sglass/.local/lib/python3.10/site-packages/tbot/machine/connector/common.py",
> line 126, in from_context
> ??????? with contextlib.ExitStack() as cx:
> ????? File "/usr/lib/python3.10/contextlib.py", line 576, in __exit__
> ??????? raise exc_details[1]
> ????? File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__
> ??????? self.gen.throw(typ, value, traceback)
> ????? File "/home/sglass/.local/lib/python3.10/site-packages/tbot/context.py",
> line 379, in request
> ??????? raise e from None
> ????? File "/home/sglass/.local/lib/python3.10/site-packages/tbot/context.py",
> line 366, in request
> ??????? yield m
> ????? File "/usr/lib/python3.10/contextlib.py", line 561, in __exit__
> ??????? if cb(*exc_details):
> ????? File "/home/sglass/.local/lib/python3.10/site-packages/tbot/machine/machine.py",
> line 186, in __exit__
> ??????? self._cx.__exit__(*args)
> ????? File "/usr/lib/python3.10/contextlib.py", line 576, in __exit__
> ??????? raise exc_details[1]
> ????? File "/usr/lib/python3.10/contextlib.py", line 561, in __exit__
> ??????? if cb(*exc_details):
> ????? File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__
> ??????? next(self.gen)
> ????? File "/home/sglass/.local/lib/python3.10/site-packages/tbot/machine/connector/common.py",
> line 133, in _connect
> ??????? with self.host.clone() as cloned, self.connect(cloned) as ch:
> ????? File "/home/sglass/.local/lib/python3.10/site-packages/tbot/machine/channel/channel.py",
> line 562, in __exit__
> ??????? self.close()
> ????? File "/home/sglass/.local/lib/python3.10/site-packages/tbot/machine/channel/channel.py",
> line 540, in close
> ??????? self._c.close()
> ????? File "/home/sglass/.local/lib/python3.10/site-packages/tbot/machine/channel/paramiko.py",
> line 58, in close
> ??????? raise channel.ChannelClosedException()
> ??? tbot.machine.channel.channel.ChannelClosedException
> ??????????????????????????????????????????
> ??FAILURE (77.579s)
> 
> 
> If I then connect again, it works, mostly. Has something changed in
> paramiko? Any clues?
> 
> Regards,
> Simon

From sjg at chromium.org  Mon Oct  9 02:52:13 2023
From: sjg at chromium.org (Simon Glass)
Date: Sun, 8 Oct 2023 18:52:13 -0600
Subject: cannot lock /dev/ttyusb_port1: Resource temporarily unavailable
In-Reply-To: <e58b7448b75dded41c7e5234dde989c586f9b739.camel@rahix.de>
References: <CAPnjgZ0L8U-hxa+bgkB4kT3L1TNmDz8QYB9QhKWcwkQcnfdjEA@mail.gmail.com>
 <e58b7448b75dded41c7e5234dde989c586f9b739.camel@rahix.de>
Message-ID: <CAPnjgZ00VFb-1Ovq4VG2WP3-mpNaTiwD14K5djEdA5cJ6NaEBQ@mail.gmail.com>

Hi Harald,

On Sun, 8 Oct 2023 at 17:07, Rahix <rahix at rahix.de> wrote:
>
> Hi Simon,
>
> On Sun, 2023-10-08 at 14:16 -0600, Simon Glass wrote:
> > Hi,
> >
> > Since an update to Ubuntu 2022.04 on my lab machine I cannot get
> > boards to connect:
> >
> > Revision 7d1e5aacdb44bdc2139197a60d8c97c9062d5064, board rpi3
> >
> > Checking revision 7d1e5aacdb44bdc2139197a60d8c97c9062d5064
> > /vid/software/devel/ubtest
> > tbot starting ...
> > ??Parameters:
> > ?     rev        = '7d1e5aacdb44bdc2139197a60d8c97c9062d5064'
> > ?     clean      = False
> > ??Calling uboot_build_and_flash ...
> > ?   ??Calling uboot_build ...
> > ?   ?   ??Calling uboot_checkout ...
> > ?   ?   ?   ??Builder: rpi3
> > ?   ?   ?   ??Done. (6.016s)
> > ?   ?   ??Configuring build ...
> > ?   ?   ??Calling uboot_make ...
> > ?   ?   ?   ??Done. (10.188s)
> > ?   ?   ??Done. (18.444s)
> > ?   ??Calling uboot_flash ...
> > ?   ?   ??Calling copy ...
> > ?   ?   ?   ??Done. (0.004s)
> > ?   ?   ??Done. (0.646s)
> > ?   ??Done. (19.390s)
> > ??????????????????????????????????????????
> > ??SUCCESS (19.486s)
> > tbot starting ...
> > ??Calling interactive_board ...
> > ?   ??POWERON (rpi3)
> > ?   ??Entering interactive shell (CTRL+D to exit) ...
> >
> > FATAL: cannot lock /dev/ttyusb_port1: Resource temporarily unavailable
>
> This is a telltale sign that there is still a session with the console
> open somewhere. This can either happen
>
>   - on the remote side (e.g. a picocom process lingering after the ssh
>     session closed)
>
>   - or locally (an ssh process not getting SIGTERMed correctly).
>
> The latter shouldn't be possible with the paramiko connector unless you
> start tbot multiple times within a single python interpreter and somehow
> manage to leak channels.

Thanks for the info.

I don't see either of those...e.g. I can see the picocom going away on
the remote machine (with the lab).

>
> > (after it hangs forever here, I press ctrl-D)
> >
> > logout
> > ?   ??POWEROFF (rpi3)
> > ?   ??Exiting poweroff
> > ?   ??Fail. (77.542s)
> > ??Warning: Found dangling <class 'rpi3.Rpi3'> instance in this context
> > ??Warning: Teardown went wrong!  A <class 'rpi3.Rpi3'> instance is still alive.
> > ? Please report this to https://github.com/rahix/tbot/issues!
>
> This here is looking very interesting to me.  Even when everything is
> broken, you should not be able to produce this error... Might be
> related.
>
> In any case, I think it would be easiest to diagnose this with a bit
> more info about your setup.  Can you share
>
>   - how are you invoking tbot?

tbot -l ${lab} -b ${board}.py -T tbot/contrib -p rev=\"${rev}\" \
-p clean=${clean} $patch $V uboot_build_and_flash
tbot -l ${lab} -b ${board}.py interactive_board

my tree is here:
https://github.com/sjg20/tbot/pull/new/lab8

>   - what terminal emulator are you using to open the console sessions?

In this case I am just running tbot and relying on it to attach the console

>   - if possible, the board & lab config which produces this error

I see it on all boards - this one happens to be a rpi4, but I can
repeat this on any of them.

The lab machine is remote (in the next room).

Regards,
Simon

>
> Best regards,
>
> Harald
> --
>
> > ??Exception:
> > ?   Traceback (most recent call last):
> > ?     File "/home/sglass/.local/lib/python3.10/site-packages/tbot/main.py",
> > line 345, in main
> > ?       func(**params)
> > ?     File "/home/sglass/.local/lib/python3.10/site-packages/tbot/decorators.py",
> > line 62, in wrapped
> > ?       return tc(*args, **kwargs)
> > ?     File "/home/sglass/.local/lib/python3.10/site-packages/tbot/tc/callable.py",
> > line 43, in interactive_board
> > ?       with tbot.ctx.request(tbot.role.Board) as b:
> > ?     File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__
> > ?       self.gen.throw(typ, value, traceback)
> > ?     File "/home/sglass/.local/lib/python3.10/site-packages/tbot/context.py",
> > line 359, in request
> > ?       with instance.request(exclusive, self._keep_alive) as m:
> > ?     File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__
> > ?       self.gen.throw(typ, value, traceback)
> > ?     File "/home/sglass/.local/lib/python3.10/site-packages/tbot/context.py",
> > line 94, in request
> > ?       self.teardown()
> > ?     File "/home/sglass/.local/lib/python3.10/site-packages/tbot/context.py",
> > line 65, in teardown
> > ?       self._cx.close()
> > ?     File "/usr/lib/python3.10/contextlib.py", line 584, in close
> > ?       self.__exit__(None, None, None)
> > ?     File "/usr/lib/python3.10/contextlib.py", line 576, in __exit__
> > ?       raise exc_details[1]
> > ?     File "/usr/lib/python3.10/contextlib.py", line 561, in __exit__
> > ?       if cb(*exc_details):
> > ?     File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__
> > ?       next(self.gen)
> > ?     File "/home/sglass/.local/lib/python3.10/site-packages/tbot/machine/connector/common.py",
> > line 126, in from_context
> > ?       with contextlib.ExitStack() as cx:
> > ?     File "/usr/lib/python3.10/contextlib.py", line 576, in __exit__
> > ?       raise exc_details[1]
> > ?     File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__
> > ?       self.gen.throw(typ, value, traceback)
> > ?     File "/home/sglass/.local/lib/python3.10/site-packages/tbot/context.py",
> > line 379, in request
> > ?       raise e from None
> > ?     File "/home/sglass/.local/lib/python3.10/site-packages/tbot/context.py",
> > line 366, in request
> > ?       yield m
> > ?     File "/usr/lib/python3.10/contextlib.py", line 561, in __exit__
> > ?       if cb(*exc_details):
> > ?     File "/home/sglass/.local/lib/python3.10/site-packages/tbot/machine/machine.py",
> > line 186, in __exit__
> > ?       self._cx.__exit__(*args)
> > ?     File "/usr/lib/python3.10/contextlib.py", line 576, in __exit__
> > ?       raise exc_details[1]
> > ?     File "/usr/lib/python3.10/contextlib.py", line 561, in __exit__
> > ?       if cb(*exc_details):
> > ?     File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__
> > ?       next(self.gen)
> > ?     File "/home/sglass/.local/lib/python3.10/site-packages/tbot/machine/connector/common.py",
> > line 133, in _connect
> > ?       with self.host.clone() as cloned, self.connect(cloned) as ch:
> > ?     File "/home/sglass/.local/lib/python3.10/site-packages/tbot/machine/channel/channel.py",
> > line 562, in __exit__
> > ?       self.close()
> > ?     File "/home/sglass/.local/lib/python3.10/site-packages/tbot/machine/channel/channel.py",
> > line 540, in close
> > ?       self._c.close()
> > ?     File "/home/sglass/.local/lib/python3.10/site-packages/tbot/machine/channel/paramiko.py",
> > line 58, in close
> > ?       raise channel.ChannelClosedException()
> > ?   tbot.machine.channel.channel.ChannelClosedException
> > ??????????????????????????????????????????
> > ??FAILURE (77.579s)
> >
> >
> > If I then connect again, it works, mostly. Has something changed in
> > paramiko? Any clues?
> >
> > Regards,
> > Simon