[RFC PATCH 03/28] cli: lil: Replace strclone with strdup

[Sean Anderson]

On 7/5/21 1:50 PM, Wolfgang Denk wrote:
> Dear Sean,
> 
> In message <5a967151-94f0-6037-2d02-0114c43b846c at gmail.com> you wrote:
>>
>> AIUI hush has diverged significantly from what U-Boot has. This would
>> not be an "update" moreso than a complete port in the style of the
>> current series.
> 
> Agreed.  However, as you write this it sounds like a big problem
> only.  I disagree here - it is also a chance to do a few things
> different than with the original port.
> 
> Please keep in mind when this original port of the hush shell was
> done:  it was a time when many systems came with a total of 4 MB
> flash memory, and not only U-Boot, but also the Linux kernel and a
> ram-disk image or such had to fit into that.  At that time 40 or 60 kB
> code size for just a fancy shell was immense!
> 
> Under such restrictions (and the need to complete the task with a
> given budget) many things were just commented out which from today's
> point of view would be nice to have.
> 
> 
> This is not a problem of the code complexity or resources, but only
> of different requirements and different resources.
> 
> 
>> I don't think sh-style shells are a good match for U-Boot's execution
>> environment in the first place. The fundamental idea of an sh-style
>> shell is that the output of one command can be redirected to the input
>> (or arguments) of another command. This cannot be done (or rather would
>> be difficult to do) in U-Boot for a few reasons
> 
> There is an old saying:
> 
> 	The loser says: "It might be possible, but it is too difficult."
> 	The winner says: "It might be difficult, but it is possible."

And yet, the winner may also say "this is too difficult to be worth it"
instead of laboring inefficiently :)

> 
> Apparently you take another position here than me.
> 
> First, I disagree that pipes are the "fundamental idea" of a shell.
> It is a fundamental idea of the UNIX operating systems, indeed.

Well, I think all programming languages at their core are about
transferring information from one part of the program to another.
Without pipes, how do bourne-style shells communicate? Sure you can
transfer data into functions using arguments, but what about the other
way around? Numeric return values? Global variables? I think these are
rather anemic compared to proper return values.

> But please keep in mind that we are here in a boot loader, not in a
> full-blown OS context.
> 
>> * U-Boot does not support multithreading. Existing shells tend to depend
>>     strongly on this feature of the enviromnent. Many of the changes to
>>     U-Boot's hush are solely to deal with the lack of this feature.
> 
> I disagree to both parts of your statement.
> 
> Multithreading (or rather, a concept of separate processes which can
> eventually even run in parallel) is very nice to have, but it is not
> mandatory in most cases.  Command pipes can almost always be
> strictly sequentialized and thus run in a single-task environment,
> too.  I'm not sure, but I think I even remember pipes in the "shell"
> of the CPM "OS" on Z80 processors a number of decades ago...

The point here is that many Hush features were written with fork. For
example, to create a new scope (such as for a subshell), Hush just
fork()s. When parsing, if it encounters a construct like $(), it fork()s
and then modifies the map variable, (correctly) assuming that the
"original" map will remain unmodified. These things are all over and
make doing a port difficult; especially when they crop up as unforseen
bugs. Though I suppose the real issue here is a lack of virtual memory.

> And the fact that our current version of hush did not attempt to
> keep these features was much more driven by strict memory footprint
> limitations that anything else.  I claim it would have been
> possible, and still is.
> 
> You also don't mention (and I guess you oversee) another critical
> fact: so far, U-Boot has no concept of files.  The classic design
> principle "everything is a file" is missing even more than the
> concept of processes.

IMO "everything is a file" is more of an API thing than a shell thing.
E.g. the idea that you can use open/read/write/close to access any
resource. But without pipes, I don't think such an abstraction is very
useful.

>> * Existing commands do not read from stdin, nor do they print useful
>>     information to stdout. Command output is designed for human
>>     consumption and is substantially more verbose than typical unix
>>     commands.
> 
> This is a statement which is correct, but it does not contain any
> pro or con for the discussion here.

In order for pipes to be useful, programs should generally follow the
Unix philosophy. In particular, many commands in U-Boot fall afoul of
rule 2:

> Expect the output of every program to become the input to another, as
> yet unknown, program. Don't clutter output with extraneous
> information. Avoid stringently columnar or binary input formats.
> Don't insist on interactive input.

Which makes it difficult to compose them.

> And if we had the features, it would be easy to fix.

I don't know about that. I think making U-Boot commands emit output
designed for machine consumption would require substantial effort.

>> * Tools such as grep, cut, tr, sed, sort, uniq, etc. which are extremely
>>     useful when working with streams are not present in U-Boot.
> 
> You are talking about OS environments here.  But we are a boot
> loader.

These programs are just as much part of the shell as the builtin
commands. They allow one to take input from one program and transform it
into the appropriate input to another program. This is especially
important for extracting information from programs which were not
designed to interface well.

> Also, this argument is not fair, as the suggested LIL does not
> provide grep, sed, awk, sort, uniq etc. functionality either, or
> does it?

Of course it is fair. These programs are just as much part of the *nix
programming environment as the bourne shells which invoke them.

> 
>> And of course, this feature is currently not present in U-Boot. To get
> 
> Correct, and for very good reasons.
> 
>> which will set my_uuid to the uuid of the selected partition. My issue
>> with this is threefold: every command must add new syntax to do this,
>> that syntax is inconsistent, and it prevents easy composition. Consider
>> a script which wants to iterate over partitions. Instead of doing
>>
>> 	for p in $(part list mmc 0); do
>> 		# ...
>> 	done
>>
>> it must instead do
>>
>> 	part list mmc 0 partitions
>> 	for p in $partitions; do
>> 		# ...
>> 	done
>>
>> which unnecessarily adds an extra step. This overhead accumulates with
>> each command which adds something like this.
> 
> Apparently you fail to understand that this "extra step" is primarily
> due to the fact that we are single tasking. Even if we has command
> substitution (like we could have with hush) the execution sequence
> would be serialized in exactly the same way under the hood - it's
> just not as directly visible.

Please don't assume what I do and do not understand ;)

My point here is that instead of having a consistent interface for
passing data from a command to its caller, we instead have an
inconsistent one which is different for each command.

>> Both of these workarounds are natural consequences of using a sh-tyle
>> shell in an environment it is not suited for. If we are going to go to
> 
> No, they are not.  They are much more the consequence of no strict
> design guidelines for commands, so everybody implements what fits his
> purposes best.  A cleaner approach does not require any of the
> additional features you've asked for - it would be possible as is.

So you would have every command have a new parameter at the end
specifying which environmental variable to put a return value in? IMO
such things should be done at the language level so that commands can
just call a function like set_result() and let the shell handle the
rest.

>> the effort of porting a new language (which must be done no matter if
>> we use Hush or some other language), we should pick one which has better
>> support for single-threaded programming.
> 
> In which way do you think "a new language" needs to be ported when
> switching from an old to a new version of hush?  It would be still a
> (mostly) POSIX compatible shell, with some restrictions.

Modern Hush is completely unrecognizable compared to what we have in
U-Boot today. Any "updating" effort would be akin to going over the
entire upstream codebase and porting it from scratch.

> I can fully understand that you defend your proposal, but let's be
> fair and stick with the facts.  Thanks.

And I'd appreciate if we could presume good faith of our peers.

--Sean