[coreboot] AMD's binary-only AGESA libraries

[Bruce Griffith]

AMD has recently made a decision to supply AGESA as binary-only for future
AGESA releases. This decision seems to have caused some grief and
disharmony within the coreboot community. I would like to make a proposal
that I hope satisfies everyone moving forward.



First some history:



The initial AMD releases to coreboot were not based on either AGESA or
CIM-X. Sometime after AMD started contributing AMD 64-bit processor
support, someone realized that core libraries were available elsewhere in
AMD to initialize the processor and chipset in a standard way (AGESA and
CIMX). Ultimately, that turned into the AGESA and CIM-X open-source
postings in vendorcode. AMD still develops AGESA for other (proprietary)
purposes and relicenses and forks AGESA for new processors for coreboot.



To get to an open-source AGESA, there were meetings and rules put in place
within AMD with the intent that AMD would never lose proprietary license or
ownership of the trunk AGESA or CIM-X codebases resulting from the
contribution. The principal rules are that:



1. No modifications derived from open-source AGESA ever go back into
AMD’s codebase.

2. Open-source AGESA always derives from a processor-specific fork of
AGESA that is reviewed and edited to remove any proprietary code,
identifiers, or concepts.



So AMD accepted that the process of releasing into open-source would be a
very large and time-consuming task. No matter how good the process, any
automated pruning effort will depend on authors to add tags to code to
identify what is proprietary and what is not. Authors tend to make
mistakes and therefore require review. So they elected to keep it a manual
process.



Jumping ahead to 2013:



AMD realized that the open-sourcing process they developed was awkward and
required lots of developer and legal involvement. The six month delay
required to implement the “scrub” prevents AMD Embedded from capitalizing
on the advertising and press excitement following the introduction of new
parts. This delays commercial shipment of systems (and therefore chips)
until well into the design cycle of the next processor. Anyone that
received early code was required to modify their application when the
scrubbed code was eventually released. And they were not allowed to
release their product until after they incorporated the scrubbed code.



Please remember that AMD’s primary goal in open-source AGESA is to enable
new platforms to sell more chips.



Binary PI was conceived with the sole intent of eliminating the delay
caused by the scrub. By donating AGESA as a binary instead of scrubbed
open-source, coreboot becomes available much sooner in the lifecycle of a
processor. This is good for AMD and good for companies that want to use
AMD’s chips.



There are downsides:

· The AGESA binary has to have a well-defined interface. Normal
mechanisms to work around interface deficiencies don’t work, so you have to
work within the API.

· Since the same BLOB is available to everyone, the API must be as
flexible as possible to accommodate varying needs.

· Because there is no scrub, the source must remain proprietary.

· There is less flexibility because you can’t change AGESA to
change when, where, or how initialization functions occur.

· There may be a performance or size impact. Binary PI currently
runs exclusively from ROM and is not split between a ROM phase and a RAM
phase.



The current binary PI implementation requires an Application Programming
Interface (API) to insulate the interface from future changes. This is the
same AGESA API that has been open-sourced for several processor
generations. The API uses processor-specific structures as arguments and
therefore must be compiled using header files that accompany the AGESA
library releases. Note that this differs from an Application Binary
Interface (ABI). I have maintained API compatibility between open-source
AGESA and binary PI. This is for two purposes:

1. To allow early reference platform development using an integrated
AGESA. The code is easier to debug when it is all contained within one
binary.

2. In the hopes that there will be future open-source AGESA
contributions from AMD late in the design cycle of a processor, well after
binary PI is released.



Proposal:



There has been a request and changelist to split the AGESA wrappers:

1. Slowly migrating open-source AGESA away from the published AGESA
API for Llano (F12), Ontario (F14), Trinity (F15tn), Orochi (F15), and
Kabini (F16kb).

2. Creating new wrappers for binary PI-based processors, namely Bald
Eagle (00630F01), and Steppe Eagle (00730F01).



The same wrappers, or at least wrapper structure, are shared by all of
these platforms today. Note that the API is completely contained with the
3rdparty/pi and src/vendorcode/amd subtree. Code in src/mainboard,
src/northbridge, src/southbridge, and src/cpu is “wrapper code” and fair
game for modification within the coreboot community.



Instead of splitting the wrappers this way, I would propose forking
open-source AGESA within the coreboot tree. Let’s COPY the existing
vendorcode/amd/agesa into another subtree, maybe soc or devices, and rename
it OSIFA (open software interface for AMD) or something similar. Move or
clone experimenter boards over to OSIFA.



Let’s leave commercial production boards alone (AMD, ASRock IMB-A180, some
tyan, some supermicro, some HP, maybe a few more) using the existing
src/
/agesa directories. That leaves the interface stable for the boards
AMD needs to demonstrate, allows AMD to continue donating AGESA with their
existing interface, yet still allows the community to improve on and
reorganize OSIFA.



What do you think 
 ?

[Kyösti Mälkki]

Post by Bruce Griffith
AMD has recently made a decision to supply AGESA as binary-only for future
AGESA releases. This decision seems to have caused some grief and
disharmony within the coreboot community. I would like to make a proposal
that I hope satisfies everyone moving forward.
The initial AMD releases to coreboot were not based on either AGESA or
CIM-X. Sometime after AMD started contributing AMD 64-bit processor
support, someone realized that core libraries were available elsewhere in
AMD to initialize the processor and chipset in a standard way (AGESA and
CIMX). Ultimately, that turned into the AGESA and CIM-X open-source
postings in vendorcode. AMD still develops AGESA for other (proprietary)
purposes and relicenses and forks AGESA for new processors for coreboot.
To get to an open-source AGESA, there were meetings and rules put in place
within AMD with the intent that AMD would never lose proprietary license or
ownership of the trunk AGESA or CIM-X codebases resulting from the
1. No modifications derived from open-source AGESA ever go back into
AMD’s codebase.
2. Open-source AGESA always derives from a processor-specific fork of
AGESA that is reviewed and edited to remove any proprietary code,
identifiers, or concepts.
So AMD accepted that the process of releasing into open-source would be a
very large and time-consuming task. No matter how good the process, any
automated pruning effort will depend on authors to add tags to code to
identify what is proprietary and what is not. Authors tend to make
mistakes and therefore require review. So they elected to keep it a manual
process.
AMD realized that the open-sourcing process they developed was awkward and
required lots of developer and legal involvement. The six month delay
required to implement the “scrub” prevents AMD Embedded from capitalizing
on the advertising and press excitement following the introduction of new
parts. This delays commercial shipment of systems (and therefore chips)
until well into the design cycle of the next processor. Anyone that
received early code was required to modify their application when the
scrubbed code was eventually released. And they were not allowed to
release their product until after they incorporated the scrubbed code.

And the rule for the customer to _not_ release product until the
scrubbed code is available remains to be a rule that is dictated by
coreboot's GPL license.

Given that there is often a 3rdparty involved, say SAGE, who does
coreboot / AGESA integration work under a contract for an end customer,
it can be claimed the time of coreboot distribution happens already when
said customer receives a copy of coreboot firmware image to test on
their platform. And this may happen several months before the release of
product to public.

The choice of GPL over LGPL implies a promise to both commercial and
non-commercial contributors of coreboot codebase that hardware vendors
will need to make necessary files available, so that the customer can
rebuild and improve said coreboot firmware for the platform.

You have verified binaryPI uses (almost) exactly the same API as
open-source AGESA so far. This complex API between AGESA component and
coreboot does not fall under the mere aggregation terms of GPL, it is a
form of runtime linking. In other words the distribution of binaryPI
violates rights of every copyright holder in the coreboot sourcebase.
I am not fond of Google's MRC binary or Intel's FSP either, but their
implementation appears to be a single entry/exit with an array of
platform configuration data.

Post by Bruce Griffith
Please remember that AMD’s primary goal in open-source AGESA is to enable
new platforms to sell more chips.

Yes. And I understand AMD's and SAGE's urges and interest for the early
distribution in binary form. The major trouble here is the question has
turned from 'When to scrub AGESA' to 'If to scrub AGESA'.

The community needs to evaluate if infrastructure of said binaryPI
brings any added value for coreboot, when there is no promise or a
schedule for scrubbed AGESA sources. There has been empty promises in
the past when it comes to updates for AGESA vendorcode. The community
has to deal with what we see going on in the tree, as neither AMD AES or
SAGE are not so willing to discuss any of their plans with community.

More importantly, we have an absurd situation here. A claimed
compatibility with a source component we have not been promised to ever
receive is currently delaying and restricting development of code we
already have in the tree.


Personally, I would be willing to flex from the strict GPL terms here
for the benefit of the industry, AMD AES, SAGE and the hardware vendor.
That would require AMD AES to publish a schedule in which scrubbed AGESA
sources under BSD -style license are released, and a regular schedule of
bugfixes to previously released open-source AGESA.

My expectations on binaryPI are that it would be virtually impossible to
do any community ports of AMD platforms with it. Also concepts of
timestamping, USB debug and CBMEM console from romstage appear to be
impossible to achieve.

Existing code in the tree suggests even AMD AES and SAGE were not able
to implement ACPI S3 within the AGESA API for fam15tn and fam16kb, but
chose to bypass the API and link directly with a couple functions in
AGESA vendorcode proper.

Post by Bruce Griffith
Binary PI was conceived with the sole intent of eliminating the delay
caused by the scrub. By donating AGESA as a binary instead of scrubbed
open-source, coreboot becomes available much sooner in the lifecycle of a
processor. This is good for AMD and good for companies that want to use
AMD’s chips.
· The AGESA binary has to have a well-defined interface. Normal
mechanisms to work around interface deficiencies don’t work, so you have to
work within the API.
· Since the same BLOB is available to everyone, the API must be as
flexible as possible to accommodate varying needs.
· Because there is no scrub, the source must remain proprietary.
· There is less flexibility because you can’t change AGESA to
change when, where, or how initialization functions occur.
· There may be a performance or size impact. Binary PI currently
runs exclusively from ROM and is not split between a ROM phase and a RAM
phase.
The current binary PI implementation requires an Application Programming
Interface (API) to insulate the interface from future changes. This is the
same AGESA API that has been open-sourced for several processor
generations. The API uses processor-specific structures as arguments and
therefore must be compiled using header files that accompany the AGESA
library releases. Note that this differs from an Application Binary
Interface (ABI). I have maintained API compatibility between open-source
1. To allow early reference platform development using an integrated
AGESA. The code is easier to debug when it is all contained within one
binary.
2. In the hopes that there will be future open-source AGESA
contributions from AMD late in the design cycle of a processor, well after
binary PI is released.

1. Can You explain SAGE's role in all of this?

2. As You write you (SAGE) "maintain open-source AGESA and binaryPI
compatibility", this must mean SAGE has early access to un-scrubbed
AGESA and the changes proposed to it? Would an early release of AGESA
sources affect negatively on SAGE's consultation revenue?

3. What is Your opinion on community ports for AMD platforms? How should
we debug or develop those in future? Or is the answer that we need to
consult with the commercial partner who have access to un-scrubbed AGESA
to debug as much as they wish / how much they are contracted for? I
assume here binaryPI is virtually impossible to debug for board
bring-up. I assume binaryPI is built with all debugging output to
console (aka IDS) disabled with no possibility to enable at runtime?


Regarding maintained API compatibility:

In my opinion maintaining an API involves a single set of header files
and a source base where bugfixes are backported to previously released
chipsets and CPU families. I can see that even within the two binaryPI
releases pushed to coreboot blobs.git, the header files are not the same.

Post by Bruce Griffith
1. Slowly migrating open-source AGESA away from the published AGESA
API for Llano (F12), Ontario (F14), Trinity (F15tn), Orochi (F15), and
Kabini (F16kb).
2. Creating new wrappers for binary PI-based processors, namely Bald
Eagle (00630F01), and Steppe Eagle (00730F01).
The same wrappers, or at least wrapper structure, are shared by all of
these platforms today. Note that the API is completely contained with the
3rdparty/pi and src/vendorcode/amd subtree. Code in src/mainboard,
src/northbridge, src/southbridge, and src/cpu is “wrapper code” and fair
game for modification within the coreboot community.

Well.. not exactly the same wrappers. Same wrapper structure? Not
exactly that either, as fam15tn and fam16kb ACPI S3 support adds
functions that do not exist for cimx/sb800 ACPI S3 implementation for fam14.

In the spirit of AGESA API, platform specifics would belong to
BiosCallOuts and no board-specific agesawrapper.c file should have ever
existed. But since board-specific agesawrapper.c files did exists,
someone though it was okay to modify those nevertheless.

Post by Bruce Griffith
Instead of splitting the wrappers this way, I would propose forking
open-source AGESA within the coreboot tree. Let’s COPY the existing
vendorcode/amd/agesa into another subtree, maybe soc or devices, and rename
it OSIFA (open software interface for AMD) or something similar. Move or
clone experimenter boards over to OSIFA.
Let’s leave commercial production boards alone (AMD, ASRock IMB-A180, some
tyan, some supermicro, some HP, maybe a few more) using the existing
src/…/agesa directories. That leaves the interface stable for the boards
AMD needs to demonstrate, allows AMD to continue donating AGESA with their
existing interface, yet still allows the community to improve on and
reorganize OSIFA.

You are effectively saying that you do not want any of the improvements
that coreboot has gained from Chromebook development, to be available on
AMD reference designs? Is this the opinion of You as an individual, the
opinion of SAGE or that of the AMD Advanced Embedded Solutions division?

The approach I am suggesting brings improvements for all the AMD
platforms where open-source AGESA is available, yet leaving a stable
environment for those who are restricted to use of binaryPI.

Mind you, we are not even talking about modifying the AGESA API per se.
Let's dig into this deeper:

A pre-requirement to improve the coreboot experience for any AMD
platform is to get rid of the excessive copy-paste approach both AMD AES
and SAGE seems to be fond of. Some of this work has already been merged
(BiosCallOuts, GetBusConf). My work to have common (per family)
agesawrapper.c files is under review, and it seemed to have positive
feedback from You. All that work is currently on hold due this on-going
dispute on the separation of open-source and binaryPI trees.

To think further, understand that a requirement to use a common
agesawrapper.c file across fam12-fam16kb is for the vendorcode to
actually expose a common API. A common API does not involve forking the
header files for each family separately. The community has the
possibility to unify the function prototypes within the different
families of open-source AGESA API, should we find any.

The requirement to build binaryPI from a specific set of header files
from blobs/ justifies the move of binaryPI to a sandbox. The matter
around binaryPI is further complicated as the header files between
different binaryPI releases are not the same. I do not want to see all
the complications and #ifdefs this may introduce to affect the quality
of the open-source AGESA -based implementations.


If I choose to bypass some of the lop layers of AGESA API for better
performance on one platform, it does not mean I would need to remove
unused layers of AGESA API from vendorcode/.

Once/if binaryPI gets scrubbed, it should not be a big deal to import
the API header file changes from binaryPI release into the single set of
open-source AGESA API header files under vendorcode/. After all, you
claim a compatibility in binaryPI and open-source AGESA APIs.

Post by Bruce Griffith
What do you think … ?

Missing signature at end. Does this mail reflect the opinions of You as
an individual developer, SAGE or the unanimous voice of SAGE and AMD
Advanced Embedded Solutions division?


Kind Regards,
Kyösti Mälkki

* Cleaning up AMD platforms sources in coreboot since 2012 *

[Patrick Georgi]

Post by Kyösti Mälkki
You have verified binaryPI uses (almost) exactly the same API as
open-source AGESA so far. This complex API between AGESA component
and coreboot does not fall under the mere aggregation terms of GPL,
it is a form of runtime linking. In other words the distribution of
binaryPI violates rights of every copyright holder in the coreboot
sourcebase. I am not fond of Google's MRC binary or Intel's FSP
either, but their implementation appears to be a single entry/exit
with an array of platform configuration data.

Please be careful with statements like that (that is: just don't. please.)

I talked with a number of lawyers over matters like that (some of them specialists in various jurisdictions' copyright laws), and they could make all kinds of compelling cases - for pretty much every outcome.
All that "does situation X violate term Y in license Z" stuff is a matter of degrees. The result may even depend on the country in which you pose the question.


Patrick

[ron minnich]

This discussion has gone way off base, as such discussions can do when
licensing comes into the picture.

Bruce asked a simple question: can we rename some directories/move
some code around in response to a change in how a vendor is delivering
their chipset support?

That's a pretty simple question.

All of a sudden we're talking about what AMD should have done, should
do now, or will do. That's not the question.

Bruce, is it possible to rephrase your question in simple terms, one
paragraph, and not bring in what 3rd parties such as Sage and AMD
might do, and just pose the question absent any thought to what might
happen at some future time?

thanks

ron

[Kyösti Mälkki]

Post by ron minnich
This discussion has gone way off base, as such discussions can do when
licensing comes into the picture.
Bruce asked a simple question: can we rename some directories/move
some code around in response to a change in how a vendor is delivering
their chipset support?

Ron, to clarify, Bruce did not originally pop this question.

See http://review.coreboot.org/#/c/7149/

Post by ron minnich
That's a pretty simple question.
All of a sudden we're talking about what AMD should have done, should
do now, or will do. That's not the question.
Bruce, is it possible to rephrase your question in simple terms, one
paragraph, and not bring in what 3rd parties such as Sage and AMD
might do, and just pose the question absent any thought to what might
happen at some future time?

Ron, see Marc's blocking review on patchset 2. That one brought the
speculations of what future might bring into this discussion.

Post by ron minnich
thanks
ron

Kyösti

[Peter Stuge]

Ron,

pose the question absent any thought to what might happen at some
future time?

I think coreboot has had BY FAR enough of thoughtless action.


//Peter

[ron minnich]

Post by Peter Stuge
Ron,

pose the question absent any thought to what might happen at some
future time?

I think coreboot has had BY FAR enough of thoughtless action.

So I'll rephrase.

"pose the question absent a bunch of bootless speculation about what
might happen at some future time?"

BTW, on reflection, I'm fine with http://review.coreboot.org/#/c/7149/.

If you really want a full source-based coreboot for a chipset
nowadays, it's not going to happen on any modern x86, I am sorry to
say. And, it won't happen on many ARMs, particularly in the v8 space.

ron

[Marc Jones]

Post by ron minnich

Post by Peter Stuge
Ron,

pose the question absent any thought to what might happen at some
future time?

I think coreboot has had BY FAR enough of thoughtless action.

So I'll rephrase.
"pose the question absent a bunch of bootless speculation about what
might happen at some future time?"
BTW, on reflection, I'm fine with http://review.coreboot.org/#/c/7149/.

Based on the community feedback and this email discussion, isolation
of the PI and AGESA source is the preferred development path. While we
were trying for a common interface, it is not necessary. I concede
that AMD has not contributed updates to AGESA releases and has had
poor community interactions. It should be more beneficial to the
development of coreboot to fix the long standing issues in the code. I
have removed my NACK(-2) on 7149 and it should be un-abandoned. Thanks
to each of you to your contributions and improvements to coreboot.

Regards,
Marc

[Peter Stuge]

Bruce,

I have no beef with you, but you are the closest to a communications
channel with AMD that the community is given access to, so I pass my
comments to you.

Post by Bruce Griffith
To get to an open-source AGESA, there were meetings and rules put in place
within AMD with the intent that AMD would never lose proprietary license or
ownership of the trunk AGESA or CIM-X codebases resulting from the
contribution.

As these contributions started to flow into the community many years
ago, you may remember that I guessed that such rules would exist and
asked *literally* what those rules were. You may also remember that
there was *NO* response whatsoever.
Three (is it four? five?) years later it is nice to see these rules
finally confirmed. Of course, by now, all of us fools have already
been able to deduce them.

Unfortunately, by not communicating with the community, AMD has
caused a significant amount of uncertanity and confusion within the
project. Because community projects are supposed to be about
collaboration and communication, this has placed AMD in a worse
position than anyone wished for.

Although coreboot has received contributions from AMD over a long
time already, it is also painfully obvious that AMD still has some
ways to go until they can get proper returns on their investment in
coreboot.

You may understand that I, as a volunteer, can only do *so* much to
help AMD make the most of coreboot. I believe I have asked exactly
the right questions but without dialogue there can not be progress.

Post by Bruce Griffith
AMD accepted that the process of releasing into open-source would be a
very large and time-consuming task.

It is obviously backwards. I wonder, if someone in those meetings
suggested "let's make the open-source code our HEAD so that we only
have to scrub once and worst-case only incrementally thereafter".

Major corporations like AMD are perfectly capable of contributing to
and even running open source projects. There are many sources of
inspiration.

Post by Bruce Griffith
AMD realized that the open-sourcing process they developed was awkward
and required lots of developer and legal involvement.

I think you understand that the community is quite frustrated since
many attempts were made right at the start to explain that this would
be the case. Community members with any meaningful open source
experience immediately recognize how resource intensive the scrubbing
process would be for AMD.

Post by Bruce Griffith
The six month delay required to implement the “scrub” prevents
AMD Embedded from capitalizing

But ultimately this is AMD's problem and not coreboot's.

Post by Bruce Griffith
Please remember that AMD’s primary goal in open-source AGESA is to
enable new platforms to sell more chips.

Please remember that the project's primary goal is to create an open
source firmware.

Post by Bruce Griffith
Binary PI

..is not open source firmware, is it?

Post by Bruce Griffith
By donating AGESA as a binary instead of scrubbed open-source,
coreboot becomes available

That's false. coreboot is an open source project. Binary PI is not
open source, so can not be part of coreboot. If it were, coreboot's
GPL would make a combination of the two difficult to use.

Post by Bruce Griffith
This is good for AMD and good for companies that want to use AMD’s chips.

It is however not good for coreboot, the open source firmware project.

Post by Bruce Griffith
I would propose forking open-source AGESA within the coreboot tree.
What do you think … ?

Let's not be in such a hurry to make changes for once, let's try to
think this thing through?


Industry contributors appear unable to see the forest for all the trees.

I notice a trend within governments to require more transparency from
IT equipment suppliers. It will take a couple of years, but those
requirements will eventually bubble up also to Intel and AMD, who
will have to either open up or lose market share. Things will
progressively get worse for anyone who insists on remaining opaque,
as industry starts adopting the very same requirements as the
government.

I predict that whoever opens up first has a great advantage in
winning contracts over the next 5 years, but what do I know.


//Peter