[coreboot] Asus KGPE-D16 with latest coreboot errors with Unsupported Hardware on Qubes 4 install missing IOMMU?

[T***@gmx.com]

Hi,
I have an Asus KGPE-D16 motherboard I am trying to get working with Coreboot and use with Qubes 4. It has a single AMD 6386 CPU and 128Gb DDR3 ECC RAM.
I have successfully cloned the git repository and built the coreboot.rom. However when I flash it on to the board and then run the Qubes installer it complains that there is an “Unsupported Hardware Error” and I get the text "This hardware lack features requred by Qubes OS. Missing features: IOMMU/VT-d/AMD-Vi , Interrupt Remapping”
If I install Fedora and run dmesg, AMD-Vi and the IOMMU all appear to be fine.

As it applies microcode updates.

I have tried flashing the board with an older version of Libreboot and can confim this proceeds through the Qubes 4 installation without issue. However, I wish to use a recent version of Coreboot for support of a Pci-E SSD and to ensure I am running the latest microcode updates.
Can anyone offer me some guidance please?
Is it possible I am not selecting the right options when I do “make nconfig” before building the rom?

63xx CPU's need a microcode update for working IOMMU and to fix a
critical security error thus you can't use libreboot.

I noted this on the wiki which is sadly now gone.

In the coreboot config enable "use binary only repo" and "generate
microcode update from tree" and then use the cbfstool and a livecd
(check dmesg) to verify it has been updated to the latest version that
includes the spectre protections as well as the IOMMU/NMI fixes.

Congrats on your libre purchase and let us know if you need help :D
I also suggest checking out the OpenPOWER9 Raptor Blackbird if you want
another owner controlled computer that is newer and faster.

[T***@gmx.com]

This post might be inappropriate. Click to display it.

[T***@gmx.com]

I followed these instructions and found that "ucode=scan" was already present.

But was it actually present in the grub.cfg you are using?

I also noticed that it had "iommu=no-igfx"

That is an intel thing

so I changed this to "iommu=on" and rebooted. Unfortunately this did not appear to have an effect and when I ran the "qubes-hcl-report" it still indicated IOMMU was > not active :-(

Can you provide me dmesg and # lspci -vvv from your system either in
qubes or xen (does not matter what version)

Post by T***@gmx.com

Have you successfully got a 6386 on this board running with Qubes and all the microcode updates?

Yeah I have with various 63xx CPU's.

To clarify, have you been successful with Qubes 4?

Yes I have and I know other people who have done it as well.

The rest of my systems are running this so I need to figure out a way to get version 4 running securely on this system.
Do you have any other suggestions?
If this is a Xen issue do you expect it to be a while before it is fixed and merged in to Qubes?

Note you can always use new versions of xen with qubes by installing
yourself but I doubt that would do anything.

If a 62xx CPU definitely works and doesn't have these problems, it may be better for me to procure one of those so I can be up and running quicker.

IMO Keep what you have and find out what is wrong the 63xx CPU's are
much faster than 62xx.

Post by T***@gmx.com

I have attached a copy of my serial output booting the system with 2 sticks of 16Gb ECC RAM (32Gb total) in case this is of any help.

Put your RAM back in that would not effect anything :]

I noticed it tended to take longer to boot with more RAM, so I took some out whilst I was troubleshooting :-)

Yeah more RAM takes longer to train.

Do you know if this system is likely to be more stable with a single CPU and 128Gb RAM in all its slots or whether it would be better to have 2 CPUs and split the 128Gb RAM between them (ie. all Orange slots populated)

Nah it doesn't matter you just can't have more than 192GB total without
potentially running in to issues.

Anyway don't worry I won't stop replying till all is well :D

[T***@gmx.com]

Oops sorry forgot I also need "sudo xl dmesg" from dom0!

[Nico Huber]

Hi Pete,

I'm wondering if my problem is related to not having any SATA drives
installed? (I just have a PCI-E SSD). It may be the case that the logic
to disable combined mode is not getting triggered in my scenario, yet it
would do if there was a SATA drive present.

it's a configuration issue. If you have nvram settings enabled (CONFIG_
USE_OPTION_TABLE), you can enable `sata_ahci_mode` with nvramtool.

No idea why combined mode is the default, it's only useful for OSes from
the '90s. It's not about the type of drives (SATA vs PATA) connected but
how the SATA controller identifies itself to the OS.

Hope that helps,
Nico

[T***@gmx.com]

Post by Nico Huber
Hi Pete,

I'm wondering if my problem is related to not having any SATA drives
installed? (I just have a PCI-E SSD). It may be the case that the logic
to disable combined mode is not getting triggered in my scenario, yet it
would do if there was a SATA drive present.

it's a configuration issue. If you have nvram settings enabled (CONFIG_
USE_OPTION_TABLE), you can enable `sata_ahci_mode` with nvramtool.

Ah so that is why it works for me but not for him, since I always
customize my CMOS options :D

Post by Nico Huber
No idea why combined mode is the default, it's only useful for OSes from
the '90s. It's not about the type of drives (SATA vs PATA) connected but
how the SATA controller identifies itself to the OS.

AHCI is faster and better as it supports NCQ, TRIM etc.

Conga-rats petey now you can fix it easy! just enable cmos settings in
menuconfig then go to the kgpe-d16 coreboot board directory and change
cmos.default, recompile/flash then reset your CMOS - OR if you already
have use cmos enabled you can simply change it via the cmos tool with
the required iomem=relaxed in the kernel command line.

I would also lower the log/debug level via menuconfig to speed boot
times and maybe change some other things in cmos.default like me

I have
experimental_memory_speed_boost enabled, my 1394 controller set to
disabled and SATA ALPM to enabled.

[Angel Pons]

Hello,

On Sun, Dec 2, 2018 at 4:07 PM petecb via coreboot

Thank you for all those details. I've now compiled a version with the default CMOS settings apart from the following changes
Minimum memory voltage = 1.35v
experimental_memory_speed_boost enabled
1394 controller disabled
SATA ALPM enabled.

Okay.

Unfortunately, this results in the Qubes installation on the SSD crashing before it gets to the password prompt and the Qubes installer crashing shortly after booting. They both produce a similar error message that has been too quick to catch so far but mentions "PCI" and "IRQ".

One of the options you selected has the word "experimental" on its
name. I would suspect it may cause issues.

Regards,

Angel Pons

[T***@gmx.com]

Post by Angel Pons
Hello,
On Sun, Dec 2, 2018 at 4:07 PM petecb via coreboot

Thank you for all those details. I've now compiled a version with the default CMOS settings apart from the following changes
Minimum memory voltage = 1.35v
experimental_memory_speed_boost enabled
1394 controller disabled
SATA ALPM enabled.

But what about the other one that you need? SATA AHCI mode? Of course I
also wouldn't start modding stuff before you get things working to start
though.

Post by Angel Pons
Okay.

Unfortunately, this results in the Qubes installation on the SSD crashing before it gets to the password prompt and the Qubes installer crashing shortly after booting. They both produce a similar error message that has been too quick to catch so far but mentions "PCI" and "IRQ".

One of the options you selected has the word "experimental" on its
name. I would suspect it may cause issues.

Ah well the good thing is that it is easy to remedy if you have added
the nvramcui to your selected payload (it is a cmdline cmos settings
utility)

My money is on the memory voltage though since the experimental option
works for me (it is an optimization related to 63xx CPUs not actually
RAM itself)

Post by Angel Pons
Regards,
Angel Pons

[Peter Stuge]

Thank you for all those details. I've now compiled a version with
the default CMOS settings apart from the following changes
Minimum memory voltage = 1.35v
experimental_memory_speed_boost enabled
1394 controller disabled
SATA ALPM enabled.

Please do not ever change more than a single thing at a time. You
waste not only your own time, but also that of those who help you.

Unfortunately, this results in the Qubes installation on the SSD crashing

Try changing only the SATA setting before moving on to anything else.


//Peter

[Nico Huber]

Hi Pete,

As the default SATA setting already appeared correct, I modified the 3
additional settings that Taiidan had already indicated worked
(memory_speed_boost, 1394 controller and SATA ALPM) so in my mind I was
only adjusting one additional setting, the memory voltage. I thought
this was fairly sensible as my RAM is definitely 1.35V.

did you have CONFIG_USE_OPTION_TABLE before? If not you potentially have
changed all settings. The defaults in the cmos.default file don't have
to be the same defaults that are hardcoded in the code and the file only
takes effect if CONFIG_USE_OPTION_TABLE is set. As I looked at the SATA
code, I know the AHCI setting is one that definitely differs.

That's a thing that could easily be improved. Check all the options and
their defaults in the source code and agree on one single reasonable
default.

I would value confirmation that it is safe to run the Crucial
CT16G3ERSLD4160B RAM I have in this board at 1.5v.

Yes, all DDR3L should be 1.5V compatible. I'm fairly confused that such
a board has a 1.35V option, though.

Nico

[T***@gmx.com]

Post by Nico Huber
Hi Pete,

As the default SATA setting already appeared correct, I modified the 3
additional settings that Taiidan had already indicated worked
(memory_speed_boost, 1394 controller and SATA ALPM) so in my mind I was
only adjusting one additional setting, the memory voltage. I thought
this was fairly sensible as my RAM is definitely 1.35V.

did you have CONFIG_USE_OPTION_TABLE before? If not you potentially have
changed all settings. The defaults in the cmos.default file don't have
to be the same defaults that are hardcoded in the code and the file only
takes effect if CONFIG_USE_OPTION_TABLE is set. As I looked at the SATA
code, I know the AHCI setting is one that definitely differs.

I would also use nvramcui to verify that settings have actually been
changed anyways.

[Nico Huber]

I have attached a text file with an overview of all the options I have
selected with the nconfig utility, on the off-chance someone spots
something I have done wrong.

Best way to find out is to disable USE_OPTION_TABLE again and leave
everything else as is. If that doesn't result in a reliable image,
the option table is not to blame. Actually, we try to discourage
developers from adding options that may result in a failure, but
you never know.

You can use `make savedefconfig` to save the options that differ from
their default. It writes into a file `defconfig`. That's probably better
than your "screenshots" (unless the D16 requires non-default settings to
work).

Regarding the cmos.default, IMHO, somebody should go through all the
options (from `interleave_chip_selects` to `maximum_p_state_limit` in
cmos.layout), check where they are used in the code (e.g. `git grep
\"interleave_chip_selects\"`) and document their defaults in absence
of the option table (i.e. when get_option() returns a failure or
doesn't do anything at all). Then we'd be able to discuss sane defaults.
Though, we'd need to take all boards using the respective code paths
into account.

Nico

[T***@gmx.com]

Please keep replies on the list - I will notice them - and like I said I
won't give up until it works :D

I wish you had OpenBMC installed so that I could access your system
remotely and try stuff myself (via VPN and a router IP whitelist for
security ofc)

Did your D16 come with an ASMB4 or ASMB5 module? if so I recommend
installing OpenBMC and its coreboot patches - note that coreboot needs
patching for them to play nice together since for some stupid reason the
required patches have not been upstreamed so you would need to cherry-pick.

I am using v4.6 on my system FYI (no reason for me to update) and the
only options I have changed are the ones I told you about before...all I
can figure is perhaps I have a different version of the SP5100 that
doesn't have the erratum or something to that effect.

FYI I saw you select build nvramcui as a secondary payload so you can
mess with cmos options without recompiling as long as next time you shut
off "Load default configuration values into CMOS on each boot"

Since your menuconfig options look fine...shot in the dark but try using
grub instead of SeaBIOS as I use grub.

[T***@gmx.com]

Hi Taiidan,
Thanks for your message.

Post by T***@gmx.com
I am using v4.6 on my system FYI (no reason for me to update) and the
only options I have changed are the ones I told you about before...all I
can figure is perhaps I have a different version of the SP5100 that
doesn't have the erratum or something to that effect.

Is it possible for me to build v4.6 in case something has changed since then that causes my issues? When I look at the branches available on git there is no 4.6 available.

Get it from the coreboot website + check signature.

Post by T***@gmx.com
FYI I saw you select build nvramcui as a secondary payload so you can
mess with cmos options without recompiling as long as next time you shut
off "Load default configuration values into CMOS on each boot"
Since your menuconfig options look fine...shot in the dark but try using
grub instead of SeaBIOS as I use grub.

I will try building it with Grub. There is an option to include a GRUB2 runtime config file into ROM image. Should I select this? If so, what options should be in this file?

I would set it to load your disk .cfg file otherwise you would have to
do so via configfile (ahci0,msdos1)/grub2/grub.cfg or what not when it
boots to the rescue shell.

Kind regards,
Pete

[Peter Stuge]

Well the good news is that I have now got this board to work with
Coreboot v4.6 with the CMOS options and SeaBIOS! :-)
The bad news is that this obviously means a bug has crept in on the
way up to 4.8

The best news is that you can track that down, since you have both a
working (good) and a failing (bad) test case.

Please clone the git repo and use git bisect to find the first bad commit.

This would clearly be a really valuable contribution to the project,
because it is not so unlikely that more boards are affected.


Thanks

//Peter

[Nico Huber]

Post by Nico Huber
No idea why combined mode is the default, it's only useful for OSes from
the '90s. It's not about the type of drives (SATA vs PATA) connected but
how the SATA controller identifies itself to the OS.>

I agree that the combined mode isn't the best default, but I can't say
that it's totally useless. In combined mode you have the first 4(?) SATA
ports connected to an AHCI controller and the last 2(?) to a legacy
IDE/ATA controller. I had some SATA->PATA bridge to connect a PATA
device to a mainboard and the adapter I had only worked on SATA ports
that were connected to a legacy IDE/ATA controller, but not on SATA
ports connected to an AHCI controller. I didn't further debug that issue
though, so I can't say why it didn't work on an AHCI controller.

Interesting use case. I missed that the combined mode is also the only
mode where you have SATA ports on AHCI and the legacy interface.

I also don't know if an IDE drive should work with AHCI. My guess would
be yes, but OS drivers probably don't expect it.

Nico

[Timothy Pearson]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Hi!

Post by Nico Huber
No idea why combined mode is the default, it's only useful for OSes from
the '90s. It's not about the type of drives (SATA vs PATA) connected but
how the SATA controller identifies itself to the OS.

I agree that the combined mode isn't the best default, but I can't say
that it's totally useless. In combined mode you have the first 4(?) SATA
ports connected to an AHCI controller and the last 2(?) to a legacy
IDE/ATA controller. I had some SATA->PATA bridge to connect a PATA
device to a mainboard and the adapter I had only worked on SATA ports
that were connected to a legacy IDE/ATA controller, but not on SATA
ports connected to an AHCI controller. I didn't further debug that issue
though, so I can't say why it didn't work on an AHCI controller.
Speaking of the KGPE-D16: I still have this patch [0] in my review queue
that is needed for coreboot to work on the board when the BMC firmware
flash is installed, but it seems that neither Timothy nor owners of that
board really care about that patch any more (I poked a few people about
that patch a few times), which I find quite sad. Would be good if
someone would address the last 3 comments and actually tests the patch
on the real hardware, so it can be merged to fix that show-stopper.
Regards
Felix

First, thanks for the patch and work on the KGPE-D16. I've been meaning
to do a better job at helping maintain the platform, but right now life
has gotten somewhat in the way in that I haven't had proper access to
the KGPE-D16 development system in quite a while (space taken up with
Talos / Blackbird boards for the most part).

Things should settle down some after Christmas, so I'll see what I can
do to pull the old D16 dev platform back out at that time and start
testing / merging patches. Are there any others that I should also help
take a look at?

- --
Timothy Pearson
Raptor Engineering
[phone] (direct line)
[phone] (switchboard)
https://www.raptorengineering.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iQEcBAEBCAAGBQJcC0PGAAoJEK+E3vEXDOFb2pkIAIhqutZr52Z591tjkMzTdCWf
HCNrJdcyg6bI4t7GAIT59F8EYVeVsZq8guc5+Sttp36ojpnLbnG7LtdSThsEzlhL
/uIRF5Fx1joi1eE9r79zQ0gl7RGaKvjzgqgvgLMXVgepMP/H03cRFc4xqyx1iuDY
4OimTJCR82Fl7aG/SnS7snYDO7DNRtOvpv4KIO6AONBKtSVcDElk/aBOW864yVgF
WEkQHq26Mg32zO6rN21MBuGFBhNsWgQ+MmFrltlmr/KqOnL4mRlKDs7QbuaXIhMl
1DcdXRz6sqXVK3GU2JgaSTca6RbK1G6dpvGUFca+mQYU7E3IH0x1JkPfbq9CV50=
=CjaW
-----END PGP SIGNATURE-----

[T***@gmx.com]

Maybe try installing a SATA drive to test things?

Worse case maybe disabling the SATA controller would fix it? as you say
you don't need it.

I hope timothy pearson sees this and replies as he would know what to do
to fix it.