RX5700XT investigation

bargo

Hello folks,

I hope you are doing well. I apologies for the English language post, but at least its a way we can communicate :), I have a couple of these red devils rx 5700XT one comes with two 8 pins and the other 6+8 pins,

The one with two 8s, has this well known issue of b2 symptoms where all voltage rails are well reported and the card is completely alive Except that it does not work.

I have managed to get to the state where I can get information from the GPU, and I believe there is some kind of security feature that this card can block all memory access and just crash if memory is accessed, which causes all the restarts and hanging with qcode 8 or another..

Mainly I'm using the "atitool" binary, which was/is available on github somewhere.

The below screen shots are from this card, which shows the security preventing memory settings, in any way or form, due to the security feature, I was not able to see where this security feature is coming from, and why its enabled on this card, I have never had a card with this feature being enabled.

I think the card has an issue with one memory chip, as it had a hard physical shock in the past, which may have caused a memory ball to go loose, but also there could nothing other than this security feature, disabling memory access..

The other 6+8, is a little bit more interesting story and got me on my feet for quite sometime already, I had a short on the gate of Q900 (lg) and had to remove both the Q900 and the U901, but then i also had a problem on U902 and its mosfet, I replaced both but I had no extra GS7260 and I thought I would use a NCP5230 as a replacement (apparently this is not working well) after running the card I had only 0.2 volts on that rail, without a PGOOD signal. Which makes me think that the NCP5230 has a different R1/R2 equation than the GS7260, so now waiting for another GS7260 to arrive to see if that was it or not :)

I will go back to the two 8 pins now, and see if i can get to know anything more about this security lock..

All my best,

Update for the two 8s: it seems the memory is currently clocking at 500Mhz, (screen shot attached)

bargo

ok some progress i guess, I managed to boot with the card, apparently the hanging/restarts symptoms/behavior is due to a stuck for loop in the amdgpu driver, the reason is mainly stuck looping some specific area as it seems to me, i hard coded it to a specific value and now it does not stuck, but also the bin/ips binary is not valid as I dont really know which ball is responsible for which bit at this stage:

Some debugging text was added to the module, but its very helpful:

Цитата:

[ 0.614669] [drm] amdgpu kernel modesetting enabled.
[ 0.614700] amdgpu: CRAT table disabled by module option
[ 0.614703] amdgpu: Virtual CRAT table created for CPU
[ 0.614711] amdgpu: Topology: Add CPU node
[ 0.614791] amdgpu 0000:0c:00.0: enabling device (0000 -> 0003)
[ 0.614822] [drm] initializing kernel modesetting (NAVI10 0x1002:0x731F 0x148C:0x2398 0xC1).
[ 0.614827] [drm] register mmio base: 0xFC800000
[ 0.614828] [drm] register mmio size: 524288
[ 0.614833] [drm] register mmio base: 0xFC800000
[ 0.614835] [drm] register mmio size: 524288
[ 0.614840] [drm] after device_Get pcie_ we are good
[ 0.614869] [drm] after device reset we are good
[ 0.614873] [drm] after hw virtualization detect we are good
[ 0.614874] [drm] after get job timeout
[ 0.614876] [drm] after enable virtual display
[ 0.614876] [drm] after request full gpu
[ 0.614877] [drm] going to default
[ 0.614879] [drm] going to have some defaults
[ 0.614881] [drm] reading binary from vram
[ 0.614883] [drm] reading binary from vram at pos: 8573091840 and size 8573157376
[ 0.614887] [drm] Inside for loop 3
[ 0.614889] [drm] Inside for loop 3
[ 0.614891] [drm] Inside for loop 3
[ 0.614894] [drm] Inside for loop 3
[ 0.614896] [drm] Inside for loop 3
[ 0.614898] [drm] Inside for loop 3
[ 0.614901] [drm] Inside for loop 3
[ 0.614903] [drm] Inside for loop 3
[ 0.614905] [drm] Inside for loop 3
[ 0.614907] [drm] Inside for loop 3
[ 0.614910] [drm] Inside for loop 3
[ 0.614912] [drm] Inside for loop 3
[ 0.614914] [drm] Inside for loop 3
[ 0.614917] [drm] Inside for loop 3
[ 0.614919] [drm] Inside for loop 3
[ 0.614921] [drm] Inside for loop 3
[ 0.614924] [drm] after reading binary from vram
[ 0.614927] amdgpu 0000:0c:00.0: amdgpu: invalid ip discovery binary checksum
[ 0.614930] [drm] Going to out after failing to read binary from vram and maybe file
[ 0.614932] [drm:amdgpu_discovery_set_ip_blocks.cold] *ERROR* amdgpu_discovery_init failed
[ 0.614937] [drm] reg base init done
[ 0.614937] [drm] finished from default
[ 0.614939] amdgpu 0000:0c:00.0: amdgpu: Fatal error during GPU init
[ 0.614943] amdgpu 0000:0c:00.0: amdgpu: amdgpu: finishing device.
[ 0.614973] amdgpu: probe of 0000:0c:00.0 failed with error -22

Posted after 1 hour 50 minutes 52 seconds:
ok, this is a little bit of what I understood so far, maybe someone can help or shed some more light, or maybe this could help someone else,

There is a binary file which seems to be a firmware that contains a list of IPs which are basically features for the running ASIC, i don't know if this assumption is correct but it looks like that, then this binary is located somewhere in the vram memory, and it seems that its located at the end of the 8Gb address space, mainly in the last 0.5Mb (according to my logging [drm] reading binary from vram at pos: 8573091840 and size 8573157376) but I also don't know if this is also correct, but this is how I see it at the moment.

bargo

that is it

bargo

Ok, its getting a bit more interesting with the two 8s card.. All the memory banks are not accessible, I could not read anything from them. Or at least I tried with what supposed to be the first memory space and the last, but with the same behavior.

I then started looking again at voltages, all voltages are present but for some reason I noticed a strange behavior on the EXTA 12v rail, it starts at 12.12, then dropping to 11.80, then back to 12.12, then stays still at at 11.80, other lower rails are not affected, but that tells me something is not working correctly, on the other hand the EXTB 12v rail, stays at 12, never below.

Did anyone experience something similar before? or after? :)

uncledim

waheed.barghouthi@gmail.com писал(а):

a strange behavior on the EXTA 12v rail

have you tried to swap A&B "tails" from PSU? it may be just poor contact
or check schematics, what's load on A vs B? they may differs significally

bargo

uncledim писал(а):

have you tried to swap A&B "tails" from PSU? it may be just poor contact

Thank you for the input, different power supplies results the same behavior and swapping A/B inputs keeps the behavior the same.

uncledim писал(а):

or check schematics, what's load on A vs B? they may differs significally

Yes, the load is indeed different on A, the memory load is added, which tells me it could be either U702/703 issue, but also it could be a bad bootstrap capacitor.

Update:
the EXTA 12v rail is ok, apparently the raiser 12V rail is not powerful enough, so setting it on the mainboard resulted in 12v staying always at 12v

uncledim

waheed.barghouthi@gmail.com писал(а):

the memory load is added, which tells me it could be either U702/703 issue

memory load ~= 45% memory interface + 45% gpu-memory interface +10% memory itself)
since "free" tserver can test good cards only - sometimes "IR-look" can help. check if some of mem chips hot vs others.

bargo

uncledim писал(а):

memory load ~= 45% memory interface + 45% gpu-memory interface +10% memory itself)

Thank you for the detailed information, this is quite possible numbers.

uncledim писал(а):

since "free" tserver can test good cards only - sometimes "IR-look" can help. check if some of mem chips hot vs others.

Yes I will go and check it with IR, I was stuck with another card and didn't get the chance to have a look with IR, i will do that soon and report,

Thanks again,

bargo

I think these IR pictures goes well with my theory that U2700 is causing some problems, but I still can not confirm, I will need to take it out and maybe reball or replace..

The U2700 is always showing heat spot first, but the difference is only milliseconds and the others start to glow as well. The only difference heat starts in a very specific location on this chip and then it spreads and not like the others.

Pandemonium6

If it is hotter you have to replace. Reballing will not help and rather don't waste a time for reballing a hot memory.

bargo

Thank you, yes I think you are correct. I will replace it.. Once I get my main motherboard working again :D

Pandemonium6 писал(а):

If it is hotter you have to replace. Reballing will not help and rather don't waste a time for reballing a hot memory.

DoktorVGA

Pandemonium6 писал(а):

If it is hotter you have to replace. Reballing will not help and rather don't waste a time for reballing a hot memory.

I did this many times not helped and turned out that the GPU was faulty.
As i remember when a VRAM module heats up on different points just look under it, you will find silicon that turned out partly conductive oil and thats produces the heat you can see on the thermal camera.
So if you have an ultrasonic cleaner you can check again with thermal camera after a deep clean and you can see the heat is gone.
Have also used the "hot" chips on an RTX 20xx card and does not have any problems even in 0-24 usage.

bargo

DoktorVGA писал(а):

I did this many times not helped and turned out that the GPU was faulty.
As i remember when a VRAM module heats up on different points just look under it, you will find silicon that turned out partly conductive oil and that produces the heat you can see on the thermal camera.
So if you have an ultrasonic cleaner you can check again with thermal camera after a deep clean and you can see the heat is gone.
Have also used the "hot" chips on an RTX 20xx card and does not have any problems even in 0-24 usage.

Thank you for your feedback, I actually would suspect the GPU is not faulty and there is a high chance only the memory chip is, the card itself is a new card never been used, it was opened by the owner after delivery to find out that it was damaged during shipping, it took a hit and its near that memory chip, nothing is damaged from the look of it, but if you look at the thermal camera, you notice the heat starting to spread from a specific location on that memory chip.

but I actually since I installed this card until this moment, I'm trying to fix:
1- The motherboard that was being the test board for this card (the board decided to die at the moment when I was doing some I2C level stuff with one graphic card)
2- The heat gun machine got shorted due to some factor, and I decided to recreate the whole heat station.

I have almost finished the heat station right now with a small esp32 microcontroller, and its connected to my browser with two big digits, one for the hot air temperature and the other for the hotplate and I can set the desired temps with filling a text box like "hotair,340" or "hotplate,200", so you can say I have been barely progressing with fixing anything except some other motherboards so I can keep going. but it turned out to be for the better, I thank the Lord for what I have got either way.

ll_r1d0_ll

I have the same Red Devil 5700xt stuck on ERROR CODE B2 while its 5V fuse was short on the top Corner's Memory mosfet's Driver VCC. Removed the cap and short was gone. Also checked the resistance of the memory line. All seemed normal to me. Tried to boot it but unfortunately stuck on B2 .. Tried to run TSERVER on it but it just hangs . I am out of clues now

bargo

ll_r1d0_ll писал(а):

Joined: 27 May 2022, 16:08
Cash on hand:
1.01
[Donate]
Posts: 21
Location: Pakistanrepublic
Has thanked: 10 times
Have thanks: 1 time
I have the same Red Devil 5700xt stuck on ERROR CODE B2 while its 5V fuse was short on the top Corner's Memory mosfet's Driver VCC. Removed the cap and short was gone. Also checked the resistance of the memory line. All seemed normal to me. Tried to boot it but unfortunately stuck on B2 .. Tried to run TSERVER on it but it just hangs . I am out of clues now

Do you see the card listed in

Цитата:

lspci

maybe you can try to test the memory with something like dmgg, or write to the memory space and see if you have success.. b2 post code cards cant be tested with tserver, as the GPU itself has failed to init successfully so it can be a specific memory chip, or GPU balls or maybe something much more smaller inside the die itself, of course if any voltage rail is missing B2 will also be present. but I'm assuming you have a similar case to mine, and all voltages are present and correct.. also bios problems are b2 as the card will fail to init with the wrong bios. as putting a banana in an apple shoe :)

I hope I said something that may help..

ll_r1d0_ll

bargo писал(а):

ll_r1d0_ll wrote:
Joined: 27 May 2022, 16:08
Cash on hand:
1.01
[Donate]
Posts: 21
Location: Pakistanrepublic
Has thanked: 10 times
Have thanks: 1 time
I have the same Red Devil 5700xt stuck on ERROR CODE B2 while its 5V fuse was short on the top Corner's Memory mosfet's Driver VCC. Removed the cap and short was gone. Also checked the resistance of the memory line. All seemed normal to me. Tried to boot it but unfortunately stuck on B2 .. Tried to run TSERVER on it but it just hangs . I am out of clues now

Do you see the card listed in
Quote:
lspci
maybe you can try to test the memory with something like dmgg, or write to the memory space and see if you have success.. b2 post code cards cant be tested with tserver, as the GPU itself has failed to init successfully so it can be a specific memory chip, or GPU balls or maybe something much more smaller inside the die itself, of course if any voltage rail is missing B2 will also be present. but I'm assuming you have a similar case to mine, and all voltages are present and correct.. also bios problems are b2 as the card will fail to init with the wrong bios. as putting a banana in an apple shoe :)

I hope I said something that may help..

Yes I can see the card in LSPCI. I have checked each and every voltage. Every voltage is present. I had tried to flash the bios as well. I have tried to switch the bios as there is switch available because I cannot assume that both bios chips can get faulty at the same time :D . Anyways, I did not try with DMGG but with TSERVER, But TSERVER hangs still. Dont know what else to do

bargo

ll_r1d0_ll писал(а):

. I have tried to switch the bios as there is switch available because I cannot assume that both bios chips can get faulty at the same time :D . Anyways, I did not try with DMGG but with TSERVER, But TSERVER hangs still. Dont know what else to do

:) I think I would not assume they are faulty neither, I think its either a missing ball contact, under a memory chip, or under the main die. I would try feeding the memory rail from an outside voltage, say give it 1.2V and tune it up to 1.6 and try to notice if there is any memory that is heating up more than the others, (the heat difference can be very small, sometimes 1C degree only. So you need to really look carefully for it) , if nothing heats up at all, then its a missing contact, either a missing ball or a missing resistor not giving a specific signal, for example, a resistor for a ALL_RAILS_UP which indicates to the GPU that everything seems to be ok, from a voltage standing point of course..

Raisca

1. Проверьте еще раз формирование всех напряжений (в идеале осциллографом)
2. Сделайте замеры снятых GPU.(других данных нет у меня) По этим данным можно косвенно оценить что GPU исправный или нет

bargo

Raisca писал(а):

Сделайте замеры снятых GPU.(других данных нет у меня) По этим данным можно косвенно оценить что GPU исправный или нет

Спасибо, это очень полезно.. (Thank you, this is very helpful)

uncledim

uncledim писал(а):

since "free" tserver can test good cards only

Oops, not "only good") Try this:
./agt -mctraininfo
(helped me to find "bad guy" - poor ball contact at "C" channel on RX5600)

bargo

uncledim писал(а):

./agt -mctraininfo

Thanks for the information, much appreciated. I actually just tried this one, It showed all channels trained successfully, I don't know if this is actually doing anything, but it could be helpful in some cases, in this case returning successfully didn't indicate any failure..

When ever I try to write anything to the memory space/address, at any offset, the card instantly hangs and drive the other cards connected to the same board to a panic call,

interestingly the reset_n pin goes lower than the usual 1.45v or vddc +- 2% hitting to 1.30v at the hang moment on all memory channels, which seems the GPU is not liking how the memories are reacting to its calls, i just assumed this information as per the datasheet of gddr6:

Сообщение не в тему

Reset: RESET_n low asynchronously initiates a full chip reset. With RESET_n LOW all
ODTs are disabled. A full chip reset may be performed at any time by pulling RE-
SET_n LOW.

, this might still indicate a memory problem, I didnt have enough time to actual replace the memory chip, I must try that as a next step though.

uncledim

bargo
Have you ever tried to run with "unlocked" (with latest beta RBE) vbios? (I haven't such a case, but who knows it may help)

bargo

uncledim писал(а):

with latest beta RBE

Thanks uncledim. I have not tried that, but I think it may be worth it. At least for adjusting memory clock and seeing if anything differs

bargo

bargo писал(а):

interestingly the reset_n pin goes lower than the usual 1.45v or vddc +- 2% hitting to 1.30v at the hang moment on all memory channels, which seems the GPU is not liking how the memories are reacting to its calls, i just assumed this information as per the datasheet of gddr6:

I think the reset_n is not going to the low threshold it seems low means 0.2v and less for this pin and what I'm seeing is not that, its something else going on, but reset is reacting to it which only comes from the gpu as a signal to the memories to reset.

Posted after 3 seconds:

bargo писал(а):

interestingly the reset_n pin goes lower than the usual 1.45v or vddc +- 2% hitting to 1.30v at the hang moment on all memory channels, which seems the GPU is not liking how the memories are reacting to its calls, i just assumed this information as per the datasheet of gddr6:

I think the reset_n is not going to the low threshold it seems low means 0.2v and less for this pin and what I'm seeing is not that, its something else going on, but reset is reacting to it which only comes from the gpu as a signal to the memories to reset.

bargo

bargo писал(а):

The other 6+8, is a little bit more interesting story and got me on my feet for quite sometime already, I had a short on the gate of Q900 (lg) and had to remove both the Q900 and the U901, but then i also had a problem on U902 and its mosfet, I replaced both but I had no extra GS7260 and I thought I would use a NCP5230 as a replacement (apparently this is not working well) after running the card I had only 0.2 volts on that rail, without a PGOOD signal. Which makes me think that the NCP5230 has a different R1/R2 equation than the GS7260, so now waiting for another GS7260 to arrive to see if that was it or not :)

Ok I finally got this card running again, This is not the b2 problem card its the one with 8+6 pins, which had a problem on both 1.8v and 0.75v rails. I thank the Lord of skies and earth for his gifts, this was needed..

I'm of course using the card to post this message :D

All my best to all,

uncledim

Great!

bargo писал(а):

rx5700_2.gif

It seems NAVI can't drop pcie mode to 1.1 when idle with actual drivers..

Pandemonium6

bargo писал(а):

which had a problem on both 1.8v and 0.75v rails

What exact problems did you have with both voltage rails?

bargo

Pandemonium6 писал(а):

What exact problems did you have with both voltage rails?

I was missing the enable signal for the 1.8v and 0.75 was getting enabled (running at 0.5v though) while it should be dependent on the 1.8v pwr_good signal. I had to replace both the GS7260 and their mosfets.

---

Also an update for the second card which is the XT version, which had the B2 issue, I currently have removed the u2700 memory chip, I still get the same results when booting with the card, I still don't have a dead card with a similar working memory chip, I thought I will be able to do something with it without the memory chip, but I actually don't know if this chip is required for the GPU to initialize correctly or not, once I have a replacement I will plant a chip there and test it again.

All my best,

bargo

bargo писал(а):

Also an update for the second card which is the XT version, which had the B2 issue, I currently have removed the u2700 memory chip, I still get the same results when booting with the card, I still don't have a dead card with a similar working memory chip, I thought I will be able to do something with it without the memory chip, but I actually don't know if this chip is required for the GPU to initialize correctly or not, once I have a replacement I will plant a chip there and test it again.

Just sharing maybe it helps someone else, or someone else help!

I have tested the memorytraining information for this rx5700xt 220W version, and did that with memtune and with agt -mctraininfo I got all success on all banks A until H. I still don't understand how its succeeding to train H0/1 if its not present. But apparently not present means successful training, but a faulty present means unsuccessful, I have no idea yet..

RX5700XT investigation

Кто сейчас на конференции