I have never used an ESP but as far as I can tell, you are right. There is no component involved for the 5V. Some boards have a diode and a fuse between VBUS (5V USB) and the 5V rail on the board, but that is not the case for the board in your picture.
The black component with “512” is just a pull-down resistor for one of the CC lines of the USB connector. This should not get particularly warm.
The brownish thing is the input capacitor of the voltage regulator. It can get a little warm due to ripple currents, but I wouldn’t expect a lot of heat.
Are you sure it is not the black 5-pin voltage regulator for the ESP? As far as I know those ESPs can be a bit power hungry depending on what you have enabled (e.g. WiFi).
Maybe you can check the temperature without your LED matrix connected and see if it is hot as well.
The voltage regulator is only used to generate the 3.3V supply for the ESP. But OP is using the 5V from the input of the regulator to power the LEDs.
At max it is wasting about 0.7W if the 400mA max current is right. But it will still get hot due to the current draw of the ESP. Even at half of the max. rated current this device is probably 30-40 degrees above your ambient temperature.