User avatar
PeterO
Posts: 4942
Joined: Sun Jul 22, 2012 4:14 pm

GPU stats (free mem etc)

Sun Nov 18, 2012 5:42 pm

I've had a look around but I can't see any tool(s) that show how much of its allocated RAM the GPU is actually using. I've just spent some time trying to fathom out why my code to draw into a frame buffer wasn't working. When I finally found the cause it turns out the GPU was running out of memory.

PeterO
Discoverer of the PI2 XENON DEATH FLASH!
Interests: C,Python,PIC,Electronics,Ham Radio (G0DZB),1960s British Computers.
"The primary requirement (as we've always seen in your examples) is that the code is readable. " Dougie Lawson

dom
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5318
Joined: Wed Aug 17, 2011 7:41 pm
Location: Cambridge

Re: GPU stats (free mem etc)

Sun Nov 18, 2012 5:45 pm

Code: Select all

sudo vcdbg reloc
is the main one.

User avatar
PeterO
Posts: 4942
Joined: Sun Jul 22, 2012 4:14 pm

Re: GPU stats (free mem etc)

Sun Nov 18, 2012 6:12 pm

Thanks dom.

I've increased GPU RAM by 32M (64M->96M) and it is telling me it has 32M free !

With my openGLES application running I see these "big" allocations:
[ 22] 0x1b882080: used 7.9M (refcount 3 lock count 9, size 8294400, d1Rua) '0x1a127b00'
[ 54] 0x1c06c080: used 7.9M (refcount 2 lock count 8, size 8294400, d1Rua) '0x1a127b00'
[ 55] 0x1c856080: used 7.9M (refcount 2 lock count 1, size 8294400, d1Rua) '0x1a127b00'
[ 50] 0x1d086c00: used 4.0M (refcount 1 lock count 0, size 4177920, d1Rua) '0x1a127b00'
[ 26] 0x1d483c00: used 8.0M (refcount 1 lock count 0, size 8355840, d1Rua) '0x1a127b00'

My screen is 1920 x 1080, so it works out at

2 x 7.9M = 2 32bit/pixel on screen Frame buffers
1 x 7.9M = 2 16bit/pixel on screen Depth buffers

1 x 4.0M = 1 16bit/pixel off screen Depth buffer
1x 8.0M = 1 32bit/pixel off screen Rendering buffer.

Is that a correct analysis ? (not that i t really matters, I'm just curious).

PeterO
Discoverer of the PI2 XENON DEATH FLASH!
Interests: C,Python,PIC,Electronics,Ham Radio (G0DZB),1960s British Computers.
"The primary requirement (as we've always seen in your examples) is that the code is readable. " Dougie Lawson

dom
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5318
Joined: Wed Aug 17, 2011 7:41 pm
Location: Cambridge

Re: GPU stats (free mem etc)

Sun Nov 18, 2012 6:49 pm

By default the GL framebuffer is triple buffered. I believe you can force double buffer with environment variable V3D_DOUBLE_BUFFER=1, but it hurts the framerate.

Santa77
Posts: 32
Joined: Wed Oct 31, 2012 2:08 pm
Location: Slovakia

Re: GPU stats (free mem etc)

Thu Dec 06, 2012 7:31 am

Hi dom,
memory allocations are clear for me now. But is there any way to get somewhere an GPU load? I need to detect somehow, which currently used OMX component consumes which percentual power of GPU.

I want to implement some functionality to my code (transcoding, image manipulating), that will tell to user something like:

"Hi dude, u can not use that configuration, because load of GPU will be high, and that configuration will not be as good as u expect..."

Santa77
Posts: 32
Joined: Wed Oct 31, 2012 2:08 pm
Location: Slovakia

Re: GPU stats (free mem etc)

Thu Dec 06, 2012 9:59 am

Small shell code to get one number, representing size of all allocated buffers by GPU in bytes

Code: Select all

sudo vcdbg reloc |grep "\[" | awk '{split($0,a," "); print a[12]}' | awk '{split($0,a,","); print a[1]}' | awk '{sum+=$1}END{print sum}'

dom
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5318
Joined: Wed Aug 17, 2011 7:41 pm
Location: Cambridge

Re: GPU stats (free mem etc)

Thu Dec 06, 2012 10:52 am

Code: Select all

sudo vcdbg hist
gnuplot task.gpt
will show you how busy the GPU is. (Usually not very...)

User avatar
PeterO
Posts: 4942
Joined: Sun Jul 22, 2012 4:14 pm

Re: GPU stats (free mem etc)

Thu Dec 06, 2012 12:36 pm

I shall have to try that tonight. Thanks Dom.
PeterO
Discoverer of the PI2 XENON DEATH FLASH!
Interests: C,Python,PIC,Electronics,Ham Radio (G0DZB),1960s British Computers.
"The primary requirement (as we've always seen in your examples) is that the code is readable. " Dougie Lawson

User avatar
PeterO
Posts: 4942
Joined: Sun Jul 22, 2012 4:14 pm

Re: GPU stats (free mem etc)

Fri Dec 07, 2012 7:03 pm

dom wrote:

Code: Select all

sudo vcdbg hist
gnuplot task.gpt
will show you how busy the GPU is. (Usually not very...)
Does it ? ;)
Is there a description anywhere that tells how to interpret all the different graphs ?
PeterO
Discoverer of the PI2 XENON DEATH FLASH!
Interests: C,Python,PIC,Electronics,Ham Radio (G0DZB),1960s British Computers.
"The primary requirement (as we've always seen in your examples) is that the code is readable. " Dougie Lawson

Santa77
Posts: 32
Joined: Wed Oct 31, 2012 2:08 pm
Location: Slovakia

Re: GPU stats (free mem etc)

Fri Dec 07, 2012 7:24 pm

PeterO,
yes, it's partialy working.

Generated template for gnuplot in is "hardcoded" in vcdbg to be used by gnuplot to show results within an X enviroment.

Better idea for author of vcdbg is to export real data to some more open and more parsable format like XML or so, that is not dependand on gnuplot and could be interpreted by any tool that is able to process some opened format like XML is. Of course with proper description of format.

I was trying to modify an task.gpt to produce an PNG files but:
1) every generated task.gpt is different because DAT file contains different structure based on components that are initialised and udes by GPU.
2) there should be some output that will produce some simple data like "load%" and "idle%". Mostly (me) do not care which component or sub-process in GPU consume an "power" of it.... i need to graph it by my own way, for example with RRD same way like i can do with CPU. something like actual data:

GPU 14% components, 8% render, 78% iddle
GPUMEM all:256MB, used: 128MB, free: 128MB

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 23392
Joined: Sat Jul 30, 2011 7:41 pm

Re: GPU stats (free mem etc)

Fri Dec 07, 2012 7:29 pm

PeterO wrote:
dom wrote:

Code: Select all

sudo vcdbg hist
gnuplot task.gpt
will show you how busy the GPU is. (Usually not very...)
Does it ? ;)
Is there a description anywhere that tells how to interpret all the different graphs ?
PeterO
Nope! It basically a graph of all the threads running on the GPU's two vector units (first graph = unit 1, 2nd =2 unit, third = combined). So you find out whether a thread is using a lot of CPU time, and usually the thread has a name that gives some idea of what it's doing. These graphs are very useful when debugging and optimising the GPU code. The threads are on the Y axis, time on the X axis. They don't cover what the HW blocks or the 3D CPU's are doing.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
"My grief counseller just died, luckily, he was so good, I didn't care."

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 23392
Joined: Sat Jul 30, 2011 7:41 pm

Re: GPU stats (free mem etc)

Fri Dec 07, 2012 7:35 pm

Santa77 wrote:PeterO,
yes, it's partialy working.

Generated template for gnuplot in is "hardcoded" in vcdbg to be used by gnuplot to show results within an X enviroment.

Better idea for author of vcdbg is to export real data to some more open and more parsable format like XML or so, that is not dependand on gnuplot and could be interpreted by any tool that is able to process some opened format like XML is. Of course with proper description of format.

I was trying to modify an task.gpt to produce an PNG files but:
1) every generated task.gpt is different because DAT file contains different structure based on components that are initialised and udes by GPU.
2) there should be some output that will produce some simple data like "load%" and "idle%". Mostly (me) do not care which component or sub-process in GPU consume an "power" of it.... i need to graph it by my own way, for example with RRD same way like i can do with CPU. something like actual data:

GPU 14% components, 8% render, 78% iddle
GPUMEM all:256MB, used: 128MB, free: 128MB
It would be difficult to get an overall GPU level of usage. It consists of dozens of HW blocks, 2 vector CPU's, a number of 3D CPU's etc. If one of those becomes saturated, then the GPU is running as fast as it can, but (for example) the vector units may NOT being running at full tilt, so you cannot just add up all the performance from each block. One measure could be overall memory bandwidth, which can sometimes be a limitation, but even that's not much use as you can max out some of the blocks without hitting a memory bandwidth issue. So as you can see, it's not like a CPU graph where you CAN say specifically how much CPU time is being use.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
"My grief counseller just died, luckily, he was so good, I didn't care."

User avatar
PeterO
Posts: 4942
Joined: Sun Jul 22, 2012 4:14 pm

Re: GPU stats (free mem etc)

Fri Dec 07, 2012 7:37 pm

jamesh wrote:These graphs are very useful when debugging and optimising the GPU code. T.
But since no one outside Broadcom is either debugging or optimising GPU code they are next to useless to everyone else.
Oh well, I guess all I can do is just keep adding more and more stuff to my program until the GPU can't manage 60fps anymore....
PeterO
Discoverer of the PI2 XENON DEATH FLASH!
Interests: C,Python,PIC,Electronics,Ham Radio (G0DZB),1960s British Computers.
"The primary requirement (as we've always seen in your examples) is that the code is readable. " Dougie Lawson

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 23392
Joined: Sat Jul 30, 2011 7:41 pm

Re: GPU stats (free mem etc)

Sat Dec 08, 2012 11:43 am

Well, you can determine if one of the VPU's is maxxed out, and tell which thread is maxing it out...
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
"My grief counseller just died, luckily, he was so good, I didn't care."

User avatar
PeterO
Posts: 4942
Joined: Sun Jul 22, 2012 4:14 pm

Re: GPU stats (free mem etc)

Sat Dec 08, 2012 12:03 pm

jamesh wrote:Well, you can determine if one of the VPU's is maxxed out, and tell which thread is maxing it out...
But from what you said above they are only one of many things that could be maxxed out, so hardly a reliable indication :-(
I'm just a little annoyed that after dom seemed to suggest this would give a useful indication it turns out that in reality it is somewhat less useful :-(
Anyway I'm currently trying to get my head around the callback mechanisms in openMAX IL so this is not too important to me at the moment.
PeterO
Discoverer of the PI2 XENON DEATH FLASH!
Interests: C,Python,PIC,Electronics,Ham Radio (G0DZB),1960s British Computers.
"The primary requirement (as we've always seen in your examples) is that the code is readable. " Dougie Lawson

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 23392
Joined: Sat Jul 30, 2011 7:41 pm

Re: GPU stats (free mem etc)

Sat Dec 08, 2012 3:10 pm

Well, the VPU's control the rest of the HW blocks - they run the 'supervisor' code that tells the HW blocks what to do, as well as doing a of of processing tasks. They are also the recipient of all the commands from the Arm(look for VCHI thread or similar). So knowing what's going on in the VPU's gives an indication of what is currently going on in the whole GPU. Doesn't give load for those parts, but certainly gives and idea where the GPU is spending a lot of time. There are a lot of internal registers in the GPU for the status of the various HW blocks, but I don't think they would be of any use without considerable knowledge of the GPU internals. TBH even the graph is of limited use without a certain level of knowledge of the GPU.

Which doesn't really help at all...
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
"My grief counseller just died, luckily, he was so good, I didn't care."

Santa77
Posts: 32
Joined: Wed Oct 31, 2012 2:08 pm
Location: Slovakia

Re: GPU stats (free mem etc)

Wed Dec 12, 2012 2:38 pm

Hello folks,

probably I have small problem there ;) It's occures sometime... My app initialize video_decoder, buffers for it, then splitter, video_encoders connected by tunnels to splitter and buffers for that encoders.
when app ends i am dissabling all buffers, deinitiate all components and application ends. but....
there are some 'ilin mapping buffer' left, that was not allocated before apps starts...

Code: Select all

[email protected]:~# vcdbg reloc
Relocatable heap version 4 found at 0xd380000
total space allocated is 44M, with 39M relocatable, 0 legacy and 4.5M offline
0 legacy blocks of size 2359296

free list at 0xdf886a0
27M free memory in 1 free block(s)
largest free block is 27M bytes

[   1] 0xd380000: used  16K (refcount 1 lock count 0, size  16384, d0ruA) 'audioplus_tmp_buf'
[   2] 0xd384020: used 679K (refcount 1 lock count 8, size 691200, d3rua) 'ARM FB'
0xd42dc20: free 544
[   4] 0xd42de40: used  544 (refcount 1 lock count 0, size    512, d0rua) 'ILCS VC buffer pool'
0xd42e060: free 5.1M
[  35] 0xd94e320: used 621K (refcount 1 lock count 0, size 635904, d1ruA) 'ilin mapping buffer'
[  36] 0xd9e9740: used  64K (refcount 1 lock count 0, size  65536, d1ruA) 'ilin mapping buffer'
0xd9f9760: free 33M
0xfac0000: corrupt entry (space 0x1000000)
small allocs not requested
Did i forgot something and that's an reason of that 2 buffers in GPU leaks?
Probably yes, because if i run that app again, it's perfectly working, but when it's ending it's hang up on next calls

Code: Select all

   tr_log_write(TR_LOG_INFO,"hw_decoder","OMX_disablebuffers");
   ilclient_disable_port_buffers(input_dec_1.video_decode, 130, NULL, NULL, NULL);
   ilclient_disable_port_buffers(input_dec_1.video_decode, 131, NULL, NULL, NULL);

dom
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5318
Joined: Wed Aug 17, 2011 7:41 pm
Location: Cambridge

Re: GPU stats (free mem etc)

Wed Dec 12, 2012 3:04 pm

Santa77 wrote: probably I have small problem there ;) It's occures sometime... My app initialize video_decoder, buffers for it, then splitter, video_encoders connected by tunnels to splitter and buffers for that encoders.
when app ends i am dissabling all buffers, deinitiate all components and application ends. but....
there are some 'ilin mapping buffer' left, that was not allocated before apps starts...
Can you run "vcgencmd cache_flush" before the vcdbg command.
vcdbg works by snooping memory. The ARM cannot see the GPU's L1 cache so can get stale data.
"vcgencmd cache_flush" helps, although doesn't guarantee correctness if allocations are in progress at the time.

"ilin mapping buffer" is created on a UseBuffer/AllocateBuffer call. Are you calling FreeBuffer?

In general GPU allocations will be released when the arm process exits, but it is sometimes possible to get the OMX components in a blocked state where that doesn't happen.

You can extract logs from GPU.
sudo vcdbg log assert
will show if any asserts have gone off (which generally indicate an unexpected code path).

For more detailed logging:
vcgencmd set_logging level=0x40
<do openmax stuff>
sudo vcdbg log msg

The openmax logging appears in the LOGGING_VMCS category. You could enable LOGGING_VMCS_VERBOSE too, but it is likely to swamp the log buffer.

Code: Select all

/* Bitfield for indicating the category and level of a logging message. */
#define LOGGING_GENERAL                   (1<<0)  /* for logging general messages */
#define LOGGING_GENERAL_VERBOSE           (2<<0)
#define LOGGING_CODECS                    (1<<2)  /* for codec messages */
#define LOGGING_CODECS_VERBOSE            (2<<2)
#define LOGGING_FILESYSTEM                (1<<4)  /* filesystem messages */
#define LOGGING_FILESYSTEM_VERBOSE        (2<<4)
#define LOGGING_VMCS                      (1<<6)  /* VMCS related messages */
#define LOGGING_VMCS_VERBOSE              (2<<6)
#define LOGGING_DISPMAN2                  (1<<8)  /* Dispman2/scalar logs */
#define LOGGING_DISPMAN2_VERBOSE          (2<<8)
#define LOGGING_GCE                       (1<<8)  /* Re-use Dispman2 for GCE logging */
#define LOGGING_GCE_VERBOSE               (2<<8)
#define LOGGING_CAMPLUS                   (1<<10) /* Camplus logs */
#define LOGGING_CAMPLUS_VERBOSE           (2<<10)
#define LOGGING_APPS                      (1<<12) /* Application logs */
#define LOGGING_APPS_VERBOSE              (2<<12)
#define LOGGING_CLOCKMAN_POWERMAN         (1<<14) /* Clockman + powerman logs */
#define LOGGING_CLOCKMAN_POWERMAN_VERBOSE (2<<14)
#define LOGGING_VCOS                      (1<<16)
#define LOGGING_VCOS_VERBOSE              (2<<16)
#define LOGGING_IMAGE_POOL                (1<<18) /* Image pool messages */
#define LOGGING_IMAGE_POOL_VERBOSE        (2<<18)
#define LOGGING_HDMI                      (1<<20) /* HDMI and HDCP messages */
#define LOGGING_HDMI_VERBOSE              (2<<20)
#define LOGGING_MINIMAL                   (1<<22) /* minimal logging for bandwidth measurement, ie all others off. */
#define LOGGING_MINIMAL_VERBOSE           (2<<22)
#define LOGGING_TUNER                     (1<<24) /* ISP Tuner logs - AGC, AWB etc */
#define LOGGING_TUNER_VERBOSE             (2<<24)
#define LOGGING_VCHI                      (1<<26) /* For all VCHI based services */
#define LOGGING_VCHI_VERBOSE              (2<<26)
#define LOGGING_FOCUS                     (1<<28) /* Focus messages */
#define LOGGING_HANDLERS                  (1<<29) /* For handler messages */
#define LOGGING_VOWIFI                    (1<<28) /* Re-use FOCUS for VOWIFI */
#define LOGGING_VOWIFI_VERBOSE            (2<<28) /* Re-use HANDLERS for VOWIFI */
// add more here
#define LOGGING_USER                      (1<<30) /* only for code under development - do not check in! */
#define LOGGING_USER_VERBOSE              (2<<30)

Santa77
Posts: 32
Joined: Wed Oct 31, 2012 2:08 pm
Location: Slovakia

Re: GPU stats (free mem etc)

Wed Dec 12, 2012 3:39 pm

dom, thank you for your help, but i do not see anything in logs

Code: Select all

[email protected]:~# vcgencmd set_logging level=0x40
level=64
[email protected]:~# vcdbg log assert
No messages available
[email protected]:~# vcdbg log msg
No messages available
even i use

Code: Select all

vcgencmd cache_flush
result with 2 left buffers is there.

I am not using direct OMX calls to alocate buffers, but i am using provided helper lib called ilclient:

Code: Select all

  printf("enabling port buffers for 200...\n");
   if (ilclient_enable_port_buffers(video_encode, 200, NULL, NULL, NULL) != 0) {
      printf("enabling port buffers for 200 failed!\n");
      exit(1);
   }

   printf("enabling port buffers for 201...\n");
   if (ilclient_enable_port_buffers(video_encode, 201, NULL, NULL, NULL) != 0) {
      printf("enabling port buffers for 201 failed!\n");
      exit(1);
   }
within ilclient_disable_port_buffers is that part of code that should free all allocated buffers for component...

Code: Select all

      while(list)
      {
         void *buf = list->pBuffer;
         OMX_BUFFERHEADERTYPE *next = list->pAppPrivate;

         error = OMX_FreeBuffer(comp->comp, portIndex, list);
         vc_assert(error == OMX_ErrorNone);

         if(ilclient_free)
            ilclient_free(private, buf);
         else
            vcos_free(buf);

         num--;
         list = next;
      }

dom
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5318
Joined: Wed Aug 17, 2011 7:41 pm
Location: Cambridge

Re: GPU stats (free mem etc)

Wed Dec 12, 2012 4:04 pm

Santa77 wrote:

Code: Select all

[email protected]:~# vcgencmd set_logging level=0x40
level=64
[email protected]:~# vcdbg log assert
No messages available
[email protected]:~# vcdbg log msg
No messages available
Did you run any openmax code between enabling the log and capturing it?

Santa77
Posts: 32
Joined: Wed Oct 31, 2012 2:08 pm
Location: Slovakia

Re: GPU stats (free mem etc)

Wed Dec 12, 2012 4:45 pm

Dear dom,
Later I found that i need to ... ;)
Meanwhile I found an error in my code. If I had more than 1 encoders initiated, i had badly coded loop for all opened encoders, I missed always last one and after ending ARM code, it was not cleared in GPU. Bug found and fixed. thank you for leading to right place to look.

Return to “Graphics programming”