User avatar
triss64738
Posts: 31
Joined: Wed Jun 16, 2021 5:13 pm
Location: masto/fedi: sys64738@hellsite.site
Contact: Website

Re: Dynamic code execution

Sat Jun 19, 2021 8:23 pm

hippy wrote:
Sat Jun 19, 2021 7:49 pm
That doesn't work because GCC adds push and pop wrappers inside that routine, around what I have in that code, stops it working. I tried to compensate for that but had no success.
You can add __attribute__((__naked__)) to stop GCC from adding these pushes and pops (aka prologues/epilogues). Though bear in mind that you will now have to do the return yourself manually, as well.

EDIT: also kilograham has a good point here, totally forgot about that .-.

pica200
Posts: 277
Joined: Tue Aug 06, 2019 10:27 am

Re: Dynamic code execution

Sat Jun 19, 2021 8:26 pm

hippy wrote:
Sat Jun 19, 2021 2:24 pm

Code: Select all

[ 16%] Building C object CMakeFiles/myasm.dir/myasm.c.obj
/tmp/ccgQGdx1.s: Assembler messages:
/tmp/ccgQGdx1.s:73: Warning: this instruction will write back the base register
/tmp/ccgQGdx1.s:150: Warning: this instruction will write back the base register
[ 17%] Linking CXX executable myasm.elf
[100%] Built target myasm
The warnings mean you are trying to write the register used as an adress.
Example: "stmfd r0!, {r0-r3}"

The assembler warns about this because "r0!" will modify this reg. This is considered undefined behavior by ARM. Also note that push and pop are aliases for stm/ldm.

See the examples at the bottom and why the bad one is bad:
https://www.keil.com/support/man/docs/a ... 906470.htm

kilograham
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 727
Joined: Fri Apr 12, 2019 11:00 am
Location: austin tx

Re: Dynamic code execution

Sat Jun 19, 2021 8:27 pm

EDIT: also kilograham has a good point here, totally forgot about that .-.
As I say, everyone always forgets the thumb bit!

(see table 293 in section 2.8.4.4.8 here https://datasheets.raspberrypi.org/rp20 ... asheet.pdf) :-)

hippy
Posts: 10239
Joined: Fri Sep 09, 2011 10:34 pm
Location: UK

Re: Dynamic code execution

Sat Jun 19, 2021 9:13 pm

Possibly - I had just discovered that tidbit from - viewtopic.php?t=300974

My 'codePtr' does have its bit set when I use 'codePtr = DummyRoutine();'.

The good news is that works, I can even alter registers if I ignore GCC getting in the way. And it also works with '__not_in_flash_func' which was what I was searching for when I discovered the Thumb bit info.

Unfortunately I can't get that to work with my code array even when forcing the Thumb bit in 'codePtr'.

Seems I'm slowly getting there but having been at if for over 12 hours solid I think I'll put it to bed and see if the Sleep Pixies work their magic overnight.

User avatar
triss64738
Posts: 31
Joined: Wed Jun 16, 2021 5:13 pm
Location: masto/fedi: sys64738@hellsite.site
Contact: Website

Re: Dynamic code execution

Sat Jun 19, 2021 9:21 pm

I guess I must be a sleep pixie then, because I just hacked something together (see this and this file) and it seems to work*: register state and machine code is received from a USB-CDC interface, and the result is printed on the stdio UART interface.

*: only tested with "adds r1, r2" (see test case) buuut that works!

EDIT: here's a photo of the UART output (read out using my RPï v1.2 B+), taken with a *really* shitty potato camera:

Image
Last edited by triss64738 on Sat Jun 19, 2021 9:28 pm, edited 1 time in total.

hippy
Posts: 10239
Joined: Fri Sep 09, 2011 10:34 pm
Location: UK

Re: Dynamic code execution

Sat Jun 19, 2021 9:25 pm

pica200 wrote:
Sat Jun 19, 2021 8:26 pm
hippy wrote:
Sat Jun 19, 2021 2:24 pm

Code: Select all

[ 16%] Building C object CMakeFiles/myasm.dir/myasm.c.obj
/tmp/ccgQGdx1.s: Assembler messages:
/tmp/ccgQGdx1.s:73: Warning: this instruction will write back the base register
/tmp/ccgQGdx1.s:150: Warning: this instruction will write back the base register
[ 17%] Linking CXX executable myasm.elf
[100%] Built target myasm
The warnings mean you are trying to write the register used as an adress.
Example: "stmfd r0!, {r0-r3}"
https://www.keil.com/support/man/docs/a ... 906470.htm
I had though of that, but why does "ldm r0, {r1-r7}" generate the warning ? And why two warnings so far apart when I only add that one "ldm" ?

I thought I would use "ldm" to do the above, do "mov r0, [r0, #0] for the last, to avoid warnings - but it wasn't to be.

User avatar
triss64738
Posts: 31
Joined: Wed Jun 16, 2021 5:13 pm
Location: masto/fedi: sys64738@hellsite.site
Contact: Website

Re: Dynamic code execution

Sat Jun 19, 2021 9:29 pm

hippy wrote:
Sat Jun 19, 2021 9:25 pm
pica200 wrote:
Sat Jun 19, 2021 8:26 pm
hippy wrote:
Sat Jun 19, 2021 2:24 pm

Code: Select all

[ 16%] Building C object CMakeFiles/myasm.dir/myasm.c.obj
/tmp/ccgQGdx1.s: Assembler messages:
/tmp/ccgQGdx1.s:73: Warning: this instruction will write back the base register
/tmp/ccgQGdx1.s:150: Warning: this instruction will write back the base register
[ 17%] Linking CXX executable myasm.elf
[100%] Built target myasm
The warnings mean you are trying to write the register used as an adress.
Example: "stmfd r0!, {r0-r3}"
https://www.keil.com/support/man/docs/a ... 906470.htm
I had though of that, but why does "ldm r0, {r1-r7}" generate the warning ? And why two warnings so far apart when I only add that one "ldm" ?

I thought I would use "ldm" to do the above, do "mov r0, [r0, #0] for the last, to avoid warnings - but it wasn't to be.
I think at least with Thumb2 on the M0+ it can only encode (some of) the ldmia/ldmdb/stmia/stmdb opcodes, and not the "regular" ldm/stm (which do exist in non-Thumb ARM) that don't modify the address register. But this is mostly an educated guess tbh.

hippy
Posts: 10239
Joined: Fri Sep 09, 2011 10:34 pm
Location: UK

Re: Dynamic code execution

Sat Jun 19, 2021 9:57 pm

Solved!

1) Forgot to add the Thumb bit, or rather never realised it would be necessary

2) When I added that, operator precedence must have been playing some part because, after tidying up my code to post it, testing before I did, it's now working.

So, Phase 2 completed. Once Phase 3 is completed, passing in a code array from MicroPython, and maybe a few tweaks to the MicroPython to C interface, I will post my code.

User avatar
triss64738
Posts: 31
Joined: Wed Jun 16, 2021 5:13 pm
Location: masto/fedi: sys64738@hellsite.site
Contact: Website

Re: Dynamic code execution

Sat Jun 19, 2021 10:11 pm

Ah nice!

(but hey did my post here go unnoticed? oh well you got it to work now)

kilograham
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 727
Joined: Fri Apr 12, 2019 11:00 am
Location: austin tx

Re: Dynamic code execution

Sat Jun 19, 2021 10:40 pm

triss64738 wrote:
Sat Jun 19, 2021 9:29 pm
hippy wrote:
Sat Jun 19, 2021 9:25 pm
I had though of that, but why does "ldm r0, {r1-r7}" generate the warning ? And why two warnings so far apart when I only add that one "ldm" ?

I thought I would use "ldm" to do the above, do "mov r0, [r0, #0] for the last, to avoid warnings - but it wasn't to be.
I think at least with Thumb2 on the M0+ it can only encode (some of) the ldmia/ldmdb/stmia/stmdb opcodes, and not the "regular" ldm/stm (which do exist in non-Thumb ARM) that don't modify the address register. But this is mostly an educated guess tbh.
From "Cortex-M0+ Devices Generic User Guide" .. quite useful for this sort of thing

Restrictions:

...

* the writeback suffix must always be used unless the instruction is an LDM where reglist also contains Rn, in which case the writeback suffix must not be used

so the assembler is telling you it has actually assembled ldm r0!, {r1-r7} which is not the same thing

hippy
Posts: 10239
Joined: Fri Sep 09, 2011 10:34 pm
Location: UK

Re: Dynamic code execution

Sun Jun 20, 2021 5:06 pm

triss64738 wrote:
Sat Jun 19, 2021 10:11 pm
but hey did my post here go unnoticed?
No, not unnoticed, but my brain was too addled at the time to absorb much more. It looks interesting and I will take a closer look later. Thanks.

The good news is I am now executing code passed in from MicroPython. Because MicroPython's 'array.array("H")' stores its data in memory as contiguous 16-bit words I don't have to faff about with creating arrays or 'malloc', can just point the function pointer to it and it works.

So that's Phase 3 done and, after a tidy-up and a review of what I have, what others might need, I will post the module and test code in the MicroPython sub-forum. Thanks to everyone who helped me get there.

hippy
Posts: 10239
Joined: Fri Sep 09, 2011 10:34 pm
Location: UK

Re: Dynamic code execution

Mon Jun 21, 2021 12:18 pm

I am stuck again ...

Code: Select all

uint32_t reg[8];

void run(void) {
  __asm volatile(
       "push {r0-r7}\n"
       "ldr  r7, =reg\n"
That sets 'r7' to the address of the 'reg[]' array. That works fine in a C Extension but doesn't when used in a Native C Module. The MicroPython build tools simply cannot handle it.

The simplest option seems to me to be pass the 'reg[]' address into the function, then I can access it via the stack pointer ...

Code: Select all

void run(uint32_t regArrayAddress) {
  __asm volatile(
       "push {r0-r7}\n"
       "ldr r7, [sp, #0xNN]\n"
       
run((uint32_t)&reg);
The problem is I am not sure how to determine the offset and I'm not entirely convinced GCC is putting the address on the stack in the first place, hasn't optimised it away. It looks to me like GCC may have moved stuff into 'run()' where it's using 'ip' which I have no idea about, but that might be something to do with the way MicroPython Native C Modules are compiled and executed. Because they are themselves dynamically loaded modules the ABI and calling convention may be different.

Any ideas on this one ?

User avatar
triss64738
Posts: 31
Joined: Wed Jun 16, 2021 5:13 pm
Location: masto/fedi: sys64738@hellsite.site
Contact: Website

Re: Dynamic code execution

Mon Jun 21, 2021 12:39 pm

hippy wrote:
Mon Jun 21, 2021 12:18 pm
I am stuck again ...

Code: Select all

uint32_t reg[8];

void run(void) {
  __asm volatile(
       "push {r0-r7}\n"
       "ldr  r7, =reg\n"
That sets 'r7' to the address of the 'reg[]' array. That works fine in a C Extension but doesn't when used in a Native C Module. The MicroPython build tools simply cannot handle it.

The simplest option seems to me to be pass the 'reg[]' address into the function, then I can access it via the stack pointer ...

Code: Select all

void run(uint32_t regArrayAddress) {
  __asm volatile(
       "push {r0-r7}\n"
       "ldr r7, [sp, #0xNN]\n"
       
run((uint32_t)&reg);
The problem is I am not sure how to determine the offset and I'm not entirely convinced GCC is putting the address on the stack in the first place, hasn't optimised it away. It looks to me like GCC may have moved stuff into 'run()' where it's using 'ip' which I have no idea about, but that might be something to do with the way MicroPython Native C Modules are compiled and executed. Because they are themselves dynamically loaded modules the ABI and calling convention may be different.

Any ideas on this one ?
You can pass C variables to inline assembly snippets as follows:

Code: Select all

void run(uint32_t array_addr) {
  uint32_t result;
  asm volatile(
    "push {r0-r7}\n"
    "mov r7, %[array]\n"
    // more code...
    "mov %[outthing], r7\n"
    : [outthing] "=r" (result) // output list
    : [array] "r" (array_addr) // input list
    : "r7", "memory" // list of destroyed registers
  );
}
The general format is: [name in asm] "[=+]storage type" (C variable name)

The "storage type" can be various things, eg "r" for integer register, "m" for a memory operand (i.e. something readable/writable by ldr/str), and a few others. The "=" prefix means it'll be written to, and "+" means it'll be both used as input and as output. See also the GCC docs for more info (if you can figure out how to decode it, it's not very well-worded or -formatted).

hippy
Posts: 10239
Joined: Fri Sep 09, 2011 10:34 pm
Location: UK

Re: Dynamic code execution

Mon Jun 21, 2021 1:35 pm

Thanks. That seems to have moved me forward.

The Native C Module ABI and calling conventions are rather different so I will have to tweak to compensate for those.

dthacher
Posts: 40
Joined: Sun Jun 06, 2021 12:07 am

Re: Dynamic code execution

Mon Jun 21, 2021 2:58 pm

This interface may be considered confusing and may be reworked. I know why you did it the way you did, however for the people that will likely use it this will be very backwards. Then again they will read the code and just rewrite it. This has started to annoy me.

hippy
Posts: 10239
Joined: Fri Sep 09, 2011 10:34 pm
Location: UK

Re: Dynamic code execution

Mon Jun 21, 2021 3:50 pm

dthacher wrote:
Mon Jun 21, 2021 2:58 pm
This interface may be considered confusing and may be reworked. I know why you did it the way you did, however for the people that will likely use it this will be very backwards. Then again they will read the code and just rewrite it.
I am not sure what you think is backwards but I am open to being enlightened. This is similar to how I have done it when compiling and creating ARM executable code on a Pi and for other devices. The challenge here is bridging from MicroPython, into C, into my own executable code. I have achieved that for my C Extension and think it's now done for my Native C Module - The problem there is in the MicroPython to Native C Module interface.

MicroPython users are my primary audience who would not likely be concerned as to how it works, would probably not look at the module code. But, given the niche domain, maybe they would. If anyone wants to rewrite things I am more than happy for them to do so.

Personally I care very little about how it works only that it does. I am more interested in compilation and ARM code generation in MicroPython than how to run that code, but having a means to run that code is fundamental. No one has done it for me, so I had to do it myself, even though not really equipped for that task.
dthacher wrote:
Mon Jun 21, 2021 2:58 pm
This has started to annoy me.
Not sure what's annoying you but I can only apologise if I am in any way to blame for that.

Anyway, pending publishing it all, this is what I currently have, MPY defined when Native C Module -

Code: Select all

uint32_t reg[8];                // Register transfer array
uint32_t codAdr;                // Address of code array

#define ASM(s) __asm volatile(                                         \
                 s                                                     \
                 ::: "r0","r1","r2","r3","r4","r5","r6","r7", "memory" \
               )

#ifndef MPY

void run() {
    ASM(
        "ldr  r7, =codAdr"      "\n"
        "ldr  r7, [r7]"         "\n"
        "mov  lr, r7"           "\n"

        "ldr  r7, =reg"         "\n"
        "ldr  r0, [r7, #0*4]"   "\n"
        "ldr  r1, [r7, #1*4]"   "\n"
        "ldr  r2, [r7, #2*4]"   "\n"
        "ldr  r3, [r7, #3*4]"   "\n"
        "ldr  r4, [r7, #4*4]"   "\n"
        "ldr  r5, [r7, #5*4]"   "\n"
        "ldr  r6, [r7, #6*4]"   "\n"
        "ldr  r7, [r7, #7*4]"   "\n"

        "blx  lr"               "\n"

        "push {r7}"             "\n"
        "ldr  r7, =reg"         "\n"
        "str  r0, [r7, #0*4]"   "\n"
        "str  r1, [r7, #1*4]"   "\n"
        "str  r2, [r7, #2*4]"   "\n"
        "str  r3, [r7, #3*4]"   "\n"
        "str  r4, [r7, #4*4]"   "\n"
        "str  r5, [r7, #5*4]"   "\n"
        "str  r6, [r7, #6*4]"   "\n"
        "pop  {r0}"             "\n"
        "str  r0, [r7, #7*4]"   "\n"
    );
}

#else

const uint32_t regAdr = (uint32_t)®

void run() {
    __asm volatile(
        "mov  lr, %[xCodAdr]"   "\n"
        :
        : [xCodAdr] "r" (codAdr)
        : "r0","r1","r2","r3","r4","r5","r6","r7", "memory"
    );
    __asm volatile(
        "mov  r7, %[xRegAdr]"   "\n"
        "ldr  r0, [r7, #0*4]"   "\n"
        "ldr  r1, [r7, #1*4]"   "\n"
        "ldr  r2, [r7, #2*4]"   "\n"
        "ldr  r3, [r7, #3*4]"   "\n"
        "ldr  r4, [r7, #4*4]"   "\n"
        "ldr  r5, [r7, #5*4]"   "\n"
        "ldr  r6, [r7, #6*4]"   "\n"
        "ldr  r7, [r7, #7*4]"   "\n"

        "blx  lr"               "\n"

        "push {r7}"             "\n"
        "mov  r7, ip"           "\n"
        "str  r0, [r7, #0*4]"   "\n"
        "str  r1, [r7, #1*4]"   "\n"
        "str  r2, [r7, #2*4]"   "\n"
        "str  r3, [r7, #3*4]"   "\n"
        "str  r4, [r7, #4*4]"   "\n"
        "str  r5, [r7, #5*4]"   "\n"
        "str  r6, [r7, #6*4]"   "\n"
        "pop  {r0}"             "\n"
        "str  r0, [r7, #7*4]"   "\n"
        :
        : [xRegAdr] "r" (regAdr)
        : "r0","r1","r2","r3","r4","r5","r6","r7", "memory"
    );
}

#endif
And this is the basics of the MicroPython interface which sets registers, calls the above, returns the registers -

Code: Select all

    for(int n=0; n < reg_list_len; n++) {
        reg[n] = mp_obj_get_int(reg_list_items[n]);
    }
    codAdr = (uint32_t)code_array.buf + (word_offset << 1) + 1;
    run();
    mp_obj_t list[] = {
        mp_obj_new_int(reg[0]),
        ...
        mp_obj_new_int(reg[7]),
    };
    return mp_obj_new_list(8, list);
Update : it's all working; as a C Extension Module and as a Native C Module. It may not be as elegant or as optimised as it could be but it works, which is good enough for me. Thanks everyone.

dthacher
Posts: 40
Joined: Sun Jun 06, 2021 12:07 am

Re: [SOLVED] Dynamic code execution

Tue Jun 22, 2021 1:57 am

I am not sure that logic is correct. This would probably be worth looking into:
https://en.wikipedia.org/wiki/Calling_c ... #ARM_(A32)

This looks like a problem:
"mov r7, ip" "\n"
Why did you not make it this?
"ldr r7, =reg" "\n"
I get that you are trying to make 8 input args and 8 output args. This function wraps your convention. This not the fastest way, but may be simpler. You could do 12 args.

hippy
Posts: 10239
Joined: Fri Sep 09, 2011 10:34 pm
Location: UK

Re: [SOLVED] Dynamic code execution

Tue Jun 22, 2021 11:47 am

dthacher wrote:
Tue Jun 22, 2021 1:57 am
I am not sure that logic is correct. This would probably be worth looking into:
https://en.wikipedia.org/wiki/Calling_c ... #ARM_(A32)
What I have created is intended to be a solution for executing the code I generate, not a universal solution for any arbitrary code, so I am not overly concerned about calling conventions used within the executed code.

If anyone wants more than what there is, they are welcome to update my code to deliver it. That should only require a change to the "run()" routine which should be an easy enough job for someone who knows what they are doing. If they don't; they have to live within the limitations I have introduced, find someone to fix it for them, or do without.

I would love someone to do a better job than I have but, until then, this is what there is.
dthacher wrote:
Tue Jun 22, 2021 1:57 am
This looks like a problem:
"mov r7, ip" "\n"
I believe that should be okay. GCC generates code which loads the value I need into 'ip' earlier and the code I will be executing won't be touching 'ip', or it will need to be pushed to preserve it.

I did try adding "push {ip}" and "pop {ip}" around my "blx lr" but GCC rejects those as invalid register lists. I am sure there is some way around that but solving that is not a priority for me.
dthacher wrote:
Tue Jun 22, 2021 1:57 am
Why did you not make it this?
"ldr r7, =reg" "\n"
I do for the C Extension Module but the MicroPython tool which packages a Native C Module for use crashes with that. I would likely have used that if it didn't, but it does.

Using "mov r7,%[reg]" does work but, if I include that after the call, the GCC generated code corrupts multiple registers before I have a chance to save them.

Rather than expend time and effort in resolving that in some kind of 'proper way' I simply use the "ip" which already contains the value I want in 'r7'.

Maybe I am saving up problems for later but I can address those when and if they arise. My priority was to get things to work with what I have so I can continue with what I am wanting to do. If what I now have is useful to others then that's great, if not I don't mind.

My belief is having something which doesn't quite do what one wants, isn't quite how one might like it to be, is still far better than having nothing at all.

dthacher
Posts: 40
Joined: Sun Jun 06, 2021 12:07 am

Re: [SOLVED] Dynamic code execution

Tue Jun 22, 2021 2:51 pm

hippy wrote: I believe that should be okay. GCC generates code which loads the value I need into 'ip' earlier and the code I will be executing won't be touching 'ip', or it will need to be pushed to preserve it.
This only works in your case. If you follow the convention you should save r12, however this could be confusing.
hippy wrote: I did try adding "push {ip}" and "pop {ip}" around my "blx lr" but GCC rejects those as invalid register lists. I am sure there is some way around that but solving that is not a priority for me.
You would want to push r7 before calling loads. Then pop it before stores. You can use r8-r11.
hippy wrote: I would love someone to do a better job than I have but, until then, this is what there is.
Here is a more standard approach. Basically the same thing, but allows GCC to do a lot of the work. You cannot use function pointer for GCC using 8 registers. (Hence your convention.) Note all of this still requires special tricks for symbols.

Code: Select all

uint32_t reg[4];
uint32_t codAdr;

void Run() {
	void (*func)(uint32_t *, uint32_t *, uint32_t *, uint32_t *);
	func = (void *) codAdr;
	
	func(&reg[0], &reg[1], &reg[2], &reg[3]);
}
hippy wrote: Using "mov r7,%[reg]" does work but, if I include that after the call, the GCC generated code corrupts multiple registers before I have a chance to save them.
Probably due to missing function pointer. This could be variable with other compilers, but I would not quote me on that. For your application this will work fine almost every time. However as a module/library, maybe not.

I think the assembly function suggestion from before would work around this. Basically convert run into an assembly function then link it with C would be the better way to go.

hippy
Posts: 10239
Joined: Fri Sep 09, 2011 10:34 pm
Location: UK

Re: [SOLVED] Dynamic code execution

Tue Jun 22, 2021 8:21 pm

dthacher wrote:
Tue Jun 22, 2021 2:51 pm
I think the assembly function suggestion from before would work around this. Basically convert run into an assembly function then link it with C would be the better way to go.
Things work in the C Extension Module as it is without any nasty tricks, using "mov r7,=reg". So figuring how to add assembler into a MicroPython build is just additional effort which is not needed at present.

The Native C Module works just as well. It's not preserving 'ip' but that's not an issue for me.

How MicroPython Native C Modules get built and packaged are something of a mystery to me, not very well documented, and seemingly very limited. Traditional linking doesn't appear to be supported so less chance of getting assembler to work with that; I haven't even been able to use multiple C files other than by using #include.

I have what I need so am calling it job done and putting it behind me - I have already moved on to the next thing on my To Do list.

Return to “General”