User avatar
scruss
Posts: 1398
Joined: Sat Jun 09, 2012 12:25 pm
Location: Toronto, ON
Contact: Website

Porting a C Python module to Python 3

Wed Jul 12, 2017 4:26 pm

I find this module very useful: migurski/atkinson: Atkinson dithering of PIL images in Python.

Unfortunately, Python 3 changed its interfacing method, and it won't build at all under Py3. So I've had to stick with using it in Python 2.7, which is a pain, as there are some Py3-only modules that I'd like to use at the same time.

It's a really simple module with just one method. There are pure-Python modules that do the same, but they're typically 10× slower and thus useless for interactive work. Where would I start looking for porting tips?
‘Remember the Golden Rule of Selling: “Do not resort to violence.”’ — McGlashan.

User avatar
paddyg
Posts: 2013
Joined: Sat Jan 28, 2012 11:57 am
Location: UK

Re: Porting a C Python module to Python 3

Wed Jul 12, 2017 10:58 pm

I've not looked too hard at what it does but many things like this can be done with numpy (very fast). I will have a more comprehensive peruse in the morning.

PS had a slightly longer look and I can see there is an issue with the way the atkinson algorithm propagates the errors meaning that looping through the array probably is necessary. It still might be possible to do an approximation with numpy but cython and ctypes are not so difficult.
also https://groups.google.com/forum/?hl=en-GB&fromgroups=#!forum/pi3d

User avatar
paddyg
Posts: 2013
Joined: Sat Jan 28, 2012 11:57 am
Location: UK

Re: Porting a C Python module to Python 3

Thu Jul 13, 2017 3:41 pm

OK, I thought it might be useful for me to figure out how to build 'proper' python3 modules... But quickly decided there were too many complications for such a simple thing. What I did was compile just the C function below into a shared object file then imported that into python using ctypes. The wrapper round the numpy array passing looks ugly, and I think I've got away with something simpler in the past, but also had issues so you could play around. Also there is no kind of testing in here so it's probably easy to cause seg faults etc!

C file 'atk_mod.c'

Code: Select all

// this is a slightly condensed version of migurski - use an if rather than  lookup table for threshold
#include <string.h>
#include <stdio.h>

#ifndef adderror
    #define adderror( b, e ) ( ((b) < -(e)) ? 0x00 : ( ((0xFF - (b)) < (e)) ? 0xFF : (b + e) ) )
#endif

void atk(int w, int h, unsigned char pixels[])
{
  int i, x, y, off, len, err;
  int threshold = 127;
  unsigned char old, new;
  
  for(y = 0; y < h; y++)
  {
      for(x = 0; x < w; x++)
      {
          // offset in the string for a given (x, y) pixel
          off = (y * w) + x;
          // threshold and get the error
          old = pixels[off];
          if (old > threshold) {
            new = 0xFF;
          } else {
            new = 0x00;
          }
          err = (old - new) >> 3; // divide by 8
          // update the image
          pixels[off] = new;

          // now distribute the error...
          if(x+1 < w) { // x+1, y
              pixels[off + 1] = adderror(pixels[off + 1], err);
          }
          if(x+2 < w) { // x+2, y
              pixels[off + 2] = adderror(pixels[off + 2], err);
          }
          if(x > 0 && y+1 < h) { // x-1, y+1
              pixels[off + w - 1] = adderror(pixels[off + w - 1], err);
          }
          if(y+1 < h) { // x, y+1
              pixels[off + w] = adderror(pixels[off + w], err);
          }
          if(x+1 < w && y+1 < h) { // x+1, y+1
              pixels[off + w + 1] = adderror(pixels[off + w + 1], err);
          }
          if(y+2 < h) { // x, y+2
              pixels[off + 2 * w] = adderror(pixels[off + 2 * w], err);
          }
      }
  }
}
compiled on command line:

Code: Select all

gcc -shared -o atk.so -fPIC atk_mod.c
used in python3 as per

Code: Select all

import numpy as np
import ctypes
from PIL import Image

PATH = '/home/patrick/raspberry_pi/atkinson/'
atklib = ctypes.CDLL(PATH + 'atk.so')
img = np.array(Image.open(PATH + 'lenna_l.png'))
atklib.atk(img.shape[0], img.shape[1], 
           img.ctypes.data_as(ctypes.POINTER(ctypes.c_ubyte)))
Image.fromarray(img).save('lenna_l.png')
lenna_l.png
lenna_l.png (40.81 KiB) Viewed 554 times
lenna_bw.png
lenna_bw.png (12.71 KiB) Viewed 554 times
PS you do know that PIL.Image.convert() will use FLOYDSTEINBERG dithering.
Last edited by paddyg on Thu Jul 13, 2017 4:24 pm, edited 1 time in total.
also https://groups.google.com/forum/?hl=en-GB&fromgroups=#!forum/pi3d

User avatar
scruss
Posts: 1398
Joined: Sat Jun 09, 2012 12:25 pm
Location: Toronto, ON
Contact: Website

Re: Porting a C Python module to Python 3

Thu Jul 13, 2017 3:58 pm

Thanks! That as a lot of work you went to. You could probably do without Numpy, as it's really only acting as an array handler. Pretty sure there's similar code in PIL to do that.
‘Remember the Golden Rule of Selling: “Do not resort to violence.”’ — McGlashan.

User avatar
paddyg
Posts: 2013
Joined: Sat Jan 28, 2012 11:57 am
Location: UK

Re: Porting a C Python module to Python 3

Thu Jul 13, 2017 4:57 pm

Yes, I feel this ought to be possible. You could use the Image.tobytes() then somehow frombytes(). However I tried for a long time to improve the efficiency of pi3d and ended up using numpy. Let me know if you figure out a way.

EDIT. OK seems to be as easy as

Code: Select all

im = Image.open(PATH + 'lenna_l.png')
img = im.tobytes()
atklib.atk(im.size[0], im.size[1], 
           ctypes.c_char_p(img))
Image.frombytes('L', im.size, img).save('lenna2_bw.png')
I will check out speed comparison.
PPS no difference in speed - presumably they're doing the same thing behind the scenes
... later still ...
Just for completeness I tried the cython route too, and it turned out to marginally faster (but the .so file was fifteen times bigger!). It was quite tedious figuring out the best way to pass the data and in the end I had to go back to numpy array. If anyone decides they need to roll their own external functionality then there is a lot to be said for using cython. Here is a record of what I did. This was the atk_mod_a.pyx file

Code: Select all

from __future__ import division
import numpy as np
cimport numpy as np

cdef inline np.uint8_t adderror(np.uint8_t b, int e):
  return min(max(b + e, 0x00), 0xFF) 

def atk(np.ndarray[np.uint8_t, ndim=2] pixels):
  cdef:
    int x, y, off, err
    int threshold = 127
    np.uint8_t old, new
    int h = pixels.shape[0]
    int w = pixels.shape[1]
  
  for y in range(h):
      for x in range(w):
          old = pixels[y, x]
          if old > threshold:
            new = 0xFF
          else:
            new = 0x00
          err = (old - new) >> 3; # divide by 8
          pixels[y, x] = new
          # now distribute the error...
          if (x+1) < w:            # x+1, y
              pixels[y, x + 1] = adderror(pixels[y, x + 1], err)
          if (x+2) < w:           # x+2, y
              pixels[y, x + 2] = adderror(pixels[y, x + 2], err)
          if x > 0 and (y+1) < h:   # x-1, y+1
              pixels[y + 1, x - 1] = adderror(pixels[y + 1, x - 1], err)
          if (y+1) < h:           # x, y+1
              pixels[y + 1, x] = adderror(pixels[y + 1, x], err)
          if (x+1) < w and (y+1) < h: # x+1, y+1
              pixels[y + 1, x + 1] = adderror(pixels[y + 1, x + 1], err)
          if (y+2) < h:            # x, y+2
              pixels[y + 2, x] = adderror(pixels[y + 2, x], err)
and this was the setup.py file

Code: Select all

from distutils.core import setup
from Cython.Build import cythonize

setup(
    ext_modules = cythonize("atk_mod_a.pyx")
)
built on the command line with

Code: Select all

python3 setup.py build_ext --inplace
and used from normal python

Code: Select all

from PIL import Image
import numpy as np
from atk_mod_a import atk
PATH = '/home/pi/atkinson/'

img = np.array(Image.open(PATH + 'lenna_l.png'))
atk(img)
Image.fromarray(img).save('lenna_bw.png')
also https://groups.google.com/forum/?hl=en-GB&fromgroups=#!forum/pi3d

User avatar
paddyg
Posts: 2013
Joined: Sat Jan 28, 2012 11:57 am
Location: UK

Re: Porting a C Python module to Python 3

Tue Jul 18, 2017 4:29 pm

Finally got round to looking at this again and I've posted the various files to https://github.com/paddywwoof/atkinson I've also put in a subdirectory python_module a 'standard' module structure which compiles for python2 or python3
also https://groups.google.com/forum/?hl=en-GB&fromgroups=#!forum/pi3d

User avatar
scruss
Posts: 1398
Joined: Sat Jun 09, 2012 12:25 pm
Location: Toronto, ON
Contact: Website

Re: Porting a C Python module to Python 3

Tue Jul 18, 2017 11:50 pm

Thank you! It was very kind of you to do this!
‘Remember the Golden Rule of Selling: “Do not resort to violence.”’ — McGlashan.

User avatar
paddyg
Posts: 2013
Joined: Sat Jan 28, 2012 11:57 am
Location: UK

Re: Porting a C Python module to Python 3

Sun Nov 19, 2017 9:38 pm

For even more completeness, and because I thought it was time to find out more about rust (touted as the successor to C/C++), I added another version here which actually turns out to be faster than the C and cython versions.

For a one-off module, rust with ctypes (or cffi) seems to have a lot going for it, in the medium term, there seems to be some work being done towards a general integration of rust into python ecosystem see here and here. It was very easy to set up and cross compile for the raspberry pi see here
also https://groups.google.com/forum/?hl=en-GB&fromgroups=#!forum/pi3d

User avatar
scruss
Posts: 1398
Joined: Sat Jun 09, 2012 12:25 pm
Location: Toronto, ON
Contact: Website

Re: Porting a C Python module to Python 3

Mon Nov 20, 2017 1:12 am

Thanks … and all those LETs are Sinclair BASIC-tastic!
‘Remember the Golden Rule of Selling: “Do not resort to violence.”’ — McGlashan.

Return to “Python”

Who is online

Users browsing this forum: elParaguayo and 17 guests