Image Processing

Matft.image provides image processing compatible with OpenCV (cv2), and preprocessing compatible with PIL / Hugging Face transformers. See NumPy Mapping › Image for the list of functions.

CGImage ⇄ MfArray

Convert a CGImage to an MfArray of shape (height, width, 4) (RGBA), process it with indexing or image functions, and convert it back. See the demo app for the complete example.

@IBOutlet weak var originalImageView: UIImageView!
@IBOutlet weak var reverseImageView: UIImageView!
@IBOutlet weak var swapImageView: UIImageView!

func reverse(){
    var image = Matft.image.cgimage2mfarray(self.reverseImageView.image!.cgImage!)

    // reverse
    image = image[Matft.reverse] // same as image[~<<-1]
    self.reverseImageView.image = UIImage(cgImage: Matft.image.mfarray2cgimage(image))
}

func swapchannel(){
    var image = Matft.image.cgimage2mfarray(self.swapImageView.image!.cgImage!)

    // swap channel
    image = image[Matft.all, Matft.all, MfArray([1,0,2,3])] // same as image[0~<, 0~<, MfArray([1,0,2,3])]
    self.swapImageView.image = UIImage(cgImage: Matft.image.mfarray2cgimage(image))
}

For more complex conversion, see OpenCV's code.

Preprocessing for vision models

Matft.image.resize(_:width:height:resample:) reproduces PIL.Image.resize (Pillow's fixed-point arithmetic), so a UInt8 image is resized to exactly the same pixels as PIL. On top of it, the preprocessing of Hugging Face transformers' image processors is available, e.g. to feed the same input as Python to a VLM running on Core ML or MLX.

let rgba = Matft.image.cgimage2mfarray(cgimage, mftype: .UInt8)     // (h, w, 4), 0...255
let rgb = rgba[Matft.all, Matft.all, 0~<3].to_contiguous(mforder: .Row)

let resized = Matft.image.resize(rgb, width: 224, height: 224, resample: .bicubic) // == PIL.Image.resize(BICUBIC)

let pixel_values = Matft.image.clip_preprocess(rgb)                       // (1, 3, 224, 224), same as CLIPImageProcessor
let (patches, grid_thw) = Matft.image.qwen2vl_preprocess(rgb)             // same as Qwen2VLImageProcessor

note

Unlike PIL, an RGBA image is not premultiplied by alpha. Convert it to RGB first as transformers does.

Visual check against OpenCV

MATFT_IMAGE_SNAPSHOT=1 swift test --filter MatftTests.ImageTest
python3 scripts/image_compare.py

note

Matft.image.warpAffine's matrix has the same meaning as cv2.warpAffine's one after v0.3.3 (v0.3.3 and earlier swapped the off-diagonal elements and used a bottom-left origin).
Interpolations differ from OpenCV's ones (vImage), so the small differences along the edges are expected.
Filters use MfBorderType.Replicate by default, because OpenCV's default border (BORDER_REFLECT_101) is not supported by vImage.