PyTorch to vectorize images, REST APIs to POST to Oracle

Back in September I made my first venture into 23ai’s VECTOR feature. We used some interesting data with a generation model loaded into the Oracle Database to compute some vectors using Hugging Face and all-MiniLM-L12-v2 (Read that now.)

So that was cool, but it was all text based.

How about doing something similar with pictures?

Now, you can’t load an LLM to Oracle that deals with pictures (yet??), so what am I do to?

Want to play with 23ai? Try it for FREE on your own machine!
 Container | Linux (OL9) | Windows | VirtualBox

Just use python!

This code takes a filename from the command line argument passed to it, and then using the Hugging Face openai/clip-vit-base-patch32 model – generates some vectors that we can then use to compare pictures to see if they’re alike, or just how alike they are.

Once I get that Vector data (it looks like the above if you print it), I then hand it over to the requests library to do a POST to an ORDS REST API.

The code – feel free to laugh, I’m only mildly aware of how it actually works.

from PIL import Image
from transformers import CLIPProcessor, CLIPModel
import torch
import numpy as np
import http.client
import sys

# Load the CLIP model and processor
model = CLIPModel.from_pretrained("openai/clip-vit-base-patch32")
processor = CLIPProcessor.from_pretrained("openai/clip-vit-base-patch32")

# Move model to GPU if available
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model = model.to(device)

filename = sys.argv[1]

# Load and preprocess the image
try:
    image = Image.open(filename).convert("RGB")  # Ensure image is in RGB mode
except FileNotFoundError:
    print("Error: The file not found.")
    exit(1)
except Exception as e:
    print(f"An error occurred while opening the image: {e}")
    exit(1)

inputs = processor(images=image, return_tensors="pt").to(device)

# Generate the image embedding
with torch.no_grad():  # Disable gradient calculation for efficiency
    image_features = model.get_image_features(**inputs)

# Convert to numpy array, detach from computation graph, and move to CPU
vector = image_features.detach().cpu().numpy().squeeze()  # .squeeze() removes any singleton dimensions
vector_str = '[' + ','.join(map(str, vector)) + ']'

#print("Shape of vector:", vector.shape)
#print("Vector:", vector_str)

conn = http.client.HTTPConnection("localhost:8181")
payload = vector_str

headers = {
     'Content-Type' : "text/plain",
     'file' : filename
    }


conn.request("POST", "/ords/hr/vectors/pics/", payload, headers)
res = conn.getresponse()
data = res.read()
print(data.decode("utf-8"))

The REST API is really, really simple –

In my python code you can see how I’m doing the POST, here –

This wasn’t all that tricky, except I had to play with the output from the model.get_image_features() function call to look like a vector – that’s why you see the string concatenation call right after that, which is also used to define the POST request PAYLOAD.

Using a REST API to load a VECTOR into the Oracle Database is super simple, you’re just doing an INSERT.

Calling my program

I was too lazy to figure out how to loop through the contents of the directory and do the POSTs, so I just called it 4 times. If I were processing 4,000 files, you bet your butt I would have taken the time to do that. But, you’re not here to learn Python, I hope!

Looking at the table data

The picture are all of my dog, and in some of them, I’m also drinking a beer.

Comparing/finding similar pictures

First I need an INDEX, the maths on dealing with these VECTORS can get expensive.

CREATE VECTOR INDEX pics_hnsw_idx ON pics_and_vectors (victor) ORGANIZATION INMEMORY NEIGHBOR GRAPH
DISTANCE COSINE
WITH TARGET ACCURACY 95;

Then I need a PL/SQL function I can use, to make it VERY easy to pass an existing VECTOR to a SQL statement –

create or replace function kareem2 (input_id in integer) return vector is
 roger vector;
begin
 select victor into roger
 from pics_and_vectors
 where id = input_id;
 return roger;
end;

I pass in an ID for a row in my table, it simply returns the VECTOR in that same row.

Now I can run the query –

SELECT file_name, content
FROM pics_and_vectors
WHERE id <> 4
ORDER BY VECTOR_DISTANCE( victor, kareem2(4), COSINE )
FETCH APPROXIMATE FIRST 2 ROWS ONLY;