REST APIs bring several inherent advantages:

  • code/platform agnostic, everything is just an HTTPS request
  • client/server separation

If you have a fleet of IOT devices deployed, how do you receive that data for processing, monitoring, and reporting…into your Oracle Database?

The Solution

Didi you know, I am not a graphic artist!
Not shown: any Load Balancers, API Gateways, or Proxies front ending your ORDS Mid-Tiers.

What’s in our diagram?

  • IOT Devices collecting … weather data
  • python function/scripts loading telemetry data via HTTP POST
  • Oracle Database for storage and processing of our data
  • REST API for ingestion, powered by simple PL/SQL stored procedure and secured…
  • …via a Pre-Authenticated Request (PAR)
  • No Oracle Client, No Oracle drivers, and no Oracle Database usernames or passwords
ORDS logo

Wait, what is this? That’s our REST API resource server, Oracle REST Data Services, or simply, ORDS. It makes things in your Oracle Database, or the database itself, available via HTTP(S). You can have one or more of them in front of your database, serving up requests from your applications, monitoring/logging infrastructure, APEX apps, and more!

The Backend Database

It’s just an Oracle Database. Any currently supported version of Oracle will do, and it can be of the FREE or licensed variety.

In that database we’ll have a USER/SCHEMA that contains a:

  • TABLE – to store the incoming data,
  • PL/SQL stored procedure – to process JSON and do table inserts,
  • and REST API – to hand off the data from device and invoke the SP

The REST APIs will be served by Oracle REST Data Services (ORDS), which is also free, and in the Oracle Cloud, is pre-provisioned and maintained as part of the Autonomous Database Service for your convenience.

Otherwise, it’s about a 5 minute download/install/deploy process, especially if you’re just tinkering around on your laptop.

I’m using a VirtualBox Appliance our team makes available that includes the Database & ORDS all ready to go for local development work (Download || Tutorial.)

If you prefer Containers, we have those, too.

Our Database Table…

…is quite simple. The ID is the primary key for the SENSOR_READINGS table, it’s value is automatically added for any row inserted, and is an ascending number, maintained by the database, for us.

The rest of the the fields should be self-explanatory. Note we’re storing our data via a relational model, but we could have also gone with JSON documents, or even the best of both worlds via a JSON Relational Duality View.

Database Stored Procedure

I’m going to show you the code visually, but no worries I’ll share the actual code down below.

The stored procedure accepts a collection of JSON objects, and then processes each object to build up an INSERT DDL operation to load the data to our table.

The trickiest thing here is to know what the incoming SENSOR_TIME values will look like so we can format it for the DATE value to be inserted into the table, that’s the TO_DATE() call on line 17.

The logic of our API is stored and executed IN the database, where the data is. That means the processing is lightning fast, and it offloads that from your browser, mobile apps, node.JS mid-tiers, etc.

The Database REST API

The RESTful Service simply invokes the above stored procedure. It’s ‘nice’ in the sense that it will report on successful invocations with a 201, and the number of records processed. And if it fails, it will share back some useful errors so the request can be ‘fixed.’

Our REST API has no query parameters, and only services a single purpose, ingesting records. If the incoming payload doesn’t match the JSON Schema enforced by the table, the records will be rejected.

Our API is accessed via HTTPS, so literally any program could use the interface to upload their data, assuming they have the proper AUTH.

There is no way to directly interact with the database, and there are no database credentials shared or used to upload our data, so there’s no way for the API to be misused, nor are there any risks for the credentials to be leaked or falling into the hands of a bad actor.

Testing the API

Before I go and deploy this API to my Python app and then have my test device start using it, I should probably make sure it ‘works,’ or see how it works when things go ‘good’ and ‘bad.’

My API link has been generated as a Pre-Authenticated Request, hence the _/par/randomlessof letters… looking URI. Note that anyone I share this PAR with, will be able to exercise the API.

Good

Good request, good response.

And if you want to double-check, browsing the TABLE with a point and click in our SQL Developer Extension for VS Code will show you the data.

But of course, sometimes things go less than good. So we need logging, error messages, etc.

Bad

Bad request, the array shouldn’t be named.

Your python script will want to log these responses to a file or table or use some sort of ‘logger’ so you can troubleshoot your IOT device as needed.

Our IOT Device Software Payload

We’ll assume Linux is onboard, and we’ll assume you have Python available, with a network connection, AND a sensor that collects the data we want to upload.

I’ve picked some common weather based metrics, that would be readily available in a real-world scenario, but you can extrapolate the data model to fit your needs.

Python 🐍

My intent here is to do something between a mockup with pseudo code, and also share what the upload and logging would look like. And I want to have a minimum of required libraries (http.client & sys.)

What the code is doing –

  • register filename argument passed in
  • construct the http request headers
  • open the file, for ‘payload’, which will be sent as the request body
  • make the request
  • print the result

Because I have attention issues, I like hitting the Go! button as quickly as possible. Let’s say I forget to add my PAR for the request resources…

It fails. Which is GOOD, because our endpoint is SECURED. Without the PAR, the API request fails because I’m not providing an access token, say if I had walked a full OAuth2 workflow (ORDS + Python Example.)

But since we DO have a PAR available for accessing our logging table API, we can simply use that in our POST request. Here’s the code stub you could use for a python function:

Python
import http.client
import sys

conn = http.client.HTTPConnection("localhost:8181")
filename = sys.argv[1]

headers = {
     'Content-Type' : "application/json",
     'file' : filename
    }

with open(filename) as payload:
    conn.request("POST", "/ords/hr/_/par/ueh1MfXggkTzH2dZGSoXR1NhIjmt8MzzuNrk1DxS3kqhZQipolL5wuXa8miYWxaylrmF2vnY5EhZxbns5Pidf0Dp7Lw09Lcuu6HdtS3mIDplmH90iy0FooAUaYJ8t/thatjeffsmith/sensor_data/", payload, headers)

res = conn.getresponse()
data = res.read()
print(data.decode("utf-8"))

And now calling our program –

Sweet!

Sending that response to the console is…pretty useless. We should ‘print’ or append that to a log file.

And OK, let’s make sure the data got into the table –

Our VS Code Extension also includes an integrated CLI for your Terminal. Because, we love developers!

Batching IOT data vs 1-by-1’s

You will want to probably have your devices send a collection of data periodically to the system vs doing a POST on every single device reading. The API and the Database will happily accept hundreds of payloads from your IOT ‘fleet,’ each having many hundreds, or thousands of records.

This example assumes you have a different set of code (python or otherwise) that will take the instrumentation readings and write them to a JSON file on your device’s storage, which is then on a regular basis sent to the ‘upload-sensor-data’ python script.

Our program will happily upload 1 records, or hundreds, as often as you invoke it.

How can I setup the PARs, or how can I shut the entire system ‘down?’

In this post I demonstrate how to create the PAR for your Oracle Database REST API, and also how you can NUKE the PAR. Otherwise, the PAR generated below will last approx 6 months before expiring.

The Database & REST API Code

PLSQL
-- assumes your database is version 23ai
-- with a few tweaks, 100% possible in 19c 

-- create the table to store the data 
CREATE TABLE IF NOT EXISTS SENSOR_READINGS 
   (	"ID" NUMBER GENERATED ALWAYS AS IDENTITY MINVALUE 1 MAXVALUE 9999999999999999999999999999 INCREMENT BY 1 START WITH 1 CACHE 20 NOORDER  NOCYCLE  NOKEEP  NOSCALE  NOT NULL ENABLE, 
	DEVICE_ID VARCHAR2(50), 
	SENSOR_TIME DATE DEFAULT sysdate, 
	TEMPERATURE NUMBER(5,2), 
	HUMIDITY NUMBER, 
	BAROMETRIC NUMBER, 
	 PRIMARY KEY (ID);
	 
-- process JSON to insert into table 
create or replace PROCEDURE add_sensor_data(
    p_json_data IN JSON, rows_added out varchar2
) AS
    v_rowcount NUMBER;
BEGIN
    -- Single INSERT statement with JSON_TABLE
    -- JSON array is read into a tabular format via JSON_TABLE Function
    -- $.notation allows youo to take attributes from JSON object and 
    --   map to columns
    -- Note that the date value is read in as a string and then formatted
    --   as a date before being inserted 
    INSERT INTO sensor_readings (
        device_id,
        temperature,
        humidity,
        sensor_time,
        barometric
    )
    SELECT device_id,
           temp,
           humidity,
           to_date(sensor_time, 'YYYY-MM-DD HH24:MI:SS'),
           barometric
    FROM JSON_TABLE(
        p_json_data,
        '$[*]'
        COLUMNS (
            device_id VARCHAR2(50) PATH '$.deviceId',
            temp number PATH '$.temp',
            humidity number PATH '$.humidity',
            sensor_time varchar2(50) PATH '$.time',
            barometric NUMBER PATH '$.mb'
        )
    );

    -- Capture the number of rows inserted
    v_rowcount := SQL%ROWCOUNT;
    rows_added := v_rowcount;

    -- Commit the transaction
    COMMIT;

    -- Output the result
    -- only useful while 'debugging' in database client 
    -- this value is also returned via OUT parameter for REST API 
    DBMS_OUTPUT.PUT_LINE(v_rowcount || ' records inserted successfully');

    -- do BETTER than WHEN OTHERS, but this will catch everything that
    --  fails and print error stack on oracle client console 
EXCEPTION
    WHEN OTHERS THEN
        -- Rollback on error
        ROLLBACK;
        DBMS_OUTPUT.PUT_LINE('Error occurred: ' || SQLERRM);
        RAISE;
END add_sensor_data;
/

-- Create the REST API 

-- Generated by ORDS REST Data Services 24.4.0.r3451601
-- Schema: HR  Date: Tue Mar 25 07:04:34 2025 
--
        
DECLARE
  l_roles     OWA.VC_ARR;
  l_modules   OWA.VC_ARR;
  l_patterns  OWA.VC_ARR;

BEGIN
  -- assumes ORDS is installed for your database
  -- replace HR for schema where your table/procedure were created
  ORDS.ENABLE_SCHEMA(
      p_enabled             => TRUE,
      p_schema              => 'HR',
      p_url_mapping_type    => 'BASE_PATH',
      p_url_mapping_pattern => 'hr',
      p_auto_rest_auth      => FALSE);
    
  ORDS.DEFINE_MODULE(
      p_module_name    => 'thatjeffsmith',
      p_base_path      => '/thatjeffsmith/',
      p_items_per_page => 25,
      p_status         => 'PUBLISHED',
      p_comments       => 'For thatjeffsmith.com Oracle Database tutorials. Please review and test all code before copy/pasting into your projects.');

  ORDS.DEFINE_TEMPLATE(
      p_module_name    => 'thatjeffsmith',
      p_pattern        => 'sensor_data/',
      p_priority       => 0,
      p_etag_type      => 'HASH',
      p_etag_query     => NULL,
      p_comments       => 'For storage of your IOT Sensor data. JSON Object Definition:

{
  "deviceId": "text",
  "temp": 65.4,
  "humidity": 56,
  "time": "DDDD-MM-DD HR24:MI:SS"",
  "mb": 1015.6
}');

-- this is where the POST handler logic is defined 
-- :body_text is an implicit :bind variable that makes POST REQUEST BODY
--    payload available to plsql program as a CLOB 
--    we convert that with JSON() function as we send to PL/SQL API
  ORDS.DEFINE_HANDLER(
      p_module_name    => 'thatjeffsmith',
      p_pattern        => 'sensor_data/',
      p_method         => 'POST',
      p_source_type    => 'plsql/block',
      p_mimes_allowed  => NULL,
      p_comments       => 'Accepts an array of our objects, example: 

[
    {"deviceId": "Sue", "temp": 65.4, "humidity": 56, "time": "2024-03-24 07:30:01", "mb": 1015.6},
    {"deviceId": "Sue", "temp": 65.8, "humidity": 55, "time": "2024-03-24 07:31:01", "mb": 1015.7},
    {"deviceId": "Sue", "temp": 66.1, "humidity": 54, "time": "2024-03-24 07:32:01", "mb": 1015.5}
]',
      p_source         => 
'begin
    add_sensor_data(p_json_data => json(:body_text), rows_added => :response);
    :status_code := 201;
EXCEPTION
        WHEN NO_DATA_FOUND
        THEN
            :message := ''Sorry cannot find who you are looking for, get with the program'';
            :status := 400;
        WHEN VALUE_ERROR
        THEN
            :message := ''Your name and message were, weird? Check inputs and try again.'';
            :status := 400;
        WHEN OTHERS
        THEN
            :message := ''That does not look quite right, ensure your payload matches this format [{"deviceId": "Sue", "temp": 65.4, "humidity": 56, "time": "2024-03-24 07:30:01", "mb": 1015.6},...]. Here is the system response: '' || SQLERRM;
            :status := 400;
end;');

-- this captures. the OUT parameter for PLSQL api to get number of rows added
  ORDS.DEFINE_PARAMETER(
      p_module_name        => 'thatjeffsmith',
      p_pattern            => 'sensor_data/',
      p_method             => 'POST',
      p_name               => 'rowsAdded',
      p_bind_variable_name => 'response',
      p_source_type        => 'RESPONSE',
      p_param_type         => 'STRING',
      p_access_method      => 'OUT',
      p_comments           => NULL);

-- this captures oracle error message returned from calling PL/SQL program

  ORDS.DEFINE_PARAMETER(
      p_module_name        => 'thatjeffsmith',
      p_pattern            => 'sensor_data/',
      p_method             => 'POST',
      p_name               => 'message',
      p_bind_variable_name => 'message',
      p_source_type        => 'RESPONSE',
      p_param_type         => 'STRING',
      p_access_method      => 'OUT',
      p_comments           => NULL);

-- SECURITY Mechanism for locking down access to API
    
  ORDS.CREATE_ROLE(p_role_name => 'DEMO');
    
  l_roles(1) := 'DEMO';
  l_modules(1) := 'thatjeffsmith';

  ORDS.DEFINE_PRIVILEGE(
      p_privilege_name => 'sensors',
      p_roles          => l_roles,
      p_patterns       => l_patterns,
      p_modules        => l_modules,
      p_label          => 'sensors',
      p_description    => 'lock down IOT device APIs',
      p_comments       => NULL);

  l_roles.DELETE;
  l_modules.DELETE;
  l_patterns.DELETE;
        
COMMIT;

END;
/

-- Creating. the Preq-Authenticated Request
-- Pick-up the PAR URL output from this block to substitute in 
--   line #13 of sensor-data.py 
DECLARE
  l_uri clob;
BEGIN
   l_uri := ORDS_PAR.DEFINE_FOR_HANDLER(
    p_module_name => 'thatjeffsmith',
    p_pattern => 'sensor_data/',
    p_method => 'POST',
    p_duration => 15769728
  );
  COMMIT;
  dbms_output.put_line(l_uri);
END;
/

Author

I'm a Distinguished Product Manager at Oracle. My mission is to help you and your company be more efficient with our database tools.

Write A Comment