Logo

dev-resources.site

for different kinds of informations.

DBOS-Cloud: Fast and Free Automatic Data Aggregator

Published at
7/20/2024
Categories
dbos
workflows
postgres
orchestration
Author
vince_hirefunnel_co
Author
19 person written this
vince_hirefunnel_co
open
DBOS-Cloud: Fast and Free Automatic Data Aggregator

This is a simple DBOS app example focusing on remote deployment to DBOS Cloud, their hosted solution with a generous free tier for devs.

The github repo sets up a cron job which: 1) performs a "SELECT COUNT" on a Supabase Postgres database table and 2) an "INSERT" to a second table. Consider it a starter for a poor man's data lake aggregating data in the receiving table.

Note: This is not the fastest, most compact and recommended way to create this workflow. DBOS-Cloud has terrific in-app full Postgres database functionality. Check out the transactions section for more. I built the script this way since I had pre-existing data stored on Supabase.


Prerequisites

  • Make sure you have Node.js 21.x

  • Sign up for DBOS Cloud (https://www.dbos.dev/dbos-cloud)

  • Have a Supabase account and get your project url (https://foobar.supabase.co/rest/v1) and project API anon key. Check out the Supabase docs to get them.

  • Make sure to have two tables set up, the first can have any data in it as we will be counting rows, and the second table should have the columns "id", "count", and "created_at".


Getting Started

Clone this repository and navigate to the project directory

  • Install the dependencies

  • To deploy to DBOS-cloud, login with this, "npx dbos-cloud login" and follow the instructions to match the uuid given in the console to the one in the browser, then standard login user/password applies.

  • Next provision a d'base instance: "npx dbos-cloud db provision database-instance-name -U database-username"

  • Register your app with the d'base instance: "npx dbos-cloud app register -d database-instance-name"


Secrets Handling

To use secrets in DBOS, add your variables in the cli like this:

  • export PGPASSWORD=password-when-you-setup-the-remote-database-here

  • export SUPABASE_URL=https://foobar.supabase.co

  • export SUPABASE_ANON_KEY=long-foo-bar-key

  • export SOURCE_TABLE=source-table-name

  • export DESTINATION_TABLE=destination-table-name

These will be picked up at build time and inserted into the dbos-config.yaml fields: ${PGPASSWORD}, ${SUPABASE_URL}, ${SUPABASE_ANON_KEY}, ${SOURCE_TABLE}, ${DESTINATION_TABLE} respectively. Don't use quotation marks wrapping the values. The PGPASSWORD is for the basic user (not admin) Postgres service, which won't be used here.

Be sure not to commit / hard code your secrets to a public repo! This setup is locally developed and directly deployed to the dbos-cloud service.


Get ready to deploy!

  • And finally deploy your app: "npx dbos-cloud app deploy"

After 5 minutes, assuming you have data in your source table, check your destination table for entries.

  • If you are not getting expected results:

Login to the monitoring dashboard using this "npx dbos-cloud dashboard url"and check the collection of logs.

  • To delete the app online,npx dbos-cloud app delete [application-name] --dropdb. Remove the '--dropdb' parameter if you want to retain the database table. If you don't delete it, the cron script will continue to run.

There are so many more benefits to mention re: workflow orchestration, observability, debugging, monitoring and overall speed, security, and costs which can be explored further here: https://docs.dbos.dev/


Reference Docs (From Official Repo)


Resources to learn more --

Awesome-dbos

The first article in this series using cron and Postmark email service provider.

workflows Article's
30 articles in total
Favicon
Security Teams : Automate Cybersecurity Workflows
Favicon
GitHub Action security hardening with OpenID (OIDC) Connect - "Password-Less"
Favicon
Optimizing Test Coverage for Oracle OTM Workflows
Favicon
Continuous Integration
Favicon
DBOS-Cloud Simple and Robust Workflow Orchestration
Favicon
DBOS-Cloud: Minimal Effort Change Data Capture (CDC) Tool
Favicon
DBOS-Cloud: Fast and Free Automatic Data Aggregator
Favicon
DBOS-Cloud: Fast and Free Automatic Supabase Table Copier
Favicon
Part 1 - LittleHorse Advanced Concept Series: Conditionals
Favicon
Free & Paid Notion templates, covers, tools and more!
Favicon
Announcing WDL 1.1.1
Favicon
Visualize and Inspect Workflows Executions
Favicon
Simplifying Workflow Management with the XWorkflows Library
Favicon
How to use GitHub Workflows to Streamline Your Software Development Process
Favicon
Building Autonomous Business Processes using AI Agent Workflows
Favicon
📦 Stay Organized and Efficient with Auto-Label-Pulls Github Action
Favicon
Temporal for VS Code
Favicon
Using the Temporal CLI
Favicon
Getting Started With Apache Airflow
Favicon
Getting Started with n8n: Integrating Typeform and Slack for Dynamic Messaging
Favicon
Calculating your potential reach on Mastodon with Google Cloud Workflows orchestrating the Mastodon APIs
Favicon
Simple Github Workflow for Lint - Prettier & Jest (yarn)
Favicon
How to view fields in an unauthorized connector's action cards
Favicon
All Okta Workflows Community-Created Flows
Favicon
Okta Workflows Connector Builder
Favicon
Manage Short Term Users with Okta Workflows
Favicon
GraphQL queries with Okta Workflows
Favicon
Event Hook Filtering and Okta Workflows
Favicon
Pagination with Okta Workflows
Favicon
A Flow to Audit Inactive Google Workspace Users in Okta

Featured ones: