AI-powered astrology platform combining Vedic birth charts, computer-vision palmistry and a persistent conversational reading assistant.
A genuinely personalized astrology and palmistry experience needs several technical disciplines that don't normally sit in one stack: precise astronomical calculation for birth charts, computer vision to validate and read a photographed palm, and natural-language AI to turn all of that into a coherent, safe reading — while making sure a phone photo of a hand is actually usable input rather than something an LLM just guesses at.
We treated this as an AI-orchestration problem rather than a single model call. A Swiss Ephemeris-based engine computes precise Vedic (sidereal) and Western (tropical) placements from date, time and place of birth. A two-stage computer-vision pipeline confirms a photo actually contains a hand — and which one — before a custom-trained model detects palm-line and mount features. An OpenAI GPT-4o layer then fuses both outputs into a single persistent reading thread per user, so every later question is answered in that same personalized context. A React-based internal tool let us validate the whole flow end-to-end — entering birth details, uploading palms, visualizing the chart, chatting with the assistant — before committing to the mobile build.
Internal client
React-based internal tool for entering birth details, uploading palm images, visualizing the birth-chart wheel and testing the AI chat — used to validate the product ahead of the mobile build.
API
Core FastAPI service for users, astrology data, palm images and chat sessions.
Data
Relational storage for user profiles, computed astro data, palm analysis results and chat history.
Astrology engine
Deterministic Vedic and Western birth-chart calculation from date, time and place of birth — planetary positions, ascendant, moon nakshatra.
Computer vision pipeline
Validates that an uploaded photo actually contains a palm — and which hand — then detects palm-line and mount features.
Conversational AI layer
Combines astrology and palm results into a persistent per-user OpenAI Assistant thread that powers every follow-up question.
A palm photo has to survive two independent checks before it becomes part of a reading — is it actually a hand, and is it the hand the user says it is — only then does feature detection and AI interpretation run.
The user uploads a left or right palm photo, typically taken with a phone camera.
MediaPipe checks the image for a real hand landmark set before anything else runs, correcting for front-camera mirroring using the image's EXIF orientation data.
The detected hand side is compared against what the user said they uploaded — a mismatch is rejected before it ever reaches paid AI processing.
A custom-trained Roboflow model, running on a self-hosted inference server, detects palm lines and mounts as structured predictions.
The palm-analysis output is combined with the user's Vedic and Western birth chart inside a persistent OpenAI Assistant thread, which answers every subsequent question in that same personalized context.
Problem 1
Anyone can upload any photo — a screenshot, a random object, the wrong hand entirely — and a naive pipeline would send it straight to an expensive AI call and get a nonsense reading back.
Solution
We put a free, fast, local check ahead of any paid AI/CV cost: MediaPipe confirms a real hand is present and identifies which one, including correcting for mirrored front-camera selfies via EXIF orientation, before the image is ever sent to feature detection.
Impact: Bad uploads are rejected immediately with a clear reason, and every image that does reach the detection model is actually worth analyzing.
Problem 2
Off-the-shelf vision models don't know what a 'heart line' or a 'mount of Jupiter' is — palmistry features aren't a category any general-purpose model was trained on.
Solution
We trained a custom Roboflow object-detection model on palm-specific features and ran it on a self-hosted inference server, rather than relying on a generic vision API and prompting alone.
Impact: Feature detection is purpose-built for the domain instead of forced out of a general model.
Problem 3
A single LLM call can't hold a deep, evolving conversation about someone's chart and palm without re-sending their entire profile — astro data, palm summary, history — on every message.
Solution
Each user gets one persistent OpenAI Assistant thread, seeded once with their full birth-chart and palm-analysis context; every later question is asked inside that same thread, so the model already has that context.
Impact: Follow-up questions stay fast and cheap, and answers stay consistent with everything already established about the user.
Problem 4
A blended astrology and palmistry AI advisor can easily drift into fear-based or definitive predictions, which is both irresponsible and a poor experience.
Solution
The assistant's system prompt explicitly bans deterministic predictions, medical/legal/financial advice and superstition, requiring hedged, psychology-oriented language across both the astrology and palmistry interpretation.
Impact: Readings stay grounded and supportive by design, not by luck.
Computer-vision validation, custom feature detection, ephemeris-based astrology and LLM synthesis were all built and integrated end-to-end.
Both Vedic (sidereal) and Western (tropical) placements, plus ascendant and moon nakshatra, are computed directly from raw date, time and location.
A React internal tool let the whole flow — birth data entry, palm upload, chart visualization, AI chat — be tested end-to-end before investing in the Flutter app.
The engagement paused with the backend and AI pipeline validated; the Flutter consumer app remains an unstarted scaffold.
Building something similar?
Let's Talk