Looking for a senior backend engineer to develop a scalable web scraping service using Python, PostgreSQL, and AWS.
The service will be part of an AI-powered chatbot system using RAG (Retrieval Augmented Generation) architecture.
What You Need to Do: Fresh Scraping Service Implementation with PostgreSQL Main Application (Django) -- SQS Queue -- Scraping Service -- S3 Storage -- SNS Notifications -- Result Processing -- PostgreSQL (RDS) -- Vector DB (pgvector) Initial Infrastructure Setup - AWS Components Configuration PostgreSQL Initial Setup - Base Configuration Service Implementation - Django Settings Improved Scraping Service Required Skills: Python (Django/FastAPI) with async programming PostgreSQL 15+ (performance tuning, connection pooling) AWS (ECS, RDS PostgreSQL, SQS, SNS) Playwright/Web Scraping at scale Docker containerization Experience with high-throughput web services Project Scope: Develop a separate scraping service that integrates with existing RAG architecture - you must create a stand-alone service and output will be markdown format vector db (RAG ready) Implement efficient PostgreSQL database design and optimization Set up AWS infrastructure with proper scaling and monitoring Ensure zero impact on main application performance Deliverables: Complete scraping service implementation (you don't need to write scraping code, we will provide it) AWS infrastructure setup including Monitoring and logging Documentation and deployment guides Knowledge transfer sessions Current System Flow: Web scraping - Markdown conversion - Data cleansing - Embedding
Duration: 1 week
Budget: Fixed rate to be discussed.
Shortlisted candidates will receive a detailed job description.
Please only apply if you have 10+ years of experience with the above-mentioned tools.
We will provide you with a Current Issues Analysis showing issues we have: Database Connection Problems, Resource Management Issues in Scraping Service, System Architecture issues, etc., so you don't repeat the same mistakes.
However, your job must come with a guarantee.
#J-18808-Ljbffr