Looking for a senior backend engineer to develop a scalable web scraping service using Python, PostgreSQL, and AWS.
The service will be part of an AI-powered chatbot system using RAG (Retrieval Augmented Generation) architecture.
What You Need to Do:Fresh Scraping Service Implementation with PostgreSQLMain Application (Django) -- SQS Queue -- Scraping Service -- S3 Storage -- SNS Notifications -- Result Processing -- PostgreSQL (RDS) -- Vector DB (pgvector)Initial Infrastructure Setup - AWS Components ConfigurationPostgreSQL Initial Setup - Base ConfigurationService Implementation - Django SettingsImproved Scraping ServiceRequired Skills:Python (Django/FastAPI) with async programmingPostgreSQL 15+ (performance tuning, connection pooling)AWS (ECS, RDS PostgreSQL, SQS, SNS)Playwright/Web Scraping at scaleDocker containerizationExperience with high-throughput web servicesProject Scope:Develop a separate scraping service that integrates with existing RAG architecture - you must create a stand-alone service and output will be markdown format vector db (RAG ready)Implement efficient PostgreSQL database design and optimizationSet up AWS infrastructure with proper scaling and monitoringEnsure zero impact on main application performanceDeliverables:Complete scraping service implementation (you don't need to write scraping code, we will provide it)AWS infrastructure setup including Monitoring and loggingDocumentation and deployment guidesKnowledge transfer sessionsCurrent System Flow:Web scraping - Markdown conversion - Data cleansing - Embedding
Duration:1 week
Budget:Fixed rate to be discussed.
Shortlisted candidates will receive a detailed job description.
Please only apply if you have 10+ years of experience with the above-mentioned tools.
We will provide you with a Current Issues Analysis showing issues we have: Database Connection Problems, Resource Management Issues in Scraping Service, System Architecture issues, etc., so you don't repeat the same mistakes.
However, your job must come with a guarantee.
#J-18808-Ljbffr