86 lines
3.6 KiB
Markdown
86 lines
3.6 KiB
Markdown
# Direct Parser Source Ingestion Implementation Plan
|
|
|
|
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
|
|
|
**Goal:** Move parser runtime reads and writes from legacy parser record tables to organization source storage.
|
|
|
|
**Architecture:** Add a focused ingestion service in `organizations` that persists normalized source-record inputs directly into polymorphic source extensions. Parser services become adapters from parser dataclasses to ingestion inputs. Runtime reads use `OrganizationSourceRecord` and extension counters.
|
|
|
|
**Tech Stack:** Django 3.2, PostgreSQL, DRF, django-polymorphic, pytest.
|
|
|
|
---
|
|
|
|
### Task 1: Direct Ingestion Core
|
|
|
|
**Files:**
|
|
- Create: `src/organizations/source_identity.py`
|
|
- Create: `src/organizations/source_ingestion.py`
|
|
- Modify: `src/organizations/source_backfill.py`
|
|
- Test: `tests/apps/organizations/test_source_ingestion.py`
|
|
- Test: `tests/apps/organizations/test_source_backfill.py`
|
|
|
|
- [ ] Write failing tests for direct generic source ingestion.
|
|
- [ ] Write failing tests for FNS report ingestion with financial lines.
|
|
- [ ] Extract identity normalization from backfill into a shared helper.
|
|
- [ ] Implement `SourceRecordInput` and `SourceFinancialLineInput`.
|
|
- [ ] Implement `OrganizationSourceIngestionService.save_records`.
|
|
- [ ] Keep backfill behavior green by using the same identity normalization helper.
|
|
|
|
### Task 2: Parser Save Services
|
|
|
|
**Files:**
|
|
- Modify: `src/apps/parsers/services.py`
|
|
- Test: `tests/apps/parsers/test_services.py`
|
|
|
|
- [ ] Switch generic source saves to `OrganizationSourceIngestionService`.
|
|
- [ ] Switch industrial certificate/manufacturer/product saves.
|
|
- [ ] Switch inspection and procurement saves.
|
|
- [ ] Switch FNS report saves and duplicate checks.
|
|
- [ ] Replace period/deduplication helpers with source-record queries.
|
|
|
|
### Task 3: Parser Tasks
|
|
|
|
**Files:**
|
|
- Modify: `src/apps/parsers/tasks.py`
|
|
- Test: `tests/apps/parsers/test_tasks.py`
|
|
|
|
- [ ] Remove source backfill queueing from parser completion.
|
|
- [ ] Keep parser load logs and background job progress unchanged.
|
|
- [ ] Return source-record identifiers for FNS processing instead of legacy report ids.
|
|
|
|
### Task 4: Runtime Reads
|
|
|
|
**Files:**
|
|
- Modify: `src/apps/parsers/source_cards.py`
|
|
- Modify: `src/apps/parsers/views.py`
|
|
- Modify: `src/apps/parsers/serializers.py`
|
|
- Modify: `src/apps/core/admin_dashboard.py`
|
|
- Modify: `src/apps/backups/services.py`
|
|
- Test: parser source-card and result endpoint tests.
|
|
|
|
- [ ] Move source card counts and timestamps to source extensions/source records.
|
|
- [ ] Move parser log organization counts to source records.
|
|
- [ ] Adapt v1 parser result endpoints to read source records.
|
|
- [ ] Move dashboard/export runtime reads off legacy parser models.
|
|
|
|
### Task 5: Frontend Record Detail
|
|
|
|
**Files:**
|
|
- Modify: `mostovik-frontend/src/pages/main/model/source-record-detail/*`
|
|
- Test: frontend source-detail/source-record-detail unit tests.
|
|
|
|
- [ ] Replace legacy generated v1 detail clients with organization source-record reads.
|
|
- [ ] Use `payload` plus top-level source-record fields for detail rendering.
|
|
- [ ] Keep source-detail lists on the new source-record list endpoint.
|
|
|
|
### Task 6: Validation
|
|
|
|
**Files:**
|
|
- No production files.
|
|
|
|
- [ ] Run focused backend parser/organization tests.
|
|
- [ ] Run frontend source-detail/source-record-detail checks.
|
|
- [ ] Run live parser smoke against one small generic source.
|
|
- [ ] Confirm legacy parser record counts do not change during the smoke.
|
|
- [ ] Confirm new organization source-record counts do change.
|