feat(organizations): migrate source storage to polymorphic records
This commit is contained in:
@@ -0,0 +1,85 @@
|
||||
# Direct Parser Source Ingestion Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** Move parser runtime reads and writes from legacy parser record tables to organization source storage.
|
||||
|
||||
**Architecture:** Add a focused ingestion service in `organizations` that persists normalized source-record inputs directly into polymorphic source extensions. Parser services become adapters from parser dataclasses to ingestion inputs. Runtime reads use `OrganizationSourceRecord` and extension counters.
|
||||
|
||||
**Tech Stack:** Django 3.2, PostgreSQL, DRF, django-polymorphic, pytest.
|
||||
|
||||
---
|
||||
|
||||
### Task 1: Direct Ingestion Core
|
||||
|
||||
**Files:**
|
||||
- Create: `src/organizations/source_identity.py`
|
||||
- Create: `src/organizations/source_ingestion.py`
|
||||
- Modify: `src/organizations/source_backfill.py`
|
||||
- Test: `tests/apps/organizations/test_source_ingestion.py`
|
||||
- Test: `tests/apps/organizations/test_source_backfill.py`
|
||||
|
||||
- [ ] Write failing tests for direct generic source ingestion.
|
||||
- [ ] Write failing tests for FNS report ingestion with financial lines.
|
||||
- [ ] Extract identity normalization from backfill into a shared helper.
|
||||
- [ ] Implement `SourceRecordInput` and `SourceFinancialLineInput`.
|
||||
- [ ] Implement `OrganizationSourceIngestionService.save_records`.
|
||||
- [ ] Keep backfill behavior green by using the same identity normalization helper.
|
||||
|
||||
### Task 2: Parser Save Services
|
||||
|
||||
**Files:**
|
||||
- Modify: `src/apps/parsers/services.py`
|
||||
- Test: `tests/apps/parsers/test_services.py`
|
||||
|
||||
- [ ] Switch generic source saves to `OrganizationSourceIngestionService`.
|
||||
- [ ] Switch industrial certificate/manufacturer/product saves.
|
||||
- [ ] Switch inspection and procurement saves.
|
||||
- [ ] Switch FNS report saves and duplicate checks.
|
||||
- [ ] Replace period/deduplication helpers with source-record queries.
|
||||
|
||||
### Task 3: Parser Tasks
|
||||
|
||||
**Files:**
|
||||
- Modify: `src/apps/parsers/tasks.py`
|
||||
- Test: `tests/apps/parsers/test_tasks.py`
|
||||
|
||||
- [ ] Remove source backfill queueing from parser completion.
|
||||
- [ ] Keep parser load logs and background job progress unchanged.
|
||||
- [ ] Return source-record identifiers for FNS processing instead of legacy report ids.
|
||||
|
||||
### Task 4: Runtime Reads
|
||||
|
||||
**Files:**
|
||||
- Modify: `src/apps/parsers/source_cards.py`
|
||||
- Modify: `src/apps/parsers/views.py`
|
||||
- Modify: `src/apps/parsers/serializers.py`
|
||||
- Modify: `src/apps/core/admin_dashboard.py`
|
||||
- Modify: `src/apps/backups/services.py`
|
||||
- Test: parser source-card and result endpoint tests.
|
||||
|
||||
- [ ] Move source card counts and timestamps to source extensions/source records.
|
||||
- [ ] Move parser log organization counts to source records.
|
||||
- [ ] Adapt v1 parser result endpoints to read source records.
|
||||
- [ ] Move dashboard/export runtime reads off legacy parser models.
|
||||
|
||||
### Task 5: Frontend Record Detail
|
||||
|
||||
**Files:**
|
||||
- Modify: `mostovik-frontend/src/pages/main/model/source-record-detail/*`
|
||||
- Test: frontend source-detail/source-record-detail unit tests.
|
||||
|
||||
- [ ] Replace legacy generated v1 detail clients with organization source-record reads.
|
||||
- [ ] Use `payload` plus top-level source-record fields for detail rendering.
|
||||
- [ ] Keep source-detail lists on the new source-record list endpoint.
|
||||
|
||||
### Task 6: Validation
|
||||
|
||||
**Files:**
|
||||
- No production files.
|
||||
|
||||
- [ ] Run focused backend parser/organization tests.
|
||||
- [ ] Run frontend source-detail/source-record-detail checks.
|
||||
- [ ] Run live parser smoke against one small generic source.
|
||||
- [ ] Confirm legacy parser record counts do not change during the smoke.
|
||||
- [ ] Confirm new organization source-record counts do change.
|
||||
@@ -0,0 +1,100 @@
|
||||
# Polymorphic Organization Sources Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** Replace source-centric parser output access with organization-centric polymorphic source extensions.
|
||||
|
||||
**Architecture:** Keep `Organization` as the root entity. Add polymorphic source extensions per product source group and a shared subordinate source-record table. Backfill legacy parser tables idempotently, then switch API v2 to the new extension data.
|
||||
|
||||
**Tech Stack:** Django 3.2, Django REST Framework, django-filter, django-polymorphic, PostgreSQL, pytest.
|
||||
|
||||
---
|
||||
|
||||
### Task 1: Dependency And Schema
|
||||
|
||||
**Files:**
|
||||
- Modify: `pyproject.toml`
|
||||
- Modify: `uv.lock`
|
||||
- Modify: `src/settings/base.py`
|
||||
- Modify: `src/organizations/models.py`
|
||||
- Create: `src/organizations/migrations/0006_polymorphic_source_extensions.py`
|
||||
- Test: `tests/apps/organizations/test_source_extensions_models.py`
|
||||
|
||||
- [ ] Add `django-polymorphic` to project dependencies.
|
||||
- [ ] Add `"polymorphic"` to `INSTALLED_APPS` before local apps.
|
||||
- [ ] Add source-group and identity-status choices.
|
||||
- [ ] Add `identity_status` and `primary_identity` to `Organization`.
|
||||
- [ ] Add `OrganizationSourceExtension` as `PolymorphicModel`.
|
||||
- [ ] Add source extension subclasses.
|
||||
- [ ] Add `OrganizationSourceRecord`.
|
||||
- [ ] Add `OrganizationSourceFinancialLine`.
|
||||
- [ ] Write tests proving:
|
||||
- one extension per `(organization, source_group)`;
|
||||
- polymorphic queries return subclass instances;
|
||||
- source records are unique by legacy model/pk;
|
||||
- financial lines attach to a source record.
|
||||
|
||||
### Task 2: Backfill Service
|
||||
|
||||
**Files:**
|
||||
- Create: `src/organizations/source_groups.py`
|
||||
- Create: `src/organizations/source_backfill.py`
|
||||
- Create: `src/organizations/management/commands/backfill_organization_sources.py`
|
||||
- Test: `tests/apps/organizations/test_source_backfill.py`
|
||||
|
||||
- [ ] Define source group mapping for all legacy parser sources.
|
||||
- [ ] Implement organization resolution by `inn + kpp`, `ogrn`, `ogrip`, unique `inn`, then normalized name.
|
||||
- [ ] Implement idempotent extension creation/update.
|
||||
- [ ] Implement idempotent source record creation/update.
|
||||
- [ ] Preserve legacy row payload and `(legacy_model, legacy_pk)`.
|
||||
- [ ] Backfill financial report lines into `OrganizationSourceFinancialLine`.
|
||||
- [ ] Report scanned, created organizations, created extensions, updated extensions, created records, updated records, unresolved rows.
|
||||
|
||||
### Task 3: API v2 Switch
|
||||
|
||||
**Files:**
|
||||
- Modify: `src/organizations/serializers.py`
|
||||
- Modify: `src/organizations/filters.py`
|
||||
- Modify: `src/organizations/views.py`
|
||||
- Delete or stop using: `src/organizations/api_enrichment.py`
|
||||
- Delete or stop using: `src/organizations/services.py` snapshot refresh paths
|
||||
- Test: `tests/apps/organizations/test_api_v2.py`
|
||||
|
||||
- [ ] Replace embedded `data` JSON with compact `sources`.
|
||||
- [ ] Add source extension list/detail serializers.
|
||||
- [ ] Add source records endpoint.
|
||||
- [ ] Rework source filters to use `OrganizationSourceExtension`.
|
||||
- [ ] Remove snapshot dependency from list/retrieve behavior.
|
||||
- [ ] Keep old snapshot management command only as deprecated/no-op until cleanup.
|
||||
|
||||
### Task 4: Parser Write Path
|
||||
|
||||
**Files:**
|
||||
- Modify: `src/apps/parsers/tasks.py`
|
||||
- Modify: `src/organizations/tasks.py`
|
||||
- Test: `tests/apps/parsers/test_tasks.py`
|
||||
- Test: `tests/apps/organizations/test_tasks.py`
|
||||
|
||||
- [ ] Replace snapshot refresh queueing with source backfill queueing for affected parser batches.
|
||||
- [ ] For each parser completion, backfill only the completed source/batch.
|
||||
- [ ] Keep full backfill command for initial migration and repair.
|
||||
|
||||
### Task 5: Frontend Contract Repair
|
||||
|
||||
**Files:**
|
||||
- Modify frontend generated API clients after backend OpenAPI changes.
|
||||
- Modify source detail table composables to consume `sources` and source records endpoints.
|
||||
|
||||
- [ ] Regenerate API client.
|
||||
- [ ] Update source pages to request extension records instead of embedded `organization.data[source]`.
|
||||
- [ ] Verify planned inspections page loads from source records.
|
||||
|
||||
### Task 6: Cleanup Phase
|
||||
|
||||
**Files:**
|
||||
- Modify migrations only after successful backfill validation.
|
||||
|
||||
- [ ] Remove `OrganizationDataSnapshot`.
|
||||
- [ ] Remove snapshot refresh schedules.
|
||||
- [ ] Decide which legacy parser tables remain as ingestion staging and which can be dropped.
|
||||
- [ ] Run full backend and frontend validation.
|
||||
Reference in New Issue
Block a user