Compare commits
4 Commits
hotfix/int
...
feature/mo
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
fedd2e00a3 | ||
| 887442e8e3 | |||
|
|
cd5a1764a8 | ||
| 245edd1fff |
15
docs/all-years-source-exports.md
Normal file
15
docs/all-years-source-exports.md
Normal file
@@ -0,0 +1,15 @@
|
||||
# Выгрузка за все годы
|
||||
|
||||
Годовая задача `organizations.tasks.refresh_source_record_export_artifacts` сохраняет прежнее поведение.
|
||||
|
||||
Новая задача `organizations.tasks.refresh_all_years_source_record_export_artifacts` собирает все доступные записи каждого поддерживаемого источника без ограничения по году, включая все финансовые строки. Для источников с заменяемыми snapshot выгружается текущее опубликованное состояние: удалённые прежние snapshot не восстанавливаются.
|
||||
|
||||
Результат размещается в `SOURCE_RECORD_EXPORT_DIRECTORY/all-years`. У него отдельные `current.json`, `generations` и очистка старых поколений. Публикация атомарна; незавершённая сборка не заменяет опубликованные файлы. Общий lock с годовой задачей ограничивает одновременную нагрузку. Manifest содержит `export_scope: all_years` и `export_year: null`; годовой manifest — `export_scope: current_year` и числовой год. Старые годовые manifest без `export_scope` продолжают читаться.
|
||||
|
||||
Ручной запуск:
|
||||
|
||||
```sh
|
||||
python src/manage.py build_source_record_exports --all-years
|
||||
```
|
||||
|
||||
Периодический запуск настраивается отдельной записью django-celery-beat либо системным таймером на контуре без Beat. Задачу следует ставить после годовой сборки. Имена файлов, группировка JSON по ОКПО и форматы CSV/XLSX/JSON сохраняются. API скачивания продолжает отдавать годовой пакет; доступ к полному пакету предоставляется отдельно.
|
||||
52
docs/dashboard-performance.md
Normal file
52
docs/dashboard-performance.md
Normal file
@@ -0,0 +1,52 @@
|
||||
# Производительность главной страницы
|
||||
|
||||
Главная страница использует `/api/v1/sources/` и `/api/v1/parsers/dashboard/`.
|
||||
Счетчики, состав задач, порядок запусков и поля ответов остаются прежними.
|
||||
|
||||
## Изменения
|
||||
|
||||
- PostgreSQL выбирает последнюю фоновую задачу для каждого `task_name` один раз
|
||||
через `DISTINCT ON`. Сортировка сохраняется: `created_at DESC, id DESC`.
|
||||
Выбор последних задач выполняется до фильтрации карточки; более старые
|
||||
активные задачи и завершенные части составного запуска остаются видимыми.
|
||||
Для SQLite сохранен совместимый вариант с подзапросом.
|
||||
- Аналитика повторно использует соответствия организаций источникам и последние
|
||||
логи в пределах одного HTTP-запроса. Между пользователями результаты не кешируются.
|
||||
- Метаданные снимка используют сохраненный `snapshot_organizations_count`,
|
||||
включая ноль. Для старых снимков без допустимого итога выполняется один
|
||||
групповой подсчет уникальных опубликованных организаций. Отклоненные строки
|
||||
и строки без организации в итог не входят. Подсчет нужен только для выбранной
|
||||
последней версии снимка каждого запуска.
|
||||
|
||||
Новых настроек, миграций и правил инвалидирования кеша нет.
|
||||
|
||||
## Проверка на данных dev, 1 октября 2026
|
||||
|
||||
| Обработчик | Кеш агрегатов | До | После |
|
||||
| --- | --- | ---: | ---: |
|
||||
| `/sources/` | Пустой | 13,200 с | 3,453 с |
|
||||
| `/sources/` | Заполненный | 10,191 с | 0,243 с |
|
||||
| `/parsers/dashboard/` | Пустой | 4,165 с | 2,864 с |
|
||||
| `/parsers/dashboard/` | Заполненный | 4,246 с | 2,880 с |
|
||||
|
||||
Это время обработчиков в отдельном процессе, без сети и аутентификационного
|
||||
middleware. Сравнение выполнено на одном снимке БД в транзакции
|
||||
`REPEATABLE READ, READ ONLY`, с фиксированным временем и отдельным кешем в памяти.
|
||||
Полные JSON-ответы совпали во всех четырех сравнениях. Учетные данные
|
||||
пользователей не использовались. Это диагностическое сравнение, не нагрузочный
|
||||
тест и не подтверждение выкатки.
|
||||
|
||||
Оставшееся время первого запроса связано с подсчетом записей источников.
|
||||
Dashboard по-прежнему рассчитывает актуальные счетчики при каждом запросе.
|
||||
|
||||
## Регрессионные проверки
|
||||
|
||||
- `test_refresh_progress.py`: порядок по времени и UUID, старые активные задачи,
|
||||
параллельные и составные запуски, глобальный выбор последней задачи.
|
||||
- `test_snapshot_metadata.py`: сохраненные и старые счетчики, нулевой итог,
|
||||
повторные организации, неопубликованные строки, выбор последнего снимка.
|
||||
- `test_views.py`: значения аналитики и отсутствие повторных обходов источников.
|
||||
- `test_refresh_concurrency.py`: проверка блокировок на PostgreSQL.
|
||||
|
||||
Проверки PostgreSQL запускаются с `--ds=settings.test_postgres --migrations`
|
||||
и отдельной тестовой БД. SQLite не проверяет `DISTINCT ON` и блокировки PostgreSQL.
|
||||
@@ -1,33 +1,41 @@
|
||||
"""Public snapshot provenance shared by cards, dashboard and parser history."""
|
||||
|
||||
from apps.parsers.models import ParserSourceArtifact
|
||||
from django.db.models import Count, Q
|
||||
from apps.parsers.models import ParserSourceArtifact, ParserStagedRecord
|
||||
from django.db.models import Count
|
||||
|
||||
|
||||
def attach_snapshot_artifacts(logs) -> None:
|
||||
"""Attach the latest artifact for each load in one query, without raw data."""
|
||||
"""Attach provenance, counting staged rows only for legacy snapshots."""
|
||||
logs = list(logs)
|
||||
if not logs:
|
||||
return
|
||||
artifacts = {}
|
||||
for artifact in (
|
||||
ParserSourceArtifact.objects.filter(
|
||||
source__in={log.source for log in logs},
|
||||
load_batch__in={log.batch_id for log in logs},
|
||||
)
|
||||
.annotate(
|
||||
published_organizations_count=Count(
|
||||
"staged_records__organization_id",
|
||||
distinct=True,
|
||||
filter=Q(staged_records__disposition="published"),
|
||||
)
|
||||
)
|
||||
.order_by("-created_at", "-uid")
|
||||
):
|
||||
for artifact in ParserSourceArtifact.objects.filter(
|
||||
source__in={log.source for log in logs},
|
||||
load_batch__in={log.batch_id for log in logs},
|
||||
).order_by("-created_at", "-uid"):
|
||||
artifacts.setdefault((artifact.source, artifact.load_batch), artifact)
|
||||
legacy_artifacts = []
|
||||
for artifact in artifacts.values():
|
||||
total = artifact.metadata.get("snapshot_organizations_count")
|
||||
if isinstance(total, int) and total >= 0:
|
||||
artifact.published_organizations_count = total
|
||||
artifacts.setdefault((artifact.source, artifact.load_batch), artifact)
|
||||
else:
|
||||
legacy_artifacts.append(artifact.pk)
|
||||
counts = {}
|
||||
if legacy_artifacts:
|
||||
counts = dict(
|
||||
ParserStagedRecord.objects.filter(
|
||||
artifact_id__in=legacy_artifacts, disposition="published"
|
||||
)
|
||||
.order_by()
|
||||
.values("artifact_id")
|
||||
.annotate(total=Count("organization_id", distinct=True))
|
||||
.values_list("artifact_id", "total")
|
||||
)
|
||||
for artifact in artifacts.values():
|
||||
if artifact.pk in legacy_artifacts:
|
||||
artifact.published_organizations_count = counts.get(artifact.pk, 0)
|
||||
for log in logs:
|
||||
log._source_artifact_cache = artifacts.get((log.source, log.batch_id))
|
||||
|
||||
|
||||
@@ -27,7 +27,7 @@ from apps.parsers.snapshot_metadata import (
|
||||
from apps.parsers.source_registry import get_source_by_model_source
|
||||
from django.conf import settings
|
||||
from django.core.cache import cache
|
||||
from django.db import transaction
|
||||
from django.db import connections, transaction
|
||||
from django.db.models import Count, Max, OuterRef, Q, Subquery
|
||||
from django.http import Http404
|
||||
from django.utils import timezone
|
||||
@@ -1033,19 +1033,28 @@ class SourceCardService:
|
||||
for task_name in definition.task_names:
|
||||
slugs_by_task_name.setdefault(task_name, []).append(definition.slug)
|
||||
slugs = {definition.slug for definition in definitions}
|
||||
latest = (
|
||||
BackgroundJobService.get_queryset()
|
||||
.filter(task_name=OuterRef("task_name"))
|
||||
.order_by("-created_at", "-id")
|
||||
.values("pk")[:1]
|
||||
)
|
||||
all_jobs = BackgroundJobService.get_queryset()
|
||||
if connections[all_jobs.db].features.can_distinct_on_fields:
|
||||
# Find each task's latest job once, rather than once per history row.
|
||||
latest = all_jobs.order_by("task_name", "-created_at", "-id").distinct(
|
||||
"task_name"
|
||||
)
|
||||
else:
|
||||
latest_for_task = (
|
||||
all_jobs.filter(task_name=OuterRef("task_name"))
|
||||
.order_by("-created_at", "-id")
|
||||
.values("pk")[:1]
|
||||
)
|
||||
latest = all_jobs.filter(pk=Subquery(latest_for_task)).order_by()
|
||||
jobs = list(
|
||||
BackgroundJobService.get_queryset()
|
||||
.filter(
|
||||
Q(task_name__in=slugs_by_task_name)
|
||||
| Q(meta__source_card__in=list(slugs))
|
||||
)
|
||||
.filter(cls._fresh_active_job_filter() | Q(pk=Subquery(latest)))
|
||||
.filter(
|
||||
cls._fresh_active_job_filter() | Q(pk__in=Subquery(latest.values("pk")))
|
||||
)
|
||||
.order_by("-created_at", "-id")
|
||||
)
|
||||
active_by_slug: dict[str, list[dict[str, Any]]] = {}
|
||||
|
||||
@@ -2242,8 +2242,9 @@ def _registry_source_matrix_rows(
|
||||
return matrix_rows
|
||||
|
||||
|
||||
def _registry_data_coverage() -> dict:
|
||||
registry_organization_ids, by_inn, by_ogrn = _active_registry_organization_indexes()
|
||||
def _registry_data_coverage(
|
||||
registry_organization_ids: set[str], source_matches: dict[str, set[str]]
|
||||
) -> dict:
|
||||
total_organizations = len(registry_organization_ids)
|
||||
descriptor_titles: dict[str, str] = {}
|
||||
for descriptor in PARSER_SOURCES.values():
|
||||
@@ -2254,9 +2255,7 @@ def _registry_data_coverage() -> dict:
|
||||
for source, _label in ParserLoadLog.Source.choices:
|
||||
if source in REGISTRY_COVERAGE_EXCLUDED_SOURCES:
|
||||
continue
|
||||
organizations_count = len(
|
||||
_matched_source_record_registry_organization_ids(source, by_inn, by_ogrn)
|
||||
)
|
||||
organizations_count = len(source_matches.get(source, set()))
|
||||
coverage_percent = (
|
||||
round(organizations_count / total_organizations * 100, 1)
|
||||
if total_organizations
|
||||
@@ -2277,10 +2276,14 @@ def _registry_data_coverage() -> dict:
|
||||
}
|
||||
|
||||
|
||||
def _registry_enrichment_analytics(jobs, schedules: list[dict]) -> dict:
|
||||
registry_organization_ids, by_inn, by_ogrn = _active_registry_organization_indexes()
|
||||
def _registry_enrichment_analytics(
|
||||
jobs,
|
||||
schedules: list[dict],
|
||||
registry_organization_ids: set[str],
|
||||
source_matches: dict[str, set[str]],
|
||||
latest_loads: list[ParserLoadLog],
|
||||
) -> dict:
|
||||
total_organizations = len(registry_organization_ids)
|
||||
source_matches = _source_registry_matches(by_inn, by_ogrn)
|
||||
descriptor_titles: dict[str, str] = {}
|
||||
for descriptor in PARSER_SOURCES.values():
|
||||
descriptor_titles.setdefault(descriptor.source, descriptor.title)
|
||||
@@ -2369,7 +2372,6 @@ def _registry_enrichment_analytics(jobs, schedules: list[dict]) -> dict:
|
||||
enrichment_sources=enrichment_sources,
|
||||
)
|
||||
|
||||
latest_loads = list(ParserLoadLog.objects.all().order_by("-created_at")[:30])
|
||||
recent_success = sum(
|
||||
1 for load in latest_loads if load.status == ParserLoadLog.Status.SUCCESS
|
||||
)
|
||||
@@ -3459,6 +3461,8 @@ class ParserDashboardDataView(APIView):
|
||||
]
|
||||
load_logs = list(ParserLoadLog.objects.all().order_by("-created_at")[:30])
|
||||
attach_snapshot_artifacts(load_logs)
|
||||
registry_ids, by_inn, by_ogrn = _active_registry_organization_indexes()
|
||||
source_matches = _source_registry_matches(by_inn, by_ogrn)
|
||||
return api_response(
|
||||
{
|
||||
"sources": sources,
|
||||
@@ -3471,10 +3475,15 @@ class ParserDashboardDataView(APIView):
|
||||
"schedules": ParserScheduleSerializer(schedules, many=True).data,
|
||||
"jobs": BackgroundJobListSerializer(jobs, many=True).data,
|
||||
"source_counts": source_counts,
|
||||
"registry_data_coverage": _registry_data_coverage(),
|
||||
"registry_data_coverage": _registry_data_coverage(
|
||||
registry_ids, source_matches
|
||||
),
|
||||
"registry_enrichment_analytics": _registry_enrichment_analytics(
|
||||
jobs,
|
||||
schedules,
|
||||
registry_ids,
|
||||
source_matches,
|
||||
load_logs,
|
||||
),
|
||||
"load_logs": ParserLoadLogSerializer(
|
||||
load_logs,
|
||||
|
||||
@@ -15,8 +15,18 @@ class Command(BaseAppCommand):
|
||||
help = "Формирует готовые CSV/XLSX/JSON выгрузки источников"
|
||||
use_transaction = False
|
||||
|
||||
def add_arguments(self, parser) -> None:
|
||||
super().add_arguments(parser)
|
||||
parser.add_argument(
|
||||
"--all-years",
|
||||
action="store_true",
|
||||
help="Все доступные годы в отдельном каталоге all-years",
|
||||
)
|
||||
|
||||
def execute_command(self, *args, **options) -> str:
|
||||
generation = build_source_record_export_artifacts()
|
||||
generation = build_source_record_export_artifacts(
|
||||
all_years=options["all_years"]
|
||||
)
|
||||
rendered = json.dumps(
|
||||
{
|
||||
"generation_id": generation.generation_id,
|
||||
|
||||
@@ -195,7 +195,7 @@ class SourceRecordExportGeneration:
|
||||
|
||||
generation_id: str
|
||||
generated_at: str
|
||||
export_year: int
|
||||
export_year: int | None
|
||||
artifacts: tuple[SourceRecordExportArtifact, ...]
|
||||
records_count: int
|
||||
source_snapshots: dict[str, str] = field(default_factory=dict)
|
||||
@@ -267,16 +267,19 @@ def build_source_record_export_artifacts(
|
||||
*,
|
||||
now: datetime | None = None,
|
||||
export_directory: str | Path | None = None,
|
||||
all_years: bool = False,
|
||||
) -> SourceRecordExportGeneration:
|
||||
"""Build all files on disk and atomically publish a new generation."""
|
||||
"""Build annual or all-years files and atomically publish their generation."""
|
||||
|
||||
root_directory = _resolve_export_directory(export_directory)
|
||||
if all_years:
|
||||
root_directory /= "all-years"
|
||||
generations_directory = root_directory / GENERATION_DIRECTORY_NAME
|
||||
root_directory.mkdir(parents=True, exist_ok=True)
|
||||
generations_directory.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
generated_at_datetime = _normalize_generation_datetime(now or timezone.now())
|
||||
export_year = _export_year(generated_at_datetime)
|
||||
export_year = None if all_years else _export_year(generated_at_datetime)
|
||||
generation_id = (
|
||||
f"{generated_at_datetime.strftime('%Y%m%dT%H%M%SZ')}-{uuid4().hex[:8]}"
|
||||
)
|
||||
@@ -739,7 +742,7 @@ def _build_source_group_file_name(*, source_group: str, file_format: str) -> str
|
||||
def _source_group_queryset(
|
||||
source_group: str,
|
||||
*,
|
||||
export_year: int,
|
||||
export_year: int | None,
|
||||
) -> QuerySet[OrganizationSourceRecord]:
|
||||
queryset = (
|
||||
OrganizationSourceRecord.objects.filter(extension__source_group=source_group)
|
||||
@@ -749,6 +752,8 @@ def _source_group_queryset(
|
||||
# source groups with millions of rows.
|
||||
.order_by()
|
||||
)
|
||||
if export_year is None:
|
||||
return queryset
|
||||
if source_group == FINANCIAL_SOURCE_GROUP:
|
||||
return queryset.filter(financial_lines__year=export_year).distinct()
|
||||
if source_group in ALL_HISTORY_SOURCE_GROUPS:
|
||||
@@ -777,7 +782,7 @@ def _iter_source_records(
|
||||
*,
|
||||
source_group: str,
|
||||
include_financial_lines: bool,
|
||||
export_year: int,
|
||||
export_year: int | None,
|
||||
) -> Iterator[OrganizationSourceRecord]:
|
||||
iterator = _source_group_queryset(
|
||||
source_group,
|
||||
@@ -793,7 +798,7 @@ def _iter_source_records(
|
||||
Prefetch(
|
||||
"financial_lines",
|
||||
queryset=OrganizationSourceFinancialLine.objects.filter(
|
||||
year=export_year
|
||||
**({"year": export_year} if export_year is not None else {})
|
||||
).order_by(),
|
||||
),
|
||||
)
|
||||
@@ -804,7 +809,7 @@ def _spool_source_group_rows(
|
||||
*,
|
||||
source_group: str,
|
||||
output_path: Path,
|
||||
export_year: int,
|
||||
export_year: int | None,
|
||||
) -> tuple[list[str], int]:
|
||||
include_financial_lines = source_group == FINANCIAL_SOURCE_GROUP
|
||||
payload_headers: set[str] = set()
|
||||
@@ -1373,6 +1378,9 @@ def _generation_manifest_payload(
|
||||
"source_snapshots": generation.source_snapshots,
|
||||
"generated_at": generation.generated_at,
|
||||
"export_year": generation.export_year,
|
||||
"export_scope": "all_years"
|
||||
if generation.export_year is None
|
||||
else "current_year",
|
||||
"records_count": generation.records_count,
|
||||
"artifacts_count": generation.artifacts_count,
|
||||
"files_count": generation.files_count,
|
||||
@@ -1403,6 +1411,19 @@ def _manifest_snapshot_ids(payload: dict) -> dict[str, str]:
|
||||
return identities
|
||||
|
||||
|
||||
def _manifest_export_year(payload: dict[str, Any]) -> int | None:
|
||||
raw_year = payload["export_year"]
|
||||
scope = payload.get("export_scope", "current_year")
|
||||
if scope == "all_years" and raw_year is None:
|
||||
return None
|
||||
if scope != "current_year" or raw_year is None or isinstance(raw_year, bool):
|
||||
raise ValueError("Source-record export scope is invalid.")
|
||||
year = int(raw_year)
|
||||
if not 1 <= year <= 9999:
|
||||
raise ValueError("Source-record export year is invalid.")
|
||||
return year
|
||||
|
||||
|
||||
def _generation_from_manifest(
|
||||
payload: Any,
|
||||
*,
|
||||
@@ -1415,9 +1436,7 @@ def _generation_from_manifest(
|
||||
source_snapshots = _manifest_snapshot_ids(payload)
|
||||
generated_at = str(payload["generated_at"])
|
||||
datetime.fromisoformat(generated_at)
|
||||
export_year = int(payload["export_year"])
|
||||
if not 1 <= export_year <= 9999:
|
||||
raise ValueError("Source-record export year is invalid.")
|
||||
export_year = _manifest_export_year(payload)
|
||||
records_count = int(payload["records_count"])
|
||||
artifact_payloads = payload["artifacts"]
|
||||
if not isinstance(artifact_payloads, list):
|
||||
|
||||
@@ -22,6 +22,16 @@ logger = logging.getLogger(__name__)
|
||||
@shared_task(bind=True, base=CorePeriodicTask)
|
||||
def refresh_source_record_export_artifacts(self) -> dict: # noqa: ARG001
|
||||
"""Build and atomically publish the nightly source-record export matrix."""
|
||||
return _refresh_source_record_export_artifacts(self, all_years=False)
|
||||
|
||||
|
||||
@shared_task(bind=True, base=CorePeriodicTask)
|
||||
def refresh_all_years_source_record_export_artifacts(self) -> dict:
|
||||
"""Publish all available years independently of the annual generation."""
|
||||
return _refresh_source_record_export_artifacts(self, all_years=True)
|
||||
|
||||
|
||||
def _refresh_source_record_export_artifacts(task, *, all_years: bool) -> dict:
|
||||
lock_key = getattr(
|
||||
settings,
|
||||
"SOURCE_RECORD_EXPORT_LOCK_KEY",
|
||||
@@ -37,7 +47,7 @@ def refresh_source_record_export_artifacts(self) -> dict: # noqa: ARG001
|
||||
countdown = 30
|
||||
max_retries = max(1, math.ceil(lock_ttl / countdown) + 2)
|
||||
try:
|
||||
raise self.retry(countdown=countdown, max_retries=max_retries)
|
||||
raise task.retry(countdown=countdown, max_retries=max_retries)
|
||||
except MaxRetriesExceededError:
|
||||
logger.error(
|
||||
"Source-record export retry budget exhausted while waiting for build lock"
|
||||
@@ -45,7 +55,7 @@ def refresh_source_record_export_artifacts(self) -> dict: # noqa: ARG001
|
||||
raise
|
||||
|
||||
try:
|
||||
generation = build_source_record_export_artifacts()
|
||||
generation = build_source_record_export_artifacts(all_years=all_years)
|
||||
result = {
|
||||
"status": "success",
|
||||
"generation_id": generation.generation_id,
|
||||
|
||||
@@ -443,6 +443,126 @@ class OrganizationSourceRecordExportApiV2Test(APITestCase):
|
||||
"Строка 1\nСтрока 2\nПолная история",
|
||||
)
|
||||
|
||||
def test_all_years_generation_is_separate_and_keeps_historical_financial_lines(
|
||||
self,
|
||||
):
|
||||
organization = Organization.objects.create(name="Export", inn="7707083899")
|
||||
extension = PlannedInspectionExtension.objects.create(organization=organization)
|
||||
for year in (2020, 2026):
|
||||
OrganizationSourceRecord.objects.create(
|
||||
extension=extension,
|
||||
record_type="inspection",
|
||||
source="inspections",
|
||||
external_id=str(year),
|
||||
record_date=f"{year}-01-01",
|
||||
)
|
||||
finance = FinancialIndicatorsExtension.objects.create(organization=organization)
|
||||
report = OrganizationSourceRecord.objects.create(
|
||||
extension=finance,
|
||||
record_type="financial_report",
|
||||
source="fns_reports",
|
||||
external_id="full",
|
||||
)
|
||||
for year in (2020, 2026):
|
||||
OrganizationSourceFinancialLine.objects.create(
|
||||
source_record=report,
|
||||
year=year,
|
||||
form_code="1",
|
||||
line_code=str(year),
|
||||
)
|
||||
source = SourceGroup.PLANNED_INSPECTIONS.value
|
||||
annual = build_source_record_export_artifacts(
|
||||
now=datetime(2026, 8, 4, tzinfo=UTC)
|
||||
)
|
||||
annual_manifest = Path(self.export_directory.name) / "current.json"
|
||||
before = annual_manifest.read_bytes()
|
||||
full = build_source_record_export_artifacts(
|
||||
now=datetime(2026, 8, 4, tzinfo=UTC),
|
||||
all_years=True,
|
||||
)
|
||||
self.assertIsNone(full.export_year)
|
||||
self.assertEqual(annual_manifest.read_bytes(), before)
|
||||
full_root = Path(self.export_directory.name) / "all-years"
|
||||
loaded = load_current_source_record_export_generation(
|
||||
export_directory=full_root
|
||||
)
|
||||
self.assertEqual(loaded.generation_id, full.generation_id)
|
||||
self.assertIsNone(loaded.export_year)
|
||||
|
||||
def rows(generation, group):
|
||||
artifact = next(
|
||||
a
|
||||
for a in generation.artifacts
|
||||
if a.source_group == group and a.file_format == "json"
|
||||
)
|
||||
return _source_records(json.loads(artifact.path.read_text()), group)
|
||||
|
||||
self.assertEqual(len(rows(annual, source)), 1)
|
||||
self.assertEqual(len(rows(full, source)), 2)
|
||||
self.assertEqual(
|
||||
{
|
||||
line["year"]
|
||||
for line in rows(full, "financial_indicators")[0]["financial_lines"]
|
||||
},
|
||||
{2020, 2026},
|
||||
)
|
||||
self.assertEqual(
|
||||
{
|
||||
line["year"]
|
||||
for line in rows(annual, "financial_indicators")[0]["financial_lines"]
|
||||
},
|
||||
{2026},
|
||||
)
|
||||
# Retention in either scope must not delete the other's files.
|
||||
for _ in range(3):
|
||||
build_source_record_export_artifacts(all_years=True)
|
||||
self.assertTrue(all(a.path.is_file() for a in annual.artifacts))
|
||||
|
||||
def test_all_years_command_does_not_publish_an_annual_manifest(self):
|
||||
call_command("build_source_record_exports", all_years=True, stdout=StringIO())
|
||||
self.assertFalse((Path(self.export_directory.name) / "current.json").exists())
|
||||
full = load_current_source_record_export_generation(
|
||||
export_directory=Path(self.export_directory.name) / "all-years",
|
||||
)
|
||||
self.assertIsNone(full.export_year)
|
||||
|
||||
def test_full_export_task_publishes_separate_manifest_and_waits_for_lock(self):
|
||||
from celery.exceptions import Retry
|
||||
from django.core.cache import cache
|
||||
from organizations.tasks import refresh_all_years_source_record_export_artifacts
|
||||
|
||||
lock = "test-full-export-lock"
|
||||
with self.settings(SOURCE_RECORD_EXPORT_LOCK_KEY=lock):
|
||||
cache.delete(lock)
|
||||
cache.add(lock, "held", timeout=30)
|
||||
try:
|
||||
with self.assertRaises(Retry):
|
||||
refresh_all_years_source_record_export_artifacts.run()
|
||||
self.assertFalse(
|
||||
(
|
||||
Path(self.export_directory.name) / "all-years/current.json"
|
||||
).exists()
|
||||
)
|
||||
finally:
|
||||
cache.delete(lock)
|
||||
result = refresh_all_years_source_record_export_artifacts.run()
|
||||
self.assertEqual(result["status"], "success")
|
||||
self.assertIsNone(result["export_year"])
|
||||
self.assertIsNone(cache.get(lock))
|
||||
self.assertFalse(
|
||||
(Path(self.export_directory.name) / "current.json").exists()
|
||||
)
|
||||
|
||||
def test_manifest_rejects_ambiguous_all_years_scope(self):
|
||||
build_source_record_export_artifacts(all_years=True)
|
||||
directory = Path(self.export_directory.name) / "all-years"
|
||||
manifest_path = directory / "current.json"
|
||||
payload = json.loads(manifest_path.read_text())
|
||||
payload["export_year"] = 2026
|
||||
manifest_path.write_text(json.dumps(payload))
|
||||
with self.assertRaises(SourceRecordExportArtifactsUnavailable):
|
||||
load_current_source_record_export_generation(export_directory=directory)
|
||||
|
||||
def test_new_calendar_year_requires_a_new_prepared_generation(self):
|
||||
generated_at = datetime(2026, 12, 31, 23, 59, tzinfo=UTC)
|
||||
build_source_record_export_artifacts(now=generated_at)
|
||||
|
||||
@@ -2,6 +2,7 @@
|
||||
|
||||
from datetime import timedelta
|
||||
from unittest.mock import MagicMock
|
||||
from uuid import UUID
|
||||
|
||||
import pytest
|
||||
from apps.core.models import BackgroundJob, JobStatus
|
||||
@@ -145,3 +146,53 @@ def test_latest_task_does_not_reactivate_a_stale_job(status, time_field):
|
||||
**{time_field: timezone.now() - timedelta(minutes=15)}
|
||||
)
|
||||
assert SourceCardService.get_card(definition.slug)["active_tasks"] == []
|
||||
|
||||
|
||||
@pytest.mark.django_db
|
||||
def test_latest_job_uses_created_time_then_uuid_and_keeps_older_active_jobs():
|
||||
definition = SourceCardService.get_definition("arbitration-cases")
|
||||
jobs = [
|
||||
BackgroundJob.objects.create(
|
||||
id=UUID(int=index),
|
||||
task_id=f"ordered-{index}",
|
||||
task_name=definition.task_names[0],
|
||||
status=JobStatus.SUCCESS,
|
||||
)
|
||||
for index in (1, 2, 3, 4)
|
||||
]
|
||||
now = timezone.now()
|
||||
BackgroundJob.objects.filter(pk__in=[job.pk for job in jobs]).update(
|
||||
created_at=now - timedelta(minutes=2)
|
||||
)
|
||||
# UUID 4 is older; UUID 2 wins the timestamp tie with UUID 1.
|
||||
BackgroundJob.objects.filter(pk__in=[jobs[0].pk, jobs[1].pk]).update(created_at=now)
|
||||
BackgroundJob.objects.filter(pk=jobs[2].pk).update(status=JobStatus.STARTED)
|
||||
active, refresh = SourceCardService._build_task_maps((definition,))
|
||||
assert [job["task_id"] for job in active[definition.slug]] == ["ordered-3"]
|
||||
assert [job["task_id"] for job in refresh[definition.slug]] == [
|
||||
"ordered-2",
|
||||
"ordered-3",
|
||||
]
|
||||
|
||||
|
||||
@pytest.mark.django_db
|
||||
def test_latest_job_is_chosen_before_filtering_source_card_metadata():
|
||||
definition = SourceCardService.get_definition("arbitration-cases")
|
||||
older = BackgroundJob.objects.create(
|
||||
task_id="old-card-job",
|
||||
task_name="custom.shared.task",
|
||||
status=JobStatus.SUCCESS,
|
||||
meta={"source_card": definition.slug},
|
||||
)
|
||||
newer = BackgroundJob.objects.create(
|
||||
task_id="new-other-card-job",
|
||||
task_name=older.task_name,
|
||||
status=JobStatus.SUCCESS,
|
||||
meta={"source_card": "bankruptcy-procedures"},
|
||||
)
|
||||
BackgroundJob.objects.filter(pk=older.pk).update(
|
||||
created_at=newer.created_at - timedelta(minutes=1)
|
||||
)
|
||||
active, refresh = SourceCardService._build_task_maps((definition,))
|
||||
assert active == {}
|
||||
assert refresh == {}
|
||||
|
||||
61
tests/apps/parsers/test_snapshot_metadata.py
Normal file
61
tests/apps/parsers/test_snapshot_metadata.py
Normal file
@@ -0,0 +1,61 @@
|
||||
"""Snapshot counts avoid rescanning staged data when a saved total exists."""
|
||||
|
||||
from datetime import timedelta
|
||||
|
||||
import pytest
|
||||
from apps.parsers.models import ParserLoadLog, ParserSourceArtifact, ParserStagedRecord
|
||||
from apps.parsers.snapshot_metadata import attach_snapshot_artifacts
|
||||
from django.db import connection
|
||||
from django.test.utils import CaptureQueriesContext
|
||||
from organizations.models import Organization
|
||||
|
||||
|
||||
@pytest.mark.django_db
|
||||
@pytest.mark.parametrize("saved_total", [0, 7, None, -1, "7"])
|
||||
def test_saved_total_or_distinct_published_organizations(saved_total):
|
||||
log = ParserLoadLog.objects.create(source="sro_membership_check", batch_id=1)
|
||||
artifact = ParserSourceArtifact.objects.create(
|
||||
source=log.source,
|
||||
load_batch=log.batch_id,
|
||||
metadata={"snapshot_organizations_count": saved_total},
|
||||
)
|
||||
organization = Organization.objects.create(name="Published fixture")
|
||||
for row, disposition in enumerate(["published", "published", "quarantined"]):
|
||||
ParserStagedRecord.objects.create(
|
||||
artifact=artifact,
|
||||
row_number=row + 1,
|
||||
organization=organization,
|
||||
disposition=disposition,
|
||||
)
|
||||
ParserStagedRecord.objects.create(
|
||||
artifact=artifact, row_number=4, disposition="published", organization=None
|
||||
)
|
||||
with CaptureQueriesContext(connection) as queries:
|
||||
attach_snapshot_artifacts([log, log])
|
||||
cached = log._source_artifact_cache
|
||||
assert cached.pk == artifact.pk
|
||||
has_saved_total = isinstance(saved_total, int) and saved_total >= 0
|
||||
assert cached.published_organizations_count == (
|
||||
saved_total if has_saved_total else 1
|
||||
)
|
||||
staged_queries = [q for q in queries if "parsers_staged_record" in q["sql"]]
|
||||
assert len(staged_queries) == (0 if has_saved_total else 1)
|
||||
|
||||
|
||||
@pytest.mark.django_db
|
||||
def test_latest_artifact_and_missing_load_keep_their_own_counts():
|
||||
log = ParserLoadLog.objects.create(source="sro_membership_check", batch_id=1)
|
||||
missing = ParserLoadLog.objects.create(source=log.source, batch_id=2)
|
||||
old = ParserSourceArtifact.objects.create(source=log.source, load_batch=1)
|
||||
latest = ParserSourceArtifact.objects.create(
|
||||
source=log.source, load_batch=1, metadata={"snapshot_organizations_count": 9}
|
||||
)
|
||||
ParserSourceArtifact.objects.filter(pk=old.pk).update(
|
||||
created_at=latest.created_at - timedelta(seconds=1)
|
||||
)
|
||||
with CaptureQueriesContext(connection) as queries:
|
||||
attach_snapshot_artifacts([log, missing])
|
||||
assert log._source_artifact_cache.pk == latest.pk
|
||||
assert log._source_artifact_cache.published_organizations_count == 9
|
||||
assert missing._source_artifact_cache is None
|
||||
assert not any("parsers_staged_record" in q["sql"] for q in queries)
|
||||
@@ -20,6 +20,8 @@ from apps.parsers.models import (
|
||||
ProcurementRecord,
|
||||
)
|
||||
from django.core.files.uploadedfile import SimpleUploadedFile
|
||||
from django.db import connection
|
||||
from django.test.utils import CaptureQueriesContext
|
||||
from django.urls import reverse
|
||||
from openpyxl import Workbook
|
||||
from organizations.models import Organization, OrganizationSourceRecord
|
||||
@@ -697,7 +699,17 @@ class ParsersViewSetTest(APITestCase):
|
||||
)
|
||||
self.client.force_authenticate(self.user)
|
||||
|
||||
response = self.client.get("/api/v1/parsers/dashboard/")
|
||||
with CaptureQueriesContext(connection) as queries:
|
||||
response = self.client.get("/api/v1/parsers/dashboard/")
|
||||
|
||||
source_scans = [
|
||||
query["sql"]
|
||||
for query in queries
|
||||
if "SELECT DISTINCT" in query["sql"]
|
||||
and "organizations_source_record" in query["sql"]
|
||||
]
|
||||
self.assertTrue(source_scans)
|
||||
self.assertEqual(len(source_scans), len(set(source_scans)))
|
||||
|
||||
self.assertEqual(response.status_code, status.HTTP_200_OK)
|
||||
coverage = response.data["data"]["registry_data_coverage"]
|
||||
|
||||
Reference in New Issue
Block a user