Compare commits

..

4 Commits

Author SHA1 Message Date
Aleksandr Meshchryakov
fedd2e00a3 fix: avoid repeated task and source scans on dashboard
Some checks failed
Mostovik Backend CI/CD / Tests and lint (pull_request) Successful in 3m16s
Mostovik Backend CI/CD / Build linux/amd64 release images (pull_request) Has been skipped
Mostovik Backend CI/CD / Deploy dev (pull_request) Has been skipped
Mostovik Backend CI/CD / Deploy and verify internal main (pull_request) Has been skipped
Mostovik Backend CI/CD / Tests and lint (push) Failing after 3s
Mostovik Backend CI/CD / Build linux/amd64 release images (push) Has been skipped
Mostovik Backend CI/CD / Deploy dev (push) Has been skipped
Mostovik Backend CI/CD / Deploy and verify internal main (push) Has been skipped
2026-10-01 13:16:11 +02:00
avm
887442e8e3 Export all available years in a separate scheduled task [skip ci] #no_deploy #no_image
Reviewed hotfix. Manual immutable release with data preservation and runtime verification.
2026-09-28 13:20:08 +03:00
Aleksandr Meshchryakov
cd5a1764a8 feat: add separate all-years source export task 2026-09-28 12:06:50 +02:00
avm
245edd1fff Import: skip organizations absent from canonical registry [skip ci] #no_deploy #no_image
Reviewed hotfix. Manual immutable release with data preservation and runtime verification.
2026-09-28 12:37:05 +03:00
12 changed files with 426 additions and 50 deletions

View File

@@ -0,0 +1,15 @@
# Выгрузка за все годы
Годовая задача `organizations.tasks.refresh_source_record_export_artifacts` сохраняет прежнее поведение.
Новая задача `organizations.tasks.refresh_all_years_source_record_export_artifacts` собирает все доступные записи каждого поддерживаемого источника без ограничения по году, включая все финансовые строки. Для источников с заменяемыми snapshot выгружается текущее опубликованное состояние: удалённые прежние snapshot не восстанавливаются.
Результат размещается в `SOURCE_RECORD_EXPORT_DIRECTORY/all-years`. У него отдельные `current.json`, `generations` и очистка старых поколений. Публикация атомарна; незавершённая сборка не заменяет опубликованные файлы. Общий lock с годовой задачей ограничивает одновременную нагрузку. Manifest содержит `export_scope: all_years` и `export_year: null`; годовой manifest — `export_scope: current_year` и числовой год. Старые годовые manifest без `export_scope` продолжают читаться.
Ручной запуск:
```sh
python src/manage.py build_source_record_exports --all-years
```
Периодический запуск настраивается отдельной записью django-celery-beat либо системным таймером на контуре без Beat. Задачу следует ставить после годовой сборки. Имена файлов, группировка JSON по ОКПО и форматы CSV/XLSX/JSON сохраняются. API скачивания продолжает отдавать годовой пакет; доступ к полному пакету предоставляется отдельно.

View File

@@ -0,0 +1,52 @@
# Производительность главной страницы
Главная страница использует `/api/v1/sources/` и `/api/v1/parsers/dashboard/`.
Счетчики, состав задач, порядок запусков и поля ответов остаются прежними.
## Изменения
- PostgreSQL выбирает последнюю фоновую задачу для каждого `task_name` один раз
через `DISTINCT ON`. Сортировка сохраняется: `created_at DESC, id DESC`.
Выбор последних задач выполняется до фильтрации карточки; более старые
активные задачи и завершенные части составного запуска остаются видимыми.
Для SQLite сохранен совместимый вариант с подзапросом.
- Аналитика повторно использует соответствия организаций источникам и последние
логи в пределах одного HTTP-запроса. Между пользователями результаты не кешируются.
- Метаданные снимка используют сохраненный `snapshot_organizations_count`,
включая ноль. Для старых снимков без допустимого итога выполняется один
групповой подсчет уникальных опубликованных организаций. Отклоненные строки
и строки без организации в итог не входят. Подсчет нужен только для выбранной
последней версии снимка каждого запуска.
Новых настроек, миграций и правил инвалидирования кеша нет.
## Проверка на данных dev, 1 октября 2026
| Обработчик | Кеш агрегатов | До | После |
| --- | --- | ---: | ---: |
| `/sources/` | Пустой | 13,200 с | 3,453 с |
| `/sources/` | Заполненный | 10,191 с | 0,243 с |
| `/parsers/dashboard/` | Пустой | 4,165 с | 2,864 с |
| `/parsers/dashboard/` | Заполненный | 4,246 с | 2,880 с |
Это время обработчиков в отдельном процессе, без сети и аутентификационного
middleware. Сравнение выполнено на одном снимке БД в транзакции
`REPEATABLE READ, READ ONLY`, с фиксированным временем и отдельным кешем в памяти.
Полные JSON-ответы совпали во всех четырех сравнениях. Учетные данные
пользователей не использовались. Это диагностическое сравнение, не нагрузочный
тест и не подтверждение выкатки.
Оставшееся время первого запроса связано с подсчетом записей источников.
Dashboard по-прежнему рассчитывает актуальные счетчики при каждом запросе.
## Регрессионные проверки
- `test_refresh_progress.py`: порядок по времени и UUID, старые активные задачи,
параллельные и составные запуски, глобальный выбор последней задачи.
- `test_snapshot_metadata.py`: сохраненные и старые счетчики, нулевой итог,
повторные организации, неопубликованные строки, выбор последнего снимка.
- `test_views.py`: значения аналитики и отсутствие повторных обходов источников.
- `test_refresh_concurrency.py`: проверка блокировок на PostgreSQL.
Проверки PostgreSQL запускаются с `--ds=settings.test_postgres --migrations`
и отдельной тестовой БД. SQLite не проверяет `DISTINCT ON` и блокировки PostgreSQL.

View File

@@ -1,33 +1,41 @@
"""Public snapshot provenance shared by cards, dashboard and parser history."""
from apps.parsers.models import ParserSourceArtifact
from django.db.models import Count, Q
from apps.parsers.models import ParserSourceArtifact, ParserStagedRecord
from django.db.models import Count
def attach_snapshot_artifacts(logs) -> None:
"""Attach the latest artifact for each load in one query, without raw data."""
"""Attach provenance, counting staged rows only for legacy snapshots."""
logs = list(logs)
if not logs:
return
artifacts = {}
for artifact in (
ParserSourceArtifact.objects.filter(
source__in={log.source for log in logs},
load_batch__in={log.batch_id for log in logs},
)
.annotate(
published_organizations_count=Count(
"staged_records__organization_id",
distinct=True,
filter=Q(staged_records__disposition="published"),
)
)
.order_by("-created_at", "-uid")
):
for artifact in ParserSourceArtifact.objects.filter(
source__in={log.source for log in logs},
load_batch__in={log.batch_id for log in logs},
).order_by("-created_at", "-uid"):
artifacts.setdefault((artifact.source, artifact.load_batch), artifact)
legacy_artifacts = []
for artifact in artifacts.values():
total = artifact.metadata.get("snapshot_organizations_count")
if isinstance(total, int) and total >= 0:
artifact.published_organizations_count = total
artifacts.setdefault((artifact.source, artifact.load_batch), artifact)
else:
legacy_artifacts.append(artifact.pk)
counts = {}
if legacy_artifacts:
counts = dict(
ParserStagedRecord.objects.filter(
artifact_id__in=legacy_artifacts, disposition="published"
)
.order_by()
.values("artifact_id")
.annotate(total=Count("organization_id", distinct=True))
.values_list("artifact_id", "total")
)
for artifact in artifacts.values():
if artifact.pk in legacy_artifacts:
artifact.published_organizations_count = counts.get(artifact.pk, 0)
for log in logs:
log._source_artifact_cache = artifacts.get((log.source, log.batch_id))

View File

@@ -27,7 +27,7 @@ from apps.parsers.snapshot_metadata import (
from apps.parsers.source_registry import get_source_by_model_source
from django.conf import settings
from django.core.cache import cache
from django.db import transaction
from django.db import connections, transaction
from django.db.models import Count, Max, OuterRef, Q, Subquery
from django.http import Http404
from django.utils import timezone
@@ -1033,19 +1033,28 @@ class SourceCardService:
for task_name in definition.task_names:
slugs_by_task_name.setdefault(task_name, []).append(definition.slug)
slugs = {definition.slug for definition in definitions}
latest = (
BackgroundJobService.get_queryset()
.filter(task_name=OuterRef("task_name"))
.order_by("-created_at", "-id")
.values("pk")[:1]
)
all_jobs = BackgroundJobService.get_queryset()
if connections[all_jobs.db].features.can_distinct_on_fields:
# Find each task's latest job once, rather than once per history row.
latest = all_jobs.order_by("task_name", "-created_at", "-id").distinct(
"task_name"
)
else:
latest_for_task = (
all_jobs.filter(task_name=OuterRef("task_name"))
.order_by("-created_at", "-id")
.values("pk")[:1]
)
latest = all_jobs.filter(pk=Subquery(latest_for_task)).order_by()
jobs = list(
BackgroundJobService.get_queryset()
.filter(
Q(task_name__in=slugs_by_task_name)
| Q(meta__source_card__in=list(slugs))
)
.filter(cls._fresh_active_job_filter() | Q(pk=Subquery(latest)))
.filter(
cls._fresh_active_job_filter() | Q(pk__in=Subquery(latest.values("pk")))
)
.order_by("-created_at", "-id")
)
active_by_slug: dict[str, list[dict[str, Any]]] = {}

View File

@@ -2242,8 +2242,9 @@ def _registry_source_matrix_rows(
return matrix_rows
def _registry_data_coverage() -> dict:
registry_organization_ids, by_inn, by_ogrn = _active_registry_organization_indexes()
def _registry_data_coverage(
registry_organization_ids: set[str], source_matches: dict[str, set[str]]
) -> dict:
total_organizations = len(registry_organization_ids)
descriptor_titles: dict[str, str] = {}
for descriptor in PARSER_SOURCES.values():
@@ -2254,9 +2255,7 @@ def _registry_data_coverage() -> dict:
for source, _label in ParserLoadLog.Source.choices:
if source in REGISTRY_COVERAGE_EXCLUDED_SOURCES:
continue
organizations_count = len(
_matched_source_record_registry_organization_ids(source, by_inn, by_ogrn)
)
organizations_count = len(source_matches.get(source, set()))
coverage_percent = (
round(organizations_count / total_organizations * 100, 1)
if total_organizations
@@ -2277,10 +2276,14 @@ def _registry_data_coverage() -> dict:
}
def _registry_enrichment_analytics(jobs, schedules: list[dict]) -> dict:
registry_organization_ids, by_inn, by_ogrn = _active_registry_organization_indexes()
def _registry_enrichment_analytics(
jobs,
schedules: list[dict],
registry_organization_ids: set[str],
source_matches: dict[str, set[str]],
latest_loads: list[ParserLoadLog],
) -> dict:
total_organizations = len(registry_organization_ids)
source_matches = _source_registry_matches(by_inn, by_ogrn)
descriptor_titles: dict[str, str] = {}
for descriptor in PARSER_SOURCES.values():
descriptor_titles.setdefault(descriptor.source, descriptor.title)
@@ -2369,7 +2372,6 @@ def _registry_enrichment_analytics(jobs, schedules: list[dict]) -> dict:
enrichment_sources=enrichment_sources,
)
latest_loads = list(ParserLoadLog.objects.all().order_by("-created_at")[:30])
recent_success = sum(
1 for load in latest_loads if load.status == ParserLoadLog.Status.SUCCESS
)
@@ -3459,6 +3461,8 @@ class ParserDashboardDataView(APIView):
]
load_logs = list(ParserLoadLog.objects.all().order_by("-created_at")[:30])
attach_snapshot_artifacts(load_logs)
registry_ids, by_inn, by_ogrn = _active_registry_organization_indexes()
source_matches = _source_registry_matches(by_inn, by_ogrn)
return api_response(
{
"sources": sources,
@@ -3471,10 +3475,15 @@ class ParserDashboardDataView(APIView):
"schedules": ParserScheduleSerializer(schedules, many=True).data,
"jobs": BackgroundJobListSerializer(jobs, many=True).data,
"source_counts": source_counts,
"registry_data_coverage": _registry_data_coverage(),
"registry_data_coverage": _registry_data_coverage(
registry_ids, source_matches
),
"registry_enrichment_analytics": _registry_enrichment_analytics(
jobs,
schedules,
registry_ids,
source_matches,
load_logs,
),
"load_logs": ParserLoadLogSerializer(
load_logs,

View File

@@ -15,8 +15,18 @@ class Command(BaseAppCommand):
help = "Формирует готовые CSV/XLSX/JSON выгрузки источников"
use_transaction = False
def add_arguments(self, parser) -> None:
super().add_arguments(parser)
parser.add_argument(
"--all-years",
action="store_true",
help="Все доступные годы в отдельном каталоге all-years",
)
def execute_command(self, *args, **options) -> str:
generation = build_source_record_export_artifacts()
generation = build_source_record_export_artifacts(
all_years=options["all_years"]
)
rendered = json.dumps(
{
"generation_id": generation.generation_id,

View File

@@ -195,7 +195,7 @@ class SourceRecordExportGeneration:
generation_id: str
generated_at: str
export_year: int
export_year: int | None
artifacts: tuple[SourceRecordExportArtifact, ...]
records_count: int
source_snapshots: dict[str, str] = field(default_factory=dict)
@@ -267,16 +267,19 @@ def build_source_record_export_artifacts(
*,
now: datetime | None = None,
export_directory: str | Path | None = None,
all_years: bool = False,
) -> SourceRecordExportGeneration:
"""Build all files on disk and atomically publish a new generation."""
"""Build annual or all-years files and atomically publish their generation."""
root_directory = _resolve_export_directory(export_directory)
if all_years:
root_directory /= "all-years"
generations_directory = root_directory / GENERATION_DIRECTORY_NAME
root_directory.mkdir(parents=True, exist_ok=True)
generations_directory.mkdir(parents=True, exist_ok=True)
generated_at_datetime = _normalize_generation_datetime(now or timezone.now())
export_year = _export_year(generated_at_datetime)
export_year = None if all_years else _export_year(generated_at_datetime)
generation_id = (
f"{generated_at_datetime.strftime('%Y%m%dT%H%M%SZ')}-{uuid4().hex[:8]}"
)
@@ -739,7 +742,7 @@ def _build_source_group_file_name(*, source_group: str, file_format: str) -> str
def _source_group_queryset(
source_group: str,
*,
export_year: int,
export_year: int | None,
) -> QuerySet[OrganizationSourceRecord]:
queryset = (
OrganizationSourceRecord.objects.filter(extension__source_group=source_group)
@@ -749,6 +752,8 @@ def _source_group_queryset(
# source groups with millions of rows.
.order_by()
)
if export_year is None:
return queryset
if source_group == FINANCIAL_SOURCE_GROUP:
return queryset.filter(financial_lines__year=export_year).distinct()
if source_group in ALL_HISTORY_SOURCE_GROUPS:
@@ -777,7 +782,7 @@ def _iter_source_records(
*,
source_group: str,
include_financial_lines: bool,
export_year: int,
export_year: int | None,
) -> Iterator[OrganizationSourceRecord]:
iterator = _source_group_queryset(
source_group,
@@ -793,7 +798,7 @@ def _iter_source_records(
Prefetch(
"financial_lines",
queryset=OrganizationSourceFinancialLine.objects.filter(
year=export_year
**({"year": export_year} if export_year is not None else {})
).order_by(),
),
)
@@ -804,7 +809,7 @@ def _spool_source_group_rows(
*,
source_group: str,
output_path: Path,
export_year: int,
export_year: int | None,
) -> tuple[list[str], int]:
include_financial_lines = source_group == FINANCIAL_SOURCE_GROUP
payload_headers: set[str] = set()
@@ -1373,6 +1378,9 @@ def _generation_manifest_payload(
"source_snapshots": generation.source_snapshots,
"generated_at": generation.generated_at,
"export_year": generation.export_year,
"export_scope": "all_years"
if generation.export_year is None
else "current_year",
"records_count": generation.records_count,
"artifacts_count": generation.artifacts_count,
"files_count": generation.files_count,
@@ -1403,6 +1411,19 @@ def _manifest_snapshot_ids(payload: dict) -> dict[str, str]:
return identities
def _manifest_export_year(payload: dict[str, Any]) -> int | None:
raw_year = payload["export_year"]
scope = payload.get("export_scope", "current_year")
if scope == "all_years" and raw_year is None:
return None
if scope != "current_year" or raw_year is None or isinstance(raw_year, bool):
raise ValueError("Source-record export scope is invalid.")
year = int(raw_year)
if not 1 <= year <= 9999:
raise ValueError("Source-record export year is invalid.")
return year
def _generation_from_manifest(
payload: Any,
*,
@@ -1415,9 +1436,7 @@ def _generation_from_manifest(
source_snapshots = _manifest_snapshot_ids(payload)
generated_at = str(payload["generated_at"])
datetime.fromisoformat(generated_at)
export_year = int(payload["export_year"])
if not 1 <= export_year <= 9999:
raise ValueError("Source-record export year is invalid.")
export_year = _manifest_export_year(payload)
records_count = int(payload["records_count"])
artifact_payloads = payload["artifacts"]
if not isinstance(artifact_payloads, list):

View File

@@ -22,6 +22,16 @@ logger = logging.getLogger(__name__)
@shared_task(bind=True, base=CorePeriodicTask)
def refresh_source_record_export_artifacts(self) -> dict: # noqa: ARG001
"""Build and atomically publish the nightly source-record export matrix."""
return _refresh_source_record_export_artifacts(self, all_years=False)
@shared_task(bind=True, base=CorePeriodicTask)
def refresh_all_years_source_record_export_artifacts(self) -> dict:
"""Publish all available years independently of the annual generation."""
return _refresh_source_record_export_artifacts(self, all_years=True)
def _refresh_source_record_export_artifacts(task, *, all_years: bool) -> dict:
lock_key = getattr(
settings,
"SOURCE_RECORD_EXPORT_LOCK_KEY",
@@ -37,7 +47,7 @@ def refresh_source_record_export_artifacts(self) -> dict: # noqa: ARG001
countdown = 30
max_retries = max(1, math.ceil(lock_ttl / countdown) + 2)
try:
raise self.retry(countdown=countdown, max_retries=max_retries)
raise task.retry(countdown=countdown, max_retries=max_retries)
except MaxRetriesExceededError:
logger.error(
"Source-record export retry budget exhausted while waiting for build lock"
@@ -45,7 +55,7 @@ def refresh_source_record_export_artifacts(self) -> dict: # noqa: ARG001
raise
try:
generation = build_source_record_export_artifacts()
generation = build_source_record_export_artifacts(all_years=all_years)
result = {
"status": "success",
"generation_id": generation.generation_id,

View File

@@ -443,6 +443,126 @@ class OrganizationSourceRecordExportApiV2Test(APITestCase):
"Строка 1\nСтрока 2\nПолная история",
)
def test_all_years_generation_is_separate_and_keeps_historical_financial_lines(
self,
):
organization = Organization.objects.create(name="Export", inn="7707083899")
extension = PlannedInspectionExtension.objects.create(organization=organization)
for year in (2020, 2026):
OrganizationSourceRecord.objects.create(
extension=extension,
record_type="inspection",
source="inspections",
external_id=str(year),
record_date=f"{year}-01-01",
)
finance = FinancialIndicatorsExtension.objects.create(organization=organization)
report = OrganizationSourceRecord.objects.create(
extension=finance,
record_type="financial_report",
source="fns_reports",
external_id="full",
)
for year in (2020, 2026):
OrganizationSourceFinancialLine.objects.create(
source_record=report,
year=year,
form_code="1",
line_code=str(year),
)
source = SourceGroup.PLANNED_INSPECTIONS.value
annual = build_source_record_export_artifacts(
now=datetime(2026, 8, 4, tzinfo=UTC)
)
annual_manifest = Path(self.export_directory.name) / "current.json"
before = annual_manifest.read_bytes()
full = build_source_record_export_artifacts(
now=datetime(2026, 8, 4, tzinfo=UTC),
all_years=True,
)
self.assertIsNone(full.export_year)
self.assertEqual(annual_manifest.read_bytes(), before)
full_root = Path(self.export_directory.name) / "all-years"
loaded = load_current_source_record_export_generation(
export_directory=full_root
)
self.assertEqual(loaded.generation_id, full.generation_id)
self.assertIsNone(loaded.export_year)
def rows(generation, group):
artifact = next(
a
for a in generation.artifacts
if a.source_group == group and a.file_format == "json"
)
return _source_records(json.loads(artifact.path.read_text()), group)
self.assertEqual(len(rows(annual, source)), 1)
self.assertEqual(len(rows(full, source)), 2)
self.assertEqual(
{
line["year"]
for line in rows(full, "financial_indicators")[0]["financial_lines"]
},
{2020, 2026},
)
self.assertEqual(
{
line["year"]
for line in rows(annual, "financial_indicators")[0]["financial_lines"]
},
{2026},
)
# Retention in either scope must not delete the other's files.
for _ in range(3):
build_source_record_export_artifacts(all_years=True)
self.assertTrue(all(a.path.is_file() for a in annual.artifacts))
def test_all_years_command_does_not_publish_an_annual_manifest(self):
call_command("build_source_record_exports", all_years=True, stdout=StringIO())
self.assertFalse((Path(self.export_directory.name) / "current.json").exists())
full = load_current_source_record_export_generation(
export_directory=Path(self.export_directory.name) / "all-years",
)
self.assertIsNone(full.export_year)
def test_full_export_task_publishes_separate_manifest_and_waits_for_lock(self):
from celery.exceptions import Retry
from django.core.cache import cache
from organizations.tasks import refresh_all_years_source_record_export_artifacts
lock = "test-full-export-lock"
with self.settings(SOURCE_RECORD_EXPORT_LOCK_KEY=lock):
cache.delete(lock)
cache.add(lock, "held", timeout=30)
try:
with self.assertRaises(Retry):
refresh_all_years_source_record_export_artifacts.run()
self.assertFalse(
(
Path(self.export_directory.name) / "all-years/current.json"
).exists()
)
finally:
cache.delete(lock)
result = refresh_all_years_source_record_export_artifacts.run()
self.assertEqual(result["status"], "success")
self.assertIsNone(result["export_year"])
self.assertIsNone(cache.get(lock))
self.assertFalse(
(Path(self.export_directory.name) / "current.json").exists()
)
def test_manifest_rejects_ambiguous_all_years_scope(self):
build_source_record_export_artifacts(all_years=True)
directory = Path(self.export_directory.name) / "all-years"
manifest_path = directory / "current.json"
payload = json.loads(manifest_path.read_text())
payload["export_year"] = 2026
manifest_path.write_text(json.dumps(payload))
with self.assertRaises(SourceRecordExportArtifactsUnavailable):
load_current_source_record_export_generation(export_directory=directory)
def test_new_calendar_year_requires_a_new_prepared_generation(self):
generated_at = datetime(2026, 12, 31, 23, 59, tzinfo=UTC)
build_source_record_export_artifacts(now=generated_at)

View File

@@ -2,6 +2,7 @@
from datetime import timedelta
from unittest.mock import MagicMock
from uuid import UUID
import pytest
from apps.core.models import BackgroundJob, JobStatus
@@ -145,3 +146,53 @@ def test_latest_task_does_not_reactivate_a_stale_job(status, time_field):
**{time_field: timezone.now() - timedelta(minutes=15)}
)
assert SourceCardService.get_card(definition.slug)["active_tasks"] == []
@pytest.mark.django_db
def test_latest_job_uses_created_time_then_uuid_and_keeps_older_active_jobs():
definition = SourceCardService.get_definition("arbitration-cases")
jobs = [
BackgroundJob.objects.create(
id=UUID(int=index),
task_id=f"ordered-{index}",
task_name=definition.task_names[0],
status=JobStatus.SUCCESS,
)
for index in (1, 2, 3, 4)
]
now = timezone.now()
BackgroundJob.objects.filter(pk__in=[job.pk for job in jobs]).update(
created_at=now - timedelta(minutes=2)
)
# UUID 4 is older; UUID 2 wins the timestamp tie with UUID 1.
BackgroundJob.objects.filter(pk__in=[jobs[0].pk, jobs[1].pk]).update(created_at=now)
BackgroundJob.objects.filter(pk=jobs[2].pk).update(status=JobStatus.STARTED)
active, refresh = SourceCardService._build_task_maps((definition,))
assert [job["task_id"] for job in active[definition.slug]] == ["ordered-3"]
assert [job["task_id"] for job in refresh[definition.slug]] == [
"ordered-2",
"ordered-3",
]
@pytest.mark.django_db
def test_latest_job_is_chosen_before_filtering_source_card_metadata():
definition = SourceCardService.get_definition("arbitration-cases")
older = BackgroundJob.objects.create(
task_id="old-card-job",
task_name="custom.shared.task",
status=JobStatus.SUCCESS,
meta={"source_card": definition.slug},
)
newer = BackgroundJob.objects.create(
task_id="new-other-card-job",
task_name=older.task_name,
status=JobStatus.SUCCESS,
meta={"source_card": "bankruptcy-procedures"},
)
BackgroundJob.objects.filter(pk=older.pk).update(
created_at=newer.created_at - timedelta(minutes=1)
)
active, refresh = SourceCardService._build_task_maps((definition,))
assert active == {}
assert refresh == {}

View File

@@ -0,0 +1,61 @@
"""Snapshot counts avoid rescanning staged data when a saved total exists."""
from datetime import timedelta
import pytest
from apps.parsers.models import ParserLoadLog, ParserSourceArtifact, ParserStagedRecord
from apps.parsers.snapshot_metadata import attach_snapshot_artifacts
from django.db import connection
from django.test.utils import CaptureQueriesContext
from organizations.models import Organization
@pytest.mark.django_db
@pytest.mark.parametrize("saved_total", [0, 7, None, -1, "7"])
def test_saved_total_or_distinct_published_organizations(saved_total):
log = ParserLoadLog.objects.create(source="sro_membership_check", batch_id=1)
artifact = ParserSourceArtifact.objects.create(
source=log.source,
load_batch=log.batch_id,
metadata={"snapshot_organizations_count": saved_total},
)
organization = Organization.objects.create(name="Published fixture")
for row, disposition in enumerate(["published", "published", "quarantined"]):
ParserStagedRecord.objects.create(
artifact=artifact,
row_number=row + 1,
organization=organization,
disposition=disposition,
)
ParserStagedRecord.objects.create(
artifact=artifact, row_number=4, disposition="published", organization=None
)
with CaptureQueriesContext(connection) as queries:
attach_snapshot_artifacts([log, log])
cached = log._source_artifact_cache
assert cached.pk == artifact.pk
has_saved_total = isinstance(saved_total, int) and saved_total >= 0
assert cached.published_organizations_count == (
saved_total if has_saved_total else 1
)
staged_queries = [q for q in queries if "parsers_staged_record" in q["sql"]]
assert len(staged_queries) == (0 if has_saved_total else 1)
@pytest.mark.django_db
def test_latest_artifact_and_missing_load_keep_their_own_counts():
log = ParserLoadLog.objects.create(source="sro_membership_check", batch_id=1)
missing = ParserLoadLog.objects.create(source=log.source, batch_id=2)
old = ParserSourceArtifact.objects.create(source=log.source, load_batch=1)
latest = ParserSourceArtifact.objects.create(
source=log.source, load_batch=1, metadata={"snapshot_organizations_count": 9}
)
ParserSourceArtifact.objects.filter(pk=old.pk).update(
created_at=latest.created_at - timedelta(seconds=1)
)
with CaptureQueriesContext(connection) as queries:
attach_snapshot_artifacts([log, missing])
assert log._source_artifact_cache.pk == latest.pk
assert log._source_artifact_cache.published_organizations_count == 9
assert missing._source_artifact_cache is None
assert not any("parsers_staged_record" in q["sql"] for q in queries)

View File

@@ -20,6 +20,8 @@ from apps.parsers.models import (
ProcurementRecord,
)
from django.core.files.uploadedfile import SimpleUploadedFile
from django.db import connection
from django.test.utils import CaptureQueriesContext
from django.urls import reverse
from openpyxl import Workbook
from organizations.models import Organization, OrganizationSourceRecord
@@ -697,7 +699,17 @@ class ParsersViewSetTest(APITestCase):
)
self.client.force_authenticate(self.user)
response = self.client.get("/api/v1/parsers/dashboard/")
with CaptureQueriesContext(connection) as queries:
response = self.client.get("/api/v1/parsers/dashboard/")
source_scans = [
query["sql"]
for query in queries
if "SELECT DISTINCT" in query["sql"]
and "organizations_source_record" in query["sql"]
]
self.assertTrue(source_scans)
self.assertEqual(len(source_scans), len(set(source_scans)))
self.assertEqual(response.status_code, status.HTTP_200_OK)
coverage = response.data["data"]["registry_data_coverage"]