diff --git a/.gitattributes b/.gitattributes new file mode 100644 index 0000000..93ff5c3 --- /dev/null +++ b/.gitattributes @@ -0,0 +1,2 @@ +# Preserve the verified official FNS schema byte-for-byte (Windows-1251, CRLF). +src/apps/parsers/schemas/fns_sme_support_structure_20230615.xsd -text whitespace=blank-at-eol,blank-at-eof,space-before-tab,cr-at-eol diff --git a/docs/frontend-integration-2026-09-13.md b/docs/frontend-integration-2026-09-13.md new file mode 100644 index 0000000..edc58ab --- /dev/null +++ b/docs/frontend-integration-2026-09-13.md @@ -0,0 +1,144 @@ +# Мостовик: интеграция frontend с backend, 13.09.2026 + +Передача Глебу. Проверен свежий frontend `origin/dev` +`c6f2ec787efef5617d6624ff83050589c3941f0d` после успешного SSH fetch через +штатный jump host. Рабочее дерево frontend не переключалось и не изменялось. +Backend-контракт: текущая поставка поверх +`c863563851642727045f58f16336aa96fb28308a`; основной контракт новых источников +описан в [registry-sources-ru.md](registry-sources-ru.md). +Документ не подтверждает публикацию backend/frontend на dev-стенде. + +Все frontend-пути ниже начинаются с `src/pages/main/`. + +**Уже подключено в свежем frontend dev:** оба новых view, таблицы и настройки +запуска, оба источника в фильтре истории и обе группы общей выгрузки. Это +подтверждено чтением `resolveSourceDetailView.ts`, `useScrapingCollectionCards.ts`, +`ReferenceDataSourcesExportCard.vue`, `updateHistoryLogs.ts` и исполнением resolver. +Старое заключение об отсутствии этих подключений к свежему dev не относится. + +| Источник | Slug | Source | Source group | Record type | +| --- | --- | --- | --- | --- | +| Поддержка МСП | `sme-support-recipients-registry` | `fns_sme_support_recipients` | `government_support` | `sme_support_measure` | +| Бюджетный процесс | `budget-process-registry` | `budget_ubpandnubp` | `budget_process_registry` | `budget_registry_organization` | + +**MV-1. Привести DTO и детали к фактическому MVP payload.** +Backend `src/apps/parsers/budget_registry.py:55` отдаёт нормализованные +`registry`, `classification`, `budget`, `address` и полный исходный `upstream` +в детали. Он не обещает нормализованные `legal`, `heads`, `contacts`, `activities`, +`authorities`, `successions`, `contracts`, `attachments`, `summary`, +`is_separate_division`. Однако frontend +`ui/SourceRecordDetail/BudgetProcessRegistryRecordDetail.vue:98` обращается к +`payload.legal.firm_name`, а +`model/source-record-detail/sourceRecordDetail.ts:400,414` — к +`payload.legal.legal_form` и `.heads.length`. На payload фактического normalizer +исполнение mapper завершилось TypeError `reading 'legal_form'`. + +Backend `src/apps/parsers/sme_support.py:78` отдаёт `recipient_type` и +`sme_category` строковыми кодами, `support_form`/`support_kind` словарями, +`provider` с name/inn, `support_sizes` со значением и единицей. `violations` и +`regulatory_documents` сохраняют оригинальные XML-атрибуты. `provenance` и +нормализованного региона в payload нет. Frontend +`model/source-detail/smeSupportRecipients.ts:61,195` ожидает словарь категории +и теряет существующий код при выводе; detail mapper +`model/source-record-detail/sourceRecordDetail.ts:525` падает на +`payload.provenance.dataset_url` (TypeError воспроизведён). + +Исправить перечисленные DTO, мапперы и detail renderer под этот контракт: +не читать несуществующие обязательные блоки; не заменять неизвестные признаки +значением «нет»; выводить подтверждённые код/значение/единицу, а не вымышленные +названия. Для бюджета полный исходный документ доступен в `payload.upstream` +детали/выгрузки. Для МСП читать реальные имена XML-атрибутов документов и +нарушений. Отдельно проверить экспорт полной карточки, использующий тот же mapper. + +Список GET `/api/v2/organization-source-records/` возвращает `data` и +`meta.pagination`, detail GET `/{uid}/` — объект записи. Краткий payload +намеренно исключает `source_document_id`, `violations`, `regulatory_documents`, +`upstream`: за ними нужен detail-запрос. Отсутствие массива в списке не означает +отсутствие данных. Полный контракт: `docs/registry-sources-ru.md`. + +**MV-2. Оставить только поддержанные сортировки и фильтры.** +Backend уже исправлен для составного `ordering` из существующего allowlist: +начальный запрос МСП `ordering=-record_date,-external_id` теперь проходит. +Каждый компонент отдельно валидируется; неизвестные поля остаются 400. +Дополнительные JSON-sort и специальные фильтры в MVP не добавлялись. + +`model/source-detail/useSmeSupportRecipientsSourceTable.ts:76–135` посылает +неподдержанные sort по support_form, support_kind, support_until, provider, +region, sme_category, has_violation. +`model/source-detail/useBudgetProcessRegistrySourceTable.ts:93–129` — по +organization_type, establishment_kind, budget.level, address.region, +is_separate_division. Убрать sortable/orderingKey с этих заголовков до отдельного +согласования backend-расширения. Не подменять сортировку смыслово другим полем. + +Допустимые простые поля для соответствующих колонок: `record_date`, `updated_at`, +`status`, `external_id`, `extension__organization__name` (также alias +`organization__name`), ИНН/ОГРН/ОКПО организации. Полный текущий allowlist находится +в `src/organizations/views.py:104`. Рекомендуемые defaults: +МСП `-record_date,-external_id`, бюджет `extension__organization__name`. + +Бюджетная таблица на строках 252–259 отправляет `budget_level`, +`establishment_kind`, `has_procurement_permission`, `is_branch`, +`organization_type`, `region_code`: текущий backend эти шесть параметров +игнорирует. Скрыть/отключить соответствующие фильтры. Поддержаны общие +source/source_group/record_type, status, organization, search и период record_date. +Колонка бюджета «Обновлено» показывает `updated_at`, но её dateFilterKey — +`record_date` (строки 120–122): согласовать подпись/значение колонки с полем +фильтра. Это разные даты. + +**MV-3. Завершать polling по статусу и считать всю группу задач.** +`model/sources/useSourceCardRefreshTracking.ts:128–153` по-прежнему считает +progress=100 завершением и успехом даже для status=running или status=error +с пустым error. Оба результата воспроизведены. Статус jobs API — +`running`, `success`, `error`; 100% не доказывает завершение. Terminal и success +определять по статусу; ошибка должна оставаться ошибкой. + +Строки 180–203 делят сумму только на уже полученные числовые ответы. Если из +двух task_ids первая задача ответила 100, а вторая временно недоступна, +расчёт показывает 100; после ответа второй с 0 даёт 50. Свежий +`ui/SourceDetailPage/components/useSourceDetailRefresh.ts:107–128` уже удерживает +показанный максимум: пользователь теперь может видеть преждевременные 100, +а не снижение до 50. Исправить знаменатель на фиксированный полный task_ids, +сохранять последние значения при временной ошибке polling. После перезагрузки, +если полный набор неизвестен, использовать агрегированный backend progress: +текущий приоритет среднего только activeTasks на строках 73–102 исключает +завершённые части группы. Строки 87–88 и 130–133 показывают success/«Обновлено» +для любого неактивного состояния: отдельно отображать error и idle. + +202 с task_ids уже поддерживается (`queued` вместо `accepted` не мешает). +При 409 `source_refresh_running` показать сообщение и перечитать карточку/ +статусы; не включать фиктивный polling без IDs. Общий error envelope совместим, +отдельного восстановления карточки на 409 в hook пока нет. + +**История и общая выгрузка уже согласованы по исходникам.** +`model/update-history/useUpdateHistoryTable.ts:55,67,148` теперь отправляет +`date_from/date_to` и в список, и в экспорт, включает период в query key и +сбрасывает страницу. `lib/update-history/updateHistoryLogs.ts:46` корректно +передаёт один день как одинаковые границы. Старый локальный date filter удалён. +Это регрессия для приёмки, не новая задача реализации. Проверить >100 строк, +совпадение периода/количества CSV и список, границу суток: backend фильтрует +updated_at в UTC, browser formatter использует локальную зону. + +Обе новые группы уже есть в `ReferenceDataSourcesExportCard.vue:86–93`. +Общий ticket 201 / native POST download совместим. В +`model/source-detail/exportSourceDetailTable.ts:111` новые таблицы остаются +на локальном CSV fallback. Если кнопка обещает полный реестр, подключить +серверную выгрузку; если только текущую страницу — явно обозначить объём. +Карточки/списки относятся к ОПК, полная выгрузка — ко всем сопоставленным +организациям справочника; эти счётчики могут отличаться по контракту. + +## Проверки и критерии готовности + +Составной allowlisted ordering уже исправлен на backend: 29 API-тестов прошли, +Ruff и `git diff --check` прошли. Несуществующие payload-sort поля дают 400. +Node probes свежих frontend-функций воспроизвели ошибки деталей (`legal_form`, +`dataset_url`), потерю кода категории и ошибки определения terminal/progress. +Payload для деталей получен фактическими backend normalizers без сети и БД; +это не полный Vue/browser тест. Frontend build/Vitest/E2E здесь не запускались. + +Готовность после MV-1–MV-3: оба реальных API списка и детали открываются без +TypeError; отсутствующие поля не подменяются утверждением «нет»; доступны только +работающие filters/sorts; running100 продолжает polling, error100 остаётся +ошибкой; две задачи с задержанным ответом не показывают преждевременные 100%; +после reload сохраняется прогресс всей группы. Проверить 409/refetch, экспорт +карточки, серверный период истории и настоящий ZIP общей выгрузки. Использовать +fixtures реального backend и стенд без frontend mock-режима. diff --git a/docs/registry-sources-ru.md b/docs/registry-sources-ru.md new file mode 100644 index 0000000..44e750a --- /dev/null +++ b/docs/registry-sources-ru.md @@ -0,0 +1,93 @@ +# Реестры поддержки МСП и участников бюджетного процесса + +Источники обновляются администратором через существующую карточку источника или +`POST /api/v1/parsers/run/{source_key}/`. Ответ `202` содержит идентификатор задачи, +доступный сразу через `/api/v1/jobs/{task_id}/`. Одновременный запуск того же +источника возвращает `409`. Автоматические расписания не добавлены. + +| Источник | Карточка | Ключ парсера | Группа записей | Тип записи | +| --- | --- | --- | --- | --- | +| Поддержка МСП | `sme-support-recipients-registry` | `fns_sme_support_recipients` | `government_support` | `sme_support_measure` | +| Бюджетный процесс | `budget-process-registry` | `budget_ubpandnubp` | `budget_process_registry` | `budget_registry_organization` | + +Записи выдаются универсальными API `/api/v2/organization-source-records/` и +`/api/v2/organization-source-records/{uid}/`. Список содержит краткий payload; +деталь сохраняет полные массивы и исходные бюджетные блоки. Стандартные выгрузки +CSV/XLSX/JSON содержат полный payload, включая данные за прошлые годы. + +## Состав организаций и счетчики + +Импорт связывает сведения только с существующими организациями канонического +справочника (`directory_imported_at` заполнено). Новые организации из внешних +реестров не создаются. Неоднозначные и конфликтующие идентификаторы не выбираются +автоматически. Для публикации нужны наименование, ИНН, ОГРН и ОКПО организации. + +Карточка, список записей и dashboard `source_counts` новых источников используют +существующий фильтр организаций ОПК. Полный импорт и стандартная выгрузка +охватывают все сопоставленные организации справочника, включая организации вне +ОПК. Поэтому `published_records_count` задачи и `published_count` исходного +артефакта могут быть больше счетчика карточки. Raw, пропуски и карантин также +учитываются отдельно. + +## Загрузка и публикация + +МСП: каталог `https://www.nalog.gov.ru/opendata/7707329152-rsmppp/` определяет +последний ZIP формата `structure-20230615`. Архив скачивается потоково во временный +файл; XML проверяется по сохраненной официальной XSD 4.04. Одна мера поддержки +создает одну стабильную запись. Размеры разных единиц остаются массивом, `amount` +содержит только рубли. Физические лица и получатели вне справочника не +публикуются; полный исходный ZIP сохраняется как артефакт. Лимиты: ZIP 2 GiB, +суммарный распакованный XML 64 GiB, без извлечения путей архива на диск. + +ZIP запрашивается диапазонами по 16 MiB; оборванное тело продолжается с последнего +записанного байта, до трех повторов подряд. Продолжение требует совпадения +`Content-Range`, полного размера и сильного `ETag` либо `Last-Modified`; сменившийся +файл не склеивается с прежним. Если сервер сразу не поддерживает Range, повтор +полного ответа начинается с нуля. Размер диапазона и число повторов задаются +внутренними параметрами `download_registry_archive`. + +В официальном архиве от 15.08.2026 `КолДок=1` встречается в XML с сотнями +документов. Полноту подтверждают CRC архива, завершенный XML и XSD, а расхождение +этого поля сохраняется в metadata артефакта: `documents_count` — фактическое число, +`declared_documents_count` — сумма заявленных значений, +`document_count_mismatch_files` — число XML с расхождением. + +Бюджет: `https://budget.gov.ru/epbs/registry/ubpandnubp/data` обходится по страницам +`blocks=info`. Полная деталь запрашивается и сохраняется только для сопоставленных +организаций. Изменение количества записей или версии во время обхода, +дублирование идентификатора и неполная страница отклоняют новый снимок. +Неизвестные коды статуса остаются `unknown`; документированный код `2` означает +`inactive`. Исходные поля и неизвестные непустые блоки сохраняются в detail +`payload.upstream`, без выдуманного толкования. + +Новые данные сначала попадают в staging. Только завершенный и проверенный вход +публикуется одной транзакцией, вместе с финальными статусами журнала и задачи. +Ошибка разбора, публикации, проверки кеша до commit или финализации сохраняет +предыдущие записи. После commit версия кеша меняется повторно, чтобы исключить +обычное заполнение кеша старым снимком во время транзакции. Ошибка этой повторной +инвалидации записывается в журнал, но уже опубликованные данные и успешная задача +не откатываются: общей транзакции базы данных и кеша нет. +Повторная доставка задачи и повторная публикация артефакта не удаляют данные. +Существующая политика хранения артефактов не изменена. + +Причина отказа проверяется по `artifact_id` в `ParserSourceArtifact.metadata`: +`error_code` содержит класс ошибки, а `rejection_reason` — ограниченный внутренний +код проверки, например `incomplete_budget_snapshot`. Такие коды также видны в +ошибке задачи и журнале загрузки. Тела HTTP-ответов и XML в ошибки не копируются. + +Перед включением на стенде нужны миграции `parsers.0034` и `organizations.0012`, +доступ worker к официальным HTTPS-источникам и место для временного и сохраненного +ZIP. Первый полный импорт и время его выполнения проверяются отдельно: успешное +чтение каталога/XSD не подтверждает успешную загрузку всего реестра. + +## Проверенный официальный снимок + +13.09.2026 полностью проверен `data-20260815-structure-20230615.zip`: +840 457 773 байта, SHA-256 +`00bc1d1ef97e1332f5e59fb04fd230f7ed0e0452ba7e90f6551d357ec891ce7a`. +CRC всех 14 140 XML и полный проход XSD успешны: 3 344 437 документов, +12 580 017 мер поддержки. Все 4 173 762 меры юридических лиц прошли нормализацию +без ошибок; 8 406 255 мер физических лиц посчитаны без нормализации. +Эта проверка не выполняла сопоставление со справочником и публикацию в рабочую +базу. Бюджетный API проверен со стенда: JSON первой страницы и нормализация записи +совместимы; доступность этого API с рабочей машины отличалась от серверной. diff --git a/docs/september-checkup-shared-api.md b/docs/september-checkup-shared-api.md new file mode 100644 index 0000000..bea58ce --- /dev/null +++ b/docs/september-checkup-shared-api.md @@ -0,0 +1,52 @@ +# Общие исправления API: сентябрь 2026 + +`GET /api/v2/organizations/` и `GET /api/v2/organization-source-records/` +используют одинаковый поиск по каноническим наименованиям (`name`, `full_name`, +`short_name`), ИНН, КПП, ОГРН, ОГРИП и ОКПО. Слова запроса соединяются через AND, +поля для каждого слова — через OR. Совпадение только в `payload`, URL, названии +записи или технических идентификаторах не включает организацию в результат. +Поиск и активные фильтры применяются до пагинации. + +`GET /api/v1/system/logs/` и `GET /api/v1/system/logs/export/` принимают +`date_from`/`date_to` в формате `YYYY-MM-DD`. Границы включительны по `updated_at` +в часовом поясе приложения (`TIME_ZONE`, сейчас UTC). Один `date_from` означает +один день; один `date_to` ограничивает только конец периода. Некорректные даты, +перевёрнутый период, неверный `batch_id` или неизвестный `ordering` возвращают 400. + +Порядок по умолчанию — `-updated_at, id`. Поддерживаются `updated_at`, `source`, +`status`, `records_count` и обратные направления; сохранены существующие поля +`id`, `batch_id`, `created_at`, `source_label`, `status_label`, +`organizations_count` и сортировка по нескольким полям через запятую. +Вторичный порядок по `id` стабилизирует строки с одинаковым значением. +`source` сортируется по публичному slug карточки. Фильтры и порядок list/export +совпадают, но export не ограничивается страницей. Права администратора сохранены. + +История экспортируется как `update-history.csv`: UTF-8 с BOM, разделитель `;`, +CRLF, русские заголовки и подписи источников/статусов. Даты отображаются в часовом +поясе приложения. Первые столбцы: «№», «Дата актуализации», «Источник», «Статус», +«Количество записей»; далее идут сведения о пакете, организациях и результатах импорта. + +В XLSX санкций заголовки `rn`, `ogrn`, `inn`, `okpo` заменены русскими подписями, +включая каждый файл при разбиении выгрузки. Порядок колонок, строковые +идентификаторы, JSON/CSV санкций и ticket/download API сохранены. + +После обновления backend необходимо пересобрать подготовленные выгрузки командой +`uv run python src/manage.py build_source_record_exports` в настроенном окружении +сервиса: скачивание отдаёт опубликованные файлы, а не формирует книгу при запросе. + +При отмене загрузки вакансий между сохранением записей и контрольной точкой +метаданные завершённой задачи остаются неизменными; следующий запуск повторяет +последнюю организацию от сохранённой позиции в том же batch, включая позицию 0, +если пакет уже содержит записи. Идемпотентное сохранение предотвращает дубликаты. + + +Прогресс фоновых задач обновляется условным атомарным запросом: позднее меньшее +значение не уменьшает процент, успешное завершение устанавливает 100%, ошибка +сохраняет достигнутый процент и причину. Успех, ошибка и отмена окончательны; +запоздавший callback или запрос отмены не меняет завершённую задачу. + +При ручном запуске карточки весь набор `refresh_task_ids` записывается до отправки +первой задачи в очередь. Среднее включает завершённые части запуска. Если старый +разрешённый запуск ещё работает, он остаётся в `active_tasks`; новая завершённая +группа его не скрывает. Новый запуск нового реестра блокируется кодом 409 до +завершения предыдущего. diff --git a/pyproject.toml b/pyproject.toml index 3961748..946b81f 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -24,6 +24,7 @@ dependencies = [ "numpy==1.24.4", "requests==2.31.0", "beautifulsoup4==4.12.3", + "lxml==6.0.2", # Web scraping "scrapy==2.11.2", "selenium==4.17.2", diff --git a/src/apps/core/models.py b/src/apps/core/models.py index b258920..bacb9bc 100644 --- a/src/apps/core/models.py +++ b/src/apps/core/models.py @@ -12,6 +12,7 @@ from typing import Any from apps.core.mixins import TimestampMixin from django.db import models +from django.db.models.functions import Coalesce from django.utils import timezone from django.utils.translation import gettext_lazy as _ @@ -141,81 +142,69 @@ class BackgroundJob(TimestampMixin, models.Model): # ==================== Методы обновления статуса ==================== + def _unfinished_queryset(self): + """Условие проверяется в UPDATE, включая события от старого экземпляра.""" + return type(self).objects.filter( + pk=self.pk, + status__in=[JobStatus.PENDING, JobStatus.STARTED, JobStatus.RETRY], + ) + + def _update_unfinished(self, **values: Any) -> None: + self._unfinished_queryset().update(updated_at=timezone.now(), **values) + self.refresh_from_db() + def mark_started(self) -> None: - """Отметить задачу как начатую.""" - self.status = JobStatus.STARTED - self.started_at = timezone.now() - self.save(update_fields=["status", "started_at", "updated_at"]) + """Начать задачу, сохранив время первого запуска после retry.""" + self._update_unfinished( + status=JobStatus.STARTED, + started_at=Coalesce("started_at", timezone.now()), + ) - def update_progress(self, progress: int, message: str = "") -> None: - """ - Обновить прогресс выполнения. - - Args: - progress: Процент выполнения (0-100) - message: Описание текущего этапа - """ - self.progress = min(max(progress, 0), 100) - self.progress_message = message - self.save(update_fields=["progress", "progress_message", "updated_at"]) + def update_progress( + self, progress: int, message: str = "", *, meta: dict | None = None + ) -> None: + """Атомарно сохранить только неубывающий прогресс незавершённой задачи.""" + progress = min(max(int(progress), 0), 100) + values = { + "progress": progress, + "progress_message": message, + "updated_at": timezone.now(), + } + if meta is not None: + values["meta"] = meta + self._unfinished_queryset().filter(progress__lte=progress).update(**values) + self.refresh_from_db() def complete(self, result: Any = None) -> None: - """ - Отметить задачу как успешно завершённую. - - Args: - result: Результат выполнения (сериализуемый в JSON) - """ - self.status = JobStatus.SUCCESS - self.progress = 100 - self.result = result - self.error = "" - self.traceback = "" - self.completed_at = timezone.now() - self.save( - update_fields=[ - "status", - "progress", - "result", - "error", - "traceback", - "completed_at", - "updated_at", - ] + """Первое финальное событие фиксирует успешный результат и 100%.""" + self._update_unfinished( + status=JobStatus.SUCCESS, + progress=100, + result=result, + error="", + traceback="", + completed_at=timezone.now(), ) def fail(self, error: str, traceback_str: str = "") -> None: - """ - Отметить задачу как завершённую с ошибкой. - - Args: - error: Текст ошибки - traceback_str: Полный traceback - """ - self.status = JobStatus.FAILURE - self.error = str(error) - self.traceback = traceback_str - self.completed_at = timezone.now() - self.save( - update_fields=[ - "status", - "error", - "traceback", - "completed_at", - "updated_at", - ] + """Зафиксировать ошибку без изменения достигнутого прогресса.""" + self._update_unfinished( + status=JobStatus.FAILURE, + error=str(error), + traceback=traceback_str, + completed_at=timezone.now(), ) - def revoke(self) -> None: - """Отметить задачу как отменённую.""" - self.status = JobStatus.REVOKED - self.completed_at = timezone.now() - self.save(update_fields=["status", "completed_at", "updated_at"]) + def revoke(self, message: str | None = None) -> None: + """Отменить незавершённую задачу, сохранив её прогресс.""" + values = {"status": JobStatus.REVOKED, "completed_at": timezone.now()} + if message is not None: + values["progress_message"] = message + self._update_unfinished(**values) def mark_retry(self) -> None: - """Отметить, что задача будет повторена.""" - self.status = JobStatus.RETRY - self.save(update_fields=["status", "updated_at"]) + """Отметить повторную попытку, не открывая завершённую задачу заново.""" + self._update_unfinished(status=JobStatus.RETRY) # ==================== Свойства ==================== diff --git a/src/apps/core/views.py b/src/apps/core/views.py index 8e88940..2ebd2ce 100644 --- a/src/apps/core/views.py +++ b/src/apps/core/views.py @@ -395,7 +395,6 @@ class BackgroundJobControlView(BackgroundJobStatusView): }, ) def post(self, request: Request, task_id: str) -> Response: - from apps.core.models import JobStatus from apps.core.services import BackgroundJobService from celery import current_app @@ -415,10 +414,7 @@ class BackgroundJobControlView(BackgroundJobStatusView): task_id, terminate=bool(request.data.get("terminate", False)), ) - if not job.is_finished: - job.status = JobStatus.REVOKED - job.progress_message = "Задача отозвана пользователем" - job.save(update_fields=["status", "progress_message", "updated_at"]) + job.revoke(message="Задача отозвана пользователем") return Response(BackgroundJobSerializer(job).data) diff --git a/src/apps/parsers/budget_registry.py b/src/apps/parsers/budget_registry.py new file mode 100644 index 0000000..5bce3f0 --- /dev/null +++ b/src/apps/parsers/budget_registry.py @@ -0,0 +1,257 @@ +"""Scan the budget registry, fetching and retaining detail only for own organizations.""" + +from __future__ import annotations + +import json +import tempfile +from collections import Counter +from datetime import datetime + +from apps.parsers.models import ParserSourceArtifact, ParserStagedRecord +from apps.parsers.registry_http import ( + limited_bytes, + registry_response, + registry_session, +) +from apps.parsers.registry_snapshots import ( + SNAPSHOT_CHUNK_SIZE, + OwnOrganizationIndex, + SnapshotResult, + SnapshotValidationError, + publish_snapshot, + reject_snapshot, + save_artifact_file, + stage_records, +) + +BUDGET_SOURCE = "budget_ubpandnubp" +BUDGET_RECORD_TYPE = "budget_registry_organization" +BUDGET_URL = "https://budget.gov.ru/epbs/registry/ubpandnubp/data" +BUDGET_HOSTS = frozenset({"budget.gov.ru"}) +BUDGET_PAGE_SIZE = 1000 +BUDGET_MAX_PAGE_BYTES = 32 * 1024**2 + + +def _text(value: object) -> str: + return "" if value is None else str(value).strip() + + +def _date(value: object) -> str: + text = _text(value) + if not text: + return "" + for pattern in ("%Y-%m-%d", "%d.%m.%Y"): + try: + return datetime.strptime(text[:10], pattern).date().isoformat() + except ValueError: + continue + raise SnapshotValidationError("invalid_budget_date") + + +def _dictionary(info: dict, code: str, name: str) -> dict: + return {"code": _text(info.get(code)), "name": _text(info.get(name))} + + +def normalize_budget_record(record: dict) -> dict: + """Expose useful list fields and keep every upstream block for detail/export.""" + info = record["info"] + status_code = _text(info.get("statusCode") or info.get("status")) + payload = { + "registry": { + "code": _text(info.get("code")), + "registration_number": _text(info.get("regNum")), + "record_number": _text(info.get("recordNum")), + "guid": _text(info.get("guid")), + "parent_record_number": _text(info.get("parentrecordnum")), + "status_code": status_code, + "status_name": _text(info.get("statusName")), + }, + "classification": { + "organization_type": _dictionary(info, "orgTypeCode", "orgTypeName"), + "establishment_kind": _dictionary( + info, "establishmentKindCode", "establishmentKindName" + ), + }, + "budget": { + "level": _dictionary(info, "budgetLvlCode", "budgetLvlName"), + "code": _text(info.get("budgetCode")), + "name": _text(info.get("budgetName")), + }, + "address": {"region": _dictionary(info, "regionCode", "regionName")}, + # Preserve the complete info and all known/unknown blocks rather than + # projecting an unverified 18-block upstream schema into lossy fields. + "upstream": record, + } + return { + "title": _text(info.get("fullName") or info.get("shortName")), + "record_date": _date( + info.get("dateUpdate") or info.get("lastRegDate") or info.get("loadDate") + ), + "status": "inactive" if status_code == "2" else "unknown", + "url": f"{BUDGET_URL}?filterid={record['id']}", + "payload": payload, + } + + +def budget_page(session, params: dict) -> dict: + with registry_response( + session, BUDGET_URL, hosts=BUDGET_HOSTS, params=params + ) as response: + body = limited_bytes(response, BUDGET_MAX_PAGE_BYTES) + result = json.loads(body) + if not isinstance(result, dict) or not isinstance(result.get("data"), list): + raise SnapshotValidationError("invalid_budget_envelope") + return result + + +def _validate_record(record: object) -> dict: + if ( + not isinstance(record, dict) + or not _text(record.get("id")) + or not isinstance(record.get("info"), dict) + ): + raise SnapshotValidationError("invalid_budget_record") + return record + + +def _own_budget_detail(session, summary: dict, index: OwnOrganizationIndex): + info = summary["info"] + organization, reason = index.resolve( + inn=_text(info.get("inn")), + ogrn=_text(info.get("ogrn")), + kpp=_text(info.get("kpp")), + okpo=_text(info.get("okpoCode")), + ) + if reason: + return None, organization, reason + external_id = _text(summary["id"]) + detail_page = budget_page(session, {"filterid": external_id, "pageSize": 1}) + detail_rows = detail_page["data"] + if len(detail_rows) != 1: + raise SnapshotValidationError("incomplete_budget_detail") + detail = _validate_record(detail_rows[0]) + if _text(detail["id"]) != external_id: + raise SnapshotValidationError("unexpected_budget_detail") + detail_info = detail["info"] + matched, detail_reason = index.resolve( + inn=_text(detail_info.get("inn")), + ogrn=_text(detail_info.get("ogrn")), + kpp=_text(detail_info.get("kpp")), + okpo=_text(detail_info.get("okpoCode")), + ) + if detail_reason or matched is None or matched.uid != organization.uid: + raise SnapshotValidationError("budget_identity_changed") + return detail, organization, "" + + +def _iter_budget_summaries(session, metadata: dict): + """Reject mixed/incomplete pages before a staged snapshot can be published.""" + page_number = 1 + expected_total = None + version = None + seen = set() + while True: + page = budget_page( + session, + { + "pageNum": page_number, + "pageSize": BUDGET_PAGE_SIZE, + "blocks": "info", + "order": "id", + "orderDirection": "asc", + }, + ) + total = int(page["recordCount"]) + if expected_total is None: + expected_total, version = total, _text(page.get("version")) + if total != expected_total or _text(page.get("version")) != version: + raise SnapshotValidationError("budget_snapshot_changed") + if int(page.get("pageNum", page_number)) != page_number: + raise SnapshotValidationError("unexpected_budget_page") + rows = page["data"] + if not rows and len(seen) != expected_total: + raise SnapshotValidationError("incomplete_budget_snapshot") + for value in rows: + summary = _validate_record(value) + external_id = _text(summary["id"]) + if external_id in seen: + raise SnapshotValidationError("duplicate_budget_record") + seen.add(external_id) + yield len(seen), summary + if len(seen) > expected_total: + raise SnapshotValidationError("budget_record_count_mismatch") + if len(seen) == expected_total: + break + page_number += 1 + metadata.update(pages_count=page_number, parsed_count=len(seen), version=version) + + +def _scan_budget(session, artifact: ParserSourceArtifact, raw_handle) -> None: + index = OwnOrganizationIndex() + pending = [] + reasons: Counter = Counter() + skipped = 0 + metadata = {} + for row_number, summary in _iter_budget_summaries(session, metadata): + detail, organization, reason = _own_budget_detail(session, summary, index) + if reason == "outside_directory": + skipped += 1 + continue + if reason: + reasons[reason] += 1 + continue + raw_handle.write( + (json.dumps(detail, ensure_ascii=False) + "\n").encode("utf-8") + ) + pending.append( + ParserStagedRecord( + artifact=artifact, + row_number=row_number, + external_id=_text(summary["id"]), + record_type=BUDGET_RECORD_TYPE, + organization=organization, + raw_data=detail, + normalized_data=normalize_budget_record(detail), + ) + ) + if len(pending) >= SNAPSHOT_CHUNK_SIZE: + stage_records(pending) + stage_records(pending) + artifact.parsed_count = metadata["parsed_count"] + artifact.quarantined_count = sum(reasons.values()) + artifact.rejection_reasons = dict(reasons) + artifact.version = metadata["version"] + artifact.metadata = { + "pages_count": metadata["pages_count"], + "skipped_records_count": skipped, + } + + +def refresh_budget_registry( + *, + load_batch: int, + uploaded_by_id: int | None = None, + session=None, + on_publish=None, +) -> tuple[ParserSourceArtifact, SnapshotResult]: + artifact = ParserSourceArtifact.objects.create( + source=BUDGET_SOURCE, + original_name=f"budget-own-organizations-{load_batch}.jsonl", + content_type="application/x-ndjson", + load_batch=load_batch, + uploaded_by_id=uploaded_by_id, + ) + client = session or registry_session() + try: + with tempfile.TemporaryFile() as raw_handle: + _scan_budget(client, artifact, raw_handle) + save_artifact_file(artifact, raw_handle) + artifact.status = ParserSourceArtifact.Status.PARSED + artifact.save() + return artifact, publish_snapshot(artifact, on_publish=on_publish) + except Exception as exc: + reject_snapshot(artifact, exc) + raise + finally: + if session is None: + client.close() diff --git a/src/apps/parsers/migrations/0034_support_and_budget_registry_sources.py b/src/apps/parsers/migrations/0034_support_and_budget_registry_sources.py new file mode 100644 index 0000000..c26b9eb --- /dev/null +++ b/src/apps/parsers/migrations/0034_support_and_budget_registry_sources.py @@ -0,0 +1,164 @@ +# Generated by Django 3.2.25 on 2026-09-13 20:12 + +from django.db import migrations, models + + +class Migration(migrations.Migration): + dependencies = [ + ("parsers", "0033_auto_20260820_1325"), + ] + + operations = [ + migrations.AlterField( + model_name="genericparserrecord", + name="source", + field=models.CharField( + choices=[ + ("industrial", "Сертификаты промышленного производства"), + ("industrial_products", "Реестр промышленной продукции"), + ("manufactures", "Реестр производителей"), + ("inspections", "Единый реестр проверок"), + ("procurements", "Единая информационная система закупок"), + ("fns_reports", "Бухгалтерская отчетность ФНС"), + ("procurements_44fz", "Закупки 44-ФЗ"), + ("procurements_223fz", "Закупки 223-ФЗ"), + ("contracts", "Контракты ЕИС"), + ("unfair_suppliers", "Недобросовестные поставщики"), + ("fas_goz", "Уклонение от ГОЗ"), + ("arbitration", "Арбитражные дела"), + ("fedresurs_bankruptcy", "Банкротства Федресурс"), + ("fstec", "Реестры ФСТЭК"), + ("trudvsem", "Вакансии Работа России"), + ( + "gosedo_address_directory", + "Глобальный адресный справочник ГосЭДО", + ), + ("media_news", "Новости СМИ"), + ("ropk_sanctions", "РОПК — Санкции"), + ( + "fns_sme_support_recipients", + "Реестр субъектов МСП — получателей поддержки", + ), + ("budget_ubpandnubp", "Реестр участников бюджетного процесса"), + ("hh", "Вакансии HeadHunter"), + ("superjob", "Вакансии SuperJob"), + ], + db_index=True, + help_text="Источник данных", + max_length=50, + verbose_name="источник", + ), + ), + migrations.AlterField( + model_name="parserbatchsequence", + name="source", + field=models.CharField( + choices=[ + ("industrial", "Сертификаты промышленного производства"), + ("industrial_products", "Реестр промышленной продукции"), + ("manufactures", "Реестр производителей"), + ("inspections", "Единый реестр проверок"), + ("procurements", "Единая информационная система закупок"), + ("fns_reports", "Бухгалтерская отчетность ФНС"), + ("procurements_44fz", "Закупки 44-ФЗ"), + ("procurements_223fz", "Закупки 223-ФЗ"), + ("contracts", "Контракты ЕИС"), + ("unfair_suppliers", "Недобросовестные поставщики"), + ("fas_goz", "Уклонение от ГОЗ"), + ("arbitration", "Арбитражные дела"), + ("fedresurs_bankruptcy", "Банкротства Федресурс"), + ("fstec", "Реестры ФСТЭК"), + ("trudvsem", "Вакансии Работа России"), + ( + "gosedo_address_directory", + "Глобальный адресный справочник ГосЭДО", + ), + ("media_news", "Новости СМИ"), + ("ropk_sanctions", "РОПК — Санкции"), + ( + "fns_sme_support_recipients", + "Реестр субъектов МСП — получателей поддержки", + ), + ("budget_ubpandnubp", "Реестр участников бюджетного процесса"), + ], + help_text="Источник данных", + max_length=50, + unique=True, + verbose_name="источник", + ), + ), + migrations.AlterField( + model_name="parserloadlog", + name="source", + field=models.CharField( + choices=[ + ("industrial", "Сертификаты промышленного производства"), + ("industrial_products", "Реестр промышленной продукции"), + ("manufactures", "Реестр производителей"), + ("inspections", "Единый реестр проверок"), + ("procurements", "Единая информационная система закупок"), + ("fns_reports", "Бухгалтерская отчетность ФНС"), + ("procurements_44fz", "Закупки 44-ФЗ"), + ("procurements_223fz", "Закупки 223-ФЗ"), + ("contracts", "Контракты ЕИС"), + ("unfair_suppliers", "Недобросовестные поставщики"), + ("fas_goz", "Уклонение от ГОЗ"), + ("arbitration", "Арбитражные дела"), + ("fedresurs_bankruptcy", "Банкротства Федресурс"), + ("fstec", "Реестры ФСТЭК"), + ("trudvsem", "Вакансии Работа России"), + ( + "gosedo_address_directory", + "Глобальный адресный справочник ГосЭДО", + ), + ("media_news", "Новости СМИ"), + ("ropk_sanctions", "РОПК — Санкции"), + ( + "fns_sme_support_recipients", + "Реестр субъектов МСП — получателей поддержки", + ), + ("budget_ubpandnubp", "Реестр участников бюджетного процесса"), + ], + db_index=True, + help_text="Источник данных", + max_length=50, + verbose_name="источник", + ), + ), + migrations.AlterField( + model_name="parsersourceartifact", + name="source", + field=models.CharField( + choices=[ + ("industrial", "Сертификаты промышленного производства"), + ("industrial_products", "Реестр промышленной продукции"), + ("manufactures", "Реестр производителей"), + ("inspections", "Единый реестр проверок"), + ("procurements", "Единая информационная система закупок"), + ("fns_reports", "Бухгалтерская отчетность ФНС"), + ("procurements_44fz", "Закупки 44-ФЗ"), + ("procurements_223fz", "Закупки 223-ФЗ"), + ("contracts", "Контракты ЕИС"), + ("unfair_suppliers", "Недобросовестные поставщики"), + ("fas_goz", "Уклонение от ГОЗ"), + ("arbitration", "Арбитражные дела"), + ("fedresurs_bankruptcy", "Банкротства Федресурс"), + ("fstec", "Реестры ФСТЭК"), + ("trudvsem", "Вакансии Работа России"), + ( + "gosedo_address_directory", + "Глобальный адресный справочник ГосЭДО", + ), + ("media_news", "Новости СМИ"), + ("ropk_sanctions", "РОПК — Санкции"), + ( + "fns_sme_support_recipients", + "Реестр субъектов МСП — получателей поддержки", + ), + ("budget_ubpandnubp", "Реестр участников бюджетного процесса"), + ], + db_index=True, + max_length=50, + ), + ), + ] diff --git a/src/apps/parsers/models.py b/src/apps/parsers/models.py index 0b3b44e..c8d3abd 100644 --- a/src/apps/parsers/models.py +++ b/src/apps/parsers/models.py @@ -43,6 +43,14 @@ class ParserLoadLog(TimestampMixin, models.Model): ) MEDIA_NEWS = "media_news", _("Новости СМИ") ROPK_SANCTIONS = "ropk_sanctions", _("РОПК — Санкции") + FNS_SME_SUPPORT_RECIPIENTS = ( + "fns_sme_support_recipients", + _("Реестр субъектов МСП — получателей поддержки"), + ) + BUDGET_UBPANDNUBP = ( + "budget_ubpandnubp", + _("Реестр участников бюджетного процесса"), + ) class Status(models.TextChoices): SUCCESS = "success", _("Успешно") diff --git a/src/apps/parsers/registry_http.py b/src/apps/parsers/registry_http.py new file mode 100644 index 0000000..9909574 --- /dev/null +++ b/src/apps/parsers/registry_http.py @@ -0,0 +1,196 @@ +"""Bounded HTTPS reads for public registry sources with explicit host allowlists.""" + +from __future__ import annotations + +import re +from collections.abc import Iterator +from contextlib import contextmanager +from dataclasses import dataclass +from typing import BinaryIO +from urllib.parse import urljoin, urlparse + +import requests +from apps.parsers.registry_snapshots import SnapshotValidationError +from requests.adapters import HTTPAdapter +from urllib3.util.retry import Retry + + +def registry_session() -> requests.Session: + session = requests.Session() + session.mount( + "https://", + HTTPAdapter( + max_retries=Retry( + total=3, + backoff_factor=1, + allowed_methods={"GET"}, + status_forcelist=(429, 500, 502, 503, 504), + ) + ), + ) + return session + + +@contextmanager +def registry_response( + session: requests.Session, + url: str, + *, + hosts: frozenset[str], + params: dict | None = None, + headers: dict[str, str] | None = None, +) -> Iterator[requests.Response]: + for _ in range(5): + parsed = urlparse(url) + if ( + parsed.scheme != "https" + or parsed.hostname not in hosts + or parsed.username + or parsed.password + or parsed.port not in (None, 443) + ): + raise SnapshotValidationError("unsafe_source_url") + response = session.get( + url, + params=params, + headers=headers, + stream=True, + allow_redirects=False, + timeout=(10, 60), + ) + if response.is_redirect: + location = response.headers.get("Location", "") + response.close() + url = urljoin(url, location) + params = None + continue + try: + response.raise_for_status() + yield response + finally: + response.close() + return + raise SnapshotValidationError("too_many_redirects") + + +def limited_bytes(response: requests.Response, maximum: int) -> bytes: + chunks = [] + size = 0 + for chunk in response.iter_content(chunk_size=64 * 1024): + size += len(chunk) + if size > maximum: + raise SnapshotValidationError("source_response_too_large") + chunks.append(chunk) + return b"".join(chunks) + + +@dataclass +class _DownloadState: + total: int | None = None + validator: tuple[str, str] | None = None + full_response: bool = False + + +def _response_validator(response: requests.Response) -> tuple[str, str]: + etag = response.headers.get("ETag", "") + if etag and not etag.startswith("W/"): + return "ETag", etag + modified = response.headers.get("Last-Modified", "") + if modified: + return "Last-Modified", modified + raise SnapshotValidationError("source_resume_validator_missing") + + +def _validate_archive_range(response, state, offset, requested_end, maximum): + match = re.fullmatch( + r"bytes (\d+)-(\d+)/(\d+)", response.headers.get("Content-Range", "") + ) + if response.status_code != 206 or match is None: + raise SnapshotValidationError("invalid_source_content_range") + start, end, total = map(int, match.groups()) + if start != offset or end != min(requested_end, total - 1) or end < start: + raise SnapshotValidationError("unexpected_source_content_range") + if total > maximum: + raise SnapshotValidationError("source_response_too_large") + validator = _response_validator(response) + if (state.total is not None and state.total != total) or ( + state.validator is not None and state.validator != validator + ): + raise SnapshotValidationError("source_changed_during_download") + state.total, state.validator = total, validator + return end + 1 + + +def _copy_archive_response(response, handle: BinaryIO, maximum: int) -> None: + for chunk in response.iter_content(chunk_size=1024**2): + if handle.tell() + len(chunk) > maximum: + raise SnapshotValidationError("source_response_too_large") + handle.write(chunk) + + +def _read_archive_response(response, handle, state, requested_end, maximum): + if response.headers.get("Content-Encoding", "identity").lower() != "identity": + raise SnapshotValidationError("unexpected_source_content_encoding") + state.full_response = response.status_code == 200 + if state.full_response: + if handle.tell() or state.validator is not None: + raise SnapshotValidationError("source_range_not_honored") + length = response.headers.get("Content-Length") + state.total = int(length) if length is not None else None + if state.total is not None and state.total > maximum: + raise SnapshotValidationError("source_response_too_large") + _copy_archive_response(response, handle, maximum) + expected_end = state.total if state.total is not None else handle.tell() + state.total = expected_end + else: + expected_end = _validate_archive_range( + response, state, handle.tell(), requested_end, maximum + ) + _copy_archive_response(response, handle, expected_end) + if handle.tell() != expected_end: + raise requests.exceptions.ChunkedEncodingError("incomplete_source_response") + + +def download_registry_archive( + session: requests.Session, + url: str, + handle: BinaryIO, + *, + hosts: frozenset[str], + maximum: int, + range_bytes: int = 16 * 1024**2, + max_retries: int = 3, +) -> int: + """Resume interrupted bodies only when the same immutable HTTP entity is proven.""" + state = _DownloadState() + retries = 0 + handle.seek(0) + handle.truncate() + while state.total is None or handle.tell() < state.total: + requested_end = handle.tell() + range_bytes - 1 + headers = { + "Range": f"bytes={handle.tell()}-{requested_end}", + "Accept-Encoding": "identity", + } + if state.validator is not None: + headers["If-Range"] = state.validator[1] + try: + with registry_response( + session, url, hosts=hosts, headers=headers + ) as response: + _read_archive_response(response, handle, state, requested_end, maximum) + retries = 0 + except ( + requests.ConnectionError, + requests.Timeout, + requests.exceptions.ChunkedEncodingError, + ): + if state.full_response: + # A server without Range support can only be retried from zero. + handle.seek(0) + handle.truncate() + state = _DownloadState() + retries += 1 + if retries > max_retries: + raise + return handle.tell() diff --git a/src/apps/parsers/registry_snapshots.py b/src/apps/parsers/registry_snapshots.py new file mode 100644 index 0000000..9e29c6d --- /dev/null +++ b/src/apps/parsers/registry_snapshots.py @@ -0,0 +1,239 @@ +"""Bounded staging and atomic publication of organization registry snapshots.""" + +from __future__ import annotations + +import hashlib +import logging +import uuid +from collections import defaultdict +from collections.abc import Callable +from dataclasses import dataclass +from decimal import Decimal +from itertools import islice +from typing import BinaryIO + +from apps.parsers.models import ParserSourceArtifact, ParserStagedRecord +from django.core.files import File +from django.db import transaction +from django.db.models import Count +from organizations.models import Organization, OrganizationSourceRecord +from organizations.source_cache import invalidate_source_data_cache +from organizations.source_groups import get_source_group_descriptor +from organizations.source_ingestion import ( + OrganizationSourceIngestionService, + SourceRecordInput, +) + +SNAPSHOT_CHUNK_SIZE = 500 +SNAPSHOT_UUID_NAMESPACE = uuid.UUID("c7b9a4ae-e2ae-47b0-9e09-d550677c598e") +logger = logging.getLogger(__name__) + + +class SnapshotValidationError(ValueError): + """The input cannot replace the last complete published snapshot.""" + + +@dataclass(frozen=True) +class SnapshotResult: + parsed: int + published: int + quarantined: int + skipped: int = 0 + + +class OwnOrganizationIndex: + """Index only the authoritative directory; never create upstream subjects.""" + + def __init__(self) -> None: + self.by_inn: dict[str, list[Organization]] = defaultdict(list) + for organization in ( + Organization.objects.filter(directory_imported_at__isnull=False) + .only("uid", "name", "inn", "ogrn", "kpp", "okpo", "is_branch") + .iterator() + ): + if organization.inn: + self.by_inn[organization.inn].append(organization) + + def resolve( + self, *, inn: str, ogrn: str, kpp: str = "", okpo: str = "" + ) -> tuple[Organization | None, str]: + candidates = self.by_inn.get(inn, []) + if not candidates: + return None, "outside_directory" + if ogrn: + candidates = [ + organization for organization in candidates if organization.ogrn == ogrn + ] + if kpp: + candidates = [ + organization for organization in candidates if organization.kpp == kpp + ] + if okpo: + candidates = [ + organization for organization in candidates if organization.okpo == okpo + ] + if not candidates: + return None, "identity_conflict" + if len(candidates) > 1 and not kpp and not okpo: + heads = [ + organization + for organization in candidates + if not organization.is_branch + ] + if len(heads) == 1: + candidates = heads + if len(candidates) != 1: + return None, "ambiguous_organization" + organization = candidates[0] + if not all( + (organization.name, organization.okpo, organization.inn, organization.ogrn) + ): + return organization, "incomplete_organization" + return organization, "" + + +def stable_registry_uid(source: str, external_id: str) -> uuid.UUID: + return uuid.uuid5(SNAPSHOT_UUID_NAMESPACE, f"{source}:{external_id}") + + +def save_artifact_file(artifact: ParserSourceArtifact, handle: BinaryIO) -> None: + """Checksum and persist the source stream without loading it into memory.""" + handle.seek(0) + digest = hashlib.sha256() + size = 0 + for chunk in iter(lambda: handle.read(1024 * 1024), b""): + digest.update(chunk) + size += len(chunk) + artifact.sha256 = digest.hexdigest() + artifact.size_bytes = size + handle.seek(0) + artifact.file.save(artifact.original_name, File(handle), save=False) + artifact.save() + + +def stage_records(rows: list[ParserStagedRecord]) -> None: + if rows: + ParserStagedRecord.objects.bulk_create(rows, batch_size=SNAPSHOT_CHUNK_SIZE) + rows.clear() + + +def _artifact_result(artifact: ParserSourceArtifact) -> SnapshotResult: + return SnapshotResult( + parsed=artifact.parsed_count, + published=artifact.published_count, + quarantined=artifact.quarantined_count, + skipped=artifact.metadata.get("skipped_records_count", 0), + ) + + +def _refresh_extension_counts(descriptor) -> None: + extensions = descriptor.extension_model.objects.annotate( + actual_count=Count("records") + ) + for extension in extensions.iterator(): + if extension.records_count != extension.actual_count: + descriptor.extension_model.objects.filter(pk=extension.pk).update( + records_count=extension.actual_count + ) + + +def _invalidate_committed_snapshot_cache() -> None: + """Close the pre-commit cache refill window without rejecting committed data.""" + try: + invalidate_source_data_cache() + except Exception as exc: + logger.error( + "Published registry cache invalidation failed (%s)", type(exc).__name__ + ) + + +def publish_snapshot( + artifact: ParserSourceArtifact, + *, + on_publish: Callable[[ParserSourceArtifact, SnapshotResult], None] | None = None, +) -> SnapshotResult: + """Publish complete staging in one transaction, preserving stable record IDs.""" + source = artifact.source + descriptor = get_source_group_descriptor(source) + staged = ParserStagedRecord.objects.filter( + artifact=artifact, disposition=ParserStagedRecord.Disposition.STAGED + ) + if staged.values("external_id").annotate(n=Count("uid")).filter(n__gt=1).exists(): + raise SnapshotValidationError("duplicate_registry_number") + + with transaction.atomic(): + # Refresh API serializes jobs per source; lock this artifact as an additional + # guard against processing the same downloaded input concurrently. + locked_artifact = ParserSourceArtifact.objects.select_for_update().get( + pk=artifact.pk + ) + if locked_artifact.status == ParserSourceArtifact.Status.PUBLISHED: + return _artifact_result(locked_artifact) + if locked_artifact.status != ParserSourceArtifact.Status.PARSED: + raise SnapshotValidationError("snapshot_not_ready_for_publication") + iterator = ( + staged.select_related("organization") + .order_by("row_number") + .iterator(chunk_size=SNAPSHOT_CHUNK_SIZE) + ) + published = 0 + while rows := list(islice(iterator, SNAPSHOT_CHUNK_SIZE)): + inputs = [] + for row in rows: + organization = row.organization + if organization is None: + raise SnapshotValidationError("organization_resolution_changed") + data = row.normalized_data + inputs.append( + SourceRecordInput( + uid=stable_registry_uid(source, row.external_id), + organization_uid=organization.uid, + external_id=row.external_id, + record_type=row.record_type, + title=data.get("title") or organization.name, + organization_name=organization.name, + inn=organization.inn, + ogrn=organization.ogrn, + record_date=data.get("record_date", ""), + amount=Decimal(data["amount"]) + if data.get("amount") is not None + else None, + status=data.get("status", ""), + url=data.get("url", ""), + payload=data["payload"], + ) + ) + result = OrganizationSourceIngestionService.save_records( + source=source, load_batch=artifact.load_batch, records=inputs + ) + if result.unresolved: + raise SnapshotValidationError("organization_resolution_changed") + published += len(inputs) + retained_ids = staged.values_list("external_id", flat=True) + OrganizationSourceRecord.objects.filter(source=source).exclude( + external_id__in=retained_ids + ).delete() + descriptor.extension_model.objects.filter(records__isnull=True).delete() + # Deletion can change counts for extensions whose remaining rows were upserted + # before old records were removed. + _refresh_extension_counts(descriptor) + staged.update(disposition=ParserStagedRecord.Disposition.PUBLISHED) + artifact.published_count = published + artifact.status = ParserSourceArtifact.Status.PUBLISHED + artifact.save() + result = _artifact_result(artifact) + if on_publish is not None: + on_publish(artifact, result) + # Cache failure must roll back the new records and task finalization too. + invalidate_source_data_cache() + transaction.on_commit(_invalidate_committed_snapshot_cache) + return result + + +def reject_snapshot(artifact: ParserSourceArtifact, error: Exception) -> None: + metadata = {**artifact.metadata, "error_code": type(error).__name__} + if isinstance(error, SnapshotValidationError): + metadata["rejection_reason"] = str(error)[:200] + ParserSourceArtifact.objects.filter(pk=artifact.pk).exclude( + status=ParserSourceArtifact.Status.PUBLISHED + ).update(status=ParserSourceArtifact.Status.REJECTED, metadata=metadata) diff --git a/src/apps/parsers/schemas/fns_sme_support_structure_20230615.xsd b/src/apps/parsers/schemas/fns_sme_support_structure_20230615.xsd new file mode 100644 index 0000000..3fe0c83 --- /dev/null +++ b/src/apps/parsers/schemas/fns_sme_support_structure_20230615.xsd @@ -0,0 +1,582 @@ + + + + + + + + + + + + "" "" + + + + + + + + + + + + + + + + + + , , + + + + + + + + + + + + + + + + + + + + + + + + + + + + E-mail + + + + + + + + + + + + + + + + + + + + , + + + + + + + + + + + + + + + + + + + + + + + + + + + + , + + + + + + , , + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + ( , ) + + + + + , , + + + + + + + + + + + + , + + + + + , + + + + + + + + + + + , + + + + + + + + + + + , , + + + + + + + + + + + , + + + + + , + + + + + + + + + + + , + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + , + + + + + + + + + + + , + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + , , + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - + + + + + + + + + - + + + + + + + + + .. (01.01.1900 - 31.12.2099) + + + + + + + diff --git a/src/apps/parsers/serializers.py b/src/apps/parsers/serializers.py index 72139c2..b48c3de 100644 --- a/src/apps/parsers/serializers.py +++ b/src/apps/parsers/serializers.py @@ -1103,6 +1103,12 @@ class FrontendApiResponseSerializer(serializers.Serializer): meta = serializers.JSONField(read_only=True, allow_null=True) +class ParserRunEnvelopeSerializer(FrontendApiResponseSerializer): + """Pollable parser task inside the standard API envelope.""" + + data = ParserRunResponseSerializer(read_only=True) + + class SourceCardListResponseSerializer(FrontendApiResponseSerializer): """Envelope для списка карточек источников.""" diff --git a/src/apps/parsers/sme_support.py b/src/apps/parsers/sme_support.py new file mode 100644 index 0000000..d064733 --- /dev/null +++ b/src/apps/parsers/sme_support.py @@ -0,0 +1,305 @@ +"""Stream the official FNS support-measures ZIP into the organization directory.""" + +from __future__ import annotations + +import re +import tempfile +import zipfile +from collections import Counter +from datetime import datetime +from decimal import Decimal +from pathlib import Path +from typing import BinaryIO +from urllib.parse import urljoin + +from apps.parsers.models import ParserSourceArtifact, ParserStagedRecord +from apps.parsers.registry_http import ( + download_registry_archive, + limited_bytes, + registry_response, + registry_session, +) +from apps.parsers.registry_snapshots import ( + SNAPSHOT_CHUNK_SIZE, + OwnOrganizationIndex, + SnapshotResult, + SnapshotValidationError, + publish_snapshot, + reject_snapshot, + save_artifact_file, + stage_records, +) +from bs4 import BeautifulSoup +from lxml import etree + +SME_SOURCE = "fns_sme_support_recipients" +SME_RECORD_TYPE = "sme_support_measure" +SME_CATALOG_URL = "https://www.nalog.gov.ru/opendata/7707329152-rsmppp/" +SME_HOSTS = frozenset({"www.nalog.gov.ru", "nalog.gov.ru", "file.nalog.ru"}) +SME_MAX_ARCHIVE_BYTES = 2 * 1024**3 +SME_MAX_XML_BYTES = 64 * 1024**3 +SME_SCHEMA_PATH = ( + Path(__file__).with_name("schemas") / "fns_sme_support_structure_20230615.xsd" +) +SME_SIZE_UNITS = { + "1": "RUB", + "2": "square_meter", + "3": "hour", + "4": "percent", + "5": "unit", +} + + +def discover_sme_archive(html: bytes) -> tuple[str, str]: + """Select the most recent dated archive from the official catalog links.""" + candidates = [] + for link in BeautifulSoup(html, "html.parser").find_all("a", href=True): + url = urljoin(SME_CATALOG_URL, link["href"]) + match = re.search(r"/data-(\d{8})-structure-20230615\.zip(?:$|\?)", url) + if match: + snapshot_date = datetime.strptime(match[1], "%Y%m%d").date().isoformat() + candidates.append((snapshot_date, url)) + if not candidates: + raise SnapshotValidationError("sme_archive_not_found") + snapshot_date, url = max(candidates) + return url, snapshot_date + + +def _date(value: str | None) -> str | None: + return datetime.strptime(value, "%d.%m.%Y").date().isoformat() if value else None + + +def _dictionary(element, code: str, name: str) -> dict[str, str]: + if element is None: + raise SnapshotValidationError("missing_support_dictionary") + return {"code": element.attrib[code], "name": element.attrib[name]} + + +def support_payload( + document_id: str, measure, snapshot_date: str +) -> tuple[dict, str | None]: + sizes = [ + { + "unit_code": element.attrib["ЕдПод"], + "unit": SME_SIZE_UNITS[element.attrib["ЕдПод"]], + "value": format(Decimal(element.attrib["РазмПод"]), ".2f"), + } + for element in measure.findall("РазмПод") + ] + if not sizes: + raise SnapshotValidationError("missing_support_size") + rub_sizes = [Decimal(size["value"]) for size in sizes if size["unit"] == "RUB"] + # Keep all upstream attributes of nested elements: no flattening or loss of + # less common regulatory/violation fields in the universal JSON payload. + violations = [dict(element.attrib) for element in measure.findall("Нарушения")] + documents = [dict(element.attrib) for element in measure.findall("РегДок")] + payload = { + "support_registry_number": measure.attrib["НомерПод"], + "source_document_id": document_id, + "recipient_type": measure.attrib["ВидПП"], + "sme_category": measure.attrib["КатСуб"], + "support_form": _dictionary(measure.find("ФормПод"), "КодФорм", "НаимФорм"), + "support_kind": _dictionary(measure.find("ВидПод"), "КодВид", "НаимВид"), + "decision_date": _date(measure.attrib["ДатаПрин"]), + "support_until": _date(measure.attrib["СрокПод"]), + "termination_date": _date(measure.get("ДатаПрекр")), + "registry_entry_date": _date(measure.attrib["ДатаСвед"]), + "source_updated_date": _date(measure.get("ДатаОбнов")), + "support_sizes": sizes, + "provider": {"name": measure.attrib["НаимОрг"], "inn": measure.attrib["ИННЮЛ"]}, + "has_violation": measure.attrib["ИнфНаруш"] == "1", + "violation_count": len(violations), + "regulatory_documents_count": len(documents), + "violations": violations, + "regulatory_documents": documents, + "source_snapshot_date": snapshot_date, + } + return payload, format(sum(rub_sizes), ".2f") if rub_sizes else None + + +def _clear_element(element) -> None: + element.clear() + while element.getprevious() is not None: + del element.getparent()[0] + + +def iter_sme_measures(handle: BinaryIO, counts: Counter): + """Validate incrementally and release each measure, including large recipients.""" + schema = etree.XMLSchema(etree.parse(str(SME_SCHEMA_PATH))) # noqa: S320 - bundled immutable schema + context = etree.iterparse( + handle, + events=("start", "end"), + schema=schema, + resolve_entities=False, + no_network=True, + huge_tree=False, + ) + expected_documents = 0 + document_id = "" + recipient = None + for event, element in context: + if event == "start" and element.tag == "Файл": + expected_documents = int(element.attrib["КолДок"]) + if event == "start" and element.tag == "Документ": + document_id = element.attrib["ИдДок"] + recipient = None + if event != "end": + continue + if element.tag == "СвЮЛ": + recipient = dict(element.attrib) + if element.tag == "СвПредПод": + counts["measures"] += 1 + yield document_id, recipient, element + _clear_element(element) + if element.tag == "Документ": + counts["documents"] += 1 + _clear_element(element) + if context.root.getroottree().docinfo.doctype: + raise SnapshotValidationError("xml_doctype_not_allowed") + counts["declared_documents"] += expected_documents + if counts["documents"] != expected_documents: + # The official 2026-08-15 archive declares КолДок=1 in files containing + # hundreds of documents. EOF, XSD and ZIP CRC establish completeness; + # this unreliable upstream counter remains an artifact diagnostic. + counts["document_count_mismatch_files"] += 1 + + +def _stage_sme_xml( + handle: BinaryIO, artifact: ParserSourceArtifact, index: OwnOrganizationIndex +) -> Counter: + counts: Counter = Counter() + pending = [] + for document_id, recipient, measure in iter_sme_measures(handle, counts): + if recipient is None: + organization, reason = None, "unsupported_recipient" + else: + organization, reason = index.resolve( + inn=recipient["ИННЮЛ"], ogrn=recipient["ОГРН"] + ) + if reason in ("outside_directory", "unsupported_recipient"): + counts["skipped"] += 1 + continue + payload, amount = support_payload( + document_id, measure, str(artifact.source_published_at) + ) + if reason: + counts[reason] += 1 + pending.append( + ParserStagedRecord( + artifact=artifact, + row_number=artifact.parsed_count + counts["measures"], + external_id=measure.attrib["НомерПод"], + record_type=SME_RECORD_TYPE, + organization=organization, + raw_data={"recipient": recipient, "measure": dict(measure.attrib)}, + normalized_data={ + "record_date": payload["decision_date"], + "amount": amount, + "payload": payload, + "url": "https://rmsp-pp.nalog.ru/search.html?m=SupportList", + }, + disposition=( + ParserStagedRecord.Disposition.QUARANTINED + if reason + else ParserStagedRecord.Disposition.STAGED + ), + reason=reason, + ) + ) + if len(pending) >= SNAPSHOT_CHUNK_SIZE: + stage_records(pending) + stage_records(pending) + return counts + + +def import_sme_support( + *, + handle: BinaryIO, + original_name: str, + snapshot_date: str, + load_batch: int, + uploaded_by_id: int | None = None, + on_publish=None, +) -> tuple[ParserSourceArtifact, SnapshotResult]: + artifact = ParserSourceArtifact.objects.create( + source=SME_SOURCE, + original_name=original_name, + source_published_at=snapshot_date, + version="4.04", + load_batch=load_batch, + uploaded_by_id=uploaded_by_id, + content_type="application/zip", + ) + try: + save_artifact_file(artifact, handle) + index = OwnOrganizationIndex() + counts: Counter = Counter() + with zipfile.ZipFile(handle) as archive: + members = [member for member in archive.infolist() if not member.is_dir()] + if ( + not members + or any( + not member.filename.lower().endswith(".xml") for member in members + ) + or sum(member.file_size for member in members) > SME_MAX_XML_BYTES + ): + raise SnapshotValidationError("invalid_sme_archive_members") + for member in members: + with archive.open(member) as xml_stream: + parsed = _stage_sme_xml(xml_stream, artifact, index) + counts.update(parsed) + artifact.parsed_count += parsed["measures"] + if not counts["documents"]: + raise SnapshotValidationError("empty_sme_snapshot") + artifact.quarantined_count = sum( + counts[key] + for key in ( + "identity_conflict", + "ambiguous_organization", + "incomplete_organization", + ) + ) + artifact.rejection_reasons = { + key: counts[key] + for key in ( + "identity_conflict", + "ambiguous_organization", + "incomplete_organization", + ) + if counts[key] + } + artifact.metadata = { + "skipped_records_count": counts["skipped"], + "documents_count": counts["documents"], + "declared_documents_count": counts["declared_documents"], + "document_count_mismatch_files": counts["document_count_mismatch_files"], + } + artifact.status = ParserSourceArtifact.Status.PARSED + artifact.save() + return artifact, publish_snapshot(artifact, on_publish=on_publish) + except Exception as exc: + reject_snapshot(artifact, exc) + raise + + +def refresh_sme_support( + *, load_batch: int, uploaded_by_id: int | None = None, on_publish=None +): + with registry_session() as session: + with registry_response(session, SME_CATALOG_URL, hosts=SME_HOSTS) as response: + url, snapshot_date = discover_sme_archive( + limited_bytes(response, 4 * 1024**2) + ) + with tempfile.TemporaryFile() as handle: + download_registry_archive( + session, url, handle, hosts=SME_HOSTS, maximum=SME_MAX_ARCHIVE_BYTES + ) + handle.seek(0) + return import_sme_support( + handle=handle, + original_name=url.split("/")[-1].split("?")[0], + snapshot_date=snapshot_date, + load_batch=load_batch, + uploaded_by_id=uploaded_by_id, + on_publish=on_publish, + ) diff --git a/src/apps/parsers/source_cards.py b/src/apps/parsers/source_cards.py index 3e83878..2d5a0c8 100644 --- a/src/apps/parsers/source_cards.py +++ b/src/apps/parsers/source_cards.py @@ -5,21 +5,24 @@ from __future__ import annotations import uuid from contextlib import suppress from copy import deepcopy -from dataclasses import dataclass +from dataclasses import dataclass, field from datetime import datetime, timedelta from hashlib import blake2s from typing import Any +from apps.core.exceptions import ConflictError from apps.core.models import JobStatus from apps.core.services import BackgroundJobService from apps.parsers.models import ( VACANCY_RECORD_SOURCES, + ParserBatchSequence, ParserLoadLog, ) from apps.parsers.source_registry import get_source_by_model_source from django.conf import settings from django.core.cache import cache -from django.db.models import Count, Max, Q +from django.db import transaction +from django.db.models import Count, Max, OuterRef, Q, Subquery from django.http import Http404 from django.utils import timezone from organizations.models import OrganizationSourceExtension, OrganizationSourceRecord @@ -96,6 +99,7 @@ class SourceCardStatsContext: latest_load_by_source: dict[str, ParserLoadLog] latest_success_load_by_source: dict[str, ParserLoadLog] active_tasks_by_slug: dict[str, list[dict[str, Any]]] + refresh_tasks_by_slug: dict[str, list[dict[str, Any]]] = field(default_factory=dict) @dataclass(frozen=True) @@ -398,6 +402,38 @@ SOURCE_CARD_DEFINITIONS: tuple[SourceCardDefinition, ...] = ( supports_refresh=False, upload_url="/api/v1/parsers/upload/media_news/", ), + SourceCardDefinition( + slug="sme-support-recipients-registry", + title="Реестр субъектов МСП — получателей поддержки", + description="Меры государственной поддержки организаций.", + order=120, + task_names=("parsers.fns_sme_support_recipients.refresh",), + source_items=( + SourceItemDefinition( + code="fns_sme_support_recipients", + title="Реестр субъектов МСП — получателей поддержки", + description="Официальный реестр ФНС России.", + parser_source=ParserLoadLog.Source.FNS_SME_SUPPORT_RECIPIENTS, + refresh_key="fns_sme_support_recipients", + ), + ), + ), + SourceCardDefinition( + slug="budget-process-registry", + title="Реестр участников бюджетного процесса", + description="Бюджетные сведения организаций канонического реестра.", + order=130, + task_names=("parsers.budget_ubpandnubp.refresh",), + source_items=( + SourceItemDefinition( + code="budget_ubpandnubp", + title="Реестр участников бюджетного процесса", + description="Официальный реестр Федерального казначейства.", + parser_source=ParserLoadLog.Source.BUDGET_UBPANDNUBP, + refresh_key="budget_ubpandnubp", + ), + ), + ), ) SOURCE_CARD_BY_SLUG = {item.slug: item for item in SOURCE_CARD_DEFINITIONS} @@ -544,7 +580,8 @@ class SourceCardService: else cls._get_latest_data_timestamp(source_items) ) active_tasks = context.active_tasks_by_slug.get(definition.slug, []) - progress = cls._get_progress(active_tasks) + refresh_tasks = context.refresh_tasks_by_slug.get(definition.slug, []) + progress = cls._get_progress(refresh_tasks or active_tasks) status = cls._get_status( definition=definition, active_tasks=active_tasks, @@ -554,6 +591,22 @@ class SourceCardService: error_message = latest_load.error_message if latest_load else "" if cls._is_stale_load(latest_load): error_message = cls._stale_load_message() + if refresh_tasks and all( + task["status"] not in ACTIVE_JOB_STATUSES for task in refresh_tasks + ): + if all(task["status"] == JobStatus.SUCCESS for task in refresh_tasks): + status = "success" + error_message = "" + else: + status = "error" + error_message = next( + ( + task.get("error", "") + for task in refresh_tasks + if task.get("error") + ), + error_message, + ) return { "slug": definition.slug, @@ -636,6 +689,7 @@ class SourceCardService: latest_success_load_by_source, ) = cls._build_latest_load_maps(parser_sources) aggregate_stats = cls._get_or_build_aggregate_stats(definitions) + active_tasks_by_slug, refresh_tasks_by_slug = cls._build_task_maps(definitions) return SourceCardStatsContext( item_stats_by_code=aggregate_stats.item_stats_by_code, card_organizations_count_by_slug=( @@ -643,7 +697,8 @@ class SourceCardService: ), latest_load_by_source=latest_load_by_source, latest_success_load_by_source=latest_success_load_by_source, - active_tasks_by_slug=cls._build_active_tasks_by_slug(definitions), + active_tasks_by_slug=active_tasks_by_slug, + refresh_tasks_by_slug=refresh_tasks_by_slug, ) @classmethod @@ -825,46 +880,102 @@ class SourceCardService: return latest_load_by_source, latest_success_load_by_source @classmethod - def _build_active_tasks_by_slug( - cls, - definitions: tuple[SourceCardDefinition, ...], - ) -> dict[str, list[dict[str, Any]]]: - task_names_by_slug = { - definition.slug: set(definition.task_names) - for definition in definitions - if definition.task_names - } + def _build_active_tasks_by_slug(cls, definitions): + return cls._build_task_maps(definitions)[0] + + @classmethod + def _build_task_maps(cls, definitions): + """Прочитать активные задачи и фиксированный состав последнего запуска.""" slugs_by_task_name: dict[str, list[str]] = {} - for slug, task_names in task_names_by_slug.items(): - for task_name in task_names: - slugs_by_task_name.setdefault(task_name, []).append(slug) - - if not slugs_by_task_name: - return {} - - active_tasks_by_slug: dict[str, list[dict[str, Any]]] = { - slug: [] for slug in task_names_by_slug - } - queryset = ( + for definition in definitions: + for task_name in definition.task_names: + slugs_by_task_name.setdefault(task_name, []).append(definition.slug) + slugs = {definition.slug for definition in definitions} + latest = ( + BackgroundJobService.get_queryset() + .filter(task_name=OuterRef("task_name")) + .order_by("-created_at", "-id") + .values("pk")[:1] + ) + jobs = list( BackgroundJobService.get_queryset() .filter( - task_name__in=list(slugs_by_task_name), + Q(task_name__in=slugs_by_task_name) + | Q(meta__source_card__in=list(slugs)) ) - .filter(cls._fresh_active_job_filter()) - .order_by("-created_at") + .filter(cls._fresh_active_job_filter() | Q(pk=Subquery(latest))) + .order_by("-created_at", "-id") ) - - for job in queryset: - for slug in slugs_by_task_name.get(job.task_name, []): - if len(active_tasks_by_slug[slug]) >= 10: - continue - active_tasks_by_slug[slug].append(cls._serialize_job(job)) - - return { - slug: active_tasks - for slug, active_tasks in active_tasks_by_slug.items() - if active_tasks + active_by_slug: dict[str, list[dict[str, Any]]] = {} + leaders = {} + active_jobs_by_slug = {} + for job in jobs: + job_slugs = slugs_by_task_name.get(job.task_name, []) + source_card = job.meta.get("source_card") + if source_card in slugs: + job_slugs = list({*job_slugs, source_card}) + for slug in job_slugs: + leaders.setdefault(slug, job) + if job.status in ACTIVE_JOB_STATUSES and not cls._job_is_stale(job): + active_by_slug.setdefault(slug, []).append(cls._serialize_job(job)) + active_jobs_by_slug.setdefault(slug, []).append(job) + jobs_by_id = {job.task_id: job for job in jobs} + run_ids_by_slug = { + slug: list( + dict.fromkeys( + task_id + for job in [leader, *active_jobs_by_slug.get(slug, [])] + for task_id in (job.meta.get("refresh_task_ids") or [job.task_id]) + if isinstance(task_id, str) + ) + ) + for slug, leader in leaders.items() } + required_ids = { + task_id for task_ids in run_ids_by_slug.values() for task_id in task_ids + } + missing_ids = required_ids - jobs_by_id.keys() + if missing_ids: + jobs_by_id.update( + { + job.task_id: job + for job in BackgroundJobService.get_queryset().filter( + task_id__in=missing_ids + ) + } + ) + refresh_by_slug = {} + for slug, task_ids in run_ids_by_slug.items(): + # Keep completed parts of every live run and of the latest run. + # Overlapping legacy refreshes must not hide one another. + run_jobs = [ + jobs_by_id[task_id] for task_id in task_ids if task_id in jobs_by_id + ] + refresh_by_slug[slug] = [ + {**cls._serialize_job(job), "error": job.error} for job in run_jobs + ] + return active_by_slug, refresh_by_slug + + @staticmethod + def _job_is_stale(job) -> bool: + now = timezone.now() + if job.status == JobStatus.PENDING: + max_age = int( + getattr( + settings, + "PARSER_STALE_PENDING_JOB_MAX_AGE_MINUTES", + STALE_PENDING_MAX_AGE_MINUTES, + ) + ) + return job.created_at < now - timedelta(minutes=max_age) + max_age = int( + getattr( + settings, + "PARSER_STALE_LOAD_MAX_AGE_MINUTES", + STALE_ACTIVE_MAX_AGE_MINUTES, + ) + ) + return job.updated_at < now - timedelta(minutes=max_age) @classmethod def _get_latest_context_load( @@ -1000,117 +1111,178 @@ class SourceCardService: requested_by_id: int | None, params: dict[str, Any], ) -> list[dict[str, str]]: + kwargs = {"requested_by_id": requested_by_id} if definition.slug == "financial-indicators": from apps.parsers.tasks import sync_fns_financial_reports - task_info = cls._enqueue_task( - task=sync_fns_financial_reports, - task_name="apps.parsers.tasks.sync_fns_financial_reports", - requested_by_id=requested_by_id, - meta={ - "source_card": definition.slug, - "source": ParserLoadLog.Source.FNS_REPORTS, - }, - kwargs={"requested_by_id": requested_by_id}, + task_specs = ( + ( + sync_fns_financial_reports, + "apps.parsers.tasks.sync_fns_financial_reports", + ParserLoadLog.Source.FNS_REPORTS, + ), ) - return [task_info] - - if definition.slug == "manufacturers-and-products": + elif definition.slug == "manufacturers-and-products": from apps.parsers.tasks import ( parse_industrial_production, parse_industrial_products, parse_manufactures, ) - return [ - cls._enqueue_task( - task=parse_industrial_production, - task_name="apps.parsers.tasks.parse_industrial_production", - requested_by_id=requested_by_id, - meta={ - "source_card": definition.slug, - "source": ParserLoadLog.Source.INDUSTRIAL, - }, - kwargs={"requested_by_id": requested_by_id}, + task_specs = ( + ( + parse_industrial_production, + "apps.parsers.tasks.parse_industrial_production", + ParserLoadLog.Source.INDUSTRIAL, ), - cls._enqueue_task( - task=parse_industrial_products, - task_name="apps.parsers.tasks.parse_industrial_products", - requested_by_id=requested_by_id, - meta={ - "source_card": definition.slug, - "source": ParserLoadLog.Source.INDUSTRIAL_PRODUCTS, - }, - kwargs={"requested_by_id": requested_by_id}, + ( + parse_industrial_products, + "apps.parsers.tasks.parse_industrial_products", + ParserLoadLog.Source.INDUSTRIAL_PRODUCTS, ), - cls._enqueue_task( - task=parse_manufactures, - task_name="apps.parsers.tasks.parse_manufactures", - requested_by_id=requested_by_id, - meta={ - "source_card": definition.slug, - "source": ParserLoadLog.Source.MANUFACTURES, - }, - kwargs={"requested_by_id": requested_by_id}, + ( + parse_manufactures, + "apps.parsers.tasks.parse_manufactures", + ParserLoadLog.Source.MANUFACTURES, ), - ] - - if definition.slug == "planned-inspections": + ) + elif definition.slug == "planned-inspections": from apps.parsers.tasks import sync_inspections - task_info = cls._enqueue_task( - task=sync_inspections, - task_name="apps.parsers.tasks.sync_inspections", - requested_by_id=requested_by_id, - meta={ - "source_card": definition.slug, - "source": ParserLoadLog.Source.INSPECTIONS, - }, - kwargs={ - "requested_by_id": requested_by_id, - **{ - key: value - for key, value in params.items() - if key in {"current_year", "current_month", "use_playwright"} - }, - }, + task_specs = ( + ( + sync_inspections, + "apps.parsers.tasks.sync_inspections", + ParserLoadLog.Source.INSPECTIONS, + ), ) - return [task_info] - - if definition.slug == "public-procurements": + kwargs.update( + { + key: value + for key, value in params.items() + if key in {"current_year", "current_month", "use_playwright"} + } + ) + elif definition.slug == "public-procurements": from apps.parsers.tasks import parse_registry_enrichment_sources - task_info = cls._enqueue_task( - task=parse_registry_enrichment_sources, - task_name="apps.parsers.tasks.parse_registry_enrichment_sources", - requested_by_id=requested_by_id, - meta={ - "source_card": definition.slug, - "source": ParserLoadLog.Source.PROCUREMENTS_44FZ, - }, - kwargs={ - "requested_by_id": requested_by_id, - }, + task_specs = ( + ( + parse_registry_enrichment_sources, + "apps.parsers.tasks.parse_registry_enrichment_sources", + ParserLoadLog.Source.PROCUREMENTS_44FZ, + ), + ) + elif definition.slug in { + "budget-process-registry", + "sme-support-recipients-registry", + }: + from apps.parsers.tasks_registry_snapshots import ( + parse_budget_registry, + parse_sme_support_recipients, ) - return [task_info] - task_specs = cls._get_simple_refresh_task_specs(definition.slug) - if task_specs: - return [ - cls._enqueue_task( - task=task, - task_name=task_name, - requested_by_id=requested_by_id, - meta={ - "source_card": definition.slug, - "source": parser_source, - }, - kwargs={"requested_by_id": requested_by_id}, + if definition.slug == "budget-process-registry": + task_specs = ( + ( + parse_budget_registry, + "parsers.budget_ubpandnubp.refresh", + "budget_ubpandnubp", + ), ) - for task, task_name, parser_source in task_specs - ] + else: + task_specs = ( + ( + parse_sme_support_recipients, + "parsers.fns_sme_support_recipients.refresh", + "fns_sme_support_recipients", + ), + ) + else: + task_specs = cls._get_simple_refresh_task_specs(definition.slug) + if not task_specs: + raise ValidationError( + {"detail": "Обновление для карточки не поддерживается."} + ) + return cls._enqueue_refresh_group( + definition, + task_specs, + requested_by_id=requested_by_id, + kwargs=kwargs, + ) - raise ValidationError({"detail": "Обновление для карточки не поддерживается."}) + @classmethod + def _enqueue_refresh_group(cls, definition, task_specs, *, requested_by_id, kwargs): + """Зафиксировать весь состав запуска до отправки первой задачи брокеру.""" + if definition.slug == "labor-vacancies": + existing = ( + BackgroundJobService.get_queryset() + .filter(task_name=task_specs[0][1]) + .filter(cls._fresh_active_job_filter()) + .first() + ) + if existing: + return [{"task_id": existing.task_id, "task_name": existing.task_name}] + run_id = str(uuid.uuid4()) + task_ids = [str(uuid.uuid4()) for _ in task_specs] + jobs = [] + with transaction.atomic(): + if definition.slug in { + "budget-process-registry", + "sme-support-recipients-registry", + }: + source = task_specs[0][2] + sequence, _ = ParserBatchSequence.objects.get_or_create(source=source) + ParserBatchSequence.objects.select_for_update().get(pk=sequence.pk) + if ( + BackgroundJobService.get_queryset() + .filter( + task_name=task_specs[0][1], + status__in=ACTIVE_JOB_STATUSES, + ) + .exists() + ): + raise ConflictError( + message="Источник уже обновляется", + code="source_refresh_running", + ) + for task_id, (_, task_name, source) in zip( + task_ids, task_specs, strict=False + ): + meta = { + "source_card": definition.slug, + "source": str(source), + "refresh_run_id": run_id, + "refresh_task_ids": task_ids, + } + jobs.append( + BackgroundJobService.create_job( + task_id=task_id, + task_name=task_name, + user_id=requested_by_id, + meta=meta, + ) + ) + tasks = [] + for index, (job, (task, task_name, _source)) in enumerate( + zip(jobs, task_specs, strict=False) + ): + try: + tasks.append( + cls._enqueue_task( + task=task, + task_name=task_name, + requested_by_id=requested_by_id, + meta=job.meta, + kwargs=kwargs, + task_id=job.task_id, + ) + ) + except Exception: + for pending in jobs[index:]: + pending.fail("Не удалось поставить обновление в очередь") + raise + return tasks @staticmethod def _get_simple_refresh_task_specs( @@ -1186,36 +1358,36 @@ class SourceCardService: requested_by_id: int | None, meta: dict[str, Any], kwargs: dict[str, Any], + task_id: str | None = None, ) -> dict[str, str]: - if task_name == "apps.parsers.tasks.parse_trudvsem_vacancies": - existing_job = ( + if ( + task_id is None + and task_name == "apps.parsers.tasks.parse_trudvsem_vacancies" + ): + existing = ( BackgroundJobService.get_queryset() .filter(task_name=task_name) .filter(cls._fresh_active_job_filter()) - .order_by("-created_at") .first() ) - if existing_job is not None: - return { - "task_id": existing_job.task_id, - "task_name": existing_job.task_name, - } - - task_id = str(uuid.uuid4()) - BackgroundJobService.create_job( - task_id=task_id, - task_name=task_name, - user_id=requested_by_id, - meta=meta, - ) + if existing is not None: + return {"task_id": existing.task_id, "task_name": existing.task_name} + if task_id is None: + task_id = str(uuid.uuid4()) + BackgroundJobService.create_job( + task_id=task_id, + task_name=task_name, + user_id=requested_by_id, + meta={**meta, "refresh_run_id": task_id, "refresh_task_ids": [task_id]}, + ) try: - async_result = task.apply_async(kwargs=kwargs, task_id=task_id) + task.apply_async(kwargs=kwargs, task_id=task_id) except Exception: with suppress(Exception): - BackgroundJobService.get_queryset().filter(task_id=task_id).delete() + job = BackgroundJobService.get_by_task_id(task_id) + job.fail("Не удалось поставить обновление в очередь") raise - - return {"task_id": async_result.id, "task_name": task_name} + return {"task_id": task_id, "task_name": task_name} @classmethod def _build_source_item( diff --git a/src/apps/parsers/source_registry.py b/src/apps/parsers/source_registry.py index 40427ad..9d23b4a 100644 --- a/src/apps/parsers/source_registry.py +++ b/src/apps/parsers/source_registry.py @@ -56,6 +56,30 @@ class ParserSourceDescriptor: PARSER_SOURCES: dict[str, ParserSourceDescriptor] = { + "fns_sme_support_recipients": ParserSourceDescriptor( + key="fns_sme_support_recipients", + source=ParserLoadLog.Source.FNS_SME_SUPPORT_RECIPIENTS, + title="Реестр субъектов МСП — получателей поддержки", + agency="ФНС России", + data_scope="Меры поддержки организаций канонического реестра", + task_name="parsers.fns_sme_support_recipients.refresh", + upstream_url="https://www.nalog.gov.ru/opendata/7707329152-rsmppp/", + access_method="open_data", + parser_strategy="streaming_xml_snapshot", + admin_only=True, + ), + "budget_ubpandnubp": ParserSourceDescriptor( + key="budget_ubpandnubp", + source=ParserLoadLog.Source.BUDGET_UBPANDNUBP, + title="Реестр участников бюджетного процесса", + agency="Федеральное казначейство", + data_scope="Бюджетные сведения организаций канонического реестра", + task_name="parsers.budget_ubpandnubp.refresh", + upstream_url="https://budget.gov.ru/epbs/registry/ubpandnubp/data", + access_method="official_api", + parser_strategy="paginated_own_organizations_snapshot", + admin_only=True, + ), "industrial": ParserSourceDescriptor( key="industrial", source=ParserLoadLog.Source.INDUSTRIAL, diff --git a/src/apps/parsers/tasks.py b/src/apps/parsers/tasks.py index a9be7ce..3a79936 100644 --- a/src/apps/parsers/tasks.py +++ b/src/apps/parsers/tasks.py @@ -72,6 +72,10 @@ from apps.parsers.services import ( ) from apps.parsers.source_artifacts import cleanup_parser_source_artifacts from apps.parsers.source_registry import PARSER_SOURCES +from apps.parsers.tasks_registry_snapshots import ( # noqa: F401 + parse_budget_registry, + parse_sme_support_recipients, +) from celery import shared_task from django.conf import settings from django.db.models import Q @@ -3907,7 +3911,7 @@ def _find_resumable_vacancy_job( if ( meta.get("targets_signature") != targets_signature or int(meta.get("total_organizations") or 0) != targets_count - or next_offset <= 0 + or next_offset < 0 or next_offset >= targets_count or batch_id is None ): @@ -3916,8 +3920,13 @@ def _find_resumable_vacancy_job( source=ParserLoadLog.Source.TRUDVSEM, batch_id=batch_id, ).first() - if load is not None: - return candidate, load + if load is None: + continue + # Отмена между записью вакансий и checkpoint оставляет прежний cursor. + # Повторяем эту организацию в том же batch: сохранение идемпотентно. + if next_offset == 0 and _vacancy_batch_records_count(batch_id) == 0: + continue + return candidate, load return None, None @@ -3931,10 +3940,10 @@ def _update_vacancy_registry_checkpoint( failed: int, targets_signature: str, ) -> int: - """Зафиксировать прогресс после полностью обработанной организации.""" + """Зафиксировать прогресс; отменённая задача сохраняет прежний cursor для replay.""" saved_count = _vacancy_batch_records_count(batch_id) progress = 100 if total == 0 else min(99, round(processed * 100 / total)) - job.meta = { + checkpoint_meta = { **(job.meta or {}), "batch_id": batch_id, "next_offset": processed, @@ -3943,12 +3952,11 @@ def _update_vacancy_registry_checkpoint( "saved_records": saved_count, "targets_signature": targets_signature, } - job.progress = progress - job.progress_message = ( + message = ( f"Обработано организаций: {processed} из {total}; " f"сохранено вакансий: {saved_count}; ошибок: {failed}" ) - job.save(update_fields=["meta", "progress", "progress_message", "updated_at"]) + job.update_progress(progress, message, meta=checkpoint_meta) ParserLoadLogService.update( load_log, status=ParserLoadLog.Status.IN_PROGRESS, diff --git a/src/apps/parsers/tasks_registry_snapshots.py b/src/apps/parsers/tasks_registry_snapshots.py new file mode 100644 index 0000000..6c3e7d5 --- /dev/null +++ b/src/apps/parsers/tasks_registry_snapshots.py @@ -0,0 +1,138 @@ +"""Celery entrypoints for the two public organization registry snapshots.""" + +from __future__ import annotations + +import uuid +from collections.abc import Callable + +from apps.core.models import BackgroundJob, JobStatus +from apps.parsers.budget_registry import BUDGET_SOURCE, refresh_budget_registry +from apps.parsers.models import ParserLoadLog +from apps.parsers.registry_snapshots import SnapshotValidationError +from apps.parsers.services import ParserLoadLogService +from apps.parsers.sme_support import SME_SOURCE, refresh_sme_support +from celery import shared_task +from django.utils import timezone + + +def _complete_snapshot_job(job, load_log, artifact, result) -> dict: + """Run inside the publication transaction, alongside the new records.""" + job = BackgroundJob.objects.select_for_update().get(pk=job.pk) + if job.status != JobStatus.STARTED: + raise SnapshotValidationError("snapshot_job_no_longer_active") + ParserLoadLogService.update( + load_log, + status=ParserLoadLog.Status.SUCCESS, + records_count=result.published, + ) + payload = { + "status": "success", + "batch_id": load_log.batch_id, + "load_id": load_log.id, + "artifact_id": str(artifact.uid), + "raw_records_count": result.parsed, + "published_records_count": result.published, + "quarantine_records_count": result.quarantined, + "skipped_records_count": result.skipped, + } + job.update_progress(100, "Снимок реестра опубликован") + job.complete(result=payload) + return payload + + +def _run_snapshot( + task, *, source: str, refresh: Callable, requested_by_id: int | None +) -> dict: + job = None + load_log = None + try: + job, _ = BackgroundJob.objects.get_or_create( + task_id=task.request.id or str(uuid.uuid4()), + defaults={ + "task_name": task.name, + "user_id": requested_by_id, + "meta": {"source": source, "source_key": source}, + }, + ) + claimed = BackgroundJob.objects.filter( + pk=job.pk, + status__in=(JobStatus.PENDING, JobStatus.RETRY), + ).update( + status=JobStatus.STARTED, + started_at=timezone.now(), + updated_at=timezone.now(), + ) + if not claimed: + job.refresh_from_db() + return job.result or {"status": job.status, "duplicate_delivery": True} + job.refresh_from_db() + load_log, batch_id = ParserLoadLogService.create_load_log_with_next_batch_id( + source=source, + status=ParserLoadLog.Status.IN_PROGRESS, + ) + metadata = { + **job.meta, + "source": source, + "source_key": source, + "batch_id": batch_id, + } + attached = BackgroundJob.objects.filter( + pk=job.pk, status=JobStatus.STARTED + ).update(meta=metadata, updated_at=timezone.now()) + if not attached: + raise SnapshotValidationError("snapshot_job_no_longer_active") + job.refresh_from_db() + job.update_progress(5, "Загрузка и проверка снимка реестра") + payload = {} + + def finalize(artifact, result): + payload.update(_complete_snapshot_job(job, load_log, artifact, result)) + + refresh( + load_batch=batch_id, uploaded_by_id=requested_by_id, on_publish=finalize + ) + return payload + except Exception as exc: + if job is not None: + job.refresh_from_db() + if job.status == JobStatus.SUCCESS: + return job.result + # Store a bounded error class, not HTTP request internals or raw source data. + message = f"Не удалось обновить реестр ({type(exc).__name__})" + if isinstance(exc, SnapshotValidationError): + message += f": {str(exc)[:200]}" + if load_log is not None: + ParserLoadLogService.mark_failed(load_log, message) + if job is not None: + job.fail(error=message) + raise + + +@shared_task( + bind=True, + name="parsers.fns_sme_support_recipients.refresh", + soft_time_limit=4 * 3600, + time_limit=4 * 3600 + 300, +) +def parse_sme_support_recipients(self, *, requested_by_id: int | None = None) -> dict: + return _run_snapshot( + self, + source=SME_SOURCE, + refresh=refresh_sme_support, + requested_by_id=requested_by_id, + ) + + +@shared_task( + bind=True, + name="parsers.budget_ubpandnubp.refresh", + soft_time_limit=4 * 3600, + time_limit=4 * 3600 + 300, +) +def parse_budget_registry(self, *, requested_by_id: int | None = None) -> dict: + return _run_snapshot( + self, + source=BUDGET_SOURCE, + refresh=refresh_budget_registry, + requested_by_id=requested_by_id, + ) diff --git a/src/apps/parsers/views.py b/src/apps/parsers/views.py index f1ac5fb..bcc4e84 100644 --- a/src/apps/parsers/views.py +++ b/src/apps/parsers/views.py @@ -9,7 +9,7 @@ import csv import json import uuid from collections import defaultdict -from datetime import timedelta +from datetime import date, datetime, time, timedelta from apps.core.filters import BaseFilterSet from apps.core.models import JobStatus @@ -56,6 +56,7 @@ from apps.parsers.serializers import ( ParserLoadLogSerializer, ParserResultQuerySerializer, ParserResultRecordSerializer, + ParserRunEnvelopeSerializer, ParserRunRequestSerializer, ParserScheduleRequestSerializer, ParserScheduleSerializer, @@ -77,7 +78,7 @@ from apps.parsers.source_cards import SourceCardService from apps.parsers.source_registry import PARSER_SOURCES from django.core.files.storage import default_storage from django.core.paginator import Paginator -from django.db.models import CharField, Count, Q +from django.db.models import Case, CharField, Count, F, Q, Value, When from django.db.models.functions import Cast, Lower from django.http import HttpResponse from django.utils import timezone @@ -430,8 +431,28 @@ PARSER_LOG_ORDERING_FIELDS = { "records_count", "created_at", "updated_at", + "source_label", + "status_label", + "organizations_count", } +PARSER_LOG_DATE_PARAMETERS = [ + openapi.Parameter( + "date_from", + openapi.IN_QUERY, + type=openapi.TYPE_STRING, + format="date", + description="Начало периода updated_at; без date_to — один календарный день.", + ), + openapi.Parameter( + "date_to", + openapi.IN_QUERY, + type=openapi.TYPE_STRING, + format="date", + description="Конец периода включительно, в часовом поясе приложения.", + ), +] + PARSER_LOG_STATUS_LABELS = { "success": "Успешно", "failed": "Ошибка", @@ -485,25 +506,23 @@ def _matches_parser_log_search(row: dict, search_term: str) -> bool: return False +def _parser_log_ordering(ordering: str) -> list[str]: + fields = [item.strip() for item in ordering.split(",") if item.strip()] + if any( + field.removeprefix("-") not in PARSER_LOG_ORDERING_FIELDS for field in fields + ): + raise ValidationError({"ordering": "Неподдерживаемое поле сортировки"}) + fields = fields or ["-updated_at"] + if not any(field.lstrip("-") == "id" for field in fields): + fields.append("id") + return fields + + def _sort_parser_log_rows(rows: list[dict], ordering: str) -> list[dict]: - allowed_fields = { - "id", - "batch_id", - "source", - "source_label", - "status", - "status_label", - "records_count", - "organizations_count", - "created_at", - "updated_at", - } sorted_rows = list(rows) - order_by_fields = [item.strip() for item in ordering.split(",") if item.strip()] + order_by_fields = _parser_log_ordering(ordering) for raw_field in reversed(order_by_fields): field_name = raw_field[1:] if raw_field.startswith("-") else raw_field - if field_name not in allowed_fields: - continue reverse = raw_field.startswith("-") sorted_rows.sort( key=lambda row: (row.get(field_name) is None, row.get(field_name)), @@ -518,7 +537,7 @@ def _build_page_url(request, page_number: int) -> str: return request.build_absolute_uri(f"{request.path}?{query_params.urlencode()}") -def _paginate_results(request, rows: list[dict]): +def _paginate_results(request, rows): page_size_raw = request.query_params.get("page_size", "20") page_raw = request.query_params.get("page", "1") try: @@ -679,11 +698,111 @@ def _apply_parser_log_filters(request): batch_id = request.query_params.get("batch_id") if batch_id: - queryset = queryset.filter(batch_id=batch_id) + try: + queryset = queryset.filter(batch_id=int(batch_id)) + except (TypeError, ValueError) as exc: + raise ValidationError({"batch_id": "Ожидается целое число"}) from exc + + queryset = _filter_parser_log_period(queryset, request.query_params) + ordering = _parser_log_ordering(request.query_params.get("ordering", "")) + return _order_parser_log_queryset(queryset, ordering) + + +def _filter_parser_log_period(queryset, params): + period = {} + for field in ("date_from", "date_to"): + raw = params.get(field) + if not raw: + continue + try: + parsed = date.fromisoformat(raw) + if parsed.isoformat() != raw: + raise ValueError + except (TypeError, ValueError) as exc: + raise ValidationError( + {field: "Ожидается дата в формате YYYY-MM-DD"} + ) from exc + period[field] = parsed + date_from = period.get("date_from") + date_to = period.get("date_to", date_from) + if date_from and date_to and date_from > date_to: + raise ValidationError({"date_from": "Начало периода не может быть позже конца"}) + app_timezone = timezone.get_default_timezone() + if date_from: + queryset = queryset.filter( + updated_at__gte=timezone.make_aware( + datetime.combine(date_from, time.min), + app_timezone, + ) + ) + if date_to: + queryset = queryset.filter( + updated_at__lte=timezone.make_aware( + datetime.combine(date_to, time.max), + app_timezone, + ) + ) return queryset +def _order_parser_log_queryset(queryset, ordering: list[str]): + if any(field.lstrip("-") == "organizations_count" for field in ordering): + return queryset.order_by("id") # Computed counts are ordered after enrichment. + + annotations = {} + for field in ("source", "source_label", "status_label"): + if not any(item.lstrip("-") == field for item in ordering): + continue + if field == "status_label": + choices = PARSER_LOG_STATUS_LABELS.items() + model_field = "status" + else: + model_field = "source" + choices = [ + ( + source, + ( + SourceCardService.get_card_slug_by_parser_source(source) + or source + if field == "source" + else SourceCardService.get_card_title_by_parser_source(source) + or str(label) + ), + ) + for source, label in ParserLoadLog.Source.choices + ] + annotations[f"log_order_{field}"] = Case( + *[ + When(**{model_field: value}, then=Value(label)) + for value, label in choices + ], + default=F(model_field), + output_field=CharField(), + ) + ordering = [ + item.replace(field, f"log_order_{field}") + if item.lstrip("-") == field + else item + for item in ordering + ] + return queryset.annotate(**annotations).order_by(*ordering) + + +def _filtered_parser_log_rows(request, queryset) -> list[dict]: + rows = [_serialize_parser_log_row(log) for log in queryset] + search_term = request.query_params.get("search", "").strip() + if search_term: + rows = [row for row in rows if _matches_parser_log_search(row, search_term)] + ordering = request.query_params.get("ordering", "") + if any( + field.lstrip("-") == "organizations_count" + for field in _parser_log_ordering(ordering) + ): + return _sort_parser_log_rows(rows, ordering) + return rows + + # ============================================================================= # Минпромторг - Сертификаты промышленного производства # ============================================================================= @@ -1399,6 +1518,7 @@ class SourceCardRefreshView(APIView): responses={ 202: SourceCardRefreshFrontendResponseSerializer, 400: CommonResponses.BAD_REQUEST, + 409: openapi.Response("Источник уже обновляется"), **ErrorResponses.ADMIN_NOT_FOUND, }, ) @@ -1427,7 +1547,12 @@ class SourceCardRefreshView(APIView): response_payload = { "task_id": tasks[0]["task_id"] if tasks else None, "task_ids": [task["task_id"] for task in tasks], - "status": "accepted", + "status": ( + "queued" + if slug + in {"budget-process-registry", "sme-support-recipients-registry"} + else "accepted" + ), } return Response( response_payload, @@ -1468,6 +1593,7 @@ class ParserLoadLogViewSet(ReadOnlyModelViewSet): "created_at, updated_at." ), manual_parameters=[ + *PARSER_LOG_DATE_PARAMETERS, openapi.Parameter( name="search", in_=openapi.IN_QUERY, @@ -1492,13 +1618,22 @@ class ParserLoadLogViewSet(ReadOnlyModelViewSet): }, ) def list(self, request, *args, **kwargs): - rows = [_serialize_parser_log_row(item) for item in self.get_queryset()] - search_term = request.query_params.get("search", "").strip() - if search_term: - rows = [row for row in rows if _matches_parser_log_search(row, search_term)] - rows = _sort_parser_log_rows(rows, request.query_params.get("ordering", "")) - serializer = ParserLoadLogListSerializer(rows, many=True) - return Response(_paginate_results(request, serializer.data)) + queryset = self.get_queryset() + ordering = _parser_log_ordering(request.query_params.get("ordering", "")) + needs_enrichment = request.query_params.get("search", "").strip() or any( + field.lstrip("-") == "organizations_count" for field in ordering + ) + if needs_enrichment: + page = _paginate_results( + request, _filtered_parser_log_rows(request, queryset) + ) + else: + page = _paginate_results(request, queryset) + page["results"] = [ + _serialize_parser_log_row(log) for log in page["results"] + ] + page["results"] = ParserLoadLogListSerializer(page["results"], many=True).data + return Response(page) @swagger_auto_schema( tags=[SYSTEM_TAG], @@ -1526,6 +1661,7 @@ class ParserLoadLogExportView(APIView): "Поддерживает те же фильтры, search и ordering, что и список логов." ), manual_parameters=[ + *PARSER_LOG_DATE_PARAMETERS, openapi.Parameter( name="source", in_=openapi.IN_QUERY, @@ -1568,37 +1704,29 @@ class ParserLoadLogExportView(APIView): }, ) def get(self, request): - rows = [ - _serialize_parser_log_row(item) - for item in _apply_parser_log_filters(request) - ] - search_term = request.query_params.get("search", "").strip() - if search_term: - rows = [row for row in rows if _matches_parser_log_search(row, search_term)] - rows = _sort_parser_log_rows(rows, request.query_params.get("ordering", "")) + rows = _filtered_parser_log_rows(request, _apply_parser_log_filters(request)) response = HttpResponse(content_type="text/csv; charset=utf-8") - response["Content-Disposition"] = 'attachment; filename="parser-load-logs.csv"' + response["Content-Disposition"] = 'attachment; filename="update-history.csv"' + response.write("\ufeff") - writer = csv.writer(response) + writer = csv.writer(response, delimiter=";", lineterminator="\r\n") writer.writerow( [ - "id", - "batch_id", - "source", - "source_label", - "records_count", - "organizations_count", - "artifact_uid", - "parsed_count", - "published_count", - "quarantined_count", - "rejection_reasons", - "status", - "status_label", - "error_message", - "created_at", - "updated_at", + "№", + "Дата актуализации", + "Источник", + "Статус", + "Количество записей", + "Пакет загрузки", + "Количество организаций", + "Идентификатор файла источника", + "Разобрано записей", + "Опубликовано записей", + "Записей в карантине", + "Причины отклонения", + "Ошибка", + "Дата создания", ] ) @@ -1606,21 +1734,23 @@ class ParserLoadLogExportView(APIView): writer.writerow( [ row["id"], - row["batch_id"], - row["source"], + timezone.localtime( + row["updated_at"], timezone.get_default_timezone() + ).strftime("%d.%m.%Y %H:%M"), row["source_label"], + row["status_label"], row["records_count"], + row["batch_id"], row["organizations_count"], row["artifact_uid"] or "", row["parsed_count"], row["published_count"], row["quarantined_count"], json.dumps(row["rejection_reasons"], ensure_ascii=False), - row["status"], - row["status_label"], row["error_message"], - row["created_at"], - row["updated_at"], + timezone.localtime( + row["created_at"], timezone.get_default_timezone() + ).strftime("%d.%m.%Y %H:%M"), ] ) @@ -2728,6 +2858,21 @@ class ParserRunView(APIView): permission_classes = [IsAuthenticated] + @swagger_auto_schema( + tags=[PARSERS_TAG], + operation_description=( + "Запускает parser source. Для budget_ubpandnubp и " + "fns_sme_support_recipients используются настройки сервера; " + "допустимо пустое тело или params: {}. Повторный активный запуск — 409." + ), + request_body=ParserRunRequestSerializer, + responses={ + 202: openapi.Response( + "Задача поставлена в очередь", ParserRunEnvelopeSerializer + ), + 409: openapi.Response("Источник уже обновляется"), + }, + ) def post(self, request: Request, source_key: str): canonical_source_key = PARSER_SOURCE_ALIASES.get(source_key, source_key) descriptor = PARSER_SOURCES.get(canonical_source_key) @@ -2735,6 +2880,29 @@ class ParserRunView(APIView): return _source_not_found_response(source_key) if descriptor.admin_only and not request.user.is_staff: return Response(status=status.HTTP_403_FORBIDDEN) + snapshot_card_slugs = { + "budget_ubpandnubp": "budget-process-registry", + "fns_sme_support_recipients": "sme-support-recipients-registry", + } + if canonical_source_key in snapshot_card_slugs: + serializer = SourceCardRefreshRequestSerializer(data=request.data) + serializer.is_valid(raise_exception=True) + payload = SourceCardService.refresh_card( + slug=snapshot_card_slugs[canonical_source_key], + requested_by_id=request.user.id, + params=serializer.validated_data.get("params", {}), + ) + task_ids = [task["task_id"] for task in payload["tasks"]] + return api_response( + { + "task_id": task_ids[0], + "task_ids": task_ids, + "status": "queued", + "source": canonical_source_key, + "task_name": descriptor.task_name, + }, + status_code=status.HTTP_202_ACCEPTED, + ) if not descriptor.supports_refresh: if canonical_source_key == ParserLoadLog.Source.ROPK_SANCTIONS: return api_error_response( @@ -3157,6 +3325,10 @@ class ParserDashboardDataView(APIView): jobs = BackgroundJobService.get_user_jobs(user_id=request.user.id, limit=30) source_counts = dict( OrganizationSourceRecord.objects.order_by() + .filter( + ~Q(source__in=("fns_sme_support_recipients", "budget_ubpandnubp")) + | Q(extension__organization__opk_registry_membership=True) + ) .values("source") .annotate(count=Count("uid")) .values_list("source", "count") diff --git a/src/organizations/cache.py b/src/organizations/cache.py index 9532f1a..ddbb1d4 100644 --- a/src/organizations/cache.py +++ b/src/organizations/cache.py @@ -7,7 +7,7 @@ import time from django.core.cache import cache ORGANIZATION_API_CACHE_PREFIX = "api:v2:organizations" -ORGANIZATION_API_CACHE_CONTRACT_VERSION = 2 +ORGANIZATION_API_CACHE_CONTRACT_VERSION = 3 ORGANIZATION_API_CACHE_VERSION_KEY = f"{ORGANIZATION_API_CACHE_PREFIX}:version" DEFAULT_ORGANIZATION_API_CACHE_VERSION = 1 DEFAULT_ORGANIZATION_API_CACHE_TIMEOUT_SECONDS = 24 * 60 * 60 diff --git a/src/organizations/migrations/0012_support_and_budget_registry_sources.py b/src/organizations/migrations/0012_support_and_budget_registry_sources.py new file mode 100644 index 0000000..1619582 --- /dev/null +++ b/src/organizations/migrations/0012_support_and_budget_registry_sources.py @@ -0,0 +1,85 @@ +# Generated by Django 3.2.25 on 2026-09-13 20:12 + +import django.db.models.deletion +from django.db import migrations, models + + +class Migration(migrations.Migration): + dependencies = [ + ("organizations", "0011_auto_20260820_1325"), + ] + + operations = [ + migrations.CreateModel( + name="BudgetProcessRegistryExtension", + fields=[ + ( + "organizationsourceextension_ptr", + models.OneToOneField( + auto_created=True, + on_delete=django.db.models.deletion.CASCADE, + parent_link=True, + primary_key=True, + serialize=False, + to="organizations.organizationsourceextension", + ), + ), + ], + options={ + "verbose_name": "участник бюджетного процесса", + "verbose_name_plural": "участники бюджетного процесса", + "db_table": "organizations_budget_process_registry_extension", + }, + bases=("organizations.organizationsourceextension",), + ), + migrations.CreateModel( + name="GovernmentSupportExtension", + fields=[ + ( + "organizationsourceextension_ptr", + models.OneToOneField( + auto_created=True, + on_delete=django.db.models.deletion.CASCADE, + parent_link=True, + primary_key=True, + serialize=False, + to="organizations.organizationsourceextension", + ), + ), + ], + options={ + "verbose_name": "государственная поддержка", + "verbose_name_plural": "государственная поддержка", + "db_table": "organizations_government_support_extension", + }, + bases=("organizations.organizationsourceextension",), + ), + migrations.AlterField( + model_name="organizationsourceextension", + name="source_group", + field=models.CharField( + choices=[ + ("financial_indicators", "Финансово-экономические показатели"), + ("government_procurements", "Государственные закупки"), + ("industrial_production", "Производители и продукция России"), + ("planned_inspections", "Плановые проверки"), + ("bankruptcy", "Сведения о процедурах банкротства"), + ("defense_suppliers", "Недобросовестные поставщики ГОЗ"), + ("arbitration", "Арбитражные дела"), + ("security_registries", "Реестры по информационной безопасности"), + ("vacancies", "Вакансии"), + ("electronic_document_exchange", "Электронный документооборот"), + ("media_mentions", "Упоминания в СМИ"), + ("sanctions", "Санкции"), + ("government_support", "Государственная поддержка"), + ( + "budget_process_registry", + "Реестр участников бюджетного процесса", + ), + ], + db_index=True, + max_length=64, + verbose_name="группа источников", + ), + ), + ] diff --git a/src/organizations/models.py b/src/organizations/models.py index ce99102..aa17625 100644 --- a/src/organizations/models.py +++ b/src/organizations/models.py @@ -38,6 +38,11 @@ class SourceGroup(models.TextChoices): ) MEDIA_MENTIONS = "media_mentions", _("Упоминания в СМИ") SANCTIONS = "sanctions", _("Санкции") + GOVERNMENT_SUPPORT = "government_support", _("Государственная поддержка") + BUDGET_PROCESS_REGISTRY = ( + "budget_process_registry", + _("Реестр участников бюджетного процесса"), + ) class SourceExtensionStatus(models.TextChoices): @@ -703,6 +708,28 @@ class SanctionsExtension(OrganizationSourceExtension): verbose_name_plural = _("санкционные признаки") +class GovernmentSupportExtension(OrganizationSourceExtension): + """Support measures for an organization in the canonical directory.""" + + source_group_value = SourceGroup.GOVERNMENT_SUPPORT + + class Meta: + db_table = "organizations_government_support_extension" + verbose_name = _("государственная поддержка") + verbose_name_plural = _("государственная поддержка") + + +class BudgetProcessRegistryExtension(OrganizationSourceExtension): + """Budget registry records for an organization in the canonical directory.""" + + source_group_value = SourceGroup.BUDGET_PROCESS_REGISTRY + + class Meta: + db_table = "organizations_budget_process_registry_extension" + verbose_name = _("участник бюджетного процесса") + verbose_name_plural = _("участники бюджетного процесса") + + class OrganizationSourceRecord(models.Model): """Subordinate source record stored under a source extension.""" diff --git a/src/organizations/serializers.py b/src/organizations/serializers.py index 9404dea..2bc91bb 100644 --- a/src/organizations/serializers.py +++ b/src/organizations/serializers.py @@ -78,10 +78,48 @@ class OrganizationSourceRecordOrganizationSerializer(serializers.Serializer): ogrip = serializers.CharField(read_only=True, allow_blank=True) +class SourceSupportDictionarySerializer(serializers.Serializer): + code = serializers.CharField(read_only=True) + name = serializers.CharField(read_only=True) + + +class SourceSupportSizeSerializer(serializers.Serializer): + unit_code = serializers.CharField(read_only=True) + unit = serializers.ChoiceField( + choices=("RUB", "square_meter", "hour", "percent", "unit"), read_only=True + ) + value = serializers.CharField(read_only=True) + + +class SourceSupportProviderSerializer(serializers.Serializer): + name = serializers.CharField(read_only=True) + inn = serializers.CharField(read_only=True) + + class OrganizationSourceRecordPayloadSerializer(serializers.Serializer): """Typed optional fields exposed by source-record list payloads.""" artifact_id = serializers.UUIDField(read_only=True, allow_null=True) + support_registry_number = serializers.CharField(read_only=True) + recipient_type = serializers.CharField(read_only=True) + sme_category = serializers.CharField(read_only=True) + support_form = SourceSupportDictionarySerializer(read_only=True) + support_kind = SourceSupportDictionarySerializer(read_only=True) + support_sizes = SourceSupportSizeSerializer(many=True, read_only=True) + provider = SourceSupportProviderSerializer(read_only=True) + decision_date = serializers.DateField(read_only=True, allow_null=True) + support_until = serializers.DateField(read_only=True, allow_null=True) + termination_date = serializers.DateField(read_only=True, allow_null=True) + registry_entry_date = serializers.DateField(read_only=True, allow_null=True) + source_updated_date = serializers.DateField(read_only=True, allow_null=True) + source_snapshot_date = serializers.DateField(read_only=True) + has_violation = serializers.BooleanField(read_only=True) + violation_count = serializers.IntegerField(read_only=True) + regulatory_documents_count = serializers.IntegerField(read_only=True) + registry = serializers.DictField(read_only=True) + classification = serializers.DictField(read_only=True) + budget = serializers.DictField(read_only=True) + address = serializers.DictField(read_only=True) rn = serializers.CharField(read_only=True, allow_blank=True, allow_null=True) uk_hm_treasury = serializers.BooleanField(read_only=True, allow_null=True) european_union = serializers.BooleanField(read_only=True, allow_null=True) @@ -207,6 +245,13 @@ class OrganizationSourceRecordDetailPayloadSerializer( ): """Detail payload including the complete media article text.""" + source_document_id = serializers.CharField(read_only=True) + violations = serializers.ListField(child=serializers.DictField(), read_only=True) + regulatory_documents = serializers.ListField( + child=serializers.DictField(), read_only=True + ) + upstream = serializers.DictField(read_only=True) + full_text = serializers.CharField( read_only=True, allow_blank=True, @@ -315,6 +360,19 @@ class OrganizationSourceRecordListSerializer(OrganizationSourceRecordSerializer) ) def get_payload(self, obj) -> dict | list | str | int | float | bool | None: payload = super().get_payload(obj) + if obj.extension.source_group in ( + SourceGroup.GOVERNMENT_SUPPORT, + SourceGroup.BUDGET_PROCESS_REGISTRY, + ): + response_payload = dict(payload) if isinstance(payload, dict) else {} + for field in ( + "source_document_id", + "violations", + "regulatory_documents", + "upstream", + ): + response_payload.pop(field, None) + return response_payload if obj.extension.source_group != SourceGroup.MEDIA_MENTIONS: return payload response_payload = dict(payload) if isinstance(payload, dict) else {} diff --git a/src/organizations/source_groups.py b/src/organizations/source_groups.py index 052e259..ec1d56a 100644 --- a/src/organizations/source_groups.py +++ b/src/organizations/source_groups.py @@ -9,10 +9,12 @@ from apps.parsers.models import ParserLoadLog from organizations.models import ( ArbitrationExtension, BankruptcyExtension, + BudgetProcessRegistryExtension, DefenseSupplierExtension, ElectronicDocumentExchangeExtension, FinancialIndicatorsExtension, GovernmentProcurementExtension, + GovernmentSupportExtension, IndustrialProductionExtension, MediaMentionExtension, OrganizationSourceExtension, @@ -36,6 +38,20 @@ class SourceGroupDescriptor: SOURCE_GROUP_DESCRIPTORS: dict[str, SourceGroupDescriptor] = { + ParserLoadLog.Source.FNS_SME_SUPPORT_RECIPIENTS: SourceGroupDescriptor( + source=ParserLoadLog.Source.FNS_SME_SUPPORT_RECIPIENTS, + source_group=SourceGroup.GOVERNMENT_SUPPORT, + record_type="sme_support_measure", + title="Государственная поддержка", + extension_model=GovernmentSupportExtension, + ), + ParserLoadLog.Source.BUDGET_UBPANDNUBP: SourceGroupDescriptor( + source=ParserLoadLog.Source.BUDGET_UBPANDNUBP, + source_group=SourceGroup.BUDGET_PROCESS_REGISTRY, + record_type="budget_registry_organization", + title="Реестр участников бюджетного процесса", + extension_model=BudgetProcessRegistryExtension, + ), ParserLoadLog.Source.FNS_REPORTS: SourceGroupDescriptor( source=ParserLoadLog.Source.FNS_REPORTS, source_group=SourceGroup.FINANCIAL_INDICATORS, diff --git a/src/organizations/source_record_export.py b/src/organizations/source_record_export.py index 98e8843..33947af 100644 --- a/src/organizations/source_record_export.py +++ b/src/organizations/source_record_export.py @@ -39,6 +39,8 @@ EXPORT_FORMAT_JSON = "json" EXPORT_FORMATS = (EXPORT_FORMAT_CSV, EXPORT_FORMAT_XLSX, EXPORT_FORMAT_JSON) FINANCIAL_SOURCE_GROUP = SourceGroup.FINANCIAL_INDICATORS.value ALL_HISTORY_SOURCE_GROUPS = { + SourceGroup.GOVERNMENT_SUPPORT.value, + SourceGroup.BUDGET_PROCESS_REGISTRY.value, SourceGroup.MEDIA_MENTIONS.value, SourceGroup.SANCTIONS.value, } @@ -64,6 +66,8 @@ EXPORT_PROVIDER_NAME_PATTERN = re.compile( ) SOURCE_GROUP_EXPORT_FILE_STEMS: dict[str, str] = { + SourceGroup.GOVERNMENT_SUPPORT.value: "sme-support-recipients-registry", + SourceGroup.BUDGET_PROCESS_REGISTRY.value: "budget-process-registry", SourceGroup.FINANCIAL_INDICATORS.value: "financial-indicators", SourceGroup.GOVERNMENT_PROCUREMENTS.value: "public-procurements", SourceGroup.INDUSTRIAL_PRODUCTION.value: "manufacturers-and-products", @@ -117,6 +121,15 @@ SANCTIONS_EXPORT_FLAG_FIELDS = { "Санкции - Великобритания UKSL": "uk_uksl", "Санкции - Украина": "ukraine", } +SANCTIONS_XLSX_HEADERS = [ + { + "rn": "Регистрационный номер", + "ogrn": "ОГРН", + "inn": "ИНН", + "okpo": "ОКПО", + }.get(header, header.replace(" - ", " — ")) + for header in SANCTIONS_EXPORT_FIELDS +] FINANCIAL_LINE_FIELDS = [ "id", "form_code", @@ -268,6 +281,11 @@ def build_source_record_export_artifacts( headers=headers, file_format=file_format, records_count=records_count, + display_headers=( + SANCTIONS_XLSX_HEADERS + if source_group == SourceGroup.SANCTIONS + else None + ), ) parts_count = len(artifact_paths) artifacts.extend( @@ -685,6 +703,7 @@ def _render_source_group_artifact( headers: Sequence[str], file_format: str, records_count: int, + display_headers: Sequence[str] | None = None, ) -> tuple[Path, ...]: if file_format == EXPORT_FORMAT_CSV: _render_csv_file( @@ -699,6 +718,7 @@ def _render_source_group_artifact( output_path=output_path, headers=headers, records_count=records_count, + display_headers=display_headers, ) _render_json_file(row_spool_path=row_spool_path, output_path=output_path) return (output_path,) @@ -736,6 +756,7 @@ def _render_xlsx_files( output_path: Path, headers: Sequence[str], records_count: int, + display_headers: Sequence[str] | None = None, ) -> tuple[Path, ...]: rows_per_file = min( EXCEL_MAX_DATA_ROWS_PER_SHEET, @@ -761,7 +782,7 @@ def _render_xlsx_files( ) part_number = 1 rows_in_file = 0 - workbook, worksheet = _new_export_workbook(headers) + workbook, worksheet = _new_export_workbook(display_headers or headers) for row in _iter_spooled_rows(row_spool_path): if rows_in_file >= rows_per_file: @@ -769,7 +790,7 @@ def _render_xlsx_files( workbook.close() part_number += 1 rows_in_file = 0 - workbook, worksheet = _new_export_workbook(headers) + workbook, worksheet = _new_export_workbook(display_headers or headers) worksheet.append( [_serialize_spreadsheet_value(row.get(key)) for key in headers] ) diff --git a/src/organizations/test_companies.py b/src/organizations/test_companies.py index 484de32..6750997 100644 --- a/src/organizations/test_companies.py +++ b/src/organizations/test_companies.py @@ -39,6 +39,8 @@ TEST_RECORD_PREFIX = "mostovik-test-company" CANONICAL_ONLY_TEST_SOURCES = { ParserLoadLog.Source.GOSEDO_ADDRESS_DIRECTORY, ParserLoadLog.Source.MEDIA_NEWS, + ParserLoadLog.Source.FNS_SME_SUPPORT_RECIPIENTS, + ParserLoadLog.Source.BUDGET_UBPANDNUBP, } FILE_UPLOAD_ONLY_TEST_SOURCES = { ParserLoadLog.Source.ROPK_SANCTIONS, @@ -546,6 +548,74 @@ class TestCompanyDatasetService: } url = f"https://example.test/{source}/{index}" values = { + ParserLoadLog.Source.FNS_SME_SUPPORT_RECIPIENTS: { + "title": f"Тестовая мера поддержки — {organization.name}", + "record_date": "2024-10-01", + "amount": Decimal("1250000.00"), + "url": url, + "payload": { + **common, + "support_registry_number": f"ТЕСТ-ПОД-{index:05d}", + "source_document_id": f"test-support-document-{index}", + "recipient_type": "1", + "sme_category": "1", + "support_form": {"code": "0001", "name": "Тестовая поддержка"}, + "support_kind": {"code": "0001", "name": "Тестовая субсидия"}, + "support_sizes": [ + {"unit_code": "1", "unit": "RUB", "value": "1250000.00"}, + {"unit_code": "3", "unit": "hour", "value": "12.00"}, + ], + "provider": {"name": organization.name, "inn": organization.inn}, + "decision_date": "2024-10-01", + "support_until": "2026-12-31", + "termination_date": None, + "registry_entry_date": "2024-10-02", + "source_updated_date": "2026-08-15", + "source_snapshot_date": "2026-08-15", + "has_violation": False, + "violation_count": 0, + "regulatory_documents_count": 1, + "violations": [], + "regulatory_documents": [{"НомерРД": f"ТЕСТ-РД-{index:05d}"}], + }, + }, + ParserLoadLog.Source.BUDGET_UBPANDNUBP: { + "title": f"Тестовая бюджетная запись — {organization.name}", + "record_date": "2026-08-15", + "status": "inactive", + "url": url, + "payload": { + **common, + "registry": { + "code": f"test-budget-{index}", + "registration_number": f"ТЕСТ-БЮД-{index:05d}", + "record_number": str(index), + "guid": "", + "parent_record_number": "", + "status_code": "2", + "status_name": "Не действует", + }, + "classification": { + "organization_type": {"code": "", "name": ""}, + "establishment_kind": {"code": "", "name": ""}, + }, + "budget": { + "level": {"code": "", "name": ""}, + "code": "", + "name": "Тестовый бюджет", + }, + "address": {"region": {"code": "77", "name": "Москва"}}, + "upstream": { + "id": f"test-budget-{index}", + "info": { + "inn": organization.inn, + "ogrn": organization.ogrn, + "fullName": organization.name, + "statusCode": "2", + }, + }, + }, + }, ParserLoadLog.Source.FNS_REPORTS: { "title": ( f"Бухгалтерская отчетность за {report_year} год — " diff --git a/src/organizations/views.py b/src/organizations/views.py index c07ab34..0201b7e 100644 --- a/src/organizations/views.py +++ b/src/organizations/views.py @@ -3,7 +3,6 @@ from __future__ import annotations import hashlib -import json import os from contextlib import suppress from datetime import datetime @@ -15,7 +14,7 @@ from django.conf import settings from django.core.cache import cache from django.db.models import Case, CharField, F, Q, Value, When from django.db.models.fields.json import KeyTextTransform -from django.db.models.functions import Cast, Coalesce, NullIf +from django.db.models.functions import Coalesce, NullIf from django.http import StreamingHttpResponse from django_filters import rest_framework as filters from drf_yasg import openapi @@ -67,6 +66,16 @@ from organizations.source_record_export import ( ) ORGANIZATIONS_TAG = swagger_tag("Организации", "Organizations") +ORGANIZATION_SEARCH_FIELDS = ( + "name", + "full_name", + "short_name", + "inn", + "kpp", + "ogrn", + "ogrip", + "okpo", +) def _query_parameter( @@ -142,7 +151,7 @@ ORGANIZATION_LIST_PARAMS = [ ), _query_parameter( "search", - description="Поиск по наименованию, ИНН, КПП, ОГРН, ОГРИП и основному идентификатору.", + description="Поиск по наименованиям организации, ИНН, КПП, ОГРН, ОГРИП и ОКПО.", ), _query_parameter( "ordering", @@ -221,8 +230,8 @@ SOURCE_RECORD_LIST_PARAMS = [ _query_parameter( "search", description=( - "Поиск по организации, реквизитам, заголовку, внешнему ID, " - "статусу, датам, URL и исходным данным записи." + "Поиск по каноническим наименованиям организации, ИНН, КПП, ОГРН, " + "ОГРИП и ОКПО. Все слова запроса должны совпадать с данными организации." ), ), _query_parameter( @@ -238,18 +247,13 @@ SOURCE_RECORD_LIST_PARAMS = [ _query_parameter( "ordering", description=( - "Сортировка по полям: record_date, record_type, external_id, " - "extension__organization__name, " - "extension__organization__full_name, created_at, updated_at, title, " - "uid, extension__organization__inn, extension__organization__ogrn, " - "extension__organization__okpo, status, " - "payload__attestation_status, payload__full_name, " - "payload__medo_address, payload__registration_number, " - "payload__sentiment, " - "payload__news_source. Для обратной сортировки используйте префикс -. " + "Сортировка по полям: " + + ", ".join(SOURCE_RECORD_ORDERING_FIELDS) + + ". Для обратной сортировки используйте префикс -. " + "Несколько полей перечисляются через запятую без пробелов, " + "например -record_date,-external_id. " "Значения record_date с null сортируются последними." ), - enum=SOURCE_RECORD_ORDERING_VALUES, ), _query_parameter( "page", description="Номер страницы.", param_type=openapi.TYPE_INTEGER @@ -317,14 +321,7 @@ class OrganizationViewSet(CachedReadOnlyMixin, ReadOnlyModelViewSet): OrderingFilter, ] filterset_class = OrganizationFilter - search_fields = [ - "name", - "inn", - "kpp", - "ogrn", - "ogrip", - "primary_identity", - ] + search_fields = ORGANIZATION_SEARCH_FIELDS ordering_fields = [ "name", "inn", @@ -547,26 +544,7 @@ class OrganizationSourceRecordViewSet(ReadOnlyModelViewSet): lookup_field = "uid" filter_backends = [] search_fields = [ - "title", - "external_id", - "record_type", - "source", - "record_date", - "status", - "url", - "legacy_model", - "legacy_pk", - "source_record_amount_text", - "source_record_load_batch_text", - "source_record_payload_text", - "extension__title", - "extension__source_group", - "extension__organization__name", - "extension__organization__inn", - "extension__organization__kpp", - "extension__organization__ogrn", - "extension__organization__okpo", - "extension__organization__ogrip", + f"extension__organization__{field}" for field in ORGANIZATION_SEARCH_FIELDS ] ordering_fields = SOURCE_RECORD_ORDERING_FIELDS ordering = ["-created_at", "-uid"] @@ -659,20 +637,28 @@ class OrganizationSourceRecordViewSet(ReadOnlyModelViewSet): @staticmethod def _order_source_records(queryset, ordering: str): - if ordering.lstrip("-") == "organization__name": - ordering = ordering.replace( - "organization__name", - "extension__organization__name", - ) - return queryset.order_by(ordering, "external_id", "uid") - if ordering.lstrip("-") == "record_date": - expression = F("canonical_record_date") - if ordering.startswith("-"): - expression = expression.desc(nulls_last=True) + fields = ordering.split(",") + field_names = {field.lstrip("-") for field in fields} + ordering_terms = [] + for field in fields: + if field.lstrip("-") == "organization__name": + ordering_terms.append( + field.replace("organization__name", "extension__organization__name") + ) + elif field.lstrip("-") == "record_date": + expression = F("canonical_record_date") + ordering_terms.append( + expression.desc(nulls_last=True) + if field.startswith("-") + else expression.asc(nulls_last=True) + ) else: - expression = expression.asc(nulls_last=True) - return queryset.order_by(expression, "uid") - return queryset.order_by(ordering, "uid") + ordering_terms.append(field) + if "organization__name" in field_names and "external_id" not in field_names: + ordering_terms.append("external_id") + if "uid" not in field_names: + ordering_terms.append("uid") + return queryset.order_by(*ordering_terms) def _validate_source_record_query(self) -> list[dict[str, str]]: errors: list[dict[str, str]] = [] @@ -713,13 +699,16 @@ class OrganizationSourceRecordViewSet(ReadOnlyModelViewSet): or self.request.query_params.get("source") == "ropk_sanctions" ): ordering = "organization__name" - if ordering and ordering not in SOURCE_RECORD_ORDERING_VALUES: + if ordering and any( + field not in SOURCE_RECORD_ORDERING_VALUES for field in ordering.split(",") + ): errors.append( { "code": "invalid_ordering", "field": "ordering", "message": ( - "Поддерживаются: " + ", ".join(SOURCE_RECORD_ORDERING_VALUES) + "Перечислите через запятую без пробелов допустимые поля: " + + ", ".join(SOURCE_RECORD_ORDERING_VALUES) ), } ) @@ -731,15 +720,6 @@ class OrganizationSourceRecordViewSet(ReadOnlyModelViewSet): @classmethod def _filter_search_queryset(cls, queryset, search_terms: list[str]): - queryset = queryset.annotate( - source_record_amount_text=Cast("amount", output_field=CharField()), - source_record_load_batch_text=Cast( - "load_batch", - output_field=CharField(), - ), - source_record_payload_text=Cast("payload", output_field=CharField()), - ) - for search_term in search_terms: queryset = queryset.filter(cls._source_record_search_query(search_term)) @@ -750,29 +730,7 @@ class OrganizationSourceRecordViewSet(ReadOnlyModelViewSet): query = Q() for field_name in cls.search_fields: query |= Q(**{f"{field_name}__icontains": search_term}) - if field_name == "source_record_payload_text": - escaped_search_term = cls._json_escaped_search_term(search_term) - if escaped_search_term != search_term: - query |= Q( - **{f"{field_name}__icontains": escaped_search_term}, - ) - return query | cls._registry_search_query(search_term) - - @staticmethod - def _json_escaped_search_term(search_term: str) -> str: - return json.dumps(search_term, ensure_ascii=True)[1:-1] - - @staticmethod - def _registry_search_query(search_term: str) -> Q: - return ( - Q(extension__organization__name__icontains=search_term) - | Q(extension__organization__full_name__icontains=search_term) - | Q(extension__organization__short_name__icontains=search_term) - | Q(extension__organization__inn__icontains=search_term) - | Q(extension__organization__kpp__icontains=search_term) - | Q(extension__organization__ogrn__icontains=search_term) - | Q(extension__organization__okpo__icontains=search_term) - ) + return query @swagger_auto_schema( tags=[ORGANIZATIONS_TAG], diff --git a/tests/apps/core/test_job_control_race.py b/tests/apps/core/test_job_control_race.py new file mode 100644 index 0000000..7730b78 --- /dev/null +++ b/tests/apps/core/test_job_control_race.py @@ -0,0 +1,78 @@ +"""Отмена через API не перезаписывает завершившуюся в это время задачу.""" + +from unittest.mock import patch + +import pytest +from apps.core.models import BackgroundJob, JobStatus +from django.urls import reverse +from rest_framework.test import APIClient + +from tests.apps.user.factories import UserFactory + + +@pytest.mark.django_db +@pytest.mark.parametrize("terminal", ["success", "failure"]) +def test_revoke_preserves_worker_terminal_result(terminal): + user = UserFactory.create_user() + client = APIClient() + client.force_authenticate(user) + job = BackgroundJob.objects.create( + task_id=f"revoke-race-{terminal}", task_name="test.task", user_id=user.pk + ) + job.mark_started() + job.update_progress(65, "Обработка") + expected = {} + + def finish_during_revoke(*args, **kwargs): + worker_job = BackgroundJob.objects.get(pk=job.pk) + if terminal == "success": + worker_job.complete({"saved": 5}) + else: + worker_job.fail("Ошибка источника", "worker traceback") + expected.update( + status=worker_job.status, + progress=worker_job.progress, + progress_message=worker_job.progress_message, + result=worker_job.result, + error=worker_job.error, + traceback=worker_job.traceback, + completed_at=worker_job.completed_at, + ) + + url = reverse("api_v1:jobs:job-control", kwargs={"task_id": job.task_id}) + with patch("celery.current_app.control.revoke", side_effect=finish_during_revoke): + response = client.post(url, {"action": "revoke"}, format="json") + + assert response.status_code == 200 + job.refresh_from_db() + assert {field: getattr(job, field) for field in expected} == expected + assert response.data["status"] == ("success" if terminal == "success" else "error") + assert response.data["progress"] == expected["progress"] + assert response.data["message"] == expected["progress_message"] + + +@pytest.mark.django_db +def test_revoke_unfinished_job_records_completion_and_user_message(): + user = UserFactory.create_user() + client = APIClient() + client.force_authenticate(user) + job = BackgroundJob.objects.create( + task_id="revoke-active", task_name="test.task", user_id=user.pk + ) + job.mark_started() + job.update_progress(65, "Обработка") + url = reverse("api_v1:jobs:job-control", kwargs={"task_id": job.task_id}) + + with patch("celery.current_app.control.revoke") as revoke: + response = client.post( + url, {"action": "revoke", "terminate": True}, format="json" + ) + + revoke.assert_called_once_with(job.task_id, terminate=True) + assert response.status_code == 200 + job.refresh_from_db() + assert job.status == JobStatus.REVOKED + assert job.progress == 65 + assert job.progress_message == "Задача отозвана пользователем" + assert job.completed_at is not None + assert response.data["message"] == job.progress_message diff --git a/tests/apps/core/test_job_progress.py b/tests/apps/core/test_job_progress.py new file mode 100644 index 0000000..1382353 --- /dev/null +++ b/tests/apps/core/test_job_progress.py @@ -0,0 +1,58 @@ +"""Регрессии для запоздалых и повторных событий фоновой задачи.""" + +import pytest +from apps.core.models import BackgroundJob, JobStatus + + +@pytest.mark.django_db +def test_stale_progress_never_reduces_persisted_value(): + job = BackgroundJob.objects.create(task_id="monotonic", task_name="test.task") + stale = BackgroundJob.objects.get(pk=job.pk) + job.update_progress(75, "Новая стадия") + stale.update_progress(25, "Старая стадия") + job.refresh_from_db() + assert (job.progress, job.progress_message) == (75, "Новая стадия") + + +@pytest.mark.django_db +@pytest.mark.parametrize("terminal", ["complete", "fail", "revoke"]) +def test_terminal_job_ignores_late_events(terminal): + job = BackgroundJob.objects.create(task_id=terminal, task_name="test.task") + stale = BackgroundJob.objects.get(pk=job.pk) + job.update_progress(65, "Обработка") + if terminal == "complete": + job.complete({"saved": 2}) + elif terminal == "fail": + job.fail("Ошибка источника") + else: + job.revoke() + expected = (job.status, job.progress, job.completed_at, job.result, job.error) + stale.mark_started() + stale.mark_retry() + stale.update_progress(5, "Запоздалое событие") + stale.complete({"saved": 0}) + stale.fail("Поздняя ошибка") + stale.revoke() + job.refresh_from_db() + assert ( + job.status, + job.progress, + job.completed_at, + job.result, + job.error, + ) == expected + + +@pytest.mark.django_db +def test_retry_preserves_progress_and_success_finishes_at_100(): + job = BackgroundJob.objects.create(task_id="retry", task_name="test.task") + job.mark_started() + started_at = job.started_at + job.update_progress(40) + job.mark_retry() + job.mark_started() + job.update_progress(0) + assert job.progress == 40 + assert job.started_at == started_at + job.complete() + assert (job.status, job.progress) == (JobStatus.SUCCESS, 100) diff --git a/tests/apps/organizations/test_api_v2.py b/tests/apps/organizations/test_api_v2.py index 2a2e4e8..6cf9061 100644 --- a/tests/apps/organizations/test_api_v2.py +++ b/tests/apps/organizations/test_api_v2.py @@ -207,21 +207,28 @@ class OrganizationsApiV2Test(APITestCase): ) self.assertIn("/api/v2/organizations/{uid}/sources/", paths) self.assertIn("/api/v2/organization-sources/{uid}/records/", paths) - source_record_list_operation = paths[ - "/api/v2/organization-source-records/" - ]["get"] + source_record_list_operation = paths["/api/v2/organization-source-records/"][ + "get" + ] source_record_parameters = { parameter["name"]: parameter for parameter in source_record_list_operation["parameters"] } - source_record_ordering_values = source_record_parameters["ordering"]["enum"] - for ordering_value in ( + source_record_ordering = source_record_parameters["ordering"] + self.assertEqual(source_record_ordering["type"], "string") + self.assertNotIn("enum", source_record_ordering) + ordering_description = source_record_ordering["description"] + for ordering_field in ( + "record_date", + "external_id", + "organization__name", "payload__sentiment", - "-payload__sentiment", "payload__news_source", - "-payload__news_source", ): - self.assertIn(ordering_value, source_record_ordering_values) + self.assertIn(ordering_field, ordering_description) + self.assertIn("префикс -", ordering_description) + self.assertIn("через запятую без пробелов", ordering_description) + self.assertIn("-record_date,-external_id", ordering_description) def test_retrieve_returns_item_by_uid(self): organization = create_frontend_organization( @@ -425,7 +432,9 @@ class OrganizationsApiV2Test(APITestCase): self.assertEqual(response.status_code, status.HTTP_200_OK) self.assertNotIn("data", response.data) - self.assertEqual(response.data["sources"][0]["source_group"], "financial_indicators") + self.assertEqual( + response.data["sources"][0]["source_group"], "financial_indicators" + ) def test_source_group_filter_limits_organizations_by_source_extension(self): organization = create_frontend_organization( diff --git a/tests/apps/organizations/test_api_v2_source_extensions.py b/tests/apps/organizations/test_api_v2_source_extensions.py index 48e7a3e..39eabbb 100644 --- a/tests/apps/organizations/test_api_v2_source_extensions.py +++ b/tests/apps/organizations/test_api_v2_source_extensions.py @@ -577,7 +577,7 @@ class OrganizationSourceExtensionsApiV2Test(APITestCase): def test_flat_source_record_date_filter_precedes_pagination(self): organization = create_frontend_organization( - name='ООО "Filtered pagination"', + name='ООО "Искомое"', inn="7707083892", ogrn="1027700132092", ) @@ -755,7 +755,7 @@ class OrganizationSourceExtensionsApiV2Test(APITestCase): self.assertEqual(response_organization["inn"], "0277106840") self.assertEqual(response_organization["ogrn"], organization.ogrn) - def test_flat_source_records_searches_payload_values_displayed_in_tables(self): + def test_flat_source_records_does_not_search_payload_values(self): target = create_frontend_organization( name='ООО "Поиск по payload"', inn="7707083817", @@ -814,11 +814,8 @@ class OrganizationSourceExtensionsApiV2Test(APITestCase): ) self.assertEqual(response.status_code, status.HTTP_200_OK) - self.assertEqual(response.data["meta"]["pagination"]["total_count"], 1) - self.assertEqual( - response.data["data"][0]["external_id"], - "INSP-PAYLOAD", - ) + self.assertEqual(response.data["meta"]["pagination"]["total_count"], 0) + self.assertEqual(response.data["data"], []) def test_flat_source_records_searches_registry_identity_displayed_in_tables(self): organization = create_frontend_organization( diff --git a/tests/apps/organizations/test_canonical_search.py b/tests/apps/organizations/test_canonical_search.py new file mode 100644 index 0000000..b1a5a13 --- /dev/null +++ b/tests/apps/organizations/test_canonical_search.py @@ -0,0 +1,140 @@ +"""Organization search must use the same canonical identity in every table.""" + +from django.core.cache import cache +from django.urls import reverse +from organizations.models import ( + Organization, + OrganizationSourceRecord, + PlannedInspectionExtension, +) +from rest_framework.test import APITestCase + +from tests.apps.user.factories import UserFactory + + +class CanonicalOrganizationSearchTest(APITestCase): + def setUp(self): + cache.clear() + self.client.force_authenticate(UserFactory.create_user()) + self.organization = Organization.objects.create( + name="Росатом", + full_name="Полное наименование Росатома", + short_name="Краткое", + inn="7707083881", + kpp="770701081", + ogrn="1027700132181", + okpo="00003881", + opk_registry_membership=True, + ) + extension = PlannedInspectionExtension.objects.create( + organization=self.organization, + title="Проверки", + ) + for index in range(2): + OrganizationSourceRecord.objects.create( + extension=extension, + source="inspections", + record_type="inspection", + external_id=f"target-{index}", + status="active", + ) + other = Organization.objects.create(name="Другая", opk_registry_membership=True) + other_extension = PlannedInspectionExtension.objects.create( + organization=other, + title="Росатом", + ) + OrganizationSourceRecord.objects.create( + extension=other_extension, + source="inspections", + record_type="inspection", + external_id="Росатом", + title="Росатом", + status="active", + url="https://example.test/Росатом", + payload={"raw_text": "Росатом"}, + ) + self.organizations_url = reverse("api_v2:organizations:organizations-list") + self.records_url = reverse( + "api_v2:organizations:organization-source-records-list" + ) + + def test_all_canonical_names_and_identifiers_are_searchable_in_both_endpoints(self): + for search in ( + "Росатом", + "Полное", + "Краткое", + "7707083881", + "770701081", + "1027700132181", + "00003881", + ): + with self.subTest(search=search): + organizations = self.client.get( + self.organizations_url, {"search": search} + ) + records = self.client.get(self.records_url, {"search": search}) + self.assertEqual(organizations.status_code, 200) + self.assertEqual(records.status_code, 200) + self.assertEqual( + organizations.data["meta"]["pagination"]["total_count"], 1 + ) + self.assertEqual(records.data["meta"]["pagination"]["total_count"], 2) + self.assertEqual( + organizations.data["data"][0]["uid"], str(self.organization.uid) + ) + + def test_entrepreneur_identifier_is_searchable_in_both_endpoints(self): + entrepreneur = Organization.objects.create( + name="ИП Иванов", + inn="500100732259", + ogrip="304500116000181", + opk_registry_membership=True, + ) + extension = PlannedInspectionExtension.objects.create( + organization=entrepreneur, + title="Проверки", + ) + OrganizationSourceRecord.objects.create( + extension=extension, + source="inspections", + record_type="inspection", + external_id="entrepreneur-record", + ) + for url in (self.organizations_url, self.records_url): + response = self.client.get(url, {"search": "304500116000181"}) + self.assertEqual(response.status_code, 200) + self.assertEqual(response.data["meta"]["pagination"]["total_count"], 1) + + def test_search_terms_and_source_filters_are_applied_before_pagination(self): + extension = self.organization.source_extensions.get() + OrganizationSourceRecord.objects.create( + extension=extension, + source="other-source", + record_type="inspection", + external_id="filtered-out", + status="active", + ) + params = { + "search": "Росатом 7707083881", + "source_group": "planned_inspections", + "source": "inspections", + "record_type": "inspection", + "status": "active", + "ordering": "external_id", + "page_size": 1, + } + first = self.client.get(self.records_url, {**params, "page": 1}) + second = self.client.get(self.records_url, {**params, "page": 2}) + self.assertEqual(first.status_code, 200) + self.assertEqual(second.status_code, 200) + self.assertEqual(first.data["meta"]["pagination"]["total_count"], 2) + self.assertEqual(first.data["data"][0]["external_id"], "target-0") + self.assertEqual(second.data["data"][0]["external_id"], "target-1") + + def test_technical_identity_is_not_a_search_field(self): + Organization.objects.filter(pk=self.organization.pk).update( + primary_identity="technical-only" + ) + response = self.client.get(self.organizations_url, {"search": "technical-only"}) + self.assertEqual(response.status_code, 200) + self.assertEqual(response.data["meta"]["pagination"]["total_count"], 0) diff --git a/tests/apps/organizations/test_source_record_composite_ordering.py b/tests/apps/organizations/test_source_record_composite_ordering.py new file mode 100644 index 0000000..45e0f28 --- /dev/null +++ b/tests/apps/organizations/test_source_record_composite_ordering.py @@ -0,0 +1,95 @@ +"""API accepts frontend composite ordering without widening its field allowlist.""" + +import pytest +from django.urls import reverse +from organizations.models import ( + GovernmentSupportExtension, + Organization, + OrganizationSourceRecord, +) +from rest_framework.test import APIClient + +from tests.apps.user.factories import UserFactory + + +@pytest.fixture +def source_records_client(db): + client = APIClient() + client.force_authenticate(UserFactory.create_user()) + organization = Organization.objects.create( + name="Alpha", + inn="7707083810", + ogrn="1027700132010", + opk_registry_membership=True, + ) + extension = GovernmentSupportExtension.objects.create(organization=organization) + for external_id, record_date in ( + ("A", "2026-01-31"), + ("Z", "2026-01-31"), + ("M", "2026-01-01"), + ("N", ""), + ): + OrganizationSourceRecord.objects.create( + extension=extension, + record_type="sme_support_measure", + source="fns_sme_support_recipients", + external_id=external_id, + record_date=record_date, + ) + return client + + +@pytest.mark.parametrize( + ("ordering", "expected"), + [ + ("-record_date,-external_id", ["Z", "A", "M", "N"]), + ("record_date,-external_id", ["M", "Z", "A", "N"]), + ("organization__name,-external_id", ["Z", "N", "M", "A"]), + ("organization__name", ["A", "M", "N", "Z"]), + ], +) +def test_source_record_composite_ordering_and_null_dates( + source_records_client, ordering, expected +): + response = source_records_client.get( + reverse("api_v2:organizations:organization-source-records-list"), + {"source": "fns_sme_support_recipients", "ordering": ordering}, + ) + + assert response.status_code == 200 + assert [record["external_id"] for record in response.data["data"]] == expected + + +def test_composite_ordering_precedes_pagination(source_records_client): + response = source_records_client.get( + reverse("api_v2:organizations:organization-source-records-list"), + {"ordering": "-record_date,-external_id", "page_size": 1, "page": 2}, + ) + + assert response.status_code == 200 + assert response.data["data"][0]["external_id"] == "A" + assert response.data["meta"]["pagination"]["total_count"] == 4 + + +@pytest.mark.parametrize( + "ordering", + [ + "-record_date,password", + "-record_date,payload__support_form__name", + "-record_date,", + ",external_id", + "record_date,,external_id", + "record_date, external_id", + ], +) +def test_composite_ordering_rejects_any_non_allowlisted_part( + source_records_client, ordering +): + response = source_records_client.get( + reverse("api_v2:organizations:organization-source-records-list"), + {"ordering": ordering}, + ) + + assert response.status_code == 400 + assert response.data["errors"][0]["code"] == "invalid_ordering" + assert response.data["data"] == [] diff --git a/tests/apps/organizations/test_source_record_export.py b/tests/apps/organizations/test_source_record_export.py index 525018e..879ea07 100644 --- a/tests/apps/organizations/test_source_record_export.py +++ b/tests/apps/organizations/test_source_record_export.py @@ -93,9 +93,9 @@ class OrganizationSourceRecordExportApiV2Test(APITestCase): call_command("build_source_record_exports", stdout=command_output) generation = load_current_source_record_export_generation() - self.assertEqual(generation.artifacts_count, 34) + self.assertEqual(generation.artifacts_count, 40) self.assertEqual(generation.export_year, timezone.localdate().year) - self.assertIn('"artifacts_count": 34', command_output.getvalue()) + self.assertIn('"artifacts_count": 40', command_output.getvalue()) def test_sanctions_xlsx_export_uses_source_contract_columns_and_boolean_flags(self): organization = Organization.objects.create( @@ -150,17 +150,17 @@ class OrganizationSourceRecordExportApiV2Test(APITestCase): rows[0], ( "Наименование", - "rn", - "ogrn", - "inn", - "okpo", - "Санкции - Великобритания HM Treasury", - "Санкции - Евросоюз", - "Санкции - США", - "Санкции - Швейцария", - "Санкции секторальные - США", - "Санкции - Великобритания UKSL", - "Санкции - Украина", + "Регистрационный номер", + "ОГРН", + "ИНН", + "ОКПО", + "Санкции — Великобритания HM Treasury", + "Санкции — Евросоюз", + "Санкции — США", + "Санкции — Швейцария", + "Санкции секторальные — США", + "Санкции — Великобритания UKSL", + "Санкции — Украина", ), ) self.assertEqual( @@ -181,6 +181,74 @@ class OrganizationSourceRecordExportApiV2Test(APITestCase): ), ) + def test_sanctions_localized_headers_apply_to_all_xlsx_parts_only(self): + self.client.force_authenticate(UserFactory.create_superuser()) + organization = Organization.objects.create( + name="АО Части выгрузки", + inn="0001234567", + ogrn="1027700132195", + okpo="00001234", + ) + extension = SanctionsExtension.objects.create( + organization=organization, title="Санкции" + ) + for index in range(2): + OrganizationSourceRecord.objects.create( + extension=extension, + source="ropk_sanctions", + record_type="organization_sanctions", + external_id=f"000{index}", + payload={"rn": f"000{index}", "ukraine": True}, + ) + with self.settings(SOURCE_RECORD_EXPORT_XLSX_ROWS_PER_FILE=1): + build_source_record_export_artifacts() + + ticket = self.client.post( + self.ticket_url, {"sources": ["sanctions"], "format": "xlsx"}, format="json" + ) + self.assertEqual(ticket.status_code, 201) + response = self.client.post( + self.download_url, {"ticket": ticket.data["ticket"]} + ) + self.assertEqual(response.status_code, 200) + with zipfile.ZipFile(BytesIO(self._response_body(response))) as archive: + self.assertEqual(len(archive.namelist()), 2) + for name in archive.namelist(): + workbook = load_workbook(BytesIO(archive.read(name)), read_only=True) + cells = list(workbook["data"].iter_rows()) + self.assertEqual( + [cell.value for cell in cells[0]][:5], + [ + "Наименование", + "Регистрационный номер", + "ОГРН", + "ИНН", + "ОКПО", + ], + ) + self.assertEqual([cell.data_type for cell in cells[1]][1:5], ["s"] * 4) + self.assertEqual(cells[1][3].value, "0001234567") + workbook.close() + + for file_format in ("csv", "json"): + response = self.client.post( + self.url, + {"sources": ["sanctions"], "format": file_format}, + format="json", + ) + self.assertEqual(response.status_code, 200) + with zipfile.ZipFile(BytesIO(self._response_body(response))) as archive: + content = archive.read(archive.namelist()[0]).decode("utf-8-sig") + rows = ( + list(csv.DictReader(StringIO(content))) + if file_format == "csv" + else json.loads(content) + ) + self.assertEqual(len(rows), 2) + self.assertEqual({row["rn"] for row in rows}, {"0000", "0001"}) + self.assertEqual(rows[0]["inn"], "0001234567") + self.assertNotIn("Регистрационный номер", rows[0]) + def test_generation_contains_only_records_from_its_calendar_year(self): export_year = 2026 generated_at = datetime(export_year, 8, 4, 6, 0, tzinfo=UTC) @@ -398,7 +466,7 @@ class OrganizationSourceRecordExportApiV2Test(APITestCase): self.assertEqual( response["X-Source-Export-Generated-At"], generation.generated_at ) - self.assertEqual(generation.artifacts_count, 34) + self.assertEqual(generation.artifacts_count, 40) self.assertIn( 'filename="planned-inspections__financial-indicators_', response["Content-Disposition"], @@ -593,8 +661,8 @@ class OrganizationSourceRecordExportApiV2Test(APITestCase): key=lambda item: item.file_name, ) - self.assertEqual(generation.artifacts_count, 34) - self.assertEqual(generation.files_count, 35) + self.assertEqual(generation.artifacts_count, 40) + self.assertEqual(generation.files_count, 41) self.assertEqual( [item.file_name for item in artifacts], [ @@ -792,7 +860,7 @@ class OrganizationSourceRecordExportApiV2Test(APITestCase): first_generation = build_source_record_export_artifacts() current_generation = load_current_source_record_export_generation() - self.assertEqual(first_generation.artifacts_count, 34) + self.assertEqual(first_generation.artifacts_count, 40) self.assertEqual( current_generation.generation_id, first_generation.generation_id ) diff --git a/tests/apps/organizations/test_tasks.py b/tests/apps/organizations/test_tasks.py index 133a352..1472ab4 100644 --- a/tests/apps/organizations/test_tasks.py +++ b/tests/apps/organizations/test_tasks.py @@ -145,7 +145,7 @@ class SourceRecordExportArtifactsTaskTest(TestCase): result = refresh_source_record_export_artifacts() self.assertEqual(result["status"], "success") - self.assertEqual(result["artifacts_count"], 34) + self.assertEqual(result["artifacts_count"], 40) self.assertEqual(result["export_year"], timezone.localdate().year) self.assertIsNone(cache.get(settings.SOURCE_RECORD_EXPORT_LOCK_KEY)) diff --git a/tests/apps/organizations/test_test_companies_commands.py b/tests/apps/organizations/test_test_companies_commands.py index 60a61ef..3c27e6d 100644 --- a/tests/apps/organizations/test_test_companies_commands.py +++ b/tests/apps/organizations/test_test_companies_commands.py @@ -138,6 +138,33 @@ class TestCompaniesCommandsTest(TestCase): expected_financial_years, ) self.assertEqual(GenericParserRecord.objects.count(), 20 * 9) + snapshot_sources = ( + ParserLoadLog.Source.FNS_SME_SUPPORT_RECIPIENTS, + ParserLoadLog.Source.BUDGET_UBPANDNUBP, + ) + snapshot_records = OrganizationSourceRecord.objects.filter( + source__in=snapshot_sources, + ) + self.assertEqual(snapshot_records.count(), 40) + self.assertEqual( + snapshot_records.filter(legacy_model="", legacy_pk="").count(), 40 + ) + self.assertFalse( + GenericParserRecord.objects.filter(source__in=snapshot_sources).exists() + ) + support = snapshot_records.filter(source=snapshot_sources[0]).first() + self.assertEqual(support.record_type, "sme_support_measure") + self.assertEqual(len(support.payload["support_sizes"]), 2) + self.assertEqual( + support.payload["support_sizes"][0]["value"], str(support.amount) + ) + budget = snapshot_records.filter(source=snapshot_sources[1]).first() + self.assertEqual(budget.record_type, "budget_registry_organization") + self.assertEqual(budget.status, "inactive") + self.assertEqual(budget.payload["registry"]["status_code"], "2") + self.assertEqual( + budget.payload["upstream"]["info"]["inn"], budget.extension.organization.inn + ) def test_create_updates_the_fixed_dataset_without_duplicates(self): call_command("create_test_companies", stdout=StringIO()) @@ -218,7 +245,7 @@ class TestCompaniesCommandsTest(TestCase): year=stale_year, ).exists() ) - self.assertEqual(OrganizationSourceRecord.objects.count(), 20 * 17) + self.assertEqual(OrganizationSourceRecord.objects.count(), 20 * 19) def test_create_replaces_stale_source_record_for_same_legacy_row(self): call_command("create_test_companies", stdout=StringIO()) diff --git a/tests/apps/parsers/test_refresh_concurrency.py b/tests/apps/parsers/test_refresh_concurrency.py new file mode 100644 index 0000000..d4d4869 --- /dev/null +++ b/tests/apps/parsers/test_refresh_concurrency.py @@ -0,0 +1,80 @@ +"""Real row-lock coverage for admission and asynchronous progress events.""" + +from concurrent.futures import ThreadPoolExecutor +from threading import Barrier +from unittest.mock import MagicMock + +import pytest +from apps.core.exceptions import ConflictError +from apps.core.models import BackgroundJob +from apps.parsers.source_cards import SourceCardService +from django.db import close_old_connections, connection + +pytestmark = pytest.mark.django_db(transaction=True) + + +def _parallel(actions): + barrier = Barrier(len(actions)) + + def run(action): + close_old_connections() + try: + barrier.wait(timeout=10) + return action() + finally: + close_old_connections() + + with ThreadPoolExecutor(max_workers=len(actions)) as executor: + return list(executor.map(run, actions)) + + +@pytest.mark.parametrize( + "slug,source", + [ + ("sme-support-recipients-registry", "fns_sme_support_recipients"), + ("budget-process-registry", "budget_ubpandnubp"), + ], +) +def test_two_simultaneous_refreshes_dispatch_exactly_one_task(slug, source): + if connection.vendor != "postgresql": + pytest.skip("Requires PostgreSQL row locks") + task = MagicMock() + definition = SourceCardService.get_definition(slug) + + def refresh(): + try: + SourceCardService._enqueue_refresh_group( + definition, + [(task, f"parsers.{source}.refresh", source)], + requested_by_id=None, + kwargs={}, + ) + return "queued" + except ConflictError: + return "conflict" + + assert sorted(_parallel([refresh, refresh])) == ["conflict", "queued"] + assert task.apply_async.call_count == 1 + assert BackgroundJob.objects.count() == 1 + + +def test_concurrent_progress_preserves_maximum_and_terminal_result(): + if connection.vendor != "postgresql": + pytest.skip("Requires PostgreSQL concurrent updates") + job = BackgroundJob.objects.create( + task_id="concurrent-progress", task_name="test.task" + ) + + def progress(value): + return lambda: BackgroundJob.objects.get(pk=job.pk).update_progress(value) + + _parallel([progress(value) for value in [10, 85, 30, 55]]) + job.refresh_from_db() + assert job.progress == 85 + _parallel([lambda: job.fail("Source unavailable"), progress(25)]) + job.refresh_from_db() + assert (job.status, job.progress, job.error) == ( + "failure", + 85, + "Source unavailable", + ) diff --git a/tests/apps/parsers/test_refresh_progress.py b/tests/apps/parsers/test_refresh_progress.py new file mode 100644 index 0000000..88d0c8e --- /dev/null +++ b/tests/apps/parsers/test_refresh_progress.py @@ -0,0 +1,147 @@ +"""Общий прогресс ручного запуска, в том числе после завершения его частей.""" + +from datetime import timedelta +from unittest.mock import MagicMock + +import pytest +from apps.core.models import BackgroundJob, JobStatus +from apps.parsers.source_cards import SourceCardService +from django.test import override_settings +from django.utils import timezone + + +@pytest.mark.django_db +def test_all_jobs_are_visible_before_dispatch_and_completed_parts_stay_in_progress(): + definition = SourceCardService.get_definition("manufacturers-and-products") + observed_counts = [] + tasks = [MagicMock(), MagicMock(), MagicMock()] + + def dispatch(**kwargs): + jobs = list(BackgroundJob.objects.all()) + observed_counts.append(len(jobs)) + assert {job.task_id for job in jobs} == set(jobs[0].meta["refresh_task_ids"]) + + for task in tasks: + task.apply_async.side_effect = dispatch + specs = tuple( + (task, name, source) + for task, name, source in zip( + tasks, + [ + "apps.parsers.tasks.parse_industrial_production", + "apps.parsers.tasks.parse_industrial_products", + "apps.parsers.tasks.parse_manufactures", + ], + ["industrial", "industrial_products", "manufactures"], + strict=False, + ) + ) + result = SourceCardService._enqueue_refresh_group( + definition, specs, requested_by_id=1, kwargs={} + ) + assert observed_counts == [3, 3, 3] + jobs = [BackgroundJob.objects.get(task_id=item["task_id"]) for item in result] + jobs[0].complete() + jobs[1].update_progress(50) + jobs[2].update_progress(0) + card = SourceCardService.get_card(definition.slug) + assert card["progress"] == 50 + assert len(card["active_tasks"]) == 2 + jobs[1].complete() + assert SourceCardService.get_card(definition.slug)["progress"] == 67 + jobs[2].complete() + for card in [ + SourceCardService.get_card(definition.slug), + next( + card + for card in SourceCardService.list_cards() + if card["slug"] == definition.slug + ), + ]: + assert (card["progress"], card["status"], card["active_tasks"]) == ( + 100, + "success", + [], + ) + + +@pytest.mark.django_db +def test_failed_run_keeps_progress_and_error_after_last_active_job_disappears(): + job = BackgroundJob.objects.create( + task_id="failed-refresh", + task_name="apps.parsers.tasks.parse_arbitration_cases", + meta={ + "source_card": "arbitration-cases", + "refresh_task_ids": ["failed-refresh"], + }, + ) + job.update_progress(60) + job.fail("Ошибка загрузки") + card = SourceCardService.get_card("arbitration-cases") + assert (card["status"], card["progress"], card["error_message"]) == ( + "error", + 60, + "Ошибка загрузки", + ) + + +@pytest.mark.django_db +def test_broker_failure_keeps_all_registered_parts_in_final_failed_state(): + definition = SourceCardService.get_definition("defense-unreliable-suppliers") + task = MagicMock() + task.apply_async.side_effect = RuntimeError("broker unavailable") + with pytest.raises(RuntimeError): + SourceCardService._enqueue_refresh_group( + definition, + [ + (task, "test.first", "unfair_suppliers"), + (task, "test.second", "fas_goz"), + ], + requested_by_id=1, + kwargs={}, + ) + jobs = list(BackgroundJob.objects.all()) + assert len(jobs) == 2 + assert all(job.status == JobStatus.FAILURE and job.error for job in jobs) + assert task.apply_async.call_count == 1 + + +@pytest.mark.django_db +def test_completed_new_run_does_not_hide_active_older_run(): + definition = SourceCardService.get_definition("manufacturers-and-products") + specs = [(MagicMock(), name, "industrial") for name in definition.task_names] + first = SourceCardService._enqueue_refresh_group( + definition, specs, requested_by_id=None, kwargs={} + ) + second = SourceCardService._enqueue_refresh_group( + definition, specs, requested_by_id=None, kwargs={} + ) + for item in second: + BackgroundJob.objects.get(task_id=item["task_id"]).complete() + card = SourceCardService.get_card(definition.slug) + assert {job["task_id"] for job in card["active_tasks"]} == { + job["task_id"] for job in first + } + assert (card["status"], card["progress"]) == ("in_progress", 50) + for item in first: + BackgroundJob.objects.get(task_id=item["task_id"]).complete() + assert SourceCardService.get_card(definition.slug)["progress"] == 100 + + +@pytest.mark.django_db +@pytest.mark.parametrize( + "status,time_field", + [(JobStatus.PENDING, "created_at"), (JobStatus.STARTED, "updated_at")], +) +@override_settings( + PARSER_STALE_LOAD_MAX_AGE_MINUTES=10, PARSER_STALE_PENDING_JOB_MAX_AGE_MINUTES=10 +) +def test_latest_task_does_not_reactivate_a_stale_job(status, time_field): + definition = SourceCardService.get_definition("arbitration-cases") + job = BackgroundJob.objects.create( + task_id="stale-task", task_name=definition.task_names[0], status=status + ) + BackgroundJob.objects.filter(pk=job.pk).update( + **{time_field: timezone.now() - timedelta(minutes=15)} + ) + assert SourceCardService.get_card(definition.slug)["active_tasks"] == [] diff --git a/tests/apps/parsers/test_registry_http.py b/tests/apps/parsers/test_registry_http.py new file mode 100644 index 0000000..1df8f80 --- /dev/null +++ b/tests/apps/parsers/test_registry_http.py @@ -0,0 +1,149 @@ +"""Transport regression cases observed while downloading the full FNS snapshot.""" + +from contextlib import contextmanager +from io import BytesIO +from types import SimpleNamespace +from unittest.mock import patch + +import pytest +import requests +from apps.parsers.registry_http import download_registry_archive +from apps.parsers.registry_snapshots import SnapshotValidationError + +URL = "https://file.nalog.ru/archive.zip" +HOSTS = frozenset({"file.nalog.ru"}) + + +def _response( + chunks, *, status=206, content_range="bytes 0-5/6", etag='"v1"', length=None +): + headers = {} + if content_range is not None: + headers["Content-Range"] = content_range + if etag is not None: + headers["ETag"] = etag + if length is not None: + headers["Content-Length"] = str(length) + + def content(**_): + for chunk in chunks: + if isinstance(chunk, Exception): + raise chunk + yield chunk + + return SimpleNamespace(status_code=status, headers=headers, iter_content=content) + + +def _download(responses, **kwargs): + calls = [] + responses = iter(responses) + + @contextmanager + def response(_session, url, *, hosts, headers): + assert url == URL + assert hosts == HOSTS + calls.append(headers) + yield next(responses) + + handle = BytesIO() + with patch("apps.parsers.registry_http.registry_response", side_effect=response): + result = download_registry_archive( + object(), URL, handle, hosts=HOSTS, maximum=1024, range_bytes=6, **kwargs + ) + return handle.getvalue(), result, calls + + +def test_interrupted_body_resumes_only_missing_bytes_with_validator(): + content, count, calls = _download( + [ + _response([b"ab", requests.exceptions.ChunkedEncodingError("interrupted")]), + _response([b"cdef"], content_range="bytes 2-5/6"), + ] + ) + assert content == b"abcdef" + assert count == 6 + assert calls[1]["Range"] == "bytes=2-7" + assert calls[1]["If-Range"] == '"v1"' + + +@pytest.mark.parametrize( + ("next_response", "reason"), + [ + ( + _response([b"cdef"], content_range="bytes 2-5/6", etag='"v2"'), + "source_changed_during_download", + ), + ( + _response([b"cdefgh"], content_range="bytes 2-7/8"), + "source_changed_during_download", + ), + (_response([b"abcdef"], status=200, length=6), "source_range_not_honored"), + ( + _response([b"cdef"], content_range="bytes 0-3/6"), + "unexpected_source_content_range", + ), + ], +) +def test_resume_rejects_changed_or_unproven_entity(next_response, reason): + with pytest.raises(SnapshotValidationError, match=reason): + _download( + [ + _response( + [b"ab", requests.exceptions.ChunkedEncodingError("interrupted")] + ), + next_response, + ] + ) + + +def test_no_range_server_restarts_entire_body_after_interruption(): + content, count, calls = _download( + [ + _response( + [b"ab", requests.exceptions.ChunkedEncodingError("interrupted")], + status=200, + length=6, + ), + _response([b"abcdef"], status=200, length=6), + ] + ) + assert content == b"abcdef" + assert count == 6 + assert calls[1]["Range"] == "bytes=0-5" + assert "If-Range" not in calls[1] + + +def test_range_without_validator_is_rejected(): + with pytest.raises( + SnapshotValidationError, match="source_resume_validator_missing" + ): + _download([_response([b"abcdef"], etag=None)]) + + +def test_last_modified_is_used_when_strong_etag_is_unavailable(): + first = _response( + [b"ab", requests.exceptions.ChunkedEncodingError("interrupted")], + etag='W/"weak"', + ) + second = _response([b"cdef"], content_range="bytes 2-5/6", etag='W/"weak"') + for response in (first, second): + response.headers["Last-Modified"] = "Sat, 15 Aug 2026 09:46:30 GMT" + content, _, calls = _download([first, second]) + assert content == b"abcdef" + assert calls[1]["If-Range"] == "Sat, 15 Aug 2026 09:46:30 GMT" + + +def test_body_retry_count_is_bounded(): + with pytest.raises(requests.exceptions.ChunkedEncodingError): + _download( + [ + _response([requests.exceptions.ChunkedEncodingError("interrupted")]) + for _ in range(2) + ], + max_retries=1, + ) + + +def test_oversized_archive_is_rejected_before_reading_body(): + with pytest.raises(SnapshotValidationError, match="source_response_too_large"): + _download([_response([], content_range="bytes 0-5/2048")]) diff --git a/tests/apps/parsers/test_registry_snapshots.py b/tests/apps/parsers/test_registry_snapshots.py new file mode 100644 index 0000000..7331617 --- /dev/null +++ b/tests/apps/parsers/test_registry_snapshots.py @@ -0,0 +1,549 @@ +"""Regression coverage for official registry ingestion without upstream IO.""" + +from __future__ import annotations + +import zipfile +from collections import Counter +from io import BytesIO +from tempfile import TemporaryDirectory +from types import SimpleNamespace +from unittest.mock import patch + +from apps.core.models import BackgroundJob, JobStatus +from apps.parsers.budget_registry import BUDGET_SOURCE, refresh_budget_registry +from apps.parsers.models import ParserLoadLog, ParserSourceArtifact, ParserStagedRecord +from apps.parsers.registry_snapshots import SnapshotValidationError, publish_snapshot +from apps.parsers.services import ParserLoadLogService +from apps.parsers.sme_support import ( + SME_SOURCE, + discover_sme_archive, + import_sme_support, + iter_sme_measures, +) +from apps.parsers.tasks_registry_snapshots import _run_snapshot +from django.test import TestCase, override_settings +from django.utils import timezone +from lxml import etree +from organizations.models import Organization, OrganizationSourceRecord +from organizations.serializers import ( + OrganizationSourceRecordListSerializer, + OrganizationSourceRecordSerializer, +) +from organizations.source_record_export import _source_group_queryset +from rest_framework.test import APIClient + +from tests.apps.user.factories import UserFactory + + +def _xml( + *, + numbers=("old-support", "new-support"), + inn="1234567890", + documents=1, + units=("1", "3"), +): + root = etree.Element( + "Файл", + ИдФайл="fixture", + ВерсФорм="4.04", + ТипИнф="РЕЕСТРМСП-ПП", + КолДок=str(documents), + ) + doc = etree.SubElement( + root, "Документ", ИдДок="recipient-doc", ДатаСост="15.08.2026" + ) + etree.SubElement( + doc, "СвЮЛ", НаимОрг="АО Проверка", ИННЮЛ=inn, ОГРН="1027700132195" + ) + for number in numbers: + measure = etree.SubElement( + doc, + "СвПредПод", + НомерПод=number, + ДатаСвед="04.10.2024", + ДатаОбнов="01.07.2026", + ВидПП="1", + НаимОрг="Поставщик", + ИННЮЛ="9876543210", + КатСуб="1", + СрокПод="01.01.2027", + ДатаПрин="01.10.2024", + ИнфНаруш="1", + ) + etree.SubElement(measure, "ФормПод", КодФорм="0001", НаимФорм="Форма") + etree.SubElement(measure, "ВидПод", КодВид="0002", НаимВид="Вид") + for unit in units: + etree.SubElement(measure, "РазмПод", ЕдПод=unit, РазмПод="2.50") + etree.SubElement(measure, "Нарушения", ВидНаруш="1", ДатаНаруш="01.02.2025") + etree.SubElement(measure, "РегДок", НомерРД="A", НаимРД="Первый документ") + etree.SubElement(measure, "РегДок", НомерРД="B", НаимРД="Второй документ") + return etree.tostring(root, encoding="UTF-8", xml_declaration=True) + + +def _archive(xml: bytes) -> BytesIO: + handle = BytesIO() + with zipfile.ZipFile(handle, "w", compression=zipfile.ZIP_DEFLATED) as archive: + archive.writestr("support.xml", xml) + handle.seek(0) + return handle + + +def _budget_record(identifier="1", *, inn="1234567890", status="2"): + return { + "id": identifier, + "info": { + "inn": inn, + "ogrn": "1027700132195", + "fullName": "АО Проверка", + "okpoCode": "12345678", + "statusCode": status, + "dateUpdate": "2024-10-01T15:00:00", + "code": "own-code", + "recordNum": "00000001", + }, + "activities": [{"name": "activity"}], + "unknown_new_block": [{"raw": "preserved"}], + } + + +class RegistrySnapshotTest(TestCase): + def setUp(self): + self.storage = TemporaryDirectory() + self.addCleanup(self.storage.cleanup) + self.settings = override_settings(MEDIA_ROOT=self.storage.name) + self.settings.enable() + self.addCleanup(self.settings.disable) + self.organization = Organization.objects.create( + name="АО Проверка", + inn="1234567890", + ogrn="1027700132195", + okpo="12345678", + directory_imported_at=timezone.now(), + opk_registry_membership=True, + ) + + def import_sme(self, xml=None, batch=1): + return import_sme_support( + handle=_archive(xml or _xml()), + original_name="snapshot.zip", + snapshot_date="2026-08-15", + load_batch=batch, + ) + + def test_sme_multiple_measures_nested_data_and_stable_ids(self): + artifact, result = self.import_sme() + self.assertEqual((result.parsed, result.published), (2, 2)) + records = list(OrganizationSourceRecord.objects.order_by("external_id")) + self.assertEqual(records[0].record_date, "2024-10-01") + self.assertEqual(str(records[0].amount), "2.50") + self.assertEqual(len(records[0].payload["support_sizes"]), 2) + self.assertEqual(len(records[0].payload["regulatory_documents"]), 2) + self.assertEqual(records[0].payload["source_updated_date"], "2026-07-01") + self.assertEqual(len(records[0].payload["violations"]), 1) + before = {record.external_id: record.uid for record in records} + self.import_sme(batch=2) + self.assertEqual( + dict(OrganizationSourceRecord.objects.values_list("external_id", "uid")), + before, + ) + self.assertEqual(Organization.objects.count(), 1) + self.assertEqual(artifact.status, ParserSourceArtifact.Status.PUBLISHED) + + def test_sme_non_rub_amount_is_null_and_foreign_subject_not_created(self): + self.import_sme(_xml(units=("3", "4"))) + self.assertIsNone(OrganizationSourceRecord.objects.first().amount) + _, result = self.import_sme(_xml(inn="5678901234"), batch=2) + self.assertEqual(result.published, 0) + self.assertEqual(result.skipped, 2) + self.assertEqual(Organization.objects.count(), 1) + self.assertEqual(OrganizationSourceRecord.objects.count(), 0) + + def test_sme_invalid_truncated_or_conflicting_snapshot_keeps_old_records(self): + self.import_sme() + before = set(OrganizationSourceRecord.objects.values_list("uid", flat=True)) + for xml in (_xml(numbers=("same", "same")), _xml()[:-15]): + with self.assertRaises((SnapshotValidationError, etree.XMLSyntaxError)): + self.import_sme(xml, batch=2) + self.assertEqual( + set(OrganizationSourceRecord.objects.values_list("uid", flat=True)), + before, + ) + + def test_official_document_count_mismatch_is_retained_as_diagnostic(self): + root = etree.fromstring(_xml()) # noqa: S320 - locally generated fixture + other = etree.fromstring(_xml(numbers=("third-measure",))) # noqa: S320 + root.append(other.find("Документ")) + artifact, result = self.import_sme(etree.tostring(root, encoding="UTF-8")) + self.assertEqual(result.published, 3) + self.assertEqual(artifact.metadata["documents_count"], 2) + self.assertEqual(artifact.metadata["declared_documents_count"], 1) + self.assertEqual(artifact.metadata["document_count_mismatch_files"], 1) + + def test_sme_publish_failure_rolls_back_updates_and_deletes(self): + self.import_sme() + before = set(OrganizationSourceRecord.objects.values_list("uid", flat=True)) + with self.assertRaises(RuntimeError), patch( + "apps.parsers.registry_snapshots.OrganizationSourceIngestionService.save_records", + side_effect=RuntimeError("stop"), + ): + self.import_sme(_xml(numbers=("replacement",)), batch=2) + self.assertEqual( + set(OrganizationSourceRecord.objects.values_list("uid", flat=True)), before + ) + + def test_repeated_publication_keeps_previous_records_and_counts(self): + artifact, original = self.import_sme() + before = set(OrganizationSourceRecord.objects.values_list("uid", flat=True)) + self.assertEqual(publish_snapshot(artifact), original) + self.assertEqual( + set(OrganizationSourceRecord.objects.values_list("uid", flat=True)), before + ) + artifact.refresh_from_db() + self.assertEqual(artifact.published_count, 2) + + def test_cache_failure_rolls_back_publication(self): + self.import_sme() + before = set(OrganizationSourceRecord.objects.values_list("uid", flat=True)) + with self.assertRaises(RuntimeError), patch( + "apps.parsers.registry_snapshots.invalidate_source_data_cache", + side_effect=RuntimeError("cache"), + ): + self.import_sme(_xml(numbers=("replacement",)), batch=2) + self.assertEqual( + set(OrganizationSourceRecord.objects.values_list("uid", flat=True)), before + ) + + def test_cache_is_invalidated_again_only_after_commit(self): + with patch( + "apps.parsers.registry_snapshots.invalidate_source_data_cache" + ) as invalidate, self.captureOnCommitCallbacks(execute=True) as callbacks: + artifact, _ = self.import_sme() + self.assertEqual(invalidate.call_count, 1) + artifact.refresh_from_db() + self.assertEqual(artifact.status, ParserSourceArtifact.Status.PUBLISHED) + self.assertEqual(len(callbacks), 1) + self.assertEqual(invalidate.call_count, 2) + + def test_after_commit_cache_failure_does_not_reject_published_data(self): + with patch( + "apps.parsers.registry_snapshots.invalidate_source_data_cache", + side_effect=[None, RuntimeError("cache")], + ), self.assertLogs( + "apps.parsers.registry_snapshots", level="ERROR" + ), self.captureOnCommitCallbacks(execute=True): + artifact, result = self.import_sme() + artifact.refresh_from_db() + self.assertEqual(artifact.status, ParserSourceArtifact.Status.PUBLISHED) + self.assertEqual(OrganizationSourceRecord.objects.count(), result.published) + + def test_job_finalization_failure_rolls_back_publication(self): + self.import_sme() + before = set(OrganizationSourceRecord.objects.values_list("uid", flat=True)) + task = SimpleNamespace( + name="task", request=SimpleNamespace(id="finalize-failure") + ) + + def refresh(**kwargs): + return import_sme_support( + handle=_archive(_xml(numbers=("replacement",))), + original_name="next.zip", + snapshot_date="2026-08-15", + **kwargs, + ) + + with self.assertRaises(RuntimeError), patch.object( + BackgroundJob, "complete", side_effect=RuntimeError("finalization") + ): + _run_snapshot( + task, source=SME_SOURCE, refresh=refresh, requested_by_id=None + ) + self.assertEqual( + set(OrganizationSourceRecord.objects.values_list("uid", flat=True)), before + ) + self.assertEqual( + BackgroundJob.objects.get(task_id="finalize-failure").status, + JobStatus.FAILURE, + ) + self.assertEqual( + ParserLoadLog.objects.get().status, ParserLoadLog.Status.FAILED + ) + + def test_revoked_job_cannot_replace_previous_published_snapshot(self): + self.import_sme() + before = set(OrganizationSourceRecord.objects.values_list("uid", flat=True)) + task = SimpleNamespace( + name="task", request=SimpleNamespace(id="revoked-import") + ) + + def refresh(**kwargs): + BackgroundJob.objects.filter(task_id="revoked-import").update( + status=JobStatus.REVOKED + ) + return import_sme_support( + handle=_archive(_xml(numbers=("replacement",))), + original_name="next.zip", + snapshot_date="2026-08-15", + **kwargs, + ) + + with self.assertRaisesMessage( + SnapshotValidationError, "snapshot_job_no_longer_active" + ): + _run_snapshot( + task, source=SME_SOURCE, refresh=refresh, requested_by_id=None + ) + self.assertEqual( + set(OrganizationSourceRecord.objects.values_list("uid", flat=True)), before + ) + self.assertEqual( + BackgroundJob.objects.get(task_id="revoked-import").status, + JobStatus.REVOKED, + ) + self.assertEqual( + ParserLoadLog.objects.get().status, ParserLoadLog.Status.FAILED + ) + + def test_opk_card_list_dashboard_counts_share_scope_while_export_keeps_own(self): + Organization.objects.create( + name="Справочная организация", + inn="5678901234", + ogrn="1027700132195", + okpo="87654321", + directory_imported_at=timezone.now(), + opk_registry_membership=False, + ) + root = etree.fromstring(_xml()) # noqa: S320 - locally generated fixture + other = etree.fromstring(_xml(numbers=("non-opk",), inn="5678901234")) # noqa: S320 + root.append(other.find("Документ")) + root.set("КолДок", "2") + _, result = self.import_sme(etree.tostring(root, encoding="UTF-8")) + self.assertEqual(result.published, 3) + client = APIClient() + client.force_authenticate(UserFactory.create_user()) + card = client.get("/api/v1/sources/sme-support-recipients-registry/") + records = client.get( + "/api/v2/organization-source-records/", {"source": SME_SOURCE} + ) + dashboard = client.get("/api/v1/parsers/dashboard/") + self.assertEqual( + (card.status_code, records.status_code, dashboard.status_code), + (200, 200, 200), + ) + self.assertEqual(card.data["data"]["records_count"], 2) + self.assertEqual(records.data["meta"]["pagination"]["total_count"], 2) + self.assertEqual(dashboard.data["data"]["source_counts"][SME_SOURCE], 2) + self.assertEqual( + _source_group_queryset("government_support", export_year=2026).count(), 3 + ) + + def test_list_omits_nested_detail_but_detail_and_export_keep_all_years(self): + self.import_sme() + record = OrganizationSourceRecord.objects.first() + self.assertNotIn( + "violations", OrganizationSourceRecordListSerializer(record).data["payload"] + ) + self.assertEqual( + len( + OrganizationSourceRecordSerializer(record).data["payload"]["violations"] + ), + 1, + ) + self.assertEqual( + _source_group_queryset("government_support", export_year=2026).count(), 2 + ) + + def test_budget_scans_foreign_without_detail_or_raw_retention(self): + own, foreign = _budget_record(), _budget_record("2", inn="5678901234") + calls = [] + + def page(_session, params): + calls.append(params) + if params.get("filterid"): + self.assertEqual(params["filterid"], "1") + return {"data": [own]} + return { + "data": [own, foreign], + "recordCount": 2, + "version": "10", + "pageNum": 1, + } + + with patch("apps.parsers.budget_registry.budget_page", side_effect=page): + artifact, result = refresh_budget_registry(load_batch=1, session=object()) + self.assertEqual((result.parsed, result.published, result.skipped), (2, 1, 1)) + self.assertEqual(calls[0]["blocks"], "info") + self.assertEqual(len(calls), 2) + self.assertEqual( + ParserStagedRecord.objects.filter(artifact=artifact).count(), 1 + ) + with artifact.file.open("rb") as handle: + raw = handle.read().decode() + self.assertNotIn("5678901234", raw) + record = OrganizationSourceRecord.objects.get(source=BUDGET_SOURCE) + self.assertEqual(record.status, "inactive") + self.assertEqual( + record.payload["upstream"]["unknown_new_block"], [{"raw": "preserved"}] + ) + self.assertEqual( + _source_group_queryset("budget_process_registry", export_year=2026).count(), + 1, + ) + self.assertEqual(Organization.objects.count(), 1) + + def test_budget_incomplete_second_page_preserves_published_snapshot(self): + own = _budget_record() + first = {"data": [own], "recordCount": 1, "version": "10", "pageNum": 1} + with patch( + "apps.parsers.budget_registry.budget_page", + side_effect=[first, {"data": [own]}], + ): + refresh_budget_registry(load_batch=1, session=object()) + before = OrganizationSourceRecord.objects.get().uid + first["recordCount"] = 2 + with patch( + "apps.parsers.budget_registry.budget_page", + side_effect=[ + first, + {"data": [own]}, + {"data": [], "recordCount": 2, "version": "10", "pageNum": 2}, + ], + ), self.assertRaises(SnapshotValidationError): + refresh_budget_registry(load_batch=2, session=object()) + self.assertEqual(OrganizationSourceRecord.objects.get().uid, before) + + def test_budget_unknown_status_stays_unknown(self): + own = _budget_record(status="1") + with patch( + "apps.parsers.budget_registry.budget_page", + side_effect=[ + {"data": [own], "recordCount": 1, "version": "10"}, + {"data": [own]}, + ], + ): + refresh_budget_registry(load_batch=1, session=object()) + self.assertEqual(OrganizationSourceRecord.objects.get().status, "unknown") + + +class RegistryTaskLifecycleTest(TestCase): + def test_revoked_after_claim_does_not_attach_late_batch_metadata(self): + task = SimpleNamespace(name="task", request=SimpleNamespace(id="revoked-claim")) + original = ParserLoadLogService.create_load_log_with_next_batch_id + + def create_log(**kwargs): + value = original(**kwargs) + BackgroundJob.objects.filter(task_id="revoked-claim").update( + status=JobStatus.REVOKED + ) + return value + + with self.assertRaisesMessage( + SnapshotValidationError, "snapshot_job_no_longer_active" + ), patch.object( + ParserLoadLogService, + "create_load_log_with_next_batch_id", + side_effect=create_log, + ), patch( + "apps.parsers.tasks_registry_snapshots.refresh_budget_registry" + ) as refresh: + _run_snapshot( + task, source=BUDGET_SOURCE, refresh=refresh, requested_by_id=None + ) + refresh.assert_not_called() + job = BackgroundJob.objects.get(task_id="revoked-claim") + self.assertEqual(job.status, JobStatus.REVOKED) + self.assertNotIn("batch_id", job.meta) + + def test_validation_failure_reports_bounded_reason(self): + task = SimpleNamespace(name="task", request=SimpleNamespace(id="invalid-page")) + with self.assertRaises(SnapshotValidationError), patch( + "apps.parsers.tasks_registry_snapshots.refresh_budget_registry", + side_effect=SnapshotValidationError("incomplete_budget_snapshot"), + ) as refresh: + _run_snapshot( + task, source=BUDGET_SOURCE, refresh=refresh, requested_by_id=None + ) + job = BackgroundJob.objects.get(task_id="invalid-page") + self.assertIn("incomplete_budget_snapshot", job.error) + + def test_duplicate_completed_delivery_returns_result_without_new_batch(self): + job = BackgroundJob.objects.create( + task_id="delivery", + task_name="task", + status=JobStatus.SUCCESS, + result={"published": 5}, + ) + task = SimpleNamespace(name="task", request=SimpleNamespace(id=job.task_id)) + with patch( + "apps.parsers.tasks_registry_snapshots.refresh_sme_support" + ) as refresh: + self.assertEqual( + _run_snapshot( + task, source=SME_SOURCE, refresh=refresh, requested_by_id=None + ), + {"published": 5}, + ) + refresh.assert_not_called() + self.assertFalse(ParserLoadLog.objects.exists()) + + def test_duplicate_started_delivery_does_not_repeat_import(self): + BackgroundJob.objects.create( + task_id="delivery", task_name="task", status=JobStatus.STARTED + ) + task = SimpleNamespace(name="task", request=SimpleNamespace(id="delivery")) + with patch( + "apps.parsers.tasks_registry_snapshots.refresh_sme_support" + ) as refresh: + result = _run_snapshot( + task, source=SME_SOURCE, refresh=refresh, requested_by_id=None + ) + self.assertTrue(result["duplicate_delivery"]) + refresh.assert_not_called() + self.assertFalse(ParserLoadLog.objects.exists()) + + def test_sequence_failure_marks_precreated_job_failed(self): + job = BackgroundJob.objects.create(task_id="delivery", task_name="task") + task = SimpleNamespace(name="task", request=SimpleNamespace(id=job.task_id)) + with self.assertRaises(RuntimeError), patch( + "apps.parsers.tasks_registry_snapshots.ParserLoadLogService.create_load_log_with_next_batch_id", + side_effect=RuntimeError("sequence"), + ): + _run_snapshot( + task, + source=SME_SOURCE, + refresh=lambda **_: None, + requested_by_id=None, + ) + job.refresh_from_db() + self.assertEqual(job.status, JobStatus.FAILURE) + + +def test_sme_discovery_selects_latest_official_snapshot(): + url, snapshot = discover_sme_archive( + b'oldnew' + ) + assert snapshot == "2026-08-15" + assert "data-20260815" in url + + +def test_sme_stream_releases_each_measure_before_reading_next_recipient(): + class BoundedReader(BytesIO): + def read(self, size=-1): + assert 0 < size <= 65536 + return super().read(size) + + counts = Counter() + stream = iter_sme_measures( + BoundedReader(_xml(numbers=tuple(f"measure-{i}" for i in range(1200)))), counts + ) + _, recipient, previous = next(stream) + assert recipient["ИННЮЛ"] == "1234567890" + _, recipient, current = next(stream) + assert len(previous) == 0 + assert not previous.attrib + assert current.attrib["НомерПод"] == "measure-1" + remaining = 0 + for _, recipient, _ in stream: + assert recipient["ИННЮЛ"] == "1234567890" + remaining += 1 + assert remaining == 1198 + assert counts == {"documents": 1, "measures": 1200, "declared_documents": 1} diff --git a/tests/apps/parsers/test_snapshot_refresh_api.py b/tests/apps/parsers/test_snapshot_refresh_api.py new file mode 100644 index 0000000..26f32ae --- /dev/null +++ b/tests/apps/parsers/test_snapshot_refresh_api.py @@ -0,0 +1,97 @@ +"""Оба публичных входа запускают один импорт и исключают конкурирующий запуск.""" + +from importlib import import_module +from unittest.mock import patch + +import pytest +from apps.core.models import BackgroundJob +from core.celery import app as celery_app +from django.urls import reverse +from rest_framework.test import APIClient + +from tests.apps.user.factories import UserFactory + +SOURCES = [ + ("budget-process-registry", "budget_ubpandnubp", "parse_budget_registry"), + ( + "sme-support-recipients-registry", + "fns_sme_support_recipients", + "parse_sme_support_recipients", + ), +] + + +@pytest.mark.django_db +def test_openapi_exposes_refresh_conflict_and_pollable_task(): + client = APIClient() + client.force_authenticate(UserFactory.create_user(is_staff=True)) + response = client.get(reverse("schema-swagger-ui"), {"format": "openapi"}) + assert response.status_code == 200 + schema = response.data + path = next(path for path in schema["paths"] if "/parsers/run/" in path) + responses = schema["paths"][path]["post"]["responses"] + assert "409" in responses + assert responses["202"]["schema"]["$ref"].endswith("/ParserRunEnvelope") + assert "task_id" in schema["definitions"]["ParserRunResponse"]["properties"] + card_path = next( + path for path in schema["paths"] if "/sources/{slug}/refresh/" in path + ) + assert "409" in schema["paths"][card_path]["post"]["responses"] + + +@pytest.mark.django_db +@pytest.mark.parametrize("slug,source,task", SOURCES) +@pytest.mark.parametrize("first", ["card", "parser"]) +def test_refresh_is_pollable_and_other_entrypoint_conflicts(slug, source, task, first): + client = APIClient() + client.force_authenticate(UserFactory.create_user(is_staff=True)) + urls = { + "card": f"/api/v1/sources/{slug}/refresh/", + "parser": f"/api/v1/parsers/run/{source}/", + } + second = "parser" if first == "card" else "card" + import_module("apps.parsers.tasks") + registered_task = celery_app.tasks[f"parsers.{source}.refresh"] + with patch.object(registered_task, "apply_async") as dispatch: + response = client.post(urls[first], {"params": {}}, format="json") + assert response.status_code == 202, response.data + payload = response.data.get("data", response.data) + job = BackgroundJob.objects.get(task_id=payload["task_id"]) + assert payload["status"] == "queued" + assert dispatch.call_args.kwargs["task_id"] == job.task_id + assert dispatch.call_args.kwargs["kwargs"] == {"requested_by_id": job.user_id} + assert job.meta["refresh_task_ids"] == [job.task_id] + assert client.get(f"/api/v1/jobs/{job.task_id}/").status_code == 200 + duplicate = client.post(urls[second], {}, format="json") + assert duplicate.status_code == 409, duplicate.data + assert BackgroundJob.objects.count() == 1 + assert dispatch.call_count == 1 + job.complete() + next_run = client.post(urls[second], {}, format="json") + assert next_run.status_code == 202 + assert BackgroundJob.objects.count() == 2 + + +@pytest.mark.django_db +@pytest.mark.parametrize("slug,source,task", SOURCES) +def test_refresh_requires_administrator_and_rejects_loader_parameters( + slug, source, task +): + client = APIClient() + client.force_authenticate(UserFactory.create_user()) + import_module("apps.parsers.tasks") + registered_task = celery_app.tasks[f"parsers.{source}.refresh"] + with patch.object(registered_task, "apply_async") as dispatch: + for url in [ + f"/api/v1/sources/{slug}/refresh/", + f"/api/v1/parsers/run/{source}/", + ]: + assert client.post(url, {}, format="json").status_code == 403 + client.force_authenticate(UserFactory.create_user(is_staff=True)) + response = client.post( + f"/api/v1/sources/{slug}/refresh/", + {"params": {"url": "https://example.invalid"}}, + format="json", + ) + assert response.status_code == 400 + dispatch.assert_not_called() diff --git a/tests/apps/parsers/test_source_cards_service.py b/tests/apps/parsers/test_source_cards_service.py index 12f5660..a85cbff 100644 --- a/tests/apps/parsers/test_source_cards_service.py +++ b/tests/apps/parsers/test_source_cards_service.py @@ -15,7 +15,7 @@ from apps.parsers.source_cards import ( ) from django.db import connection from django.http import Http404 -from django.test import SimpleTestCase, TestCase, override_settings +from django.test import TestCase, override_settings from django.test.utils import CaptureQueriesContext from django.utils import timezone from organizations.models import Organization @@ -75,7 +75,7 @@ def _save_source_record( ) -class SourceCardServiceUnitTest(SimpleTestCase): +class SourceCardServiceUnitTest(TestCase): def test_list_cards_exposes_all_frontend_category_slugs_in_menu_order(self): self.assertEqual( [card.slug for card in SOURCE_CARD_DEFINITIONS], @@ -92,6 +92,8 @@ class SourceCardServiceUnitTest(SimpleTestCase): "labor-vacancies", "gosedo-global-address-directory", "media-mentions", + "sme-support-recipients-registry", + "budget-process-registry", ], ) self.assertEqual( @@ -109,6 +111,8 @@ class SourceCardServiceUnitTest(SimpleTestCase): "Вакансии Работа России", "ГосЭДО: Глобальный адресный справочник", "Новости СМИ", + "Реестр субъектов МСП — получателей поддержки", + "Реестр участников бюджетного процесса", ], ) @@ -331,16 +335,11 @@ class SourceCardServiceUnitTest(SimpleTestCase): "Обновление для карточки не поддерживается", str(error.exception.detail) ) - def test_enqueue_task_deletes_background_job_on_async_error(self): + def test_enqueue_task_preserves_failure_for_polling_on_async_error(self): task = MagicMock() task.apply_async.side_effect = RuntimeError("broker down") - queryset = MagicMock() - with patch( "apps.parsers.source_cards.uuid.uuid4", return_value="task-id-1" - ), patch("apps.parsers.source_cards.BackgroundJobService.create_job"), patch( - "apps.parsers.source_cards.BackgroundJobService.get_queryset", - return_value=queryset, ), self.assertRaisesMessage(RuntimeError, "broker down"): SourceCardService._enqueue_task( task=task, @@ -350,8 +349,9 @@ class SourceCardServiceUnitTest(SimpleTestCase): kwargs={"region_code": "77"}, ) - queryset.filter.assert_called_once_with(task_id="task-id-1") - queryset.filter.return_value.delete.assert_called_once_with() + job = BackgroundJob.objects.get(task_id="task-id-1") + self.assertEqual(job.status, JobStatus.FAILURE) + self.assertTrue(job.error) def test_helper_methods_cover_unknown_codes_and_status_variants(self): self.assertEqual(SourceCardService._get_source_records_count("unknown"), 0) diff --git a/tests/apps/parsers/test_source_cards_views.py b/tests/apps/parsers/test_source_cards_views.py index 79369bf..28e06a5 100644 --- a/tests/apps/parsers/test_source_cards_views.py +++ b/tests/apps/parsers/test_source_cards_views.py @@ -275,7 +275,7 @@ class SourceCardsApiTestCase(APITestCase): with patch( "apps.parsers.tasks.parse_registry_enrichment_sources.apply_async", return_value=SimpleNamespace(id="task-procurements"), - ): + ) as task_mock: response = self.client.post( reverse( "api_v1:sources:source-cards-refresh", @@ -286,7 +286,9 @@ class SourceCardsApiTestCase(APITestCase): ) self.assertEqual(response.status_code, status.HTTP_202_ACCEPTED) - self.assertEqual(response.data["task_id"], "task-procurements") + self.assertEqual( + response.data["task_id"], task_mock.call_args.kwargs["task_id"] + ) def test_refresh_forbidden_for_regular_user(self): response = self.client.post( diff --git a/tests/apps/parsers/test_sources_api_e2e.py b/tests/apps/parsers/test_sources_api_e2e.py index 73eb514..2552e47 100644 --- a/tests/apps/parsers/test_sources_api_e2e.py +++ b/tests/apps/parsers/test_sources_api_e2e.py @@ -1,6 +1,5 @@ from __future__ import annotations -from types import SimpleNamespace from unittest.mock import patch from apps.core.models import BackgroundJob, JobStatus @@ -182,25 +181,7 @@ class SourcesApiE2ETest(APITestCase): def test_refresh_endpoints_cover_multiple_source_cards(self): self.client.force_authenticate(self.admin) - with patch( - "apps.parsers.source_cards.uuid.uuid4", - side_effect=[ - "request-minprom", - "task-industrial", - "task-products", - "task-manufactures", - "request-procurements", - "task-procurements", - ], - ), patch( - "celery.app.task.Task.apply_async", - side_effect=[ - SimpleNamespace(id="task-industrial"), - SimpleNamespace(id="task-products"), - SimpleNamespace(id="task-manufactures"), - SimpleNamespace(id="task-procurements"), - ], - ): + with patch("celery.app.task.Task.apply_async") as dispatch: minprom_response = self.client.post( reverse( "api_v1:sources:source-cards-refresh", @@ -229,30 +210,34 @@ class SourcesApiE2ETest(APITestCase): set(minprom_response.data.keys()), {"task_id", "task_ids", "status"}, ) - self.assertEqual(minprom_response.data["task_id"], "task-industrial") + first_task_ids = [ + call.kwargs["task_id"] for call in dispatch.call_args_list[:3] + ] + procurement_task_id = dispatch.call_args_list[3].kwargs["task_id"] + self.assertEqual(minprom_response.data["task_id"], first_task_ids[0]) self.assertEqual( minprom_response.data["task_ids"], - ["task-industrial", "task-products", "task-manufactures"], + first_task_ids, ) self.assertEqual( set(procurements_response.data.keys()), {"task_id", "task_ids", "status"}, ) - self.assertEqual(procurements_response.data["task_id"], "task-procurements") + self.assertEqual(procurements_response.data["task_id"], procurement_task_id) self.assertEqual( procurements_response.data["task_ids"], - ["task-procurements"], + [procurement_task_id], ) self.assertEqual( BackgroundJob.objects.filter( - task_id__in=["task-industrial", "task-products", "task-manufactures"], + task_id__in=first_task_ids, user_id=self.admin.id, ).count(), 3, ) self.assertTrue( BackgroundJob.objects.filter( - task_id="task-procurements", + task_id=procurement_task_id, task_name="apps.parsers.tasks.parse_registry_enrichment_sources", user_id=self.admin.id, ).exists() diff --git a/tests/apps/parsers/test_system_logs_checkup.py b/tests/apps/parsers/test_system_logs_checkup.py new file mode 100644 index 0000000..41518fe --- /dev/null +++ b/tests/apps/parsers/test_system_logs_checkup.py @@ -0,0 +1,190 @@ +"""Regression coverage for the routed update-history list and Excel export.""" + +import csv +from datetime import UTC, datetime +from io import StringIO +from unittest.mock import patch +from urllib.parse import parse_qs, urlsplit + +from apps.parsers import views +from apps.parsers.models import ParserLoadLog +from django.test import override_settings +from django.urls import reverse +from rest_framework.test import APITestCase + +from tests.apps.parsers.factories import ParserLoadLogFactory +from tests.apps.user.factories import UserFactory + + +@override_settings(TIME_ZONE="UTC") +class SystemLogsCheckupTest(APITestCase): + def setUp(self): + self.client.force_authenticate(UserFactory.create_superuser()) + self.list_url = reverse("api_v1:system:parser-logs-list") + self.export_url = reverse("api_v1:system:parser-logs-export") + + @staticmethod + def create_log(updated_at, **kwargs): + kwargs.setdefault("source", "inspections") + log = ParserLoadLogFactory(**kwargs) + ParserLoadLog.objects.filter(pk=log.pk).update(updated_at=updated_at) + return log + + def csv_rows(self, params): + response = self.client.get(self.export_url, params) + self.assertEqual(response.status_code, 200) + return list( + csv.reader(StringIO(response.content.decode("utf-8-sig")), delimiter=";") + ) + + def test_date_filter_is_inclusive_and_single_start_means_one_day(self): + self.create_log(datetime(2026, 9, 6, 23, 59, 59, tzinfo=UTC)) + first = self.create_log(datetime(2026, 9, 7, tzinfo=UTC)) + last = self.create_log(datetime(2026, 9, 7, 23, 59, 59, 999999, tzinfo=UTC)) + self.create_log(datetime(2026, 9, 8, tzinfo=UTC)) + params = {"date_from": "2026-09-07", "ordering": "updated_at", "page_size": 1} + response = self.client.get(self.list_url, params) + self.assertEqual(response.status_code, 200) + self.assertEqual(response.data["count"], 2) + self.assertEqual(response.data["results"][0]["id"], first.id) + self.assertIsNone(response.data["previous"]) + next_query = parse_qs(urlsplit(response.data["next"]).query) + self.assertEqual(next_query["date_from"], ["2026-09-07"]) + self.assertEqual(next_query["page"], ["2"]) + page_two = self.client.get(self.list_url, {**params, "page": 2}) + self.assertEqual(page_two.data["results"][0]["id"], last.id) + self.assertIsNone(page_two.data["next"]) + self.assertIsNotNone(page_two.data["previous"]) + self.assertEqual( + [int(row[0]) for row in self.csv_rows(params)[1:]], [first.id, last.id] + ) + + def test_period_combines_with_source_status_and_search(self): + moment = datetime(2026, 9, 7, 12, tzinfo=UTC) + expected = self.create_log( + moment, source="manufactures", status="failed", error_message="маркер" + ) + self.create_log( + moment, source="manufactures", status="success", error_message="маркер" + ) + self.create_log( + moment, source="inspections", status="failed", error_message="маркер" + ) + self.create_log( + moment, source="manufactures", status="failed", error_message="другое" + ) + params = { + "date_from": "2026-09-01", + "date_to": "2026-09-07", + "source": "manufacturers-and-products", + "status": "failed", + "search": "маркер", + } + response = self.client.get(self.list_url, params) + self.assertEqual(response.status_code, 200) + self.assertEqual(response.data["count"], 1) + self.assertEqual(response.data["results"][0]["id"], expected.id) + self.assertEqual( + [int(row[0]) for row in self.csv_rows(params)[1:]], [expected.id] + ) + + def test_date_validation_matches_between_list_and_export(self): + for params in ( + {"date_from": "2026-02-30"}, + {"date_to": "07.09.2026"}, + {"date_from": "20260907"}, + {"date_from": "2026-09-08", "date_to": "2026-09-07"}, + {"ordering": "unknown"}, + {"ordering": "--source"}, + {"batch_id": "invalid"}, + ): + for url in (self.list_url, self.export_url): + with self.subTest(params=params, url=url): + self.assertEqual(self.client.get(url, params).status_code, 400) + + def test_ordering_is_global_stable_and_enriches_only_the_page(self): + moment = datetime(2026, 9, 7, tzinfo=UTC) + middle = self.create_log(moment, records_count=20) + low = self.create_log(moment, records_count=10) + tie = self.create_log(moment, records_count=20) + high = self.create_log(moment, records_count=30) + for ordering, expected in ( + ("records_count", [low.id, middle.id, tie.id, high.id]), + ("-records_count", [high.id, middle.id, tie.id, low.id]), + ): + ids = [] + for page in (1, 2): + with patch.object( + views, + "_serialize_parser_log_row", + wraps=views._serialize_parser_log_row, + ) as serialize: + response = self.client.get( + self.list_url, + {"ordering": ordering, "page_size": 2, "page": page}, + ) + self.assertEqual(response.status_code, 200) + self.assertEqual(response.data["count"], 4) + self.assertEqual(serialize.call_count, 2) + ids.extend(row["id"] for row in response.data["results"]) + self.assertEqual(ids, expected) + self.assertEqual( + [int(row[0]) for row in self.csv_rows({"ordering": ordering})[1:]], + expected, + ) + + def test_sort_by_source_uses_public_slug_consistently_with_export(self): + moment = datetime(2026, 9, 7, tzinfo=UTC) + manufacture = self.create_log(moment, source="manufactures") + industrial = self.create_log(moment, source="industrial") + report = self.create_log(moment, source="fns_reports") + for ordering, expected in ( + ("source", [report.id, manufacture.id, industrial.id]), + ("-source", [manufacture.id, industrial.id, report.id]), + ): + response = self.client.get(self.list_url, {"ordering": ordering}) + self.assertEqual(response.status_code, 200) + self.assertEqual([row["id"] for row in response.data["results"]], expected) + self.assertEqual( + [int(row[0]) for row in self.csv_rows({"ordering": ordering})[1:]], + expected, + ) + + def test_csv_preserves_cyrillic_delimiters_quotes_and_newlines(self): + message = 'Ошибка; "детали"\r\nСледующая строка' + self.create_log( + datetime(2026, 9, 7, 12, 34, tzinfo=UTC), + status="failed", + error_message=message, + ) + response = self.client.get(self.export_url) + self.assertEqual(response.status_code, 200) + self.assertEqual(response["Content-Type"], "text/csv; charset=utf-8") + self.assertEqual( + response["Content-Disposition"], 'attachment; filename="update-history.csv"' + ) + self.assertTrue(response.content.startswith(b"\xef\xbb\xbf")) + decoded = response.content.decode("utf-8-sig") + self.assertNotIn("\n", decoded.replace("\r\n", "")) + rows = list(csv.reader(StringIO(decoded), delimiter=";")) + self.assertEqual( + rows[0][:5], + ["№", "Дата актуализации", "Источник", "Статус", "Количество записей"], + ) + self.assertEqual(rows[1][1], "07.09.2026 12:34") + self.assertEqual(rows[1][3], "Ошибка") + self.assertEqual(rows[1][rows[0].index("Ошибка")], message) + + def test_openapi_documents_period_for_list_and_export(self): + response = self.client.get(reverse("schema-swagger-ui"), {"format": "openapi"}) + self.assertEqual(response.status_code, 200) + paths = response.data["paths"] + for suffix in ("/system/logs/", "/system/logs/export/"): + path = next(path for path in paths if path.endswith(suffix)) + parameters = { + parameter["name"]: parameter + for parameter in paths[path]["get"]["parameters"] + } + for name in ("date_from", "date_to"): + self.assertEqual(parameters[name]["type"], "string") + self.assertEqual(parameters[name]["format"], "date") diff --git a/tests/apps/parsers/test_tasks.py b/tests/apps/parsers/test_tasks.py index 817977a..4f6c8ee 100644 --- a/tests/apps/parsers/test_tasks.py +++ b/tests/apps/parsers/test_tasks.py @@ -3224,7 +3224,7 @@ class ParseVacanciesTaskTestCase(TestCase): batch_id=stopped_job.meta["batch_id"], ) self.assertEqual(stopped_result["status"], "revoked") - self.assertEqual(stopped_job.meta["next_offset"], 1) + self.assertEqual(stopped_job.meta["next_offset"], 0) self.assertEqual(stopped_load.status, ParserLoadLog.Status.SKIPPED) self.assertEqual(stopped_load.records_count, 1) self.assertEqual(OrganizationSourceRecord.objects.count(), 1) @@ -3236,7 +3236,7 @@ class ParseVacanciesTaskTestCase(TestCase): self.assertEqual(resumed_result["status"], "success") self.assertEqual(resumed_result["batch_id"], stopped_result["batch_id"]) self.assertTrue(resumed_result["resumed"]) - self.assertEqual(captured_fetches, ["7701000601", "7701000602"]) + self.assertEqual(captured_fetches, ["7701000601", "7701000601", "7701000602"]) self.assertEqual(OrganizationSourceRecord.objects.count(), 2) def test_registry_run_fails_when_trudvsem_fails_for_every_organization(self): diff --git a/tests/apps/parsers/test_vacancy_progress_race.py b/tests/apps/parsers/test_vacancy_progress_race.py new file mode 100644 index 0000000..f341e11 --- /dev/null +++ b/tests/apps/parsers/test_vacancy_progress_race.py @@ -0,0 +1,113 @@ +"""Запоздалые контрольные точки вакансий не меняют актуальное состояние задачи.""" + +from unittest.mock import patch + +import pytest +from apps.core.models import BackgroundJob +from apps.parsers.models import ParserLoadLog +from apps.parsers.tasks import ( + _find_resumable_vacancy_job, + _update_vacancy_registry_checkpoint, +) + + +def _checkpoint(job, load_log, *, processed): + with patch("apps.parsers.tasks._vacancy_batch_records_count", return_value=17): + return _update_vacancy_registry_checkpoint( + job=job, + load_log=load_log, + batch_id=load_log.batch_id, + processed=processed, + total=100, + failed=0, + targets_signature="test-targets", + ) + + +@pytest.fixture +def vacancy_run(db): + job = BackgroundJob.objects.create( + task_id="vacancy-checkpoint", + task_name="apps.parsers.tasks.parse_trudvsem_vacancies", + meta={"source_card": "labor-vacancies"}, + ) + load_log = ParserLoadLog.objects.create( + source=ParserLoadLog.Source.TRUDVSEM, + batch_id=1, + status=ParserLoadLog.Status.IN_PROGRESS, + ) + return job, load_log + + +def test_lower_stale_vacancy_checkpoint_preserves_progress_message_and_meta( + vacancy_run, +): + job, load_log = vacancy_run + stale = BackgroundJob.objects.get(pk=job.pk) + assert _checkpoint(job, load_log, processed=75) == 17 + expected = (job.progress, job.progress_message, job.meta) + + assert _checkpoint(stale, load_log, processed=25) == 17 + + job.refresh_from_db() + assert (job.progress, job.progress_message, job.meta) == expected + assert job.meta["next_offset"] == 75 + assert job.meta["source_card"] == "labor-vacancies" + load_log.refresh_from_db() + assert load_log.records_count == 17 + assert load_log.status == ParserLoadLog.Status.IN_PROGRESS + + +@pytest.mark.parametrize("terminal", ["complete", "fail", "revoke"]) +def test_terminal_vacancy_job_ignores_late_checkpoint(vacancy_run, terminal): + job, load_log = vacancy_run + _checkpoint(job, load_log, processed=65) + stale = BackgroundJob.objects.get(pk=job.pk) + if terminal == "fail": + job.fail("Ошибка источника") + else: + getattr(job, terminal)() + expected = ( + job.status, + job.progress, + job.progress_message, + job.meta, + job.completed_at, + ) + + assert _checkpoint(stale, load_log, processed=90) == 17 + + job.refresh_from_db() + assert ( + job.status, + job.progress, + job.progress_message, + job.meta, + job.completed_at, + ) == expected + + +@pytest.mark.parametrize("saved_records", [0, 17]) +def test_zero_cursor_resumes_only_when_batch_has_saved_records( + vacancy_run, saved_records +): + job, load_log = vacancy_run + job.meta.update( + batch_id=load_log.batch_id, + next_offset=0, + total_organizations=100, + targets_signature="test-targets", + ) + job.save(update_fields=["meta"]) + job.revoke() + + with patch( + "apps.parsers.tasks._vacancy_batch_records_count", return_value=saved_records + ): + result = _find_resumable_vacancy_job( + exclude_task_id="new-run", + targets_signature="test-targets", + targets_count=100, + ) + + assert result == ((job, load_log) if saved_records else (None, None)) diff --git a/tests/apps/parsers/test_views.py b/tests/apps/parsers/test_views.py index f3836a0..85c21b1 100644 --- a/tests/apps/parsers/test_views.py +++ b/tests/apps/parsers/test_views.py @@ -1241,7 +1241,7 @@ class ParsersViewSetTest(APITestCase): self.assertEqual(response.status_code, status.HTTP_200_OK) self.assertEqual(response["Content-Type"], "text/csv; charset=utf-8") content = response.content.decode("utf-8") - self.assertIn("organizations_count", content) + self.assertIn("Количество организаций", content) self.assertIn("333", content) def test_system_logs_support_search_by_source_label(self): diff --git a/uv.lock b/uv.lock index 2ecb86e..e1debd6 100644 --- a/uv.lock +++ b/uv.lock @@ -1397,6 +1397,7 @@ dependencies = [ { name = "drf-yasg" }, { name = "factory-boy" }, { name = "faker" }, + { name = "lxml" }, { name = "model-bakery" }, { name = "numpy" }, { name = "openpyxl" }, @@ -1528,6 +1529,7 @@ requires-dist = [ { name = "gevent", marker = "extra == 'dev'", specifier = "==23.9.1" }, { name = "gunicorn", marker = "extra == 'dev'", specifier = "==21.2.0" }, { name = "isort", marker = "extra == 'dev'", specifier = "==5.13.2" }, + { name = "lxml", specifier = "==6.0.2" }, { name = "model-bakery", specifier = ">=1.17.0" }, { name = "numpy", specifier = "==1.24.4" }, { name = "openpyxl", specifier = ">=3.1.5" },