diff --git a/docs/source-export-catalog-operations.md b/docs/source-export-catalog-operations.md index 7200435..9493558 100644 --- a/docs/source-export-catalog-operations.md +++ b/docs/source-export-catalog-operations.md @@ -52,8 +52,8 @@ Django settings (при необходимости задать в настро | Setting | Default | | --- | --- | -| `SOURCE_EXPORT_TARGET_FILE_BYTES` | 256 MiB | -| `SOURCE_EXPORT_MAX_FILE_BYTES` | 512 MiB | +| `SOURCE_EXPORT_TARGET_FILE_BYTES` | 256 MiB; для v5 ограничивается также ZIP cap 99 000 000 bytes | +| `SOURCE_EXPORT_MAX_FILE_BYTES` | 512 MiB; для v5 каждый файл должен помещаться в ZIP 99 000 000 bytes с manifest | | `SOURCE_EXPORT_ROWS_PER_PART` | 100000 | | `SOURCE_RECORD_EXPORT_XLSX_ROWS_PER_FILE` | 100000, совместимый верхний предел | | `SOURCE_EXPORT_MIN_FREE_BYTES` | 1 GiB, резерв проверяется между batch/render | @@ -147,3 +147,17 @@ SOURCE_EXPORT_TRANSPORT_LOAD=1 SOURCE_EXPORT_TRANSPORT_REPORT=/tmp/source-export У Orval/Zod для export-ticket включён strict mode: иначе legacy вариант union с тремя полями способен принять новый ответ и silently strip selection fields. Generated файлы не патчатся. Canonical timestamps публикуются в UTC с `Z`, поддерживаемым generated validator. + + +## Окно лет и готовые ZIP (manifest v5) + +После migrate `0023_source_export_settings` в Django admin доступна singleton-настройка +«Глубина выгрузки, лет» (по умолчанию 5). Применяется при следующей сборке, без рестарта. +Подробный контракт и исключение старых/будущих/неопределимых годов: +[source-export-year-partitions.md](source-export-year-partitions.md). + +Сборка дополнительно сохраняет готовые ZIP каждого формата до 99 000 000 байт. +При расчёте диска учитывать и исходные CSV/XLSX/JSON части, и ZIP_STORED-копии их содержимого, +а также staging и retained generations. Все файлы остаются в приватном generation directory. +`--verify-only` проверяет размеры и SHA-256 как частей, так и ZIP. Cleanup удаляет их +вместе с поколением, защищённым действующими leases. Ночная задача и расписание не меняются. diff --git a/docs/source-export-window-verification.md b/docs/source-export-window-verification.md new file mode 100644 index 0000000..ef3a1b1 --- /dev/null +++ b/docs/source-export-window-verification.md @@ -0,0 +1,47 @@ +# Окно лет и ZIP до 99 МБ: проверка 2026-11-03 + +Локальная реализация в backend `dev`. Фронт, dev-сервер и ночные задания на сервере +не изменялись; push/выкладка и первая сборка реальных данных в этот этап не входят. + +## Проверки + +- Основной backend-набор `PYTHONPATH=src uv run pytest -q --tb=short`: + **2102 passed, 49 skipped**, exit 0. После него добавлены ещё две проверки каталога + и параметризация PostgreSQL race-теста; окончательная целевая проверка ниже включает их. +- Окончательный каталог: **54 passed**, exit 0. +- Изолированный PostgreSQL 15 с настоящими миграциями, каталог + snapshot contract + + concurrent tickets: **74 passed**, exit 0. В том числе схема/seed `0023`, repeatable-read, + публикация при ingestion и два конкурентных native POST одного ticket — для обычного + файла и для нового готового ZIP. Тестовый контейнер удалён после прогона. +- Canonical OpenAPI build и `--check`, Orval generation, TypeScript `type-check` — exit 0. + Generated Zod: **5 реальных API-ответов** без потери полей, включая новый archives detail + и ticket готового ZIP. +- Ruff, format check, Django system check, `makemigrations --check --dry-run` и + `git diff --check` — exit 0. + +Полный набор включает пропуски opt-in transport load и PostgreSQL-only тестов на SQLite; +они не считаются PASS. PostgreSQL-only проверки выполнены отдельно. Длительная нагрузка +через публичный proxy и browser/UI в этом этапе не запускались. + +## Проверенные сценарии + +- Окно 2021–2026, изменение глубины из БД, смена календарного года, created_at fallback, + исключение старых/будущих/неопределимых лет без удаления исходных записей. +- Финансовые проекции содержат только свой год; unique source count отличается от + количества включённых годовых проекций. +- Самостоятельный ZIP **ровно 99 000 000 байт** с реальными заголовками и manifest; + превышение на один байт отклоняется. Использован локальный sparse synthetic fixture. +- Несколько архивов, несколько лет в одном архиве, большой год, согласованные варианты + CSV/XLSX/JSON, CRC/SHA-256 и отсутствие потерь/дубликатов. +- Несколько уровней рекурсивного разбиения с разными временными spool paths: исправлена + коллизия имён `.left`/`.right` при повторном делении. +- Слишком большая одиночная запись: понятная ошибка, прежний current сохраняется, + staging удаляется. Изменение настройки во время render не меняет captured policy. +- archive_id обязателен для v5; неизвестный ID и несовпадающий формат отклоняются. + Tickets закреплены за поколением; v3/v4 и прежняя streaming ZIP схема доступны. +- Verify обнаруживает повреждение готового ZIP; retention/leases и ночной dispatch всех + 22 источников продолжают работать по существующим правилам. + +Эксплуатация и контракт: [годовые выгрузки](source-export-year-partitions.md), +[операции](source-export-catalog-operations.md). Готовые ZIP_STORED дополнительно занимают +место рядом с исходными частями; это нужно учитывать перед будущей серверной сборкой. diff --git a/docs/source-export-year-partitions.md b/docs/source-export-year-partitions.md index ba9b8eb..63eb2e4 100644 --- a/docs/source-export-year-partitions.md +++ b/docs/source-export-year-partitions.md @@ -3,6 +3,48 @@ Уточнение пользователя от 2026-10-22 заменяет правило единого набора за все годы в исходном [задании](source-data-export-api-spec.md). Фронтенд не изменяется. +## Изменение схемы от 2026-11-03 + +Для новых поколений manifest v5 действует окно годов и несколько самостоятельных ZIP. +В админке **Организации → Настройки выгрузки источников** поле «Глубина выгрузки, лет» +по умолчанию равно 5. Включаются годы от текущего минус глубина до текущего включительно +(2021–2026 в 2026 году); 0 означает только текущий год. Текущий год определяется в TIME_ZONE +сервиса. Старые, будущие и неопределимые годы исключаются из выгрузки, исходные записи +не удаляются. Пустая дата сохраняет прежний fallback на год created_at. У финансов +отбираются проекции и строки показателей внутри этого окна; records_count считает +уникальные включённые исходные записи, exported_records_count — их годовые проекции. + +Настройка читается один раз при начале сборки источника. Параметры фиксируются в +`export_policy`: years_back, min_year, max_year, archive_max_bytes. Изменение настройки +или календарного года помечает существующее поколение stale до следующей сборки. + +Каждый ZIP одного формата содержит папки годов и manifest.json и не превышает +**99 000 000 байт** вместе со служебными данными ZIP. ZIP_STORED, самостоятельные архивы, +не многотомный ZIP. Большой год делится по границам записей; один архив может включать +несколько лет. Файлы частей согласованы по записям между форматами. Финансы — JSON-only. +Размер архива проверяется перед атомарной публикацией. Слишком большая одиночная запись +даёт `source_export_record_too_large`, прежнее current поколение сохраняется. + +Detail API дополнен `export_policy` (null для прежних поколений) и `archives` (пустой +список для прежних поколений). Каждый элемент содержит archive_id, artifact_id, +archive_number/archives_count (внутри формата), format, years, file_name, mime_type, +size_bytes, sha256 и part_ids. Список архивов полный, независимо от страницы files. +Для manifest v5 selector `source_archive` требует `archive_id`: + +```json +{"selection_kind":"source_archive","source_group":"government_procurements","generation_id":"","archive_id":"","format":"json"} +``` + +Отсутствие archive_id — 400, неизвестный ID — 404, неверный формат — 409. Ticket выдаёт +точный размер и имя готового ZIP; download стримит его с Content-Length. Выданный ticket +закрепляет поколение и архив; leases/retention сохраняются. Старые manifest v3/v4 и их +архивные tickets используют прежний streaming ZIP без archive_id. Отдельные file tickets +работают по прежнему selector. Public schema_version остаётся 2 с добавленными полями. +Фронт в этой задаче не меняется; клиенту для новых архивов нужен выбор archive_id. + +Далее описана исходная схема v4; правила полноты всех лет и одного ZIP относятся +к прежним поколениям. Определение года, годовые проекции и структура папок сохранены. + Ночная задача `organizations.tasks.refresh_source_export_catalog` обрабатывает весь registry из 22 источников и все доступные годы их канонических опубликованных данных. Миграция `0021_enable_yearly_partitioned_exports` включает recurring Beat schedule diff --git a/src/core/openapi.json b/src/core/openapi.json index 1f92645..9e33f7b 100644 --- a/src/core/openapi.json +++ b/src/core/openapi.json @@ -10297,6 +10297,9 @@ }, { "$ref": "#/components/schemas/SourceExportArchiveTicketRequest" + }, + { + "$ref": "#/components/schemas/SourceExportNumberedArchiveTicketRequest" } ] } @@ -25766,6 +25769,112 @@ "has_previous" ] }, + "SourceExportPolicy": { + "type": "object", + "additionalProperties": false, + "properties": { + "years_back": { + "type": "integer", + "minimum": 0, + "maximum": 9007199254740991 + }, + "min_year": { + "type": "integer", + "minimum": 1, + "maximum": 9007199254740991 + }, + "max_year": { + "type": "integer", + "minimum": 1, + "maximum": 9007199254740991 + }, + "archive_max_bytes": { + "type": "integer", + "minimum": 1, + "maximum": 9007199254740991 + } + }, + "required": [ + "years_back", + "min_year", + "max_year", + "archive_max_bytes" + ] + }, + "SourceExportArchive": { + "type": "object", + "additionalProperties": false, + "properties": { + "archive_id": { + "type": "string", + "format": "uuid" + }, + "artifact_id": { + "type": "string", + "format": "uuid" + }, + "archive_number": { + "type": "integer", + "minimum": 1, + "maximum": 9007199254740991 + }, + "archives_count": { + "type": "integer", + "minimum": 1, + "maximum": 9007199254740991 + }, + "format": { + "type": "string", + "enum": [ + "xlsx", + "csv", + "json" + ] + }, + "years": { + "type": "array", + "items": { + "type": "integer", + "minimum": 1, + "maximum": 9007199254740991 + } + }, + "file_name": { + "type": "string" + }, + "mime_type": { + "type": "string" + }, + "size_bytes": { + "type": "integer", + "minimum": 1, + "maximum": 9007199254740991 + }, + "sha256": { + "type": "string" + }, + "part_ids": { + "type": "array", + "items": { + "type": "string", + "format": "uuid" + } + } + }, + "required": [ + "archive_id", + "artifact_id", + "archive_number", + "archives_count", + "format", + "years", + "file_name", + "mime_type", + "size_bytes", + "sha256", + "part_ids" + ] + }, "SourceExportSourceDetail": { "type": "object", "additionalProperties": false, @@ -25850,6 +25959,16 @@ "format": "uuid", "nullable": true }, + "export_policy": { + "$ref": "#/components/schemas/SourceExportPolicy", + "nullable": true + }, + "archives": { + "type": "array", + "items": { + "$ref": "#/components/schemas/SourceExportArchive" + } + }, "description": { "$ref": "#/components/schemas/SourceExportDescription" }, @@ -25876,6 +25995,8 @@ "data_scope", "partitioning", "current_generation_id", + "export_policy", + "archives", "description", "datasets", "files" @@ -26082,6 +26203,68 @@ "format" ] }, + "SourceExportNumberedArchiveTicketRequest": { + "type": "object", + "additionalProperties": false, + "properties": { + "selection_kind": { + "type": "string", + "enum": [ + "source_archive" + ] + }, + "source_group": { + "type": "string", + "enum": [ + "financial_indicators", + "government_procurements", + "industrial_production", + "planned_inspections", + "bankruptcy", + "defense_suppliers", + "arbitration", + "security_registries", + "vacancies", + "electronic_document_exchange", + "media_mentions", + "sanctions", + "government_support", + "sro_membership", + "budget_process_registry", + "integrated_structure_documents", + "fedresurs_company_profile", + "enforcement_proceedings", + "stock_exchange_activity", + "leasing", + "company_messages", + "company_licenses" + ] + }, + "generation_id": { + "type": "string", + "format": "uuid" + }, + "archive_id": { + "type": "string", + "format": "uuid" + }, + "format": { + "type": "string", + "enum": [ + "xlsx", + "csv", + "json" + ] + } + }, + "required": [ + "selection_kind", + "source_group", + "generation_id", + "archive_id", + "format" + ] + }, "SourceExportSelectionTicket": { "type": "object", "additionalProperties": false, diff --git a/src/organizations/admin.py b/src/organizations/admin.py index dbf4438..d24dc97 100644 --- a/src/organizations/admin.py +++ b/src/organizations/admin.py @@ -2,7 +2,7 @@ from django.contrib import admin -from organizations.models import Organization +from organizations.models import Organization, SourceExportSettings @admin.register(Organization) @@ -20,3 +20,16 @@ class OrganizationAdmin(admin.ModelAdmin): name_short.short_description = "Организация" name_short.admin_order_field = "name" + + +@admin.register(SourceExportSettings) +class SourceExportSettingsAdmin(admin.ModelAdmin): + fields = ["years_back"] + + def has_add_permission(self, request): + return not SourceExportSettings.objects.exists() and super().has_add_permission( + request + ) + + def has_delete_permission(self, request, obj=None): + return False diff --git a/src/organizations/migrations/0023_source_export_settings.py b/src/organizations/migrations/0023_source_export_settings.py new file mode 100644 index 0000000..1ef15c5 --- /dev/null +++ b/src/organizations/migrations/0023_source_export_settings.py @@ -0,0 +1,59 @@ +# Generated by Django 3.2.25 on 2026-11-03 08:54 + +import django.core.validators +from django.db import migrations, models + + +def seed_settings(apps, schema_editor): + apps.get_model("organizations", "SourceExportSettings").objects.using( + schema_editor.connection.alias + ).get_or_create(singleton_key=1) + + +class Migration(migrations.Migration): + dependencies = [ + ("organizations", "0022_current_directory_membership"), + ] + + operations = [ + migrations.CreateModel( + name="SourceExportSettings", + fields=[ + ( + "id", + models.BigAutoField( + auto_created=True, + primary_key=True, + serialize=False, + verbose_name="ID", + ), + ), + ( + "singleton_key", + models.PositiveSmallIntegerField( + default=1, editable=False, unique=True + ), + ), + ( + "years_back", + models.PositiveSmallIntegerField( + default=5, + validators=[django.core.validators.MaxValueValidator(9998)], + verbose_name="Глубина выгрузки, лет", + ), + ), + ], + options={ + "verbose_name": "настройки выгрузки источников", + "verbose_name_plural": "настройки выгрузки источников", + }, + ), + migrations.RunPython(seed_settings, migrations.RunPython.noop), + migrations.AddConstraint( + model_name="sourceexportsettings", + constraint=models.CheckConstraint( + check=models.Q(("singleton_key", 1)), + name="source_export_settings_singleton", + ), + ), + ] diff --git a/src/organizations/models.py b/src/organizations/models.py index 05f8b3a..f28c08f 100644 --- a/src/organizations/models.py +++ b/src/organizations/models.py @@ -2,6 +2,7 @@ import uuid +from django.core.validators import MaxValueValidator from django.db import models from django.db.models import Q from django.utils.translation import gettext_lazy as _ @@ -1090,6 +1091,29 @@ class StockExchangeActivityExtension(OrganizationSourceExtension): verbose_name_plural = _("биржевая активность") +class SourceExportSettings(models.Model): + """Database-backed export policy, read once for each immutable generation.""" + + singleton_key = models.PositiveSmallIntegerField( + default=1, unique=True, editable=False + ) + years_back = models.PositiveSmallIntegerField( + _("Глубина выгрузки, лет"), default=5, validators=[MaxValueValidator(9998)] + ) + + class Meta: + verbose_name = _("настройки выгрузки источников") + verbose_name_plural = _("настройки выгрузки источников") + constraints = [ + models.CheckConstraint( + check=Q(singleton_key=1), name="source_export_settings_singleton" + ) + ] + + def __str__(self): + return "Настройки выгрузки источников" + + class SourceExportState(models.Model): """Current pointer and renewable owner lock for one all-available source.""" diff --git a/src/organizations/source_export_catalog.py b/src/organizations/source_export_catalog.py index c9c89ca..8dd34a3 100644 --- a/src/organizations/source_export_catalog.py +++ b/src/organizations/source_export_catalog.py @@ -20,6 +20,7 @@ import time import zipfile from contextlib import ExitStack, contextmanager, suppress from datetime import UTC, timedelta +from io import BytesIO from pathlib import Path from typing import BinaryIO, cast from uuid import UUID, uuid4 @@ -37,6 +38,7 @@ from organizations.models import ( SourceExportGeneration, SourceExportLease, SourceExportLimiter, + SourceExportSettings, SourceExportState, SourceExportTicket, SourceGroup, @@ -71,6 +73,7 @@ IDLE_BUILD = { "finished_at": None, "error_code": None, } +ARCHIVE_MAX_BYTES = 99_000_000 CHUNK_SIZE = 1024 * 1024 HEARTBEAT_RENEW_SECONDS = 30 TICKET_PATTERN = re.compile(r"se1_[A-Za-z0-9_-]{43}") @@ -99,6 +102,20 @@ def _limit(name, default): return max(1, int(getattr(settings, f"SOURCE_EXPORT_{name}", default))) +def _export_policy(): + years_back = SourceExportSettings.objects.values_list( + "years_back", flat=True + ).first() + years_back = 5 if years_back is None else years_back + year = timezone.localtime(timezone.now(), timezone.get_default_timezone()).year + return { + "years_back": years_back, + "min_year": max(1, year - years_back), + "max_year": year, + "archive_max_bytes": ARCHIVE_MAX_BYTES, + } + + def registry(): # Importing definitions does not execute their database-backed card service. from apps.parsers.source_cards import SOURCE_CARD_DEFINITIONS @@ -175,7 +192,7 @@ def rate_limit(user_id, kind): row.save(update_fields=["window", "count"]) -def summary(entry, state=None, generation=None, revision=None): +def summary(entry, state=None, generation=None, revision=None, policy=None): manifest = generation.manifest if generation and not generation.deleted_at else None build = state.last_build if state and state.last_build else dict(IDLE_BUILD) reason = None @@ -193,7 +210,8 @@ def summary(entry, state=None, generation=None, revision=None): stale = bool( manifest and ( - build["state"] == "failed" + manifest.get("export_policy") != (policy or _export_policy()) + or build["state"] == "failed" or generation.created_at < timezone.now() - timedelta(hours=36) or generation.data_revision != ( @@ -246,6 +264,7 @@ def catalog(): ) } entries = registry() + policy = _export_policy() sources = [ summary( entry, @@ -254,6 +273,7 @@ def catalog(): if group in states else None, get_source_data_cache_version(group), + policy, ) for group, entry in entries.items() ] @@ -327,6 +347,8 @@ def detail(group, query): if state and state.current_generation else None, "description": manifest.get("description", _description(entry)), + "export_policy": manifest.get("export_policy"), + "archives": manifest.get("archives", []), "datasets": [ {"year": None, **dataset} for dataset in manifest.get("datasets", []) ], @@ -458,7 +480,13 @@ def _year_rows(record, row, financial): yield year, {**row, "financial_lines": lines} -def _spool(group, path): +def _window_rows(record, row, financial, policy): + for year, annual in _year_rows(record, row, financial): + if year is not None and policy["min_year"] <= year <= policy["max_year"]: + yield year, annual + + +def _spool(group, path, policy): """Use a disk-backed sort, bounded RAM and one repeatable-read snapshot. The canonical table has extension joins, so an external local index is used @@ -513,10 +541,14 @@ def _spool(group, path): row = _build_record_row( record, include_financial_lines=group == FINANCIAL_SOURCE_GROUP ) - headers.update(key for key in row if key.startswith("payload.")) - for year, annual_row in _year_rows( - record, row, group == FINANCIAL_SOURCE_GROUP + included = False + for year, annual_row in _window_rows( + record, row, group == FINANCIAL_SOURCE_GROUP, policy ): + included = True + headers.update( + key for key in annual_row if key.startswith("payload.") + ) encoded = _json_bytes({"year": year, "row": annual_row}).decode( "utf-8" ) @@ -537,7 +569,7 @@ def _spool(group, path): encoded, ) ) - count += 1 + count += int(included) _check_capacity(path.parent) db.executemany("INSERT INTO rows VALUES (?, ?, ?, ?, ?)", rows) db.commit() @@ -561,7 +593,109 @@ def _spool(group, path): ordered.unlink(missing_ok=True) -def _render_part(group, spool, directory, headers, count, formats, serial, year=None): +def _archive_manifest(parts, fmt, policy=None): + # Only stable, selected-part metadata belongs in each archive. In particular + # no circular archive checksum/size or unrelated generation parts are included. + return { + "version": 5, + "partitioning": "year", + "supported_formats": [fmt], + "records_count": sum(p["records_count"] for p in parts), + "export_policy": policy or _export_policy(), + "parts": [ + { + **{ + key: part[key] + for key in ("part_id", "dataset_key", "year", "records_count") + }, + "variants": [v for v in part["variants"] if v["format"] == fmt], + } + for part in parts + ], + } + + +def _archive_entries(parts, fmt): + for part in parts: + variant = next(v for v in part["variants"] if v["format"] == fmt) + yield variant, f"{part['year']:04d}/" + variant["file_name"] + + +def _archive_size(parts, fmt, policy=None): + # ZIP_STORED has no compression variability. Empty entries give the precise + # seekable ZIP64 headers/central directory; payload sizes are then additive. + buffer = BytesIO() + payload_bytes = 0 + with zipfile.ZipFile(buffer, "w", compression=zipfile.ZIP_STORED) as archive: + for variant, name in _archive_entries(parts, fmt): + with archive.open(name, "w", force_zip64=True): + pass + payload_bytes += variant["size_bytes"] + archive.writestr( + "manifest.json", _json_bytes(_archive_manifest(parts, fmt, policy)) + ) + return len(buffer.getvalue()) + payload_bytes + + +def _build_archives(directory, manifest): + result = [] + for fmt in manifest["supported_formats"]: + batches, batch = [], [] + for part in manifest["parts"]: + if ( + _archive_size([*batch, part], fmt, manifest["export_policy"]) + > ARCHIVE_MAX_BYTES + ): + if not batch: + raise ExportError("record_too_large", 503) + batches.append(batch) + batch = [] + batch.append(part) + if batch: + batches.append(batch) + for index, parts in enumerate(batches, 1): + _check_capacity(directory) + name = f"{SOURCE_GROUP_EXPORT_FILE_STEMS[manifest['source_group']]}-{fmt}-archive-{index:06d}.zip" + path = directory / name + with zipfile.ZipFile(path, "w", compression=zipfile.ZIP_STORED) as archive: + for variant, member in _archive_entries(parts, fmt): + with (directory / variant["file_name"]).open( + "rb" + ) as source, archive.open(member, "w", force_zip64=True) as target: + shutil.copyfileobj(source, target, CHUNK_SIZE) + archive.writestr( + "manifest.json", + _json_bytes( + _archive_manifest(parts, fmt, manifest["export_policy"]) + ), + ) + size = path.stat().st_size + if size > ARCHIVE_MAX_BYTES: + raise ExportError("record_too_large", 503) + with path.open("rb") as handle: + checksum = hashlib.file_digest(handle, "sha256").hexdigest() + uid = str(uuid4()) + result.append( + { + "archive_id": uid, + "artifact_id": uid, + "archive_number": index, + "archives_count": len(batches), + "format": fmt, + "years": sorted({p["year"] for p in parts}), + "file_name": name, + "mime_type": "application/zip", + "size_bytes": size, + "sha256": checksum, + "part_ids": [p["part_id"] for p in parts], + } + ) + return result + + +def _render_part( + group, spool, directory, headers, count, formats, serial, year=None, policy=None +): """Render then split every variant together when actual bytes exceed the cap.""" max_bytes = min(_limit("MAX_FILE_BYTES", 512 * 1024 * 1024), 512 * 1024 * 1024) stem = SOURCE_GROUP_EXPORT_FILE_STEMS[group] + ( @@ -584,12 +718,39 @@ def _render_part(group, spool, directory, headers, count, formats, serial, year= if len(paths) != 1: raise ExportError("internal_error", 500) files.append((fmt, paths[0])) - if any(path.stat().st_size > max_bytes for _, path in files): + candidate = { + "part_id": str(uuid4()), + "dataset_key": _year_key(year), + "year": year, + "part_number": serial, + "parts_count": 1, + "title": "", + "records_count": count, + "default_format": formats[0], + "variants": [ + { + "artifact_id": str(uuid4()), + "format": fmt, + "file_name": path.name, + "mime_type": MIMES[fmt], + "size_bytes": path.stat().st_size, + "sha256": "0" * 64, + } + for fmt, path in files + ], + } + # Reserve the exact singleton ZIP envelope as well as the raw variant cap. + oversized = any( + _archive_size([candidate], fmt, policy) > ARCHIVE_MAX_BYTES for fmt in formats + ) + if oversized or any(path.stat().st_size > max_bytes for _, path in files): for _, path in files: path.unlink() if count == 1: raise ExportError("record_too_large", 503) - halves = [spool.with_suffix(".left"), spool.with_suffix(".right")] + halves = [ + spool.with_name(spool.name + suffix) for suffix in (".left", ".right") + ] try: with halves[0].open("w", encoding="utf-8") as left, halves[1].open( "w", encoding="utf-8" @@ -599,7 +760,15 @@ def _render_part(group, spool, directory, headers, count, formats, serial, year= _json_bytes(row).decode("utf-8") + "\n" ) parts = _render_part( - group, halves[0], directory, headers, count // 2, formats, serial, year + group, + halves[0], + directory, + headers, + count // 2, + formats, + serial, + year, + policy, ) parts.extend( _render_part( @@ -611,6 +780,7 @@ def _render_part(group, spool, directory, headers, count, formats, serial, year= formats, serial + len(parts), year, + policy, ) ) return parts @@ -646,7 +816,9 @@ def _render_part(group, spool, directory, headers, count, formats, serial, year= ] -def _render_year_parts(group, staging, annual_rows, headers, formats, year): +def _render_year_parts( + group, staging, annual_rows, headers, formats, year, policy=None +): max_rows = min( _limit("ROWS_PER_PART", 100000), _limit( @@ -656,6 +828,7 @@ def _render_year_parts(group, staging, annual_rows, headers, formats, year): 100000, ) target = min( + ARCHIVE_MAX_BYTES, _limit("TARGET_FILE_BYTES", 256 * 1024 * 1024), _limit("MAX_FILE_BYTES", 512 * 1024 * 1024), ) @@ -678,6 +851,7 @@ def _render_year_parts(group, staging, annual_rows, headers, formats, year): formats, len(parts) + 1, year, + policy, ) ) output = part_spool.open("w", encoding="utf-8") @@ -697,6 +871,7 @@ def _render_year_parts(group, staging, annual_rows, headers, formats, year): formats, len(parts) + 1, year, + policy, ) ) finally: @@ -705,7 +880,7 @@ def _render_year_parts(group, staging, annual_rows, headers, formats, year): return parts -def _render_parts(group, staging, spool, headers, formats): +def _render_parts(group, staging, spool, headers, formats, policy=None): from itertools import groupby parts = [] @@ -713,7 +888,13 @@ def _render_parts(group, staging, spool, headers, formats): envelopes = (json.loads(line) for line in source) for year, rows in groupby(envelopes, key=lambda item: item["year"]): annual = _render_year_parts( - group, staging, (item["row"] for item in rows), headers, formats, year + group, + staging, + (item["row"] for item in rows), + headers, + formats, + year, + policy, ) title = _entry(group)["title"] + ( f" — {year:04d} год" if year else " — год не определён" @@ -767,13 +948,17 @@ def build_source(group): raise OSError("Export disk reserve exhausted") staging.mkdir(mode=0o700) spool = staging / ".rows.jsonl" - headers, count, example, as_of, revision, example_key = _spool(group, spool) + policy = _export_policy() + headers, count, example, as_of, revision, example_key = _spool( + group, spool, policy + ) formats = ("json",) if group == FINANCIAL_SOURCE_GROUP else FORMATS - parts = _render_parts(group, staging, spool, headers, formats) + parts = _render_parts(group, staging, spool, headers, formats, policy) spool.unlink() _check_capacity(staging) manifest = { - "version": 4, + "version": 5, + "export_policy": policy, "partitioning": "year", "exported_records_count": sum(p["records_count"] for p in parts), "data_scope": "all_available", @@ -787,6 +972,7 @@ def build_source(group): "datasets": _annual_datasets(parts, entry["title"]), "parts": parts, } + manifest["archives"] = _build_archives(staging, manifest) (staging / "manifest.json").write_bytes(_json_bytes(manifest)) if lost.is_set(): raise ExportError("not_ready", 409) @@ -877,11 +1063,32 @@ def cleanup(group): return removed +def _select_archive(manifest, selection): + if "archive_id" not in selection: + raise ExportError("invalid_request", field="archive_id") + archive = next( + (a for a in manifest["archives"] if a["archive_id"] == selection["archive_id"]), + None, + ) + if archive is None: + raise ExportError("archive_not_found", 404, "archive_id") + if archive["format"] != selection["format"]: + raise ExportError("format_unavailable", 409, "format") + return [archive] + + def _select(generation, selection): manifest = generation.manifest if not manifest["parts"]: raise ExportError("empty", 409) parts = manifest["parts"] + if ( + selection["selection_kind"] == "source_archive" + and manifest.get("version", 0) >= 5 + ): + return _select_archive(manifest, selection) + if "archive_id" in selection: + raise ExportError("archive_not_found", 404, "archive_id") if selection["selection_kind"] == "file": parts = [part for part in parts if part["part_id"] == selection["part_id"]] if not parts: @@ -958,7 +1165,10 @@ def _issue_ticket(user, selection): issuer_id=str(user.pk), expires_at=now + timedelta(seconds=360), ) - file = selection["selection_kind"] == "file" + file = ( + selection["selection_kind"] == "file" + or generation.manifest.get("version", 0) >= 5 + ) name = ( artifacts[0]["file_name"] if file @@ -970,9 +1180,12 @@ def _issue_ticket(user, selection): "expires_in": 300, "mime_type": artifacts[0]["mime_type"] if file else "application/zip", "size_bytes": artifacts[0]["size_bytes"] if file else None, - "parts_count": len(artifacts), + "parts_count": len(artifacts[0]["part_ids"]) + if "part_ids" in artifacts[0] + else len(artifacts), } response.pop("part_id", None) + response.pop("archive_id", None) SourceExportTicket.objects.create( token_hash=hashlib.sha256(token.encode()).hexdigest(), lease=lease, @@ -1129,6 +1342,7 @@ def verify_generation(generation): artifacts = [ v for part in generation.manifest["parts"] for v in part["variants"] ] + artifacts.extend(generation.manifest.get("archives", [])) for artifact, handle in _open_artifacts(generation, artifacts, stack): if ( hashlib.file_digest(handle, "sha256").hexdigest() @@ -1183,7 +1397,9 @@ def _admission_error(generation, lease, now): def _prepare_stream(generation, lease, payload, stack, now): opened = _open_artifacts(generation, payload["artifacts"], stack) - if generation.manifest.get("partitioning") == "year": + if generation.manifest.get("partitioning") == "year" and not payload["artifacts"][ + 0 + ].get("archive_id"): years = { v["artifact_id"]: f"{p['year']:04d}" if p["year"] else "undated" for p in generation.manifest["parts"] @@ -1215,7 +1431,12 @@ def _prepare_stream(generation, lease, payload, stack, now): ], } return DownloadStream( - lease, opened, stack, manifest, payload["request"]["selection_kind"] == "file" + lease, + opened, + stack, + manifest, + payload["request"]["selection_kind"] == "file" + or bool(payload["artifacts"][0].get("archive_id")), ) diff --git a/src/organizations/source_export_serializers.py b/src/organizations/source_export_serializers.py index 7b98ea8..3c6c26d 100644 --- a/src/organizations/source_export_serializers.py +++ b/src/organizations/source_export_serializers.py @@ -7,12 +7,15 @@ class SourceExportSelectionSerializer(serializers.Serializer): source_group = serializers.CharField(max_length=64) generation_id = serializers.UUIDField() part_id = serializers.UUIDField(required=False) + archive_id = serializers.UUIDField(required=False) format = serializers.ChoiceField(choices=("xlsx", "csv", "json")) def validate(self, attrs): expected = {"selection_kind", "source_group", "generation_id", "format"} if attrs["selection_kind"] == "file": expected.add("part_id") + elif "archive_id" in self.initial_data: + expected.add("archive_id") if set(self.initial_data) != expected: raise serializers.ValidationError("Некорректный состав selector.") return {key: str(value) for key, value in attrs.items()} diff --git a/tests/apps/organizations/test_source_export_catalog.py b/tests/apps/organizations/test_source_export_catalog.py index 9c0fcae..cf11d29 100644 --- a/tests/apps/organizations/test_source_export_catalog.py +++ b/tests/apps/organizations/test_source_export_catalog.py @@ -93,9 +93,8 @@ class SourceExportCatalogTest(APITestCase): side_effect=render_while_source_changes, ), patch.object( refresh_source_export_catalog, "retry", side_effect=Retry() - ) as retry: - with self.assertRaises(Retry): - refresh_source_export_catalog.run(source_group=GROUP) + ) as retry, self.assertRaises(Retry): + refresh_source_export_catalog.run(source_group=GROUP) retry.assert_called_once_with(countdown=30, max_retries=1000) self.assertEqual(SourceExportGeneration.objects.count(), 1) source = next(s for s in catalog()["sources"] if s["source_group"] == GROUP) @@ -166,6 +165,16 @@ class SourceExportCatalogTest(APITestCase): } if kind == "file": result["part_id"] = generation.manifest["parts"][0]["part_id"] + if ( + kind == "source_archive" + and generation.manifest.get("version", 0) >= 5 + and generation.manifest["archives"] + ): + result["archive_id"] = next( + a["archive_id"] + for a in generation.manifest["archives"] + if a["format"] == fmt + ) return result def bytes(self, response): @@ -321,7 +330,7 @@ class SourceExportCatalogTest(APITestCase): self.user, self.selector(generation, "source_archive", "csv") ) response = consume_ticket(ticket["ticket"]) - self.assertNotIn("Content-Length", response) + self.assertEqual(int(response["Content-Length"]), ticket["size_bytes"]) with zipfile.ZipFile(BytesIO(self.bytes(response))) as archive: self.assertEqual(len(archive.namelist()), 4) self.assertEqual( @@ -329,7 +338,9 @@ class SourceExportCatalogTest(APITestCase): ) manifest = json.loads(archive.read("manifest.json")) self.assertNotIn("relative_path", json.dumps(manifest)) - self.assertEqual(manifest["generation_id"], str(generation.pk)) + self.assertEqual( + manifest["export_policy"], generation.manifest["export_policy"] + ) def test_ticket_survives_new_current(self): self.records() @@ -801,13 +812,12 @@ class SourceExportCatalogTest(APITestCase): 2021: {str(records[0].uid)}, 2022: {str(records[1].uid)}, 2024: {str(records[2].uid)}, - None: {str(records[3].uid), str(records[4].uid)}, }, ) - self.assertEqual(generation.manifest["exported_records_count"], 5) + self.assertEqual(generation.manifest["exported_records_count"], 3) self.assertEqual( [d["year"] for d in generation.manifest["datasets"]], - [2021, 2022, 2024, None], + [2021, 2022, 2024], ) def test_year_parts_numbers_formats_and_zip_directories(self): @@ -845,7 +855,7 @@ class SourceExportCatalogTest(APITestCase): self.assertEqual(manifest["partitioning"], "year") self.assertEqual(manifest["records_count"], len(records)) - def test_financial_record_without_lines_and_invalid_line_year_are_retained(self): + def test_financial_records_outside_window_and_invalid_line_year_are_excluded(self): extension = FinancialIndicatorsExtension.objects.create( organization=self.organization ) @@ -866,22 +876,16 @@ class SourceExportCatalogTest(APITestCase): source_record=invalid, year=0, form_code="1", line_code="1" ) generation = build_source("financial_indicators") - self.assertEqual( - [d["year"] for d in generation.manifest["datasets"]], [2019, None] + self.assertEqual(generation.manifest["datasets"], []) + self.assertEqual(generation.manifest["records_count"], 0) + self.assertEqual(generation.manifest["exported_records_count"], 0) + self.assertEqual(generation.manifest["archives"], []) + self.assertTrue( + OrganizationSourceRecord.objects.filter( + pk__in=[empty.pk, invalid.pk] + ).count(), + 2, ) - self.assertEqual(generation.manifest["records_count"], 2) - self.assertEqual(generation.manifest["exported_records_count"], 2) - rows = [] - for part in generation.manifest["parts"]: - variant = part["variants"][0] - rows.extend( - r - for org in json.loads( - (_directory(generation) / variant["file_name"]).read_bytes() - ) - for r in org["financial_indicators"] - ) - self.assertEqual({r["uid"] for r in rows}, {str(empty.uid), str(invalid.uid)}) def test_pre_partition_generation_metadata_and_ticket_stay_readable(self): self.records(1) @@ -921,7 +925,7 @@ class SourceExportCatalogTest(APITestCase): ).first() or descriptor.extension_model.objects.create( organization=self.organization ) - for year in [2020, 2021]: + for year in [2021, 2022]: record = OrganizationSourceRecord.objects.create( extension=extension, source=descriptor.source, @@ -944,7 +948,7 @@ class SourceExportCatalogTest(APITestCase): for generation in SourceExportGeneration.objects.all(): self.assertEqual(generation.manifest["partitioning"], "year") self.assertEqual( - [d["year"] for d in generation.manifest["datasets"]], [2020, 2021] + [d["year"] for d in generation.manifest["datasets"]], [2021, 2022] ) self.assertEqual(generation.manifest["records_count"], 2) @@ -997,3 +1001,341 @@ class SourceExportCatalogTest(APITestCase): PeriodicTasks.objects.filter(ident=1).update(last_update=old) migration.disable(apps, None) self.assertGreater(PeriodicTasks.objects.get(ident=1).last_update, old) + + def test_export_window_admin_setting_and_calendar_rollover(self): + from datetime import UTC, datetime + + from organizations import source_export_catalog as exports + from organizations.models import SourceExportSettings + + records = self.records(6) + for record, value in zip( + records, + ["2020-12-31", "2021-01-01", "2026-12-31", "2027-01-01", "invalid", ""], + strict=True, + ): + OrganizationSourceRecord.objects.filter(pk=record.pk).update( + record_date=value, created_at=datetime(2025, 7, 1, tzinfo=UTC) + ) + with patch.object( + exports.timezone, "now", return_value=datetime(2026, 11, 3, tzinfo=UTC) + ): + generation = build_source(GROUP) + self.assertEqual( + [d["year"] for d in generation.manifest["datasets"]], [2021, 2025, 2026] + ) + self.assertEqual(generation.manifest["records_count"], 3) + self.assertFalse( + next(s for s in catalog()["sources"] if s["source_group"] == GROUP)[ + "is_stale" + ] + ) + policy, _ = SourceExportSettings.objects.update_or_create( + singleton_key=1, defaults={"years_back": 2} + ) + self.assertTrue( + next(s for s in catalog()["sources"] if s["source_group"] == GROUP)[ + "is_stale" + ] + ) + narrowed = build_source(GROUP) + self.assertEqual( + [d["year"] for d in narrowed.manifest["datasets"]], [2025, 2026] + ) + policy.years_back = 5 + policy.save() + with patch.object( + exports.timezone, "now", return_value=datetime(2027, 1, 1, tzinfo=UTC) + ): + self.assertTrue( + next(s for s in catalog()["sources"] if s["source_group"] == GROUP)[ + "is_stale" + ] + ) + self.assertEqual(OrganizationSourceRecord.objects.count(), 6) + + def test_financial_window_counts_unique_records_and_filters_lines(self): + extension = FinancialIndicatorsExtension.objects.create( + organization=self.organization + ) + record = OrganizationSourceRecord.objects.create( + extension=extension, + source="fns_reports", + record_type="financial_report", + external_id=str(uuid4()), + ) + for year in [2020, 2021, 2026, 2027, 0]: + OrganizationSourceFinancialLine.objects.create( + source_record=record, year=year, form_code="1", line_code="1" + ) + generation = build_source("financial_indicators") + self.assertEqual(generation.manifest["records_count"], 1) + self.assertEqual(generation.manifest["exported_records_count"], 2) + self.assertEqual( + [d["year"] for d in generation.manifest["datasets"]], [2021, 2026] + ) + for part in generation.manifest["parts"]: + rows = json.loads( + (_directory(generation) / part["variants"][0]["file_name"]).read_bytes() + ) + lines = [ + line + for org in rows + for row in org["financial_indicators"] + for line in row["financial_lines"] + ] + self.assertEqual({line["year"] for line in lines}, {part["year"]}) + + def test_archive_selector_requires_id_and_checks_format(self): + self.records() + generation = build_source(GROUP) + selection = self.selector(generation, "source_archive") + for change, code in [ + ({"archive_id": str(uuid4())}, "source_export_archive_not_found"), + ({"format": "csv"}, "source_export_format_unavailable"), + ]: + with self.assertRaises(ExportError) as error: + issue_ticket(self.user, {**selection, **change}) + self.assertEqual(error.exception.code, code) + selection.pop("archive_id") + response = self.client.post( + self.base + "export-ticket/", selection, format="json" + ) + self.assertEqual(response.status_code, 400) + self.assertIn("archive_id", str(response.data)) + response = self.client.post( + self.base + "export-ticket/", + self.selector(generation, "source_archive"), + format="json", + ) + self.assertEqual(response.status_code, 201, response.data) + + def test_split_archives_are_complete_verified_and_pinned(self): + from organizations import source_export_catalog as exports + + records = self.records(20) + with patch.object(exports, "ARCHIVE_MAX_BYTES", 12500): + generation = build_source(GROUP) + archives = [a for a in generation.manifest["archives"] if a["format"] == "json"] + self.assertGreater(len(archives), 1) + observed = [] + tickets = [] + for index, metadata in enumerate(archives, 1): + self.assertLessEqual(metadata["size_bytes"], 12500) + self.assertEqual(metadata["archive_number"], index) + self.assertEqual(metadata["archives_count"], len(archives)) + selection = self.selector(generation, "source_archive") + selection["archive_id"] = metadata["archive_id"] + tickets.append((issue_ticket(self.user, selection), metadata)) + build_source(GROUP) + for ticket, metadata in tickets: + body = self.bytes(consume_ticket(ticket["ticket"])) + self.assertEqual(hashlib.sha256(body).hexdigest(), metadata["sha256"]) + self.assertEqual(len(body), metadata["size_bytes"]) + with zipfile.ZipFile(BytesIO(body)) as archive: + self.assertIsNone(archive.testzip()) + archived = json.loads(archive.read("manifest.json")) + self.assertEqual( + {p["part_id"] for p in archived["parts"]}, set(metadata["part_ids"]) + ) + for member in archive.namelist(): + if member == "manifest.json": + continue + year = int(member.split("/")[0]) + for org in json.loads(archive.read(member)): + for row in org[GROUP]: + self.assertEqual(int(row["record_date"][:4]), year) + observed.append(row["uid"]) + self.assertCountEqual(observed, [str(r.uid) for r in records]) + self.assertTrue(exports.verify_generation(generation)) + damaged = _directory(generation) / archives[0]["file_name"] + data = bytearray(damaged.read_bytes()) + data[40] ^= 1 + damaged.write_bytes(data) + self.assertFalse(exports.verify_generation(generation)) + + def test_single_record_too_large_preserves_current_generation(self): + from organizations import source_export_catalog as exports + + self.records(1) + previous = build_source(GROUP) + with patch.object(exports, "ARCHIVE_MAX_BYTES", 100), self.assertRaises( + ExportError + ) as error: + build_source(GROUP) + self.assertEqual(error.exception.code, "source_export_record_too_large") + self.assertEqual( + SourceExportState.objects.get(pk=GROUP).current_generation, previous.pk + ) + self.assertEqual(SourceExportGeneration.objects.count(), 1) + self.assertFalse(list(_directory(previous).parent.glob(".building-*"))) + + def test_v4_archive_ticket_remains_streamable_after_new_generation(self): + self.records(3) + old = build_source(GROUP) + old.manifest["version"] = 4 + old.manifest.pop("archives") + old.manifest.pop("export_policy") + old.save(update_fields=["manifest"]) + ticket = issue_ticket(self.user, self.selector(old, "source_archive")) + build_source(GROUP) + with zipfile.ZipFile( + BytesIO(self.bytes(consume_ticket(ticket["ticket"]))) + ) as archive: + self.assertIsNone(archive.testzip()) + self.assertEqual( + json.loads(archive.read("manifest.json"))["generation_id"], str(old.pk) + ) + + def test_exact_99mb_zip_limit_includes_headers_and_manifest(self): + from organizations import source_export_catalog as exports + + self.records(1) + generation = build_source(GROUP) + part = generation.manifest["parts"][0] + variant = next(v for v in part["variants"] if v["format"] == "json") + directory = _directory(generation) + # Sparse fixture proves the real 99,000,000-byte boundary without a huge ORM payload. + for delta in [0, 1]: + variant["size_bytes"] = 98_000_000 + while True: + measured = exports._archive_size( + [part], "json", generation.manifest["export_policy"] + ) + correction = 99_000_000 + delta - measured + if not correction: + break + variant["size_bytes"] += correction + with (directory / variant["file_name"]).open("wb") as handle: + handle.truncate(variant["size_bytes"]) + manifest = { + **generation.manifest, + "parts": [part], + "supported_formats": ["json"], + } + if delta: + with self.assertRaises(ExportError): + exports._build_archives(directory, manifest) + else: + built = exports._build_archives(directory, manifest) + self.assertEqual(built[0]["size_bytes"], 99_000_000) + with zipfile.ZipFile(directory / built[0]["file_name"]) as archive: + self.assertIsNone(archive.testzip()) + + def test_admin_singleton_policy_validation(self): + from django.contrib.admin.sites import AdminSite + from django.core.exceptions import ValidationError + from django.db import IntegrityError, transaction + from django.test import RequestFactory + from organizations.admin import SourceExportSettingsAdmin + from organizations.models import SourceExportSettings + + model_admin = SourceExportSettingsAdmin(SourceExportSettings, AdminSite()) + request = RequestFactory().get("/") + request.user = self.user + SourceExportSettings.objects.update_or_create(singleton_key=1) + self.assertFalse(model_admin.has_add_permission(request)) + self.assertFalse(model_admin.has_delete_permission(request)) + with transaction.atomic(), self.assertRaises(IntegrityError): + SourceExportSettings.objects.create(singleton_key=2) + with self.assertRaises(ValidationError): + SourceExportSettings(years_back=10000).full_clean() + + def test_large_year_is_split_at_record_boundaries_for_all_formats(self): + from organizations import source_export_catalog as exports + + records = self.records(8) + for record in records: + record.record_date = "2025-01-01" + record.payload = {"subject": "x" * 1500} + record.save(update_fields=["record_date", "payload"]) + with patch.object(exports, "ARCHIVE_MAX_BYTES", 14000), override_settings( + SOURCE_EXPORT_ROWS_PER_PART=100 + ): + generation = build_source(GROUP) + self.assertGreater(len(generation.manifest["parts"]), 1) + for fmt in ("json", "csv", "xlsx"): + archives = [ + a for a in generation.manifest["archives"] if a["format"] == fmt + ] + selected_parts = [] + for artifact in archives: + self.assertLessEqual(artifact["size_bytes"], 14000) + selected_parts.extend(artifact["part_ids"]) + with zipfile.ZipFile( + _directory(generation) / artifact["file_name"] + ) as archive: + self.assertIsNone(archive.testzip()) + self.assertCountEqual( + selected_parts, [p["part_id"] for p in generation.manifest["parts"]] + ) + self.assertEqual( + sum(p["records_count"] for p in generation.manifest["parts"]), 8 + ) + + def test_recursive_render_splits_keep_all_rows(self): + from organizations import source_export_catalog as exports + from organizations.source_record_export import _build_record_row + + records = self.records(64) + directory = Path(self.directory.name) + spool = directory / ".nested.jsonl" + spool.write_bytes( + b"".join( + exports._json_bytes( + _build_record_row(record, include_financial_lines=False) + ) + + b"\n" + for record in records + ) + ) + # Bypass the initial size heuristic: force several actual render/split levels. + with override_settings(SOURCE_EXPORT_MAX_FILE_BYTES=7000): + parts = exports._render_part( + GROUP, spool, directory, [], len(records), ("json",), 1, 2025 + ) + self.assertGreater(len(parts), 4) + observed = [] + for part in parts: + variant = part["variants"][0] + self.assertLessEqual(variant["size_bytes"], 7000) + observed.extend( + r["uid"] + for org in json.loads((directory / variant["file_name"]).read_bytes()) + for r in org[GROUP] + ) + self.assertCountEqual(observed, [str(r.uid) for r in records]) + self.assertFalse(list(directory.glob("*.left*"))) + self.assertFalse(list(directory.glob("*.right*"))) + + def test_build_uses_one_policy_even_if_admin_changes_during_render(self): + from organizations import source_export_catalog as exports + from organizations.models import SourceExportSettings + + self.records() + render = exports._render_parts + + def change_policy(*args, **kwargs): + SourceExportSettings.objects.update_or_create( + singleton_key=1, defaults={"years_back": 0} + ) + return render(*args, **kwargs) + + with patch.object(exports, "_render_parts", side_effect=change_policy): + generation = build_source(GROUP) + self.assertEqual(generation.manifest["export_policy"]["years_back"], 5) + for metadata in generation.manifest["archives"]: + with zipfile.ZipFile( + _directory(generation) / metadata["file_name"] + ) as archive: + self.assertEqual( + json.loads(archive.read("manifest.json"))["export_policy"][ + "years_back" + ], + 5, + ) + self.assertTrue( + next(s for s in catalog()["sources"] if s["source_group"] == GROUP)[ + "is_stale" + ] + ) diff --git a/tests/apps/organizations/test_source_export_postgres.py b/tests/apps/organizations/test_source_export_postgres.py index 11c79bd..f9780d6 100644 --- a/tests/apps/organizations/test_source_export_postgres.py +++ b/tests/apps/organizations/test_source_export_postgres.py @@ -17,12 +17,13 @@ from rest_framework.test import APIClient from tests.apps.user.factories import UserFactory +@pytest.mark.parametrize("selection_kind", ["file", "source_archive"]) @pytest.mark.django_db(transaction=True) @pytest.mark.skipif( connection.vendor != "postgresql", reason="Requires PostgreSQL row locks and independent connections", ) -def test_two_simultaneous_native_posts_one_ticket(tmp_path): +def test_two_simultaneous_native_posts_one_ticket(tmp_path, selection_kind): with override_settings(SOURCE_EXPORT_DIRECTORY=str(tmp_path)): user = UserFactory.create_superuser() organization = Organization.objects.create( @@ -41,10 +42,20 @@ def test_two_simultaneous_native_posts_one_ticket(tmp_path): ticket = issue_ticket( user, { - "selection_kind": "file", + "selection_kind": selection_kind, "source_group": "government_procurements", "generation_id": str(generation.pk), - "part_id": generation.manifest["parts"][0]["part_id"], + **( + {"part_id": generation.manifest["parts"][0]["part_id"]} + if selection_kind == "file" + else { + "archive_id": next( + a["archive_id"] + for a in generation.manifest["archives"] + if a["format"] == "json" + ) + } + ), "format": "json", }, ) diff --git a/tools/openapi/source-export-catalog.mjs b/tools/openapi/source-export-catalog.mjs index a3698f1..614b650 100644 --- a/tools/openapi/source-export-catalog.mjs +++ b/tools/openapi/source-export-catalog.mjs @@ -36,12 +36,16 @@ export function addSourceExportCatalog(document) { schemas.SourceExportPart = object({ part_id: uuid, year, dataset_key: string, title: string, part_number: positive, parts_count: positive, records_count: positive, default_format: format, variants: { ...array(ref('SourceExportArtifact')), minItems: 1 } }); schemas.SourceExportPagination = object({ page: positive, page_size: { ...positive, maximum: 200 }, total_count: integer, total_pages: positive, has_next: bool, has_previous: bool }); - schemas.SourceExportSourceDetail = object({ ...snapshot, current_generation_id: nullable(uuid), description: ref('SourceExportDescription'), datasets: array(ref('SourceExportDataset')), files: array(ref('SourceExportPart')) }); + schemas.SourceExportPolicy = object({ years_back: integer, min_year: positive, max_year: positive, archive_max_bytes: positive }); + schemas.SourceExportArchive = object({ archive_id: uuid, artifact_id: uuid, archive_number: positive, archives_count: positive, format, + years: array(positive), file_name: string, mime_type: string, size_bytes: positive, sha256: string, part_ids: array(uuid) }); + schemas.SourceExportSourceDetail = object({ ...snapshot, current_generation_id: nullable(uuid), export_policy: nullable(ref('SourceExportPolicy')), archives: array(ref('SourceExportArchive')), description: ref('SourceExportDescription'), datasets: array(ref('SourceExportDataset')), files: array(ref('SourceExportPart')) }); schemas.SourceExportSourceDetailResponse = envelope(ref('SourceExportSourceDetail'), object({ pagination: ref('SourceExportPagination') })); schemas.SourceExportAuthenticationError = object({ detail: string }); schemas.SourceExportError = object({ detail: string, code: string, field: nullable(string), retryable: bool, request_id: uuid }); schemas.SourceExportFileTicketRequest = object({ selection_kind: enumeration(['file']), source_group: group, generation_id: uuid, part_id: uuid, format }); schemas.SourceExportArchiveTicketRequest = object({ selection_kind: enumeration(['source_archive']), source_group: group, generation_id: uuid, format }); + schemas.SourceExportNumberedArchiveTicketRequest = object({ selection_kind: enumeration(['source_archive']), source_group: group, generation_id: uuid, archive_id: uuid, format }); schemas.OrganizationSourceRecordExportRequest.additionalProperties = false; schemas.SourceRecordExportTicket.additionalProperties = false; schemas.SourceRecordExportTicket.required = Object.keys(schemas.SourceRecordExportTicket.properties); @@ -63,7 +67,7 @@ export function addSourceExportCatalog(document) { ] : []; } const ticket = document.paths[base + 'export-ticket/'].post; - ticket.requestBody = { required: true, content: json({ oneOf: ['OrganizationSourceRecordExportRequest', 'SourceExportFileTicketRequest', 'SourceExportArchiveTicketRequest'].map(ref) }) }; + ticket.requestBody = { required: true, content: json({ oneOf: ['OrganizationSourceRecordExportRequest', 'SourceExportFileTicketRequest', 'SourceExportArchiveTicketRequest', 'SourceExportNumberedArchiveTicketRequest'].map(ref) }) }; ticket.responses = { ...errorResponses, 201: response({ oneOf: ['SourceRecordExportTicket', 'SourceExportSelectionTicket'].map(ref) }), 401: response(ref('SourceExportAuthenticationError'), 'Authentication required') }; const download = document.paths[base + 'export-download/'].post; download.requestBody = { required: true, content: { 'application/x-www-form-urlencoded': { schema: ref('SourceRecordExportDownloadRequest') } } };