Skip to main content
The payload_loader module is the single owner of the payload pipeline. One JSON payload moves through this order: session memo, persistent cache, CDN fetch, validation, then memo and cache write-back. The ci_mode and offline_mode config flags apply on this path, the same as in tif1.async_fetch. Session routes its synchronous fetches through this module; the async fan-out (get_many) delegates to tif1.async_fetch, which shares the CDN fallback loop of tif1.cdn.CDNManager.

Overview

The module contains one protocol, two transport implementations, one loader class, and one factory function:
  • HttpTransport — the HTTP seam, defined as a typing.Protocol.
  • NiquestsTransport — the production transport, built on the shared niquests session.
  • InMemoryTransport — a test transport that serves payloads from an in-memory mapping.
  • PayloadLoader — the two-method interface (get for one path, get_many for async fan-out) that hides the whole pipeline.
  • get_url_loader() — a thread-safe factory for the shared loader used for absolute-URL payloads.
Session constructs one PayloadLoader per session and injects its own memo, cache callables, and overridable fetch delegates (Session._fetch_from_cdn and Session._fetch_from_cdn_fast). CDN source ordering and fallback live in tif1.cdn.CDNManager, not in this module. The loader supplies only the per-URL fetch callable built on the transport. Validation failures always raise. There is no patched-callable escape hatch.

HttpTransport

HttpTransport is a runtime-checkable typing.Protocol. Implementations fetch one URL and return the parsed JSON object.
Implementations must raise these exceptions:
  • DataNotFoundError — the resource does not exist (HTTP 404).
  • NetworkError — the request failed at the transport level.
  • InvalidDataError — the response body was not a JSON object.
Write a custom transport to route fetches through a proxy, a recording layer, or a local fixture set.

NiquestsTransport

NiquestsTransport is the production transport. It sends every request through the pooled niquests session owned by tif1.http_session.

Methods

get_json

Fetch url with the pooled niquests session and parse the JSON body. Each response is added to the request counters of tif1.http_session.get_connection_stats().
str
required
Absolute URL to fetch.
float | None
default:"None"
Request timeout in seconds. When None, the transport reads the timeout value from the configuration. The configuration default is 30.
Behavior:
  • A transport-level failure raises NetworkError with the URL and no status code.
  • HTTP 404 raises DataNotFoundError with the URL.
  • Any other HTTP error status raises NetworkError with the URL and the status code.
  • A response body that decodes to a non-dict JSON value raises InvalidDataError.

InMemoryTransport

InMemoryTransport serves payloads from an in-memory mapping. Use it in tests to run the full pipeline without network access. Every requested URL is recorded in the calls attribute.

Constructor

Mapping[str, Any] | None
default:"None"
Mapping of URL fragment to payload. A fragment can be a path, a full URL, or any substring of a URL. A value can be a dict, any other payload value, an Exception class or instance to raise, or a callable. A callable takes the URL and returns any of those value types.
Any
default:"sentinel"
Fallback entry used when no mapping matches. When unset, an unmatched URL raises DataNotFoundError.
Matching prefers an exact URL match, then the longest fragment contained in the requested URL.

Methods

add

Register a payload for a URL fragment, or replace the payload registered for that fragment.

get_json

Resolve url against the in-memory mapping and record url in calls. Raises DataNotFoundError when no mapping matched and no default was provided. Raises the matched exception when the matched entry specifies one.

key_for

Build the fragment key of a fixture file: its path under root as a URL path with forward slashes on every platform. str() of a Windows path renders backslashes, which never match the URL of the mirrored file.

PayloadLoader

PayloadLoader owns the pipeline order: memo, cache, CDN fetch, validation, write-back. The loader is parameterized by session coordinates (year, gp, session) that build cache keys and CDN URLs. All storage and the fetch step are injectable as callables.

Constructor

int | None
default:"None"
Season year. None is valid only for absolute-URL loaders.
str | None
default:"None"
Grand Prix identifier, URL-encoded.
str | None
default:"None"
Session identifier, URL-encoded.
HttpTransport | None
default:"None"
HTTP seam implementation. Defaults to NiquestsTransport().
CDNManager | None
default:"None"
CDN fallback manager. Defaults to the process-global tif1.cdn.get_cdn_manager() instance, resolved at call time. Also settable through the cdn_manager property.
SessionMemo | None
default:"None"
Per-session memo tier used by the default memo callables. When None and no memo_get/memo_set override is given, the memo step is skipped.
bool
default:"True"
Gate for the default persistent-cache callables.
Callable[..., Any] | None
default:"None"
Override for the fetch step, with signature fetch(path, *, fast: bool). Defaults to PayloadLoader.fetch_from_cdn.
Callable[[str], dict[str, Any] | None] | None
default:"None"
Override for the memo read, path -> dict or None.
Callable[[str, dict[str, Any]], None] | None
default:"None"
Override for the memo write, (path, dict).
Callable[[str], Any | None] | None
default:"None"
Override for the persistent-cache read, cache key -> data or None.
Callable[[str, dict[str, Any]], None] | None
default:"None"
Override for the persistent-cache write, (cache key, dict).

Methods

get

Fetch one session-relative JSON payload through the full pipeline. The order is: session memo, persistent cache, CDN fetch, validation, memo and cache write-back. Two config flags change the pipeline:
  • ci_mode: Skips the persistent cache read and write, as if use_cache and write_cache were False. The session memo still applies.
  • offline_mode: Serves memo and cache hits. On a miss, raises NetworkError (with the cache key as url) before any request.
str
required
Session-relative payload path, for example "drivers.json".
bool
default:"True"
Run payload validation before returning. Validation is path-based and uses the pydantic-free fast validators of tif1.validation. The validate_data, validate_lap_times, and validate_telemetry configuration values switch it on per payload type. All three default to False.
bool
default:"True"
Read the persistent cache before a network fetch.
bool
default:"True"
Write fetched payloads to the persistent cache.
bool
default:"False"
Skip per-source retry and backoff delays on the CDN fetch.
Raises DataNotFoundError when the payload does not exist. Raises InvalidDataError when the payload is not a JSON object or failed validation. Raises NetworkError when all CDN sources failed, or when offline_mode is on and the payload is not cached. Validation failures always raise. A payload found in the memo or cache returns without a network request. A cached dict is also written to the memo. The default memo keeps only paths that contain a / (such as VER/laptimes.json) and these session-level paths: drivers.json, session_laptimes.json, weather.json, rcm.json, session_info.json, car_data.json, and position.json. Other paths skip the memo and go to the persistent cache on every call.

fetch_from_cdn

Fetch a payload over HTTP with CDN fallback. CDN source ordering and fallback live in CDNManager.try_sources; this method only builds the per-URL fetch callable on the configured transport. When fast is False, tif1.retry.retry_with_backoff wraps the fetch callable. It reads the max_retries, retry_backoff_factor, and retry_jitter configuration values and retries only NetworkError. A max_retries value below 1 makes one attempt. It also runs each call through the process-wide circuit breaker. When fast is True, each CDN source is tried once with no delay, and the circuit breaker is bypassed. Raises DataNotFoundError when every CDN source answered 404. Raises NetworkError when the sources failed for any other reason. Raises TypeError when the loader was built without year, gp, and session.

get_url

Fetch a JSON object from an absolute URL. This method bypasses the memo, the cache, and CDN fallback; it calls the transport directly. Use it for resources outside the per-session CDN layout. The timeout default comes from the configuration.

get_many

Fetch many session-relative payloads concurrently. This coroutine delegates to tif1.async_fetch.fetch_multiple_async. That function uses the process-global SQLite cache and the shared niquests session directly. The injected transport, memo, and cache callables of this loader are not used on this path. When max_retries, timeout, or max_concurrent_requests is None, the configured value applies. The returned list has one entry per requested path; a failed fetch produces None at its position. Raises TypeError when the loader has no session context.

cache_key

Build the persistent-cache key for a session-relative path. The key format is "{year}/{gp}/{session}/{path}".

cdn_manager

The CDN manager used for fallback. The property is settable. When no manager was injected, the property returns the process-global instance from tif1.cdn.get_cdn_manager().

get_url_loader

Return the shared PayloadLoader for absolute-URL payloads. The loader is created on first call under a lock, so the function is safe to call from multiple threads. It has no session context, so only get_url works on it. tif1.events uses it to fetch the live schedule_{year}.json file of the current season.

Examples

Offline pipeline run with InMemoryTransport

This example runs the full get pipeline without network access. The enable_cache=False argument keeps the run off the persistent cache. Validation is disabled because the fixture payload is not a real drivers.json document. The memo tier is active only when a SessionMemo is passed; without one, every get call reaches the transport.

Absolute-URL fetch

This example requires network access. It fetches one session payload straight from the primary CDN, with no memo, cache, or fallback. (tif1.events uses the same method for the one non-CDN URL tif1 reads: the live schedule_{year}.json of the current season on GitHub.)
  • Core API: Session and its overridable _fetch_from_cdn delegates
  • CDN API: CDNManager source ordering and fallback
  • Cache API: SessionMemo and the persistent SQLite cache
  • Async Fetch API: fetch_multiple_async used by get_many
  • Config API: timeout and retry configuration values
  • Validation API: the validators used by the pipeline
Last modified on October 28, 2026