payload_loader module is the single owner of the payload pipeline. One JSON payload moves through this order: session memo, persistent cache, CDN fetch, validation, then memo and cache write-back. The ci_mode and offline_mode config flags apply on this path, the same as in tif1.async_fetch. Session routes its synchronous fetches through this module; the async fan-out (get_many) delegates to tif1.async_fetch, which shares the CDN fallback loop of tif1.cdn.CDNManager.
Overview
The module contains one protocol, two transport implementations, one loader class, and one factory function:HttpTransport— the HTTP seam, defined as atyping.Protocol.NiquestsTransport— the production transport, built on the shared niquests session.InMemoryTransport— a test transport that serves payloads from an in-memory mapping.PayloadLoader— the two-method interface (getfor one path,get_manyfor async fan-out) that hides the whole pipeline.get_url_loader()— a thread-safe factory for the shared loader used for absolute-URL payloads.
Session constructs one PayloadLoader per session and injects its own memo, cache callables, and overridable fetch delegates (Session._fetch_from_cdn and Session._fetch_from_cdn_fast). CDN source ordering and fallback live in tif1.cdn.CDNManager, not in this module. The loader supplies only the per-URL fetch callable built on the transport.
Validation failures always raise. There is no patched-callable escape hatch.
HttpTransport
HttpTransport is a runtime-checkable typing.Protocol. Implementations fetch one URL and return the parsed JSON object.
DataNotFoundError— the resource does not exist (HTTP 404).NetworkError— the request failed at the transport level.InvalidDataError— the response body was not a JSON object.
NiquestsTransport
NiquestsTransport is the production transport. It sends every request through the pooled niquests session owned by tif1.http_session.
Methods
get_json
url with the pooled niquests session and parse the JSON body. Each response is added to the request counters of tif1.http_session.get_connection_stats().
Request timeout in seconds. When
None, the transport reads the timeout value from the configuration. The configuration default is 30.- A transport-level failure raises
NetworkErrorwith the URL and no status code. - HTTP 404 raises
DataNotFoundErrorwith the URL. - Any other HTTP error status raises
NetworkErrorwith the URL and the status code. - A response body that decodes to a non-dict JSON value raises
InvalidDataError.
InMemoryTransport
InMemoryTransport serves payloads from an in-memory mapping. Use it in tests to run the full pipeline without network access. Every requested URL is recorded in the calls attribute.
Constructor
Mapping of URL fragment to payload. A fragment can be a path, a full URL, or any substring of a URL. A value can be a dict, any other payload value, an
Exception class or instance to raise, or a callable. A callable takes the URL and returns any of those value types.Fallback entry used when no mapping matches. When unset, an unmatched URL raises
DataNotFoundError.Methods
add
get_json
url against the in-memory mapping and record url in calls. Raises DataNotFoundError when no mapping matched and no default was provided. Raises the matched exception when the matched entry specifies one.
key_for
root as a URL path with forward slashes on every platform. str() of a Windows path renders backslashes, which never match the URL of the mirrored file.
PayloadLoader
PayloadLoader owns the pipeline order: memo, cache, CDN fetch, validation, write-back. The loader is parameterized by session coordinates (year, gp, session) that build cache keys and CDN URLs. All storage and the fetch step are injectable as callables.
Constructor
CDN fallback manager. Defaults to the process-global
tif1.cdn.get_cdn_manager() instance, resolved at call time. Also settable through the cdn_manager property.Per-session memo tier used by the default memo callables. When
None and no memo_get/memo_set override is given, the memo step is skipped.Override for the fetch step, with signature
fetch(path, *, fast: bool). Defaults to PayloadLoader.fetch_from_cdn.Override for the memo read,
path -> dict or None.Override for the memo write,
(path, dict).Override for the persistent-cache read,
cache key -> data or None.Override for the persistent-cache write,
(cache key, dict).Methods
get
ci_mode: Skips the persistent cache read and write, as ifuse_cacheandwrite_cachewereFalse. The session memo still applies.offline_mode: Serves memo and cache hits. On a miss, raisesNetworkError(with the cache key asurl) before any request.
Run payload validation before returning. Validation is path-based and uses the pydantic-free fast validators of
tif1.validation. The validate_data, validate_lap_times, and validate_telemetry configuration values switch it on per payload type. All three default to False.DataNotFoundError when the payload does not exist. Raises InvalidDataError when the payload is not a JSON object or failed validation. Raises NetworkError when all CDN sources failed, or when offline_mode is on and the payload is not cached. Validation failures always raise.
A payload found in the memo or cache returns without a network request. A cached dict is also written to the memo. The default memo keeps only paths that contain a / (such as VER/laptimes.json) and these session-level paths: drivers.json, session_laptimes.json, weather.json, rcm.json, session_info.json, car_data.json, and position.json. Other paths skip the memo and go to the persistent cache on every call.
fetch_from_cdn
CDNManager.try_sources; this method only builds the per-URL fetch callable on the configured transport.
When fast is False, tif1.retry.retry_with_backoff wraps the fetch callable. It reads the max_retries, retry_backoff_factor, and retry_jitter configuration values and retries only NetworkError. A max_retries value below 1 makes one attempt. It also runs each call through the process-wide circuit breaker. When fast is True, each CDN source is tried once with no delay, and the circuit breaker is bypassed.
Raises DataNotFoundError when every CDN source answered 404. Raises NetworkError when the sources failed for any other reason. Raises TypeError when the loader was built without year, gp, and session.
get_url
timeout default comes from the configuration.
get_many
tif1.async_fetch.fetch_multiple_async. That function uses the process-global SQLite cache and the shared niquests session directly. The injected transport, memo, and cache callables of this loader are not used on this path. When max_retries, timeout, or max_concurrent_requests is None, the configured value applies. The returned list has one entry per requested path; a failed fetch produces None at its position. Raises TypeError when the loader has no session context.
cache_key
"{year}/{gp}/{session}/{path}".
cdn_manager
tif1.cdn.get_cdn_manager().
get_url_loader
PayloadLoader for absolute-URL payloads. The loader is created on first call under a lock, so the function is safe to call from multiple threads. It has no session context, so only get_url works on it. tif1.events uses it to fetch the live schedule_{year}.json file of the current season.
Examples
Offline pipeline run with InMemoryTransport
This example runs the fullget pipeline without network access. The enable_cache=False argument keeps the run off the persistent cache. Validation is disabled because the fixture payload is not a real drivers.json document. The memo tier is active only when a SessionMemo is passed; without one, every get call reaches the transport.
Absolute-URL fetch
This example requires network access. It fetches one session payload straight from the primary CDN, with no memo, cache, or fallback. (tif1.events uses the same method for the one non-CDN URL tif1 reads: the live schedule_{year}.json of the current season on GitHub.)
Related APIs
- Core API:
Sessionand its overridable_fetch_from_cdndelegates - CDN API:
CDNManagersource ordering and fallback - Cache API:
SessionMemoand the persistent SQLite cache - Async Fetch API:
fetch_multiple_asyncused byget_many - Config API: timeout and retry configuration values
- Validation API: the validators used by the pipeline