Skip to main content

Overview

tif1 downloads each payload once and serves later reads from local tiers. The tiers are:
  1. Session memo - per-Session dictionaries that hold the payloads and frames of the current session.
  2. In-memory LRU tiers - bounded process-wide caches in front of SQLite.
  3. SQLite frame tiers - persistent, pre-built laps and telemetry DataFrames.
  4. SQLite payload tables - persistent JSON and telemetry payloads in cache.sqlite.
  5. Negative-verdict tier - payload paths that every CDN reported as missing.
  6. CDN fetch - jsDelivr first, then Hugging Face buckets, then StaticDelivr.
A load stops at the first hit:
  • session.laps checks the backend lap LRU, then the laps frame tier, then the payloads.
  • A telemetry load checks the session memo, then the telemetry frame tier, then the payloads.
  • A single lap (lap.get_telemetry()) reads its driver’s frame-tier row and memoizes every lap of that driver. Later laps of the same driver are memo hits.
  • session.load() reads and decodes every driver row but builds a lap’s frame only when the lap is first touched. fetch_all_laps_telemetry() builds all of them and returns real frames.
  • The process-wide laps LRU stores a session-free copy of the laps frame. Each session gets its own bound copy, so a deleted session and its telemetry can be garbage-collected.
  • A load served from the memo and the frame tiers never starts an event loop. The async fetch pipeline runs only for laps the tiers do not hold.
  • A payload read checks the session memo, the LRU tiers, and the SQLite payload tables, in that order. On a miss, the fetch path checks the negative-verdict tier and then the CDN.
A fresh network result is written back to the memo, the LRU tiers, and SQLite. Every tier lives in src/tif1/cache.py.
Positive cache entries never expire. The data behind a finished session does not change, so tif1 keeps it until clear() or invalidate() removes it.

Tiers

Session memo

Each Session owns one SessionMemo. The memo stores three kinds of data: The memo also records the persistent-cache probe result and per-driver telemetry failure counts. It is not shared between sessions and is released with the session object.

In-memory LRU tiers

The SQLite Cache object fronts its database with four LRUCache instances. LRUCache is an OrderedDict-based bounded LRU. Reads on the hot path do not take a lock. A hit in a parsed tier skips the orjson parse completely.

SQLite payload tables

The database file is cache.sqlite inside the cache directory. Two tables hold payloads:
JSON-tier keys have the form "{year}/{gp}/{session}/{path}". The payload paths are: Telemetry payloads ({driver}/{lap}_tel.json) are routed to telemetry_cache. Their key is the tuple (year, gp, session, driver, lap). All other payloads go to the cache table.

SQLite frame tiers

The frame tiers store assembled DataFrames so a warm load skips JSON parsing and frame construction. Each backend has its own tables with the same columns:
session.laps reads the processed laps table from the laps frame table of its backend (Cache.get_laps_frame()) on a warm cache. After a build from the payload tier, it writes the table there (Cache.set_laps_frame()). Session.fetch_all_laps_telemetry() assembles the telemetry frames and then hands the bulk write to a background daemon thread. register_frame_writer() starts that thread. Cache.close(), invalidate(), clear(), and the atexit handler join it, so the connection never closes under a live writer. The payload tables stay the source of truth. A frame row that fails to load becomes a tier miss and the loader falls back to the payload tier. The telemetry frame tables are versioned through SQLite PRAGMA user_version (_FRAME_TIER_SCHEMA_VERSION). When the stored version is older, tif1 drops both telemetry frame tables once. Later loads rebuild the frames from the payload tier. The laps frame tables are not versioned.

Negative-verdict tier

When every CDN answers a payload request with a 4xx, tif1 marks the payload as missing. A 408 or 429 does not count. The async fetch path and the synchronous PayloadLoader.get path both record the key in missing_payloads:
Later fetches on both paths consult is_known_missing() after a cache miss and raise DataNotFoundError without a request. A single lap or a driver’s telemetry with a missing lap therefore costs no network request after the first process records the verdict. A verdict expires after missing_payloads_ttl_days (default 7.0). A successful write for the same key discards the verdict.

Live schedule row

tif1.events stores the raw schedule_{year}.json of a live season in the json tier under f1schedule/schedule_{year}.json, together with its fetch time. The row is re-fetched from the theOehrly/f1schedule repository once it is older than schedule_ttl_hours (default 6.0); a failed refresh keeps serving the stale row. Session payloads never expire; this row and the negative verdicts are the two TTL-scoped entries.

Backend lap LRU

Parsed lap tables are kept in one process-global LRUCache per backend (pandas and polars). The limit is MAX_CACHE_SIZE = 100 sessions per backend and the key is "{year}_{gp}_{session}_laps".

Storage format and compression

Payload rows

Payload rows hold orjson-encoded bytes. A row of 4096 bytes or more (_SQLITE_COMPRESS_MIN_BYTES) is compressed with zstd at level 1 (_ZSTD_LEVEL). Smaller rows are stored as plain TEXT. Legacy zlib rows stay readable because zstd frames carry a 4-byte magic prefix. The zstd binding depends on the interpreter:
  • Python 3.14 and later use the standard-library compression.zstd module (PEP 784).
  • Python 3.11 to 3.13 use the zstandard package.
Both bind the same libzstd and emit identical single-frame blobs. A cache written under one binding reads under the other. zstd-1 compresses about 4.5x faster and decompresses about 2x faster than the previous zlib-3 codec. It also stores 23.4% of the original size against 26.2% for zlib-3. These figures were measured on 1452 telemetry payloads from the 2026 Monaco race.

Frame rows

Pandas telemetry frames are stored per driver in telemetry_frames_drv. The row holds a typed-columnar encoding of every lap of the driver. The laps are grouped by block layout. Each group stores one array per block over all of its rows, a dictionary for the string columns, and a small header. The header lists the column names, the lap numbers, the row count of each lap, and the block layout. The reader builds each lap frame from views into the group arrays, without the DataFrame constructor and without a copy. A driver’s merged telemetry (Laps.telemetry) is a slice of the same arrays. A partial write merges with the laps already stored for the driver. A frame with a column outside the telemetry dtype contract is pickled into the per-lap telemetry_frames table instead. Rows written by earlier releases sit in telemetry_frames as per-lap pickles or per-lap typed-columnar blobs. The reader still serves them. When a load serves 32 or more laps of a session from these rows, a background thread rewrites them as driver rows. The per-lap rows are then deleted. The reader tells the formats apart by a magic prefix. Laps frame rows hold a pickle (protocol 5). The blob is compressed with lz4 frame at level 0. When lz4 is not installed, the writer falls back to zstd-1. Python 3.15 has no lz4 wheels yet, so the frame tier uses zstd there. The reader detects the codec from the magic prefix, so lz4 and zstd rows coexist in one table. The reader decompresses lz4 rows through pyarrow’s codec, which releases the GIL. A batch read decompresses in a thread pool. Each chunk is decoded while the others are still in flight. Without pyarrow’s lz4 support, the reader uses the serial lz4 codec. Polars telemetry frames are stored per driver in telemetry_frames_pl_ipc as one Arrow IPC file with lz4 inside the stream. String columns are written as Arrow string_view, which polars reads without a conversion. Rows written by earlier releases store large_string and read back to the same frames. A lap is a slice of the driver frame; the row’s laps column lists the lap numbers and row counts in storage order. A partial write merges with the laps already stored for the driver. A batch read reads the IPC files in a thread pool and converts them to polars in one step. The laps are slices at a global offset. On the 1452-frame 2026 Monaco race set, a fresh process reads the pandas driver rows in 57 ms against 179 ms for per-lap rows. The polars driver rows read in 53 ms against 87 ms. The frame tier is write-once and read-many, so read speed wins over ratio.

Cache location and configuration

Default directory

tif1 creates the directory and sets its mode to 0o700 (owner access only). The frame tier unpickles blobs from this directory, so the restricted mode is part of the security model.

Configuration keys

Configuration is read through tif1.get_config().get(key) and changed with .set(key, value). The Config object has no attribute-style access and tif1.Config is not exported. Boolean environment variables accept 1, true, yes, and on.
Change cache_dir before the first call to tif1.get_cache() or tif1.get_session(). The SQLite Cache reads the directory once when it is created.

The .tif1rc file

.tif1rc is a JSON object. tif1 reads ~/.tif1rc by default. A .tif1rc in the current directory is read only when TIF1_TRUST_CWD_CONFIG=true. TIF1_CONFIG_FILE points at an explicit file instead. Environment variables override file values.

Managing the cache from Python

The SQLite cache object

tif1.get_cache() returns the process-wide tif1.cache.Cache instance.

Invalidation scopes

Every scope except "memory" also deletes the negative verdicts. Both laps frame tables go with either payload scope because the laps table is derived from payloads of both tiers. Each scope first joins a pending frame writer, so a background write cannot re-insert rows after the wipe.

The FastF1-compatible tif1.Cache

tif1.Cache is not the SQLite class. It is the FastF1-compatible facade from src/tif1/fastf1_compat.py and exposes class methods only:
The methods delegate to tif1.get_config() and tif1.get_cache(). Code that already targets FastF1 can keep its cache calls unchanged.

Command line

Disabling the cache

Pass enable_cache=False to get_session() to bypass every persistent tier for one session. Nothing is written for that session.
Set TIF1_ENABLE_CACHE=false or tif1.get_config().set("enable_cache", False) to change the default for every new session. offline_mode=True keeps cache reads enabled and skips the network for payload fetches. A miss raises NetworkError. ci_mode=True skips persistent cache reads and writes for payload fetches.
The first access that needs two or more of drivers.json, weather.json, and rcm.json prefetches them in one batch, for example session.load(). That batch does not check offline_mode or ci_mode, so it can still use the network and write to the cache. The laps frame tier also ignores ci_mode.

Behaviour notes

One sqlite3 connection is opened with check_same_thread=False and guarded by Cache._sqlite_lock. Writes are batched and committed every cache_commit_interval operations (default 100) and again on close(). The connection sets journal_mode=WAL, synchronous=NORMAL, and cache_size=-64000 (a 64 MB page cache). Memory-tier reads do not take a lock.
On first use, a session probes has_session_data(). A session with no cached rows skips cache reads for the rest of its life. It still writes everything it fetches. The next Session object for the same event finds a warm cache.
close() flushes pending writes, closes the connection, and sets conn to None. The object does not reconnect. Later get calls return None and later set calls do nothing. The atexit handler closes the global cache and resets get_cache(), so a later get_cache() call creates a new instance.
A row that fails to decompress, parse, or unpickle is logged and treated as a miss. The row is not deleted. The next successful write for the same key replaces it.
Payload rows, frame rows, and memory entries never expire. Only the negative-verdict tier (missing_payloads_ttl_days) and the live schedule row (schedule_ttl_hours) are TTL-scoped. Use invalidate() or clear() to remove data.
The cache directory is forced to mode 0o700. Keep the directory private: the frame tier unpickles blobs from it and trusts their origin.
Python 3.14 and later compress payload rows with the standard-library compression.zstd module. Python 3.15 has no lz4 wheels yet, so frame rows are zstd-1 there. Payload rows read under every binding, so one cache directory can be shared between interpreter versions. Frame rows are the exception. An interpreter without lz4 treats lz4 rows from another interpreter as misses. It then rebuilds the frames from the payload tier.

Cache API

Reference for the Cache class and module functions.

Best Practices

Patterns for fast and reliable data access.

Architecture

How payloads move from the CDN into DataFrames.

Configuration

Every configuration key and environment variable.

Error Handling

Cache, network, and data exceptions.

Troubleshooting

Fixes for common cache problems.
Last modified on October 28, 2026