TRIAL 2026-11-18
  • Kotlin 79.7%
  • HTML 16.4%
  • Shell 3.9%
Find a file
Repository files (latest commit first)
Filename Latest commit message Latest commit date
Travis Johnson 096abb8da8 README: consume as a Gradle source dependency
Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01X3rSFQsdekekENuCtuKWSN
2026-11-18 21:04:48 -06:00
bin misk-feedz: table-backed feeds for misk services 2026-11-18 20:49:09 -06:00
gradle misk-feedz: table-backed feeds for misk services 2026-11-18 20:49:09 -06:00
src Decode path parameters; CursorStore.get; dashboard tab without misk's iframe probe 2026-11-18 20:58:03 -06:00
.gitignore misk-feedz: table-backed feeds for misk services 2026-11-18 20:49:09 -06:00
build.gradle.kts misk-feedz: table-backed feeds for misk services 2026-11-18 20:49:09 -06:00
gradle.properties misk-feedz: table-backed feeds for misk services 2026-11-18 20:49:09 -06:00
gradlew misk-feedz: table-backed feeds for misk services 2026-11-18 20:49:09 -06:00
LICENSE misk-feedz: table-backed feeds for misk services 2026-11-18 20:49:09 -06:00
README.md README: consume as a Gradle source dependency 2026-11-18 21:04:48 -06:00
settings.gradle.kts misk-feedz: table-backed feeds for misk services 2026-11-18 20:49:09 -06:00

misk-feedz

Feeds for a misk service: the tables are the feed. Every mutation NULLs a row's feed_sync_id; a publisher hands out fresh, increasing ids under a row lock; consumers page feed_sync_id > cursor and see every row's latest state at least once, in an order with no gaps behind the cursor.

tcj:misk-feedz:0.1.0, package tcj.feedz. MySQL, misk-jdbc. It is not on Maven Central: consumers pull it from this repository as a Gradle source dependency in settings.gradle.kts (version 0.1.0 resolves to the git tag 0.1.0), or build it from a sibling checkout:

sourceControl {
  gitRepository(uri("https://github.com/traviscj/misk-feedz.git")) { producesModule("tcj:misk-feedz") }
}
// or: file("../misk-feedz").takeIf { it.isDirectory }?.let { includeBuild(it) }

Wiring

install(FeedzModule(
  AppDb::class,                                   // the JdbcModule qualifier
  config.feedz,                                   // FeedzConfig: tick_ms, publish_batch, fetch_endpoint, admin_page
  read = FeedzAccess(capabilities = listOf("user")),     // fetch, status, a consumer's own commit
  admin = FeedzAccess(capabilities = listOf("admin")),   // publish, pause/resume, cursor operator actions
))

// A feed served straight from its table (entries are rows):
multibind<FeedSource>().toInstance(tableFeed("widgets"))

// A feed with its own DTO and filters, built from injected stores:
@ProvidesIntoSet fun orders(store: OrderStore): FeedSource =
  feedSource<OrderDto>("orders", "orders", syncId = { it.feed_sync_id }) { q ->
    store.feed(q.after, q.limit, q.shardSql("o"), q.params["region"])
  }

// An in-process consumer of one of the service's own feeds (at-least-once, committed after handle returns):
multibind<FeedConsumer>().toInstance(FeedConsumer("indexer", "orders") { page -> index(page) })

Needs misk's AccessControlModule, and the tables in schema.sql in your own migrations: the library checks for them at startup and refuses to start without them, but never creates anything. Each feed table needs id, feed_sync_id BIGINT NULL (unique), and shard INT if anyone fetches by shard.

FeedzService runs one thread every tick_ms: publish every feed, then run the in-process consumers. tick_ms: 0 turns the loop off (tests call publisher.publishAll() / feedz.runConsumers()).

Endpoints

GET /_feeds/fetch/{feed}?after=&limit=&shard=&shard_count= stateless page {feed, entries, cursor}; other query params reach the feed's fetch as q.params
GET /_feeds/fetch/{feed}?consumer=NAME[&shard=&shard_count=] page after NAME's server-side cursor (created on first use); 409 while the cursor is disabled
POST /_feeds/{feed}/cursors/{consumer}/commit?token=[&shard=] {advanced}; forward-only, refused while disabled or past the feed's head
GET /_feeds/status feeds (sequence, rows, pending, held, publishing_enabled) and cursors (token, lag, enabled, timestamps, in-process poll/error state)
POST (admin) /_feeds/publish[?feed=] publish now, even if paused
POST (admin) /_feeds/{feed}/pause · /resume per-feed publishing switch, persisted in sequences
POST (admin) /_feeds/{feed}/cursors/{consumer}/enable · /disable [?shard=] per-cursor switch
POST (admin) /_feeds/{feed}/cursors/{consumer}/seek?token=[&shard=] rewind or skip; 0 or absent = from the start
POST (admin) /_feeds/{feed}/cursors/{consumer}/reshard?shard_count= fresh shards, all at the lowest old position
DELETE (admin) /_feeds/{feed}/cursors/{consumer}
GET /_admin/feedz/ (and /_feeds/ui) the page: feeds, cursors, and per-row commands shown as the curl they send

The page is markup only; what it shows comes from the calls above with the browser's own credentials. For header-authenticated dev setups, ?hdr.<Header-Name>=<value> is sent on every call.

With misk-admin's dashboard installed, set admin_page: false and install(FeedzDashboardTabModule()) to get the page as the tab Container Admin / Feeds at the same path.

Cursor columns

column moves when
token a consumer commits (forward only), or an operator seeks. Decimal feed_sync_id; NULL = from the start
enabled an operator enables/disables this cursor (the consumer side; sequences.publishing_enabled is the producer side)
updated_at operator changes only: enable/disable, seek, reshard
last_advanced_at only when a commit moves token

So "turned off at 14:02" (enabled = 0, updated_at) and "last made progress at 13:10" (last_advanced_at) are separate facts, and an operator rewind never looks like progress. There is no per-poll write. What the table cannot say, the process does for its own consumers: /_feeds/status carries last_poll_at / last_error for in-process ones.

lag is sequence − token: an upper bound on entries left (ids belonging to other shards and to republished rows count too).

Not here

  • A client for consuming another service's feed over HTTP; FeedConsumer reads local feeds only.
  • Databases other than MySQL.
  • More than one FeedzModule per injector.

Build

. bin/activate-hermit && ./gradlew test: needs database misk_feedz_test on the local mysqld (root, no password).