- Kotlin 79.7%
- HTML 16.4%
- Shell 3.9%
| Filename | Latest commit message | Latest commit date |
|---|---|---|
Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01X3rSFQsdekekENuCtuKWSN |
||
| bin | ||
| gradle | ||
| src | ||
| .gitignore | ||
| build.gradle.kts | ||
| gradle.properties | ||
| gradlew | ||
| LICENSE | ||
| README.md | ||
| settings.gradle.kts | ||
misk-feedz
Feeds for a misk service: the tables are the feed. Every mutation NULLs a row's feed_sync_id; a
publisher hands out fresh, increasing ids under a row lock; consumers page feed_sync_id > cursor
and see every row's latest state at least once, in an order with no gaps behind the cursor.
tcj:misk-feedz:0.1.0, package tcj.feedz. MySQL, misk-jdbc. It is not on Maven Central: consumers
pull it from this repository as a Gradle source dependency in settings.gradle.kts (version 0.1.0
resolves to the git tag 0.1.0), or build it from a sibling checkout:
sourceControl {
gitRepository(uri("https://github.com/traviscj/misk-feedz.git")) { producesModule("tcj:misk-feedz") }
}
// or: file("../misk-feedz").takeIf { it.isDirectory }?.let { includeBuild(it) }
Wiring
install(FeedzModule(
AppDb::class, // the JdbcModule qualifier
config.feedz, // FeedzConfig: tick_ms, publish_batch, fetch_endpoint, admin_page
read = FeedzAccess(capabilities = listOf("user")), // fetch, status, a consumer's own commit
admin = FeedzAccess(capabilities = listOf("admin")), // publish, pause/resume, cursor operator actions
))
// A feed served straight from its table (entries are rows):
multibind<FeedSource>().toInstance(tableFeed("widgets"))
// A feed with its own DTO and filters, built from injected stores:
@ProvidesIntoSet fun orders(store: OrderStore): FeedSource =
feedSource<OrderDto>("orders", "orders", syncId = { it.feed_sync_id }) { q ->
store.feed(q.after, q.limit, q.shardSql("o"), q.params["region"])
}
// An in-process consumer of one of the service's own feeds (at-least-once, committed after handle returns):
multibind<FeedConsumer>().toInstance(FeedConsumer("indexer", "orders") { page -> index(page) })
Needs misk's AccessControlModule, and the tables in
schema.sql in your own migrations: the library checks
for them at startup and refuses to start without them, but never creates anything. Each feed table
needs id, feed_sync_id BIGINT NULL (unique), and shard INT if anyone fetches by shard.
FeedzService runs one thread every tick_ms: publish every feed, then run the in-process
consumers. tick_ms: 0 turns the loop off (tests call publisher.publishAll() / feedz.runConsumers()).
Endpoints
| GET | /_feeds/fetch/{feed}?after=&limit=&shard=&shard_count= |
stateless page {feed, entries, cursor}; other query params reach the feed's fetch as q.params |
| GET | /_feeds/fetch/{feed}?consumer=NAME[&shard=&shard_count=] |
page after NAME's server-side cursor (created on first use); 409 while the cursor is disabled |
| POST | /_feeds/{feed}/cursors/{consumer}/commit?token=[&shard=] |
{advanced}; forward-only, refused while disabled or past the feed's head |
| GET | /_feeds/status |
feeds (sequence, rows, pending, held, publishing_enabled) and cursors (token, lag, enabled, timestamps, in-process poll/error state) |
| POST (admin) | /_feeds/publish[?feed=] |
publish now, even if paused |
| POST (admin) | /_feeds/{feed}/pause · /resume |
per-feed publishing switch, persisted in sequences |
| POST (admin) | /_feeds/{feed}/cursors/{consumer}/enable · /disable [?shard=] |
per-cursor switch |
| POST (admin) | /_feeds/{feed}/cursors/{consumer}/seek?token=[&shard=] |
rewind or skip; 0 or absent = from the start |
| POST (admin) | /_feeds/{feed}/cursors/{consumer}/reshard?shard_count= |
fresh shards, all at the lowest old position |
| DELETE (admin) | /_feeds/{feed}/cursors/{consumer} |
|
| GET | /_admin/feedz/ (and /_feeds/ui) |
the page: feeds, cursors, and per-row commands shown as the curl they send |
The page is markup only; what it shows comes from the calls above with the browser's own
credentials. For header-authenticated dev setups, ?hdr.<Header-Name>=<value> is sent on every call.
With misk-admin's dashboard installed, set admin_page: false and install(FeedzDashboardTabModule())
to get the page as the tab Container Admin / Feeds at the same path.
Cursor columns
| column | moves when |
|---|---|
token |
a consumer commits (forward only), or an operator seeks. Decimal feed_sync_id; NULL = from the start |
enabled |
an operator enables/disables this cursor (the consumer side; sequences.publishing_enabled is the producer side) |
updated_at |
operator changes only: enable/disable, seek, reshard |
last_advanced_at |
only when a commit moves token |
So "turned off at 14:02" (enabled = 0, updated_at) and "last made progress at 13:10"
(last_advanced_at) are separate facts, and an operator rewind never looks like progress. There is
no per-poll write. What the table cannot say, the process does for its own consumers: /_feeds/status
carries last_poll_at / last_error for in-process ones.
lag is sequence − token: an upper bound on entries left (ids belonging to other shards and to
republished rows count too).
Not here
- A client for consuming another service's feed over HTTP;
FeedConsumerreads local feeds only. - Databases other than MySQL.
- More than one
FeedzModuleper injector.
Build
. bin/activate-hermit && ./gradlew test: needs database misk_feedz_test on the local mysqld
(root, no password).