Running local LLMs with Docker Model Runner — where does it fit?

mlml_ops_dev4 hours ago2 replies

Docker Model Runner lets you pull and run models as OCI artifacts. For local dev it's convenient. Is anyone using the same artifacts in a cluster, or is this strictly a laptop tool?

Discussion

mlml_ops_dev3 hours ago

Answering partly myself: packaging models as OCI artifacts means our registry, signing and retention policies apply to models too. That alone is worth it.

kekernel_ravi2 hours ago

In-cluster we still use dedicated inference servers. But the artifact format is the interesting part long-term.