We released NOMYO Router

Recently we released a key piece of our infrastructure as open source software: Hello world NOMYO Router !

The router is managing multiple Inference Endpoints on both Ollama and OpenAI V1 API compatible endpoints and routes request from your frontend application to the configured endpoints for inference.

Router is aware of which model is served by which endpoint and how many (configurable) sessions will saturate an endpoint before using a different endpoint or API for a given large language model.

This way you can use local and remote models with the same interface your frontend or application is already using. No need for additional connection-, client- or session management. All fully asynchronuous and - this might be the best feature - it talks Ollama OR OpenAI API and translates your request into the necessary request for the endpoint the requested model is hosted on.

What does this mean?

When you built your application for OpenAI API and you now want to add an open source large language model hosted locally on Ollama you can just config both endpoints (OpenAI & Ollama) on NOMYO Router, alter one line of code in your application and talk to both endpoints via the same connection! Also the other way round: if your application only talks Ollama, you can keep your existing Ollama client connection and talk to any other OpenAI V1 compatible API without changing your application!

NOMYO Router will translate Ollama -> OpenAI or OpenAI -> Ollama when necessary, maintaining existing syntax and supported features. The Router is always transparent to both frontends/applications and endpoints.

Extend your modelverse! Just by putting NOMYO Router in the middle.

It can handle hundreds of simultaneous connections and the routing logic will select the best endpoint for the given request, which let’s you handle more different model-specific request at the same time compared to common least-connection routing.

Why are we doing this?

We believe in and use ourselves open source software. Together we can build something extra ordinary!

And for us it is time to give back to the community.

We hope you will find our Router as useful as we do…

_