Dynamo Hello World
Created a PR to add multi-node hello world example for Dynamo.
1 Frontend
Frontend class use @api decorator to setup API endpoint to take HTTP request. The augument of fegenerate should match with HTTP request’s data field keys.
class Frontend
@api
async def fegenerate(self, prompt, request_id): # from request data keys
...
Would take request from HTTP request
curl -X 'POST' \
'http://localhost:3000/fegenerate' \
-H 'accept: text/event-stream' \
-H 'Content-Type: application/json' \
-d '{
"prompt": "test prompt str",
"request_id":"request id str"
}'
2 Processor
The proessor is called in the Frontend.Its method name should match the method decorated by @dynamo_endpoint(). It seems we can only have one endpoint for each component.
@service(...)
class Frontend:
processor = depends(Processor)
...
self.processor.mdgenerate(...)
@service(...)
class Processor(Protocol):
@dynamo_endpoint()
async def mdgenerate(self, raw_request: GeneralRequest):
3 Worker
The worker can be referred from runtime in processor, as a client. The client can use direct, random, or round_robin methods to call workers.
@service(...)
class DummyWorker:
@dynamo_endpoint()
async def begenerate(self, request: GeneralRequest):
...
### In Processor or Router
runtime = dynamo_context["runtime"]
comp_ns, comp_name = DummyWorker.dynamo_address() # type: ignore
self.worker_client = (
await runtime.namespace(comp_ns)
.component(comp_name)
.endpoint("begenerate")
.client()
)
engine_generator = await self.worker_client.direct(..., worker_id)
engine_generator = await self.worker_client.random(...)
engine_generator = await self.worker_client.round_robin(...)
4. More examples
There are more examples for Dynamo under python bindings.