I’m finishing up my GUI app (dashboard photo below to get excited about) for hosts to easily deploy Akash clusters and found what seems like a bug.
I’m running into the issue in the provider when I place a bid manually (placing the bid outside of the provider auto-bidder). Basically when the bid gets accepted/lease is created the provider either wont pick up the lease or wont find the manifest. I have some pretty detailed notes below:
How to replicate:
- In a terminal (lets call it client CLI for naming purposes) : Create an order for providers to bid on
- On the host CLI: start up the provider, and for the sake of this bug: set your price per milliCPU to something that won’t bid on the order you created (aka: 1000000uakt)
- After the provider is up/synced, in a separate host-side terminal: create a new bid on the client CLI order for a sane price.
- On the client CLI: you’ll see the bid come in, and then create a lease for it.
- The lease tx will go thru successfully on the client CLI
- Now when the client CLI sends a manifest, the client receives an error and the provider receives the manifest, however claims it does not have a lease (despite you can clearly lookup open leases for the provider in the CLI outside the provider).
- On restart of the provider it gets weirder: 1 open lease, 0 manifests found. Now when the client CLI sends the manifest, the provider sees it come in, however throws an error “error looking up manifest : not found“
Some specifics on the session:
Provider Log when it syncs (indicating host prices are too high for my targeted order) :
I[2021-08-25|22:11:38.155] Price too high, not bidding module=bidengine-order order=akash1fnc04mjln6y0y7qgkkz9nwkjane50nnjxq32yf/2381629/1/1 price=10000000uakt max-price=3000uakt
After client sends manifest:
- Client receives error:
error: remote server returned 404
Error: submit manifest to some providers has been failed
Host logs for manifest: Provider receives manifest, however doesn’t have a lease for it:::
I[2021-08-25|22:24:30.411] manifest received module=manifest-manager deployment=akash1fnc04mjln6y0y7qgkkz9nwkjane50nnjxq32yf/2381629
D[2021-08-25|22:24:30.411] emit received events skips due to no leases module=manifest-manager deployment=akash1fnc04mjln6y0y7qgkkz9nwkjane50nnjxq32yf/2381629 data= leases=0 manifests=0
I[2021-08-25|22:24:30.568] data received module=manifest-manager deployment=akash1fnc04mjln6y0y7qgkkz9nwkjane50nnjxq32yf/2381629 version=334c0e8a8d9b593fbbe9167ad8798dc14fc11e5cae2065a6f2e4663dea551d17
D[2021-08-25|22:24:30.568] emit received events skips due to no leases module=manifest-manager deployment=akash1fnc04mjln6y0y7qgkkz9nwkjane50nnjxq32yf/2381629 data="deployment:<deployment_id:<owner:“akash1fnc04mjln6y0y7qgkkz9nwkjane50nnjxq32yf” dseq:2381629 > state:active version:“3L\016\212\215\233Y?\273\351\026z\330y\215\301O\301\036\\\256 e\246\362\344f=\352U\035\027” created_at:2381631 > groups:<group_id:<owner:“akash1fnc04mjln6y0y7qgkkz9nwkjane50nnjxq32yf” dseq:2381629 gseq:1 > state:open group_spec:<name:“earth-lab0” requirements:<signed_by:<> > resources:<resources:<cpu:<units:<val:“100” > > memory:<quantity:<val:“536870912” > > storage:<quantity:<val:“536870912” > > endpoints:<> > count:1 price:<denom:“uakt” amount:“3000” > > > created_at:2381631 > escrow_account:<id:<scope:“deployment” xid:“akash1fnc04mjln6y0y7qgkkz9nwkjane50nnjxq32yf/2381629” > owner:“akash1fnc04mjln6y0y7qgkkz9nwkjane50nnjxq32yf” state:open balance:<denom:“uakt” amount:“5000000” > transferred:<denom:“uakt” amount:“0” > settled_at:2381760 > " leases=0 manifests=0
And after restarting the provider: Now it sees the lease but no manifest.
I[2021-08-25|22:27:36.789] found existing leases module=provider-manifest count=1
I[2021-08-25|22:27:36.789] new lease module=manifest-manager deployment=akash1fnc04mjln6y0y7qgkkz9nwkjane50nnjxq32yf/2381629 lease=akash1fnc04mjln6y0y7qgkkz9nwkjane50nnjxq32yf/2381629/1/1/akash1mqnj2euks0aq82q0f2tknz6kua6zdfn97kmvhj
D[2021-08-25|22:27:36.789] emit received events skipped module=manifest-manager deployment=akash1fnc04mjln6y0y7qgkkz9nwkjane50nnjxq32yf/2381629 data= leases=1 manifests=0
I[2021-08-25|22:27:36.947] data received module=manifest-manager deployment=akash1fnc04mjln6y0y7qgkkz9nwkjane50nnjxq32yf/2381629 version=334c0e8a8d9b593fbbe9167ad8798dc14fc11e5cae2065a6f2e4663dea551d17
And when the client sends the manifest, it sees it as a pass, however the provider logs an error: E[2021-08-25|22:28:45.947] error looking up manifest module=provider-cluster cmp=service err=“not found” lease=akash1fnc04mjln6y0y7qgkkz9nwkjane50nnjxq32yf/2381629/1/1/akash1mqnj2euks0aq82q0f2tknz6kua6zdfn97kmvhj group-name=earth-lab0