Dangling containers after "unreserve failed" due to duplicate ServicePort

When you are reusing the port in the deployment manifest, this causes an error on the provider side leaving the dangling container due to “unreserve failed” (when the client closes the deployment).
The client won’t see anything upon “lease-status” command.
But the client is able to close the deployment.

 19   mysql:                                                                        
 20     image: mysql:5.7                                                            
 21     expose:                                                                     
 22       - port: 3306                                                              
 23         as: 3306                                                                
 24         to:                                                                     
 25           - service: php                                                        
 26       - port: 3306                                                              
 27         as: 3306 ###  <<<<<<<<<<<<<<<<< bug trigger.
 28         to:                                                                     
 29           - service: phpmyadmin    

Dangling container:

root@lamborghini:~# kubectl  get pods --all-namespaces -o wide
NAMESPACE                                       NAME                                        READY   STATUS      RESTARTS   AGE   IP               NODE          NOMINATED NODE   READINESS GATES
r9fs1sul2mlbo36oeb1ggim5a2rviak11pngdrvstntls   mysql-67f9bd6f7b-vftrn                      1/1     Running     0          12m   10.233.85.156    lamborghini   <none>           <none>

Workaround

Remove namespace manually:

kubectl delete ns r9fs1sul2mlbo36oeb1ggim5a2rviak11pngdrvstntls

Logs from the provider (only the error part):

Aug 10 09:00:17 lamborghini start-provider.sh[2946968]: E[2021-08-10|09:00:17.462] applying local service                       module=provider-cluster-kube err="Service \"mysql\" is invalid: spec.ports[1]: Duplicate value: core.ServicePort{Name:\"\", Protocol:\"TCP\", AppProtocol:(*string)(nil), Port:3306, TargetPort:intstr.IntOrString{Type:0, IntVal:0, StrVal:\"\"}, NodePort:0}" lease=akash1h24fljt7p0nh82cq0za0uhsct3sfwsfu9w3c9h/2162789/1/1/akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0 service=mysql
Aug 10 09:00:17 lamborghini start-provider.sh[2946968]: E[2021-08-10|09:00:17.462] execution error                              module=provider-cluster cmp=service cmp=deployment-manager lease=akash1h24fljt7p0nh82cq0za0uhsct3sfwsfu9w3c9h/2162789/1/1/akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0 manifest-group=westcoast state=deploy-active err="Service \"mysql\" is invalid: spec.ports[1]: Duplicate value: core.ServicePort{Name:\"\", Protocol:\"TCP\", AppProtocol:(*string)(nil), Port:3306, TargetPort:intstr.IntOrString{Type:0, IntVal:0, StrVal:\"\"}, NodePort:0}"
Aug 10 09:00:17 lamborghini start-provider.sh[2946968]: D[2021-08-10|09:00:17.462] waiting on dm.wg                             module=provider-cluster cmp=service cmp=deployment-manager lease=akash1h24fljt7p0nh82cq0za0uhsct3sfwsfu9w3c9h/2162789/1/1/akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0 manifest-group=westcoast
Aug 10 09:00:17 lamborghini start-provider.sh[2946968]: I[2021-08-10|09:00:17.463] shutdown complete                            module=provider-cluster cmp=service cmp=deployment-manager 

After the client closes the deployment:

Aug 10 09:12:03 lamborghini start-provider.sh[2946968]: E[2021-08-10|09:12:03.086] unreserve failed                             module=provider-cluster cmp=service lease=akash1h24fljt7p0nh82cq0za0uhsct3sfwsfu9w3c9h/2162789/1/1/akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0 err="not found"

Logs from the provider (complete):

Aug 10 08:57:20 lamborghini start-provider.sh[2946968]: I[2021-08-10|08:57:20.523] order detected                               module=bidengine-service order=order/akash1h24fljt7p0nh82cq0za0uhsct3sfwsfu9w3c9h/2162789/1/1
Aug 10 08:57:20 lamborghini start-provider.sh[2946968]: I[2021-08-10|08:57:20.553] group fetched                                module=bidengine-order order=akash1h24fljt7p0nh82cq0za0uhsct3sfwsfu9w3c9h/2162789/1/1
Aug 10 08:57:20 lamborghini start-provider.sh[2946968]: I[2021-08-10|08:57:20.553] requesting reservation                       module=bidengine-order order=akash1h24fljt7p0nh82cq0za0uhsct3sfwsfu9w3c9h/2162789/1/1
Aug 10 08:57:20 lamborghini start-provider.sh[2946968]: D[2021-08-10|08:57:20.553] reservation requested                        module=provider-cluster cmp=service cmp=inventory-service order=akash1h24fljt7p0nh82cq0za0uhsct3sfwsfu9w3c9h/2162789/1/1 resources="group_id:<owner:\"akash1h24fljt7p0nh82cq0za0uhsct3sfwsfu9w3c9h\" dseq:2162789 gseq:1 > state:open group_spec:<name:\"westcoast\" requirements:<signed_by:<> > resources:<resources:<cpu:<units:<val:\"100\" > > memory:<quantity:<val:\"536870912\" > > storage:<quantity:<val:\"536870912\" > > > count:1 price:<denom:\"uakt\" amount:\"100\" > > resources:<resources:<cpu:<units:<val:\"100\" > > memory:<quantity:<val:\"536870912\" > > storage:<quantity:<val:\"536870912\" > > endpoints:<> > count:1 price:<denom:\"uakt\" amount:\"100\" > > resources:<resources:<cpu:<units:<val:\"100\" > > memory:<quantity:<val:\"536870912\" > > storage:<quantity:<val:\"536870912\" > > endpoints:<> > count:1 price:<denom:\"uakt\" amount:\"100\" > > > created_at:2162792 "
Aug 10 08:57:20 lamborghini start-provider.sh[2946968]: I[2021-08-10|08:57:20.554] Reservation fulfilled                        module=bidengine-order order=akash1h24fljt7p0nh82cq0za0uhsct3sfwsfu9w3c9h/2162789/1/1
Aug 10 08:57:20 lamborghini start-provider.sh[2946968]: D[2021-08-10|08:57:20.554] submitting fulfillment                       module=bidengine-order order=akash1h24fljt7p0nh82cq0za0uhsct3sfwsfu9w3c9h/2162789/1/1 price=2uakt
Aug 10 08:57:20 lamborghini start-provider.sh[2946968]: I[2021-08-10|08:57:20.619] bid complete                                 module=bidengine-order order=akash1h24fljt7p0nh82cq0za0uhsct3sfwsfu9w3c9h/2162789/1/1
Aug 10 08:59:19 lamborghini start-provider.sh[2946968]: D[2021-08-10|08:59:19.407] ignoring group                               module=bidengine-order order=akash15yd3qszmqausvzpj7n0y0e4pft2cu9rt5gccda/1346631/1/1 group=akash1h24fljt7p0nh82cq0za0uhsct3sfwsfu9w3c9h/2162789/1
Aug 10 08:59:19 lamborghini start-provider.sh[2946968]: D[2021-08-10|08:59:19.407] ignoring group                               module=bidengine-order order=akash15egam89a3qruarxh3xe24fuuadea0ajre85meh/1965165/1/1 group=akash1h24fljt7p0nh82cq0za0uhsct3sfwsfu9w3c9h/2162789/1
Aug 10 08:59:19 lamborghini start-provider.sh[2946968]: I[2021-08-10|08:59:19.407] lease won                                    module=bidengine-order order=akash1h24fljt7p0nh82cq0za0uhsct3sfwsfu9w3c9h/2162789/1/1 lease=akash1h24fljt7p0nh82cq0za0uhsct3sfwsfu9w3c9h/2162789/1/1/akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0
Aug 10 08:59:19 lamborghini start-provider.sh[2946968]: I[2021-08-10|08:59:19.407] shutting down                                module=bidengine-order order=akash1h24fljt7p0nh82cq0za0uhsct3sfwsfu9w3c9h/2162789/1/1
Aug 10 08:59:19 lamborghini start-provider.sh[2946968]: I[2021-08-10|08:59:19.408] lease won                                    module=provider-manifest lease=akash1h24fljt7p0nh82cq0za0uhsct3sfwsfu9w3c9h/2162789/1/1/akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0
Aug 10 08:59:19 lamborghini start-provider.sh[2946968]: I[2021-08-10|08:59:19.408] new lease                                    module=manifest-manager deployment=akash1h24fljt7p0nh82cq0za0uhsct3sfwsfu9w3c9h/2162789 lease=akash1h24fljt7p0nh82cq0za0uhsct3sfwsfu9w3c9h/2162789/1/1/akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0
Aug 10 08:59:19 lamborghini start-provider.sh[2946968]: D[2021-08-10|08:59:19.408] emit received events skipped                 module=manifest-manager deployment=akash1h24fljt7p0nh82cq0za0uhsct3sfwsfu9w3c9h/2162789 data=<nil> leases=1 manifests=0
Aug 10 08:59:19 lamborghini start-provider.sh[2946968]: I[2021-08-10|08:59:19.690] data received                                module=manifest-manager deployment=akash1h24fljt7p0nh82cq0za0uhsct3sfwsfu9w3c9h/2162789 version=386dae2b84473902f7efc37154e62314395a29a2a622bc1b8582d6439d2e8bc1
Aug 10 08:59:19 lamborghini start-provider.sh[2946968]: D[2021-08-10|08:59:19.690] emit received events skipped                 module=manifest-manager deployment=akash1h24fljt7p0nh82cq0za0uhsct3sfwsfu9w3c9h/2162789 data="deployment:<deployment_id:<owner:\"akash1h24fljt7p0nh82cq0za0uhsct3sfwsfu9w3c9h\" dseq:2162789 > state:active version:\"8m\\256+\\204G9\\002\\367\\357\\303qT\\346#\\0249Z)\\242\\246\\\"\\274\\033\\205\\202\\326C\\235.\\213\\301\" created_at:2162792 > groups:<group_id:<owner:\"akash1h24fljt7p0nh82cq0za0uhsct3sfwsfu9w3c9h\" dseq:2162789 gseq:1 > state:open group_spec:<name:\"westcoast\" requirements:<signed_by:<> > resources:<resources:<cpu:<units:<val:\"100\" > > memory:<quantity:<val:\"536870912\" > > storage:<quantity:<val:\"536870912\" > > > count:1 price:<denom:\"uakt\" amount:\"100\" > > resources:<resources:<cpu:<units:<val:\"100\" > > memory:<quantity:<val:\"536870912\" > > storage:<quantity:<val:\"536870912\" > > endpoints:<> > count:1 price:<denom:\"uakt\" amount:\"100\" > > resources:<resources:<cpu:<units:<val:\"100\" > > memory:<quantity:<val:\"536870912\" > > storage:<quantity:<val:\"536870912\" > > endpoints:<> > count:1 price:<denom:\"uakt\" amount:\"100\" > > > created_at:2162792 > escrow_account:<id:<scope:\"deployment\" xid:\"akash1h24fljt7p0nh82cq0za0uhsct3sfwsfu9w3c9h/2162789\" > owner:\"akash1h24fljt7p0nh82cq0za0uhsct3sfwsfu9w3c9h\" state:open balance:<denom:\"uakt\" amount:\"5000000\" > transferred:<denom:\"uakt\" amount:\"0\" > settled_at:2162811 > " leases=1 manifests=0
Aug 10 09:00:17 lamborghini start-provider.sh[2946968]: I[2021-08-10|09:00:17.302] manifest received                            module=manifest-manager deployment=akash1h24fljt7p0nh82cq0za0uhsct3sfwsfu9w3c9h/2162789
Aug 10 09:00:17 lamborghini start-provider.sh[2946968]: D[2021-08-10|09:00:17.303] requests valid                               module=manifest-manager deployment=akash1h24fljt7p0nh82cq0za0uhsct3sfwsfu9w3c9h/2162789 num-requests=1
Aug 10 09:00:17 lamborghini start-provider.sh[2946968]: D[2021-08-10|09:00:17.303] publishing manifest received                 module=manifest-manager deployment=akash1h24fljt7p0nh82cq0za0uhsct3sfwsfu9w3c9h/2162789 num-leases=1
Aug 10 09:00:17 lamborghini start-provider.sh[2946968]: D[2021-08-10|09:00:17.303] publishing manifest received for lease       module=manifest-manager deployment=akash1h24fljt7p0nh82cq0za0uhsct3sfwsfu9w3c9h/2162789 lease_id=akash1h24fljt7p0nh82cq0za0uhsct3sfwsfu9w3c9h/2162789/1/1/akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0
Aug 10 09:00:17 lamborghini start-provider.sh[2946968]: E[2021-08-10|09:00:17.462] applying local service                       module=provider-cluster-kube err="Service \"mysql\" is invalid: spec.ports[1]: Duplicate value: core.ServicePort{Name:\"\", Protocol:\"TCP\", AppProtocol:(*string)(nil), Port:3306, TargetPort:intstr.IntOrString{Type:0, IntVal:0, StrVal:\"\"}, NodePort:0}" lease=akash1h24fljt7p0nh82cq0za0uhsct3sfwsfu9w3c9h/2162789/1/1/akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0 service=mysql
Aug 10 09:00:17 lamborghini start-provider.sh[2946968]: E[2021-08-10|09:00:17.462] execution error                              module=provider-cluster cmp=service cmp=deployment-manager lease=akash1h24fljt7p0nh82cq0za0uhsct3sfwsfu9w3c9h/2162789/1/1/akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0 manifest-group=westcoast state=deploy-active err="Service \"mysql\" is invalid: spec.ports[1]: Duplicate value: core.ServicePort{Name:\"\", Protocol:\"TCP\", AppProtocol:(*string)(nil), Port:3306, TargetPort:intstr.IntOrString{Type:0, IntVal:0, StrVal:\"\"}, NodePort:0}"
Aug 10 09:00:17 lamborghini start-provider.sh[2946968]: D[2021-08-10|09:00:17.462] waiting on dm.wg                             module=provider-cluster cmp=service cmp=deployment-manager lease=akash1h24fljt7p0nh82cq0za0uhsct3sfwsfu9w3c9h/2162789/1/1/akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0 manifest-group=westcoast
Aug 10 09:00:17 lamborghini start-provider.sh[2946968]: I[2021-08-10|09:00:17.463] shutdown complete                            module=provider-cluster cmp=service cmp=deployment-manager lease=akash1h24fljt7p0nh82cq0za0uhsct3sfwsfu9w3c9h/2162789/1/1/akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0 manifest-group=westcoast
Aug 10 09:00:17 lamborghini start-provider.sh[2946968]: I[2021-08-10|09:00:17.463] manager done                                 module=provider-cluster cmp=service lease=akash1h24fljt7p0nh82cq0za0uhsct3sfwsfu9w3c9h/2162789/1/1/akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0
Aug 10 09:00:17 lamborghini start-provider.sh[2946968]: D[2021-08-10|09:00:17.463] unreserving capacity                         module=provider-cluster cmp=service cmp=inventory-service order=akash1h24fljt7p0nh82cq0za0uhsct3sfwsfu9w3c9h/2162789/1/1
Aug 10 09:00:17 lamborghini start-provider.sh[2946968]: I[2021-08-10|09:00:17.463] attempting to removing reservation           module=provider-cluster cmp=service cmp=inventory-service order=akash1h24fljt7p0nh82cq0za0uhsct3sfwsfu9w3c9h/2162789/1/1
Aug 10 09:00:17 lamborghini start-provider.sh[2946968]: I[2021-08-10|09:00:17.463] removing reservation                         module=provider-cluster cmp=service cmp=inventory-service order=akash1h24fljt7p0nh82cq0za0uhsct3sfwsfu9w3c9h/2162789/1/1
Aug 10 09:00:17 lamborghini start-provider.sh[2946968]: I[2021-08-10|09:00:17.463] unreserve capacity complete                  module=provider-cluster cmp=service cmp=inventory-service order=akash1h24fljt7p0nh82cq0za0uhsct3sfwsfu9w3c9h/2162789/1/1

Aug 10 09:12:03 lamborghini start-provider.sh[2946968]: I[2021-08-10|09:12:03.086] deployment closed                            module=provider-manifest deployment=akash1h24fljt7p0nh82cq0za0uhsct3sfwsfu9w3c9h/2162789
Aug 10 09:12:03 lamborghini start-provider.sh[2946968]: I[2021-08-10|09:12:03.086] manager done                                 module=provider-manifest deployment=akash1h24fljt7p0nh82cq0za0uhsct3sfwsfu9w3c9h/2162789
Aug 10 09:12:03 lamborghini start-provider.sh[2946968]: I[2021-08-10|09:12:03.086] unreserving unmanaged order                  module=provider-cluster cmp=service lease=akash1h24fljt7p0nh82cq0za0uhsct3sfwsfu9w3c9h/2162789/1/1/akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0
Aug 10 09:12:03 lamborghini start-provider.sh[2946968]: D[2021-08-10|09:12:03.086] unreserving capacity                         module=provider-cluster cmp=service cmp=inventory-service order=akash1h24fljt7p0nh82cq0za0uhsct3sfwsfu9w3c9h/2162789/1/1
Aug 10 09:12:03 lamborghini start-provider.sh[2946968]: I[2021-08-10|09:12:03.086] attempting to removing reservation           module=provider-cluster cmp=service cmp=inventory-service order=akash1h24fljt7p0nh82cq0za0uhsct3sfwsfu9w3c9h/2162789/1/1
Aug 10 09:12:03 lamborghini start-provider.sh[2946968]: E[2021-08-10|09:12:03.086] unreserve failed                             module=provider-cluster cmp=service lease=akash1h24fljt7p0nh82cq0za0uhsct3sfwsfu9w3c9h/2162789/1/1/akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0 err="not found"

What providers should do now?

Now that providers can see whether they potentially have some dangling containers by searching for “unreserve failed” in the akash provider logs, they cannot immediately/easily identify which exact containers have they got in the dangling state.

Akash providers can check each namespace and whether there was an error right after that namespace in the provider logs:

root@lamborghini:~# kubectl get ns
NAME                                            STATUS   AGE
ecvg9hatnmn6hdql6gmah41geqf9mbgpaa9a61tffsddm   Active   6d1h
jiabgbmg43ffbbuu9edtmr958vu5rkjg5detd4gjqbpl0 .....

# journalctl -u akash-provider | grep -A1 jiabgbmg43ffbbuu9edtmr958vu5rkjg5detd4gjqbpl0
Aug 10 09:14:47 lamborghini start-provider.sh[2946968]: D[2021-08-10|09:14:47.944] provider/cluster/kube/builder: created service module=kube-builder service="&Service{ObjectMeta:{mysql      0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[akash.network:true akash.network/manifest-service:mysql akash.network/namespace:jiabgbmg43ffbbuu9edtmr958vu5rkjg5detd4gjqbpl0] map[] [] []  []},Spec:ServiceSpec{Ports:[]ServicePort{ServicePort{Name:0-3306,Protocol:TCP,Port:3306,TargetPort:{0 3306 },NodePort:0,AppProtocol:nil,},ServicePort{Name:1-3306,Protocol:TCP,Port:3306,TargetPort:{0 3306 },NodePort:0,AppProtocol:nil,},},Selector:map[string]string{akash.network: true,akash.network/manifest-service: mysql,akash.network/namespace: jiabgbmg43ffbbuu9edtmr958vu5rkjg5detd4gjqbpl0,},ClusterIP:,Type:ClusterIP,ExternalIPs:[],SessionAffinity:,LoadBalancerIP:,LoadBalancerSourceRanges:[],ExternalName:,ExternalTrafficPolicy:,HealthCheckNodePort:0,PublishNotReadyAddresses:false,SessionAffinityConfig:nil,IPFamily:nil,TopologyKeys:[],},Status:ServiceStatus{LoadBalancer:LoadBalancerStatus{Ingress:[]LoadBalancerIngress{},},},}"
Aug 10 09:14:47 lamborghini start-provider.sh[2946968]: E[2021-08-10|09:14:47.948] applying local service                       module=provider-cluster-kube err="Service \"mysql\" is invalid: spec.ports[1]: Duplicate value: core.ServicePort{Name:\"\", Protocol:\"TCP\", AppProtocol:(*string)(nil), Port:3306, TargetPort:intstr.IntOrString{Type:0, IntVal:0, StrVal:\"\"}, NodePort:0}" lease=akash1h24fljt7p0nh82cq0za0uhsct3sfwsfu9w3c9h/2162946/1/1/akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0 service=mysql

And then just delete that namespace:

kubectl delete namespace jiabgbmg43ffbbuu9edtmr958vu5rkjg5detd4gjqbpl0

Have just found one more zombie deployment on my provider I had to kill:

running 2 wordpress deployments

root@lamborghini:~# kubectl  get pods --all-namespaces
NAMESPACE                                       NAME                                        READY   STATUS      RESTARTS   AGE
h3ev9speqq986nrfki54h2oj432mc15dh9753g1hhdl4i   db-6b88577f95-dph2v                         1/1     Running     0          26h
h3ev9speqq986nrfki54h2oj432mc15dh9753g1hhdl4i   wordpress-86558c56c6-k995g                  1/1     Running     0          26h
kjdldis94997a84v689n7nvbjnjumm7103ovtsad3b78s   db-7c5db46774-82nzp                         1/1     Running     0          26h
kjdldis94997a84v689n7nvbjnjumm7103ovtsad3b78s   wordpress-676c9fc457-jr6mt                  1/1     Running     0          26h
...

however my provider should be running just one deployment

$ akash query market lease list --node $AKASH_NODE --provider akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0 --state=active --page 1 --limit 999999 -o json | jq -r '.leases[] | [(.lease.lease_id | .provider, .owner, .dseq, .gseq, .oseq), (.lease.price.amount|tonumber), .lease.state, (.escrow_payment.withdrawn.amount|tonumber / pow(10; 6))] | @csv' | column -s',' -t | sort -n -k6,6
"akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0"  "akash1yd0e6wkrc6h0xj75yru0eqfr7hsqlenl9jecw6"  "2334461"  1  1  4  "active"  0.058288

found in the logs the following “E” (error)

# journalctl -u akash-provider  |grep -C2 h3ev9speqq986nrfki54h2oj432mc15dh9753g1hhdl4i
Aug 22 18:32:49 lamborghini start-provider.sh[1305543]: D[2021-08-22|18:32:49.447] publishing manifest received for lease       module=manifest-manager deployment=akash1yd0e6wkrc6h0xj75yru0eqfr7hsqlenl9jecw6/2334422 lease_id=akash1yd0e6wkrc6h0xj75yru0eqfr7hsqlenl9jecw6/2334422/1/1/akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0
Aug 22 18:32:49 lamborghini start-provider.sh[1305543]: I[2021-08-22|18:32:49.448] manifest received                            module=provider-cluster cmp=service
Aug 22 18:32:49 lamborghini start-provider.sh[1305543]: D[2021-08-22|18:32:49.705] provider/cluster/kube/builder: created service module=kube-builder service="&Service{ObjectMeta:{db      0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[akash.network:true akash.network/manifest-service:db akash.network/namespace:h3ev9speqq986nrfki54h2oj432mc15dh9753g1hhdl4i] map[] [] []  []},Spec:ServiceSpec{Ports:[]ServicePort{ServicePort{Name:0-3306,Protocol:TCP,Port:3306,TargetPort:{0 3306 },NodePort:0,AppProtocol:nil,},},Selector:map[string]string{akash.network: true,akash.network/manifest-service: db,akash.network/namespace: h3ev9speqq986nrfki54h2oj432mc15dh9753g1hhdl4i,},ClusterIP:,Type:ClusterIP,ExternalIPs:[],SessionAffinity:,LoadBalancerIP:,LoadBalancerSourceRanges:[],ExternalName:,ExternalTrafficPolicy:,HealthCheckNodePort:0,PublishNotReadyAddresses:false,SessionAffinityConfig:nil,IPFamily:nil,TopologyKeys:[],},Status:ServiceStatus{LoadBalancer:LoadBalancerStatus{Ingress:[]LoadBalancerIngress{},},},}"
Aug 22 18:32:50 lamborghini start-provider.sh[1305543]: D[2021-08-22|18:32:50.100] provider/cluster/kube/builder: created service module=kube-builder service="&Service{ObjectMeta:{wordpress      0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[akash.network:true akash.network/manifest-service:wordpress akash.network/namespace:h3ev9speqq986nrfki54h2oj432mc15dh9753g1hhdl4i] map[] [] []  []},Spec:ServiceSpec{Ports:[]ServicePort{ServicePort{Name:0-80,Protocol:TCP,Port:80,TargetPort:{0 80 },NodePort:0,AppProtocol:nil,},},Selector:map[string]string{akash.network: true,akash.network/manifest-service: wordpress,akash.network/namespace: h3ev9speqq986nrfki54h2oj432mc15dh9753g1hhdl4i,},ClusterIP:,Type:ClusterIP,ExternalIPs:[],SessionAffinity:,LoadBalancerIP:,LoadBalancerSourceRanges:[],ExternalName:,ExternalTrafficPolicy:,HealthCheckNodePort:0,PublishNotReadyAddresses:false,SessionAffinityConfig:nil,IPFamily:nil,TopologyKeys:[],},Status:ServiceStatus{LoadBalancer:LoadBalancerStatus{Ingress:[]LoadBalancerIngress{},},},}"
Aug 22 18:32:50 lamborghini start-provider.sh[1305543]: D[2021-08-22|18:32:50.366] provider/cluster/kube/builder: created rules module=kube-builder rules="[{Host:YOURDOMAIN.COM IngressRuleValue:{HTTP:&HTTPIngressRuleValue{Paths:[]HTTPIngressPath{HTTPIngressPath{Path:/,Backend:IngressBackend{Resource:nil,Service:&IngressServiceBackend{Name:wordpress,Port:ServiceBackendPort{Name:,Number:80,},},},PathType:*Prefix,},},}}} {Host:ee9f12bva98v34skc3icoou0ck.provider.akash.nixaid.com IngressRuleValue:{HTTP:&HTTPIngressRuleValue{Paths:[]HTTPIngressPath{HTTPIngressPath{Path:/,Backend:IngressBackend{Resource:nil,Service:&IngressServiceBackend{Name:wordpress,Port:ServiceBackendPort{Name:,Number:80,},},},PathType:*Prefix,},},}}}]"
Aug 22 18:32:50 lamborghini start-provider.sh[1305543]: E[2021-08-22|18:32:50.371] applying ingress                             module=provider-cluster-kube err="Ingress.extensions \"wordpress\" is invalid: spec.rules[0].host: Invalid value: \"YOURDOMAIN.COM\": a lowercase RFC 1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character (e.g. 'example.com', regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?(\\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*')" lease=akash1yd0e6wkrc6h0xj75yru0eqfr7hsqlenl9jecw6/2334422/1/1/akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0 service=wordpress expose="{Port:80 ExternalPort:0 Proto:TCP Service: Global:true Hosts:[YOURDOMAIN.COM]}"

Found relevant DSEQ = 2334422.

killed the leftover

root@lamborghini:~# kubectl delete ns h3ev9speqq986nrfki54h2oj432mc15dh9753g1hhdl4i
namespace "h3ev9speqq986nrfki54h2oj432mc15dh9753g1hhdl4i" deleted

I’ve opened an issue on Github Dangling containers when manifest's got an error · Issue #1353 · ovrclk/akash · GitHub

Nice find, thanks for filing a GitHub issue!

1 Like