MCORD Installation Issues


#1

Hi All,

Reference Link :- https://guide.opencord.org/profiles/mcord/install.html

Server Details :-
RAM - 96 GB
CPUs - 2 X Octacore Intel® Xeon® CPU E5-2690 0 @ 2.90GHz (32 vCPUs)
OS - Ubuntu 16.04

  1. On one server in our lab, I tried to configure mcord using the convenient script “mcord-in-a-box.sh”.

  2. Actually, the script output is displayed on the console. After the script execution completed, based on the available console output, it seemed that there were no issues.

Note:- But, looks like when Jimmy executed the same script on one of his server, he saw some issues in the console output.

  1. Post convenient script execution, I had to perform few validation checks and I observed the below issues.

3.1. VTN nodes initializing command fails. Please see below itself.

biarca@ubuntu:~/cord/helm-charts$ ssh -p 8101 onos@onos-cord-ssh.default.svc.cluster.local cordvtn-nodes
Password authentication
Password:
Command not found: cordvtn-nodes

3.2. I see only one “Cirros 0.3.5 64-bit” image when I execute “openstack image list”. Where there should be more.

3.3. Bad request error when trying to create vepc service for mcord. Below is the output.

http -a admin@opencord.org:letmein POST http://xos-gui.default.svc.cluster.local:4000/xosapi/v1/vepc/vepcserviceinstances blueprint=mcord_5 site_id=1

HTTP/1.1 400 Bad Request
Connection: keep-alive
Content-Type: text/html
Date: Wed, 01 Aug 2018 13:13:16 GMT
Server: nginx/1.13.12
Transfer-Encoding: chunked

{“fields”: {}, “specific_error”: “Cannot find eligible owner of class VEPCService”, “error”: “XOSValidationError”}

There is not much help when I try to google on the above error. Also, I am not seeing any logs on the “xos-gui” and “mcord-epc-service” pods when I execute the above command.

3.4. Only management network is displayed when I execute “openstack network list”.

3.5. Nothing is displayed when I execute “openstack server list --all-projects”.

3.6. I observed issues while accessing UI. Then I raised the same with cordev community. One Mr. Andy provided a workaround and after following it, I could access UI.

3.7. Will continue to work on the issues and keep you all updated with the progress.

  • Manoj

#2

Today, I performed mcord installation on a fresh setup again and I too observed the “containers failed to start.” message. Upon debugging, I figured out that as part of convenient script execution, a job named mcord-tosca-loader is started, but it failed. Actually, the job is creating a pod with naming convention “mcord-tosca-loader-”. I started a new mcord-tosca-loader job and it was successful. Below is the output.

biarca@ubuntu:~$ kubectl get jobs
NAME DESIRED SUCCESSFUL AGE
base-openstack-tosca-loader 1 1 4h
mcord-mcord-subscriber 1 1 4h
mcord-tosca-loader 1 0 4h
mcord-tosca-loader-test 1 1 1h ----- I created it
mcord-tosca-loader-test2 1 1 16m ----- I created it

Even after this, I continue to see the validation errors, which I mentioned earlier.

Best Regards,
Manoj


#3

I raised the issue with Cord-dev community on Friday (3rd Aug 2018) and below is the link.

https://groups.google.com/a/opencord.org/forum/#!topic/cord-dev/oJEvOYTyJho

  • Manoj

#4

On my single node m-cord cluster, I currently have the below issues.

  1. When I execute the below command, I see a message “Command not found cordvtn-nodes”
    ssh -p 8101 onos@onos-cord-ssh.default.svc.cluster.local cordvtn-nodes

  2. In openstack, I see only 2 networks - management & sgi_network. Where as the below should be displayed.

    s11_network
    management
    s6a_network
    spgw_network
    flat_network_s1u
    db_network
    sgi_network
    flat_network_s1mme

  3. No servers (instances or VMs) were created in Openstack. Where as the below 5 should have been created.

    mysite_vmme-2
    mysite_vspgwu-1
    mysite_hssdb-5
    mysite_vhss-4
    mysite_vspgwc-3

The issue seems to be with the Openstack helm configuration. We will dig more in that angle and update here.

  1. In the UI, now I see lots of information, which also includes ServiceInstance notifications (both successful and failures).

  2. Had a call with Jimmy yesterday and showed him the current state of the set.

Thanks & Regards,
Manoj


#5

The issue seems to be with openstack-helm. To root cause the issue, we tried just openstack-helm installation (from official openstack site also) on a VM & also on a baremetal and we see the below issues, which we are seeing on the m-cord server also.

  1. The host on which QEMU was configured is not listed. Below is the output.

    biarca@ubuntu:~/cord$ openstack hypervisor list
    ±—±--------------------±----------------±--------±------+
    | ID | Hypervisor Hostname | Hypervisor Type | Host IP | State |
    ±—±--------------------±----------------±--------±------+
    | 1 | ubuntu | QEMU | None | up |
    ±—±--------------------±----------------±--------±------+

  2. When we try to create an instance, it fails with the error “No valid host was found. There are not enough hosts available.”.

After good amount of googling, I posted the issue in CORD’s slack and #openstack-helm slack. One person responded in CORD slack saying he is also seeing the same issue and looking for help. No responses in the #openstack-helm slack yet.

Thanks & Regards,
Manoj


#6

We are also pursuing OpenStack Ansible and I am sure we will have better success since this is a “known beast” :smile: Once OpenStack Ansible is operational we will look into the Integration aspect.