Ticket #355 (closed task: fixed)

Opened 5 years ago

Last modified 5 years ago

Testing 5.0 on RCI rack

Reported by: ibaldin Owned by: yxin
Priority: major Milestone:
Component: External: Testing and Redeployment Version: baseline
Keywords: Cc: yxin, anirban, pruth, vjo, jonmills, claris

Description (last modified by ibaldin) (diff)

This ticket takes over from #341. We are concentrating on

1. Controller recovery (#341, #345, #349, #351, #352)

2. Bare-metal node and storage (#359)

3. Hybrid mode (#350)

4. SM recovery issue (#358)

5. Core accounting issue (#364)

6. Static VLAN issue (#362)

7. Multiple storage volumes in a slice (#365)

Change History

  Changed 5 years ago by ibaldin

4. Need to figure out indexing strategy for mysql

  Changed 5 years ago by ibaldin

There is some new code checked in, however Yufeng is still looking at some parts of recovery. Specifically, after 2 recoveries certain things don't come back correctly (some interfaces are erroneously generated with new names). He is looking to fix that.

  Changed 5 years ago by pruth

I'm trying to look at storage for baremetal on the RCI rack. It seems that that cannot allocate any baremetal nodes. It always says that there are none available.

I checked pequod and it says that no slivers are on the rack.

I restarted and then clean restarted all of the actors and I still get this error.

Does anyone have any ideas about this? Can someone fix it?

Paul

  Changed 5 years ago by ibaldin

That's actually good, this means we can close ticket #342. To get around this we need to modify config.xml to shift the delegation of bare-metal nodes from exo broker to local broker.

  Changed 5 years ago by pruth

I added code to the xcat handler push the script to the node that will handle attaching storage. I tested it in emulation by printing the script to the log and it seems ok. I can't test this for real until we can create baremetal nodes on rci.

That said, I suspect that there will be a problem with the network ifaces. When I create a baremetal node with storage on one of the production racks, I do not see the network configuration in the configuration script. I suspect that the storage network information is never passed to the xcat handler. Further, baremetal nodes that have dataplane networks are passed the network information using the "hosteth" technique that we used before we had quantum. We need to pass the "data" or "storage" as hosteth and add a property for each in the site properties.

In summary, I'm stuck until we can create baremetal nodes.

Paul

  Changed 5 years ago by pruth

Baremetal is working on RCI now (thanks Yufeng). Working on baremetal with storage now.

  Changed 5 years ago by anirban

There is an issue with vms not coming up with the dataplane interfaces. This has been tracked down to a property incorrectly being passed to the handler. yxin is looking into it. Holding off topology testings until that is fixed.

- Anirban

  Changed 5 years ago by yxin

r6745 fixed the data plane interface name problem, also fixed a manifest problem for mesh inter-rack topology, all caused by using the wrong model for the resources.

  Changed 5 years ago by ibaldin

Not exactly an RCI test- this is in emulation, but here is what I observed

For MP slice with a node groups (5-nodes) in RCI, BBN and a single node at OSF

1. Recovery of controller assigned labels seems to work

2. Recovery of the slice doesn't work - Flukes complains of not being able to cast OrcaNode? into OrcaCrossconnect? - Yufeng said he understands the issue (it is with the model) and is fixing it

3. Regardless of recovery the single OSF node sometimes appears disconnected in the manifest.

4. Regardless of recovery NLR node appears not to have state or reservation information associated with it.

  Changed 5 years ago by anirban

Issues we are working on right now:

1. Modify after recovery: After recovery addition and deletion of nodes doesn't work. Three things happen - (a) Addition of multiple nodes doesn't happen at all. Addition of a single node changes state of an existing node to Ticketed. (b) When a single node is added, the ec2 handler doesn't get imageproxy properties and bombs out without throwing the right error code; hence the Ticketed state..

2. IP address on storage interfaces absent.

  Changed 5 years ago by anirban

Update:

The ec2 handler is returning correct exit code on imageproxy failure.

There is an issue of deleted nodes reappearing in closed state in the manifest after recovery. Here is the scenario. Run slice with nodegroup with 2 nodes. Add 2 nodes, delete one of the old nodes. Restart everything. Querying manifest is fine at this point. Try to add one node. When you query manifest now, even if the new node failed to come up (because of problem specified in the previous update), the manifest now starts showing that the old deleted node is in "closed" state. Ideally, it should have shown it in "Failed" state.

  Changed 5 years ago by ibaldin

In reply to my own status 2 entries above:

Yufeng says casting issue should be handled - committed not tested.

Disconnected nodes are a flukes issue - it is a problem for all inter-domain connections. It is fixed in my local code, however it still isn't working for the stitch ports, I'm investigating (stitchports appear disconnected)

NLR state is still for Yufeng to look at.

  Changed 5 years ago by ibaldin

In reply to my own status 3 entries above about inter-MP slices

Issue 2 is resolved - recovery appears to fully work now as best I can tell - labels and all

Issue 3 is resolved in Flukes

Issue 4: NLR node does not have state or other reservation information - still a problem. This is *regardless* of recovery.

  Changed 5 years ago by ibaldin

Added a new rule to the request checker - Stitchport cannot have an IP address assigned to it. Yay rule engine!

follow-up: ↓ 16   Changed 5 years ago by yxin

Anirban,

I checked in a fix for the modify/recover/modify state and the associated interface problem. Pls give it a try.

in reply to: ↑ 15   Changed 5 years ago by anirban

Yufeng,

I tried the same test for modify. Run slice with one node and nodegroup with 2 nodes. Add 2 nodes, delete one of the old nodes. All interfaces appear fine. Restart everything. Querying manifest is fine at this point. Interfaces and slice fine. Add one node. Now, the new node fails because of the following. The new node shows correct name, with correct interfaces in "View properties". The handler correctly fails now and "query manifest" shows the node in Failed state. So, naming issues seem to be have been fixed. The issue is that after recovery the imageproxy properties are not passed to the handler for newly added node.

Controller did not pass image proxy properties ${config.image.url} or ${config.image.guid}, using default emi ami-00000009

Paul,

Can you please check storage with bare metal and vms on rci rack now. Check if al data and storage interfaces come up and also test recovery for that slice.

Regards,
- Anirban

Replying to yxin:

Anirban,

I checked in a fix for the modify/recover/modify state and the associated interface problem. Pls give it a try.

  Changed 5 years ago by pruth

Storage isn't working at all anymore. The problem seems to be that the 1009 vlan isn't on the vlan half of the switch OR that the 1009 vlan is not plumbed to the port that leads to the baremetal nodes. I manually tried to configure both of the ports of a baremetal node to contact the iSCSI device and neither was able to ping the iSCSI device. I'm not sure how to check this.

There was anther problem with the data plane for baremetal nodes. I changed the xcat site properties file to have the following line:
xcat.interface.map=data:p2p2,storage:p2p2

This needs to be set for all racks that will be configured this way. Probably we need to add the openflow network to p2p1 as well.

Paul

  Changed 5 years ago by ibaldin

I believe the correct network names are vlan-data, of-data and vlan-storage (as decided by Jonathan and Victor). The RDF should be referencing those names as well. On Monday we will make sure this is all consistent, but this is the likely cause of the storage problem.

follow-up: ↓ 20   Changed 5 years ago by pruth

The problem is not with the workers. Even a baremetal node configured by hand cannot ping the iSCSI device. I tried configuring each of the NICs by hand and cannot ping the iSCSI device over VLAN 1009.


in reply to: ↑ 19   Changed 5 years ago by jonmills

Replying to pruth:

The problem is not with the workers. Even a baremetal node configured by hand cannot ping the iSCSI device. I tried configuring each of the NICs by hand and cannot ping the iSCSI device over VLAN 1009.


I attempted at least to log into 8264.renci.xo to investigate the port configs for baremetal, but I was unable to log in with any passwords I know of. It may have to wait until Monday for a serial connection, or to pick Chris's brain about it.

  Changed 5 years ago by ibaldin

There is a new flukes (0.5) that comes from the url I've given to most of you (http://geni-images.renci.org/webstart/0.5-SNAPSHOT/flukes.jnlp), however it has been broken into two separate JNLPs, one main and one extension that downloads bouncycastle jars. This was needed for obscure reasons of signing jars and Victor's strange setup on his laptop.

You will see now that two jnlp's are downloaded - the second one called bcprov.jnlp automatically triggers when you try to start Flukes. Nothing to worry about.

  Changed 5 years ago by ibaldin

The issue with Stitch-Port path visualization appears to be related to StitchPort? not being in the NetworkConnection? element. Here is an example of a valid point-to-point inter-domain path (number of elements should always be odd):

http://geni-orca.renci.org/owl/bbnNet.rdf#bbnNet/Domain/vlan/c9ca290a-ddc5-45a7-83ee-f6013221467f/vlan
http://geni-orca.renci.org/owl/bbnNet.rdf#BbnNet/IBM/G8052/GigabitEthernet/1/0/ethernet-AL2S/BBN/Juniper/MX/TenGigabitEthernet/1/1/ethernet
http://geni-orca.renci.org/owl/ion.rdf#ion/Domain/vlan/75facd94-29cf-4d97-ad6f-0b6f6b8903c7/vlan
http://geni-orca.renci.org/owl/ion.rdf#ION/DD/Cisco/6509/TenGigabitEthernet/1/2/ethernet-NLR/DD/Juniper/QFX3500/xe/0/0/1/ethernet
http://geni-orca.renci.org/owl/nlr.rdf#nlr/Domain/vlan/2b4079a0-4c9c-4c26-b03a-a2e282bad6fa/vlan
http://geni-orca.renci.org/owl/nlr.rdf#NLR/DD/Juniper/QFX3500/xe/0/0/0/fiber-Renci/Cisco/6509/TenGigabitEthernet/3/2/fiber
http://geni-orca.renci.org/owl/ben.rdf#ben/Domain/vlan/1e850a03-84a9-4801-aadd-f87efda0635c/vlan
http://geni-orca.renci.org/owl/ben-6509.rdf#Renci/Cisco/6509/TenGigabitEthernet/3/6/ethernet-RciNet/IBM/G8264/TenGigabitEthernet/1/0/ethernet
http://geni-orca.renci.org/owl/rciNet.rdf#rciNet/Domain/vlan/c62d3205-6c06-4743-bbed-4806ee323fe5/vlan

And here is a a path between OSG RENCI stitch port and BBN (notice the even number of entries and the missing stitchport):
http://geni-orca.renci.org/owl/bbnNet.rdf#bbnNet/Domain/vlan/eb2080ff-3b07-4fe2-8e16-b4438834bec2/vlan
http://geni-orca.renci.org/owl/ion.rdf#AL2S/BBN/Juniper/MX/TenGigabitEthernet/1/1/ethernet-BbnNet/IBM/G8052/GigabitEthernet/1/0/ethernet
http://geni-orca.renci.org/owl/ion.rdf#ion/Domain/vlan/f54def51-c75a-4857-8c5a-3e3468bee502/vlan
http://geni-orca.renci.org/owl/nlr.rdf#NLR/DD/Juniper/QFX3500/xe/0/0/1/ethernet-ION/DD/Cisco/6509/TenGigabitEthernet/1/2/ethernet
http://geni-orca.renci.org/owl/nlr.rdf#nlr/Domain/vlan/91e24001-e521-4686-8b56-974920b0ff8a/vlan
http://geni-orca.renci.org/owl/ben-6509.rdf#Renci/Cisco/6509/TenGigabitEthernet/3/2/fiber-NLR/DD/Juniper/QFX3500/xe/0/0/0/fiber
http://geni-orca.renci.org/owl/ben.rdf#ben/Domain/vlan/3bf13b9b-7a59-4e4f-8bf6-9782f32cf4c2/vlan
http://geni-orca.renci.org/owl/2ec74aec-5090-42fe-b6b7-dc8439875786#StitchPort0/intf-Renci/Cisco/6509/TenGigabitEthernet/3/4/ethernet
~

  Changed 5 years ago by ibaldin

Latest status:

1. Yufeng reports no known problems with modify (needs testing)

2. Inter-domain slices have some problems with visualization with stitchports (see above)

3. Some inter-domain slices are running into request validator rules (seems a new problem, rules have been in place for a while)

4. Storage (especially with bm) needs more testing

5. Hybrid mode needs more testing

  Changed 5 years ago by ibaldin

  • description modified (diff)

  Changed 5 years ago by ibaldin

  • description modified (diff)

Summary for 09/09:

1. Tested storage for VMs and bare-metal nodes. After some twiddling with rack configuration reflected in the release notes got storage to work with VMs and bare-metal, so this issue is done.

2. Tested QoS support in VLANs - does not seem to work. Measuring between BM and VM, from BM to VM can get as much as 5Gbps, while the other way the QoS is respected, probably due to OVS metering. Twiddling switch configuration gave no effect, Chris is contacting the vendor for explanation.

At issue appears to be ways of associating a VMAP with a VLAN and port types (server-ports vs. non-server-ports). Changing port types or not specifying the type of vmap port did not produce any result.

Also uncovered issue #359 with bare-metal resource accounting.

  Changed 5 years ago by pruth

5 Gbps sounds a lot like the limit of what can be received by a VM using a tap device. The QoS from VM to BM is being policed the OVS ingress policy on the VMs host.

This makes me think that QoS on the switch isn't working at all.

Paul


  Changed 5 years ago by ibaldin

@Paul - yes this is the working assumption. We don't know why yet, but Chris will contact IBM.

  Changed 5 years ago by ibaldin

The issue with stitchport visualisation in flukes has been resolved in the newly reployed flukes 0.5-snapshot (you all should have the URL for it).

Summary of known issues:

1. BM node accounting (#359)

2. QoS provisioning in the 8264 (IBM has already replied; not a showstopper)

3. SM unable to recover ticketed reservations (#358; technically not a showstopper)

Things still to be tested

1. Hybrid mode

2. Recovery with modify

3. General embedding in one rack (regression tests)

I will ask ops guys to prepare for converting a small set of racks to ORCA5 next week (likely to be UCD, UH and FIU or UFL - speak up if you have better candidates).

  Changed 5 years ago by ibaldin

  • cc claris added

  Changed 5 years ago by ibaldin

Hybrid mode works as reflected in ticket #350.

One more issue needs to be addressed - attaching VMs to static VLANs. Waiting for Yufeng to respond.

  Changed 5 years ago by ibaldin

Remaining issues

1. VMs and static VLANs - Chris needs to plumb vlan 1750 into the hybrid part (make the port of the vlan side that serves as uplink to OF side be a member of this vlan). Then test that if an OF reservation is requested and a VM is attached to VLAN 1750 it happens to connect via the OF port on worker node. Should work, but hasn't been tested.

2. QoS provisioning is broken after the switch firmware update. I will turn it off in the next patch for now, until we figure out with IBM what to do. Will leave VLANs as best effort for now.

3. Ticket #358 still require investigation (this is for me)

4. Ticket #351 - there is a minor issue with not being able to extend modified slices after recovery (also mine). The other issues with modified have been fixed.

5. DAR - needs to be enabled on RCI

On monday Jonathan will start upgrading TAMU, UCD and WVN. He will need help modifying config.xml and RDF (for hybrid) for those sites.

  Changed 5 years ago by ibaldin

QoS provisioning has been disabled, VLAN provisioning works with the new firmware. We will wait for IBM to fix this (or tell us how to do it right), but it isn't a showstopper.

  Changed 5 years ago by ibaldin

I believe #351 is fixed on r6803, needs testing. Also added code to report true reservation expiration in the GENI SliverStatus? call, instead of reporting slice expiration time, which is becoming irrelevant. Worth testing GENI AM API especially after extend.

  Changed 5 years ago by ibaldin

  • description modified (diff)

  Changed 5 years ago by anirban

Ticket #351 has been updated with the issue of interfaces not getting correct network names on modify after recovery.

Ilya is investigating how to report the most accurate slice end times for "sliverstatus" call to users when they are using GENI tools.

I am not testing anything else on RCI now.

Claris will test DAR on RCI tonight.

  Changed 5 years ago by ibaldin

GENI reporting should be fixed in r6806

  Changed 5 years ago by anirban

GENI reporting tested. It's working as expected now.

  Changed 5 years ago by pruth

See ticket 364. VM cores accounting is not being handled correctly.

  Changed 5 years ago by ibaldin

  • description modified (diff)

  Changed 5 years ago by ibaldin

Core accounting appears to work as designed, although it is less than ideal, because the controller is unable to reject certain types of requests outright, and SM satisfied partial successes, so the user may find out that something failed only as part of manifest query.

  Changed 5 years ago by ibaldin

Bare-metal nodes aren't working - both w9 and w10 refuse to come up failing ping test.

  Changed 5 years ago by ibaldin

Bare-metal nodes are working now. Updated xCat handler to attempt rebooting if fails the first try r6817. Also added sleep to iSCSI process in bare-metal nodes to make storage attachment more reliable r6820.

  Changed 5 years ago by ibaldin

I believe the last ORCA issue remaining with RCI rack is DAR (Claris to test more)

The last non-ORCA issue is the behavior of meso-scale (Chris is testing that)

After those two we can reopen.

  Changed 5 years ago by claris

I am testing DAR on RCI this morning. I am seeing some issues with replication (TAMU->WSU) that I need to check. I'll be reporting back soon.

  Changed 5 years ago by claris

Bad assumption made on my test case. Everything is fine with replication TAMU->WSU and WSU->TAMU.
I am done testing on RCI. Next is to test on UCD, TAMU and WVN simultaneously.

  Changed 5 years ago by ibaldin

Prior to anticipated return of RCI rack back into the pool (via local controller) of available racks, we should remember to revert the config.xml change delegating bare-metal nodes to rack-local broker.

  Changed 5 years ago by ibaldin

  • description modified (diff)

One more issue cropped up in detailed testing (#365)

  Changed 5 years ago by ibaldin

  • status changed from new to closed
  • resolution set to fixed

RCI maintenance complete modulo remaining two issues - static meso-scale vlans and SM recovery.

Note: See TracTickets for help on using tickets.