Ticket #360 (closed defect: fixed)

Opened 5 years ago

Last modified 5 years ago

ExoLayer ORCA 5 testing

Reported by: ibaldin Owned by: ibaldin
Priority: major Milestone:
Component: Don't Know Version: baseline
Keywords: Cc: anirban, yxin, pruth, claris, jonmills, vjo

Description

Need to resize VM and JVMs on geni.renci.org (controller and SM) and update all relevant databases on geni, geni2, geni-net, geni-ben and deploy ORCA5.

Need to test

1. Recovery of ION AM

2. Recovery of DD/NLR AM

3. Recovery of BEN AM

4. Recovery of controller with inter-domain slices

5. Extend on ION circuits

Change History

  Changed 5 years ago by ibaldin

geni-net.renci.org (ndl-broker, es-net) and old geni.renci.org (exo-sm and controller) have been coalesced into a single Centos6.5 host also called geni.renci.org. Es-net actor has been disabled.

There is a CNAME entry pointing from geni-net to geni.

Actor registry and image registry moved to port 15080 (from 12080 on the old host).

  Changed 5 years ago by ibaldin

JVM sizes set to 2G for broker and SM containers and 10G for controller.

  Changed 5 years ago by ibaldin

topology-embedding-sm has been renamed to 'exo-sm'. Both ndl-broker and exo-sm are running and responding to pequod. Broker appears to have formed edges to AMs already in DAR.

  Changed 5 years ago by ibaldin

We'll hold off on returnung TAMU and WVN to users, so we can have two racks to try the full range of things on. Once point-to-point is happy, we'll transition to MP by adding a couple more racks (RCI, UCD) to ndl-broker.

  Changed 5 years ago by ibaldin

TAMU to WVN comes up all active, but traffic doesn't pass, Chris is looking into it.

  Changed 5 years ago by ibaldin

Traced it to a misconfigured upstream port on TAMU switch

  Changed 5 years ago by ibaldin

Testing simple inter-domain.

Context - test the recovery of AM NET controls (VlanControl? and NdlInterfaceVlanControl?) in individual racks, ION control running on geni2.renci.org and controller running on geni.renci.org

What to test for - primarily leakage of tags in AM controls as well as ability to restore general slice state in ION control. The controller does issue tags for p-to-p connections - instead they are issued by controls and passed to ION for stitching. Controller is expected to preserve information about slice state and allow deletion and modification of slices after recovery.

follow-up: ↓ 10   Changed 5 years ago by ibaldin

After restarting ION actor, handler does not execute LEAVE for deleted slices.

  Changed 5 years ago by ibaldin

Despite changing blowhole configuration on control, can't see exo-sm slices in history db.

in reply to: ↑ 8   Changed 5 years ago by ibaldin

Replying to ibaldin:

After restarting ION actor, handler does not execute LEAVE for deleted slices.

THis needs further testing - appeared to work after all except for one vlan.

  Changed 5 years ago by ibaldin

Encountered what looks like TDB corruption issue on slice recovery (of a p-to-p slice that was not modified or extended and was recovered several times sucessfully before). The problem manifests by a large number of exceptions

INFO   | jvm 1    | 2014/09/24 16:17:09 | 2014-09-24 16:17:08,930 [WrapperSimpleAppMain] INFO  controller.orca.controllers.xmlrpc.XmlrpcOrcaState - Opening the request model /etc/orca/controller-11080//modelState/controller/request-6f9b2dda-aea1-4313-9948-6ceae431711b
INFO   | jvm 1    | 2014/09/24 16:17:09 | 2014-09-24 16:17:08,939 [WrapperSimpleAppMain] INFO  controller.orca.controllers.xmlrpc.XmlrpcOrcaState - Opening the manifest model /etc/orca/controller-11080//modelState/controller/manifest-6f9b2dda-aea1-4313-9948-6ceae431711b
INFO   | jvm 1    | 2014/09/24 16:17:09 | 2014-09-24 16:17:09,295 [WrapperSimpleAppMain] DEBUG controller.orca.controllers.xmlrpc.XmlrpcOrcaState - Slice bound elements are
INFO   | jvm 1    | 2014/09/24 16:17:09 | java.lang.NullPointerException
INFO   | jvm 1    | 2014/09/24 16:17:09 |       at java.util.Calendar.setTime(Calendar.java:1106)
INFO   | jvm 1    | 2014/09/24 16:17:09 |       at orca.ndl.elements.OrcaReservationTerm._setEnd(OrcaReservationTerm.java:59)
INFO   | jvm 1    | 2014/09/24 16:17:09 |       at orca.ndl.elements.OrcaReservationTerm.setStart(OrcaReservationTerm.java:83)
INFO   | jvm 1    | 2014/09/24 16:17:09 |       at orca.embed.workflow.RequestWorkflow.recover(RequestWorkflow.java:328)
INFO   | jvm 1    | 2014/09/24 16:17:09 |       at orca.embed.workflow.RequestWorkflow.recover(RequestWorkflow.java:293)
INFO   | jvm 1    | 2014/09/24 16:17:09 |       at orca.controllers.xmlrpc.XmlrpcControllerSlice.recover(XmlrpcControllerSlice.java:461)
INFO   | jvm 1    | 2014/09/24 16:17:09 |       at orca.controllers.xmlrpc.XmlrpcOrcaState.recoverSlice(XmlrpcOrcaState.java:443)
INFO   | jvm 1    | 2014/09/24 16:17:09 |       at orca.controllers.xmlrpc.XmlrpcOrcaState.recover(XmlrpcOrcaState.java:405)
INFO   | jvm 1    | 2014/09/24 16:17:09 |       at orca.controllers.xmlrpc.XmlRpcController._recover(XmlRpcController.java:196)
INFO   | jvm 1    | 2014/09/24 16:17:09 |       at orca.controllers.OrcaController.recover(OrcaController.java:62)
INFO   | jvm 1    | 2014/09/24 16:17:09 |       at orca.controllers.xmlrpc.XmlRpcController.access$100(XmlRpcController.java:24)
INFO   | jvm 1    | 2014/09/24 16:17:09 |       at orca.controllers.xmlrpc.XmlRpcController$ControllerContextListener.start(XmlRpcController.java:174)
INFO   | jvm 1    | 2014/09/24 16:17:09 |       at orca.controllers.xmlrpc.XmlRpcController.start(XmlRpcController.java:156)
INFO   | jvm 1    | 2014/09/24 16:17:09 |       at orca.controllers.xmlrpc.XmlRpcController.main(XmlRpcController.java:201)
INFO   | jvm 1    | 2014/09/24 16:17:09 |       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
INFO   | jvm 1    | 2014/09/24 16:17:09 |       at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
INFO   | jvm 1    | 2014/09/24 16:17:09 |       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
INFO   | jvm 1    | 2014/09/24 16:17:09 |       at java.lang.reflect.Method.invoke(Method.java:606)
INFO   | jvm 1    | 2014/09/24 16:17:09 |       at org.tanukisoftware.wrapper.WrapperSimpleApp.run(WrapperSimpleApp.java:240)
INFO   | jvm 1    | 2014/09/24 16:17:09 |       at java.lang.Thread.run(Thread.java:745)
INFO   | jvm 1    | 2014/09/24 16:17:09 | 2014-09-24 16:17:09,296 [WrapperSimpleAppMain] ERROR controller.orca.controllers.xmlrpc.XmlrpcOrcaState - Unable to recover slice 6f9b2dda-aea1-4313-9948-6ceae431711b/test-rci-wvn-1 due to: java.lang.NullPointerException

The slice however somehow does get recovered at least to the point of being queriable and the start time and end times look correct.

The large number of exceptions has been due to the fact that the slice re-recovery attempts are made for every reservation for every property in it, which we now reduced to only doing it for every reservation (r6868).

The core problem appears to be a partial corruption of the TDB manifest model. TDB dump identified two problems with it:

  • First the start time in the manifest model looks corrupted:
<http://geni-orca.renci.org/owl/88c31864-776d-40aa-a844-781747421a6b#Link0> <http://geni-orca.renci.org/owl/collections.owl#item> <http://geni-orca.renci.org/owl/wvnvmsite.rdf#WVN/ExoGeni/TenGigabitEthernet/1/ethernet-wvnNet/Cisco/UCSFI/TenGigabitEthernet/1/1/ethernet> .
<http://geni-orca.renci.org/owl/88c31864-776d-40aa-a844-781747421a6b#Term> <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://www.w3.org/2006/time#Interval> .
<http://geni-orca.renci.org/owl/88c31864-776d-40aa-a844-781747421a6b#Term> <http://www.w3.org/2006/time#hasDurationDescription> <http://geni-orca.renci.org/owl/88c31864-776d-40aa-a844-781747421a6b#TermDuration> .

The below statement looks incorrect:
<http://geni-orca.renci.org/owl/88c31864-776d-40aa-a844-781747421a6b#Term> <http://www.w3.org/2006/time#hasBeginning> <http://geni-orca.renci.org/owl/88c31864-776d-40aa-a844-781747421a6b#f0040e81-6a85-40f3-ac9c-caadbb12fd30> .

<http://geni-orca.renci.org/owl/topology.owl#NetworkConnection> <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://www.w3.org/2002/07/owl#Class> .
<http://geni-orca.renci.org/owl/topology.owl#NetworkConnection> <http://www.w3.org/2000/01/rdf-schema#subClassOf> <http://geni-orca.renci.org/owl/collections.owl#List> .
<http://geni-orca.renci.org/owl/topology.owl#NetworkConnection> <http://www.w3.org/2000/01/rdf-schema#subClassOf> <http://geni-orca.renci.org/owl/topology.owl#Connection> .
<http://geni-orca.renci.org/owl/88c31864-776d-40aa-a844-781747421a6b#TermDuration> <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://www.w3.org/2006/time#DurationDescription> .
<http://geni-orca.renci.org/owl/88c31864-776d-40aa-a844-781747421a6b#TermDuration> <http://www.w3.org/2006/time#days> "1"^^<http://www.w3.org/2001/XMLSchema#decimal> .
<http://geni-orca.renci.org/owl/collections.owl#List> <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://www.w3.org/2002/07/owl#Class> .

Also tdbdump refused to fully read the model:

_:BX2D799fb425X3A148a44b3cc2X3A55cd <http://www.w3.org/2002/07/owl#onProperty> <http://geni-orca.renci.org/owl/compute.owl#memoryCapacity> .
_:BX2D799fb425X3A148a44b3cc2X3A55cd <Exception in thread "main" org.apache.jena.atlas.lib.InternalErrorException: Invalid id node for object (null node): ([00000000000055CF], [00000000000001DB], [0000000000012C14])
	at com.hp.hpl.jena.tdb.lib.TupleLib.triple(TupleLib.java:139)
	at com.hp.hpl.jena.tdb.lib.TupleLib.triple(TupleLib.java:114)
	at com.hp.hpl.jena.tdb.lib.TupleLib.access$000(TupleLib.java:45)
	at com.hp.hpl.jena.tdb.lib.TupleLib$3.convert(TupleLib.java:76)
	at com.hp.hpl.jena.tdb.lib.TupleLib$3.convert(TupleLib.java:72)
	at org.apache.jena.atlas.iterator.Iter$4.next(Iter.java:323)
	at org.apache.jena.atlas.iterator.Iter$4.next(Iter.java:323)
	at org.apache.jena.atlas.iterator.Iter.next(Iter.java:921)
	at org.apache.jena.atlas.iterator.IteratorCons.next(IteratorCons.java:92)
	at org.apache.jena.riot.system.StreamRDFLib.quadsToStream(StreamRDFLib.java:90)
	at org.apache.jena.riot.writer.NQuadsWriter.write$(NQuadsWriter.java:62)
	at org.apache.jena.riot.writer.NQuadsWriter.write(NQuadsWriter.java:45)
	at org.apache.jena.riot.writer.NQuadsWriter.write(NQuadsWriter.java:91)
	at org.apache.jena.riot.RDFDataMgr.write$(RDFDataMgr.java:1274)
	at org.apache.jena.riot.RDFDataMgr.write(RDFDataMgr.java:1148)
	at org.apache.jena.riot.RDFDataMgr.write(RDFDataMgr.java:1138)
	at tdb.tdbdump.exec(tdbdump.java:50)
	at arq.cmdline.CmdMain.mainMethod(CmdMain.java:102)
	at arq.cmdline.CmdMain.mainRun(CmdMain.java:63)
	at arq.cmdline.CmdMain.mainRun(CmdMain.java:50)
	at tdb.tdbdump.main(tdbdump.java:32)

however obviously the corruption is not so bad as to make the model completely unreadable, since the slice is queriable by user. We need to look more carefully at where manifest model is updated and be sure to call sync on TDB model after every modification. Also we need to make sure there are no concurrent modifications to a manifest model.

  Changed 5 years ago by ibaldin

Possible causes for above are

  • Concurrent updates /race condition in manifest update
  • Incomplete sync of the model
  • Jena bug (we're on Jena 2.11, 2.12 is available)

  Changed 5 years ago by ibaldin

Another issue is the controller appears (at least after recovery -needs further testing) to honor the reservation limits set by the user (defaulting to 1 day regardless of the setting). Testing in emulation now.

  Changed 5 years ago by ibaldin

The reservation limit should be fixed in r6869. This must've been not working at least a month.

I also upped the Jena version to 2.12.0 and tdb to 1.1.0

  Changed 5 years ago by ibaldin

Looks like Yufeng fixed MP issue in r6870.

I uncovered one issue with recovery - if you have a dumbbell that you create (very repeatable and is not caused by the updated to Tdb 1.1.0) and then without requesting slice manifest (without clicking Query) restart the controller, the Link reservation disappears. It is drawn in the manifest, however has no state or reservation notices. It looks like the initial pre-query manifest generation is slightly incomplete.

Victor is looking at the issue with build hosts and Tdb 1.1.0 build problem we're currently experiencing. I noticed no adverse effects from upgrading to TDB 1.1.0/Jena 2.12.0 but we should watch carefully.

  Changed 5 years ago by anirban

There is a NPE for deleting a MP slice after recovery. This means, my multi-point slice could not be deleted. So, the switches need to be cleared before I launch another slice ?

INFO   | jvm 1    | 2014/09/25 10:34:25 | 2014-09-25 10:34:25,643 [qtp1732503677-26 - /orca/xmlrpc] ERROR controller.orca.controllers.xmlrpc.OrcaXmlrpcHandler - deleteSlice(): Exception encountered: null
INFO   | jvm 1    | 2014/09/25 10:34:25 | java.lang.NullPointerException
INFO   | jvm 1    | 2014/09/25 10:34:25 |       at orca.embed.workflow.RequestWorkflow.clearGlobalControllerAssignedLabel(RequestWorkflow.java:236)
INFO   | jvm 1    | 2014/09/25 10:34:25 |       at orca.controllers.xmlrpc.XmlrpcControllerSlice.close(XmlrpcControllerSlice.java:263)
INFO   | jvm 1    | 2014/09/25 10:34:25 |       at orca.controllers.xmlrpc.XmlrpcOrcaState.removeSlice(XmlrpcOrcaState.java:141)
INFO   | jvm 1    | 2014/09/25 10:34:25 |       at orca.controllers.xmlrpc.OrcaXmlrpcHandler.deleteSlice(OrcaXmlrpcHandler.java:793)
INFO   | jvm 1    | 2014/09/25 10:34:25 |       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
INFO   | jvm 1    | 2014/09/25 10:34:25 |       at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
INFO   | jvm 1    | 2014/09/25 10:34:25 |       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
INFO   | jvm 1    | 2014/09/25 10:34:25 |       at java.lang.reflect.Method.invoke(Method.java:606)
INFO   | jvm 1    | 2014/09/25 10:34:25 |       at org.apache.xmlrpc.server.ReflectiveXmlRpcHandler.invoke(ReflectiveXmlRpcHandler.java:115)
INFO   | jvm 1    | 2014/09/25 10:34:25 |       at org.apache.xmlrpc.server.ReflectiveXmlRpcHandler.execute(ReflectiveXmlRpcHandler.java:106)
INFO   | jvm 1    | 2014/09/25 10:34:25 |       at org.apache.xmlrpc.server.XmlRpcServerWorker.execute(XmlRpcServerWorker.java:46)
INFO   | jvm 1    | 2014/09/25 10:34:25 |       at org.apache.xmlrpc.server.XmlRpcServer.execute(XmlRpcServer.java:86)
INFO   | jvm 1    | 2014/09/25 10:34:25 |       at org.apache.xmlrpc.server.XmlRpcStreamServer.execute(XmlRpcStreamServer.java:200)
INFO   | jvm 1    | 2014/09/25 10:34:25 |       at org.apache.xmlrpc.webserver.XmlRpcServletServer.execute(XmlRpcServletServer.java:112)
INFO   | jvm 1    | 2014/09/25 10:34:25 |       at org.apache.xmlrpc.webserver.XmlRpcServlet.doPost(XmlRpcServlet.java:196)
INFO   | jvm 1    | 2014/09/25 10:34:25 |       at orca.controllers.OrcaXmlrpcServlet.doPost(OrcaXmlrpcServlet.java:151)
INFO   | jvm 1    | 2014/09/25 10:34:25 |       at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
INFO   | jvm 1    | 2014/09/25 10:34:25 |       at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
INFO   | jvm 1    | 2014/09/25 10:34:25 |       at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:527)
INFO   | jvm 1    | 2014/09/25 10:34:25 |       at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:423)
INFO   | jvm 1    | 2014/09/25 10:34:25 |       at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:119)
INFO   | jvm 1    | 2014/09/25 10:34:25 |       at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:493)
INFO   | jvm 1    | 2014/09/25 10:34:25 |       at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:926)
INFO   | jvm 1    | 2014/09/25 10:34:25 |       at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:358)
INFO   | jvm 1    | 2014/09/25 10:34:25 |       at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:860)
INFO   | jvm 1    | 2014/09/25 10:34:25 |       at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117)
INFO   | jvm 1    | 2014/09/25 10:34:25 |       at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:113)
INFO   | jvm 1    | 2014/09/25 10:34:25 |       at org.eclipse.jetty.server.Server.handle(Server.java:331)
INFO   | jvm 1    | 2014/09/25 10:34:25 |       at org.eclipse.jetty.server.HttpConnection.handleRequest(HttpConnection.java:588)
INFO   | jvm 1    | 2014/09/25 10:34:25 |       at org.eclipse.jetty.server.HttpConnection$RequestHandler.content(HttpConnection.java:1046)
INFO   | jvm 1    | 2014/09/25 10:34:25 |       at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:764)
INFO   | jvm 1    | 2014/09/25 10:34:25 |       at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:217)
INFO   | jvm 1    | 2014/09/25 10:34:25 |       at org.eclipse.jetty.server.HttpConnection.handle(HttpConnection.java:418)
INFO   | jvm 1    | 2014/09/25 10:34:25 |       at org.eclipse.jetty.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:476)
INFO   | jvm 1    | 2014/09/25 10:34:25 |       at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:436)
INFO   | jvm 1    | 2014/09/25 10:34:25 |       at java.lang.Thread.run(Thread.java:745)

  Changed 5 years ago by yxin

we'll work on it. Was your original MP slice pingable?

  Changed 5 years ago by anirban

Before redeploying, I was trying to delete the MP slice that I tried yesterday. That's when it threw the NPE. I have redeployed controller and DD/ION AM with newest code. Are the switches clean so that I can try a MP slice now ? Or, does it not matter ?

  Changed 5 years ago by yxin

It's an NPE in controller. The control LEAVE worked and cleaned the switches. So you can go ahead to test.

  Changed 5 years ago by anirban

The issue with slice expiry is fixed. I also verified that the idm model goes away on its own after automatic slice expiry.

  Changed 5 years ago by anirban

MP request involving rci, wvn and tamu, is throwing NPE at the controller. Slice doesn't go through.

INFO   | jvm 1    | 2014/09/25 11:18:19 | 2014-09-25 11:18:19,336 [qtp136719068-30 - /orca/xmlrpc] DEBUG controller.orca.controllers.xmlrpc.OrcaXmlrpcHandler - Found abstract model for resource pool: rcivmsite.baremetalce
INFO   | jvm 1    | 2014/09/25 11:18:19 | 2014-09-25 11:18:19,413 [qtp136719068-30 - /orca/xmlrpc] INFO  controller.orca.controllers.xmlrpc.OrcaXmlrpcHandler - discoverTypes: wvnvmsite/vm rt=wvnvmsite.vm available units=51
INFO   | jvm 1    | 2014/09/25 11:18:19 | 2014-09-25 11:18:19,413 [qtp136719068-30 - /orca/xmlrpc] DEBUG controller.orca.controllers.xmlrpc.OrcaXmlrpcHandler - Found abstract model for resource pool: wvnvmsite.vm
INFO   | jvm 1    | 2014/09/25 11:18:21 | java.lang.NullPointerException
INFO   | jvm 1    | 2014/09/25 11:18:21 |       at orca.embed.cloudembed.controller.InterCloudHandler.runEmbedding(InterCloudHandler.java:132)
INFO   | jvm 1    | 2014/09/25 11:18:21 |       at orca.embed.workflow.RequestWorkflow.run(RequestWorkflow.java:138)
INFO   | jvm 1    | 2014/09/25 11:18:21 |       at orca.controllers.xmlrpc.OrcaXmlrpcHandler.createSlice(OrcaXmlrpcHandler.java:342)
INFO   | jvm 1    | 2014/09/25 11:18:21 |       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
INFO   | jvm 1    | 2014/09/25 11:18:21 |       at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
INFO   | jvm 1    | 2014/09/25 11:18:21 |       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
INFO   | jvm 1    | 2014/09/25 11:18:21 |       at java.lang.reflect.Method.invoke(Method.java:606)
INFO   | jvm 1    | 2014/09/25 11:18:21 |       at org.apache.xmlrpc.server.ReflectiveXmlRpcHandler.invoke(ReflectiveXmlRpcHandler.java:115)
INFO   | jvm 1    | 2014/09/25 11:18:21 |       at org.apache.xmlrpc.server.ReflectiveXmlRpcHandler.execute(ReflectiveXmlRpcHandler.java:106)
INFO   | jvm 1    | 2014/09/25 11:18:21 |       at org.apache.xmlrpc.server.XmlRpcServerWorker.execute(XmlRpcServerWorker.java:46)
INFO   | jvm 1    | 2014/09/25 11:18:21 |       at org.apache.xmlrpc.server.XmlRpcServer.execute(XmlRpcServer.java:86)
INFO   | jvm 1    | 2014/09/25 11:18:21 |       at org.apache.xmlrpc.server.XmlRpcStreamServer.execute(XmlRpcStreamServer.java:200)
INFO   | jvm 1    | 2014/09/25 11:18:21 |       at org.apache.xmlrpc.webserver.XmlRpcServletServer.execute(XmlRpcServletServer.java:112)
INFO   | jvm 1    | 2014/09/25 11:18:21 |       at org.apache.xmlrpc.webserver.XmlRpcServlet.doPost(XmlRpcServlet.java:196)INFO   | jvm 1    | 2014/09/25 11:18:21 |       at orca.controllers.OrcaXmlrpcServlet.doPost(OrcaXmlrpcServlet.java:151)
INFO   | jvm 1    | 2014/09/25 11:18:21 |       at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
INFO   | jvm 1    | 2014/09/25 11:18:21 |       at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
INFO   | jvm 1    | 2014/09/25 11:18:21 |       at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:527)
INFO   | jvm 1    | 2014/09/25 11:18:21 |       at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:423)
INFO   | jvm 1    | 2014/09/25 11:18:21 |       at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:119)
INFO   | jvm 1    | 2014/09/25 11:18:21 |       at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:493)
INFO   | jvm 1    | 2014/09/25 11:18:21 |       at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:926)
INFO   | jvm 1    | 2014/09/25 11:18:21 |       at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:358)
INFO   | jvm 1    | 2014/09/25 11:18:21 |       at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:860)
INFO   | jvm 1    | 2014/09/25 11:18:21 |       at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117)
INFO   | jvm 1    | 2014/09/25 11:18:21 |       at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:113)
INFO   | jvm 1    | 2014/09/25 11:18:21 |       at org.eclipse.jetty.server.Server.handle(Server.java:331)
INFO   | jvm 1    | 2014/09/25 11:18:21 |       at org.eclipse.jetty.server.HttpConnection.handleRequest(HttpConnection.java:588)
INFO   | jvm 1    | 2014/09/25 11:18:21 |       at org.eclipse.jetty.server.HttpConnection$RequestHandler.content(HttpConnection.java:1046)
INFO   | jvm 1    | 2014/09/25 11:18:21 |       at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:764)
INFO   | jvm 1    | 2014/09/25 11:18:21 |       at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:217)
INFO   | jvm 1    | 2014/09/25 11:18:21 |       at org.eclipse.jetty.server.HttpConnection.handle(HttpConnection.java:418)
INFO   | jvm 1    | 2014/09/25 11:18:21 |       at org.eclipse.jetty.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:476)
INFO   | jvm 1    | 2014/09/25 11:18:21 |       at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:436)
INFO   | jvm 1    | 2014/09/25 11:18:21 |       at java.lang.Thread.run(Thread.java:745)
INFO   | jvm 1    | 2014/09/25 11:18:21 | 2014-09-25 11:18:21,241 [qtp136719068-30 - /orca/xmlrpc] ERROR controller.orca.controllers.xmlrpc.OrcaXmlrpcHandler - createSlice(): N
o reservations created for this request; Error:java.lang.NullPointerException

  Changed 5 years ago by ibaldin

Exchange, BEN and DD switches are cleaned. There was nothing in the Exchange switch, which makes me wonder if it worked.

  Changed 5 years ago by ibaldin

The deleting MP NPE should be resolved in r6873

  Changed 5 years ago by yxin

Could not repeat the NPE in emulator, so just added an error log and it won't throw NPE. Need to keep an eye on it.
Please redeploy controller and try again.

  Changed 5 years ago by ibaldin

Most issues appear resolved. DAR is somewhat behaving although requires further investigation. Manifest publishing now works too.

Before we go live need to

  1. Update ExoSM to latest code (likely r6890), as well as geni-ben and geni2 and probably do a clean restart for good measure
  1. Get new ION RDF from Yufeng with 10 placeholders for future ports. He says that vlan tag ranges are specified on the Net side, which means we should be able to add new sites without restarting ION actor. There is a new capability in ION handler to remap URNs, without a need for a restart.

3. Remove SONIC and UCD bridges from dd

  Changed 5 years ago by ibaldin

geni, geni2 and geni-ben are up to r6890, not been restarted.

  Changed 5 years ago by yxin

found a minor error (for future site) in ion.rdf, please restart genie w/ the latest rdf.

  Changed 5 years ago by ibaldin

SONIC and UCD bridges have been removed from DD.

  Changed 5 years ago by ibaldin

  • status changed from new to closed
  • resolution set to fixed

Maintenance and upgrade completed

Note: See TracTickets for help on using tickets.