Ticket #384 (closed task: fixed)

Opened 5 years ago

Last modified 4 years ago

Pre-2015 clean restart and GENI stitching update

Reported by: ibaldin Owned by: vjo
Priority: major Milestone:
Component: External: Testing and Redeployment Version: baseline
Keywords: Cc: ckh, yufeng

Description

Clean restart and update controllers and RDF at a number of sites. This accommodates ION/AL2S transition and GENI stitching changes.

The following rack sites are affected: rci, bbn, fiu, ufl, ucd, osf, sl, tamu, uh, wvn

Also ben, dd/nlr and ion

Relevant links: https://www.evernote.com/l/AK5rjnPF3VtAWY0yDGpDszaZ8KVAG_jLT4I

and

https://www.evernote.com/l/AK4FRlpRZc9DgZUap4RbwXi3-z-4aeJbbGU

Change History

Changed 5 years ago by vjo

Encountered an issue with a p2p link between FIU and UCD.

Embedding workflow reported a "path not found" - suspect issue in RDF.

Have mailed Yufeng.

Changed 5 years ago by vjo

Encountered two issues with OSCARS/ION during testing.

1)
Last lease update: all units failed priming: Error code 1 during join for unit: F8FC8507 with message: Unable to create circuit: start-oscars-v06.sh: OSCARS did not return a GRI to createReservation request due to: "Error: Generic exception: OSCARSA reservation failed with status FAILED due to Unable to find link with id al2s.net.internet2.edu:sdn-sw.clev.net.internet2.edu:eth3/2:* on reservation al2s.net.internet2.edu-42471 in Dijkstra PCE ", exiting

2)
Last lease update: all units failed priming: Error code 1 during join for unit: 6D54303A with message: Unable to create circuit: start-oscars-v06.sh: OSCARS did not return a GRI to createReservation request due to: "General exception: org.apache.cxf.service.factory.ServiceConstructionException?: Failed to create service.
javax.xml.ws.WebServiceException?: org.apache.cxf.service.factory.ServiceConstructionException?: Failed to create service.

at org.apache.cxf.jaxws.ServiceImpl?.<init>(ServiceImpl?.java:150)
at org.apache.cxf.jaxws.spi.ProviderImpl?.createServiceDelegate(ProviderImpl?.java:65)
at javax.xml.ws.Service.<init>(Service.java:77)
at net.es.oscars.api.soap.gen.v06.OSCARSService.<init>(OSCARSService.java:48)
at net.es.oscars.client.OSCARSClient.<init>(OSCARSClient.java:56)
at net.es.oscars.client.OSCARSClient.<init>(OSCARSClient.java:62)
at orca.drivers.oscars.v06.Driver.<init>(Driver.java:68)
at orca.drivers.oscars.v06.DriverMain?.main(DriverMain?.java:226)

Caused by: org.apache.cxf.service.factory.ServiceConstructionException?: Failed to create service.

at org.apache.cxf.wsdl11.WSDLServiceFactory.<init>(WSDLServiceFactory.java:93)
at org.apache.cxf.jaxws.ServiceImpl?.initializePorts(ServiceImpl?.java:205)
at org.apache.cxf.jaxws.ServiceImpl?.<init>(ServiceImpl?.java:148)
... 7 more

Caused by: javax.wsdl.WSDLException: WSDLException (at /wsdl:definitions/wsdl:types/xsd:schema): faultCode=PARSER_ERROR: Problem parsing '/etc/oscars/wsdl/OSCARSService/r-2-idc.xsd'.: java.io.FileNotFoundException?: /etc/oscars/wsdl/OSCARSService/r-2-idc.xsd (No such file or directory)

at com.ibm.wsdl.xml.WSDLReaderImpl.getDocument(Unknown Source)
at com.ibm.wsdl.xml.WSDLReaderImpl.parseSchema(Unknown Source)
at com.ibm.wsdl.xml.WSDLReaderImpl.parseSchema(Unknown Source)
at com.ibm.wsdl.xml.WSDLReaderImpl.parseTypes(Unknown Source)
at com.ibm.wsdl.xml.WSDLReaderImpl.parseDefinitions(Unknown Source)
at com.ibm.wsdl.xml.WSDLReaderImpl.readWSDL(Unknown Source)
at com.ibm.wsdl.xml.WSDLReaderImpl.readWSDL(Unknown Source)
at org.apache.cxf.wsdl11.WSDLManagerImpl.loadDefinition(WSDLManagerImpl.java:229)
at org.apache.cxf.wsdl11.WSDLManagerImpl.getDefinition(WSDLManagerImpl.java:179)
at org.apache.cxf.wsdl11.WSDLServiceFactory.<init>(WSDLServiceFactory.java:91)
... 9 more

Caused by: java.io.FileNotFoundException?: /etc/oscars/wsdl/OSCARSService/r-2-idc.xsd (No such file or directory)

at java.io.FileInputStream?.open(Native Method)
at java.io.FileInputStream?.<init>(FileInputStream?.java:146)
at java.io.FileInputStream?.<init>(FileInputStream?.java:101)
at sun.net.www.protocol.file.FileURLConnection.connect(FileURLConnection.java:90)
at sun.net.www.protocol.file.FileURLConnection.getInputStream(FileURLConnection.java:188)
at com.sun.org.apache.xerces.internal.impl.XMLEntityManager.setupCurrentEntity(XMLEntityManager.java:613)
at com.sun.org.apache.xerces.internal.impl.XMLVersionDetector.determineDocVersion(XMLVersionDetector.java:189)
at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:812)
at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:777)
at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:141)
at com.sun.org.apache.xerces.internal.parsers.DOMParser.parse(DOMParser.java:243)
at com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderImpl?.parse(DocumentBuilderImpl?.java:347)
... 19 more

Error: Generic exception: null", exiting

In the second issue, the backtrace is not from ORCA, but seems to be from the OSCARS service itself.

Have contacted Chris (and copied Yufeng) so that he can contact the right people.

Changed 5 years ago by vjo

  • cc ckh, yufeng added

Issue #1 has been traced to a dumbbell between TAMU and WVN.

Issue #2 has been traced to a dumbbell between OSF and SL.

Both are repeatable.

Leaving ExoSM closed until these issues are resolved with I2.

Changed 5 years ago by vjo

WSU <-> WVN replicates issue #1, as does a RCI <-> WVN dumbbell.
This appears to be primarily associated with WVN.

Changed 5 years ago by vjo

One more note:
None of BBN's inter-rack VLANs are passing traffic, though they do succeed in provisioning.

Changed 5 years ago by vjo

OK - ExoSM is back open, along with UH and NICTA.

As of right now, this is the summary of what's still not working:
1) BBN doesn't pass traffic on any VLANs - possibly due to an incorrect URN.
2) WVN seems to have an incorrect URN.
3) CENIC's URN leading to NICTA claims to have no VLANs available.
4) The OSCARS instance serving OSF seems to have crashed.

Changed 4 years ago by ibaldin

  • status changed from new to closed
  • resolution set to fixed

All issues have been addressed (NICTA is a separate problem due to NICTA losing its L2 connection to CENIC - L2TP tunnels are being investigated as a possible workaround.).

Note: See TracTickets for help on using tickets.