After the CUNYfirst computer system crashed repeatedly at the beginning of Fall semester, CUNY officials identified three main steps for preventing such problems in the future. One of those steps, a software upgrade for one of CUNY-first’s main components, will not be in place by the start of Spring semester, Clarion has learned. But CUNY information technology (IT) staff say other measures have been taken to make CUNYfirst more stable, and they expect that Spring term will get off to a smoother start.
 |
If the system does crash, users may have better options than they did when CUNYfirst was out of service last semester: a new system called MyInfo will be able to provide certain basic information even when CUNYfirst is down.
CUNYfirst controls everything from course registration to employee payroll – so when it failed at the start of the Fall 2014 semester, the outages caused a wave of problems. Students could not find out whether a course was still open for registration, or even where it was supposed to meet; faculty could not get access to class rosters; staff were unable to update student records (see Clarion, October 2014).
In a September email to senior CUNY administrators, Associate Vice Chancellor Brian Cohen promised action “to prevent such problems from recurring in the future.” Cohen, the University’s chief information officer, wrote that the Fall semester problems had been caused by “Oracle’s Identity Management System (IMS), which provides user access to CUNYfirst” – that is, it controls user log-ins. He identified an upgrade of the IMS to the latest version, 11g, as one of three key tasks for ensuring that such crashes did not recur.
Behind Schedule
Version 11g will “be ready for launch later this year, prior to the next peak volume in January 2015,” Cohen wrote – but Clarion has learned that this target will not be met. Instead, CUNYfirst will start the Spring semester with the old version of the IMS, version 10g.
Other measures have been taken to make the system more stable, however, and CUNY IT staff are cautiously optimistic about how Spring semester will begin. “From what I’ve seen of the testing, there should not be the problems we saw in the Fall,” said one CUNY IT staffer (who, like others interviewed for this article, asked to remain anonymous).
“CUNY decided to postpone the upgrade of the IMS system to version 11g,” said a January 6 statement from CUNY’s Office of Media Relations, in response to questions from Clarion. “Development and configuration delays would have meant that the 11g system would be deployed just prior to the start of the Spring academic semester. Since CUNY wanted to conduct more testing on 11g prior to deployment, CUNY decided that IMS 10g – after fine-tuning and testing – was a better option….CUNY believes, based on the test results, that the system can meet the demands of the Spring 2015 semester.”
While the administration and CUNY information technology staff both say that Spring semester should start more smoothly, CUNY has also been working to be more prepared than it was last Fall if CUNYfirst does crash. When CUNYfirst was out of service last Fall, most colleges had no way to look up information as basic as the location of a given class. At one senior college, staff helped students locate their classes using “paper copies of the class schedule that one of our managers was farsighted enough to download during one of the few times CUNYfirst was working,” a PSC member told Clarion. “Can you imagine a business as large as CUNY not having a back-up system?” another asked.
Alternative System
Taking such comments to heart, CIS (CUNY Information Services) has been developing a limited alternative to CUNYfirst, a separate system that would continue to operate even if CUNYfirst is down. Originally dubbed “CUNYfirst Lite,” this new system has been renamed “MyInfo,” CUNY IT staffers told Clarion; it will provide students and employees with access to a limited set of basic information on a “read-only” basis.
MyInfo will use data from the previous day, which CUNYfirst uploads each night to a separate CUNY data warehouse. It should allow students to see things like a list of the classes they are signed up for, and where those classes are supposed to meet. Faculty could see their class rosters. But because MyInfo will be based on yesterday’s data, it will not indicate whether a course is full or still open for additional registration – an important limitation at the start of the semester. Since MyInfo will make information available on a “read-only” basis – that is, it will not provide a way to update CUNYfirst data – it will be less useful for CUNY professional staff who are normally entering changes in CUNYfirst throughout the day.
“It won’t be perfect,” said one IT staffer, “but it’s definitely better than nothing.” Ironically, another employee told Clarion, MyInfo “is a ‘shadow system’ – one of the things that CUNYfirst was supposed to get rid of.”
MyInfo’s response time could be significantly better than the normal operation of the problem-plagued CUNYfirst system, IT staff told Clarion. “MyInfo will use what’s called ‘materialized views’ – basically it’s a pre-processing of the data, generated each night for every user, that lists a lot of basic information in one place,” a CUNY IT specialist explained. “That means you won’t need to wait for a lot of separate queries to look up each piece of information – it’s already there, in one place. So instead of a student waiting, say, 15 or 30 seconds, they’ll just wait for one second.”
Poor Performance
That difference highlights the poor performance of CUNYfirst even when the system is operating normally and is not out of service. Users often complain about screens taking a long time long to load, or a given process requiring many more screens than before – with the result that it takes much longer to do many of the same tasks.
Besides upgrading the Identity Management System to version 11g, a move that is now postponed, Cohen’s September email identified two other key tasks for avoiding system crashes in the future: doing improved load testing before the start of Spring semester and “tuning” the system – adjusting various parameters to achieve more stable performance.
Falling Short
Oracle had said that CUNYfirst passed its “stress tests” prior to Fall 2014, and that the system was robust enough to handle as many as 10,000 concurrent users. But in fact CUNYfirst failed last Fall well before that level was reached: in an October 8 report obtained by Clarion, Cohen told CUNY’s Council of Presidents that the system had failed “when load reached 8,500 to 9,000 users.”
(These are figures that the CUNY administration had refused to disclose. As Clarion reported in October, “the administration declined to say how many simultaneous users the system had experienced this Fall.” At the time, CUNY would say only that the system had crashed “during peak loads created by an unprecedented number of concurrent user sessions.”)
Why was Oracle’s original load testing so wrong? Answering that question was the second of the three key measures identified in Cohen’s September email.
“What they need to do is test the mix of transactions that were being done when the system failed,” said an independent IT consultant with experience in academic systems integration. “What the system can handle depends on the types of transactions,” the consultant told Clarion. “If it’s just a simple query, you might have 20,000 people do it at once and not have a problem. In Oracle’s earlier load testing, most likely someone assumed a certain group of transactions, but that didn’t match what actually happened.”
“Our examination of the load testing conducted by Oracle showed a number of factors that needed to be changed in the methodology,” CUNY’s Office of Media Relations told Clarion in January. “CIS [CUNY Information Services] has conducted additional load tests…based on load data from the Fall 2014 semester,” the administration said. As parameters were adjusted to reflect those specific conditions, CUNY said, “these tests have shown improved IMS performance.”
The university “has also hired an independent, third-party consulting company with expertise in Oracle IMS and higher education,” the statement added. “These consultants have been…working with CUNY to ensure performance testing is conducted appropriately.”
‘Tuning’ the System
The third key task named by Cohen was “tuning” the system – adjusting different parameters in the Identity Management System. “Oracle did put a lot of work into fixing some of the problems,” a CUNY IT staffer said, and CUNYfirst should now be more stable in the face of sudden increases in user demand.
The administration’s January statement said that CUNY has “made substantial effort” to “address the problems experienced at the start of the Fall 2014 semester, and prevent those problems from recurring at the start of the Spring 2015 semester.”
Hundreds of thousands of CUNYfirst users are hoping that will be enough.
