Loading...
HomeMy WebLinkAbout _ 4.11(d)--Authorize Sole-Source Purchase of Stratus Servers CITY OF REDDING REPORT TO THE CITY COUNCIL Recommendation Authorize and approve the sole source procurement of two Stratus Fault Tolerant server systems in an amount not to exceed $250,000 for the City of Redding Water Utility. Fiscal Impact There is adequate funding appropriated in the Fiscal Year 2023-24 Water Utility budget for the $250,000 purchase of the proposed equipment. Alternative Action The City Council (Council) may choose not to authorize the purchase of server systems directly from Stratus and instead request quotes and award the various items through the standard competitive procurement process. This alternative may result in multiple manufacturers that are not compatible with the necessary equipment for the City of Redding’s (City) water system network design. The Council may also provide alternative direction to staff. Background/Analysis In August 2023, the City’s Water Utility contracted with TJC/JEGO Systems, communications consultants, to standardize and redesign the network/communication systems installed over the last 30 years. The existing computer servers are more than 10 years old, have become unreliable, and reached the end of their reliable useful life. The current network system allows water treatment facilities, water storage reservoirs, groundwater wells, pump stations, and more to communicate via various types of radio telemetry and wide-area network systems. The scope of work included evaluating the existing Supervisory Control and Data Acquisition (SCADA) compute and storage infrastructure and consider the long-term future of the systems dependent on this infrastructure. MEETING DATE: November 7, 2023 ITEM NO. 4.11(d) FROM: ***APPROVED BY*** caukland@ci.redding.ca.us btippin@cityofredding.org SUBJECT: 4.11(d)--Authorize Sole-Source Purchase of Stratus Servers for the Public Works Department Chuck Aukland, Public Works Director Report to Redding City Council October 31, 2023 Re: 4.11(d)--Authorize Sole-Source Purchase of Stratus Servers Page 2 The water network design plan provided by TJCAA/JEGO Systems specifically recommends Stratus Fault Tolerant (Stratus) servers due to their industry-leading quality, product availability, existing equipment compatibility, and support capabilities. The attached SCADA Infrastructure Compute and Storage Specification recommendation by TJC/JEGO Systems details the server recommendation and includes all the information a vendor will need to prepare a quote. The City will procure quotes from three vendors using this specification. The suggested specification is for one server to be located at each of the City’s water treatment plants for a total of two Stratus servers. Each server system is estimated to cost roughly $100,000 to $125,000. Environmental Review On December 6, 2022, Council found the Water Utility Network and Communication System Project to be exempt from review under the California Environmental Quality Act Guidelines, pursuant to Section 15301 (a & b) – Existing Facilities. No further action is required. Council Priority/City Manager Goals • Public Safety – “Work to improve all aspects of public safety to help people feel secure and safe where they live, work, and play in the City of Redding.” • Government of the 21st Century – “Be relevant and proactive to the opportunities and challenges of today’s residents and workforce. Anticipate the future to make better decisions today.” Attachments ^SCADA Infrastructure Specifications Previous Staff Report City of Redding TJC| JEGO Systems SCADA Infrastructure Compute & Storage Specification August 24, 2023 Page [1] - SCADA Infrastructure Compute & Storage Specification For Water Supply, Treatment, & Distribution Systems Prepared by: Stephen Goldsworth, CISSP TJC | JEGO Systems Prepared for: City of Redding Attn: Jim Lewis, Electrical Technician Supervisor – Public Works August 24, 2023 City of Redding TJC| JEGO Systems SCADA Infrastructure Compute & Storage Specification August 24, 2023 Page [2] 1 City of Redding SCADA Compute & Storage Infrastructure 1.1 Compute and Storage Recommendation Basis TJC and JEGO Systems evaluated the existing SCADA infrastructure and considered the long-term future of the systems dependent on this infrastructure in developing this recommendation. The recommendation is for two fault- tolerant (FT) server system(s) to host new and existing computers as virtual machines (VMs) that comprise the water supply, treatment, and distribution SCADA Systems. This specification is for a quantity of (2) two fault-tolerant (FT) servers to form (2) two independent systems. One system is to be located at Foothills WTP (System 1) and one at Buckeye WTP (System 2), each consisting of (1) FT server with (2) two CRUs1 (duplex hardware). Stratus ftServer Model 4920 is the recommended solution for the City's SCADA infrastructure. This recommendation is based on several factors that include critical operational technology (OT) system availability of 99.999%+ realized on the installed base, ease of maintenance by OT staff, 24x7 ongoing support, and the long-term support of this equipment by the manufacturer, which will extend the lifecycle of this hardware refresh to 15+ years. 1.1.1 Availability Factor Stratus FT servers are designed to handle hardware problems automatically, with no application restart or data loss. Lockstep technology uses replicated fault-tolerant hardware components that simultaneously process instructions. In Stratus chipsets, hardware logic provides the lockstep architecture's error detection, fault isolation, and synchronization logic. Errors are detected and isolated to the component level, and the system itself then notifies of any problems so that corrective action can be initiated. Stratus systems are also built to withstand unpredictable transient errors that lead to downtime or cause silent data corruption on conventional systems and cl usters. In the event of a component malfunction, the redundant component acts as an active spare that continues normal operation, thus averting system downtime. While other servers may offer duplicated power supplies, fans, and disk drives, only Stratus protects core system components, including motherboards, processors, memory, I/O buses, and I/O adapters. This equipment also provides an OPC connector for health notifications directly on SCADA screens, enhancing monitoring and early detection of issues. 1.1.2 Maintenance & Support Factor Stratus systems combine platform innovations with a support model explicitly designed for critical infrastructure. The Stratus Customer Assistance Center provides 24/7 essential support to ensure continuous operation. In the event of component failure, a replacement component is shipped from Stratus for next-day arrival. Hot-pluggable components are easy to replace by users without tools, entering keyboard commands, or a visit from a field engineer. Once the replacement part is inserted, it is automatically integrated into the running system. Throughout this process, your system and applications will continue running normally. Stratus has provided platform support for 15+ years and has been installed in critical infrastructure environments that are 30+ years old and continued to be supported by Stratus. Although comprised of duplex hardware, Stratus hardware is licensed as a single system, further reducing software costs and simplifying updates and maintenance activities. Stratus provides onsite installation as a standard offering and training on the new infrastructure. 1.2 Planned VM Inventory/Resources for Systems 1 and 2 1.2.1 System 1: Foothills Water Treatment Plant SCADA Infrastructure VM Application/Function vCPU vMem (GB) vDisk (GB) 1 Windows Domain Controller 1 4 60 2 Ignition (medium) project: 10,000 tags / 20 concurrent clients 4 4 400 3 Historian (medium) 500 - 2,500 tags SQL Based 4 8 300 4 (Existing P2V) FWTP1 (GE SCADA, Win911 Server lives at Foothill) 2 8 300 1 CRU: Customer Replaceable Units; (2) two CRU in each FT server City of Redding TJC| JEGO Systems SCADA Infrastructure Compute & Storage Specification August 24, 2023 Page [3] VM Application/Function vCPU vMem (GB) vDisk (GB) 5 (Existing P2V) Historian 1 (GE Historian Server lives at Foothill) 2 20 1500 6 (Existing P2V) ETSHOP (SCADA Development Server, GE, Ignition, Reporting, Etc; lives at Foothill) 2 16 1000 7 (Existing P2V) REDW1 (Server for 'What's Up' and WIN911 Mobile App, Interface to the Business Network; lives at Foothill) 2 16 60 8 Veeam or Replication Server Appliance 2 8 500 9 VCenter Appliance 2 8 500 10 Windows Domain Controller 1 4 60 11 Ignition (medium) project: 10,000 tags / 20 concurrent clients 4 4 400 12 (Existing P2V) FWTP2 (GE SCADA, Win911 Server lives at Foothill) 2 8 300 13 (Existing P2V) Historian 2 (Wonderware Historian Server lives at Foothill) 2 8 250 1.2.2 System 2: Buckeye Water Treatment Plant SCADA Infrastructure VM Application/Function vCPU vMem (GB) vDisk (GB) 1 Windows Domain Controller 1 4 60 2 Ignition (medium) project: 10,000 tags / 20 concurrent clients 4 4 400 3 Historian (medium) 500 - 2,500 tags SQL Based 4 8 300 4 (Existing P2V) BWTP1 (GE SCADA Server lives at Buckeye) 2 8 300 5 Veeam or Replication Server Appliance 2 8 500 6 VCenter Appliance 2 8 500 7 Windows Domain Controller 1 4 60 8 Ignition (medium) project: 10,000 tags / 20 concurrent clients 4 4 400 9 (Existing P2V) BWTP2 (GE SCADA Server lives at Buckeye) 2 8 300 1.3 Hardware & Software Specifications Stratus Build Summary Quantity of (2) fault-tolerant, duplex hardware servers (Stratus ftServer 4920), each with the following features: • 2-socket, 2.4 GHz 10-core processor; Intel Hyperthreading Technology enabled. (40 vCPU) [2 processors per CRU] • 128GB Memory [128GB each CRU] • 2 800GB SSD [1 each CRU] • 8 1.6TB NVMe [4 each CRU] • 8 Logical 1Gb RJ45 Network Ports [4 per CRU] • 4 Logical 10Gb SFP+ Network Ports [2 per CRU] • VMware Essentials • Customer-supplied VM OS • Stratus Field Engineering Installation - Onsite Training • Partner Relocation Service - Install at SI re-verification at Customer Site (3 Months Staging) • 3 Years prepay EPS Support Processors Processor(s) 2 x Intel Xeon Silver 4210R processor 2.4 Ghz (40 vCPU) [2 processors per CRU] Intel Hyper-threaded Cores 10 per processor [20 per CRU] Threads 20 per processor [40 per CRU] Memory Minimum Usable Memory 128 GB DDR4 I/O Subsystem Integrated PCIe adapter slots 4 PCIe 3 x8 Additional PCIe adapter slots 4 PCIe 3 x8 Storage Subsystem Internal system drives 2 800GB SSD [1 each CRU] 8 1.6TB NVMe [4 each CRU] Minimum Raw Storage 6TB RAID levels 1, 5, 6, 10 Embedded I/O Manageability Baseboard management controller Standard Virtual Technician Module (VTM) Standard Graphics adapter 1 VGA port per system PCI Adapters 1 Gb dual-port Ethernet 8 Logical 1Gb RJ45 Network Ports [4 per CRU] 10 Gb dual-port Ethernet (copper) 4 Logical 10Gb SFP+ Network Ports [2 per CRU] Serviceability Hot-swappable components CPU / I/O module, disks Operating System VMware vSphere Essentials Virtual Machine (VM) Operating Systems Customer-supplied VM OS2 2 MS Windows [2022] Data Center to be supplied separately along with software outside of the infrastructure scope. City of Redding TJC| JEGO Systems SCADA Infrastructure Compute & Storage Specification August 24, 2023 Page [4] Manufacturer Installation, Training, & Support Field Engineering Onsite Installation & Training System Integrator (SI) Engineering Partner Relocation Service - Install at SI re-verification at Customer Site Staging Support (for SI) 3 Months Customer Extended Platform (EPS) Support 3 Years prepay EPS Support: Uptime assurance software upgrades Software support response (critical): < 2 hours 24/7/365, Hardware support response (critical): < 2 hours 24/7/365 Initial incident response: 24/7/365, Advanced parts exchange/parts repair: Next business day Longterm Support Platform Support (Hardware) Minimum platform support for 15+ years Installed Product Base Uptime Availability on Installed Base Proven Uptime 99.999%+ Operational Technology (OT) Install Base Documented OT Customer Installed Base Fault Tolerance Technology Hardware Fault Tolerance (FT) Each Server must provide local computing, storage, and networking Fault Tolerance (FT) Workload Support Independence Each Server CRU shall be capable of supporting the total system workload while its partner CRU is out of service. 1.4 General Requirements 1.4.1 Embedded monitoring and diagnostics The system must provide a web-accessible diagnostic platform capable of self-monitoring and providing detailed information on the system, including alerts and component failures. The diagnostic report should be detailed enough to identify the specific failed component or service by non-technical operations staff. As an option, the system shall be capable of automatically initiating the next-day delivery of correct repair parts without impacting operations. 1.4.2 Hot-swappable servers Hot-swappable servers (CRUs) that comprise the computer and storage system must be capable of easy replacement live without any preliminary steps or intervention, without any special tools, and while applications continue to run. 1.4.3 Fault-tolerant (FT) technology Fault-tolerant (FT) technology must be a hardware-based solution for computing and storage. System availability promised by the manufacturer must be at least five nines uptime (99.999%) or better for the compute and storage system. 1.4.4 Internal Disk Storage Storage shall be built using multiple hot-swappable SSDs in a redundant array (RAID). RAID configuration options must tolerate multiple drive failures. The Server shall accommodate a minimum of eight hard drives. 1.4.5 Power Redundant power supplies with separate power cords. Power supplies must operate from 120 VAC. 1.4.6 Startup Assistance An option for startup service shall consist of onsite setup, configuration, and technical training including, but not limited to, the following tasks are required: Unpacking all system components, hardware module setup, base operating system installation, network configuration, internal data disk setup, and the testing of individual components. Following setup, test the system to ensure the system and software function correctly. Provide a written report documenting the testing results, the initial configuration of the system, and any reference documentation. GI " Y F CITY OF REDDING REPORT TO THE CITY COUNCIL MEETING DATE: December 6, 2022 FROM: Chuck Aukland,Public ITEM NO. 4.11(a) Works Director APPROVED BY*** n ..._ s.N,. ukl n l,H'ciblic Wc aks L'reGt ' t/?1/242? ry ip}in,C"i ana 1 L,'2/242? caulcland@ci.redding.ca.us btippin@cityofredding.org SUBJECT: 4.11(a)--A uthorize Sole-Source Purchase of Network Equipment for the Public Works De artment Recommendation Authorize the following actions relative to sole-source procurement of equipment for the City of Redding Water Utility: 1) Approve networking equipment from Cisco and Palo A1to in an amount not to exceed 450,000 for Fiscal Year 2022-23 to include system routers, ethernet switches, firewalls, and other network equipment and their associated licensing software and support service; and 2) Find that the action is exempt from review under the California Environmental Quality Act Guidelines,per Section 15301 —Existing Facilities. Fiscal Impact There are adequate funds in the Fiscal Year 2022-23 Water Utility budget for the $450,000 purchase of eguipment. Funding this purchase will not impact other activities. Alternative Action The City Council (Council) may choose not to authorize the purchase of network equipment directly from Cisco and Palo Alto and instead request quotes and award the various items through the standard competitive procurement process. This alternative may result in multiple manufacturers that are not compatible with the necessary equipment for the City's water system network design nor meet the City of Redding's (City) Information Technology (IT) Department's standard requirements to properly function with existing equipment. The Council may also provide alternative direction to staff. Report to Redding City Council November29,2022 Re: 4.11(a)--Authorize Sole-Source Purchase of Network Ec uipment Page 2 Background/Analysis In March 2021, the City's Water Utility contracted with communications consultants, TJCAA/JEGO Systems, to standardize and redesign the network/communication systems installed over the last 30 years. The current network system allows water treatment facilities, water storage reservoirs, groundwater wells, pump stations, and more to communicate via various types of radio telemetry and wide area network systems. The scope of work included evaluating the existing communications/networking system, identifying vulnerabilities, and recommending equipment to get to an acceptable level of cybersecurity protection. The redesign identified many obsolete products such as the cellular based Cradlepoint network that is past its useful end-of-life. Other potential cybersecurity risks were identified and new equipment is needed to resolve these issues. In 2020, the Water Utility also performed a risk and resilience assessment (RRA), per the United States Environmental Protection Agency's America's Water Infrastructure Act — Section 2013. Cybersecurity objectives identi ed in the RRA were designed into the new communication/networking system and will be implemented with this well-organized, reliable, and efficient communication system using the latest proven equipment, methods, and technologies. The City's IT Department was consulted to identify the most reliable and supported network equipment needed for the system including routers, ethernet switches, firewalls, and their associated licensing software and support services. The City uses a cooperative procurement method through the National Association of State Procurement Officials which would result in the purchase of these products at the lowest cost. The Cisco and Palo Alto prod ucts are a standard product offering that is used and integrated within the City's existing network system platform. IT staff are trained and certified to operate Cisco products. The water network design plan provided by TJCAA/JEGO Systems specifically recommends Cisco and Palo A1to due to their industry-leading quality and product availability. Based on previous experience, the consultant does not recommend building a network with an assortment of manufacturers that are not compatible with the network design strategy nor meet the IT Department's existing networking platform. Environmental Review Staff has reviewed the project and determined that the project is exempt from review under the California Environmental Quality Act Guidelines, per Section 15301 — Existing Facilities. Class 1 exemptions include the operation, repair, maintenance, or minor alteration of existing public structures, equipment, and facilities. The existing network and communication system will be replaced to meet modern security and efficiency standards. The project will not result in expansion of use, as the project does not include the construction of new facilities. The action area does not have significant values for wildlife habitat or other environmental resources and the project has no potential to have a significant effect on the environment. Report to Redding City Council November29,2022 Re: 4.11(a)--Authorize Sole-Source Purchase of Network Ec uipment Page 3 Council Priority/City Manage Goals Public Safety — "Work to improve all aspects of public safety to help people feel secure and safe where they live, work, and play in the City of Redding." Government of the 21st Century — `Be relevant and proactive to the opportunities and challenges of today's residents and workforce. Anticipate the future to make better decisions today." Attachments Notice of Exeinption lYQTIGE OF `XE PTION Tv; Office of Planning and Research F1toM: City af Redding 1400 Tenth Street,Room I21 Public Works Department Sacramento,GA 95814 777 Cypress Avenue Redding,CA 96001 Shasta County C1erk County of Shasta P.O:Box 990880, 1643 Market Street Redding, CA 96099-0880 Proaect Title; Water Utilit Network and Communications S stem Proiect J.4:#2209-51 Project Location—Specific: Various Water Utilitv faeilities Project Location—Cityt Reddin Project Location-County: Shasta Description of'rajecta The Citv of Redding Water Utiti will replace the existin network and communicatrons svstem with a new quipment and svstem that will provide efficient and reiiable camrnanicatlon usin;modern technolo,y The existin svstem wrll be replaced with a new network includin ethernet core switches ports routers software mountin clip, and licensin;. Work mav require the removal and ret laeement of equipment and cabling electrical madifications and renlacement or reconnection of associated equipment.All work will be conducted on Ci propertv Name af Public Agency Approving Project:City of Reddin Name of Persan or Agency Carrying Out Praject: Rvan Bailey,Assistant Public Works Director Exempt Status: (check one) Ministerial [Section 21Q80(b}(1); 15268] D clared Emergancy[Section 21Q80(b)(3); 15269(a)] Q Emergency Pro ect[Section 21080(b)(4); 15269(b)j tatutory Exemptions. State cade number: Common Sense Exemption(This project does not fall within an exempt class,but it can be seen with certainty that there is na possibility that the activity may have a significant effect on the enuironment(14 CCR 15061[b][3]). 0 Categorical Exemption. State type-and section number; Section 15301(a&b.Existin Facilities Na exceptions apply that would bar the use af a categorical xemption{Pl D 21 U84 and 14 CCR 1 S3Q0.2). Reasans why project is exempta Class 1 exemptians include the operation, repair maintenance or minar alteration of existin public structures, equipment and facilities. The existing network and communicatian svstem will be replaeed to meet modem security and efficiency standards. The proiact will nat result in expansion of use as the project does not include the construction of new facilities. The action area does not have si nzfcant vatues for wildlife habitat or other enviranmental resources and the proiect has nta potential ta have a si nificant effect on the environment Lead Agency Contact Person; Amber Kelley Telephone: 530.225.4046 If I+iied by Applicant: l. Attach certified document of exemption finding. 2. Has a notica of exemption been filed by the public agency approving the pro ect`?a Yes o No Sxgnature: Date: 1'`c C t c i Titles Envir nrnental Cam iiance na er O Signed by Lead Agency Date received far filing at OPR. Signed by Applicant i?OA i OG rev. 6/3??