AWD is requesting proposals for general integration and programming services to support the utility Supervisory Control and Data Acquisition System. Perspective Automation Solution Providers shall consider the following See Compensation Table Summary: Integration providers should take into account all possible scope requirements listed below while also considering travel expenses, changes in technology and working inside of both the Legacy HMI and new Rockwell SCADA architectures of the AWD system.
Compensation Table Summary
SCADA Service Hours
500 Hours Provide quotation for typical non-emergency SCADA Services and Integration
Emergency Hours
100 Hours Provide quotation for on-call emergency services. This will include any after-hours work once defined by AWD and the vendor.
Engineering Services
100 HoursPerspective Automation providers may be required to assist in the development of new technologies and SCADA control
Networking, IT and Cyber Security
100 HoursBidders shall provide network design, consulting, architecture development, communications and network security services based on NIST & CSET
Administrative
100 HoursInvoicing preparation, As-builts, drawings, P&ID Control Narratives or other related clerical service
Travel ExpensesBased on 800 hours of services bidders shall consider travel expenses and include these expenses in the bid base total.
Bidders shall provide a base bid for 900 hours of services based on the included compensation table. Bidders shall provide an hourly rate table defining service within the compensation table and accompanying total base bid. AWD reserves the right to utilize all or a percentage of those hours within the fiscal year based on programming service needs, emergencies and project development. All ancillary costs should be considered and added into service rates and proposed in the total base bid. AWD will not incur the cost for any integrator to become certified, qualified or trained in any SCADA software related, HMI related or other contracted services technology or exercise. AWD will not compensate vendors for any time needed to familiarize integration staff or technicians with the system.
Special Considerations:
Concepts SCADA
Bidders shall understand the concept and requirement of open architecture and be experienced and well versed in all of the technologies in this specification including: Wonderware, Sci-Tech XL Reporter, Allen Bradley RS View, Plant PAX, FactoryTalk, RSLogics5000, Inductive Automation Ignition, Sonic Wall, Hyper-V Manager, Modicon Concept, Unity, Windows Server 2102 as well as legacy operating systems such as, Windows Server 2003 and Windows XP.
The existing system consists of the Legacy Wonderware SCADA system residing on a dedicated server with a parallel Rockwell System currently being developed on separate servers. The methodology employed as the basis of design is to develop the HMI and transition each process area of the treatment facility and all remote SCADA locations into the new system. Upon completion, final decommission and deletion of the Wonderware HMI shall take place. With each process transition, upgrade of each process PLC and any other obsolete device shall also be upgraded or replaced as needed to realize a system and hardware that is up the latest technology and standards. Perspective bidders will be required to work in conjunction with in-house SCADA and instrumentation staff to continue; design, build and deployment of the new system. Within the scope of these services the vendor shall also work to develop and maintain an upgraded SQL Based automated reporting package and historian.
In FY2020 AWD deployed VEEAM as a backup and recovery solution for the Wonderware system. This deployment was based on findings of the DOHS Cyber Security Evaluation performed in 2017/2018. Perspective bidders shall have experience working with VEEAM software and be able to work to transition the VEEAM Server onto the Rockwell system as development makes this work priority. This work shall include further development to backup these systems to a remote location for further reliability and security.
Since FY2017 all legacy equipment has been replaced including the original Wonderware Tower Server, switching devices and firewalls. The vendor will be allowed to install a developmental license on either of the Wonderware or Rockwell SCADA servers in order to perform all tasks. At the end of each work session any DEV license shall be removed and the AWD runtime license shall be entered back into production. Under no circumstance shall the SCADA system operate for a sustained duration under a vendor’s license.
Ignition SCADA is an ever expanding control system within the AWD Waste Water Collections Utility. Ignition was initially installed as a data collection tool and view only HMI and will now be utilized as a fully functional control system with deployment into the Beaver Creek Storm Water Treatment Facility. As an extension of AWD, the contracted SCADA services provider shall support any and all needs as related to this system to the same level all SCADA systems are maintained within the scope of this specification. AWD works with multiple Ignition Certified Engineers and expects each vendor to have the ability to work with and maintain professional relationships with each other in order to provide continued support and reliability of the Ignition System.
Concepts Network Security
Perspective bidders shall have extensive knowledge of the Cyber Security Evaluation Tool and NIST Framework and have a strong networking group in order to support future network security measures. The vendor shall also work with in-house SCADA personnel to develop a Standard Operating Procedure (SOP) in which all programs related to PLCs, HMI, programming/engineering and developmental software and any related tools are compiled into one engineering device (i.e. Laptop or engineering server) owned and managed and properly licensed by AWD. This will eventually be the only tool allowed to connect to the control system unless a device is first completely scrubbed and approved by AWD.
Concepts Level of Effort
All perspective vendors will work closely with SCADA and Instrumentation staff from AWD. It is the goal of AWD to manage, maintain and develop a robust SCADA system in-house, with less reliance on vendors for purposes other than emergency services in the future. Without divulging in-house trade secrets or violating software related licensing conflicts, it is expected that the vendor shall work in close concert with in-house SCADA and Instrumentation on any and all maintenance activities as well as any newly developed project related to SCADA automation and control system appurtenances. In house personnel will work and assist vendors with any and all projects as needed. This concerted level of effort allows for greater production and efficiency in returning systems to production in the event of failure. In turn the SCADA services provider shall train in house staff in any basic functionality that allows efficiency in repairing and maintaining all systems.
Transparency
The SCADA Services vendor WILL be completely transparent in any and all services performed. To this requirement the vendor shall continue to develop in the open architecture environment. The system shall remain free of any from of proprietary application, software, password protection or means of authentication. The vendor shall work to create an environment in which any future integration firm or integrator could develop with proper credentials. The vendor shall work in concert with in-house SCADA Management and transmit/divulge any password development or lockdown mechanism to AWD. Password and all IP scheme documentation as such will be recorded in spreadsheets and handed down to AWD immediately upon deployment into system production. All passwords and IPs must be available at all times and easily accessible to AWD Management. Any software, materials or any other purchase made for the purposes of servicing, maintaining or developing the system shall be made in the name of the Albany Department of Water and its specified representative. Neither purchases nor service shall be performed without prior authorization from AWD in writing. AWD reserves the right to void the SCADA Services Contract upon any deviation to these requirements.
On Site and In House Work Requirements
While it is understood by AWD that there will be instances of in-house emergency work that may be performed by the SCADA Services vendor, it is a requirement that the largest percentage of work will be performed on-site at any one of the previously named locations. Due to new network security architecture, and AWD SOPs, remote access will only be granted via Global VPN access routed to the centralized network hub and from there, outward into the system. NO direct connections to specific locations will be constructed or allowed now or in any future network architectural configuration. Any in-house work performed by the vendor shall first be approved by an AWD representative in writing. Approval can be given through e-mail correspondence. Fully defined justification for in-house hours along with an explicitly detailed description of work performed must be accompanied with invoicing. Invoicing for materials procurement must also include the original receipt form the vendor. Any work or materials purchased and invoiced without prior justification WILL NOT be paid by AWD.
Mobilization
AWD will NOT be responsible for mobilization and or training costs for integrators to become familiar with the SCADA System. The existing system is based around an open architecture environment and shall remain free of any proprietary applications that would prohibit future integrators from affectively servicing, programming and maintaining the SCADA System. AWD will allow any and all required on-site time for these purposes and assist in providing information, documentation and related materials in order to work efficiently in program development of the system. These hours used to train and familiarize vendors with the system shall not be charged to AWD. AWD SCADA/Instrumentation Technicians shall work in close concert with the eventual services provider to assure transitional efficiency and continued SCADA functionality.
Materials Procurement
In the event materials procurement is required to support SCADA functionality, AWD must agree to pricing provided by the vendor before the purchase is made. This approval must be made in writing. AWD will not accept any material purchased without prior approval. Vendors will allow complete transparency to all purchasing invoicing. AWD will not accept any materials in excess of a 10% markup by the vendor to their original quotation provided by the manufacturer. Invoicing for materials shall include the original receipt from the manufacturer or the vendor. Any software purchased or deployed in the functional or engineering environment shall be purchased and licensed in the name of AWD. All logic developed in the HMI, PLC or other SCADA related field shall be the property of AWD and utilized for the sole purpose of functionality in the control automation system.
Extended Qualifications
Upon request all prospective Integration Solution Providers must be able to produce examples and detailed justification related to the following qualifications. Integrators must have or can demonstrate:
- Integrated three (3) or more Allen Bradley SCADA/HMI projects with tag counts in excess of 500. These examples should include custom and packaged graphic development, PLC programming, compound PID control loops and multiple communication protocols including but not limited to (MODBUS, Ethernet TCP/IP, Analog and Digital I/O).
- Integrated or serviced three (3) or more Wonderware Systems or projects.
- At least one HMI programmed in Rockwell Automation Plant PAX and 5 years of verifiable experience with Factory Talk View and RSLogix5000.
- At least (2) examples of projects integrating Inductive Automation Ignition SCADA with tag counts in excess of 300. Must be able to prove experience with both Vision and Perspective Modules.
- The ability to build, construct, assemble and label panels according to UL 508 standards. If the perspective integrator is not a UL 508 rated shop, confirmation must be provided of the ability to provide panels, control cabinets and systems that are appropriately rated to this standard. Examples, pictures, drawings etc. of control panels manufactured and labeled under the UL 508 standard.
- Adequate staffing to dedicate one (1) engineer/integrator and one (1) additional technician at any given time to emergency services, planned services or project development. In the event of overlapping projects, the integrator must be able to dedicate multiple engineers and technicians to dedicated projects at the same time. Emergency services include any after-hours time on nights or weekends. This is considered essential on-call emergency services as identified in the Compensation Table. Ability to respond in a duration of no greater than 4 hours from notification of an emergency to any AWD location is required.
- Have an experienced network administrator or associated network services provider, with at least 10 years of service, experienced in the security standards proposed within the NIST Standard. This administrator must have the ability to create, service and maintain all VPNs, switching devices, routers, firewalls, backup devices, systems and software (VEEAM), structured cabling systems communication protocols (cellular, Leased fiber) etc. in order to lockdown, maintain and or optimize the network architecture of the associated SCADA system.
