Career Opportunity

PRINCIPAL CLOUD ENGINEER

Recruitment: RTF0135168-01130947

Published: August 08, 2023

Apply using SmartRecruiters, the City and County of San Francisco's application portal.

About:

IMPORTANT: Your application MUST include responses to the supplemental questionnaire found here: https://forms.gle/QCGomdhbKiomViVt5

Applicants must complete the supplemental questionnaire prior to starting the online application process. The application will not be reviewed without a completed supplemental questionnaire. 

Why Work for the Department of Technology (DT)? DT is the centralized technology services provider in the City and County of San Francisco (CCSF). We deliver technology infrastructure and services to approximately 33,000 employees! With an annual operating budget of over $140M and approximately 260 employees, DT provides a host of services that range from Public Safety radio and wiring and Network services to Enterprise Support and the Cloud.

Benefits of Working for CCSF:  In addition to challenging and rewarding work, the City provides a generous suite of benefits to its employees.

• Job security, pension, and robust retirement options
• Competitive pay with consistent bi-yearly or yearly increases
• Hybrid Work with a minimum of 20% of time spent in our office in San Francisco, California for all IT related roles
• Generous paid time off, family leave, and more!
• Diverse work environment in a diverse city
• Union protections and representation
• Career development and growth — move between departments, learn on the job, or take subsidized/reimbursed classes!

PEOPLE-CENTERED SOLUTIONS — have a powerful, meaningful effect on the community each day with people at the heart of every solution!

CLOSING THE DIGITAL DIVIDE — bring the benefits of the internet to low-income and marginalized residents!

SHINE A LIGHT ON WHAT MATTERS — join an award-winning production team at SFGovTV to help residents watch legislators or learn more about what makes this City great!

DRIVE INNOVATION — deliver new, cutting-edge technology to residents and city partners to help San Francisco serve its residents!

Role description

The Cloud Center of Excellence (CCoE) is responsible for mission-critical infrastructure used San Francisco wide to provide essential public services. The CCoE provides a centrally managed infrastructure service that is offered to 50+ City and County of San Francisco departments. The service offering includes both design and architecture, as well as daily operational support to ensure optimal performance, security, availability, resiliency, and cost for commercial cloud providers and services.

The ideal candidate is excited about bringing their technical savviness to serve San Franciscans through improvement and modernization of IT infrastructure, as well as advocacy of public cloud services.

Position:

The City and County of San Francisco, Department of Technology (DT), is seeking a highly experienced Principal Cloud Engineer to help design, develop, and maintain commercial cloud infrastructure servicing multiple City and County of San Francisco departments. This position will have a focus on Disaster Recovery and increasing resiliency of City services by leveraging of public cloud infrastructure.

Job Responsibilities:

• Act as a cloud team lead.
• Contribute to the thought-leadership for building cloud infrastructure for resiliency.
• Proactively work with the DT Disaster Recovery (DR) team and city departments to lead the cloud infrastructure development of application and system DR including data migration.
• Assist leadership in the documentation of process and procedures for system DR recovery during an incident or emergency.
• During any emergency or outage, provide technical leadership, troubleshooting, and system recovery.
• Contribute cloud expertise and knowledge for the development of city commercial cloud services that are optimized for multiple city departments and applications.
• Consult with and advise business partners on  best practices, efficiencies, and economies of cloud technologies.
• Architect, build, operate, deploy, and maintain secure, scalable, and highly available commercial cloud infrastructure.
• Migrate business systems and data to commercial cloud infrastructure for production and disaster recovery environments.
• Develop and maintain software solutions/frameworks to automate cloud configuration and administration in commercial cloud platforms.
• Design and implement cloud-native solutions for rapid but reliable feature enhancements and conduct cloud capacity planning.
• Collaborate with city development, applications, operations, and security teams to ensure solutions meet functional and non-functional requirements.
• Gather and analyze metrics from both operating systems and applications to assist in performance tuning and fault finding.
• Configure and deploy cloud cost management, cloud budgeting, and cloud capabilities at scale.
• Create and implement cloud infrastructure best practices and guidelines to support audits and various compliance.
• Develop and leverage automation tools to ensure consistent, efficient, transparent, and secure operational and fiscal management of commercial cloud environments.
• Enhance and maintain disaster recovery and business continuity plans, including identifying critical systems and designing and implementing backup/restore processes.
• Implement and maintain operations monitoring and alerting systems to proactively identify and resolve issues/outages and serve as 24x7 lead for escalating problem resolution with cloud providers.
• Create, manage, and maintain Operations Scope of Procedures (SOPs) consisting of standard operating procedures, configurations, lessons learned, root cause analysis, diagnostic steps, and solutions to resolve incidents.
• Research and evaluate industry trends to continuously improve infrastructure solutions.
• Provide 24-hour on-call support to ensure rapid recovery from software or hardware problems for mission-critical systems and networks.

Job Type:

The Permanent Exempt - Full Time position is excluded by the Charter from the competitive civil service examination process and shall serve at the discretion of the appointment officer. The anticipated duration of this project position is thirty-six (36) months and will not result in an eligible list or permanent civil service hiring.

Nature of Work:

Incumbent must be willing to work (specify the work schedule; for example, a 40-hour week) as determined by the department.  Travel within San Francisco may be required. 

The incumbent must be a resident of the State of California or be willing to relocate within 4 weeks of beginning employment with the City and County of San Francisco.

Work Location:

Incumbent will conduct the majority of work at the Department of Technology, (1 S Van Ness, Ave San Francisco, CA 94103).  However, there may be situations where the incumbent will be required to work at other sites throughout the City of San Francisco as necessary.

This position does not support fully remote work. Employees may be permitted to work a hybrid schedule with supervisor approval, after which they must work at least two days in the office every two weeks.

How to qualify

Education:

An associate degree in computer science or a closely related field from an accredited college or university OR its equivalent in terms of total course credits/units [i.e., at least sixty (60) semester or ninety (90) quarter credits/units with a minimum of twenty (20) semester or thirty (30) quarter credits/units in computer science or a closely-related field].

Experience:

Five (5) years of experience analyzing, installing, configuring, enhancing, and/or maintaining the components of an enterprise network.

Substitution:

Additional experience as described above may be substituted for the required degree on a year-for-year basis (up to a maximum of two (2) years). One (1) year is equivalent to thirty (30) semester units/ forty-five (45) quarter units with a minimum of 10 semester / 15 quarter units in computer science or a closely related field.

Desired Qualifications:

• Bachelor's degree in Computer Science, Information Systems, or related field OR 5+ years of recent and verifiable experience building and managing commercial cloud environments.
• Experience with Disaster Recovery planning and designing resilient cloud systems within large enterprise environment.
• Experience supervising small teams and providing leadership.
• One or more cloud certifications (e.g. AWS/Azure/GCP Architect or Developer) strongly desired.
• Strong knowledge of at least one or more major cloud platform (AWS, Azure, Google Cloud, OCI).
• Experience with multi-cloud networking (e.g. VPC, VPN, Direct Connect, ExpressRoute).
• Hands-on experience with cloud-native technologies and concepts (serverless, containers, etc.).
• Experience with Infrastructure as Code automation tools and languages (Terraform, ARM/Bicep, Puppet, PowerShell, Bash, Python, etc.)
• Experience with disaster recovery technologies and tools.
• Solid knowledge of networking technologies and concepts.
• Experience with data center consolidations, relocations, or moves. 
• Experience in government and/or public safety environment. 
• Familiarity with DevOps principles and methodologies.
• Self-motivated with time management skills.
• Excellent problem-solving and troubleshooting skills.
• Strong communication and collaboration skills to work effectively with cross-functional teams.

Applicants must meet the minimum qualification requirement by the final filing date unless otherwise noted.

Note:

1)  Security Clearances & Background Investigations: Criminal Justice Information Services (CJIS) Security Clearance may be required. Positions in this classification may require that successful candidates who become eligible for appointment may be required to go through a background investigation to determine the candidate's suitability for employment in this classification. Factors considered in the investigation may include employment history, use of illegal/controlled substances. Reasons for rejection based on this investigation may include, but not limited to applicable convictions, repeated or serious violations of the law, inability to accept supervision, inability to follow rules and regulations, falsification of application materials and/or other relevant factors. Failure to obtain and maintain security clearance may be basis for termination.

Verification: Applicants may be required to submit verification of qualifying education and experience at any point in the application and/or departmental selection process. Written verification (proof) of qualifying experience must verify that the applicant meets the minimum qualifications stated on the announcement. Written verification must be submitted on employer’s official letterhead, specifying name of employee, dates of employment, types of employment (part-time/full-time), job title(s), description of duties performed, and the verification must be signed by the employer. City employees will receive credit for the duties of the class to which they are appointed. Credit for experience obtained outside of the employee’s class will be allowed only if recorded in accordance with the provisions of the Civil Service Commission Rules. Experience claimed in self-employment must be supported by documents verifying income, earnings, business license and experience comparable to the minimum qualifications of the position. Copies of income tax papers or other documents listing occupations and total earnings must be submitted. If education verification is required, information on how to verify education requirements, including verifying foreign education credits or degree equivalency, can be found at http://sfdhr.org/index.aspx?page=456.

Note: Falsifying one’s education, training, or work experience or attempted deception on the application may result in disqualification for this and future job opportunities with the City and County of San Francisco. 

What else should I know?

Compensation: $150,124 - $188,812 (annually) / $72.1750 - $90.7750 (hourly)

How to Apply:

• Applicants are encouraged to apply immediately as this recruitment may close at any time, but not before May 10, 2023 at 11:59PM.
  • Your application MUST include a resume.  To upload, please attach using the "additional attachments" function.
  • In additional to submitting your application, you MUST submit this supplemental questionnaire: https://forms.gle/QCGomdhbKiomViVt5

You may contact Carol Wong via email at carol.x.wong@sfgov.org with questions regarding this opportunity.

Late or incomplete submissions will not be considered. Mailed, hand delivered or faxed documents/applications will not be accepted.

Right to Work:

All persons entering the City and County of San Francisco workforce are required to provide verification of authorization to work in the United States.  

CONDITION OF EMPLOYMENT:  All City and County of San Francisco employees are required to be fully vaccinated against COVID-19 as a condition of employment. Someone is fully vaccinated when 14 days have passed since they received the final dose of a two-shot vaccine or a dose of a one-shot vaccine. Any new hire must present proof of full vaccination status to be appointed. Any new hire who will be routinely assigned or occasionally enter High-Risk Settings, must provide proof of having received a COVID-19 booster vaccine by March 1, 2022, or once eligible.

The City and County of San Francisco encourages women, minorities and persons with disabilities to apply. Applicants will be considered regardless of their sex, race, age, religion, color, national origin, ancestry, physical disability, mental disability, medical condition (associated with cancer, a history of cancer, or genetic characteristics), HIV/AIDS status, genetic information, marital status, sexual orientation, gender, gender identity, gender expression, military and veteran status, or other protected category under the law.