Christopher Litsinger |

Background

I started my career as an engineer to build things, and loved doing it. I worked at the intersection of coding and infrastructure before there were terms like “DevOps”, “SRE”, or “Platform Engineering” to describe what I did. I built things. For years, managers would see something in me that made them ask, “would you like to be a manager?” My easy answer was always, “no thanks, I like building things.” I built bigger and bigger things until I found myself working on an amazing team, building a public cloud platform that would become that company’s first product to earn $100 million in annual recurring revenue.

It became clear this team needed a leader who would stick around long enough to fix some problems. And I saw a new opportunity to build something I hadn’t before — an environment where a diverse global team could thrive and deliver. So, this time, when I was asked, I answered, “yes, I like building things.” I built it using the same iterative team-driven exploration that built great platforms.

I haven’t looked back. Building teams that build platforms that help engineers build software works for me. Like I said, I like building things.

----alt header-----

I build strong engineering teams that develop and support critical back-end services at a global scale of billions of transactions per day. I have a history of assembling diverse, talented teams and creating an environment where they thrive and deliver. I improve processes and culture through iterative, team-driven exploration. I bring a mix of leadership, collaboration, technical expertise, attention to detail, and exceptional communication skills. I have experience building services, platform solutions, and multi-tenant Infrastructure as a Service (IaaS) platforms with co-located, remote, and hybrid teams. I work to continually improve myself as a leader and technologist and share what I have learned.

I have developed and proven these skills in a broad range of settings. Most recently, I led a small team working across Comcast Cybersecurity to develop better DevSecOps and agile processes, improve product mindset, and support line-level managers in developing their leadership capabilities.

Professional Experience

Comcast | Philadelphia, PA

Sr. Director, Software Development & Engineering; Comcast Cybersecurity | July 2020 — May 2024

I was responsible for four Platform Engineering scrum teams inside of Comcast’s Cybersecurity department, building and operating five projects impacting every part of the business. We delivered on business requirements for these internally developed tools while continually improving the security posture of Comcast’s services and reducing developer friction.

In addition to continued leadership of the API gateway solution used broadly throughout Comcast, I was selected as a turnaround leader for critical platforms that were underperforming. I worked with my teams to deliver significant and measurable technology, culture, and process improvements.

Most recently, I led a small team working to assist software engineering teams throughout Comcast Cybersecurity build better processes and approaches to software engineering, focusing on DevSecOps and agile practices, product mindset, and development leadership.

• Maintained a continual focus on improving CI/CD processes: expanding platform presence into a new internal cloud platform as well as extending our public cloud presence internationally in support of global expansion efforts enabling blue/green deployments and faster release delivery.
• Migrated a legacy service from jars deployed to virtual machines to blue/green containers running in AWS, resulting in an 83% reduction in mean time to repair.
• Oversaw improved eNPS scores, with workplace satisfaction increasing 156.4% while motivation scores increased 64.8% year-on-year in the midst of the global pandemic.
• Improved mean time between failure (MTBF) for an inherited solution by 383% year-on-year.
• Delivered features required for the commercial release of XClass TV (a retail smart tv platform) in 33% of the requested timeline; allowing for an accelerated development schedule and a successful market release.
• Focused on latency reduction: delivered support for connection reuse; reducing median latency for critical services by 40% while also reducing overall engine CPU utilization by 20%, delivered support for topology based routing. Reduced latency for a critical API by 76.6%
• Built support for open standards on critical platforms such as Open ID Connect (OIDC) support.
• Delivered a token exchange and related features, allowing syndication partners to deliver unique features such as partner-specific purchase flows or chat support in syndicated applications.
• Increased utilization of our API platform to over four billion transactions per day, with peak throughput reaching over 157k requests per second across heterogeneous virtualization environments throughout Comcast’s footprint. Increased utilization to 2500 Services with tens of thousands of credentials.
• As a member of the Comcast Cybersecurity DE&I Working group, developed and executed programs such as executive listening sessions and a trans identity movie discussion group.
• Worked with teams across Cybersecurity to improve product management awareness and practices, establishing a product guild and helping teams across Cybersecurity develop and execute Customer Empathy Workshops where we observed developers using in-house tools and developing action plans based on our findings.
• Technologies: TypeScript, React, Mantine, Node.js, Lua, Java, Spring Boot, Concourse, AWS, ECS, ECP, VPC, Route53, IAM, RDS, OpenStack, VMWare, Docker, git, Ansible, Terraform, PostgresDB, DocumentDB, MongoDB, Prometheus, Grafana, ElK, OIDC, OAuth, JWT, Linux

Comcast | Philadelphia, PA

Director, Core Application Platforms | December 2014 — July 2020

I was brought into this position to build an in-house replacement to a commercial API management gateway with a small DevOps team. We successfully designed, developed, and delivered a platform built with custom Lua plugins implemented in nginx, seamlessly migrated all services, built a self-service user interface, and have operated and enhanced it through sustained growth during my tenure.

Working in the office of the Chief Software Architect, I participated in the creation of our software strategy: kicking off an internal SRE community, sitting as Chair of the API Security Working Group, and participating in the enterprise’s evolving cloud strategy. I also led research projects in distributed observability and edge computing, and developed hiring practices that have been broadly influential within the company.

• Maintained stability through a 300% increase in peak traffic for the previous platform over the first 7½ months of production and over 30x growth in five years with flat staffing levels while delivering an overall 9 nines of reliability.
• Delivered customer-centric products: meeting with key stakeholders as Product Owner, executing customer empathy workshops, and chairing working groups to ensure continual feedback.
• Migrated over 4,000 keys to 400 APIs from our legacy platform; a transition completed without requiring changes to the service or calling application.
• Improved consistency and reduced bias in our interviewing process while streamlining the candidate experience by implementing research-driven enhancements.
• Met stringent security requirements for code, design, and operational procedures — earning a “Most Boring Security Audit” award.
• Participated in various enterprise-level bodies including the API Security Working Group, SRE and DevOps Communities, Coaching and Mentoring through Employee Resource Groups, the Diversity and Inclusion Steering Committee, and Rotational Program Advisory Panel.
• Technologies: React, Node.js, Lua, nginx, Jenkins, Concourse, AWS, EC2, ECS, VPC, Route53, IAM, RDS, OpenStack, VMWare, Docker, git, Ansible, Terraform, PostgresDB, Prometheus, Grafana, ElK, Splunk, OIDC, OAuth, JWT, Linux

SunGard Availability Services | Philadelphia, PA

Director, Cloud Platforms | October 2013 — December 2014
Sr. Manager, Cloud Engineering | September 2012 — September 2013

I led a 27-member global technical team of managers and engineers responsible for full lifecycle for software and platform development for Cloud Services.

We delivered a new Public Cloud service offering to market, built on Cisco’s ACI platform and Apache CloudStack’s provisioning engine, while maintaining and enhancing a legacy virtualization platform that achieved 85% growth in 2013.

• Delivered product earning $100 million in annual recurring revenue.
• Delivered a 66% improvement in release times and a 50% decrease in testing times.
• Transformed relationships with our India-based team through structured programs increasing productivity, participation, and leadership.
• Achieved PCI compliance for our legacy Cloud platform on time, meeting contractual deadlines.
• Maintained high retention during significant technical change and transition of executive leadership.
• Recruited top talent — filling key management, software development, and infrastructure engineering positions.
• Partnered with an outside vendor to deliver an elegant and efficient user interface for our IaaS Platform on-time and within budget.
• Coached individual engineers to drive career and personal growth.
• Collaborated with managers to maintain team morale, retention, and productivity.
• Improved communication between QA, operations, and program and product management.
• Oversight and procurement for a $3.5 million engineering lab to support Cloud Engineering efforts.
• Prioritized projects, new features, and production defect fixes and supervising all work through release.
• Reduced obstacles to engineering productivity.
• Engaged other development teams within SunGard to share code and technologies.
• Organized monthly sprint planning, review of product requirements, and roadmap overview sessions.
• Balanced engineering assignments across multiple critical projects to ensure success while accounting for personalities, career development, and work styles.
• Partnered with the QA team to improve levels of automated testing for all projects.
• Technologies: Openstack, Cloudstack, Python, AQMP, git, Docker, Puppet, Cisco ACI, Cisco Nexus, EMC Clarion Storage, SolidFire, Juniper Routers, Juniper Switches, Fedora Linux

SunGard Availability Services | Philadelphia, PA

Systems Engineer — Enterprise Cloud Platforms Team | July 2011 — September 2012

Working as a member of the product development team for Enterprise Cloud Services (ECS), an IaaS platform built in-house — with an orchestration engine written primarily in Python, an AMQP message queue, and a Java/GWT front-end; controlling an infrastructure platform featuring VMWare, Cisco UCS servers, Nexus switches, and EMC storage.

• Contributed code and unit tests to Python- and Java-based APIs and services to improve ECS automation.
• Developed onboarding plans and training new engineers.
• Supported an AMQP environment for dev and QA orchestration environments.
• Conducted and contributed to code reviews.
• Wrote, edited, and organized appropriate documentation for contributed and improved code and processes.
• Assisted as needed with production issues, platform documentation, quality assurance, integration testing, and release management.
• Developed and implemented User Acceptance Testing procedures for the ECS platform.
• Completed overhaul of our puppet manifests and node classifiers to improve maintainability.
• Updated orchestration software to run on a new revision of Fedora, while maintaining backwards compatibility.
• Assisted QA in automating technical testing of VMware hosts.
• Built a SOAP interface to the Service-Now ticketing system using Python and suds.
• Trained new software engineers on all aspects of the orchestration system.
• Technologies: Cloudstack, VMWare, VCenter, VSphere, Python, Django, AQMP, RabbitMQ, git, Docker, Puppet, MySQL, EMC Clarion Storage, Juniper Routers, Juniper Switches, Fedora Linux, RHEL Linux

SunGard Availability Services | Philadelphia, PA

Service Catalog Manager | March 2009 — July 2011. Philadelphia, PA

Built processes and user interfaces for Service Catalog Management. Participated in strategic planning, customer-focused initiatives, and process and policy design.

• Established Business Service Catalog Processes and Policies.
• Designed, built, and deployed an in-house Technical Service Catalog using PHP, MySQL, Zend, and JQuery.
• Defined Service Owner roles and responsibilities related to the Service Catalog.
• Trained and worked with Service Owners to populate and verify the Service Catalog.
• Developed automated systems to provide lifecycle reporting for Service Assets, interfacing with vendor sources to automate access to Service Contract and Equipment end-of-life information using Python, modPython, and MySQL.
• As a senior developer and team lead, participated in the design of a Service-Now based ITSM and ticketing system.
• Reduced a set of over 2,000 complex ticket assignment rules to a simple lookup table of less than 200 rows.
• Conducted an application and code security review of the Service-Now deployment — identifying and correcting system issues.
• Led code reviews for a small team of Service-Now developers.
• Tracked and reported for management teams across a diverse organization.
• Established strategy, objectives, and execution plans for the Service Delivery team and overall organization.
• Implemented improved service delivery processes and procedures.
• Provided technical project management and team leadership.
• Established metrics and reporting on key elements of Service Catalog.
• Technologies: Python, modPython, Django, git, MySQL, Zend, JQuery, Fedora Linux, Service-Now

Dechert LLP | Philadelphia, PA

Senior Development Engineer — Network Operations | September 1999 — December 2008

Designed and built automation and tools for a Network Operations department in a medium-sized enterprise. Managed and supported a wide variety of technologies including servers, routers, switches, email, firewalls, VoIP, supporting applications, and monitoring tools.

• Automated Windows and Exchange account creation procedures.
• Automated patch notification and scheduling for over 200 Windows servers.
• Designed and deployed a wide area clustering solution for disaster recovery utilizing Neverfail, NAS technology, and VMWare; managing resources from multiple departments within IT over several months.
• Rebuilt a SNORT based IDS solution and built custom reports with PHP.
• Assisted in the design, construction, and move of a world-class data center.
• Designed and deployed a Voice over IP infrastructure built around Cisco Call Manager.
• Developed custom applications for automating the build of Cisco IOS configuration files for routers and switches.
• Built a centralized repository of security events using Perl, VBScript, syslog, and MS SQL Server.
• Designed and implemented a Network Operations knowledge management system using PHP, MySQL, and Apache.
• Contributed updates to open source software such as Reptor.
• Provided 24/7 end-user support for those environments.
• Developed business cases for the purchase and deployment of technology solutions.
• Managed projects such as the deployment of new or updated technologies.
• Developed and executed departmental training sessions for new technologies.
• Developed technical documentation and trained IT staff for multiple projects, including commercial and in-house software.
• Technologies: Windows, Exchange, PERL, Python, VBScript, VMWare, MS SQL

CTNY Creativity | New York, NY

Director of Development | September 1998 — September 1999

Responsible for oversight of the creation of a development division within a New York-based start-up. Responsible for hiring and training a software engineering team to provide custom solutions as well as complete internal development projects such as “ThinAir Mail”, a PalmOS mail client that was later acquired by Palm Computing.

• Developed standards and policies for software development projects.
• Recruited and built a team of developers.
• Completed oversight over the full application lifecycle for projects written in VB, C++, Java, and ASP.
• Technologies: Visual Basic, C++ Palm SDK, CodeWarrior, Delphi, Java, Active Server Pages

Dechert, Price & Rhoads | Philadelphia, PA

Internal Technology Consultant | July 1995 — September 1998

Worked across internal departments automating processes with a small consulting team reporting to the CIO.

• Led a development standardization effort and managed legacy conversion to client server applications.
• Evaluated development tools, selected firm-wide development toolsets.
• Solidified the development team into a cohesive and effective department.
• Authored coding standards, documentation, and development processes.
• Designed and coded all aspects of the first firm-wide intranet using ASP and MS SQL.
• Created automation workflow software for document conversion leveraging Ole Automation, VBA, COM+, SMTP, and SQL.
• Created a set of Microsoft Office templates integrating back end-data sources.
• Recruited and built a team of developers.
• Completed oversight over the full application lifecycle for projects written in VB, C++, Java, and ASP.

The Technology Group | Baltimore, MD

Help Desk Technician | June 1994 — July 1995

Provided customers of word processing and document automation software with customer support.

• Installed and configured a small, PC based PBX controlling 30 phones.
• Using Delphi, developed call tracking workflow software with an API interface to the phone system.
• Performed QA testing of commercial offerings’ performance on pre-release operating systems, including Windows 95 and OS/2.

Education

University of Maryland | Baltimore, MD

B.A., English

Community / Talks

Philadelphia Software Leadership Bookclub | Organizer

OSCON: Building Shared Services with an Open Source Approach | https://youtu.be/AYVYeI1hj98

PANMA: How and Why We Diversified our Team | https://youtu.be/oqOKfACN2iw

NGINX Conf: Help Yourself! | https://youtu.be/VruwciRwGbw

Devopsdays Philadelphia: Ten Product Mindset Rules for DevOps Teams | https://bit.ly/4fEglEZ