Intelius, Bellevue, Washington
Senior System Administrator/Tools & Release Engineering Manager
November 2007 - present
- Support web, database, infrastructure, and development services on more than 400 Linux servers
- Perform administrative tasks on F5 BIG-IP load balancers and Juniper NetScreen firewalls
- Manage a team of three people responsible for developing operations tools, releasing code to production, and supporting web applications
- Serve as the secondary MySQL database administrator
- Track down errors and bugs in production PHP code
- Work with developers to identify operational requirements, characterize software behavior, and drive improvements and bug fixes
- Prompted improved communication and change management processes within the operations department
- Implemented a sophisticated centralized logging solution accepting up to 18 million messages per day from servers, firewalls, load balancers, and switches
- Upgraded website services to PHP 5 and Apache 2.2, making OS and application configurations more consistent in the process
- Participated in migrating the production data center with no downtime
- Improved the effectiveness and organization of a configuration management infrastructure consisting of Cfengine, Kickstart, MySQL, and DHCP
- Streamlined and standardized the Linux and Windows DNS architecture for better manageability
- Implemented secure, fault-tolerant LDAP directory services
- Initiated the removal of NFS dependencies in production
- Deployed a highly secure external access server, and drove access policy restrictions for another
- Redesigned automated network operations according to least-privilege principles
- Ensured ubiquitous deployment of strictly configured host-based firewalls
- Reorganized and simplified the configuration of a Request Tracker (RT) trouble ticketing installation, and supported outsourcing the internal help desk
- Developed a tool for continuous real-time recording of performance statistics from hundreds of Linux servers
Philips Electronics North America, Bothell, Washington
Senior UNIX System Administrator/UNIX Security Lead
January 2004 - November 2007
- Supported corporate infrastructure, engineering, and ERP services on more than 500 Solaris, SunOS, HP-UX, AIX, and Linux systems
- Performed administrative tasks on Network Appliance filers
- Introduced measures to significantly enhance security and meet Sarbanes-Oxley (SOx), ISO, FDA, and other audit requirements, including:
- Establishing administrative control over all UNIX systems at local site
- Disabling all non-essential network services, and applying access controls to those remaining
- Eliminating the use of cleartext network communication protocols
- Improving password encryption methods
- Disabling inactive login accounts
- Restricting privileged access
- Instituting a comprehensive and consistent system configuration process
- Managed automated password testing using John the Ripper, enhancing performance and effectiveness by:
- Improving the scalability, usability, and overall design of custom scripts
- Expanding the scope of testing from 3 to over 200 systems, and increasing frequency to near-daily
- Writing custom rules to catch common password patterns
- Implementing a policy of user notification and account locking
- Supported the rollout of ITIL processes for incident and change management, including development of a scalable tool for recording system changes
- Consolidated services and data to reduce maintenance costs and administrative overhead
- Performed basic ClearCase administrative functions, including VOB and view registry cleanup and data migrations
- Deployed servers used to support DNS, perimeter security, centralized logging, remote console access, manufacturing, and business applications
- Provided primary support for UNIX directory services, including:
- Streamlining and standardizing the DNS server architecture for better manageability and security
- Deploying and retiring NIS domains
- Devising a method to incorporate UNIX systems into an Active Directory infrastructure using Samba and custom scripts
- Establishing consistent DNS and NIS client configurations
- Developed a centralized logging infrastructure based on syslog-ng, Simple Event Correlator (SEC), and custom tools, which provided:
- Aggregation of system and application logs from dozens of UNIX systems
- Automated log reduction and analysis
- Real-time alerts and regular reporting of anomalous behavior, leading to greatly improved operational insight
- Secure source of event data for use in forensic analysis
- Implemented a simplified, secure UNIX email infrastructure
GeoPacket, Seattle, Washington
Software Developer / Project Manager
March 2003 - October 2003
- Developed software to analyze packet characteristics for real-time determination of call quality on voice-over-IP (VoIP) networks
- Formulated a patented call quality assessment method
- Coordinated the efforts of a small team of software and network engineers
- Managed relations with the customer supporting application development
- Served as a member of the board of directors
Amazon.com, Seattle, Washington
UNIX Systems Engineer
November 2001 - November 2002
- Supported business-critical back-end services on over 400 Linux, HP-UX, Solaris, and Tru64 servers
- Managed highly available services utilizing Cisco load balancers, HP ServiceGuard clusters, and other HA technologies
- Accepted increasing responsibilities as several sysadmin functions were merged while headcount and capital expenditures were reduced
- Provided primary server support for crucial business functions, which at various times included:
- Inventory and supply chain management
- Corporate finance and human resources
- Electronic payments
- Fraud detection
- Customer service
- Oracle database services
- Delivered escalated response, resolution, and root cause analysis for critical events
- Conducted dependency and performance analysis and optimization
- Completed scaling requirements analysis and server deployment for the holiday season
- Participated in planning and troubleshooting for HP storage arrays
- Coordinated the migration of more than half a terabyte of live data between NetApp filers with minimal interruption to services
- Planned and deployed the server architecture for a redundant secure payments site
- Assisted in developing and testing a corporate disaster recovery plan
- Initiated an effort to minimize server login access
- Implemented regular network scanning of outsourced customer service sites to monitor compliance with contractual agreements
- Improved the effectiveness and organization of a centralized configuration management system
edge2net, Kirkland, Washington
Network Services Engineer
March 2001 - October 2001
- Managed services for a global telecommunications network on about 20 Solaris, Linux, and AIX servers
- Provided primary support for many network services, including:
- Network monitoring (OpenView NNM, Big Brother)
- Directory services (DNS, LDAP)
- Voice-over-IP calling card IVR service
- Fax-over-IP mail service
- Conducted a transition of directory service from NIS to LDAP
- Secured network communications via a scalable, cross-platform IPsec solution
- Extended the capabilities of host and network monitoring platforms with custom software
- Implemented an automated file distribution mechanism used in providing IVR service and in device configuration
- Instituted a comprehensive and consistent system configuration process
Bazillion, Seattle, Washington
Applications Group Manager
August 2000 - January 2001
- Managed a team of six system administrators supporting Internet and messaging services on about 80 Linux, Solaris, CMC UNIX, and Windows servers
- Coordinated a merger of personnel and responsibilities resulting from a corporate acquisition, and grew the team to its final composition
- Conducted budget planning and made purchasing decisions
- Designed a highly scalable, fault-tolerant distributed service infrastructure
- Supervised testing and deployment of new unified messaging services
- Implemented comprehensive monitoring of hosts and network services
- Supervised documentation of troubleshooting and escalation procedures
System Administrator
March 2000 - August 2000
- Supported Internet services for a nationwide startup communications provider on about 30 Linux and Solaris servers
- Provided primary support for system administration functions, including:
- Network information and authentication (DNS, DHCP, NTP, RADIUS), email, and Web services
- Service provisioning and usage collection
- Server capacity and budget planning
- Procedural documentation
- Abuse incident response
- Developed software to implement robust and secure provisioning and mediation of distributed Internet and VoIP services
- Automated gathering of metrics to aid in proactive troubleshooting and capacity planning
- Enhanced host and network security
- Instituted a comprehensive and consistent system configuration process
- Specified standard server, storage, and backup systems
Metapath Software International, Bellevue, Washington
Lead Security Administrator / Senior UNIX System Administrator
April 1998 - March 2000
- Supported corporate infrastructure services and software development on about 70 Solaris, HP-UX, Linux, DYNIX/ptx, and AIX servers
- Participated in managing a global network of over 1000 nodes using Cisco, Ascend, Bay, 3Com, and Livingston network gear
- Managed a distributed perimeter security system composed of firewalls (Check Point, Gauntlet), choke routers, and secure Internet links (VPNs)
- Provided primary support for information services (DNS, NIS, NTP), electronic mail, and network news
- Rearchitected the global network infrastructures for email, information services, logging, and perimeter security
- Instituted a comprehensive and consistent system configuration process
- Automated Linux-based archival backups of PCs for use by support personnel
- Implemented a security policy restricting privileged access
- Introduced a coherent system support policy
- Participated in a long-term program to improve corporate IT effectiveness
- Assisted in planning the relocation of the headquarters office
- Made purchasing recommendations and participated in budget planning
Test Engineer
February 1998 - April 1998
- Led testing of network driver software critical to a customer release
- Documented test results and procedures
- Trained colleagues in software testing techniques used
Customer Support Engineer
September 1997 - February 1998
- Supported operations in production network environments for wireless carriers using proprietary software and hardware
- Performed administrative functions on customer DYNIX/ptx, AIX, and LynxOS systems
- Performed basic Oracle database administration on production systems
- Installed telecommunications equipment at customer sites
- Composed documentation for use by customer support staff
GTE Network Management Organization, Bothell, Washington
Technical Consultant
June 1997 - August 1997
- Performed administrative tasks for a network of over 150 Solaris and IRIX systems
- Automated network management tasks using shell scripts
- Implemented network performance monitoring
- Conducted performance tests of network file system protocols
- Administered a Network Appliance filer
- Assisted in planning and implementation of a transition between
NIS+ and NIS
PSW Technologies, Austin, Texas
Associate Software Engineer
July 1996 - June 1997
- Supported corporate infrastructure services and software development on over 300 NEXTSTEP, AIX, Solaris, Auspex, IRIX, and Windows systems
- Performed administrative tasks for a nationwide network using Cisco, Ascend, and Telebit gear
- Provided primary support for network security, account management, backups, network news, and resolution of internal support issues
- Maintained directory services (DNS, NIS, NetInfo, NT Domains), file services (NFS, Samba), printing facilities, and electronic mail
- Established a common software repository
- Implemented a security policy restricting privileged access
- Aided in the planning and execution of a successful corporate relocation, including:
- Restructuring the local network for a higher-bandwidth, switched architecture
- Assigning IP addresses in the renumbering of all network hosts
- Writing scripts and checklists used in reconfiguring hosts for the new network
- Provided NEXTSTEP, NetInfo, and general network consulting for external clients
Zilker Internet Park, Austin, Texas
System Administrator
November 1995 - July 1996
- Provided technical support to customers of a respected local Internet service provider
- Managed DNS, Web, email, Usenet, FTP, and other Internet services on SunOS and Solaris servers
- Maintained network connectivity for customers using Livingston, Ascend, and Specialix gear
- Developed Web tools for customers using Perl CGI scripts
- Organized staff meetings and training sessions
University of Texas Computation Center, Austin, Texas
Computer Programmer / Services Assistant
March 1995 - March 1996
- Provided front-line technical support to users working in UNIX, VMS, Macintosh, and Windows environments
- Composed documentation for use in standardized email responses
- Developed software used in the customer support environment
- Performed limited system administration tasks on SunOS, Solaris, AIX, Digital Unix, OSF/1, Ultrix, and VMS servers
- Trained and supervised new hires
Advanced Micro Devices, Austin, Texas
Co-op Associate Engineer
August 1990 - May 1991
- Constructed and documented computer models used to refine microprocessor fabrication processes
- Trained colleagues in the use of a Domain/OS engineering workstation and wrote a user's manual for its operation
- Performed backups and other routine system administration tasks
