Transformation of an Engineering Organization
I have completed 5 years at Dow Jones today and its been an incredible learning journey, where I got to work with some of the best teams a leader could ask for and a great Leadership team that enabled and empowered me to drive and be part of many transofrmation initiatives. In this post I am sharing our engineering principles that has allowed our engineering team to transform into a world class engineering organization.
Dow Jones Engineering organization is composed of a diverse and global team of professionals, who collaborate to achieve desirable business outcomes by building world-class products and systems.
As part of Technology’s transformation journey, in the past few years our Engineering team has continued to focus on culture, invest in people and adopt best practices across all functions.
Our leadership has enabled this cultural transformation by empowering teams and the broader engineering community to experiment with their solutions, design and implement best practices and reduce friction.
We have achieved a lot in this journey, but have clear goals on where we want to go next and are continuing to identify and explore opportunities to evolve and take new challenges. Our engineering practices that have helped us immensely in this journey follow these principles.
Collaboration
Collaboration has been the key to our success in recent years, enabling us to share a unified vision and deliver on our roadmaps with very lean teams.
Sharing Over Standardization: We follow a non-prescriptive approach to our development practices and have driven cultural shifts at a grassroots level to share our solutions and ideas, whilst breaking development islands in an effort to foster innovation without diluting our technology stack and adding maintenance cost.
Center of Excellence and Architecture Reviews: As part of our technology evolution we share our designs and architecture decisions across the department at multiple stages to gather feedback and ideas that help us accelerate our development cycle and build solutions that can scale and evolve. Discussions with the Architecture Technology Committee (ATC) that I am currently Chair of and Distinguished Software Engineers (DSEs) have been at the forefront of these reviews along with the engineering leadership and these reviews have been really valuable for cross team collaboration and preventing many teams from reinventing the wheel.
Innersourcing and Open sourcing: We trust in the modern software development community and prefer to use open source software and solutions where possible. We have also given back to the community by open sourcing our own solutions that has improved our brand recognition within the open source community. People recognize DJ for the Dow Jones tools and Dow Jones contributors contributing to open source projects. We also innersource to share solutions across the teams with DJ.
Agility
We are truly agile in our approach and prefer agility over following agile processes, where agility implies character or behaviour that you possess, or that you don’t.
People over processes: Every team adopts and evolves their processes using true agile principles to employ best practices that suit individual team needs, stakeholders requirements and the stage of product life cycle. Our program and project management are decentralized to enable autonomy within teams and more impactful delivery for the business.
Roles and titles: Our career ladders and Engineering pillars framework have enabled us to create an engineering team focusing on growth of individuals and building strong teams. Teams and individuals adapt to the roles and responsibilities required for them to be efficient and productive. Engineers have a leadership choice between people management and recognized engineering expertise as an individual contributor (e.g. DSEs).
Empowerment and Accountability: Teams and individuals are empowered to take the best approach for their products & platforms in the context of the business challenges they face. This is coupled with a high degree of transparency and insistence on accountability to stakeholder needs which the teams communicate via our Objectives and Key Results framework (OKRs) which includes KPIs.
Continuous Improvement
We believe in an iterative approach incorporating retrospection and continuous feedback in all aspects of running our teams and delivery efforts.
Resiliency: Every team continuously focuses on resiliency of their systems. Our Product health initiative has incorporated a framework that enables teams to prioritize resiliency efforts in their regular sprints.
Continuous Learning: With ever evolving technologies and new tools available, we enable our engineers to grow and acquire these skills through various means of learning. Speaking at and attending external events, hosting internal talks and learning sessions, training in industry standards and earning technical certifications are all an important part of our culture.
Platforms as Products: In an effort to maximise the value our platforms and shared services provide while minimizing the delivery time, risk and waste involved we are evolving our approach toward building, leveraging and maintaining these services by adopting a product approach. Where the customers to these platforms are internal teams and stakeholders.
Modernization
Dow Jones has a long history of serving our customers and has continuously evolved our platforms and services to meet the customer needs and requirements. We believe in modernization with pragmatism and have adopted new technologies and tools to deliver quality products.
Cloud Presence & optimization : We have modernized the majority of our legacy systems by moving them into the cloud under our LAMP (Legacy Architecture Modernization Program) program and now evolving our cloud-first approach as we build new systems and platforms. Through our cloud cost councils we continuously find ways to use the best of tools available to build the systems while keeping the costs down. Mosco is one of the initiatives that is in progress and would help in further reducing cloud costs.
Microservices: As we build new systems and re-architect existing systems to optimize and bring in efficiency of cloud environments, we are building our Microservices architecture strategy. This will allow us to evolve our technology stack and enable the rapid, frequent and reliable delivery of large, complex applications.
Data Pipelines: Data is at the heart of what we do and is the foundation for our personalization and customer experience efforts. There are multiple steps involved to create AI-ready data for machine-learning algorithms and any other regression analyses that need to be performed on the data. To enable this process we are building our Data Pipeline that would eventually give us easier access to information and insights, enable faster decision-making and provide agility to meet peaks in demand.
Embedded Data Science: We have embedded data scientists across product and business teams to integrate data science in our people, processes and products. Data scientists partner with engineers to build applications ranging from predictive machine learning models to descriptive analytics tools. Outstanding examples include WSJ’s dynamic paywall, MarketWatch content automation, look-alike audience segments for digital ads, and separate, specialized analytics tools for editorial staff and advertising sales. There’s further data science development powering Personalization which will be our big focus going forward.
Quality
We strive for quality above all else in everything that we do, and we have been very successful in delivering high quality products and platforms to our customers and building world-class products
Continuous Integration and Delivery: Our CI/CD adoption has helped us release small incremental software updates to deliver quality software. We have extensively shared our CI/CD solutions across teams and championed adoptions of various technologies including Terraform, Docker, Jenkins and built our own solutions like Chronoa to help deploy software.
Automation: Our automation and overall testing efforts help us to be truly agile. Where needed, we have built and shared our own custom frameworks and strategies to test our applications while leveraging open source solutions. Our SRE, Devops and QUEST efforts truly are the torch bearers of quality software and continuously share their solutions and learnings across the teams.
Security Champions: Security is one of the most important parts of quality software, and we have built a culture through training and security champions programs to include security aspects in early stages of the software development life cycle. We also continuously enforce these during architecture reviews.We developed and open sourced Reapsaw a continuous security devsecops tool, which helps in enabling security into CI/CD Pipeline.
Published Dec 7, 2020