Drive Productivity with Engineering Pillars
Today every company in the software industry is trying to explore ways and ideas to make their engineering teams more effective and productive. Google did an extensive research across their teams to find out how to build the perfect team and found that psychological safety plays a very important role in making engineers more effective and happy in their teams. At Dow Jones we have been exploring our own ways to not only build effective teams but also increase overall employee satisfaction.
While working with my teams based out of Minneapolis for last 10 months I have been building a framework which allowed me to not only effectively manage teams from NY but also improve overall productivity of engineers. I have been evolving this framework to be able to implement this across teams irrespective of size, location, product and skills of the teams.
The four engineering pillars framework covers all aspects of an engineering team under 4 themes: People, Delivery, Resources and Architecture. Strengthening each pillar and its main features helps in building an effective and productive engineering team. We are using these pillars to identify the areas we need to improve upon for each team, how to identify and allocate right people to own these areas and build upon them, they are also serving as the main foundation for our goal setting exercise for engineering teams.
Engineering Pill
PEOPLE
Humans have been scared of technology and automation advancement for number of years, but irrespective of the progress “People” have and would remain the most important resource for any company and this strongly applies for our software industry as well. Among all the pillars I am most passionate about People’s pillar and consider this as the core responsibility of my role as an engineering leader. This is the reason the first area that we chose to work on within Dow Jones was to make Dow Jones a great place to work for all our engineers. I have been inspired by “The Advantage” book written by Patrick Lencioni, he talks about organization health as the greatest opportunity for improvement and competitive advantage in businesses. I have been focusing a lot on my team’s health through open communication, politics free environment and providing clarity around each areas of individual and teams.
Career Ladder
This month we rolled out Dow Jones career ladder to define clear career tracks with unified titles for our “single engineering” team in both leadership and technical areas. This has been a huge achievement given the size of our company and how much variations we had in our titles for different locations with many different teams introduced to the company through acquisitions over the last 30 years. I have been working with our People team on this for over 6 months and the result have been overwhelming. This could be a no brainer for some startups but for enterprise companies this one is easier said than done.
Roles & Responsibilities
While rolling out the career ladder we also defined clear path for progression and movement across two tracks, one for people leadership and one for technical leadership, with clear objective roles and responsibilities. These roles and responsibilities ultimately tie up to the overall engineering pillars structure and helps in defining what role an individual can play to strengthen the pillars on their team. These roles and responsibilities also help in defining clear goals for individuals and prepare the groundwork for future promotions. We are also supporting career growth through our training and conference programs and also encouraging engineers to speak and participate at industry events.
Empowerment
“As we look ahead into the next century, leaders will be those who empower others.” - Bill Gates
You can give best tools and technologies to someone but if they are not empowered they cannot be really productive. The above two aspects are meaningless if engineers are not empowered to lead, make decisions or ask questions in their individual roles. A lot of my focus while working with my teams has been to empower and enable them in their roles to make decisions and drive results. I learnt the hard way as an engineer to delegate but it has been so far the best trait I have developed as a leader. Delegation with empowerment has been the key to successfully manage teams remotely.
Collaboration
The most effective teams are the ones which collaborate at all levels to reduce duplication, avoid miscommunication and improve overall productivity. While working with my teams we continuously come up with ideas and focus areas to increase collaboration with other teams. This becomes more critical given that we are distributed across continents and have lot of vendor engagement. The biggest area we are championing here is around engineering productivity, sharing best practices on CI/CD, AWS migrations, Testing, automation and release infrastructure. We are already seeing lot of benefits in a short period of 6 months in the form of cost savings and learnings.
DELIVERY
Delivery of product and services successfully is the true measure of any team’s effectiveness.There are many roles in an organization that play a role to deliver a great product. Identifying and clarifying each person’s role and responsibilities through the product life cycle is very critical to achieve desired results. These roles help drive various benefits across different aspects of delivery. Delivery aspect was particularly challenging as some of the teams I am working with had not adopted agile and have been working in a specific way and team structure for years.
SDLC
This was one of the very initial tasks I undertook at Dow jones where we reached on an agreement to have a generic SDLC (Software Development Life Cycle) established that all teams can adapt in their own ways to deliver products and services. Although loosely defined it has clear checkmarks that every team must follow to complete certain defined processes and tasks along the development life cycle.
Standards
As we have been working on our collaboration piece we came up with certain engineering standards and practices that each teams must follow and implement as part of their technical roadmaps. Some of the important areas we are addressing here are :
Code review process
Security
Testing (TDD, Automation Pyramid)
Metrics
Monitoring
Logging
Performance
Continuous Integration and Delivery
Communication
As part of our overall delivery process an effective communication strategy is critical, we have been practicing open communication within the team around goals, milestones, budget, issues and any other common topics. As some of our teams are co located and have been working on their own for years our first strategy was to get everyone adopt a common agile workflow. Within less than a year teams are fully integrated and communicate regularly through standups, planning meeting, triage, agile boards. We formulated a business request intake process and channel that into our backlog for effective tracking of regular production issues and feature requests.
Roadmaps
Since we have a better communication strategy it leads into defining of clear roadmaps and milestones for each team. We now have very clear roadmap through the year, that gives us clear understanding of work that needs to be done and an increased visibility to business around deliverables along with timelines. Having clear roadmaps was more important for some teams as their work spans across multiple Dow Jones products with different stakeholders.
Releases
With adoption of agile and moving our legacy on prem systems into cloud we are moving towards continuous delivery and are breaking some of our huge code bases into micro services to allow more independent releases. With some of our new teams we are building our strategy to be able to do smaller releases to a subset of user. Smaller and frequent releases with fewer changes are less risky and more effective. To be able to support multiple releases with every team we have targeting to get fully automated testing and deployment in place.
RESOURCES
For engineering teams to operate they need various types of resources and support. For many teams it’s either internal full time employees or they partner with other companies/vendors to provide specialized resources. This pillar provides the foundation for any given team and would define their success with right skills and people in place.
Hiring
As I mentioned earlier people are the most important resource for any company and the investments made during hiring process result in building right people resource. As an engineering organization we heavily depend on referrals to hire new people into our teams. For some of my teams 100% of our hires in last year were referrals which has resulted in amazing and efficient teams. We are also focusing a lot on diversity as part of current hiring strategy and across the company we are driving initiatives to encourage more inclusive hiring process.
Budget
As an enterprise company our budget process is a little challenging, it’s an extensive process and we rely heavily on our project managers for budget planning and tracking. We are slowly moving away from our annual budget planning to quarterly planning at team level. This is helping us to allocate and move money to right projects through the year. Our clear roadmap strategy is also driving better budget planning, with an effective and simple budget teams focus more on deliverables then worrying about money.
Capital (SOP)
Being a company with history of number of years we have a huge systems which require both operations and capital work. We invest equally on both type of projects to keep our products running and innovate to be ahead of the industry. This year we have invested huge amount of time, money and resources on major part of our systems to get rid of legacy. As part of this program one of my teams is moving all our systems into AWS which would allow us to scale and improve performance of our systems supporting real time data on our websites.
Time tracking
As I mentioned earlier that we are keeping a close track of our capital, we have to track our time spent doing various activities under it. Although considered as boring and time consuming time tracking helps stay within the budget. To make this process less painful we have simplified our budget systems and also have taken the responsibility away from engineers to log time in most of the scenarios. This allows engineers to focus on the work they are good and passionate about which is true engineering work.
Vendors
Being a really large organization we rely heavily on partners and vendors to drive our roadmaps and support our internal teams. We prefer to work with limited partners to maintain high quality and easier movement of resources. We use partners in both specialized areas and staff augmentation to handle our burst requirements. We also use crowdsourced testing services to meet some of our QA needs and rely heavily on internal automation testing for regression.
ARCHITECTURE
Architecture is the backbone of every product and needs utmost attention from the engineering team. As part of our career ladder roll out we have strengthened our individual contributor and senior engineering roles. This has empowered people in these roles to make more effective decisions that are right from technology perspective and keep us on track with tech debt in long run. Every role on our technical track has an architect’s responsibility with respect to the level and experience in a given system and team.
Technical Roadmap
Along with product roadmap we have built technical roadmap for each team to help them prioritize tech tasks and address technical issues on time. The technical roadmaps include tasks related to cloud infrastructure, CI/CD setup, automation, testing, releases, monitoring and various others. We make sure to involve our project and program manager in technical decisions to help them understand priorities and drive the backlogs through the sprints.
Chief Architect
Although we don’t have a role as chief architect within our organization we have our senior most engineers play these roles across teams to define and share our technical designs and decisions. Specifically on my teams multiple people play this role as we have to integrate with multiple products. We have formed internal architects group who meet on regular basis to exchange learnings and make decisions. We also partner with AWS engineers to help us make right technical decisions for our cloud applications.
Governance
We have set up internal governance groups to facilitate peer based collaboration and to promote alignment over architecture, engineering, and technology choices. These governance groups are more of an informal process to help us increase transparency and improve cross organizational visibility, sharing and inputs. We are also in the process of building our Developer handbook which would establish various technology and development standards.
This is the beginning of an exciting phase for Dow Jones engineering teams with our primary focus on Innovation and Delivery we will be able to achieve further excellence across all the engineering pillars. As we strengthen our engineering teams and underlying technology for our world class news products we hope to bring the best of it to our customers and end users.