Author: Anthony Bull

Categories
Backend eCommerce Frontend Performance Engineering Process

What To Do With a 9s Page Load?

(Decorist : 9/18-12/19)

Challenge

Inherited a two-fold challenge: 1) less-than-optimal user experience of 9.01s Avg. Page Load Speed and 2) a culture that did not yet value performance engineering.

Action

Idenfitied initial frontend and backend low-hanging fruit (e.g. page structure, image resolutions, N+1 queries, lazy-loading, etc.) Identifed and delegated KRs to FE leads. Introduced process to methodically follow up each week, pressing the case for performance engineering over months.

Below, a sampling …

Some initial efforts shaved ~3s of the Homepage:

Lighthouse score soared from 5 to 61:

There were multiple speed improvements of up to 50% on various pages across the site as a result of backend query improvements and a +300% bump YoY in SEO traffic (verified by 3rd party SEO consultant)

After Dec 2019, we shifted focus to non-converted UXes after having addressed low-hanging fruit for all UXes.
Sept 2018 to Dec 2019

Results

  • Record Avg. Site Page Load Speed of prevous four years : 9.03s (Sep 2018) to 4.13s (Dec 2019.)
Categories
Architecture eCommerce Management Process Re-platforming Roadmapping SOA

Changing Engine Mid-flight : Again

(Decorist : 8/18-12/19)

Challenge

Take a half-baked v1.5 SOA and continue re-platforming to support scaled integration with corporate parent and its subsidiaries.

Action

Aug 2018

When I joined, it was with the idea that I would be instrumental in helping scale the Decorist relationship-creation paradigm leading to higher AOV to its corporate parent – Bed Bath and Beyond – and its subsidiaries: Buy Buy Baby and Cost Plus World Market.

I inherited the beginnings of a movement away from a Django/Angular/MySQL monolith and towards a Django/React/Postgres multi-tenant platform; a v1.5 of the application architecture. My predecessor had extracted some facets of the monolith to build out the initial stages of the SOA in partnership with another Bed Bath and Beyond subsidiary: One Kings Lane.

Although labeled a multi-tenant platform, it had been built quickly – effectively as a prototype – to service one tenant and was not truly extensible for other silos, though it did encompass solid SOA principles for reuse.

Dec 2018

As part of roadmapping for 2019, wanting to strategically position ‘scalability’ for the corporate org, I worked with my boss – SVP Prod/Tech – to detail a project plan to leverage the exisitng nascent SaaS paradigm for the next generation of the Decorist user experience.

We were sanctioned by the CEO for reduced scope on the roadmap; it would be the very first effort internally for moving towards a new platform to future-power the site. We focused on porting a back-office, manual task of determinging supply-side match and availability with demand-side need.

Wanting to keep a engineer engaged, I shepherded him and helped him understand the Cost-Benefit trade-offs we would need to make in order to meet the deadline, setting a course of re-using the User service for authentication, builidng out a new React-based UI, and integrating with the monolith.

Feb 2019

A few months later, we delivered the feature of supply-side Matching and Availability:

Shortly after launch, business priorities shifted towards a focus on a new eCom offering and we had to back-burner re-platforming efforts.

Dec 2019

Given guidance that 2020 might be the year we could re-visit re-platforming, I ideated around what was still lacking in v1.5 and what would need to be built out, creating a project plan and technical roadmap for the go-forward.

I worked with my boss again around planning and prioritization and we again provided guidance to CEO. Ultimately, we could not secure the necessary corporate integration buy-in for further progress and have had to defer any further progress given other priorities.

Results

  • Mentored Lead Engineer towards implementation of first feature for new SOA paradigm.
Categories
Distributed Teams eCommerce Management Process Release Management

Managing Distributed Teams (pre-COVID)

(Decorist : 8/18-12/19)

Challenge

Inherited a new, flat, remote, full-time team (web) of 25 (mostly) junior engineers in Delhi when dominant local (SF) office culture was not optimized for remote work. Needed to transform them into Bay Area-level talent while maneuvering them into a manageable hierarchy while at the same time incorporating a contracting team (VR) of 13 in Lahore

Action

Team Organization

Re-organized (and became productive) as Tribes.

Transitioned work culture to remote by getting stakeholders more comfortable with off-peak-hour meetings.

Processes

Introduced Agile paradigm / SCRUM meetings.

Leveraged Google Forms to craft surveys towards quantitatively and qualitatively bettering processes.

Introduced story-point estimations.

Began mastering Jira / work-breakdown through Epics, etc. Started Automated Testing.

Introduced Coda for Program Management

Performance

Tried Koan for performance management and remote visibility.

Introduced reviews and performance tracking with PeopleGoal, empowering local Indian General Manager with structural tools.

Introduced OKRs.

Introduced accountability through personal / sprint goals.

Introduced bi-weekly series email “Get to Know Me!” (as managed through Google Forms); highlighting two Delhi team members per installment. Also eventally created team videos using smartphone-captured content, stitched together through iMovie.

Releases

Created branching and release strategies.

Provided team with Release Notes template for improving communication with stakeholders.

Speed

Incorporated Performance Engineering mindset.

Quality

Led by example by including hyperlinks and project IDs in correspondance.

Reminded people to provide more context in Jira stories.

Reminded QA to provide easy, concise steps-to-verify and testing credentials to stakeholders.

Introduced the concept of “Bug Severity,” Estimated Time to Resolution, and a simple Google Form for reporting (and emailing) bugs to multiple engineering stakeholders who can triage.

Overall, the whole company has seen a qualitative improvement of content of communications between distributed offices; we are all now incorporating more details leading to faster turn-around of higher-quality code.

Results

  • Created outcome-driven processes, best practices, structure, and mentorship for a new, offshore engineering team.
Categories
Distributed Teams Hiring Management

Hard Leadership Lesson

(Decorist : 11/18-12/19)

Challenge

In spite of multiple attempts to provide coaching over six months with repeated support and patience, realized team’s General Manager wasn’t working out as the technical peer/partner I needed for local representation.

Action

Given his importance, was wary of exiting him so did my best to coach him. After 6 months, realized the fit was never going to materialize and worked with local HR to exit him in an amicable fashion.

Then, worked with Manger’s line supervisor and Account Manager towards finding a replacement. Was delighted to find someone more technical, less traditionally represented in such roles, and seemingly eager to take on the challenge.

Worked with her for several months until she exited of her own accord.

Results

  • Replaced remote non-performant General Manager, twice.
Categories
Emails Process Site Reliability Troubleshooting

Flying Blind

(Decorist : 5/19-10/19)

Challenge

Lack of logging/monitoring (including for email sending/deliverability) made it impossible to know if integrations were affecting system/application uptime.

Action

Odd for an email marketing company: stakeholders would ask about email deliverability and engineering had no insight because no loggging/tracking had ever been instrumented.

Planned initiatve for adding integration monitoring while digging deep into application code and Amazon SES, instrumenting new features going forward, but given time and resource constratints, not retrofitting existing ones.

Results

  • Created visibility into metrics for company’s key component: email marketing.
Categories
Database Monitoring ORM Performance Engineering Troubleshooting

40s to 10s

(Decorist : 10/19-10/19)

Challenge

Business-critical page used internally and externally was taking 40s to load, then started timing out for everyone.

Action

  • Dug in, realized DB CPU Util was pegging at 100%, found and killed runaway DB process.
  • Setup alarm to be notified whenever CPU over 70%.
  • Page was still taking 40s, realized queries weren’t being logged, turned quereies on, saw LEFT OUTER JOINS across eight large tables.
  • Removed six lines of Django ORM select_related syntax without affecting page functionality.

Results

  • Dropped page load speed for critical internal Admin UX from 40s to 10s.
Categories
Distributed Teams Hiring

Needle in a Haystack

(Decorist : 9/19-9/19)

Challenge

Kept receiving candidates sourced by onsite India HR more corporate than the startup profile actually needed for the team.

Action

Posted Technical Architect, Scrum Master, and Full-stack Django Engineer JDs to Linkedin India, then screened potential candidates.

Results

  • Paid ~$100 total for well-qualified Technical Architect (4/35) and Scrum Master (12/50+) applicants.
Categories
Architecture CMS eCommerce SEO

Better SEO

(Decorist : 9/19-9/19)

Challenge

Deliver SEO value faster.

Action

Prior, the dominant in-house paradigm did not realize the potential of Django to serve in a CMS capacity.

I introduced the idea to to leverage Django as a CMS to power Landing Pages, like this one:

Results

  • Empowered Marketing to manage content and reduced time-to-delivery for Landing Pages by 30%.
Categories
Process

Env Consistency

(Decorist : 8/19-8/19)

Challenge

Since the beginning of the company, six disparate environments had organically come into being, each powered by their own DB – unnecessarily complicating testing.

Action

Introduced daily process for prod data replication with scrubbed PII propagating to other environments.

Results

  • Homogenized environments, empowering staff to use prod login on any env, greatly simplifying testing and
Categories
Building buy-in Distributed Teams Performance Engineering Process Roadmapping Strategy

Leveling Up

(Bluxome Labs : 6/18-7/18)

During the course of June and July 2018, I supported leadership (CTO/VP ENGR/Head of Platform/VP PROD/CEO) around engineering topics such at Brazilian-startup Pipefy around the codebase, Product Engineering, SDLC process, culture, and roadmap(s,) transitioning from a monolith to Service Oriented Architecture, platform strategy, scalability, maintainability, and uptime.

Among other contributions, I:

  • Coached CTO and VP ENGR around operational excellence, particularly thinking in terms of AS-IS versus TO-BE.
  • Introduced idea of maturing IT processes towards forecasting, in particular via a Capacity Plan.
  • Provided thought-leadership around managing remote teams, partly out of own experience, partly as informed by Best Practices.

Results

  • Identified strategies and tactics to qualitatively improve processes.