Author: Anthony Bull

New VR UX For Nurseries

Post author By Anthony Bull
Post date October 22, 2020

(3/19-10/20)

Challenge

Ship new, web-based, VR-powered, eCom experience as requirements changed.

Action

Act 1 : 3/19-5/19

While on my trip to India in April 2019, I formed a Tiger team of one of my best Lead Engineers, a project manager, and two junior engineers, coaching them to see the similarity between what was the business was asking for and current existing components of the system (PLP, PDP, NUX, and Checkout,) setting a plan in motion towards delivering an MVP for the 6/26/19 deadline.

They went heads-down and we successfully shipped v1 (following) on 5/29/20.

ecom landing page

Business priorities shifted and the project was moth-balled, leading to Act 2.

Act 2 : 8/19-10/19

Having shifted focus to more product-based eCom (see Act 1,) the business decided to leverage existing shop-the-room modeling infrastructure in a more user-friendly, web-based purchase flow.

While the original plan was to have them spin up a completely new POC with a new checkout flow, I intervened and met with the remote Technical Project Manager and Architect, providing guidance around the existing monolith marketplace system, knowing it could serve as enough of a “buy” to meet requirements so as not to have to “build” a custom solution.

Shipped v1 in Oct 2019:

Landing Page

Room Detail Page

I saved $40K in redo work after guiding the non-primary, remote, web team around component re-use while then shipping web-based, VR-powered shop-the-room.

Act 3 : 3/20-10/20

Under tight deadline, coached the Pakistani team to iterate and improve perceived and actual load times using CSS Sprites, caching via HTTP headers, use of a spinner, and gzipping in order to get a usable UI to market sooner:

Landing Page

Drilling down, a user looking to design a nursery can swap out items (made possible by a compositing technique with Three.js and photo spheres)

Room Page

Lastly, recognizing future strategic value-add within corporate partnerships, guided the team to decouple the frontend as a Single Page App for iframe embedding after having decreased page load times, introduced progressive enhancement / graceful degradation, and led the SPA strategy.

Results

New VR-powered site finally launched in Feb 2021.

Tags Django, Three.js, Unity

Collaboration Database Troubleshooting

Fixing Database Replication

Post author By Anthony Bull
Post date June 21, 2020

(6/20-6/20)

Challenge

Usually our MySQL->Postgres data replication powering the business’s analytics dashboard runs without issue. One day, it completely failed.

Action

Spotted AWS alert notification and worked with Data Engineer to realize it wasn’t a normal hiccup in our replication pipeline.
Stopped the AWS DMS task for replication.
Examined Cloudwatch logs to see if we could find direction as to where the problem was. This told us there was a table (awsdms_apply_exceptions) that didn’t exist.
Dug into online documentation about the issue.
Created a new Postgres copy target of the analytics database.
Created a new AWS DMS task with the copy target DB which should create table public.awsdms_apply_exceptions.
Grabbed the DDL statement (e.g. CREATE TABLE) for the awsdms_apply_exceptions table.
In the (original) analytics DB, created the ‘public’ schema
Also in the (original) analytics DB, applied the CREATE TABLE for awsdms_apply_exceptions.
Deleted 1) the new Postgres copy target and 2) the AWS DMS task as cleanup.
(Never did figure out why the public schema and table disappeared.)

Results

Resolved data replication issue leading to minimal downtime for analytics dashboard.

Tags AWS DMS, Cloudwatch, Postgres, SQL

Forecasting Management Process Troubleshooting

Rolling With the Punches

Post author By Anthony Bull
Post date June 2, 2020

(11/18-5/20)

Challenge

Finding the right-sized engineering team as the business ebbed and waned.

Action

Nov 2018 – Mar 2019

Two months after joining, in Oct 2018, was surprised by request to provide KLO budget slashing engineering by 60%.

Having only a basic understanding of team members’ strengths and weaknesses, I anticipated the following year’s needs and then presented guidance w/SVP PROD & CEO to corporate parent COO.

We secured fiscal year funding to ensure team/business continuity @ 38 headcount.

Dec 2019

In Nov 2019, was informed we needed to reduce our 25 person Delhi team to 8 for the 2020 fiscal year. I looked over the skillsets of the team and – with a solid year under my belt and experience of who top performers were – decided who would stay.

Saved company $800K by reducing engineering headcount from 38 to 25.

Mar 2020

Then, with the onset of COVID, needed to further reduce team-size, for both India and Pakistan teams.

Given impact of COVID, made decisions leading to add’l $840K reduction from 25 to 9.

Results

Adjusted team size as necessary to meet needs of the business.

Architecture Chat Collaboration Integration Site Reliability SOA Troubleshooting

Connecting Supply with Demand

Post author By Anthony Bull
Post date June 1, 2020

(10/18-5/20)

Challenge

Ongoing reliability issues with 3rd party chat solution crucial for business operation w/o documentation and integration monitoring.

Action

The key aspect of the Decorist experience is the connection between the Client (Demand-side) and Designer (Supply-side.) To facilitate that connection, many years prior, the business had included a (at that time) nascent Chat-as-a-Solution provider as part of the user experience; it made sense to “Buy” instead of “Build.”

Architecture Overview

The chat UX is loaded into an iFrame. When users chat, their payload is posted to the 3rd party’s backend. The 3rd party then fires a webhook to Decorist which tracks the event in the DB and fires a transactional email depending on business logic.

The Problem

The ways the bugs manifested boiled down to:

chat UX not appearing (like below, often as the result of a 3rd party deploy gone bad)
emails not sending because webhooks not called

Lacking integration monitoring, issues often bubbled up through first-tier support.

Improving the Dependency

Noticing the reliability issues, I first delegated to one experienced engineer to triage and then another.

I also dug in on my own and discovered/remedied bugs in our own webhooks while also providing data-backed reliability outage information to 3rd party, escalating to 3rd party’s CTO when necessary

Almost quarterly, as a company, the decision to continue using the 3rd party is re-visited given the reliability issues. Each time, though all stakeholders are aware of the pain collectively experienced, the decision has been made to punt replacing the solution.

Results

Ensured remediation of 3rd party issues in 24-36h, even on lowest support tier.

Tags Django

Capacity Planning Collaboration Culture Forecasting Process Program Management Roadmapping

Introducing Program Management

Post author By Anthony Bull
Post date April 1, 2020

(11/19-3/20)

Challenge

Absence of project or program management made it challenging to forecast delivery and could not get engineer time set-aside for maintenance.

Action

Investigated a few tools (Portfolio for Jira, BigPicture, Aha) for visually depicting dependencies and estimating team capacity.

Ultimately introduced lightweight (spreadsheet-based) processes to support capacity planning and crafted a business case for the CEO on why we should invest in infrastructure.

Results

Empowered the organization by increasing capacity planning visibility.

Emails Execution Process

A Tale of Two Email Providers

Post author By Anthony Bull
Post date March 1, 2020

(Decorist : 8/18-2/20)

Challenge

Transctional emails were being inefficiently managed via two email providers.

Action

Worked with Product and Design to take inventory of transactional emails needing to be migrated, created line items in Epic as project plan, and incorporated sprint-by-sprint as capacity allowed.

Results

Saved business $120K/yr by enabling shutdown of Iterable account.

Tags Campaign Monitor, Iterable

Process Quality Troubleshooting

Zero to Hero QA

Post author By Anthony Bull
Post date February 1, 2020

(Decorist : 10/18-1/20)

Challenge

Inherited a codebase having no unit nor functional testing nor CI/CD.

Action

Recruited a Sr. Automation QA Engineer, faciliating the introduction of automated testing to the company, and had him create Happy-Path coverage of main UXes.

After three months, all Happy Paths of demand- and supply-side user experiences were covered.

After six months, 100% of site (positive and negative scenarios) has been covered.

Results

Introduced automated QA testing, saving the team from shipping show-stopping bugs.

Tags Jenkins, Selenium, TestRails

Analytics Growth Troubleshooting

Jumping In For Analytics

Post author By Anthony Bull
Post date January 6, 2020

(Decorist : 11/19-11/19)

Challenge

A newly-added Custom Dimension VWO wasn’t tracking.

Action

We had just introduced VWO into the stack for A/B testing when one of the experiements wasn’t reporting correctly.

I jumped in with PROD to debug and worked with VWO to resolve and figured out there was a race condition with our Angular code preventing the beaconing of the Custom Dimension.

I fixed the issue and deployed to prod.

Results

Collaborated w/Product Manager to get VWO experimentation working.

Tags Angular, JavaScript, VWO

eCommerce Performance Engineering Site Reliability

Always Be Improving

Post author By Anthony Bull
Post date January 5, 2020

(Decorist : 8/18-12/19)

Challenge

Leading-by-example to own and improve systems as sole ENGR having SRE/DevOps/Frontend/Backend experience.

Action

Mar 2019

Watching our AWS costs rise ~8% monthly…

Costs Rising

I learned about and subscribed to Reserved Instances to realize costs savings for our hosting spend:

Dec 2019

Though not leading to cost savings or revenue generation, part of my responsbilities have been database administration, jumping in when the production DB would spike like below, figuring out if a runaway process needed to be terminated, if a slow query was bringing it to its knees, if a cron job was introducing load, or whatever needed to be done to keep the site up.