Category: Backend

Categories
Backend eCommerce Frontend Performance Engineering Process

What To Do With a 9s Page Load?

(Decorist : 9/18-12/19)

Challenge

Inherited a two-fold challenge: 1) less-than-optimal user experience of 9.01s Avg. Page Load Speed and 2) a culture that did not yet value performance engineering.

Action

Idenfitied initial frontend and backend low-hanging fruit (e.g. page structure, image resolutions, N+1 queries, lazy-loading, etc.) Identifed and delegated KRs to FE leads. Introduced process to methodically follow up each week, pressing the case for performance engineering over months.

Below, a sampling …

Some initial efforts shaved ~3s of the Homepage:

Lighthouse score soared from 5 to 61:

There were multiple speed improvements of up to 50% on various pages across the site as a result of backend query improvements and a +300% bump YoY in SEO traffic (verified by 3rd party SEO consultant)

After Dec 2019, we shifted focus to non-converted UXes after having addressed low-hanging fruit for all UXes.
Sept 2018 to Dec 2019

Results

  • Record Avg. Site Page Load Speed of prevous four years : 9.03s (Sep 2018) to 4.13s (Dec 2019.)
Categories
Architecture Backend Chat Cloud demand-side eCommerce Frontend Innovation Prototyping Strategy supply-side

Launched MVP

I was brought on initially part-time into a two-sided marketplace in the holistic services space to assist the team to go from an environment where code wasn’t being shipped to a place that could launch an MVP.

Prior to joining, I commissioned my own Market Landscape (via UpWork) around holistic services potential; one finding: 2015 global revenue = ~$40B

Better Engineering

When I arrived, the POC had been built by an offshore team in WordPress via cPanel and PHPAdmin and transferred from AWS to Digital Ocean. There was no source control. The first thing I did was `git init`.

I spent time attempting to decompose the production instance into a Docker container (of WordPress and MySQL.) I learned how tricky it is when an application is not already 12-factored to spin-up an alternative instance having functional parity. In case you didn’t know, much of the application state of a WordPress site is maintained as JSON config blobs IN THE DATABASE.

Seeing opportunity to prioritize feature creation (in support of product-market fit) in lieu of operational excellence, I put the Docker initiative on hold and spun up dev and staging instances using cPanel and PHPMyAdmin, simply copying the prod DB.

dev instance after trying to decouple

CaaS : Chatroom as a Service

First Iteration

My initial task was to integrate a chat UX with a broadcast streaming experience being built out and powered by Wowza. I settled on Converse.js and ejabberd given decent documentation, an acceptable UX, and (what seemed to be) easy integration; delivering the following POC

chat, working

Next Iteration

With the web server running under CentOS on one Droplet, I had thought it smart to run ejabberd on a separate instance, installing v14 on an Ubuntu Droplet. I ran into issues opening port 5281 from the CentOS VPS to Ubuntu so I doubled-back on my strategy, opting to co-host the daemons on one Droplet (CentOS.) I then ran into package manager issues and wound up having to build and install ejabberd 18 from scratch while upgrading OTP to v20.

While this was happening, the junior engineer was working to get the Wowza live-broadcast video solution working and time wasn’t on our side to get the MVP launched.

(Additionally, at one point, I looked at using the YouTube/Hangout API didn’t find support for our use case.)

Attempt to leverage YouTube/Hangouts

Once I got ejabberd 18 built, installed, and running, we started experiencing timeout issues; manifesting in FE with a spinner that spun for 1.5 mins before the user was finally able to get into the chatroom. I dove into the ejabberd code, and realized that there was something blocking going on server-side. I never did figure out what was happening, but about that same time I realized that the junior engineer had integrated Twilio Video for the 1:1 experience and Wowza Streaming for the 1:many broadcast experience.

Seeing opportunity to reduce waste and consolidate coding paradigms, I researched Twilio Video and realized that we could, for the MVP, likely replace the Wowza experience for both of the 1:1 and 1:many experiences WHILE ALSO leveraging Twilio Chat to replace Converse.js/ejabberd.

Twilio Video, 1:many POC

Next Iteration

I found and customized React Programmable Chat

Twilio Chat, POC

The next step was to integrate with Wowza.

Wowza UX, now w/Twilio Chat

As their were some production issues with the Wowza implementation, I forged ahead creating an alternative Twilio Video React component using React Scripts.

1:1 UX, Twilio Video Chat (desktop and mobile)

At soft (internal) launch, I posited the Twilio solution and the other team members were delighted.

Though the Twilio Video quickstart project was useful, I ran into challenges around the video codecs (VP8 and H.264) between Chrome and Safari, namely getting desktop and mobile video interaction to work as expected. Digging around produced Github links helping lead to resolution (e.g. simply switching to H.264 to ensure cross-browser support.)

On 4/6/18, we had our first successful supply-side (1:many) broadcast, powered by Twilio Video and Chat.

MVP

After helping the team achieve MVP launch and when funding could not be raised in a timely fashion, I moved on.

Results

  • Swooped in when feature-delivery was woefully behind, built out (desktop/mobile) UX, and launched MVP to paying customers.
Categories
APIs Architecture Backend Collaboration Frontend Innovation SOA SPAs

Next Gen FE

(Bluxome Labs : 7/16-7/16)

Leading the Charge

Upon returning from vacation, I disovered that the VP ENGR gave the recently-joined Team Lead carte blanche around moving ahead with the next phase of re-engnineering the codebase for a further SOA-influenced frontend and backend split.

Being intimately familiar with the current system architecture, in a race against time for Week One (only had a week before VP ENGR wanted Best Practices/new workflows/adoption of new technolgies and because one substanitally differing proposal – likely much more costly from development and operational standpoints – was on the table,) I set out to formulate a concrete strategy towards delivering in the short-term while providing a basis to iterate against a longer-term architectural shift.

Architecting the Solution

First, I advocated for the adoption of the ‘Launch Page’ as the testbed with which to move forward. Then, I reviewed that page in order to reverse-engineer whatever model attributes I might be able to use (in a Backbone model) as exposed by in a simplistic GET request on page load for initial state.

I then modeled that simplistic endpoint using the Swagger Editor

Next, I dumped the YAML of that endpoint and leveraged Swagger code gen tools to create an initial version of the endpoint in Node.js (and Ruby and Java) and fired up that Node.js for serving HTTP requests.

At that point, having proved to myself the utility of Swagger for spinning-up some simple Node.js services, I turned to building out a placeholder SPA using React that would consume a Node.js service. I opted to stay within the Rails paradigm (instead of adopting an approach of statically hosting assets elswhere) and first used react_on_rails to incorporate React and then the react-rails-hot-loader for HMR.

I wound up needing to patch the latter locally in order to get Web Sockets working correctly in my environment.

Having put the Big Rocks in place, I could hand off the SPA POC to the F2E consultant for further refinement and could focus on building more of a case for the overall approach which the VP ENGR then gave his blessing to.

Building the API

During this phase (Weeks Two and Three,) I implemented GET, POST, and PUT JSON-serving routes in Rails (not to have to worry about CORS) while also developing in the context of Amazon’s Lambda and API Gateway, taking the Node.js server stub above and dropping it in as an HTTP Proxy to collate information from existing API endpoints (as microservices.)

I also worked with five backend specialists and my work guided the conversation around establishing Best Practices for API development.

Results

  • Championed bottoms-up approach to decouple microservices, influenced VPE/Team Lead/F2E to adopt, and prototyped using React / Hot Module Reloading
Categories
Architecture Backend demand-side eCommerce Frontend Full-Stack Innovation SPAs

Demand-side UX Refresh

(Bluxome Labs : 4/16-4/16)

In a unique position given my previous experience with the UX, I took an opportunity when tasked by Product and Design to not only reskin the Listing but also to upgrade it.

Given that no one besides me really knew how RequireJS was working in the application and given its falling-out-of-favor in the general community as a module-loading solution, it was time to upgrade it to Webpack.

Here’s what the Task Listing looked like before

Here’s the mock from Design

In chronological order, here’s what I did

  1. Upgraded DataTables from 1.9 to 1.10
  2. Refactored JavaScript towards more of an OO paradigm
  3. Applied new skin
  4. Ported JavaScript for RequireJS to CoffeeScript for Webpack
  5. Deployed

…and here’s what I delievered

with modal

Results

  • Successfully advocated for adoption of Webpack to replace RequireJS and then ported most-trafficked page, while achieveing near-pixel-perfect re-skinning.

 

Categories
Backend Innovation Prototyping supply-side

Community Building

(Bluxome Labs : 5/14-6/14)

To facilitate migration between Yahoo Groups and Kinsights group functionality for a key account, I created a Command Line Interface tool for a paginated approach to parsing and importing the member list in addition to parsing and importing the archived messages (for the last seven years.)

Many iterations were necessary and several heuristic methods were applied in order to ensure the cleanest import of the messages

Additionally, I created the activation flow for the transition, partially by piecing together signup components from the existing codebase, partially by crafting new logic in the business layer.

Results

  • Created CLI tool to ingest 1,000s of Yahoo Groups users given numerous different, undocumented email schemas.
Categories
APIs Backend Innovation

Data for Everyone

(CrowdFlower : 2/14-3/14)

Results

  • Built API endpoint in Grape for external consumption to power an SPA built by company’s partner marketing firm.
Categories
Affiliate Backend eCommerce Frontend Full-Stack Growth supply-side

Selling More

(Shop It To Me : 6/10-7/12)

Results

  • Iterating with Product to scope, implement (Rails w/MySQL,) and A/B-test pixel-perfect sign-up/refer-a-friend/search/browse/opt-out/profile experiences
  • Drove conversions in the form of signups, virality, and clicks for not only our flagship web and email products (used by 4M+ users) but also eight new product launches
  • Architected company’s newest Ember.js-based product
  • Quickly integrated into a small, fast-moving, startup engineering team
  • Became proficient in all things Rails
  • Ensuring quality through the use of code reviews, TDD, unit, functional, integration, and regression tests under continuous integration, testing plans, and mentoring/pairing to deliver functionality, fix bugs, refactor legacy code, and transfer knowledge
  • Have assumed lead (primarily frontend) responsibilities while reporting directly to CTO
  • Established F2E guidelines and best practices
  • Architected the company’s newest product, an Ember.js-based Single Page Application
  • Leveraged Facebook, Twitter, and Pinterest APIs to increase our social reach (including the use of Facebook Connect and the Like Button during signup and tell-a-friend experiences)
  • Prototyped iPhone app for user to navigate item stream during in-house Hackathon
  • Contributed improvements to our Nokogiri-based data-harvesting framework.
Categories
Backend eCommerce Emails Frontend Innovation Prototyping supply-side

Deal of the Day

(Shop It To Me : 6/11-8/11)

Conceived as a framework for local deals, we pivoted to a daily deal paradigm leveraging our apparel inventory when the market opportunity presented itself. This lead to revenue generation via several 100K impressions/day.

Initially, Product curated deals by hand and as market potential was proven for the idea, we built an infrastructure for algorithmically selecting 100s of deals/day (based on signals such as open and click activity) and gave Product a way to QA the deals.

Here is an example of the ‘Deal of the Day’ in the ‘spotlight’ position of our flagship email product (click to view)

Results

  • built the original (local, pre-Groupon) deal POC infrastructure, front-to-back
  • assumed maintenance responsbilities
  • collaborated with Product to address bugs and implement additional features, most notably: ‘stackable’ deals
  • implemented admin tool for Product to QA deals
  • coined the term ‘spotlight’ now used internally to refer to the real estate occupied by the deal
Categories
Backend demand-side Frontend Full-Stack SOA Troubleshooting

(Executive) Enterprise Dashboarding

(Yahoo! : 5/07-7/07)

An external consultant to the team had been brought in to develop an überdashboard to aggregate data from another project. The learning curve was steep and the consultant wasn’t familiar with the base data so his dashboard had some major shortcomings.

Results

  • Jumped in two weeks before release date, took ownership, and still delivered a web-based, executives’ dashboard solution for experience monitoring without the benefits of system documentation or tests
  • Reduced 74% page load-time through refactoring backend, improving database queries, introducing pagination for an 80 KLOC (LAMPerl) data warehousing web app
  • Discovered and remedied major data quality issues before going live
  • Radically improved the look and usability of the tool
Categories
Backend demand-side Innovation Troubleshooting

Web-Based Document Management

(IAEA : 4/06-4/06)

We were charged with building an application to allow the customer to perform three essential tasks

  • upload documents (as PDFs) to the web
  • manage those same documents via a web-based interface
  • Apply business rules to the documents based on embedded properties (e.g. PDF XMP metadata)

The first version of the system has already allowed the customer to publish and manage 10 years worth of documents. The next 40 years of documents were to be uploaded as the customer’s schedule allowed.

Results

  • Delivered a Java-based document management system for managing 50 years of PDFs on-time after assuming project leadership when the Project Manager fell ill
  • Delivered a user-friendly custom solution for publishing PDFs directly to the web via the enterprise document management system
  • Managed the project to completion when the original Project Manager had to excuse himself during a crucial phase of the project