(The RealReal : 3/17-9/17)
In a bid to
- improve deliverability of our email campaigns (by sending from same IP) and
- free Engineering resources (by shifting email template management to Marketing)
… a(n under-resourced) team1 I was head of was tasked with further migrating from SendGrid templates to ExactTarget.
Given competing priorities, resource availability, and trying to support the engineer’s professional development around further full-stack experience, I tasked the Sr. Frontend Engineer (newest member of the team and who had never done email marketing before) with the port.
For some further context, by the time I had arrived in Q3/16, approximately 10% of system-generated transactional emails had already been ported (such as the demand-side signup confirmation email,) so there was already some precedent and knowledge in the team on how to do such ports.
Scope Change
Working with Product and Marketing, we identified the accounting email sent monthly to supply-side users (30K recipients, arguably the most important email in the system) as the next candidate for porting. According to what we could deduce from the documentation and making certain assumptions about implementation based on similar (previous) endeavors, it seemed like the port would be a slam-dunk in two weeks.
As the engineer got into implementation, I realized that the scope was much larger than originally anticipated. Instead of a one-off API call like had previously been the case (e.g. for the demand-side signup confirmation email,) we were looking at a fundamentally different business process – not of a one-off send but a batch send of 30K emails as part of the monthly payout managed by the Director of Accounting (as kicked-off by a single admin button-push.)
Furthermore: if a demand-side signup confirmation email did not land in a user’s inbox, certainly that was problematic, but if a supply-side user’s monthly accounting email (or, worst case, multiple 1000s of them) did not arrive, that would cause an exponential increase in Customer Service requests and – much worse – violate the company’s contractual SLA with supply-side customers around informing them what their monthly payout would be.
We (all of us – me, the engineer, Marketing, Design, the Product Manager) had wandered into uncharted territory by tackling a template that was blasted out at such volume. We had to revisit our assumptions about how to build out the feature.
Feature Diligence
To emulate a send at a user level was fairly trivial; simply invoke a method call from a console. To truly emulate the batch blast on staging proved much more complicated, as two conditions were necessary: a block of 30K test users and integration with (and kick-off of) the monthly payout (which took four hours to finish.)
Though I assumed the engineer would be able to navigate an unfamiliar domain because he had some full-stack abilities and had been able to reason systematically about architecture in other circumstances, he had difficulties iterating at a unit and functional level to get a local (development) email send working. I coached him on the first tactic – providing guidance to test code at various levels (unit, integration) to mitigate risk – focusing on a method call as it would SMTP email from a local dev environment and then the invoke the ExactTarget API. Then, I coached him to deploy a feature branch integrating with the payout onto staging (where users had already been anonymized) and setup a test blast of 200.
With that blast, we ran into a race-condition around data extensions – which no one in either Engineering or Marketing had complete mastery of – where emails were being delivered empty (simply explained: because no sequential dependency existed between data population and emailing.) Over the following weeks, we tried to determine the best way forward while interacting with our ExactTarget sales representative and then their engineers. We experienced suboptimal support/delays in her getting back with answers to our questions.
Result
After two months of development effort, back-and-forths with Product and Marketing, and several back-and-forths with ExactTarget sales reps and engineers, in a meeting shortly before production go-live of the port (involving the VP PROD, CTO, and VP Marketing,) given the potential risk to the business of such important emails not being delivered, an executive decision was made to abandon the port.
Takeaways
If I had the opportunity to manage it again, I would have:
- found a way to free up time for the more experienced Full-stack engineer – who had had some experience with ExactTarget but was overcommitted at that point on other priorities – to initially pair with the Sr. Frontend Engineer. That would have exposed the initial underestimation in scope earlier, leading to a better cost-benefit analysis (for all stakeholders) given the information we had.
- advocated for refactoring, given what I learned about the state of the code, architecture, and process, and how tightly-coupled the mailing code was with the payout code (making even a semblance of proxy integration testing arduous.) Refactoring would have allowed us the opportunity to identify business critical functions and ensure we had reliable automated testing around them.
In reflecting on what we could take away around process improvements, I have learned to leverage these ideas:
- kick off any new project with a technical design review
- understand the typical load of a feature before making a change
- identify system changes affecting user experience as one-off or batch (which impacts scope accordingly)
- for anything involving a 3rd party, scope work at 3X as a rule-of-thumb (this heuristic is informed not only by this project but by others that involved 3rd party APIs)
I hope that one further follow-on highlighted by the experience for Product and Engineering executives was that, given not only the difficulties in this project but others involving Salesforce/ExactTarget, smaller vendors with less confusing/better/easier technical documentation could be more cost-effective in the long run.
[1] : a Jr. Full-stack Engineer (2 yrs. exp.,) a Full-stack Engineer (7 yrs. exp.,) and a Sr. Frontend Engineer (7 yrs. exp.)
Results
- Led engineering effort to improve email deliverability (which we ultimately had to abandon.)