Test Questions have long been the quality assurance mechanism of the CrowdFlower platform.
The interactivity behind them was created around 2008 in MooTools as part of a Merb application; that interactivity was never ported, even as the company increasingly adopted Rails and jQuery. For over three years, discussion has been of porting the Test Question interface out of Merb (the last presentation-layer in that app) and into a company-standard Rails app. Given the expected amount of effort with little user-benefitting ROI to be realized for simply a straight port, it’s easy to understand why it had only ever remained a discussion.
Finally, in April, 2016, the stars aligned when the VP of Product expressed a desire to spend time on improving Test Questions usability while the VP of Engineering decided the time was right for moving forward on a micro-service architecture, dividing “frontend” and “backend” responsibilities accordingly. I seized on the opportunity (without being mandated to do so) because I saw that we could kill two birds with one stone
Following is an inventory (I created the initial format for) of JS libs across three different routes (completed by a junior engineer)
I then used this inventory in conjunction with a Jira report to help scope the effort and parcel out work to likely candidates, an example of which can be seen below
Finally, I tracked progress in a wiki page, providing status updates to key stakeholders.
Results
Laid groudwork for complete FE overhaul, including success criteria and risks, after shopping technical ideas around with Lead engineers, VP PROD, VP ENGR.
In late August 2015, given previous successes in the year, I was tapped to lead the engineering team for delvering a visual identiy refresh (in conjunction with conference-ready AI deliverable) by early October.
Week 1
took Bootstrap 3/Flat UI/custom styling from Designer and created a static page as ‘gold standard’ for other engineers to reference
identified priority routes on which the new design would need to be rolled out
reference page
Week 2
created a new layout for and and began rolling out new design on the Rails app
drafted a plan for updating the Merb app seamlessly
began to onboard other engineers
Weeks 3-5
prototyped and tested the idea for asset precompiling in the Rails app and replacing the base assets of the Merb app
continued polishing
guided other engineers on implementation
continued polishing
Week 6
coordinated bug-free deploy in conjunction with Marketing (who was working for similarly refreshing the third-party-hosted home page)
Results
Organized work of four engineers (two local incl. CTO, two remote) as Tech Lead while planning (and tracking against) engineering sprints and deliverables over two months.
In late August 2015, given previous successes in the year, I was tapped to lead the engineering team for delvering a conference-ready AI deliverable by early October.
In the months leading up to that, the CTO had been prototyping an intial verion of the app in Rails which, for the conference, was to supposed to be integrated with other legacy apps (Rails 3.2 and Merb) and have its UI overhauled to be compliant with the newly-created company Styleguide.
Week 1
Given Balsamiq wireframes, put together a few layouts
Put basic routes in place
Began architecting common styling solution between AI app and legacy apps
basics coming together
Week 2
Continued work on common styling
Made choices aobut JS libs and prototyped interactions given wireframes; got buy-in from CTO, Product, and Design
Began work integrating with ML Python web service
first index page of models
Week 3
Given higher-resolution mockups by Designer, started to polish look-and-feel
With architecture in place, began to parcel work out to other engineers
first version of export
Week 4
As conference neared, knew we weren’t going to be able to deliver everything; worked with Product to focus on MVP
Oversaw work of other engineers
adding data to the model
Week 5
Continued to lead other engineers and refine interactions
annotating a model
Week 6
Applied final polish
Delivered for the conference! Following are a few screenshots demonstrating some of the deliverables
Results
Led team in coordination with CTO to deliver AI application (Rails) for company-sponsored conference on Machine Learning, Artificial Intelligence, and Data Science.
In the Spring of 2015, seven years after the company’s inception and three years after the intital movement towards an SOA paradigm and away from the monolith Merb app (essential to the company’s business,) the architectual shift was still not finished.
Towards further paying down the tech debt of needing to maintain that app, I lobbied for, organized, and oversaw the extraction of the final presentation layer components into a more modern, more maintainable Rails app.
While not quite the magnitude of the previous major refresh, I recognized it would still be a mammoth effort. In order to attack the port, I created a spreadsheet project plan, inventorying all platform routes torwards prioritizing for the eight most-important, portable, customer-facing routes.
Following is a before-and-after example of what was achieved across those eight routes over four months.
Results
Secured adoption by CTO, VP PROD, Lead Engineer, and VPE to port view layer from legacy Merb app to Rails app.
Contributors, as they are called, are the +5M people around the world who do work on CrowdFlower’s platform. The application that enables them to do work is one of the company’s heavily trafficked as well as most complicated – blending a Rails backend with MooTools, jQuery, and RequireJS in the frontend.
The application’s UX
…had largely stayed the same for the last five years. In Q1/2014, we decided to enhance it by making it more interactive and towards engaging our users more and conveying the just how much work there is in our system.
Working with the Product Manager and an external Designer, we came up with the following high-resolution mock
Because the application is so heavily used, we knew we couldn’t merely throw the switch on a new design overnight; both from a community management standpoint as well as application performance. Instead, we chose a strategy of introducing a first at the company: use of A/B Testing to determine a design that would perform as well as if not better than the original.
Our key metric for performance in that regard had to do with contributor’s performance after being exposed to the new UX, particular the messaging around our forthcoming gamification and introduction of Levels. In the beginning, we did not have the infrastructure to determine the value for that metric so we simply settled on ‘clicks’ as a (conversion) proxy to understand if the new design was having an impact.
Infrastructure
Without an A/B Testing framework in place, I needed to choose one. As requirements were not concrete for such, I did some due diligence in vetting several options, coming up with a review of A/B testing frameworks for Rails.
It became obvious that Vanity was best suited to our needs. (Since it doesn’t yet have the ability to throttle a percentage of the traffic receiving experiments, I augmented it with Flipper.)
Once that was in place, we could begin iterating on the design, knowing with confidence how we were impacting the user experience.
Server-side
We knew we wanted the experience to be snappy, but completely replacing the existing experience with a Rich Internet Application was far out of the scope for the first month, particularly as there were infrastructure changes to be made to retrofit the stack with A/B Testing. We decided to make progress iteratively over several sprints.
In our first test, we pitted the control (original) against a bare-bones implementation version of the high-resolution mock as the new design.
original
The new version out-performed control (in terms of clicks) 21.3% vs 20.3% (at 95% confidence) so I continued to iterate on the implementation, coming up with the following
To calculate the overall satisfaction by other contributors for a task (denoted by the stars) proved to be too inefficient in this iteration; it wound up losing.
Client-side
On the assumption that we needed to make the experience snappier in order to drive engagement, it was obvious that we would need to have more (and faster) interaction and therefore, an interactive client-side implementation.
As what was essentially a completely parallel product, leveraging only some of the infrastructure that the server-side rendition was utiziling, I begin to flesh out the following
Further refinement (an actual data) was necessary to get it looking more like the high-res mock (and like its server-side-rendered peer)
At this point, we implemented and integrated with our own homemade badging solution, beginning to display badges in the following iteration
The new version out-performed control (in terms of clicks) 21.3% vs 20.3% (at 95% confidence) so I continued to iterate on the implementation, coming up with the following
Testing the impact of particular messaging was also of interest, so we added a Guiders variation as well. At this time we also leveraged Google Analytics Events on the Guider buttons to track how the far the user got in our messaging.
Letting the experiments run a few days with sufficent traffic, we found that client-side-rendered version peformed no worse than the server-side-rendered version (23.9% vs 22.9%) and that having guiders also performed not significantly worse (23.1% vs 23.7%) so we decided to keep both.
By that time, the new version was out-performing control (the original design) 22.2% vs 20.7% (at 99% confidence) so a decision was made to move forward rolling out the new experience to 100% of contributors, doing some polishing (copy/styling) work before finally settling on the following
Results
Used A/B testing to upgrade company’s most highly-trafficked page (5+M views/month,) increasing user engagement by 5% and saving $2K/month (in Bunchball costs) by rolling own simple badging solution.
Test Questions are used as the gold standard of quality in the CF platform, but they can be laborious to create, particularly for work that’s periodically repeated.
As no templatized solution existed, a team of three of us (me as F2E, Product Manager, and Backend Engineer) tackled creation of an internal product to simplify the workflow.
The user flow was to create “Cases” of Test Questions that got sent to jobs as “Batches”; where the composite idea of a “Mold” encompassed all “Cases” and “Batches” for a particular set of target jobs.
(“The Forge” was the product’s original name, derived from a time when “Test Questions” were known as “Gold.”)
One of the more challenging aspects of the project was the testing of the app. Selenium has always been a robust solution for testing even JS-heavy experiences, but given its heft, Poltergeist was used instead.
The product was to eventually be made available externally but never was.
Shop It To Me has been a beloved service for years when it comes to finding the best deals across retailers, but we wanted to enable the user finer-grained abilities to find exactly the item(s) of interest for them.
We often had feedback about being able to drilldown by clothing type and brand (e.g. “Rebecca Minkoff handbags”) but our previous tools didn’t enable that kind of granularity.
Enter “Threads.”
As the brainchild of our CEO, project “Threads” has become a way for users to track the specific items of interest to them. We now empower the user with the ability to get as general (e.g. “Black dresses”) or as specific as they want. (e.g. “Rebecca Minkoff handbags under $300.”)
Below, you’ll see the hi-res) mockup we came up with (after having previously white-boarded the idea) in early March (click to view.)
inspiration (as mocked-up in Photoshop)
Over the following two weeks, I ramped up on Ember.js and implemented the first version of the app while the other engineer built out the pseudo-RESTful backend. I won’t claim that the code we produced was the cleanest either one of us ever wrote; while doing the best we could, getting to market was the true driving motivator.
We went live – that is, deployed to production for internal and a select group of test users – on March 27th, 2012, with the following (click to view)
very first landing page
Based on user tester feedback, we decided to bolt-on onboarding and improve the look, so four days later on March 31st, 2012, we deployed the following (click to view)
revised landing page with header and pseudo-onboarding
Based on observations during user testing, we realized the opportunity to incorporate a feed of site-wide activity as that would benefit discovery. We also took a few cycles to flesh out onboarding and a “Things To Do” list given challenges users’ had in making sense of the new product. We also tweaked styling to highlight the very newest items in the users’ results; screenshots follow (click to view.)
onboarding modal on landing page (4/11/2012)
1st try: TTD/activity feed on landing page (4/11/2012)
2nd try: TTD/activity feed on landing page (4/12/2012)
emphasizing new items (released 4/12/2012)
We brought in more and more user testers. Here’s a screenshot from April 17th, 2012, showing progress we made from our observations
4/17/2012 release of results page
While observing user testers, we discovered that they wanted tastemakers to help them navigate the overwhelming number of apparel items, so we introduced the concept of curation. We also made the assets/images larger by popular demand.
curation (4/23/2012)
results page (4/23/2012)
results page w/larger assets/images (next day : 4/24/2012)
After April 24th, we stepped back to assess how we were doing. I was rotated onto our bread-and-butter projects but the team was augmented with other talent and continued forward, contracting the look-and-feel out to a third party.
The product has since been retired.
Results
To revolutionize personalized shopping, a team of three (the CEO as product manager, me as the F2E/architect, another as backend engineer,) created a new shopping experience using Ember.js over two sprints
Ramped up quickly on Ember.js
Architected frontend as a Single Page Application
Worked with backend engineer to agree upon/implement integration between FE and BE
Drove development through four major iterations during a two one-month sprints
Suggested (though not responsible for implementing) lazy-load as a way to improve performance
Picked up design slack when we lost our Graphic Designer
Scoped frontend deliverables and managed expectations
Worked with CTO to coordinate out-of-cycle releases to production
