Developing a publishing platform for 9 million users… on your own

Creo was a publishing platform I developed and launched which powered the the VideoGamer sites between 2016 and 2018. This was a stupidly big task. This article is basically a rundown of why it was developed, what it was capable of, and a few demonstrations.

When I sold the company that started VideoGamer, the new owners and I decided to launch Candy Banana, which grew to a competitive size within the video games publishing sector — approx 9m unique users and 140m pages a per month worldwide.

As is often the story, the sites and associated CMS were running on the same core homebrew code (albeit heavily modified and improved over the years) as they were at their launch way back in 2006. There was also no physical redundancy to our rack in the event of a multiple hardware failure or datacentre powercut. To say it was creaking would be a gross understatement.

And with the way things were in video games publishing in 2014, it transpired that we didn’t have the resource to hire a talented dev team, so I figured I’d do an extended hackathon and just develop a scalable cloud-based cutting edge new publishing platform over the course of a few months myself. “How hard can it be?” I thought, mapping the core concepts to ultimately entirely unrealistic timescales in my head.

It wasn’t something I would recommend to anyone, especially whilst running a business, but honestly at the time I couldn’t see any other solution — we needed to grow the site and it was soon going to get hacked or just blow up, which would have probably killed the company.

123,109 lines of code, 11,600 hours crammed into ~20 months, several open-source bugfixes, and quite a few additional grey hairs later, and the brand new site launched.

On the plus side, I was able to flex my backend, frontend, architecture, and devops muscles and learn a great deal along the way. The only thing I didn’t end up doing was the site design, was was expertly put together by Dan Edwards of No Divide — highly recommended, by the way.

The most tedious task was basically an ETL migration system — normalising terrabytes of crappy old data/metadata and media files to insert into the shiny new API. It took over 6 months of work for just a few days to migrate everything — and now that code is basically useless. The joy of migrating.

What ended up being launched was the first step in a grand plan, and a significantly cut down version of the ‘big idea’ (TM), but Candy Banana ultimately sold VideoGamer in 2017 and the rest was never realised.

As an illustration of a pretty massive task that’s technically possible to do on your own with the right motivation, bust mostly for vain posterity, I have listed the key features and put together a few demos of VideoGamer running on Creo.

Creo Platform Features

End-user features

• Fully embedded content management — no separate CMS or admin site. I see that as a needless anti-pattern to publishing, and yet besides Medium, it still seems to be the norm (see Wordpress and Chorus). What you see when editing is what you get when publishing — even with more complex articles that have metadata like reviews (overall score, quote, pros and cons). You can also upload large numbers of images on the fly, and upload HD videos via YouTube or JWPlayer.
• Powerful rich content editor — with equivalent functionality to Medium (inline editing, drop-in media), but more — for instance you can upload your own images and videos with metadata that gets put into a searchable central image library. View galleries within articles. Adding related items (games, in the case of VideoGamer). Auto-cropping of images within articles, with point of interest & facial detection to determine the right area to crop. Also has support for embedding many more media sources than Medium (60+) thanks to Embera.
• Multi-language support — the platform powered both VideoGamer.com (English speaking) and VideoGamer.pt (Portugal)
• Responsive design — for the entire site, including delivering up mobile-bandwidth friendly images.
• Single-sign-on — no need for users to login again if visiting the same sites running on the platform, even if the sites have different root domains.
• Publishing friendly — Everything marked up with JSON-LD, AMP versions of all article pages, metadata for social network sharing, ability to easily customise article permalinks.

Technical feature/capability highlights

• PHP/Python/Postgresql/Redis/AWS/Docker stack. API running on a heavily modified Laravel, JS frontend built from scratch using requireJS (AMD pattern), custom devops/deployment utilising Jenkins and python scripts.
• Entirely API driven — the business logic resides entirely in the API, and each site is a client of the API. This allows you to launch new sites on the platform very quickly, and local-language sites within minutes with a custom domain, localised content, images, and URLs.
• Hugely flexible and Hypermedia RESTful API — as well as the usual filtering/ordering/fields manipulation functionality you may expect from a JSON API, you’re also able to make requests that join related data from other endpoints — this dramatically lowered the number requests clients had to make. Note this was before GraphQL was big, but offered most of the same benefits.
• Extensible and performant. You’re able to add an API endpoint in seconds, and the API utilises a sophisticated Redis caching mechanism (which was a huge ball-ache to design, by the way). The search functionality on the API utilised AWS CloudSearch.
• Cross-domain single-sign-on OAuth 2 authentication system. Utilising 256bit SSL / JSON Web Tokens. Able to identify suspicious activity, e.g. unauthorised login from unusual country, and kill/invalidate specific user sessions.
• Enterprise level fine-grained permissions and access control based on API scopes. Designed for granular levels of authority and user-generated content (user content editors, moderators, super moderators etc).
• Scalable and replicable cloud-hosted infrastructure. Utilising Infrastructure as Code concepts and AWS CloudFormation. Hardware and configuration changes, even large ones to the underlying platform, can be made with a simple code change, despite a relatively complex setup. The stack can also be replicated in different AWS regions if necessary.
• Continuous integration with automatic deploys for both code, database, and infrastructure changes.

Demos

Browsing the site

Adding and editing articles

Adding a Video

Adding a review

If you got this far, well done — and thanks for reading/watching. I’m now working on my new startup Claimer — if you’re a UK-based startup or SME, you may want to check it out.

I’m @AssembledAdam on Twitter.