Category: Tips

Our experience with Backerkit advertising

Leave a comment

We’ve been running TTRPG crowdfunding campaigns for almost a decade and we like to think we’re quite good at it. We’ve been gradually and organically growing our audience, but it gets harder and harder to connect with people as the TTRPG world fractures into zillions of little communities. We had dipped our toe in the water of advertising previously, but never had much success with it and viewed it as a waste of money. Enter Backerkit advertising – a service that proved very effective for us. In this article I’ll break down the experience and the outcomes we saw.

The TL;DR here is that we got a lot more money, both before and after taking out the cost of the ads. Wreck This Deck looks likely to have been unusually successful for a TTRPG zine even without the ads, but there’s clear evidence that the ads increased that.

I’m not affiliated with Backerkit, I’m not getting anything from them for doing it, I’m just sharing this because I think it might be helpful for fellow creators.

Backerkit’s advertising pitch is, they buy advertising on your behalf (mostly Facebook/Instagram ads) and improve the targeting using their presumably very impressive storehouse of data from all the millions of crowdfunding campaigns they’ve been involved with. You tell them a target return you want on your ads, and they then increase or decrease spend depending on how well they’re meeting that target. They charge you a commission on any resulting pledges. You don’t pay for anything until the campaign closes and you’ve received your pledge money.

By the way, this is in-campaign advertising. Backerkit (and others, probably) do pre-campaign advertising to build up followers on your launch page. We haven’t tried that, and it isn’t covered here.

We weren’t sure if this service was likely to work for us, but – spoiler alert – it absolutely did. We saw at least a 50% increase in our backers compared to our most optimistic expectations, and there’s very clear evidence to show that this was generated by the ads, as I’ll explain below.

Before going any further, let’s talk about the ick factor. If you’re like me, you probably don’t like the idea of advertising. It’s horrible, intrusive stuff that feels sort of spammy and slightly dirty. You just want to be left alone to enjoy the internet without this stuff, and you don’t want to be a part of it. You maybe feel like your product should be so good that it doesn’t need advertising.  There was a definite emotional barrier we had to push through to get started with this. But the truth is, well-targeted adverts for a quality product are a way of finding people who want something and helping them to find out about it. They’re gonna see some ads anyway, so it might as well be for a cool new game. Provided the things you promise in your pitch are accurate, and your game is good, you’re not hurting anyone by using it.

What we did

We were pretty wary of pouring a ton of money into something for no return. The Backerkit model – tell us a target return on your ad and we’ll spend like crazy as long as you’re meeting it – was kind of terrifying to us. We set up advertising early on in our campaign, saw some fairly middling results, and told them to switch the ads off.

Later on in the campaign, for no reason I can articulate, we decided to give them another go. We switched them on again, at a low level of spend, and saw an immediate increase in pledges. Bumping the spend up a bit, we saw even better results.

Throughout the periods where we were advertising, we set a target return on advertising spend (ROAS) of 3 – meaning the aim is for each £1 spent on adverts to yield £3 or more of pledges. This is the amount we’d worked out, after costs, should ensure we made extra money rather than a loss. Although the ROAS jumped around a lot over the course of the campaign, the final ROAS was 3.04.

The results

The graph below tracks our pledges each day of the campaign for Wreck This Deck.

The blue bit of the chart is pledges that Backerkit identify as not being ad driven. Orange is pledges that Backerkit identify as being ad driven. The tiny almost-invisible grey bit is pledges Backerkit identify as being driven by their newsletter.

You might ask: why should we trust Backerkit’s assessment of whether a pledge was ad-driven? They get a commission on the ad-driven pledges so it’s in their interests to round those up isn’t it? That is indeed an anxiety that we had. But in a way, the fact that we had a gap in the middle where we weren’t using ads was incredibly helpful, in that it clearly demonstrated that the ads were working. You can easily see the point that we turned the ads back on in the graph below, even without the big red arrows, and you could probably guess how much revenue was ad-driven even without the colour-coding.

Graph showing pledges each day for Wreck This Deck. On day 20 we restarted our ad spend and there's an immediate large increase in pledges, through to the end of the campaign on day 29.

The first few days of a crowdfunding campaign always see lots of pledges as existing fans and highly enthusiastic backers jump in. After day 3 or so, things naturally quieten down, and you see a trickle of pledges from folk who have only just heard about the campaign. During this mid-campaign period – days 4-20 on the graph – we saw about £200 of new pledges per day. Once we turned the advertising on this leapt up by a factor of 4, even excluding the last few days when, again, you always see a big increase in pledges.

Interestingly even the organic pledges increased by about 75% during the period we were advertising. Presumably some people were seeing the ads and then pledging on a different device or similar, hiding them from Backerkit’s tracking algorithm.

It’s a lot harder to feel confident about the impact of the ads during the last few days, because you’d expect a big spike anyway. Look at any successful crowdfunding campaign, there’s always a rush of pledges at the end. But it is possible to estimate the effect of advertising here. I looked at our previous campaigns and a few carefully-chosen third-party campaigns that I deemed to be similar to Wreck This Deck. The difference is fairly obvious.

Table showing the percentage share of revenue taken in the last 3 days of various crowdfunding campaigns. The figure ranges from 16-29%, except for Wreck This Deck where 66% of revenue came in the last 3 days.

We also asked our backers in the post-campaign survey whether they’d seen ads. Obviously the data here is subject to the caveat that people might not remember correctly, or might have thought something was an ad when it wasn’t, and so forth. With that said:

  • 35.2% said they didn’t see any ads
  • 13.2% said they saw an ad after they’d already backed
  • 5.7% said they saw an ad after they’d already heard about the campaign
  • 9.5% said they saw an ad but probably would have heard about the campaign anyway
  • 32.9% said they came to the campaign because they’d seen an ad

Backerkit’s marketing stats claim that 57% of our pledges came from advertising. That matches reasonably well to the 61% above who said they’d seen an ad, though just under half of these had already heard of the campaign or think they would have done so anyway.

Did it pay off?

The above analysis seems to pretty clearly indicate that we raised a large amount of revenue from advertising. But of course, that’s before costs.

Based on Backerkit’s own analysis, the fees we paid them for the advertising – covering the cost of the ads themselves and Backerkit’s commission – added up to 39.8% of what the pledges that they identified as being ad-generated. So we got to keep 60.2% of what we raised.

Once you take out our own costs, that number comes down, but because we’d already paid off a lot of our costs (art etc) from organic pledges alone, it still leaves a decent % of money left over for paying ourselves for the work on the project.

The possible fly in the ointment here is what I term “wasted ad spend”. This is essentially my attempt to work out how many ad-driven pledges would have happened anyway, and are therefore wasted money. This is really really hard to know.

The survey data above suggest that only about half of our advertising driven pledges were people who hadn’t already pledged, hadn’t yet heard of the campaign and wouldn’t have likely done so anyway. If all that is counted as “wasted ad spend” then we came in very close to break-even – probably making a small amount of extra money, but just possibly making a small loss once all costs have been counted.

However, if you’d heard about the campaign before but not backed, maybe the ad was what tipped the balance, reminding you about this cool game and getting you to pledge. Only those who already backed can be considered definitely as “wasted ad spend”. If you only count these as waste, that’s only a 21.5% rate of wasted ad spend. That might seem over-optimistic, but if you compare what we made in the late stages of the campaign with what we would have expected, based on comparison with other campaigns, you’d guess that only about 19% of the ad-driven pledges were “wasted ad spend”. At any rate, at a 21.5% rate of wasted spend, the ads would have driven a healthy amount of extra money – meaning we would have kept about 23.5% of the ad-driven revenue after costs.

So we can’t ever really know how effective the ads were taking into account wasted spend. Indeed, there are other unknowns: could it be that the ad-driven folks would eventually have bought the game after the campaign closed? Might we be robbing our future selves? Conversely, might ad-driven backers have reshared the campaign a generated more organic sales from people who would never have heard of it otherwise? It’s all pretty hard to estimate.

What we do know is that this was our most successful campaign, in terms of number of backers, ever. Even though it was a small zine project, it was the most revenue we’ve ever raised from a crowdfunding campaign. And even if we can’t quite prove it, the overall trend in the data suggests that the advertising was well worth it for us.

A small further addendum to the above is that obviously a % of our ad-driven backers will come back and support future projects. We can’t know what this is worth to us, but in the scenario where we actually had very high wasted ad spend, and made a small loss overall, this would be the silver lining to the cloud.

What about you?

Before closing out, I want to pile in some caveats to the above.

First off, this was just one example. Wreck This Deck appears to already have been fairly unusual as zine projects go, with nearly 600 backers before the ads kicked in. It had low overheads, and indeed once you’ve got 600 backers the extra cost of delivering additional copies of the game is very low. This makes it easier for ads to be cost-effective. This might not be a representative example.

Second, we’re a relatively mature gaming company. We’re still absolutely tiny in the scheme of things, but we knew we could afford to take some risks with a relatively small project and if we made a loss then it wouldn’t destroy us. It’s wonderful that Backerkit don’t charge you until after the campaign, but they do charge you, and the bill can be quite high. You have to decide your own appetite for risk.

Third, your costs are an absolutely vital part of the calculation here. Not just the cost of the ads, but the cost of providing your product to all those extra people, including shipping and all the other horrible costs that notoriously turn out to be higher than you expected. We made a spreadsheet to add all these costs up, and work out how high a % return on advertising spend we’d need to turn a profit. We looked at nightmare scenarios where that % turned out to be too low, and how much that would cost us. I strongly recommend you do that too, if you’re thinking about using ads.

Fourth, advertising can be a bit anxiety-inducing. You get real-time data about advertising spend, including how effective it’s been today, and sometimes the numbers can be quite alarming. Returns on spend zigzag around. If you’re in the UK like us, it’s doubly alarming as you can’t communicate with West Cost US-based Backerkit until they get to work in your late afternoon. This goes back to your risk appetite – are you comfortable watching your money being spent, and sometimes feeling unsure if it’s worth it?

Obviously I wouldn’t think to tell anyone “go and spend a pile of money on ads” – that has to be your decision, based on your particular circumstances. All I can say is: it worked well for us, and we will likely be doing more of it.

How I run investigation games (part 2: in-session)

Josh Fox Leave a comment

Last time I wrote about how I prep investigation games. I’ll talk a bit more about that here, but I also want to move on to talk about what I do during sessions.

My aim in running an investigative game is twofold:

  • Make the players feel smart
  • Make the investigation challenging

Those two aims seem kind of contradictory, but in a way they support each other. You cannot feel smart if the game hands you everything on a plate. You cannot feel challenged if the game is simplistic or handled entirely through dice rolls.

As discussed in a previous article, you can break investigation down into various components:

  • Leads (where should I look next?)
  • Imprints (what clues are there?)
  • Patterns (how do the clues fit together?)
  • Conclusions (what is my theory of the case?)

I want as much as possible of the above to feel like they were investigated by the players themselves, using their own brainpower, without running up against the perennial problems of analysis paralysis and the thing that seems obvious to the GM not being at all obvious to the players. I’m not going to pretend those problems aren’t real – and more on how I tackle them below. But for now, let’s talk about how I handle the components above.

My initial lead is always given away for free. That’s a given: the game will be no fun if you can’t get started.

From that point on, I follow a simple set of rules:

  • If a clue is obvious, you don’t have to roll to find it
  • If a clue is hidden but you look in the place it’s hidden, you find it without rolling
  • You can find any clue with an appropriate roll
  • Once you’ve got the clues, it’s mostly down to you to figure out the logical leads, patterns, and conclusions

You can probably see for yourself how the above could easily lead to an investigation stalling. If the players don’t look in the right place, or roll badly, or can’t figure out the next lead, then everything grinds to a halt. There are three principal ways that I solve this:

  • Critical mass. I make sure there are enough clues available that it’s unlikely they’ll fail to find anything.
  • Keep some leads obvious. Signposting specific characters or locations as being of interest will ensure there’s always a next step to follow (but there’s always the potential to discover more)
  • Move the clock on. If the players are taking too long, then I look to the next event on my timeline and make it happen – so even if they get stuck, the story doesn’t

The aim here isn’t to make it impossible to fail the investigation. That wouldn’t be challenging, and it wouldn’t make the players feel smart. The aim, instead, is to make sure that they never get completely stuck – even if they’re failing, they’re moving forwards. So there’s always enough clues to find something out, and there’s always enough obvious leads that you have somewhere to look next.

Equally, my aim is to create a potential dividend from being smart, from being lucky, and from being quick. Players who get lucky on the dice find more clues; players who think their way around those clues and ask good questions discover patterns and start to reach conclusions; and those clues and conclusions can enable them to get ahead of my timeline. Those who move at the minimum pace enabled by following the obvious links, probably find themselves fighting for their lives at the finalé, having left a trail of murders in their wake. Those who leverage luck and judgement may be able to save some lives and catch the perpetrator unawares.

What this means is: being open to the players failing – so that another person is killed (or whatever consequences I established in my timeline happen); but also being open to them wildly succeeding, so that my villains fail and their plans are completely foiled. The critical mass of clues and obvious leads means that I’m hopefully leaning towards success over the medium-term, with occasional frustrating blocks that make that success more satisfying when it comes.

I cannot overemphasise how important it is for failure to come with consequences. If they get stuck, then those consequences mean that the game doesn’t get stuck; instead of their next lead being a witness they want to investigate or a place they want to investigate, the next lead comes in a body bag. And of course, this also means that when they succeed, they’ll know that it was earned, because they know what happens when they fail.

Very occasionally this means the players fail utterly. The villains complete their plan entirely, and escape. That’s great. It means there’s now a future recurring villain, who the players really want to take down, because they feel responsible for not catching them the first time. As long as things didn’t grind to a halt during the session, so there were always fresh leads to follow and tense pacing created by my timeline events, then failure is ok.

One last thing: do not let things drift towards out of character discussion of clues. To a degree all theorising is out of character, since you don’t actually have the skills, knowledge and brainpower of your characters. But try to keep people talking as their characters, because that will help to reinforce the sense that any frustration they may be feeling is fictional, it’s part of the story. They’re not sat on your couch feeling worn down by the investigation, they’re stood in a dark alley looking at a corpse and wondering when the next one will show up.

This article is supported through the Black Armada Patreon

Become a Patron!

How I run investigation games (part 1: prep)

Josh Fox 1 Comment

A really juicy mystery, with the cool feeling of piecing together clues and coming to the correct conclusion, is one of my favourite things in roleplaying. It’s also something that I feel isn’t well delivered by existing RPG systems. Here I’m going to talk about my approach to building a mystery and enabling real investigation.

This isn’t the first time I’ve explored this terrain. Back in 2013 I talked about how mysteries are like stones falling into a pool, creating ripples. And I went on to talk about how investigation isn’t just about clue detection, but about deduction and reaching conclusions. But I stopped short of talking about how to construct a satisfying mystery, which is what I want to do now.

Just for the moment, let’s assume I have an ok system that will cover the business of discovering clues, and an ok premise that make sure the players want to investigate this mystery. I may come back to these later, but let’s imagine they’re solved problems for the purposes of this article. Let’s also assume I’m running something that has a substantial investigative focus, so there’s more than just one simple mystery to solve.

I then create my mystery in a number of fairly discrete steps:

  • Decide what the fundamental driver of the mystery is. Something like “There’s a cult trying to summon a demon through a series of ceremonial sacrifices”, or “House Rukh are planning to assassinate the governor and take over the planetary government”
  • Generate from this driver a series of events. These can be past events which the players are (presumably) going to be investigating, or future events which the players are (presumably) going to be trying to avert.
  • For each past event, I generate a footprint, that is, a set of clues which are out there waiting to be discovered by the players.
  • The footprint breaks down into physical clues and witnesses, which are obviously investigated in different ways. Each of these is amenable to assigning a location and/or time. I’m also thinking about the ways in which the players might discover the clues, though I’m leaving myself open to other ways as well.
  • For future events, I generate a timing and/or trigger, some consequences, and (in case the players don’t find out about it until after it happens) a footprint, exactly as for a past event.

For instance, let’s look at the cult example:

  • For events, I decide that the cult has already sacrificed two victims. One of them was pursued through a particular district in the city in the night, and then murdered in a junkyard. The other was killed previously and more quietly, in their apartment.
  • The pursuit generated some witnesses along the route it followed – people who heard screams for help and some who looked out of their windows to see a group of figures pursuing the victim.
  • Both the murders generate a corpse, some messy bloodstains, perhaps a footprint. They also include the identity of the corpse – for the junkyard murder that may not be obvious, while the apartment victim (if the players discover it) is in their apartment so probably can easily be ID’d.
  • The junkyard murder will be reported, which is the trigger for the players’ investigation. The apartment murder will likely lie fallow for a while, but might show up later.
  • I also create three future events: a near miss where someone is cornered by the cult and nearly killed, but escapes by jumping out of a window; and a murder that involves an initial kidnapping and the victim being brought to a specific site for the final sacrifice. Perhaps the near-miss will report in to the authorities and the players can find out about it that way. Perhaps the kidnapping will be reported, perhaps not.
  • At this stage I might also add in some kind of link between the various murders, be it geographical (the locations form a shape on the map, with the final sacrifice in the middle) or social (the victims are all highly religious people, say), or whatever.
  • If the final sacrifice is completed then the demon will be summoned and a whole new set of events will be generated after that (but I don’t bother thinking about that right now, because I’m expecting that the players will stop the sacrifice happening and/or kill the demon after it’s summoned.

Once I’ve planned all this out, I’ll review what I’ve got to make sure there’s enough there to give the players a fighting chance of cracking the mystery, but not so much that they’ll solve it in five seconds flat. I can add or remove witnesses and clues until I think I have got that right. Of course, my future events ensure that, no matter what happens, the players will have something to do. If time passes and they haven’t made progress, the next event happens.

I’ll then break the information down into a number of components I can use:

  • A timeline of events
  • A list of locations with clues that can be found there
  • A list of characters with motivations, information they might have and any key abilities

Once I’ve got all that in place, the game more-or-less runs itself. The players move from location to location as prompted by clues and/or a future event becoming a present event. Perhaps they discover clues which help them to get ahead of the timeline, perhaps the timeline runs ahead of them and they’re forced to confront a scary situation unprepared.

I’ll talk in a future article about how I use this prep in practice.

This article is supported through the Black Armada Patreon.

Become a Patron!

Lovecraftesque – a kickstarter retrospective (part 3)

Josh Fox 2 Comments

This is part 3 in a multi-part series looking back on our RPG kickstarter, what went well, what we’d change, and so on. Part 1 is here. Part 2 is here. Part 4 is here. This time, we’re talking about how we delivered the product, particularly how we managed the delivery timetable and our costs.

Delivery phase – time

We did ok with keeping to our timetable. We delivered it late, but only by about 1.5 months. In the end this came down to a single delayed workstream which we couldn’t have known in advance would hit problems, and which was too critical to the project to work around. Seen in that context, 1.5 months delay is not too bad – though we would aim to get that down to zero next time.

We did several things to keep the project on track, and they largely worked. First, we calculated that there might be delays, and included that in our project timetable. We’d seen so many Kickstarters delivering many months or even years late that we were trying very hard to be realistic or even pessimistic about our timetable. We actually doubled the time estimate we’d been given by one of our contributors, because we felt it was too optimistic.

Second, we ensured that everything that could be done early was. So we wrote the game in advance of kickstarter launch. We had planned all of the art and layout in advance, and initiated work on them as soon as it was apparent we were going to fund. We had the layout completed for everything we could, even while waiting for some material to come through. In short, we ensured that we could concentrate on delivering the difficult stuff, by getting the easier stuff out of the way fast.

Now before I mention the next factor, this is a good point to recognise that people – and you should include you, the designer, in this category – are a major risk factor for any project. Everyone has a real life that can distract and delay you or even take you out of commission. Health issues can spring up, family tragedies… these are realities in an industry where almost nobody does this as their day job, so we’re all trying to squeeze out work in our spare time. (In fact, our second child was born right in the middle of fulfilment – though we did at least get 9 months notice of that!) I hope it’s clear that I don’t regard this as something to blame or shame people over: it’s something that you as the project manager have to do your best to mitigate. It turned out to be a significant factor for Lovecraftesque.

So, third, we had said up front that we weren’t guaranteeing any of our stretch goals. We would drop a stretch goal or deliver it electronically rather than let its non-delivery delay the project as a whole. I think that was a good thing to do. We didn’t have to drop any, but because we’d said we might, we didn’t need to feel too bad about having to take on authorship of one stretch goal ourselves, supported by a couple of additional consultants that we took on at short notice. We didn’t get any complaints about this, and we think the resulting product was high quality and – crucially – didn’t delay publication.

Fourth, we were very active in managing our contributors. We set deadlines, we reminded people about them, we nagged them if they were late, we negotiated additional time for those who needed it. We worked with contributors to make sure their work meshed with the vision for Lovecraftesque, giving comments and drafting assistance. I hope and believe that this was done in a supportive way, to get the very best finished product.

There is a final thing to recognise in our delivery timetable. While the books were ready and delivered to many backers in October, some of them went missing for about 7 weeks. We don’t know exactly what happened. The records suggest that books destined for the US were shipped from Britain to continental Europe, and then – for some reason – send to Budapest, where they sat for a long time. They were eventually (most of them) sent on to their US recipients. But this caused us a lot of stress and worry, since for a long while we thought they’d simply vanished. And it meant that some of our customers received the product 3 months late rather than 1.5 months. My guess is that this could happen to any project (it certainly seems to be a common problem, watching other projects). In future we would consider paying out on more expensive shipping to allow us to have greater confidence about this.

We’d originally chosen Lightning Source as our printer because they had branches in the UK and US. We planned to ship our US backers from the US branch, which would probably not have been subject to delay in this way. It was a great plan – but one which we had to abandon because of cost. Fluctuating currency values, which we’ll return to later, made the US print branch unviable for us. Post-kickstarter, we’ve offered our US customers the option to pay a (small) premium to get the product printed in the US and therefore delivered faster. Perhaps we could offer that as an option in future kickstarters – though as against that, this strikes me as potentially confusing for customers.

Image of a sleeping person in bed. Behind their back we see luminous creepy-crawlies coming out of their phone and climbing into their ear.
One of the illustrations from Lovecraftesque

Delivery phase – cost

Next, let’s talk about cost. Our carefully costed project was almost exactly on-budget. Our costings – which all included error margins for inflation, currency conversion and suchlike – came in literally on the money, pretty much every time. And yes, that includes shipping: as mentioned earlier, we shipped directly from the printer at very reasonable cost. Runaway shipping costs were my biggest fear throughout the project, and we dodged that bullet thanks to a forensic examination of the costs in advance.

And yet, as I mentioned earlier, we spent our entire 10% contingency fund. Why? Well, the single biggest factor was: Brexit. The value of the pound dropped by a total of 20% between the launch of our project and delivery. 10% of that was pre-referendum, but in retrospect was probably reflecting uncertainty about the referendum result. The rest came after. Most of our stretch goal writers, plus both our artists, were paid in dollars. The resulting exchange costs came to about £600.

The other major factor was to do with the Special Cards. Basically, we underpriced them. Or to look at it another way, we underestimated demand for them. Let me unpack that. From examining previous RPG kickstarters, we thought about half of people who purchased the physical book would want cards as well. In reality, well over 80% did – and in response to backer demand we created a “PDF + cards” tier that we hadn’t planned on. Now, the cards had a very low profit margin on them, which we’d taken into account in our planning – but when we sold a lot more of them than we anticipated, that cost us a bit. Plus they were priced in dollars, so this came together with the Brexit factor in a bad way. After the kickstarter we raised the price of the cards from £5 to £8, because of this.

The cards also added complications to delivery. They were printed and shipped separately to the books (Drivethru Cards is a separate printer from DrivethruRPG). They were an extra bit of admin, an extra delivery risk, one more thing to track and worry about. So, with the above… I’d dearly like to avoid using cards in future. Or rather, I’d like to try and stick to one physical product: cards or a book, probably not both.

So with all this, plus a handful of much smaller things, our contingency fund was spent in full – plus £5 over, to be precise. Sigh. At least we had one. But despite this, we did not make a loss. We included payment for ourselves in our funding goal, and increased it with each stretch goal – something I would always urge you to do, if you’re planning to kickstart something – and so we actually made a very respectable amount of money from the kickstarter itself. And of course, we never had any money-related problems delivering the product.

By the way, I should also mention that we used Backerkit to do our customer survey. This allowed non-Kickstarter backers to purchase books after the campaign ended and enabled existing backers to upgrade their initial pledges. True to what Backerkit estimate, we sold a sizeable additional chunk of books through them. A good choice, which I’d recommend to others.

The next article in this series is here.

Lovecraftesque – a kickstarter retrospective (pt 2)

Josh Fox 3 Comments

This is part 2 in a multi-part series looking back on our RPG kickstarter, what went well, what we’d change, and so on. Part 1 is here. Part 3 is here. Part 4 is here. This time, we’re talking about how we worked to support diversity and inclusion, and tackle Lovecraft’s bigotry; and how we ran the Kickstarter campaign itself.

Diversity and inclusion

One of our goals for Lovecraftesque was to be an exemplar on diversity and inclusion, in as many ways as we could. Aside from this aligning with our politics generally, we specifically wanted to punch Lovecraft’s bigotry in the face – to make the kind of game he would have thoroughly disapproved of.

We did this in a number of different ways:

  • We directly addressed Lovecraft’s racism and the attitudes towards mental illness that are embodied in his writing and (to a much greater extent) Lovecraftian roleplaying games. We had written some basic guidance on this ourselves, but included as stretch goals more detailed work on this, which (once funded) we included in the game book.
  • We included in the art specification a requirement that the art be diverse, showcasing characters who are female, ethnically diverse, LGBTQ and disabled. We specified ratios for these characters to deliberately put white dudes in a minority in the book’s art. We also asked our artist to avoid casting these characters as weak, submissive or sex objects – although the nature of the game meant none of the characters were kicking ass, as all were potentially going to be victims of something nasty.
  • When advertising for artists we specifically asked PoC and LGBTQ folks to put themselves forward. When approaching individual artists and writers, we aimed to draw on a diverse pool, again keeping white dudes in a minority. We weren’t sacrificing quality to do this – there is a great pool of diverse talent out there.
  • We paid all of our contributors the same highly competitive rates – well above what we understand to be the industry average – because we’d like roleplaying design and writing to be something people get properly rewarded for, especially for people who don’t have the privilege of a rockstar reputation (i.e. most of us).
  • Although we didn’t exactly set out to do this, the diverse team we’d recruited was probably directly responsible for the very diverse set of scenarios that came with the game – from those with a fairly traditional Lovecraft-ey setup to stuff HPL would never have dreamed of.

To judge by the enraged response of some less reconstructed gamers, and those who affect to be tired of political correctness, we succeeded in branding the product as diverse, inclusive and tackling bigotry. I believe we did more than that – we embedded those values in the game itself.

One of the illustrations for Lovecraftesque: a priest stands amongst the pews of a church, comforting a visibly stressed man, in whose lap is a notebook filled with drawings of a scary figure. In the background, the same scary figure is just visible pressed up against a stained glass window.
Did I mention I love our art?

The Kickstarter

Thanks to Kickstarter preserving everything in aspic, you can go and look at our campaign page now if you like. We tried to hit all the information someone might want to understand the product, get enthused and feel confident in us. So: a clear pitch that highlights what the game feels like to play, what makes it different, and what makes it fun; a clear explanation of what you’ll get if you back; a simplified version of the game for free download, to try-before-you-buy; samples of art and layout work; a detailed explanation of how we planned to spend the money; and an explanation of the risks including how we planned to mitigate them. Plus a video, because apparently that’s a good thing to have (I personally am not at all motivated by videos, but statistically I’m in a minority apparently). We got feedback from friends and more experienced people on the page prior to the launch and made changes in response. You know, this stuff isn’t really rocket science, but you do need to check these boxes to maximise chance of success, and we did.

We worked hard to publicise the game from a relatively weak starting position. We had established a presence in Indie G+ circles, over a period of 2-3 years, so we weren’t exactly invisible. Even so, as far as I can tell, we weren’t well known, nor did we have (much of) a following. So we wanted to counteract the newcomer disadvantage.

One way that we did that was by approaching people who were better known than us for stretch goals. This conflicted a bit with another goal of ours, which was to have a very diverse stretch goal team. Early on we deliberately went for an approximately 50/50 split between people with star power and other designers – all of whom, to be clear, we had chosen primarily for their talent and the fact we admired them. We also leaned towards people who seemed interested in horror gaming. I think the presence of both of these groups will probably have helped to generate interest from a broader community than we could have reached ourselves. The “star” writers will have reassured people who like reliable industry quantities. In addition to helping fulfill our mission to create a diverse and inclusive Lovecraft game (see above), the broader set of writers probably attracted backers who like to see a diverse team and who were fans of the somewhat more niche indie game design community.

Another thing we did was to approach a wide range of podcasts and blogs. In retrospect, I wonder if we actually undersold ourselves here. Having seen a friend’s recent media engagement plan, it was significantly more wide-ranging than ours. Possibly we didn’t reach as broad a roleplaying audience as we might have. But that’s not to underplay the level of activity – and quite frankly it’s hard to imagine how we could have done much more than we did, given the stage of life we were at (tiny children limited what we could realistically do). We are enormously grateful to people who hosted us on their shows or interviewed us for their blog, and I strongly suspect that this had an impact on our support – though I can’t prove it. Also, talking to people about our game was fun!

On top of that, we managed to attract interest from some big-name websites including the Mary Sue, Boing Boing and (after the campaign) Geek & Sundry. It’s still a bit of a mystery how we did this, other than having the bare-faced cheek to ask. I suspect the concept of a Lovecraft game that tackles diversity and inclusivity head-on helped. With that said, I saw precious little evidence in our referral data that these websites had an impact on our sales. We had google analytics set up on our kickstarter page, showing where people viewing the page had come from, together with kickstarter’s own analytics saying where our actual backers came from. So for example, Boing Boing got us a whopping 600 website referrals over the course of the campaign – but to judge from Kickstarter’s analytics, zero backers. On the other hand, the Mary Sue got us 160 views and 17 backers. These media splashes widened our reach beyond our wildest dreams, and probably did get us some backers (after all, some may have come back to the page later on, rather than being directly referred by an article). But it didn’t generate the deluge you might imagine, perhaps because they mainly extended our reach to people who weren’t that interested in roleplaying games – and so less likely to actually buy a copy.

While we’re on the subject of where backers came from… where did they come from, anyway? About 250 of our backers had got there from Kickstarter itself: the search page, the “discovery” pages, other campaigns, and so on. That’s nearly half our backers. We also got a lot from G+ and Facebook; some of these will have been friends, but not all. Again, not knowing if people were coming back having seen the campaign through a link but then looked it up later on, means we can’t be confident how they originally found out about the project.

We aimed for a steady flow of activity during the campaign, including stretch goal announcements, interviews and so on. We tried to strike a balance between regular updates, keeping things ticking over, and not overloading our backers (and social media feeds) with constant Lovecraftesque stuff. Again, this didn’t really have a visible impact on our stats, which conformed closely to the standard pattern, i.e. a big rush at the start, a lot of people jumping on at the end, and a pretty steady (low) rate of uptake in between. Again, we’re enormously grateful to everyone to shared our campaign and contributed to the buzz by talking about it, and I assume it had some impact. But nothing we did showed up in our stats particularly.

So… did any of this publicity work help? I mean, in a way it must have. Clearly, if you don’t talk about your game, nobody will know about it, so nobody will buy it. But despite collecting lots of data, there is very little evidence on which of this activity had an impact, and which didn’t. So I can’t say we really learned much, other than that the campaign obviously succeeded, so we got something right.

One thing we did not do is advertise. I’ve noticed recently some Kickstarters showing up as adverts in my Facebook feed. And I know at least one designer who has had some pretty good sales figures (not on Kickstarter) from advertising their RPG that way. We did consider doing this, but decided that spending a non-existent budget on advertising probably wasn’t the best plan. But I do wonder, having seen other people’s success, whether this was a mistake. It’s certainly something we’ll consider for the future.

One final interesting thing to ponder is the grabbiness of the game. People say that you can sell roleplayers anything with Cthulhu on it. We were in the weird position of effectively saying “Come and buy our Cthulhu game! Surprise, it doesn’t have Cthulhu in it, and it doesn’t work like a normal Cthulhu RPG, and we spend a big chunk of the text criticising and addressing HPL’s bigotry.” We know that at least a few people bought the game having not looked beyond the Lovecraftian brand (though we were excruciatingly clear on the campaign page what we were selling) and were then disappointed when they got the product. We could equally conjecture that some people were put off by the Lovecraftian brand, not having registered the ways that the game addressed the things that give Lovecraft a bad name. Still other people may have been drawn in by Cthulhu and not backed us when they saw what the game was really about.

In the end, we just don’t know how these things interacted. We had a very successful first Kickstarter, and we’re delighted by that.

In part 3, we’ll look at how we delivered our successfully funded kickstarter. Click here to read it.

Lovecraftesque – a kickstarter retrospective

Josh Fox 3 Comments

Just over 2 years ago, we launched the Lovecraftesque kickstarter. If you didn’t follow it at the time, it was a success – we raised £15000, over triple our funding goal. As we’re slowly moving towards a position where we might kickstart some more games[*], it seems timely to talk a bit about what went well, and what went less well.

It turns out there’s quite a lot to say, so this is part one in a multi-part series. Part two is here. Part 3 is here. Part 4 is here. This part looks at the planning we did ahead of the kickstarter.

Design phase

We started writing the game a good year before the kickstarter. We had a clear concept, and clear design goals.

We were very fortunate to get a lot of interest from potential playtesters, many of whom actually playtested the game. We asked clear, focused playtest questions. As a result, we got great feedback which allowed us to refine and streamline the game. I think this is visible in the quality of the design.

We already had a completed draft ready when we launched the kickstarter. I suspect that knowing the game was written was a selling point, for a debut project. It also meant that we had one less thing to worry about in the delivery phase.

Planning phase – printing and shipping costs

We put a great deal of thought into planning the kickstarter.

We identified potential printers, and took quotes from a few.  We plumped for delivery through Drivethru‘s printer, Lightning Source aka Ingram Spark. (We went direct to the printer rather than through Drivethru because the prices were slightly lower.) We’ve been delighted by the quality of the books they produced, and the price was reasonable – albeit a lot higher than if we had done a print run. They ship directly to customers, which is keeps things simple for us logistically.

In retrospect, however, we may have got this wrong. We now know that we could have got a significantly lower price per unit than we actually got if we had gone with a print run instead of POD. The sample of non-POD printers we looked at didn’t appear competitive, but perhaps our sample was too small, or maybe we should have asked for recommendations. I don’t want anyone to get the wrong idea; for smaller numbers, which is what we had anticipated, Print On Demand makes a lot of sense. It reduced our exposure to the risks entailed by up-front expenditure. It allowed us to produce as many copies as we wanted after the Kickstarter, because we didn’t have to decide on a number of copies in advance. However, knowing what we know now, we probably could have saved £1000s with a print run. This would be particularly true if we had gone for a single format rather than offering both softcover and hardcover.

Specifically: we priced our initial funding goal for just over 100 physical copies. In practice, we funded at triple our goal, including 400 physical copies. But perhaps more importantly, we underestimated the number of copies we might sell after the campaign. As of now, we’ve sold 1100 copies, and are still selling a steady stream. That’s easily enough for a print run. This would have entailed an up-front cost. But the cost reduction per unit (excluding shipping) would have been huge, at around a £5 saving per unit. A print run of 1500 would not have been much more expensive than producing 400 copies by POD. This excludes shipping, so may overestimate the benefits of a print run: but the lesson is that you should think ahead and at least consider whether a print run might work out better for you.

On which subject: the huge plus of using Lightning Source was that their shipping costs were very, very low. We had done our research and got the message: shipping can be expensive, especially international shipping. Lightning Source delivered to anywhere in the world (with a very small number of exceptions) for about £2-3. If we had a comparably successful product in future, this would be the one factor that might decide in favour of POD over a print run.

Planning phase – art and layout costs

For both art and layout, we researched and selected a shortlist of 3-4 individuals to approach for a quote, based on work they’d done before. We actively solicited a diverse pool of artists to shortlist from – something I’ll cover in more detail later on. For layout artists, this was based on looking at games we owned; for illustrators, we looked at the artist’s online portfolio. We selected partly based on price, and partly based on how much we liked the artist’s work. They were all great, or we wouldn’t have looked at them. We are really happy with the work our artists did.

One of the illustrations for Lovecraftesque: three children and their dog swim in a deep blue waterhole. A massive alien eye lurks at the very bottom of the picture.
I love commissioning art SO MUCH.

We set specific goals for book length, and number of illustrations, for our basic funding goal and each of our stretch goals. took into account the extra printing costs that this would mean, and got specific quotes for each goal, including for extra illustrations, scenarios and so on, from our artists, so that we knew in advance exactly how much we would be paying, even for stretch goals that we hadn’t set at the start of the project.

Planning phase – other stuff

We had looked at a dozen other (completed) RPG kickstarters of a range of types, to get an idea of what the likely distribution of rewards chosen would be. We had a good idea of what percentage would be PDF-only, softcover, hardcover and so on: and these predictions were largely right. This helped us to ensure we were setting the correct funding goal, because the amount of money you keep varies by reward level (simply put, PDF has no print or shipping costs, so you keep 100% of the money, while physical copies vary in their costs).

We sought advice and views from lots of people in the run-up to the campaign. The biggest source of advice was Google Plus, where people generously gave their views on a wide range of issues we were worrying about – including many experienced people sharing their knowledge of how to run a kickstarter. We were super-grateful for all the advice we got, which hugely enhanced the research we ourselves had done.

We approached a handful of designers we knew and admired to write scenarios and essays for our stretch goals. We were gratified by the response – almost all said yes. We had set ourselves a goal to offer rates that would be significantly above the market rate for RPG writing, which may have been a factor in people accepting work from a couple of randos they’d never heard of. We set a word limit for our stretch goal materials, and stuck very closely to it, which meant that again, we knew in advance how much we would be paying for this work.

Finally, we had a contingency fund of 10% of our total funds raised, to cover stuff that might go wrong. This was a very good decision that we took early on, and – by fluke – we spent it almost exactly. (I’ll mention here that we’d planned to give an equal share of any remaining contingency money to ourselves and our artists. It didn’t happen, because there was nothing left, but we still think it was a pretty good idea, and will likely take the same approach in future campaigns.)

The next part covers how we ran the campaign itself – and how we supported diversity and inclusion. Click here to read it.

[*] Yes, plural.

Oi, rules, get out the way!

Josh Fox Leave a comment

A long time ago, in a blog post, Vincent Baker wrote about mechanics which are driven by the game fiction, and mechanics which aren’t. He used some fancy diagrams to make the point, but I think it’s not much more complicated than that. His point (or at least a point that he made) was that if your mechanics aren’t, on some level, driven by the fiction, then you end up ignoring the fiction.

Why is this? I think it’s reasonably straightforward. If the game’s mechanics can manage quite well without the fiction, the fiction becomes an inconvenience. You can’t have your hit roll until you’ve described your attack. You can’t have your damage roll until you’ve described some gore. The description makes no difference to anything, and you may well not be that interested in detailed descriptions of combat. You want to skip to the stuff that actually matters, the hit roll and the damage roll. And so, with the best will in the world, it becomes tempting to skip over, you know, the actual roleplaying. And as your descriptions become more perfunctory, they seem ever more unnecessary, the colour drains from your combat (or investigation, or whatever mechanic it might be) in favour of lifeless dice rolling.

(Incidentally, I’m not talking about mechanics that model the fiction. Nice probability curves and mechanics broken down in a way that maps onto the fictional “reality” are not relevant here. I’m not against them. But what I’m talking about is mechanics that engage because of circumstances somebody narrated, and which are sensitive to the detail of that narration.)

Once I’d seen the phenomenon Baker describes, I could not unsee it. Everywhere I looked were designs which violated the “fiction first” principle, where a conscious effort is required to keep describing, at least when the game’s mechanics are engaged. And, conversely, many an hour of dull die-rolling seemed explicable, even inevitable, given the rules of the games I had been playing.

To bring this back to the title, many roleplayers would prefer that the rules just “get out of the way”. And I think Baker’s analysis is highly relevant to understanding why. When your mechanics suck the colour out of your roleplaying in this way, every time you find yourself in a mechanics-free scene, everything will seem that much more vibrant. You have no choice but to describe, because the mechanics aren’t there to pick things up; and the fiction no longer seems a burden, because it isn’t getting in the way of your resolution system. In the absence of those mechanics, that resolution system will probably be GM fiat or collective agreement, probably based on what is plausible in the fiction, making description key.

Don’t get me wrong, I’m not saying the entirety of mechanics-averse play is down to a lack of “fiction first” in the rules. A significant amount of it is down to clunky, cumbersome mechanics, cognitive load and tedious book-keeping, for example. But it is certainly a part of it. When the fictional situation drives the mechanics, when fictional logic is put at the centre of the rules, this problem falls away. And so, whenever I design a mechanic, I always look at it through this prism, watchful for anything that might tear the players away from the fiction.

Crowdfunding calculator

Josh Fox 2 Comments

In the run up to our planned Kickstarter of Lovecraftesque, I’ve been busily crunching numbers to make sure that the whole thing will fly financially. All of our figures have gone into a spreadsheet, where I can easily update the costs as I get better information, and tweak the prices of the rewards to reflect those costs. In turn the spreadsheet works out how much we need to ask for.

Anyway, having gone to the effort of creating it, it seemed like other people might find it useful. If you give it basic information about your crowdfunding campaign (like reward levels, costs and so on), it will work out your reward levels for you.

I’ve included instructions on how to use it (at the top and in comments on the relevant cells). In brief: fill in the yellow bits and then read off the information in the green bits.

Crowdfunding calculator

Crowdfunding calculator

EU VAT for indie roleplaying publishers

Josh Fox Leave a comment

I first noticed the changes to EU VAT when they started being shared all over my social meejas around about December last year. Everyone was all doomy about it. It didn’t seem that relevant to me at the time, partly because I don’t sell digital stuff on the internet, and I suppose mostly because I assumed it would blow over.

It hasn’t blown over, and now I’m contemplating selling digital stuff on the internet. So I’ve been doing some research into this. For the uninitiated, it basically means that anyone, anywhere in the world who sells digital products or services to customers in the EU is liable to pay VAT at the local rate to the member state where the customer lives. Phew! That’s a lot of people. And in due course – scheduled for 2016 – it will apply to physical goods too, though one might hope they’ll make some improvements to this confusing and opaque law in the interim.

I have seen some people, mostly Americans I think, rolling their eyes and saying “how do you plan to enforce that”. Well, I wouldn’t want to put my business in the position of breaking the law in the EU, even if I was in America; but as it happens I am also in the EU, so that makes it… pretty easy to enforce, actually.

Anyway: my research is, essentially, a bit of googling, a fair bit of looking at the website of Her Majesty’s Revenue and Customs (HMRC), and an email conversation with same. HMRC’s answers can probably be considered as definitive as it gets at this point, for UK businesses at least, and probably for everyone.

And before I go any further: I’m not a lawyer. This is not legal advice. I am not responsible if you mess this up.

Why is this so hard?

Well for one, it’s not that easy to determine where your customer is. You’re selling them digital goods, so it isn’t like you can easily tell by looking at them. And the VAT rate is different in different EU Member States, so it’s pretty much easy to mess this up.

The HMRC online guide to the new rules says that except in some weirdly specific circumstances (it’s along the lines of, if the customer is buying through their mobile bought from country A while flying in a plane from country B to country C but is in the airspace of country D when the sale is made…) then you have to “obtain and keep 2 pieces of non-contradictory information to support and evidence the member state where the customer is normally located.”

And, “Examples of the type of supporting evidence that tax authorities will accept include:

  • the billing address of the customer
  • the Internet Protocol (IP) address of the device used by the customer
  • customer’s bank details
  • the country code of SIM card used by the customer
  • the location of the customer’s fixed land line through which the service is supplied
  • other commercially relevant information (for example, product coding information which electronically links the sale to a particular jurisdiction)”

So, that might be easy or difficult, depending on how you handle your payments and so forth. Administratively, it’s definitely not going to be easy for anyone who has an appreciable number of EU sales.

Oh, and you’re required by law to include the correct rate of VAT in your prices, too. In theory that means you need to get the above bits of information in advance of sale and then work out the correct VAT and include it in your price which, let’s forget about that. I can’t see any reason why you can’t just quote a blanket price to all customers and say “if you’re in the EU, we’ll pay the VAT”, but then you have a tricky balancing act to set a price which ensures you can afford to pay the VAT, when you won’t know how many customers you’re going to get from each country; plus you’ll be marking up your prices for non-EU customers to make sure you have the funds to pay VAT for the EU customers.

I don’t like the sound of this! What can I do about it?

There’s a lot of ways you might legally avoid having to worry about all this.

First off, if you’re selling bundled products, and the digital product is only “ancillary” to the main product, then you don’t have to pay, at least as long as the law only applies to digital stuff. But you have to be careful, because the definition of when a component in a bundle is ancillary and when it’s part of the main thing you’re selling, doesn’t seem to exist. My guess is that a hard copy RPG sold with a PDF version counts as primarily a physical product, but I wasn’t able to get a straight answer out of HMRC so I’m not 100% confident.

There’s a sort of loop-hole in the law – I say loop-hole, it seems to have been deliberately designed this way to catch the likes of Amazon but miss smaller operations – which says that if your product isn’t delivered automatically then you don’t have to pay VAT. Emailling an attachment doesn’t count as delivering automatically. Emailling a link to a download site does. I can’t really see the logic, but that seems to be the rule. Still, you probably won’t want to manually email attachments to large numbers of people if you can help it. Also, you may want to check up on your email providers policies, as some email providers (e.g. gmail, or so I’m told) blacklist people as spammers who send too many such emails.

What about selling through a third party? Or crowdfunding?

A third way to get around the tax is if there is an intermediary in your sales process. This is crucial for a lot of roleplaying games, because they are frequently sold through online platforms such as DriveThruRPG, Indie Press Revolution, or through crowdfunding platforms like Kickstarter. I wasn’t clear on whether digital products sold through such a platform would be subject to the EU VAT rules, so I asked HMRC. I am going to quote their reply in full:

“If the platform operator identifies you as the seller but sets the general terms and conditions, or authorises payment, or handles delivery/download of the digital service [my emphasis], the platform is considered to be supplying the consumer. They are therefore responsible for accounting for the VAT payment that is charged to the consumer.”

I think that’s pretty clear. In the case of sites like DrivethruRPG and Indie Press Revolution, they will handle the payment and delivery of the digital service, so they are liable to pay the correct rate of VAT. In the case of sites like Kickstarter, they set the general terms and conditions and authorise payment, so they are liable to pay the correct rate of VAT.

[edited to add:] The HMRC website also has this to say ”

If you operate a digital platform through which third parties sell e-services you are liable to account for the VAT on those sales unless every one of the following conditions are met:

  • the digital platform and everyone else involved in the supply must identify who the supplier is in their contractual arrangements
  • the invoice, bill or sales receipt must identify that supplier and the service supplied
  • the digital platform must not authorise the charge to the consumer
  • the digital platform must not authorise the delivery
  • the digital platform must not set the general terms and conditions of the sale”

Again, this seems pretty clear to me. [/edit]

Kickstarter have specifically said that they will not do this, and that the campaign promoters are responsible for dealing with EU VAT. However, I think HMRC’s advice makes it pretty clear that they are wrong about that. This may not be particularly good news: after all Kickstarter is a big enough target that at some point someone may go after them for the VAT they have said they won’t pay. VAT is generally paid at around 20-25%, so the bill could be pretty big. If I were Kickstarter I think I’d be thinking pretty hard about this. But HMRC’s advice strongly suggests to me that you, as  the campaign promoter, do not need to think about this.

Finally, it’s worth noting that there are payment platforms out there like Payhip, who as well as being legally bound to deal with EU VAT, have said that they will do so. Where possible, you should consider using such providers, simply because those who are still in denial may be on a rocky road, and if you use them you may take some bumps along the way.

Playtesting: some reflections

Josh Fox Leave a comment

Lovecraftesque playtests

I’ve collated the information from the first Lovecraftesque external playtest and I thought it might be useful to discuss it here. I’m not going to talk about our game, instead I’ll be talking about the playtest in more general terms, in the hopes of deriving some more general lessons about playtesting.

Recruitment

We advertised the playtest through our website, Black Armada, and through G Plus, Twitter and Facebook. We put the files in a public drop box but only provided the link on request to people who expressed an interest in playtesting.

We received 31 expressions of interest. 29 of these were from people who appeared to be men, 2 from women. 6 were from people who we know quite well in real life, and another 3 from people we’ve met a few times in the flesh. The rest were from comparative strangers.

We allowed six weeks for playtesting from the day we announced it. We sent a reminder out at the midway point to anyone who we hadn’t interacted with for at least a week, and another one a few days before the deadline.

Of 31 expressions of interest, 19% sent in a report.

We received 6 playtest reports within the playtest period – just under a 20% response rate. All of these were submitted by men. 2 came from friends, 4 from comparative strangers. Between these we got 22 session-hours of playtesting, or 72 person-hours.

It seems to me that we were fairly fortunate to get as many as we did. In previous playlists using a similar method I only had a 10% response rate, from a smaller number of expressions of interest. The improved success comes, I think, from a combination of us being better connected within the indie roleplaying community than I was back then, and having a game pitch that was always likely to be a bit more popular.

Method

None of the playtesters received any guidance from us or clarification. They were given a set of detailed questions covering 10 aspects of the game, which were rather bossily labelled “READ THIS FIRST”, in addition to the rulebook and some supporting materials.

None of the playtests involved us, either as a participant or a witness.

Results

All six playtest reports responded to the questions we asked fairly assiduously. I wouldn’t say they were all completely comprehensive, but none of them ignored the structured questions, and all responded to most of the points we wanted covered. One came with a blow-by-blow actual play report (which was quite valuable beyond what our questions elicited).

I shall now provide a breakdown of the issues identified by the playtest. (Either identified by the playtesters themselves or apparent from their report whether they themselves realised it or not.) I have classified them as follows:

  • A critical issue is one which would make the game unplayable.
  • A serious issue is one which would make the game not fun or prevent the design goals of the game from being realised. If even one group identified a serious issue, I’d count it.
  • A major issue is one which makes the game very clunky or interferes with realising the design goals of the game.
  • A minor issue is one which doesn’t interfere with the design goals or make the game avery clunky, but rather is a matter of polish. Minor rules clarifications also fall into this category.

I’ve obviously had to exercise judgement as to whether an issue identified by a group is attributable to the design, and whether there’s anything that can be done in the design to ameliorate the issue. In one or two cases, because different groups reported radically different observations, I haven’t recorded an issue, but will instead watch for these recurring in the next round of playtesting.

Here’s what our groups found:

  • Critical issues – 0 (phew!)
  • Serious issues – 1
  • Major issues – 2
  • Minor issues – 16

50% of our groups caught all three major or serious issues, but 33% only caught one and 17% didn't catch any.

A note here about consistency: not all our issues were detected by all of our groups. Two groups (one of which played twice) did not pick up the serious issue identified above, and the two major issues were each picked up by only three of the six groups (arguably one of them was detectable in a fourth group, but I think we might have dismissed it based on their evidence alone, as it didn’t look that serious). More importantly, these were clustered: 3 groups caught all the serious and major issues, 3 groups missed at least two of these issues.

I want to be clear, by the way, that I don’t consider the above to be a poor reflection on any of our groups. I suspect the ones that missed issues did so because they were more familiar with the style of game or the genre. Some of our clearest and most helpful feedback came from groups that didn’t catch a lot of the bigger issues, but did notice many smaller ones. All the feedback was immensely useful.

The above suggests to me that you want at least three groups to test a game to be reasonably confident of picking up on major and serious issues. With fewer, you might get them, or you might be unlucky. (Of course in our case, we would need four groups to guarantee catching them all.)

By the way, I haven’t analysed the minor issues, but my impression is that they were sprinkled liberally through all six groups. I doubt if there’s a single group that didn’t pick up some minor issues missed by the rest.

Conclusions

The top line conclusion is that you need to playtest, and not just with one or two groups. The comparison with the playtesting on my previous game is instructive. I only had one response, which added a little to my own efforts at playtesting. But clearly, my analysis above means that there is a high risk of failing to catch even quite serious issues with such a low level of response. There would be innumerable smaller issues that will have slipped the net.

Getting playtesters isn’t at all easy. I think we were fortunate this time around. Our voices carry a bit further as a result of a few years circulating in the online indie gaming community. We got support from a couple of people with a very wide reach, and although it’s hard to say how much impact this had, I would guess a lot. And our game concept was more grabby – though whether we would have been taken as seriously if we’d proposed such a concept three years ago, I can’t say.

One thing I would observe is that it’s a lot easier to make playtests happen if you offer to organise them yourself. That’s pretty obvious, but it is worth saying anyway. You can tackle the tendency for the game to get cancelled by providing a venue, making sure you pick people you can rely on and above all not dropping out yourself. And you can make sure decent notes are taken and guarantee to take them away with you. It’s more effort, and if you want it to have the same value as an external test you’ll have to be disciplined about not facilitating the game itself, but it dramatically increases your sample size, which reduces the chances of missing a given issue.