Beta testing and beyond

Ravelry is still officially in “beta”. This word has been drained of most of its meaning in today’s web but here is what it means to me (and hopefully our users): the site is being very actively developed and things will change, unfinished “trial” features will be introduced, there will be bugs. In the last 365 days, there have been 241 releases where a new version of the code is pushed out to the production site.

As an aside – I’m looking into doing away with the beta label and having beta testing being an opt-in thing for users who want to be guinea pigs. We’re approaching 150,000 registered users and not everyone realizes that they often part of an experiment – this ends up creating more work for us in the form of questions (and sometimes, complaints)

Our users have been incredible – they report bugs of all kinds, come up with tons of great ideas and suggestions, enter things into our issue tracking system, and comment on existing issue. Here is how we do it:

Collecting feedback

Some sites collect feedback from beta testers with an email form. Don’t do this. All of your users need to be able to see and read the bug reports and ideas that everyone else is sending in. Working as a group is more efficient (less duplication), more productive (because small what-ifs can turn into great ideas through conversation) and more fun.

We set up one board in our forums that is specifically for talking about Ravelry itself. The board isn’t limited to feature requests and bug reports – it is a place for any kind of Ravelry talk like “Who has the biggest stash on Ravelry?”. In the last 30 days, 2300 people have posted on this board and 18000 people have read this board. hm… I guess I have to come up with some ideas to drum up participation – people who aren’t reading are missing out on their chance to shape the site.

Keeping Track

Our Ravelry discussion board contains 97,000 posts. We can’t use the board itself to help us track, categorize, and update all of the bug reports and suggestions. Even if we tried to do it by keeping the threads themselves excessively organized and moderated, we’d be sure to lose things.

So… we have an issue tracking system (The List) built in to Ravelry. Jess, Mary-Heather, myself, and 30ish other Ravelry users called “trackers” have the ability to turn posts in the forums into new issues in our system. When people post bug reports or suggestions we all try to either add them to The List or connect them up to existing entries. You can see that the post below has 1 agree vote and 1 disagree – sometimes the agree/disagree votes that are part of our forums can be helpful when we are looking at suggestions.

Now what?

When I am not actively working on them, the list items serve several purposes:

  • Users can search The List to see if their suggestion or bug report has already been brought up
  • Trackers can connect future forum posts to list items in case people have more or different information to share. This happens a lot and it is handy to have slightly different bug reports attached to bugs and further thoughts attached to suggestions.
  • All users can comment on items and I read these comments when I set out to work on something.

Although I do have some basic prioritization type features to The List itself, I do all of my work-gathering and organizing outside of Ravelry. I tend to go into the list, gather a set of items that fit with what I am working on, and take those back to my virtual work area so that I can sort through them, read comments, and organize.

I’m really happy with this. I enjoy talking with Ravelers about the site and getting ideas and help from them. I’m also very glad that I don’t have to worry as much about losing good ideas and bug reports in a sea of posts. Ravelry Users: You can find the For the Love of Ravelry (FTLoR) board and a link to The List on the forums tab. As always – if you have ideas on how we can improve tracking and The List, we’d love to hear them. Just make sure to post them in FTLoR so that we can track them ;)

Comments (21)

  1. Cathy-Cate wrote:

    This was extremely helpful — and inspiring.

    On Ravelry, including FTLoR, I tend to read and often agree, occasionally disagree, but I virtually never chime in (unless something comes up that I vehemently disagree with! which is rare), because I feel like you and your partners in crime have so much on your plate already. So I feel like asking for more or better is — greedy! Since there are already so many planned enhancements as well as things we don’t even know about happening constantly.

    But if you are actively soliciting comments on The List, feedback, other ideas — you got it! I really like the way you have the feedback loop, if you will, set up, and I agree that many minds are more powerful and creative than any one alone, as far as brainstorming. Plus one person’s strong opinion won’t drown others out in this exchange of ideas. (Well, unless it’s yours, I suppose, wink. Just kidding!)

    Thanks for all you do; again, most interesting behind-the-scenes look.

    Sunday, June 29, 2008 at 1:07 pm #
  2. Jillie wrote:

    OMG! Mah avatar is on Casey’s blag! I is famous!

    This is a very cool and educational post.

    Sunday, June 29, 2008 at 2:57 pm #
  3. sandy wrote:

    thanks for this. I know you work hard and this just shows how much you care. rav user sandyknits

    Sunday, June 29, 2008 at 8:42 pm #
  4. nerdchik wrote:

    Ravelsourcing – is there any need for codemonkey interns/assistants? I am an Oracle DBA by trade – but am dipping my fingers into mysql. I’d be happy to volunteer. (However, I know Ravelry is your intellectual property, so I understand if you decline.)

    Monday, June 30, 2008 at 6:29 am #
  5. JoAnne wrote:

    It’s all google’s fault! Gmail is still beta after over four years. People tend to forgot what beta really means. I’ll be opting for the beta option (hopefully I’ll have the choice). You’ve done a great job! Thanks!

    Monday, June 30, 2008 at 6:55 am #
  6. destiknit wrote:

    Thanks for the extended info Casey… I often wonder how the wheels are greased and turned on Ravelry… You have a great network here and I love all the tools you’ve added to the site… For certain, I’d opt-in for Beta! Thanks again!

    Monday, June 30, 2008 at 7:33 am #
  7. VisMajor wrote:

    I read separately that there might be a guinea-pig opt-in option on our profile pages, but if you’re tracking volunteers some other way, I officially volunteer to be a guinea pig. I would be able to give feedback on Windows, Mac, and iPhone OS’s.

    Monday, June 30, 2008 at 10:39 am #
  8. Veriton wrote:

    Thank you, this was a very interesting read. I will, too, be a beta tester if I have that option, being part of the growth of Ravelry is such a fun ride, I wouldn’t want to miss it. Also to be given the chance to help out, even just a tiny little bit, is great.

    thank you and congratulations, once again, for all your hard work Vero

    Monday, June 30, 2008 at 11:42 am #
  9. Lindsay wrote:

    Thanks Casey,

    I will definitely be opting in for the beta, if i get the choice. I am sure that it is alot of work to keep things going, and i would be glad to help in any way i can.

    Monday, June 30, 2008 at 12:35 pm #
  10. khm wrote:

    Very cool—I appreciate knowing how you pull all the info together to frame the “problem” and make sure your solution hits the target. Looks like very very cool stuff.

    Monday, June 30, 2008 at 1:49 pm #
  11. KathleenC wrote:

    This was very helpful… thanks. Much like the first commenter I rarely bring up (minor) issues because I know you have plenty to do and you’re still working on things (I do understand what being part of beta means). But if you want us to speak up I will. And I never used the agree/disagree button but if you’re also tracking that then I surely will. Ravelry is such a big place… sometimes it’s had to know where to go to post the questions and to find the answers. Thanks for the directional pointers.

    Monday, June 30, 2008 at 3:47 pm #
  12. Monica wrote:

    Thanks for this Casey. It’s really neat to get an idea about how you work on the various updates for the site.

    Monday, June 30, 2008 at 4:28 pm #
  13. Andrea wrote:

    I like the idea of having opt-in for beta testers. That way it’s more productive/serious; as it is now, most of my compatriots wouldn’t care to report a bug/problem unless it was major. Sign me up.

    Monday, June 30, 2008 at 6:22 pm #
  14. Iris Moon wrote:

    I thought it was interesting to see how you track issues. There’s so much depth to Ravelry’s code and everyone does enjoy your little details (like the little icons for cake and wine).

    I’ve been trying to behave and not cause you headaches. It’s so hard. -sigh-

    Tuesday, July 1, 2008 at 6:46 am #
  15. Maud wrote:

    I love the beta. I’ve been on plenty of development teams as monkey, tech lead, or systems analyst and know that as long as you are in beta you are still creating.

    Folks love it, you are getting all sorts of testing and feedback, features are still flowing, long standing bugs are rarer and rarer… we would say Ravelry is now in its Master Bata period.

    Let me know if you could use my opting-in

    –Dancing Gardener

    Tuesday, July 1, 2008 at 8:06 am #
  16. Tonya wrote:

    Casey – I love everything you do for Ravelry and this was definitely informative!

    I read through some of your back posts and admire how much you know and care about being a “code monkey.”

    Tuesday, July 1, 2008 at 9:17 pm #
  17. Michelle/LadyMacha wrote:

    Wow Casey Thank you for the very educational look inside of this wonderful community you and jess have created. Thank you for keeping us on track and up to date on the latest happenings.

    Wednesday, July 2, 2008 at 4:36 am #
  18. Michelle wrote:

    Dear Mr Codemonkey,

    You do a great job by keeping me from working (at work that is!). Have a banana.

    Tuesday, July 8, 2008 at 3:32 am #
  19. Mary Berry wrote:

    Dare I ask: where are we on adding spinning? How do we (users) keep track of progress (if any) on a specific idea?

    Sunday, July 13, 2008 at 10:31 pm #
  20. 241 releases in 365 days? That’s crazy. How in the world do you do that?

    …wait a minute. Superhuman abilities? You’re the final Cylon, aren’t you?

    (Er. Sorry. The BSG geek just slipped out there. But still. I am vastly impressed!)

    Monday, July 14, 2008 at 7:07 pm #
  21. is rav down? Can’t get on today and had probs yesterday with both firefox and IE

    Monday, August 4, 2008 at 11:03 am #

Trackback/Pingback (1)

  1. Baby name meaning and origin for Beta on Wednesday, December 17, 2008 at 3:50 pm

    [...] Beta testing and beyond < Code Monkey Island [...]