2018 Janitor Survey


#1

It would be useful to create a survey (similar to Rust’s annual surveys) to get feedback from everyone interested in Janitor (users and non-users), in order to:

  1. Understand what people use Janitor for
  • e.g. to fix small bugs; analyze the code base; try quick hacks?
  • Understand what we should work on next
    • e.g. improvements, new features?
  • Evaluate the impact Janitor has on supported projects
    • e.g. are many bugs getting fixed? do users feel happier or more productive?

Here is what such a survey could look like:

  • Have you ever used Janitor?
    • Yes >> usage specifics
      • ‎What are you using Janitor for? (Check all that apply)
        • quick hacks/experiments
        • developing new features
        • reproducing & fixing bugs
        • doing code reviews
        • other (please specify)
      • ‎What do you like most about Janitor?
      • Do you feel like Janitor changes your productivity?
        • -15% less productive
        • ~ about as productive
        • +15% more productive
        • other (please specify)
      • ‎Finally, please help us evaluate impact of Janitor by pasting links to any contribution you’ve made with Janitor that you remember (commits, bugs reported…)
    • No >> product fit
      • ‎Why don’t you use Janitor? Check all that apply
        • I want to work offline
        • I prefer using my local editor/tools/environment
        • I haven’t tried it yet / I don’t need it yet
        • Other (please specify)
      • ‎What feature would convince you to try using a cloud development environment such as Janitor?
  • ‎What’s not so great about Janitor? Or Most annoying thing about Janitor?
  • ‎What new feature would you like to see the most?
    • Windows support
    • MacOS support
    • Review tool integration (GitHub PR, Phabricator, etc.)
    • Automated hub authentication (GitHub)
    • Automated Bugzilla/Phabricator authentication (Firefox)
    • Treeherder integration (Firefox)
    • rr support
    • gdb/lldb/rr debugging UI
    • better Rust support with rls
    • other (please specify))

If you have any ideas or suggestions about this survey, I’d love to read them! :slight_smile:


#2

Looks good! Here is a Google Form I put together based on your example to help visualize the survey layout and functionality. Let me know what you think.

Survey (https://goo.gl/forms/FGjYPybIpfUN3M0m2)


#3

This is amazing! Thanks a lot for doing that. :smiley: (Now I really want to reply to it! :stuck_out_tongue:)

Small nits (because I couldn’t resist):

  • The productivity question on the 1-10 scale seems to work too, but I feel like it’s too granular (maybe 3 or 5 levels would be enough)
  • The “Why don’t you use Janitor?” question probably needs a “Other: ___” answer (I forgot it in my draft, will add it)
  • The “What’s not so great about Janitor? Or most annoying thing about Janitor?” question should be in common for Yes/No repliers
  • I should probably trim the “new feature” options list because it’s way too long

We have a User Research team at Mozilla, and I think I’ll email them about this survey (with a link to your Google Form) to ask them what they think of the questions / about what we’re trying to accomplish.


#4

I should probably trim the “new feature” options list because it’s way too long

Do you think it’s too broad to have the following?

  • Additional OSs (Windows and macOS support)
  • External tool integration (ie GitHub)
  • Improved code analysis tools (debuggers, rls and the like)
  • Other (please specify)

#5

Ah, very nice categories! They make sense to me, thanks. :smile:


#6

It’s obviously missing:

  • Additional projects

#7

We might want to make the first question more specific so that people who only tried Janitor, but don’t use it for actual open source work, go through the “No” section.


#8
  • Have you used Janitor in the last month? By being specific about the timeframe, we make sure everybody works from the same idea of what “I currently use Janitor” means
    • No
      • Why haven’t you used Janitor? Check all that apply
        • I want to work offline
        • I prefer using my local editor/tools/environment
        • I haven’t tried it yet / don’t need it yet
        • I haven’t had time to contribute to OSS projects added
        • I haven’t been interested in any of the projects that Janitor supports added
        • I needed support for an Operating System other than Linux added
        • It’s too slow / I’m not productive in it added
        • Other (please specify)

#9

Interesting suggestion, thanks! The only thing that worries me about it is that some users use Janitor whenever they have Linux bugs to fix, which might be as rare as once or twice a year. We’d cause these users to respond “No”. (And I’m not sure if it’s a good idea to change the timeline to “Have you used Janitor in 2017?”.)


#10

Someone from Mozilla’s User Research team did a quick pass on the survey and had the following notes:

I would change “What’s not so great about Janitor? Or most annoying thing about Janitor?” to one question.

I also feel that the “Do you feel that using Janitor changes your productivity?” could be different. Maybe, are you more or less productive after changing to Janitor?

His suggestions make sense to me, so I’ll apply them to the survey. Thanks Tyler!


#11

I’ve applied the two suggested changes, and renamed the survey to “2018”… :smile:

I’d like to send it out with our next newsletter.


#12

I was looking over the survey and I noticed that when a person answers “No” to the first question, it brings them to the appropriate questions (in section 3) but then section 3 continues to section 4. In my opinion, the 2 first questions from section 4 are irrelevant to someone who hasn’t used Janitor yet. So, what are your thoughts on submitting the survey after section 3 for someone who answered “No”?

Also, for section 2, last question, if we expecting people to provide URLs, what are your thoughts on restricting this input with the use regex to ensure that they only submit URLs?


#13

Hm, I looked at the survey again, and your concern makes sense to me. I agree that it’s a bit weird to ask people who haven’t tried Janitor what they like about it or not. That being said, these two questions are optional, so anyone can skip them. And also, couldn’t non-users reply things like “this looks like a cool project and I might use it one day”, or “I love that my team can use it to do X”, or “I hate that its website uses bootstrap”, etc? (Please tell me if my ideas are too far-fetched :stuck_out_tongue_closed_eyes:).

Actually, I personally feel like this question looks too much like it accepts only URLs. I’d love to encourage people to enter anything they want here, e.g. “all my contributions to X since 2017-06-03” or “not sure if relevant but I did X, Y and Z with Janitor”, etc.


#14

Hmm, now that I have simplified the question at the bottom of Page 3, my concern for the “No” flow is that the bottom questions on Page 3 and 4 are very similar.

I wonder if there is a way to tweak the “what would convince you to try Janitor” question to make it more different from the “what new feature should we ship next?”.

EDIT: Or, as you suggested, we could also skip Page 4 for people who reply “No”. We’d miss their input on the “next features” vote, but we remove 2 slightly irrelevant questions + 1 slightly redundant question.


#15

Actually this is a great suggestion. I implemented that change. Thanks!

I believe the Survey is now ready to accept replies. I’ve just submitted mine. :slight_smile: