The process of building a platform to help users practice programming challenges guided by AI: A summary from the first few user tests

I built a platform helping people practice programming challenges guided by AI to help improve interviewing outcomes. My rationale behind this stemmed from an anecdotal experience: I practice programming challenges and found myself regularly asking GPT or Bard to help me break down solutions I didn't fully understand.

Breaking problems and solutions down as simply as possible, helps me build a better understanding as I problem solve, and to be honest, it most importantly helps me stay more consistent.

My platform leverages AI to facilitate this process. How? It enables you to receive a random programming challenge based on the difficulty level you preselect. I use AI to send you pseudo-code phrased as questions.

Hypothesis One

Hypothesis:

Breaking the pseudo-code for building an optimal solution for a programming challenge down into questions can help you think through the process of constructing an optimal solution for a programming challenge.

This will facilitate the learning process as this retains the element of 'thinking for yourself' required to retain information, while simultaneously reducing the friction involved in practicing these problems.

In this scenario, friction refers to the optimal balance between struggling through a problem and realizing you need help and leveraging that help to move quicker.

Video Review:

https://youtu.be/sbtGBlQ-zeM?si=QhIPW9Ej_cSqUv7c&embedable=true

Reviewing Feedback

From analyzing feedback from the first 4 tests, I sense that there is some veracity to the above hypothesis. The operative word in the phrase, 'pseudo code broken down into useful questions' is 'useful'. The questions were only useful in one test. More work needs to be carried out to ensure consistency in question quality.

This means, 'reducing question redundancy' and 'increasing the relevance of the questions' This is a top priority as it is core to validating this hypothesis. I categorize this as a high-priority hypothesis underlying a core assumption of this entire platform.

Hypothesis:

Having an AI assistant to review your code and help you understand where you might be going wrong and generate questions that get incrementally descriptive, would help accentuate the core value of this platform.

Isolating things I am working on, helps me zone into the core pillar of this platform. Since this hypothesis accentuates the core value, it is necessary to first test the core value before worrying about an ancillary value.

Long Wait Time

It takes a very long time to receive a response from LLM models, particularly with longer prompts. This is certainly a problem that affects the entire usability of the platform. I plan to run this platform through at least 20 - 50 user tests, incrementally exposing more and more of the platform.

I believe this process will expose the platform to enough eyes to ensure the solution is based on tested assumptions with real-world validation. Additionally, this process would expose and enable me to fix a fair amount of bugs affecting overall functionality without falling into the trap of perfectionism. Most importantly, it is insanity not to talk to your first users.

This process will help me learn to ask better questions about my assumptions, understand the problem I am solving from a more holistic perspective backed by tangible data, and understand my target audience substantially more. The intuition to know what to build and how will come from this experience.

Potential Solutions

Potential solutions at this point would be creating a function retrieving each programming challenge from my database, using AI to generate pseudo code questions for each challenge, and storing the result in a cache with some references for the level of difficulty of the problem (for the user selection), other useful tags and a unique identifier.

With these results cached, I would speed up the time it would take for the initial start time.

Current WaitTime Video

https://vimeo.com/883456606/f301f02e2b?embedable=true

With the AI assistant - A few offline conversations with engineers along with a few users make me lean towards streaming responses based on a timer. This means experimenting with keeping track of the length of time it takes for a user to solve a problem and sending clues through the AI Assistant in intervals, with a 'Help me' button showing up every 5 minutes to trigger the AI Assistant to return this response.

This could involve keeping track of the time and sending a request with whatever code has been written on the editor every 4:10 minutes (assuming the user has stopped typing for say about 10-15 seconds). This would be used to generate the AI Assistance tips based on the state of their code a few seconds prior.

Methodologies employed to generate valuable user feedback

In testing this platform, I will go through a few phases of tests which will serve different purposes:

Give users a link to the platform, and ask for feedback after they have played around with it. This will be the primary mode of acquiring feedback at the 200-500 user tests threshold. Most of these tests will be followed up with short calls to get user feedback through conversational questions. This is reserved for when my platform is a lot more robust, and I have solved issues related to wait times for responses.
Give users access access to the platform, and ask them to play around with it in real time asking questions as they test it out. This is good to get deeper feedback regarding usability and the quality of the questions. This is one of the formats I currently use.
Schedule a Zoom call, and ask them to solve a programming challenge and then ask questions. I intend to shift to this phase between user tests 15-50. This is necessary to observe how users are interacting with the platform at a phase where I would be a little more comfortable with the overall usability of the platform (fixing the major issues to fully test the core assumptions of the platform).
Asking users to pair program a programming challenge and asking questions after the session. I believe a greater value for this platform comes when you enable users to pair program using the core functionality already built into the platform. Anecdotally, this has made practicing programming challenges more exciting for me, encouraging more consistency while helping you practice explaining your logic as you would in an interview.

Simply put, this will facilitate more consistency by allowing friends to pair programs in preparation for interviews. Having spoken to graduates from App Academy in particular, I know this is a concept they already incorporate into their curriculum. However, this is done purely via VSCode and a Slack huddle/Zoom call.

Next Steps Moving Forward

Improve the quality of questions generated.

Fixing core internal bugs. There are 2-3 at this moment that are top priority, specifically when I first receive a programming challenge, the code editor shows the pseudo-code questions from the previous question, assuming I had been using the platform before. I need to either make changes as to how I am persisting with programming challenges and the solutions generated or restructure my code to clear up the previous question when a user solves a challenge, closes the browser, and reopens the platform again. This would probably solve other issues I have regarding programming challenges not always being in sync with the challenges shown and having to click the reset button to force this consistency.

I have a hunch that I will need to disable the ability to run your code against randomly generated test cases. I don't think this functionality is necessary now; it adds more complications to things that need to be tested, especially since I am leveraging AI to generate these test cases.

The more layers of complexity I include so soon, the more complex the platform becomes without deeply validating its core assumption

Here's a breakdown of past issues discovered and resolved:

Also published here

I Built a Platform to Help Users Practice Programming Challenges Guided by AI

The process of building a platform to help users practice programming challenges guided by AI: A summary from the first few user tests

Hypothesis One

Video Review:

Reviewing Feedback