What is scu-course-evals.com?

SCU Course Evals is a searchable aggregated directory of ~20k PDFs generated by student surveys of classes on a quarterly basis. Students and alumni can login and search for classes or professors and find the aggregated ratings of each class.

Story Time

The Beginning

SCU Course Evals was a multi year project that started in 2018, my junior year of college. It started with a thought along the lines of "Wouldn't it be great if I could take easy classes for requirements that I didn't care too much about, and find good professors for classes that I do care about?" Luckily for me, Santa Clara University already surveys its students on a quarterly basis asking them to rate the class and professor. These surveys are then released as PDFs to SCU Course Evaluations.

So the plan was simple. I would scrape these PDFs, parse the text, and then figure out which class to take next time registration came around. But all plans are subject to change, and this one was no exception. With a lot of elbow grease and asking around online, I successfully wrote a script to download all of the PDFs available on the SCU Course Evalutations directory. But then I hit a wall when I moved onto the stage of converting the PDFs to text. I tried using the Python textract package but it ended up giving me a lot of garbage output mixed in with the actual information I was looking for. I couldn't find anything else at the time, so the project languished.

The Middle

6 months passed and Senior Design 2019 had rolled around. All of the engineering students of the SCU class of 2019 were showing off, pitching, and explaining their work to anyone who would come by to their booth. Lucky for me, my team abandoned the $1500 of funding we applied for and were granted. As such, we were free to roam and ask other students how their projects worked. I went around nodding approvingly while I picked away at my overflowing mini plate of free hors d'oeuvres. It was during this period I was able to catch up with another computer science and engineering student who I had been good friends with my freshman year. He was very knowledgable, and when presented with my dilemma he suggested a tool he had used in Node.js, which wrapped a tool called pdftotext. I kept this in the back of my mind.

The Middle (Part 2)

Yet another 6 months passed, and I was 2/3rds of the way to being done with my masters. I had just finished TA'ing Software Engineering and Computer Architecture. I was at home in December 2019 on winter break, idling. I finally looked into the pdftotext tool my friend had told me about, and it worked. No garbage, just text. Soon, I found myself fiendishly consumed by this project. With the text extracted from the PDFs, I wrote a script to extract the quality, difficulty, and time spent for each class. Once this was complete, I had an an Excel spreadsheet of all of the class information I could ever want. Here I learned of the magic of pivot tables. I used it to average the stats of a class and professor across all of the classes and professors that had ever taught it. This was what I was looking for. But this was so good, I just had to share it.

So, I started rapidly prototyping the website and adding features over my winter break. In order to comply with the data policy set by the SCU Course Evaluations page by the school, I had to limit the distribution of this information only to people affiliated with SCU. Forunately for me, students that I TA'd for Software Engineering already wrote code for Google OAuth in Flask which did exactly that. So with their permission, I copied their code into my codebase. The hardest part about building the website now was figuring out the UI. I solicited ideas and feedback from my girlfriend so that I could have a semi-unbiased idea of what an actual user might want.

The End, and Now

With feedback from friends excited for the concept, I was hoping for the site to take off like wildfire across campus. I launched the site and got the word out as far as I could with my friends and organizations. It didn't quite blow up the way I hoped it would. But, it made me feel good when I saw my friends browsing the site in the engineering common space (Heafey atrium) without my prompting. Why did it feel good? Because it meant that I had shipped code that people used, even if it was a small group.

As of this writing (3/16/2021), 400+ unique users have logged in to view the site (there are likely slightly more because I cleared the user table a while back). Some users are friends, one is a professor (who viewed their own rating and then logged off!), and many are people who I don't know.

Since the website launched, no new evaluations have been posted for me to scrape and incorporate into the dataset. So the website is a little stale because only information up to Fall 2019 is included. I'm not sure if this is a reaction to this site, or COVID, or just a coincidence. But because of this, the docker container that hosts the website has now been up and untouched for 12 months, which seems like a pretty big achievement to me (aside from the neglect and potential security risks).

Conclusion

This project is a source of pride for me because it is an example of code I (mostly) wrote which to this day is helping a handful of people get the most of their $60K/yr education. I learned a lot about web technologies through this project, and hope to do something similar sometime in the future. I had a lot of fun writing the code. I also found seeing people using the site very rewarding, and as such, I really would like to create something similar soon.

Credits

Thank you to the people that I mentioned in the story:

• My friend knowledgable in Node.js
• The Software Engineering dynamic duo who let me steal/copy their code
• My girlfriend for letting me bounce ideas off her

And also thank you to other SCU alumni who inspired me and whose footsteps I followed:

• Scott Hardy - creator of scuclasses.com and scubooks.com
  • Scott, I personally benefitted from your websites and you inspired me to create something that would help other students at SCU. (also I copied HTML from your blog to mine, so double thanks!)
• Fredrik Blomqvist and Joesph Théberge - creator of scuevals.com
  • Fredrik, Joesph - I remember asking you guys about your project during a poster session for your senior design and was really intrigued. Also, your guys' code is amazing for a senior design project, and I am very jealous of the domain you got for your site!