Howdy y'all,

In an effort to un-rust my SWE skills and learn more about Recommendation Systems, I decided to try my hand at developing one called By-Its-Cover

TLDR:

• Check out the site and roast me: https://by-its-cover.com/
• Check out the GitHub project and roast me: https://github.com/ByItsCover

---

## Recommendation System

The recommendation system has two major parts:

• the semantic searches for books (by cover images)
• a neural collaborative-filtering model for personalized recommendations

Both systems solely utilize CLIP embeddings to make decisions on book covers, as I wanted to see if that information alone was sufficient for finding and recommending books accurately.

For the semantic search system, each query is passed to both a CLIP-based semantic searching function as well as an NER-based keyword search. The NER parsing is powered by a GLiNER model, which was ported to ONNX (as are most models in this system). Extracted entities are then used to search for books using the Hardcover API, which is the original source of each of the books in the site. Reciprocal Rank Fusion combines the two results.

The current system actually only has a couple thousand books in it, which makes both rhe recommendations and semantic search results quite limited. However, authors and book titles that are passed into keyword searches return new books that are in-turn asynchronously added to the cover vector database, making the system grow more useful only as more people search for books (which is where y'all can help *wink wink*). Searches can be made with or without an account.

For the collaborative-filtering system, I used a two-tower neural hybrid collaborative filtering model which trains on user feedback. I then use a Determinantal Point Process to diversify the results a bit before displaying them to the user (so they don't get 5 editions of the same cover presented consecutively). For now, the only feedback possible are explicit ratings of "Dislike", "Like", and "Love". I'm aware that this likely isn't ideal, and some more implicit feedback would make for some more natural user interactions and likely better recommendations as well.

Currently, while you are able to see recommendations even without an account, they are the generic "default user" recommendations. Once you sign up and rate a few books, you should see personalized recommendations within 2 hours. Following the suggestions of Eugene Yan, I implemented an offline recommendation update-system. New recommendations are fine-tuned on every 2 hours, while the full re-training of the two-tower model happens once a day at 8:30 AM EST. Each of the current configurations for the recommendation model can be found here: https://github.com/ByItsCover/bic-learn

## Software Architecture (boring stuff)

The site (both frontend and backend) is entirely deployed to AWS, with a number of different resources used for each functionality:

• Lambda -> API deployments
• ECS -> both book scraping and model training jobs
• SQS -> queueing of cover embedding calls
• Cognito -> auth
• CloudFront -> site caching
• S3 -> just about everything else, from site hosting to vector db storage

Everything was deployed using Terraform + GitHub Actions for CI/CD: https://github.com/ByItsCover

## Next Steps

While the fundamental system currently works (kinda), there are already a lot of improvements that I think may be necessary in the future:

• Replacing CLIP with SigLIP (or more appropriate model) for better visual representations of covers
• Implementing a cover-edition comparison interface to allow users to choose preferred covers for a given book, introducing one source of implicit for the system
• Begging one of my frontend developer friends to help make the site look good (I am not a frontend developer, if that wasn't already clear)
• Make a better authentication experience, as currently a generic verification code email is sent to users (and likely sent to spam, please double check!)
• Update the README's for repositories (I'm tired boss)
• Write more unit tests (see parentheses above)
• Once Hardcover releases OAUTH support, utilize that for book search (as only my rate-limited API key is currently being used)

In any case, I've already learned a ton and I'm glad that I have a real system that I can play around with and tweak now. All I need are actual users to test with!

Please let me know if you have any questions about my process at all, and also if you have any suggestions. Also please check out the site if you're at all curious: https://by-its-cover.com/

P.S.: If something crashes, or the searches load forever, or something else equally dumb happens, just let me know or open a GitHub issue, and I'll try my best to address it.

P.P.S.: No AI-Generated code was used to develop this project (to my knowledge), as that would have defeated the purpose of sharpening my skills and learning about recommendation systems.