Today I want to tell yall about a new Android app I've been prototyping this summer called Turtle Shelf. You take a photo of books and it tells you which books you'd most like. This solves real (first world?) problems in my life:

1. Sometimes you're with friends or family and they have a bookshelf. Which do you borrow? Or, maybe you've already read one of the books but don't see it yet, so you don't know to talk about it.
2. Sometimes you're out for a walk and there's one of those community book shelfs, or a free books give away pile. Which, if any, are worth taking?
3. Sometimes you're in a book store, or a thrift shop, and there's endless books. Most are unappealing, some are certainly fantastic, but how could you possible tell them apart?

I'll start non-technical and then dig into the details a bit.

Screenshots

It lacks polish but today the app works. Here it is!

Roadmap

All the basic pieces are in place to make this app barely useful to me today, which is great. However next I need to identify which components (AI, UI, design) need improvement so that I have a product. Then, make those improvements and publish an open beta on Google Play. I've already authored most of the Play Store product and pages, which I have experience with from ChessCraft.

Market Analysis

There's actually already a few apps out there that do (basically) exactly this. So why make Turtle Shelf?

1. They don't use edge compute and must pay pennies for every query. So there's no free trial.
2. Features. I actually didn't find any app that did the whole design as described here.
3. Recommendations. AI tools are so powerful these days that I'm able to only use recommendation algorithms that I fully control.
4. Quality & Design. I didn't find anything that looked spectacular.

Like with ChessCraft, this is a real problem and I want this real solution. So the first and most important market test is how much I like and use my own app. This is called dogfooding - using your own services.

AI Written Code

Remarkably - I haven't written a single line of code for this entire app. It's all Claude. Fable at first to lay a strong foundation then smaller models like Opus to bugfix and tweak. Guiding the AI to build this has been a very different experience than the other programming projects I've for 20+ years. Overall it's been a truly joyful and engaging experience to guide the AI through the design, architecture, and computer science tradeoffs for this project.

My experience as a full stack developer, app product manager, and computer scientist certainly helped. Yes - anyone can one shot this app today with a good prompt and it will kind of work, but if you don't know what you're doing you'll still run into major product and milestone problems. Here are some where I had to put my foot down despite Claude's recommendations:

1. Licenses. I don't use non-commercial datasets or models. Claude thought this would be okay "because it's just a prototype".
2. Edge compute. It was often steering me towards paid AI services in the cloud because it's "so much easier" or gives "more accurate results".
3. Complexity. It wanted me to sign up for a lot of enterprise grade services as if I were building the next Uber or Tinder.
4. Datasets. The AI hasn't been great at transforming datasets into simpler and more usable schemas. For example it struggles to understand that here ISBN is "not" useful. A user will never be able to confirm that a book scan from a photo perfectly matches a specific ISBN. There may be 200 books called "Turtle" and we can never distinguish them. So ISBN is never a useful index - even though this is all about books!

Architecture

Here's the basics of how this works:

1. The Android phone takes a picture of books.
2. An "image segmentation model" runs completely on the phone (edge computing). This creates many tiny images of one book each, from the big image of many books.
3. An "optical character recognition model" (OCR) also runs completely on the phone, and gets all the text it can from each of the book images.
4. The text (not the image) is sent to my home server at: https://turtleshelf.frameofmindsoftware.ca/api
5. My home server has a 15 GB Open Library database of hundreds of thousands of books. It does a fuzzy text lookup to (probably) match the OCR text to an actual book.
6. The home server sends the book titles, descrpitions, ratings, tags, etc back to the phone.
7. The phone runs its own simple recommendation algorithm to predict your own rating, and shows you the results.

A critical piece of this design is the edge computing. I can eventually share this app with many people for free because I don't need any expensive compute or AI services running in the cloud. The only reason I need the home server is I can't deliver a 15 GB book database to mobile users. But that's just a simple text lookup and even my small Lenovo ThinkCentre M75q-1 16GB could handle thousands of daily users.

AI Models

• Segmentation: RF-DETR-Seg Nano. Fine-tuned on the desktop from the COCO checkpoint using rfdetr and PyTorch CUDA and my wonderful mediocre GPU. The training data is Harald Varner's "book-spine-instance-segmentation" dataset from Roboflow Universe (1463 images, CC BY 4.0). The model is exported to ONNX, and the phone runs it with ONNX Runtime (RfDetrSegmenter). The model file is an asset bundle separate from the APK.
• OCR: Google ML Kit Text Recognition v2, a pretrained SDK that isn't trained here, bundled in the APK.
• Fuzzy matching: a bunch of heuristics are run on the phone and server to clean the text (like replacing "8rilliant" with "Brilliant" because it's a common word) and fuzzy match to the 15 GB book database. Nothing extraordinary here - just a bunch of hacks tied together and benchmarked for accuracy.

Send me Pics!

I've been taking lots of photos of random books: family bookshelves, outdoors community libraries, giveaway piles, bookstores and thrift shops in Dublin, you name it. These serve as my foundation dataset, with good real world examples of differences in lighting, book quality, and arrangements. If you'd like to help out with Turtle Shelf, please send me your pics to help me develop the app! And if you like, send me an email to ask to be added as an open beta tester, whenever that happens.