Goodreads Data Search/Scraping?

[kylecarroll]

Hi all! I get all of my book metadata from Goodreads, but it's not one of the defaults on Bookpedia. I attempted to create a search plugin to do so...but my coding experience is in Python (and limited) and I have absolutely no idea what I'm doing in cocoa. I have an API key (you can get one free from goodreads.com/api/keys ) if someone would like to use it to create the plugin and help me out. This is literally the ONLY feature missing to make Bookpedia a perfect app in my opinion.

[Conor]

Since you are comfortable in python you can create two functions in Python that would search and return details. I can try to write a plugin that would integrate that info into Cocoa and Bookpedia.

One function that takes a search term and returns a list of titles and IDs.
The second function takes an ID and returns a dictionary with all the values of the fields for each key.

The field names are what you would think, but there is a list for the common ones, and the ones for Bookpedia.

It's something I been meaning to do, as there a number of open source search plugins for Plex on Git for DVDpedia that are all in Python and would be a way to integrate that.

If we can get it working together, I might be able to refine it then with your input so that it becomes a flexible interface for other languages and other users that want to create their own plugins.

I have not created a Goodreads plugin, since I sell Bookpedia for money I don't fit in their API terms of no commercial use. It's fine for you as a user to create one. They do say they would give consent for special cases, but have never gotten back to me when I write to them.

[kylecarroll]

I will definitely get working on that over the next couple days. I think together we can definitely create something useful! So, what would you need from me specifically? Just the code for the two functions, or a working .py script that can be run independently?

[kylecarroll]

Conor,

I wrote a simple Python script that takes an ISBN input and gets the Goodreads book ID, and then passes that to another function that will parse the metadata from that ID and put it into a dictionary using Bookpedia's tags. The variable "metadata" in the main function is the final output/return from the two working functions. Let me know if you can work with this, or if there is anything else I can do.

Side note, the reason I have it take only an ISBN as an input is that Goodreads is great in that it can give you metadata specific to your individual edition (cover, page count, publisher, etc.) based on ISBN. If it was just a simple Title/Author search, you would get the most popular edition on their site and not necessarily the one matching your own and for me personally, and many other collectors I'm sure, it's nice to get all the info at once for your specific copy.

https://github.com/LiquidHamilton/Testi ... %20Scraper

[Conor]

Thank you. Let me finish some work on CDpedia and Pocketpedia and then jump on this and figure out if there a quick way to bridge a .py script so that I can import all those details into the search plugin, then you can carry on tweaking. I can tell a ISBN apart from a keyword search, so I can call a different function based on that info, so the plugin still works with a regular author (and/or) title search.

[kylecarroll]

Hey! Just wanted to check in and see if any progress has been made on this front? I'm REALLY loving the app itself, but I hate having to use my own Python app to scrape the data from Goodreads and then manually add books on Bookpedia (The default searches don't get all the correct info 99% of the time it seems when working with a specific ISBN).

I know you can't do GR API Integration due to licensing issues, however, couldn't you at least add it as a plug-in option if users provide their own API Key, similar to how it's set up for Amazon? I would REALLY love if that could be worked on. I'm willing to help contribute code if needed since I'm familiar with the Goodreads API.