website goes down after handful of submissions

[vjl]

I'm trying to submit a good portion of my Bookpedia collection and I am noticing that when I select a collection to submit, after a few minutes, the website front end goes down [just a blank white page] - this is for the main Books page, as well as the books I've submitted page [as well as the rest of the site].

Any idea what's happening?

Thanks!

[Conor]

Thank you for the bug report. It's one that I am aware of as I have an computer program monitoring Doghouse.

What happens is that Doghouse has become so popular I am starting to hit the limits of ClearDB (The MySQL database provider I use). I have it on the to do list to migrate to PostgreSQL on Heroku that does not have the limits, but it's not an easy change. It also happens not very often, when there are lots of uploads from several users and lots of searches and then I start reaching the limit. Luckily I did set it to back down and then it comes back up a few minutes later and the new entries get processed later, when searches are not so high.

The website goes down, cause I have not added something to detect that there no results from the database and show an error message. I plan to fix the issue and then not need that error message at all.

[vjl]

Hi Conor,

Thanks for the info - glad it wasn't me that was doing any kind of crashing! I am unsure they would be a good match, but I have a DreamHost account and have used their mysql database as a backup for some stuff, and the service and speed is good for my needs. Maybe you can give them a try, as migrating from one install of MySQL to another is a lot simpler than going to a different db!

https://wnc.li/z has more info if you're interested.

I've been slowly working on uploading a 2,000 item collection [contributing number 289 so far]. When I notice Bookpedia stops displaying the contributing text at the bottom, I just wait a while and try again. Hopefully some of the years of using Bookpedia will be useful for the Doghouse!

[Conor]

Thank you. I hear good things about Dreamhost. Our regular infrastructure is at Site5 and I could use a MySQL database there. Doghouse runs and scales on Heroku on bigger plans. But I do think you are right that I could make a quick move to JawsDB MySQL without a migration. The postgre was just always an option as it's the native database to Heroku and not part of the add-on market place. The Thresher plan looks interesting and I shall investigate, I like the idea of SSD.

[Conor]

You may all now go wild and submit simultaneous requests. I have moved the database on to a dedicated infrastructure with no query limits. Lets see how it performs.

[sitenoise]

Wow that was fast. I don't often submit a bunch at the same time but I had noticed it was a little up and down lately. So far, so good.

[Conor]

Last two weeks have been upgrades to Doghouse, with more planned for April. When I am working on it, sometimes I break it for a few minutes, every couple of hours. I do have a pipeline for bigger changes, but for small changes, it's better to test them live against a big search load, being created by users. But then I always miss that one edge case that brings the server crashing. Luckily for every error I get an email and I hate getting error emails, so Doghouse stays quite stable.

[vjl]

Hey Conor,

Just a quick heads up that while the Music and Movies sections seem to be working okay, the Books front end just produces a white page [it's been like this for a few days, as I've been spot checking every few hours]. https://doghouse.bruji.com/books/doghouse

[Conor]

Thank you, looking into it.

[FineWine]

Problem confirmed

[Conor]

Problem squashed.