Pine.blog

Pine.blog was down for a while today (~6 hours). What follows is a bit about why and what I've learned from the experience. In the end, nothing was lost and the site is back up. My apologies to all of my users; thanks for your patience.

Today did not go as planned. As I do, this morning I opened Pine.blog to read the news and instead of seeing a newly updated timeline, the app had logged me out. Worse still, I couldn't log back in, and the website wasn't responding. Sigh.

Some quick investigation revealed that the disk had filled, which caused the creation of new sessions and access tokens to fail. Admittedly, I don't have any sort of disk usage monitoring on the database server, but because I'd just doubled the amount of storage on that server less than two months ago, I wasn't really thinking it would fill up anytime soon. Regardless, it had.

The Causes

From what I can tell, a combination of external factors and my own design decisions led to the disk filling up much faster than I was expecting:

Feeds are kinda huge (sometimes). A few Pine.blog users have feeds that seem to contain hundreds of new items every hour, which Pine.blog will faithfully parse and store in their timelines. In some cases, this meant that such a feed could contain 200,000+ items in only a few weeks. What's worse is that because of a recent change to how timelines work, a timeline can't hold that many items anymore which means Pine.blog was storing a ton of data it would never actually use.

Pine.blog stores a lot of data. By design, Pine.blog doesn't really delete items that it has previously parsed and it stores multiple copies of certain data to make it easier to serve, but this drastically inflates storage requirements.

I misunderstood my tools. I'm not a database guy; I'm an app developer. To me Postgres has always been a bit magical, and today that bit me. About a month ago, I thought I'd read somewhere that the newest versions of Postgres didn't need to be routinely vacuumed, and that running a daily vacuum was an old, outdated practice, so I stopped doing it. This was a bad idea. Without that daily cleaning, the Pine.blog database couldn't reuse space that it no longer needed, and so it kept asking the OS for more disk space, right up until there wasn't any more.

In the end it was the mundane, but powerful combination of a few small things that caused the issues today: some overzealous design choices, a few misunderstandings, and a few overactive feeds.

The Solutions

Right away, I should admit this whole thing could have been solved a lot sooner. I underestimated the issue at first and tried to keep the site up while I fixed the issue. In hindsight it would have been a lot faster and easier to resize the server to something that had more disk space, vacuum the database, resize the filesystem and scale back down. This whole thing could have been solved in 1-2 hours instead of 6-8.

That said, here's the two things that I'll be changing about how Pine.blog works in order to prevent this problem in the future:

The nightly vacuum is back. Removing it was a dumb idea.

Pine.blog will no longer store the entire history of external feeds. This one isn't fully in effect yet, but expect the change soon.

As of recently, each Pine.blog timeline has a maximum length of 3000 posts. Once a post falls off the end, you won't be able to find it by scrolling backward in your timeline anymore. This is exactly what Twitter does (although their limit is a modest 800 posts). Feeds will soon do something similar. Pine.blog will keep just the most recent posts in a feed (~3-5000 posts in total) and anything beyond that point will be removed from Pine.blog's database.1

I suspect that no one will actually notice these changes once they're rolled out, but to me they mark the end of the road for a naive dream I had back when I started Pine.blog.

tl;dr Turns out parsing and storing entire portions of the web is hard.

1. Hosted blogs are uneffected. Pine.blog will store your entire blog . This change only effects feeds from outside of Pine.blog.

This is pretty technical, but up till now Pine.blog has been running off of Django 1.11 LTS which is going out of support in April and DRF 3.6 which is 3 years out of date. I’ve largely avoided upgrading major versions because it can be a major pain and I was using an LTS anyway so it wasn’t out of support.

However with April coming fast, I decided to take the plunge and upgrade Django, Celery, DRF, and much more to the latest versions (the latest LTS for Django).

Upgrading was still a pain and took time away from feature-development but it’s a necessary thing to do. All in all since the Python Web community marches in step with Django, upgrading Pine.blog far less painful than I was expecting and the new version went live last week. With that out of the way it’s time to get back to features!

Thanks everyone for making 2019 a really great year for Pine.blog. To celebrate the end of 2019, the first 50 new users can use the code HACKER_19 after signing up to get a free month of Pine.blog's premium features for free!

Pine.blog is a feed reader and blogging app that lets you start a blog with 1 click! Upload photos and write about your interests! See likes, comments, and replies from across the web right on your very own blog!

Sign up today to find and follow sites for free. Use the code HACKER_19 to get one month free of blogging on Pine.blog.

Blogrolls are an old idea. They originated back with the original blogs back in the early aughts. The idea was simple: bloggers wanted to point their readers towards other blogs they liked. It was a recommendation system built on people and on trust. Blogrolls spread far and wide helping readers and bloggers discover and link to each other in a meaningful way.

In an effort to re-kindle some of the old ideas that made blogging so great in the old days, Pine.blog now supports adding a blogroll to your blog with 1-click! Simply enable the blogroll for your blog in your account settings, and you’re done!

Read the full post here: https://blog.pine.blog/2019/11/the-blogrolls-are-back/

I passed a really significant personal milestone today: I've finally released the features that I envisioned when I started Pine.blog.

In 2014 when I first thought of the idea behind what would become Pine.blog, I had a set of features in my head that the service would eventually have. Today I've released the last of those features.

If you're worried that Pine.blog is feature complete, don't worry, I'm not stopping development any time soon. In the last 5 years I've come up with ideas and plans for features far beyond what Pine.blog supports now. There's a bright future ahead. It's just that now I'm no longer chasing my original vision, I'm moving beyond it.

It feels good.

The wait is finally over: Pine.blog users can now start their own blog! Blogs are designed to be lighting fast and easy to use, and there’s also a brand new editor and full drafts support. With this update, Pine.blog way more than just a feed reader, it’s a way to stay updated and to share your thoughts with everyone.

All blogs have built-in RSS and JSON feeds so anyone, even if they don’t use Pine.blog, can follow what you post, and all blogs are automatically added to the Feed Directory so that other Pine.blog users can find your site quickly and easily.

It’s happening! In the next version It’s happening! In the next version of Pine.blog premium users will be able to start their very own blog right from the app or the web! The blogs are designed to be lightning fast and easy to use, and I’ve been working really hard to make everything as simple, clean, and intuitive as possible.

There’s a ton of work that’s gone into this release. I’ve been working on it since February, and I’ve crammed a whole lot of new features, fixes, and improvements into this latest version. Everything is in the finishing stages now: I’m mostly just tidying things up and doing a lot of testing to make sure everything is rock solid for release.

I’ve always said that Pine.blog was more than just a feed reader:

I’ve wanted Pine.blog to support custom blogs since the beginning, and that’s largely because I think that by making it easier to both read and write on the open web, we can give people a viable alternative to more traditional social networks and offer them an escape from the problems those platforms have…

This is why I’m so excited by what I see as the future for Pine.blog. If we want people to move off of the platforms we think are the cause of so many problems in our world, we need to give those people a place to go. Most people aren’t going to start a blog and use a feed reader unless we, as the people with the skills to do so, make the Social Web better than Social Media.

Recently I've been focused on finishing the next set of major features for Pine.blog, but as happens from time-to-time I got a bit distracted and ended up knocking out a whole lot of little features that I've wanted to build for a long time. Today's update was entirely server-side, and I'll have a new update for the iOS app coming in a few days.

Wordpress Enhancements

In addition to making a lot of things just plain faster, I've also added a new feature for Wordpress users so that when users post on their blog, Wordpress will automatically let Pine.blog know and their posts show up much more quickly in their (and other user's) timelines.

For me, this feature involved yet another foray into undocumented XML-RPC APIs from over a decade ago. So much of the information just isn't easy to find anymore so building features that use them is more archeology than software development.

API Keys and Webhooks

Users that want to write scripts using Pine.blog (or build custom applications) can now get an API key quickly and easily that lets you access the full range of Pine.blog APIs.

For those who want to dive even deeper into their data with Pine.blog, users can now add a Webhook URL if they'd like to receive updates from Pine.blog whenever a feed they're subscribed to changes.

API Documentation Revamp

I've completely redone the Pine.blog API documentation. Hopefully this makes it much easier for developers to discover and use the Pine.blog/Feed Directory API. The new documentation includes much more detailed information about the throttling limits, what to expect from each endpoint, and much much more. If you're looking to use the Pine.blog/Feed Directory API in your app or service you can find out more from the documentation here.

A lot of these developer-focused features are test runs for much broader features that I have cooking in the background. Meanwhile I'm trying to keep the majority of what I do with Pine.blog a lot more user-focused.

https://brianschrader.com/archive/lots-of-little-things/

https://pine.blog/

Pine.blog has been out for over a year now and it's been getting better and better over that time. However, although quite a few people have signed up, most have stopped short of signing up for a premium subscription. At first I thought that adding a free trial to Pine.blog would help, but that doesn't seem to do much to encourage signups.

In that light, Pine.blog no longer requires a subscription to use: just create a free account and you're good to go! If you want to organize your feeds into multiple timelines or start a blog* you'll still need a premium subscription, and don't worry, you still get a free trial.

Pine.blog has come a long way since launch and today I'm announcing the next big step: Version 1.3, which is available now. This version is pretty jam-packed with features and improvements, but the most notable addition is the ability to organize the sites you follow using timelines.

Check out the full blog post here!

https://brianschrader.com/archive/pineblog-version-13-multiple-timelines-/