Here is how we built our influencer directory app before Facebook shut down Instagram API
Hi, I am one of the creators of an influencer directory app called audiencemojo.com. As of April 5th, 2018, our otherwise full scope API access to Instagram was all of sudden returning almost nothing useful. Our influencer database hence stopped growing at 800K mark.
It has never been easy to work with Instagram API because it's not documented clearly and we had to basically learn by trial and error. Following are some of the useful solutions we come up with that I'd like to share: if you are not interested in the solution itself, skip to the bottom of this post about what we want to do with AudienceMojo.
- API quota: how to scale with "only" 5000 calls per hour.
Many developers complain this is too few, especially when you want to do a follower analysis for a popular account. The truth is, 5000 calls per hour limit was NOT on your app's singular key. In fact, each user on our site was assigned an Instagram API token: as long as s/he logged onto our site using Instagram login, all API requests were counted against that unique token!!!! Therefore, 5000 call quota does not limit our user growth.
Further: when a user requested data without logging with their Instagram account, we created a dozen or so Instagram account, each has an unique token that qualifies for 5000 calls per hour. With 10 accounts rotating randomly for each API access, our quota limit was more like 50k per hour, which is plenty even for follower analysis assuming each follower call needs one API call.
- How to identify industry category of an influencer?
Instagram user's profile is simple: just a bio and a URL, plus a few stats. How do you decide what niche this influencer has her influence on?
There are many solutions, but nothing is an absolute approach. Similar to scientific research, sometime, you have to come up with your own model.
The model we came up with was by categorizing the hashtags that influencer uses frequently. We come up with following mySQL tables:
a. Main category: this small table includes just a few categories like fashion, beauty, travel, food, health etc.
b. Subcategory: within each category, we add sub-categories. For example, under food, we have subcategories like "vegan, food stylist, etc.".
c. Hashtag table: we didn't just come up with hashtags out of blue. With Instagram API, we collected thousands hashtags people use in their media. We further built a backend CMS where we enter different hashtags to different sub categories. A hashtag such as #bestofvegan gets assigned to "vegan", and automatically belongs to category "food".
d. Match hashtags to influencer:
We analyze the frequent hashtags an influencer uses, because a hashtag is assigned with sub-category and category name, an influencer automatically gets assigned with those categories and sub-categories.
Wala, now we have an influencer directory with category and subcategory!
- How to identify the location of an influencer?
There is no straight forward way, but we come up with 2 models that we use routinely (there are other ways, obviously):
a. Identify location by the most frequently used geoTag from that influencer's most recent 100 posts. Our theory is: if a user's last 100 posts have 10 geo tags, 5 are tagged as San Francisco, he should be based in San Francisco. In this approach, after we get Instagram post's detail, we need to call Google map's "reverse geocoding API" to fetch city information, which is NOT free:). Using this method, we can find the location data for about 25% of influencers.
b. location hashtags: this is the most effective method but often ignored. Location hashtag is something like "ChicagoFoodies". If you have 100 city names, combined with another 100 hashtags, you instantly get 10k hashtags! The rest is to continuously call Instagram API to gather posts on each hashtag. Our algorithms further picked the top posts out of, say, 1 million posts we analyzed daily, extract the user info from those top posts.
Coming up with models was fun. But the biggest challenge was figuring out the perfect balance to get as many data point as possible without going over quota using our rotating tokens. We build a dozen cron job running every minute from our AWS EC2 medium node, because of the server capacity, we often were limited on the number of influencers we can get daily.
-
How to render influencer directory on our website relatively fast?
By joining numerous mySQL tables: we came to realize how important indexing our table was! Without indexing, it was literally impossible to display any data in acceptable time frame in our frontend. -
Now what?
It was lots of struggle to build such app with all these resource limitations. We realized back in Nov. 2017 that we have to bring data into our database precisely to prepare for the worst scenario like Facebook cut off Instagram API access. Our vision was correct. In about 4 months time, we were able to build an influencer database of over 800k (we only render total 687K at frontend) with over 200k of them have provided contact info in their public bio for brands to collaborate with.
Many much bigger analytic firms built their own Influencer database with their own creative approach and models, but everyone is affected by Instagram API change. No doubt!
As to what to do with AudienceMojo.com? We are looking for experienced entrepreneurs who can come up with creative ways to leverage our database. We are open to any ideas as long as it's legal, ethical.
Shoot me an email if you are interested or want to discuss.
