microlink.io

All Microlink Pro plans come with a built-in proxy layer that does automatic proxy resolution and rotation based on the input URL.

What's wrong with URLs

The Internet is a wild place where every URL is different. That's an issue especially when you are doing things at scale.

When you are surfing the net and visit a website, there are situations you can't handle at all:

• The are geographic location restrictions (e.g., visiting a site from China).
• Your IP address is blacklisted (e.g., performing a request from a data center).
• You have to validate your identity (e.g., filling a CAPTCHA).

You can quickly face these problems when sending enough traffic to popular sites, like Instagram or Bloomberg, which, in turn, makes getting a successful response inconsistent.

Resolving URLs gracefully

Wouldn't it be nice if every time you want to retrieve data from any URL, the Microlink API just returned the data all while handling any of the problems above transparently?

That's exactly what Microlink Proxy layer does. No failures. No maintenance. You don't need to do anything additional. It just works™.

The Microlink Proxy is enabled for all Microlink Pro plans and it will handle all the situations for the Top 500 most popular worldwide websites, in a transparent way, by default.

Just for informational purposes, you can check if a specific request was resolved using the proxy layer by checking x-fetch-mode response header.

$ microlink https://www.bloomberg.com/news/articles/2016-05-24/as-zenefits-stumbles-gusto-goes-head-on-by-selling-insurance

 SUCCESS  3.64 kB in 329ms

 cache HIT (19h 20m 54.8s left)
  mode prerender-proxy (8.5s)
   uri https://pro.microlink.io/?url=https://www.bloomberg.com/news/articles/2016-05-24/as-zenefits-stumbles-gusto-goes-head-on-by-selling-insurance&filter=statusCode
    id iad:paw98q36bHEIlAgsFBzy5

As you can see, the proxy layer respects other query parameters, such as prerender or ttl.

Also, you can always provide your own proxy, being HTTP, HTTPS, or SOCKS (v4 and v5) protocols supported.

Last but not least, Microlink Proxy is a cross-feature: it's available for all the Microlink products.

Even some early adopters knew about this service almost one year ago one year ago, Microlink Cards has been officially launched today.

During this time, Microlink Cards has been refined until reaching its final form, taking some engineering decisions under the hood.

There are some decisions worth to mention, specially when you're building things at scale.

The online editor

Microlink Cards claims to use an online code editor.

How real is that? A code editor can be huge and hard to embed in a web application.

Well, Microlink Cards is using Monaco Editor which is a core component used by VS Code, meaning you will have the same code editor features, like autocomplete, among others.

Although Monaco Editor is lighter than it may seem, it has to load asynchronously to prevent blocking the rendering step in a web browser.

We found monaco-react a very well React wrapper that handles the setup process in a simple way.

We combined it with react-live for rendering the code in a live preview mode, giving you the instant code change experience and making it possible to load React components inside the embedded code editor. How crazy is that?

State without database

We wanted to avoid any database interaction since it will introduce a failure point and extra cost in terms of money and maintenance.

If you don't have a database, you don't need to maintain it.

Every time you do a code or data change, it will be encoded on the query parameters, generating a unique URL to retrieve the state.

That's great for data, but what about code? can we encode all the editor code inside the query parameters?

Well, you can do it, but that will produce so long URLs:

We didn't find an official URL length limitation but looks like it could be around 2,000 characters. Also, we are not particularly interested in long URLs, they look terrible.

In order to minimize the size of the URLs, we apply a compression algorithm, being MessagePack one of the best for JSON payloads.

MessagePack will take the query parameters object as input, generating a compressed buffer representation as output.

It's important to note that MessagePack will produce a buffer of binary data as output and it isn't safe to encode it directly as part of the URL payload.

Instead, it should be encoded into safe base64 in order to retrieve the state with no data mutations.

The Base64 representation will increase the size ~33%, but even with that, MessagePack is still worth it.

The image generation

Lastly, Microlink Cards is part of the Microlink ecosystem.

Microlink's mission is to provide cloud-based browsers, ready to be used for end-users to enable any browser flow, such as taking a screenshot, getting full HTML, generating PDF, and more.

Here it is the inception: Microlink Cards is a product built using Microlink.

You can see Microlink Cards as a free canvas to draw any image composition that generates a unique URL.

That URL is provided to Microlink API enabling screenshot feature to produce an image as output, distributed over Microlink CDN.

After 2 years since we launched the last version, I'm super excited to share Microlink 3.0 with you – the most ambitious release until now. This release includes not only many improvements, but also a bunch of new cool stuff.

The entire product has been redesigned, including the mission, so I want to use this space to take an overview of everything that has been shipped until today. Let me give you a bit of backstory.

Back in 2018, Microlink started as an API service that offers to get metadata for any URL. Eventually, the product grew and it was able to take screenshots or generate PDFs on demand.

That started to scale quickly (no meme), serving nowadays more than 8 million requests per month. You can see monthly live stats on the main website.

In order to support that, some key features were added, such as Microlink Cache Layer or Microlink CDN for assets, which now are a being part of the product tier offer. It's explained deeply in microlink.io/blog/edge-cdn.

The inflection point was, the more feedback users gave us about the product, the more quickly I realized that the real product is the browser.

At that moment, the product just provided a tiny subset of common browser workflows.

By making the browser the main product to offer, we were able to activate a lot more interesting use cases, like running Lighthouse on the cloud, getting the rendered markup of any website, or extracting specific data over a target URL.

These are just examples.

Any browser workflow can be eventually done by Microlink.

🤯

You could think of Microlink as a cloud of browsers solution: we provide you the necessary infrastructure to run a full securitized cloud browser, forwarding you back the result in a cost-effective way, being no worried about auto-scaling, load balancers, or paying for capacity you don't use.

With that on mind, all the product has been redesigned to accomplish the new mission: Microlink is a remote browser that can be accessed via API.

Fundamentally, Microlink 3.0 enables the appropriate space to explore every browser workflow you can do.

Also, most of the software running under Microlink is open source, so you can run it as a standalone solution if you want.

Check it at https://microlink.io/oss!

I'm a programmer been using open source software since I started my professional career.

Over time, I acquired the habit of writing open source software by default, having around 300 npm packages published.

That's why I can't imagine running a company without keeping OSS as an engineering principle.

It's a privilege to me build a company, but now is a privilege to build it based on open source software and I want to give the same effort back.

https://microlink.io/oss

From the beginning, Microlink API shipped with a built-in cache layer for speeding up consecutive API calls, aiming to improve response times for the same resource.

How cache works

When you query against Microlink API, the first time you query for a resource that wasn't previously served it will be created, what is known as a cache MISS.

The successive requests for the resource will consume the cached version, what is known as a cache HIT.

What's new

We revisited the cache layer and implemented some improvements, making it more powerful than ever.

Cache saves your API quota

Starting from today, any response served from the cache won't count towards your API quota.

This drastically changes how users consume the API. Let me clarify with an example.

One of the Microlink API use cases is to embed it directly in your HTML markup:

<meta name="og:image" content="https://api.microlink.io?url=https://example.com&screenshot&embed=screenshot.url">

That code snippet is generating a screenshot generated on the fly, always up to date.

In the past, if you did this and your website had 1000 pageviews, it meant you consumed 1000 requests from your API quota.

Now, you only consume \one request\, serving the rest from the cache and not counting them in your API quota plan.

Cache is served around the world

The first time you hit Microlink API, the response will come from one of the servers located somewhere.

Once cached, the successive requests to the same resource will be served using the CloudFlare Network, meaning the cached response will come from the nearest edge server in the world (and they're a lot, more than 240 edge servers in over 90 countries).

A little benchmark between cached response (blue) vs. Uncached response (gray):

You can see how cached resources always have the lowest response time associated, no matter where you are.

Cache optimizes screenshots on the fly

Microlink Screenshots is one of the most used product features these days: we're serving around 100K fresh screenshots every day, generated in an average of ~1.5 seconds.

When you take a screenshot the image generated is hosted by us, additionally taking some special cache considerations into account.

First, the image will apply lossless compression on the fly. The image will be the same as the original but the size will be smaller, saving some bytes there.

Second, WebP will be served if the browser supports it. Most modern browsers support WebP and it can decrease up to 42% in average image size.

A web browser is one of the most complex pieces of software, with some internal sub-systems that work together for resolving any kind of URL on the Internet, even if the content was written with HTML tables in 1992.

Microlink API is a service that provides a high-level API to control a browser instance in the simplest way possible, where the different features can be enabled or disabled using query parameters.

When we started the service, just a few things could be done. Now, we're supporting +30 query parameters.

Just url is the only parameter that needs to be specified, but also any of the following query parameters:

Data

Enrich the response payload detecting data from the target URL.

• audio: enables audio source detection from the target URL.
• data: gets specific content extraction from the target URL.
• iframe: gets, if it's possible, the embedded representation of the target URL.
• insights: gets lighthouse perfomance metrics from the target URL.
• meta: gets unified medata from the target URL.
• palette: gets color information over any image present on the response data.
• pdf: gets a PDF over the target URL.
• screenshot: takes a screenshot over the target URL.
• video: enables video source detection from the target URL.

Browser

Tell the browser to act in a certain way or perform some tasks.

• adblock: enable/disable adblock over abusive third-party content over the browser page.
• animations: enable/disable CSS animations and transitions into the browser page.
• click: clicks DOM elements matching the given CSS selectors.
• device: emulates an specific device (viewport, user agent, dimensions, etc).
• hide: sets visibility: hidden on the matched elements.
• javascript: enable/disable the javascript engine on the entire browser page.
• mediaType: changes the CSS media type of the page.
• modules: injects <script type="module"> into the browser page.
• prerender: enable/disable browser navigation.
• proxy: uses a proxy server as an intermediary during the requests.
• remove: sets display: none on the matched elements.
• scripts: injects <script> into the browser page.
• scrollTo: scrolls to the DOM element matching the given CSS selector.
• styles: injects <style> into the browser page.
• viewport: establishes a set of properties related with the browser visible area.
• waitFor: waits a quantity of time or selector before processing the content of the browser page.
• waitUntil: ensures to wait browser event(s) before considering navigation succeeded.

Response

Apply some modifications over the response data for better accommodation.

• embed: embed a specific response data field respecting the content type.
• filter: filters a list of properties from the response data for bandwidth saving.
• force: forces a new fresh response data bypassing the cache layer.
• headers: customizes requests using custom HTTP headers.
• ttl: configures the cache layer specifying the time to live.

Six months ago we launched Microlink for Screenshot as part of our product iteration to offer browser capabilities ready to be used directly consuming our API.

Today, we're thrilled to announce Microlink for PDF 🔥.

You can see a live demo at microlink.io/pdf.

Microlink for PDF is based on the same engineering principles as the rest of the functionality behind Microlink API: making it trivial to get something back from any URL.

Enabling the new pdf query parameter will add a pdf data field as part of the response payload:

The PDF file created as output of the process is hosted on Microlink CDN and it can be combined with ttl to automatically refresh in the background, providing an updated PDF representation of the target URL.

Additionally, a set of new query parameters have been added for better PDF accommodation, such as scale, margin and media.

These query parameters are specially helpful when the target URL is not actually prepared to be properly exported as PDF.

Another thing worth mentioning is the ability to consume the PDF directly from HTML markup using embed mode:

Also as a button:

This way, the PDF file will be created on the fly, making it a backendless solution: just HTML.

We released Microlink SDK v4.5.0, introducing a powerful functionality: the ability to embed native iframes 🔥.

Every time you use Microlink SDK, it turns any link a beauty link preview, where the card can be displayed with three size variations: 'small', 'normal' and 'large'.

The card approach improves a lot of the URL preview experience, making it possible for you to adopt it on your own website or application.

However, some domains on Internet (like Instagram, SoundCloud, Spotify, Facebook, Twitter,...) have their own way to embed their content, being the Microlink card displayed less recognizable than using their own way.

Microlink SDK v4.5 introduces the ability to set iframe as media property for using native embed solution always it's available.

The property media takes into consideration the URL preferences, setting up the Microlik API call in order to satisfy the media requirements.

For example, when the URL is an audio provider (like SoundCloud or Spotify) if you set media: ['audio'] you are telling at Microlink API level that you want to detect the streaming source of audio behind the URL, enabling audio for that purpose.

If the audio detection is done successfully, Microlik API will return an audio data field as part of the response that will be used by Microlik SDK for creating the audio preview.

In the same way, in case you prefer to use provider iframes, just need to set media: ['iframe'] for enabling iframe detection.

After that, Microlink API does the magic and returns you the iframe, leveraging into Microlink SDK for embedding it properly.

Note that we are specifying media as a collection. That's because you can add more than one value to be used as fallbacks.

That's specially useful for the cases where the URL provider doesn't support iframe, or you don't know if the URL is exposing and audio/video streaming source to consume.

Providers supported

That's the best part: Any provider that implements oembed specification is supported.

A non exhaustive list of the most common providers could be:

• Instagram
• SoundCloud
• Spotify
• YouTube
• Genial.ly
• Twitter
• Vimeo
• Flickr
• Facebook
• TED
• CodePen
• CodeSandbox
• Dailymotion
• Reddit

Microlink SDK v4.4.0 was released, introducing a bunch of functionalities for turning any link into a beautiful preview.

This release has been focused on improving the media player experience when the target URL has video or audio content.

Jumping into the details, the improvements have been:

Audio support

Although Microlink API was offering audio detection previously, this detection has been improved to ensure to cover most of the popular audio providers on the web, including podcasting.

You can see some of the tested services in our storybook.

Media preview

The media player has been extended to include some common and useful functionalities as buttons:

• rewind/forward
• mute/unmute
• playback rate adjustment

These functionalities make the media experience closer to the experience offered by mobile applications, brought on the web.

Finally, controls are exposed in a way they can customize based on the user case necessities.

Extensive documentation

Last but not least, our documentation site has been revisited, ensuring all examples and code snippets are complete, creating new examples and demos, making possible bring Microlink to any website.

That also includes react and vanilla templates at CodeSandbox for instant playing with the UI component.

The Microlink API Parameter screenshot has been updated reflecting more specific configurable parameters, such as adding browser overlay & background customization:

https://microlink.io/docs/api/api-parameters/screenshot

Additionally, we built a new landing page for showcasing the new functionality:

https://microlink.io/screenshot