How To Fix: Missing map from feedback button

We have noticed that some reporting tools and “send feedback” buttons create screenshots with a blank image instead of a beautiful map capture, and we have decided to investigate why.

For this, we are going to:

• use usersnap.com to create reports with a screenshot
• create a simple web page with a MapTiler SDK map
• try out a few things until it works!

How do report services create screenshots?

There are mainly two ways:

• the built-in/permission-heavy way
• the stealth/no-user-permission-needed way

The built-in way is essentially the window-sharing pop-up that shows up when we, end-users, want to share our screen or join a video call (Zoom, Google Chat, etc.). It is very reliable and people you share your screen with see exactly what you see in your browser. Why even bother with another way then? Well, the permission popup is annoying, it’s very often confusing, and depending on the browser, you may be asked to further select your entire window (multi-screen?), a specific browser window, or just a tab. In addition, it may give the end user the impression that their privacy is a bit at threat.

For these reasons, the stealth way is usually preferred by most services, including Usersnap, but it is sometimes less reliable. Roughly, it consists of “redrawing” in a (offscreen) canvas the entire UI of the webpage, with custom code, adapting every element with the CSS, pasting images, and hoping that it looks similar to what the user sees. While many things can go wrong with such a crafty approach, I must say Usersnap does a pretty good job and this is the default method. Yet, in some niche cases, this approach cannot work, and high-performance rendering falls right into it!

Canvas, where are your pixels?

MapTiler SDK for JavaScript/TypeScript uses Maplibre GL JS and the WebGL API under the hood to provide a great-looking and high-performance rendering.

Without diving into deep technical details, let’s just say that JS code runs on the main computing chip (CPU) while WebGL to render graphics runs on the GPU, a special kind of chip that has its own memory and its own way of computing things that’s very specific for 3D and 2D data processing.

You may be familiar with the <canvas> element from HTML5, but what is less known by non-developers is that a canvas can be used in two very different ways:

• pixel values computed on CPU with canvas.getContext("2d")
• pixel values computed on GPU with canvas.getContext("webgl2")

In the second case (WebGL), the pixel values (aka rendering buffer) are stored only in the GPU’s memory and are not directly accessible to processes running on the CPU, such as JavaScript code. To access this data, we must explicitly request a copy of the rendering buffer to be transferred to CPU-accessible memory. This process is called a GPU-to-CPU readback, and both the GPU and CPU must be in specific states to make this possible. This is a question of whether the rendering buffer on the GPU is cleared to leave room for the next frame to render, and also about when it is cleared.

The trick

In the previous section, we have covered why using the generic canvas method .toDataURL() is not enough to make the pixel values accessible to the CPU, so how do we do that?

There are two ways:

1. with preserveDrawinBuffer set to true in the Map constructor. This option propagates to the WebGL creation context and allows the GPU to keep the previous frame in memory, not clearing it until the very last moment when the next frame needs some room to render. This is by far the easiest way and it guarantees that calling .toDataURL() on the map canvas will always yield a valid image. Unfortunately, this has a heavy cost on performance so it’s not the method we will explore here.
2. explicitly create the conditions to make the rendering buffer available, and grab a frame while we can. This is what we want!

With MapTiler SDK, we can create such conditions by doing this:

// Initialize the map const map = new maptilersdk.Map({  container: "map-container", }); // Explicitely triggering a redraw of the map map.redraw(); // Waiting for the next moment, just after redraw,  // when the GPU will not be computing a new frame map.once("idle", () => {  // Grab the frame now!  const imgBase64 = map.getCanvas().toDataURL(); });

The idle event happens when the end-user has stopped interacting with the map and there is no new frame to compute. During a short period (that we do not control), the rendering buffer on the GPU is not cleared yet and we can perform a GPU-to-CPU readback, even with preserveDrawingBuffer being false.

Building upon this

Now that we’ve covered the foundations of the issue and found a generic workaround to still grab a frame, we will see how to adapt this to be compatible with feedback services such as Usersnap.

One thing to know is that Usersnap will call the function .toDataURL() on the canvas element that hosts the map and that we don’t control exactly when. This may sound like a detail, but it will actually lead us to walk on the thin ice of sketchy code. Let’s call that a hack.

As we have seen, the idle event only occurs when there is nothing else to compute or render. To some extent, since the user is not interacting with the map at this moment, we can slip in some tiny extra computing and the user will not notice it.

The hack consists of two things:

• plugging in a routine that grabs a frame at every idle event
• make sure the canvas method .toDataURL() returns the data corresponding to the latest idle event rather than the data at the moment of the call

Here is one way of doing it:

// This will contain the frame data, // and will be replaced at every idle event let lastIdleDataURL = ""; // We keep the original .toDataURL method because we still // need to call it! // (dont't forget to bind it) const originalToDataURL = mapCanvas.toDataURL.bind(mapCanvas); // Replacing the original .toDataURL() of the map canvas with a fake one that // returns the latest idle frame data. // The fake version is now the one being called by Usersnap. mapCanvas.toDataURL = function(type, encoderOptions) {  return lastIdleDataURL; } // When the map is idle, we get the dataURL to be stored for later. // Note how we are using the "originalToDataURL" function,  // since it's the one that does the actual frame grabbing map.on("idle", () => {  lastIdleDataURL = originalToDataURL(); });

Note: it’s possible that some alternatives to Usersnap use .toBlob() [doc] instead of .toDataURL() [doc], as this is a popular way to grab binary content of a canvas element, directly encoded and compressed as PNG or JPEG image. The main difference between these two methods is that .toBlob() relies on an asynchronous callback logic to yield the result while .toDataURL() synchronously returns the result.

A couple of illustrations

Here is what my basic web page looks like:

Without any hack, we click on the feedback button:

And sadly, this is how the screenshot shows up in the Usersnap dashboard:

But with our hack, Usersnap gets the map:

Despite this approach being somewhat hacky, especially the part where we replace the standard .toDataURL() with a custom one, it’s still a better alternative than preserving the drawing buffer, performance-wise. If you are in full control of your web app, you can add this logic at the application level. Bear in mind that this should not be added to a library that could be used by multiple apps without the developer being aware of it, since their app may require the canvas object to behave in the standard way.

Performance-wise, calling the built-in .toDataURL() requires from 50 to 100 ms to fetch the image buffer from the GPU on an Apple Macbook Air M2, so it’s not something to use lightly. Yet, it’s acceptable to place this inside an idle callback due to the inherent nature of the event.