Roopesh

How RDS Optimized Writes doubles MySQL write throughput

Unknown — Sun, 28 Jun 2026 00:00:00 +0000

AWS markets RDS Optimized Writes as a free toggle that delivers “up to two times higher write transaction throughput” on RDS for MySQL and MariaDB. Same instance class, same engine, same workload. Flip a parameter and write throughput can roughly double, at no additional charge.

That is a tall claim for what looks like a configuration knob. Where does the 2x come from? The short answer: AWS is letting MySQL skip a long-standing safety mechanism called the InnoDB doublewrite buffer, and the underlying hardware now provides the guarantee the doublewrite buffer was protecting against.

The rest of this post is the long answer. It walks through what torn pages are, why MySQL has a doublewrite buffer in the first place, why removing it can be worth up to 2x on write-bound workloads, and what specifically changed at the hardware layer that makes this safe.

The problem: torn pages#

A 16 KiB InnoDB page is written as four 4 KiB sector writes. Lose power partway through and you get a page that's half-new, half-old. That is a torn page.

InnoDB stores all of its data in 16 KiB pages by default. Disks and filesystems, however, work in smaller units, typically 4 KiB sectors. When MySQL writes a 16 KiB page to disk, the operating system breaks it into four 4 KiB sector writes underneath. If something goes wrong partway through (a power outage, a kernel panic, a disk failure), the page on disk ends up with a mix of new and old bytes. This is called a torn page or partial page write.

A torn page is unrecoverable from MySQL’s redo log alone. The redo log records changes to pages (for example, “in page X, at offset Y, set the value to Z”), not the full page contents. Replaying those changes assumes the page itself is intact to begin with. If the page is half-new and half-old, the redo log has nothing clean to apply against, and recovery fails.

Databases solve this two ways. PostgreSQL writes full page copies into its write-ahead log on first modify after a checkpoint (the full_page_writes setting). MySQL takes a different route: it keeps a redundant copy of every dirty page in a dedicated on-disk area called the doublewrite buffer. The trade-offs differ; the goal of surviving torn pages is the same. Percona has a great side-by-side.

MySQL’s solution: the doublewrite buffer#

The doublewrite buffer write path: each dirty page is written to a reserved on-disk area first, fsynced, then written to its tablespace location, fsynced again.

The official MySQL definition sums it up:

The doublewrite buffer is a storage area where InnoDB writes pages flushed from the buffer pool before writing the pages to their proper positions in the InnoDB data files. If there is an operating system, storage subsystem, or unexpected mysqld process exit in the middle of a page write, InnoDB can find a good copy of the page from the doublewrite buffer during crash recovery.

Every dirty page that InnoDB flushes goes through four steps: write the page into the doublewrite area on disk, fsync() the doublewrite area, write the page to its actual tablespace location, fsync() the tablespace.

On crash, InnoDB compares both copies during recovery. If the tablespace copy is torn but the doublewrite copy is intact, InnoDB rewrites the page from the doublewrite copy. If the tablespace copy is intact, the doublewrite copy is discarded. Percona’s explainer on the recovery mechanic goes deeper.

Despite the name, the latency overhead on a well-tuned MySQL is not 2x. The MySQL manual is explicit:

Although data is written twice, the doublewrite buffer does not require twice as much I/O overhead or twice as many I/O operations. Data is written to the doublewrite buffer in a large sequential chunk, with a single fsync() call to the operating system.

On a healthy system, the steady-state cost is roughly 5–10%. MySQL 8.0.20 also made the doublewrite path scale better by moving the buffer out of the system tablespace (ibdata1) and into dedicated files like #ib_16384_0.dblwr, with one set per buffer pool instance. That removed a long-standing concurrency bottleneck.

The doublewrite buffer is controlled by a single variable, on by default:

SHOW VARIABLES LIKE 'innodb_doublewrite';
-- +--------------------+-------+
-- | Variable_name      | Value |
-- +--------------------+-------+
-- | innodb_doublewrite | ON    |   ← stock MySQL
-- +--------------------+-------+

Since MySQL 8.0.30, the parameter accepts a richer set of values: ON (the default, equivalent to DETECT_AND_RECOVER), DETECT_ONLY (writes metadata, doesn’t recover full pages), and OFF.

Why “2x”, then? The write amplification angle#

There is a contradiction lurking here: the MySQL manual says the doublewrite buffer adds modest I/O overhead, yet AWS markets removing it as a 2x win. Both are true; they are talking about different things.

The “modest overhead” framing measures latency on a system with I/O headroom. The buffer’s writes are sequential and batched, so they amortise into a small per-transaction cost.

The “up to 2x” framing measures throughput on a system that’s running out of I/O headroom. The doublewrite buffer doubles the volume of page data hitting durable storage; every dirty page lands on disk twice. On write-bound workloads, the binding constraint isn’t latency, it’s bytes-per-second to storage. Halve the bytes and you can serve roughly twice the writes before saturating the volume.

That is exactly the regime AWS’s benchmark targets. The AWS Database Blog deep dive describes a sysbench write-only workload on a db.r6g.8xlarge with 50,000 provisioned IOPS in a Multi-AZ setup, deliberately write-bound. Mixed or read-heavy workloads see far less benefit because they were never bottlenecked on the second write to begin with.

Percona has separately measured how badly a misconfigured doublewrite path can hurt. They show up to 55% write IOPS reduction on write-bound loads when the legacy single-buffer design becomes a contention point. The fix in stock MySQL is tuning innodb_doublewrite_pages upward. AWS’s fix is to remove the buffer entirely.

When the doublewrite buffer is redundant#

If the doublewrite buffer exists to protect against torn pages, then any storage layer that already guarantees atomic 16 KiB writes makes the buffer redundant. This is not a new idea, and MySQL has long known how to skip the buffer when it is safe to:

Fusion-io with NVMFS. The MySQL manual says it directly: “If the doublewrite buffer is located on a Fusion-io device that supports atomic writes, the doublewrite buffer is automatically disabled and data file writes are performed using Fusion-io atomic writes instead.”
ZFS with recordsize=16K. ZFS’s copy-on-write semantics mean a 16 KiB write either completes to a new block or never publishes. There is no in-place overwrite to be interrupted. OpenZFS tuning docs recommend recordsize=16K precisely for InnoDB.
MariaDB’s innodb_use_atomic_writes. Since 10.2, MariaDB auto-detects compatible hardware at startup and disables the doublewrite buffer when atomic writes are available (MariaDB KB).

The pattern is consistent: when the storage stack can atomically commit at least an InnoDB page’s worth of data, the application-level safety net becomes redundant. The atomicity moves from software to hardware, and the doublewrite traffic disappears.

Enter AWS Nitro#

With Optimized Writes on, the doublewrite step disappears entirely. The page travels straight from the buffer pool to its final location in one atomic 16 KiB write.

AWS Nitro is the hardware platform underneath modern EC2 (and therefore RDS) instances. It combines dedicated hardware, lightweight firmware, and a stripped-down hypervisor that handles networking, storage, and security functions outside the main host CPU.

For Optimized Writes, the relevant property is that the Nitro storage path guarantees atomic 16 KiB writes. From the AWS RDS documentation:

These databases run on DB instance classes that use the AWS Nitro System. Because of the hardware configuration in these systems, the database can write 16-KiB pages directly to data files reliably and durably in one step. The AWS Nitro System makes RDS Optimized Writes possible.

When Optimized Writes is on, RDS sets the underlying MySQL variable innodb_doublewrite to FALSE (0). That is the same knob you would flip locally, just with hardware that backs the safety guarantee. ACID is preserved; atomicity simply moves from the doublewrite buffer to the Nitro layer.

The result: every dirty page goes to durable storage once instead of twice. On the same write-bound workload that saturates the doublewrite path, that frees up roughly half the write bandwidth, which is where AWS’s up-to-2x throughput claim comes from.

You can verify Optimized Writes is active on your instance the same way you would verify the doublewrite buffer locally:

SHOW VARIABLES LIKE 'innodb_doublewrite';
-- +--------------------+-------+
-- | Variable_name      | Value |
-- +--------------------+-------+
-- | innodb_doublewrite | OFF   |   ← Optimized Writes active
-- +--------------------+-------+

Turning it on in practice#

Optimized Writes is controlled by a single RDS parameter:

rds.optimized_writes = AUTO   # Turn on when version × instance class supports it (default)
rds.optimized_writes = OFF    # Force off; falls back to the doublewrite buffer

For new instances, the feature is on by default on any combination that supports it:

MySQL: version 8.0.30 and later (8.0.x and 8.4). Supported instance classes cover the modern M and R families: db.m5/m6i/m6g/m6gd/m7g/m7i/m8g/m8gd, db.r5/r5b/r5d/r6g/r6gd/r6i/r7g/r7i/r8g/r8gd, plus db.x2idn/db.x2iedn. The docs page has the canonical list; AWS adds new instance families over time.
MariaDB: 10.6.10+, 10.11.4+, 11.4.3+, or 11.8+. Same shape of instance class coverage, currently without the 8th-generation Graviton families (MariaDB-specific docs).
No additional charge.

Existing instances created before the feature launched (November 27, 2022 for MySQL, March 7, 2023 for MariaDB) have an incompatible underlying file system layout and cannot be flipped in place. The migration path is an RDS Blue/Green Deployment with the “Enable Optimized Writes on green database” and “Upgrade storage file system configuration” options ticked. Cut over once the green environment is in sync.

The catches#

A few things worth keeping in mind:

“Up to 2x” is a ceiling, not a floor. AWS says it directly: “the amount of benefit that can be achieved depends on the type of workload.” Read-heavy or mixed workloads will see a fraction of the gain.
RDS only, not Aurora. Aurora has its own purpose-built storage layer that doesn’t use the InnoDB doublewrite buffer at all, so there’s nothing to optimise here.
No PostgreSQL equivalent. Postgres handles torn pages via full-page writes in WAL, which is a different architecture entirely. RDS for PostgreSQL has a separate feature called Optimized Reads, which is unrelated.
Existing instances need migration. The on-disk file system layout differs from the pre-feature state, so a Blue/Green deployment is the only path forward.
Snapshot restore is constrained. Optimized Writes can only restore into an instance if the source snapshot was created from one that already supported it.

The doublewrite buffer was a clever software workaround for a storage stack that could not promise atomic 16 KiB writes. AWS Nitro can promise that, end-to-end. So MySQL no longer needs the workaround, and write throughput approximately doubles on write-bound workloads, for free. The buffer still earns its keep on stock MySQL; on RDS, the hardware now does the job. 🍻

Reddify

Unknown — Fri, 06 Dec 2024 00:00:00 +0000

Reddify is a simple web utility to create a Spotify playlist out of a Reddit post’s comments.

It features a minimal UI built without any javascript frameworks.

Why I built Reddify?#

Quite often, on subreddits like r/music and r/spotify, there are huge threads that are filled with comments featuring great music suggestions. I used to listen to a few of the top comments and forget about the rest. This felt like leaving a treasure trove of new music experiences behind.

While most of the days I stick to listening familiar tunes, there are specific days when I am very excited about discovering new songs and having a playlist of these suggestions from Reddit looked like a wonderful way to go about it.

How does Reddify work?#

Once a Reddit thread is submitted, Reddify extracts the comments out of the post and uses the Spotify Search API to narrow down the song and picks the top result. This idea was born out of my own experiences of finding Spotify’s Search pretty spot on and worked as well when I tried feeding it Reddit comments.

How to use Reddify?#

Reddify is archived and is no longer hosted or actively developed. The source code remains available on GitHub for anyone who wants to run it themselves — clone the repo and follow the instructions in the README to get it up and running locally.

Network Wide Ad Blocking With Pi-hole

Unknown — Sat, 02 Mar 2024 00:00:00 +0000

Today, many of us rely on browser extensions to block annoying ads and tracking while browsing on our computers. They work well for desktop browsers, but what about our mobiles, tablets, smart TVs, and game consoles? They’re left vulnerable to ads because these extensions don’t work on them. We’ve come to accept ads as a necessary evil, but it doesn’t have to be that way.

With a simple setup using a Raspberry Pi and a tool called Pi-hole, you can block ads and tracking across all devices on your home network. It’s not limited to a Raspberry Pi; you can run it on any machine using Docker, but the Pi is preferred for its low power consumption as it needs to run 24x7.

My frustration with ads reached its peak while watching a Dota 2 tournament on the Twitch app in my Xbox Series X, where a browser-level or device-level solution for ads do not exist. Every time I switched channels, the same unskippable ad played, and I’d had enough. I remembered hearing about Pi-hole and decided to give it a try with my idle Raspberry Pi 400.

Installing Pi-hole was straightforward using their one-step automated install guide here.

Pi-hole’s capabilities can be extended by adding more domain blocklists from Firebog, a site that provides lists for blocking ads, tracking, crypto mining, and more here, which I did.

After installation, I configured Pi-hole as the DNS server for all devices on my home network through my router’s admin page. I also reserved a static IP address for the Raspberry Pi to avoid network interruptions.

The result? No more ads on Twitch, and when I switched from mobile data to WiFi, Pi-hole blocked ads on websites like CNN.

Pi-hole works by checking each DNS query against its blocklist, allowing or blocking queries accordingly. It’s effective because many sites use separate domains for ads and tracking. However, ads served from the same domains as regular content, like on YouTube, can’t be blocked by Pi-hole.

Notice that over one out of ten DNS queries are getting blocked by Pi-hole.

Now, what about blocking ads and trackers when you’re away from home? You can set up a VPN server on the same Raspberry Pi to enjoy ad-free browsing anywhere. I tried setting up OpenVPN, but my ISP’s Carrier-Grade NAT (CGNAT) blocked port forwarding. After some research, I found Tailscale, a beautiful and easy solution for connecting to Pi-hole remotely, even behind different NATs. You can read in detail about their NAT Traversal solution here.

Tailscale allows devices to connect to Pi-hole from anywhere, and setting it up took just a few minutes with their guide. With my mobile device connected to Tailscale, I was now able to view CNN’s site without ads even when I was using my mobile data.

No more ads or tracking, anywhere! 🎉

Automatically Scaling Down Lambda Provisioned Concurrency

Unknown — Fri, 01 Mar 2024 00:00:00 +0000

If you are using AWS Lambda to serve real-time traffic and your Lambda initialization times are high, minimizing response time becomes crucial. One option to achieve this is by utilizing provisioned concurrency. Provisioned Concurrency refers to the number of pre-initialized execution environments allocated to your Lambda function.

Maintaining a constant provisioned concurrency capacity of 200 for a 512 MB Lambda for a month can cost approximately $1,116 USD! This cost is in addition to the pricing for requests and duration. AWS offers the option to scale the provisioned concurrency of Lambda based on schedule and demand. We implemented provisioned concurrency scaling for our Lambdas based on the ProvisionedConcurrencyUtilization metric.

Note: At the time of writing this, provisioned concurrency scaling is not available from the AWS Management Console but can be configured using APIs, SDKs, or CLIs.

One of our Lambdas has an initialization time of over 40 seconds as it loads several ML models as pickled Python objects. Without a pre-initialized Lambda instance available, any request coming through the API Gateway would fail due to the API Gateway timeout of 29 seconds.

We configured the provisioned concurrency capacity to scale from a minimum of 10 to a maximum of 200. This ensures that there are at least 10 pre-initialized Lambda instances ready to serve requests at all times, with the capacity scaling up to 200 pre-initialized instances when necessary. During a quick load test, the provisioned concurrency scaled up effectively, maintaining the Lambda response time at less than 100 ms throughout. Even with 30,000 requests per minute, only about 100 provisioned concurrency capacity was required.

Everything seemed to be working well, but there was a catch: While it was reassuring to see the provisioned concurrency scale up to 100 to handle traffic without errors, it did not automatically scale down afterward. This was concerning, and we only noticed it 48 hours later, resulting in unnecessary costs during periods of low or no traffic. To address this, we quickly ran an AWS CLI command in a loop for all Lambdas to bring down their provisioned concurrency to the minimum value:

aws lambda put-provisioned-concurrency-config \
    --function-name generic-function-name \
    --qualifier latest \
    --provisioned-concurrent-executions 10

Why didn’t the provisioned concurrency of the Lambdas automatically scale down when there were no requests? Provisioned concurrency scaling works with CloudWatch Alarms managed by AWS when we create the scaling policy. Upon reviewing the alarms, we found that the action to scale down was not triggered because the alarm for low provisioned concurrency utilization never activated due to “insufficient data.” When there are no requests and provisioned concurrency is not being utilized, the alarms transition to the “insufficient data” stat and alarm actions do not get triggered in this state. There is an option in CloudWatch Alarms to set how to treat missing data. Unfortunately, this configuration option is not available when creating an autoscaling policy, only when creating an alarm directly. Also, editing the alarm directly is not recommended by AWS for alarms created with auto-scaling target tracking policies.

After extensive research, we found limited discussion on this issue online. Broadening our search, we looked for cases where CloudWatch alarms did not trigger due to insufficient data when there were no metrics to report. We came across a Stack Overflow answer suggesting the use of the FILL function to create a new metric that returns 0 whenever the original metric returns “insufficient data.” This idea seemed promising.

We created the following Target Tracking policy JSON to implement this solution:

// target-tracking.json
{
    "CustomizedMetricSpecification": {
        "Metrics": [
            {
                "Label": "ProvisionedConcurrencyUtilization",
                "Id": "m1",
                "MetricStat": {
                    "Metric": {
                        "MetricName": "ProvisionedConcurrencyUtilization",
                        "Namespace": "AWS/Lambda",
                        "Dimensions": [
                            {
                                "Name": "FunctionName",
                                "Value": "generic-function-name"
                            },
                            {
                                "Name": "Resource",
                                "Value": "generic-function-name:latest"
                            }
                        ]
                    },
                    "Stat": "Maximum"
                },
                "ReturnData": false
            },
            {
                "Label": "ProvisionedConcurrencyUtilization where Missing Data = 0",
                "Id": "e1",
                "Expression": "FILL(m1, 0)",
                "ReturnData": true
            }
        ]
    },
    "TargetValue": 0.7,
    "ScaleOutCooldown": 60,
    "ScaleInCooldown": 60,
    "DisableScaleIn": false
}

We tested the Lambda function using the following put-scaling-policy AWS CLI command:

aws application-autoscaling put-scaling-policy --service-namespace lambda \
    --scalable-dimension lambda:function:ProvisionedConcurrency \
    --resource-id generic-function-name \
    --policy-name generic-target-tracking-scaling-policy --policy-type TargetTrackingScaling \
    --target-tracking-scaling-policy-configuration file://target-tracking.json

Upon reviewing AWS CloudWatch Alarms and filtering for generic-function-name, we were pleased to see the new metric ProvisionedConcurrencyUtilization where Missing Data = 0 populating 0 even when ProvisionedConcurrencyUtilization returned “insufficient data.” Running another load test, this time waiting for no requests, the alarm triggered as expected, gracefully bringing down the provisioned concurrency capacity.

With the solution confirmed to be working, we updated our Terraform Lambda module to apply this change to all Lambdas. You can find a snippet of the Terraform resource aws_appautoscaling_policy here.

Now, even when there are no incoming requests, the CloudWatch alarm for provisioned concurrency autoscaling triggers using the new metric, automatically scaling down and leading to lower cloud bills! 🍻

How Sherlock knows if you are on Facebook

Unknown — Mon, 12 Sep 2022 00:00:00 +0000

Sherlock is a CLI tool that can be used to find usernames across many social networks. In this post, I share how Sherlock was able to overcome a couple of hurdles in reliably finding if a username existed on Facebook. Sherlock works by having a JSON file as the source where a large collection of sites are listed with a few attributes.

A sample entry from the file looks like this:

"Reddit": {
    "errorMsg": "Sorry, nobody on Reddit goes by that name.",
    "errorType": "message",
    ...
    "url": "https://www.reddit.com/user/{}",
    "urlMain": "https://www.reddit.com/",
    ...
  }

Each site has its own way of handling usernames that do not exist. To know if a particular username is on the site, Sherlock sends a request to the URL defined in the url or the urlProbe field and evaluates the response depending on the type of error defined in the errorType field. The errorType field could be one of status_code - HTTP Response Error Codes, message - HTTP Response Body that says there is no user by that username or response_url - Redirecting to a different page if the username is not found, depending on the method appropriate for consistently finding users without false positives.

Sherlock was using the status_code error type to figure out if the username existed on Facebook.

"Facebook": {
    "errorType": "status_code",
    ...
    "url": "https://www.facebook.com/{}",
    "urlMain": "https://www.facebook.com/",
    ...
  }

One user raised a GitHub issue saying Sherlock could not find his username on Facebook. I vividly remembered Sherlock being able to find my username on Facebook.

Probing into this mystery, I discovered there was a privacy setting on Facebook that disallowed search engines outside of Facebook to link to one’s profile.

Only when the search engines were allowed to link to one’s facebook profile at facebook.com/{username}, the requests to that profile returned 200 OK otherwise even when a user with that username existed on facebook the response was 404 NOT FOUND. This is why the status code approach was inconsistent.

One of the trivial ways to bypass this issue would be to login to facebook and then check for the username. However, for Sherlock, authenticating was not an option!

I wanted to check if there was a profile path Sherlock can reliably use without authenticating that would also bypass the privacy option. With turning on and off the privacy option on my facebook profile, I started testing various paths like /about, /images, /photo and their responses.

Few minutes into the search I stumbled upon /videos and I was able to see my expectations come alive. facebook.com/{username}/videos showed the login page when the username existed and only responded with This page isn't available when the username was not yet taken on Facebook. As this path did not care for the search engine privacy option as well, it was perfect!

The site resource file was updated to check for an error message & use the /videos path and voila! This now allows Sherlock to check if a username is on facebook without authentication and irrespective of the search engine privacy option they’ve opted to go with.

"Facebook": {
    "errorMsg": "This page isn't available",
    "errorType": "message",
    ...
    "urlProbe": "https://www.facebook.com/{}/videos/",
    ...
  }

A hacky and elementary fix to help Sherlock find your facebook ID.