DocumentDB Management

A lot of this page relies on a rather clumsy analogy explained in the introduction section - skip over at your peril!

Any work on this requires an AWS login - ask if you don't have! The AWS login page can be found here.

Intro to DocumentDB

DocumentDB is the Amazon hosting service that we use to serve the Mongo database (i.e. all the database objects you can see and edit on the admin site).

The best way to think of DocumentDB is as a doorway to a library; the library in question being the database values we have stored (all the relevant information users might want: competition results; entries; etc.).

At non-peak times, we have no issues with the number of people coming in and out of the door, but at busier times sometimes we have so many people wanting to visit the library that we simply cannot handle the number of requests for information

It should also be noted that uploading information (i.e. adding results) is more intensive than downloading it. (Much like it takes librarians a lot longer to put books out on the shelves than it takes you to pull a book you want off the shelf!)

It is for these reasons that at peak times, we need to 'widen the doorway', so to speak.

Instances of DocumentDB

We currently run one 'instance' of DocumentDB on AWS. An instance in this scenario is directly analogous to a doorway.

AWS has a large number of instance sizes to choose from, and it is managing this that allows us to scale for the number of users we have on a given competition day.

Note: It may be the case in future that we move to a different architecture of having multiple instances of DocumentDB; a double-width doorway can handle the same traffic as two single width doorways, but having two doorways means that if you are repairing one, people can still use the other! However, this is not the case at the moment (June 2022).

How to Modify a DocumentDB Instance

If at any point you don't like these instructions, Amazon also have some for you here.

First, log in to AWS here.

You can search for DocumentDB using the search bar. When there, the 'Clusters' option is where you can make changes.

Simply tick the checkbox next to the instance you want to modify. Note: You can only modify an individual instance, rather than a cluster. You should then see the 'Actions' button become available.

If you click on 'Actions', you will see 'Modify', 'Reboot', and 'Delete'.

Note: It is possible that the 'Reboot' option under 'Actions' may become relevant in time to deal with memory leak, detailed below.

After clicking 'Modify', you will see a section headed 'Instance specifications'. In there, you want to change the instance class. Each instance has a name, and a subheading detailing the amount of memory and number of CPUs it has.

If upgrading in a hurry, simply choose one either twice or four times the size. If trying to plan for a competition period, see the section below on instance sizes.

You are likely to have up to 10 minutes of downtime as AWS moves to the new instance - this is expected, but it is good to notify key users if you are doing it during a competition period.

You can set the time in which you want the instance change to take place in the 'Maintenance' section. If you want to apply the change instantly, when you click through to the next page you can select 'Apply immediately'.

Finally, click the 'Modify Instance' button, and wait - monitor the graphs, and everything should be back up and improved very shortly! You can revisit the clusters page to check whether the instance is still being modified, or whether you are back live.

Instance Sizes and Practical Applications

db.r6g.large (16GB mem, 2vCPUs) - This is where we sit most of the time.

db.r6g.xlarge (32GB mem, 4vCPUs) - This is a good idea for busy periods with multiple competitions

db.r6g.2xlarge (64GB mem, 8vCPUs) - This is a good idea for highly intensive periods with demanding processes (i.e. the NAL or other leagues, or multiple national championships)

It is possible to scale well above both of these.

You can use this page to look at different instance levels and costs.

Memory Leak

One thing that has been noted in May and June of 2022 is the tendency of Document DB to 'leak' memory. While the number of requests on the system can stay constant across a day, the CPU usage tends to rise until the instance becomes throttled and performance drops off rapidly.

To reduce the impact of this, we have adopted some fairly aggressive caching on key pages as of late June 2022, but we have no long-term solutions yet, and have not tracked down any major leaks.