We use cookies and other tracking technologies to improve your browsing experience on our site, analyze site traffic, and understand where our audience is coming from. To find out more, please read our privacy policy.

By choosing 'I Accept', you consent to our use of cookies and other tracking technologies.

We use cookies and other tracking technologies to improve your browsing experience on our site, analyze site traffic, and understand where our audience is coming from. To find out more, please read our privacy policy.

By choosing 'I Accept', you consent to our use of cookies and other tracking technologies. Less

We use cookies and other tracking technologies... More

Login or register
to publish this job!

Login or register
to save this job!

Login or register
to save interesting jobs!

Login or register
to get access to all your job applications!

Login or register to start contributing with an article!

Login or register
to see more jobs from this company!

Login or register
to boost this post!

Show some love to the author of this blog by giving their post some rocket fuel 🚀.

Login or register to search for your ideal job!

Login or register to start working on this issue!

Login or register
to save articles!

Login to see the application

Engineers who find a new job through WorksHub average a 15% increase in salary 🚀

You will be redirected back to this page right after signin

Blog hero image

Troubleshooting Tomcat in a memory-lean environment

Waqqas Dadabhoy 20 January, 2021 | 4 min read

onemusicapi.com runs on AWS Elastic Beanstalk with a Tomcat 8.5 platform, using t3.micro instances. t3.micro instances have 1GB of RAM, and Elastic Beanstalk's monitoring puts instances in a Warning state if memory use exceeds 90% of available memory. We found that about a day or two after an EC2 instance was brought online, it would go into Warning state, due to memory use, even if the service wasn't actually affected.

TL;DR: We found Tomcat was using more memory than expected, and the following helped:

  • Reducing -Xmx
  • Limiting the code cache size
  • Reducing the number of threads

Step one: measuring heap

The first step was to reduce the heap size (-Xmx) and monitor memory use. During this step, we realized that we should be able to reproduce the problem by setting -Xms & -Xmx to the same value, so that the heap allocation wouldn't change. However, the memory issue continued the same way it had - a day or two after being created, the instance would go into Warning state.

Step two: beyond the heap

The next step was to check what was using memory on the system. Using the command:

ps -eo size,pid,user,command --sort -size | head

... we noticed that Java process was using several hundred megabytes of memory over what -Xmx had been configured to. We were able to figure out how much of the memory was going to:

  • Thread stacks.
  • JVM overhead (by comparing it to the memory of a freshly started jshell).

However, this still left about 150MB of memory unexplained.

We assumed that this might be due to Tomcat using native memory for its internal buffers, so we tried changing every setting in that might seem like it might affect memory use.

Join our newsletter
Join over 111,000 others and get access to exclusive content, job opportunities and more!

Step three: measuring native memory

The next step was to enable Native Memory Tracking, and get "before & after" snapshots of memory use. This immediately showed us that the code cache was using over 64MB of memory. We then set the initial & maximum code cache size to get more predictable memory use. Here is the Native Memory Tracking output:

Total: reserved=1716451KB, committed=378495KB                                  |  Total: reserved=1753206KB, committed=441570KB
  • Java Heap (reserved=235520KB, committed=235520KB) - Java Heap (reserved=235520KB, committed=235520KB) (mmap: reserved=235520KB, committed=235520KB) (mmap: reserved=235520KB, committed=235520KB) | - Class (reserved=1120844KB, committed=82468KB) | - Class (reserved=1136217KB, committed=101681KB) (classes #11852) | (classes #13397) ( instance classes #11125, array classes #727) | ( instance classes #12600, array classes #797) (malloc=2636KB #35841) | (malloc=3673KB #55415) (mmap: reserved=1118208KB, committed=79832KB) | (mmap: reserved=1132544KB, committed=98008KB) ( Metadata: ) ( Metadata: ) ( reserved=69632KB, committed=67840KB) | ( reserved=83968KB, committed=83200KB) ( used=66088KB) | ( used=80220KB) ( free=1752KB) | ( free=2980KB) ( waste=0KB =0.00%) ( waste=0KB =0.00%) ( Class space:) ( Class space:) ( reserved=1048576KB, committed=11992KB) | ( reserved=1048576KB, committed=14808KB) ( used=10314KB) | ( used=12109KB) ( free=1678KB) | ( free=2699KB) ( waste=0KB =0.00%) ( waste=0KB =0.00%) | - Thread (reserved=82615KB, committed=9175KB) | - Thread (reserved=102238KB, committed=12062KB) (thread #80) | (thread #99) (stack: reserved=82232KB, committed=8792KB) | (stack: reserved=101764KB, committed=11588KB) (malloc=289KB #482) | (malloc=358KB #596) (arena=94KB #158) | (arena=116KB #196) | - Code (reserved=249146KB, committed=23006KB) | - Code (reserved=251533KB, committed=64609KB) (malloc=1458KB #8342) | (malloc=3845KB #17739) (mmap: reserved=247688KB, committed=21548KB) | (mmap: reserved=247688KB, committed=60764KB) | - GC (reserved=1211KB, committed=1211KB) | - GC (reserved=1323KB, committed=1323KB) (malloc=439KB #3369) | (malloc=551KB #4476) (mmap: reserved=772KB, committed=772KB) (mmap: reserved=772KB, committed=772KB) | - Compiler (reserved=424KB, committed=424KB) | - Compiler (reserved=991KB, committed=991KB) (malloc=294KB #1015) | (malloc=861KB #2304) (arena=131KB #5) (arena=131KB #5) | - Internal (reserved=938KB, committed=938KB) | - Internal (reserved=1295KB, committed=1295KB) (malloc=906KB #1817) | (malloc=1263KB #2206) (mmap: reserved=32KB, committed=32KB) (mmap: reserved=32KB, committed=32KB) | - Other (reserved=134KB, committed=134KB) | - Other (reserved=170KB, committed=170KB) (malloc=134KB #22) | (malloc=170KB #40) | - Symbol (reserved=17441KB, committed=17441KB) | - Symbol (reserved=19006KB, committed=19006KB) (malloc=14684KB #157347) | (malloc=16153KB #175495) (arena=2757KB #1) | (arena=2853KB #1) | - Native Memory Tracking (reserved=3356KB, committed=3356KB) | - Native Memory Tracking (reserved=4162KB, committed=4162KB) (malloc=19KB #243) | (malloc=28KB #372) (tracking overhead=3338KB) | (tracking overhead=4135KB) | - Arena Chunk (reserved=4368KB, committed=4368KB) | - Arena Chunk (reserved=180KB, committed=180KB) (malloc=4368KB) | (malloc=180KB) | - Logging (reserved=4KB, committed=4KB) - Logging (reserved=4KB, committed=4KB) (malloc=4KB #191) (malloc=4KB #191) | - Arguments (reserved=19KB, committed=19KB) - Arguments (reserved=19KB, committed=19KB) (malloc=19KB #504) (malloc=19KB #504) | - Module (reserved=132KB, committed=132KB) | - Module (reserved=146KB, committed=146KB) (malloc=132KB #1597) | (malloc=146KB #1668) | - Synchronizer (reserved=290KB, committed=290KB) | - Synchronizer (reserved=393KB, committed=393KB) (malloc=290KB #2444) | (malloc=393KB #3310) | - Safepoint (reserved=8KB, committed=8KB) - Safepoint (reserved=8KB, committed=8KB) (mmap: reserved=8KB, committed=8KB) | (mmap: reserved=8KB, committed=8KB)
  • Step four: threads cost memory

    During the above investigation, we noticed that we had 146 threads running, which was excessive for our use-case. So we removed & combined thread pools where we could, to reduce the number of threads. We also found that our web framework was creating large thread pools, but we did not have control over them. This was one of the learning outcomes of this exercise ?? limiting the number of threads to conserve memory.

    One usually doesn't find articles on the Web about running Tomcat in memory-constrained environments - our searches all turned up results about multi-gigabyte heaps, where Tomcat's default settings are a much smaller fraction of the memory use. So it was an interesting experience to troubleshoot a situation where we didn't have any search results to use as a starting point.

    Originally published on www.onemusicapi.com

    Related Issues

    open-editions / corpus-joyce-ulysses-tei
    open-editions / corpus-joyce-ulysses-tei
    • Started
    • 0
    • 4
    • Intermediate
    • HTML
    open-editions / corpus-joyce-ulysses-tei
    open-editions / corpus-joyce-ulysses-tei
    • Started
    • 0
    • 3
    • Intermediate
    • HTML
    open-editions / corpus-joyce-ulysses-tei
    open-editions / corpus-joyce-ulysses-tei
    • Started
    • 0
    • 4
    • Intermediate
    • HTML
    open-editions / corpus-joyce-ulysses-tei
    open-editions / corpus-joyce-ulysses-tei
    • Started
    • 0
    • 1
    • Intermediate
    • HTML

    Get hired!

    Sign up now and apply for roles at companies that interest you.

    Engineers who find a new job through WorksHub average a 15% increase in salary.

    Start with GitHubStart with Stack OverflowStart with Email