WilliamLam.com

  • About
    • About
    • Privacy
  • VMware Cloud
  • Tanzu
    • Application Modernization
    • Tanzu services
    • Tanzu Community Edition
    • Tanzu Kubernetes Grid
    • vSphere with Tanzu
  • Home Lab
  • Nested Virtualization
  • Apple
You are here: Home / vRealize Suite / How to purge all logs in vRealize Log Insight?

How to purge all logs in vRealize Log Insight?

02.01.2017 by William Lam // 17 Comments

I had just deployed a new vRealize Log Insight (vRLI) 4.0 instance in my home lab environment to investigate a behavior that I was seeing with another product, non-vRLI related. Due to the nature of the work, I needed to have a pristine vRLI environment each time to study the results. I had already forwarded some logs into vRLI and rather than deploying another instance or re-deploy the current instance, what I really wanted to be able to do is to just wipe all the logs in vRLI but did not see an option within the UI. I also could have used VM snapshots, but was hoping there was a cleaner solution that vRLI provided out of the box.

The next place I looked immediately after was Mr. Log Insight's site aka Steve Flanders blog but there was nothing there about this other than archiving. After a few Google searches, I came across this exact same question on the vRLI Ideas site but sadly there was no solution and it was dated back in 2014. Though Steve makes a good point about just letting the logs rotate out automatically, in my case, this was not an option and I needed a pristine environment.

Being the curious one, I figured there has to be a way, even if it is not officially recommended nor supported. As you probably have guessed, I did find a way but I would caution that you read the disclaimer below before proceeding further. This was something I needed to do in my lab to test a few scenarios that was non-vRLI related, but I needed syslog target, so this is why I am using vRLI 🙂

Disclaimer: This is probably not officially supported nor recommended by VMware. Please use at your own risk. YOU WILL LOSE ALL YOUR LOGS

Step 1 - SSH to your vRLI instance and stop the Log Insight service by running the following command:

/etc/init.d/loginsight stop

Step 2 - Run the following command which will list all the buckets (where your logs are stored) and their associated IDs which we will need in next step:

/usr/lib/loginsight/application/sbin/bucket-index show


Step 3 - For each of the bucket IDs returned in Step 2, you will go ahead and run the delete operation and specify the bucket ID (you will be prompted to confirm deletion):

/usr/lib/loginsight/application/sbin/bucket-index delete [BUCKET-ID]


Step 4 - Once all the buckets have been deleted, you can now start the Log Insight service by running the following command:

/etc/init.d/loginsight start

Once vRLI has started back up, you can log back into the vRLI UI and you should have a pristine environment with no logs as shown in the screenshot below.


In case you are lazy to type all those commands manually or if you have a large number of buckets, I have also created a quick bash script that will automate the entire process (why not, right?). Simply copy/paste the script into a file called purge.sh and make sure it has executable permissions and then run it.

#!/bin/bash

/etc/init.d/loginsight stop

cat > /tmp/vrli-purge-answer << __ANSWER__
y

__ANSWER__

for bucket in $(/usr/lib/loginsight/application/sbin/bucket-index show | tac | awk '{split($0,a,"id="); split(a[2],b,","); print b[1]}')
do
   echo "Deleting bucket $bucket ..."
   /usr/lib/loginsight/application/sbin/bucket-index delete $bucket < /tmp/vrli-purge-answer
done

rm -f /tmp/vrli-purge-answer
/etc/init.d/loginsight start

Here is a screenshot of running the script to automatically purge all the logs from vRLI:


I suspect this is probably not a common vRLI request but if you ever need to wipe all your vRLI logs without needing to re-deploy, there is an option. Perhaps this is something the team could consider as a super duper advanced option? 🙂

More from my site

  • New SDDC Certificate Replacement Fling
  • Getting started with VMware Pivotal Container Service (PKS) Part 9: Logging
  • Getting started with VMware Pivotal Container Service (PKS) Part 8: Monitoring Tool Overview
  • Automating vRealize stack based on VVD using new vRealize Suite Lifecycle Management
  • Stage Only & Stage and Install buttons disabled when updating to vSphere 7.0 Update 2a 

Categories // vRealize Suite Tags // vRealize Log Insight

Comments

  1. Craig Spreha says

    08/03/2017 at 5:19 am

    Just the answer I needed. Was getting an alert that one of my LI nodes was having a space warning. I was confused by this as I believed LI to always keep it's spaced trimmed to 93% usage. I dug through my LI alert emails and found one that reported two buckets as corrupted and I assumed it left them to stagnate. I used your commands to purge those two buckets to bring the storage back to 93%.
    Thanks as always for your knowledge.

    Reply
  2. Tasos Fragopoulos says

    10/24/2017 at 1:53 am

    Dear William

    It worked with no problem in my case.
    Thanks a lot for sharing.

    Reply
  3. Paul Allan Harrington says

    11/08/2017 at 10:36 am

    Thanks William.

    This worked great. If you do a follow-up post, understanding how to relate bucket ids to agent/log source would likely be useful also.

    Thanks again.

    Reply
  4. Budi Supriatna says

    02/20/2018 at 6:43 pm

    Hi William,
    How create CLI for autmatically delete logs from vRLI, base on time frame
    for example each month or date ?

    Reply
  5. mr says

    04/10/2018 at 7:30 pm

    Perhaps this is something the team could consider as a super duper advanced option? ????

    Absolutely, we need this as an advacned option and or to configure at vami, to delete any buckets older than 30 days or so.

    Reply
    • mrde says

      04/30/2019 at 4:55 am

      In Loginsight 4.8 you can configure retention period to delete older than 30 days or so.

      Reply
  6. Virat Kamboj says

    04/11/2018 at 10:00 pm

    Same thing can be achieved through Winscp. You are going to find bucket under path (storage/core/loginsight/cidata/store)

    Reply
  7. Elad says

    05/31/2018 at 4:06 am

    exectly what i needed !!!
    thank!

    Reply
  8. jstreit2017jim Streit says

    09/24/2018 at 9:49 am

    This can also be used free up some space like when trying to upgrade to a newer version and you get a "not enough space on /storage/core". Same process, but select a couple of your oldest archives and delete those. Archives can be identified by "status=archived".

    Thanks William.

    Reply
  9. Paul says

    11/17/2018 at 8:02 am

    Hi William,

    Do you know if its possible to keep last week data only. I also have a lab for testing purpose so I don't want to erase everything but just everything older than a week like this I still have data to work on it without consuming disk space in my lab ?

    Thanks
    Paul

    Reply
  10. Rohit says

    04/29/2021 at 1:02 am

    Vrli 8.1, you can select retention period under Administration-> Management -> Partitions. Select the default partition 'edit' button and give the number of days for retention of data.

    Reply
  11. Luis Dominguez says

    04/29/2021 at 9:46 am

    HI William

    The script does not appear to work in LogInsight 8.3, when I run it, it does not prompt for the question or continue to delete buckets

    Reply
    • William Lam says

      05/06/2021 at 4:36 pm

      Hi Luis,

      I was also informed by Mike Foley about this issue and he was kind enough to figure out the solution which required a tweak to the awk statement. The article has been updated and Mike confirmed this allowed him to delete the buckets. Please give that a try

      Reply
  12. Action Jackson says

    08/18/2021 at 11:38 pm

    William I checked the article again and didn't see any new changes, the commands do not work for LI 8.3 directories don't seem to be the same and even the stop\start commands don't work for that version.

    are you able to update this article for 8.x?

    Reply
  13. Mark Evans says

    06/13/2022 at 10:02 pm

    Thanks for the inspiration! Looking at having a dedicated instance for bringing logs back out of NFS archive. Not sure about previous versions, but with 8.8 there is a "bucket-tools" utility at /usr/lib/loginsight/application/sbin in addition to the aforementioned "bucket-index". bucket-index gives you a bit of control as to what buckets to delete, bucket-tools is a bit less refined but can do the job. bucket-tools --totalCount then bucket-tools --delete oldBucketsCount={insert value from bucket-tools --totalCount} will also clean it out! There's some other neat functionality with it too.

    Reply
  14. Josh says

    09/29/2022 at 7:57 am

    I tried running this as-is and received a syntax error pointing to the comma after split(a[2],b,. This was caused by an extra double-quote following the semicolon in the first split statement. Remove the " and it works as expected.

    #!/bin/bash

    /etc/init.d/loginsight stop

    cat > /tmp/vrli-purge-answer << __ANSWER__
    y

    __ANSWER__

    for bucket in $(/usr/lib/loginsight/application/sbin/bucket-index show | tac | awk '{split($0,a,"id="); split(a[2],b,","); print b[1]}')
    do
    echo "Deleting bucket $bucket ..."
    /usr/lib/loginsight/application/sbin/bucket-index delete $bucket < /tmp/vrli-purge-answer
    done

    rm -f /tmp/vrli-purge-answer

    /etc/init.d/loginsight start

    Reply
    • William Lam says

      09/29/2022 at 8:51 am

      Thanks Josh! I’ve fixed snippet

      Reply

Thanks for the comment! Cancel reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Search

Author

William Lam is a Senior Staff Solution Architect working in the VMware Cloud team within the Cloud Infrastructure Business Group (CIBG) at VMware. He focuses on Cloud Native technologies, Automation, Integration and Operation for the VMware Cloud based Software Defined Datacenters (SDDC)

Connect

  • Email
  • GitHub
  • LinkedIn
  • RSS
  • Twitter
  • Vimeo

Recent

  • vSphere with Tanzu using Intel Arc GPU 01/26/2023
  • Quick Tip - Automating allowed and not allowed Datastores for use with vSphere Cluster Services (vCLS) 01/25/2023
  • ESXi with Intel Arc 750 / 770 GPU 01/24/2023
  • How to bootstrap vSAN Express Storage Architecture (ESA) on unsupported hardware? 01/19/2023
  • Automating Virtual Machine screenshots in vSphere 01/18/2023

Advertisment

Privacy & Cookies: This site uses cookies. By continuing to use this website, you agree to their use.
To find out more, including how to control cookies, see here: Cookie Policy

Copyright WilliamLam.com © 2023

 

Loading Comments...